fix: address Task 3 code review (bool_or, invariant tests, guards, docs)
Nine review items on the expenditure_concept = direct|total feature: - bool_and(is_aggregate) -> bool_or(is_aggregate) for aggregate_fallback: bool_and silently misreported $5,740,775,000 of aggregate-sourced IG dollars (AL state 2011) as aggregate_fallback = FALSE, because the dense wide-era data puts a $0 leaf row in the same group as the real aggregate row. bool_or is a no-op for Direct/Revenue (verified: 0 mismatched groups across both tables) and correct for the IG leg. - Added a year-disjointness invariant test for the four legacy aggregate/leaf IG pairs (M47/M94, M89/M91-93, L47/L94, L89/L91-93), scoped to the aggregate flag rather than bare code presence (M89/L89 continue past 2011 as independent, non-aggregate leaves). - Extended the real-SQL-text/synthetic-parquet harness in test-views.R to pin ig_long/ig_long_harmonized's predicates directly (aggregate rows retained, NULL harmonized_code coalesced, L-- excluded), rather than relying on one fixture row's incidental shape. - Added a test proving the .harmonization_view_files schema-v5 guard is necessary (not just incidental) against a corpus whose `long` genuinely lacks a harmonized_code column, and rewrote the misleading "v5-only parquet files" comment to name both real reasons a file is gated. - Fixed an NA-fragile subtype filter, extended the expected-view-list test, guarded .verb_spendrev() against total on a non-spending view_base, added a roxygen caveat against summing total across levels of government, and replaced an uncheckable corpus-wide SQL comment figure with a fixture-verifiable one. Full suite: 485/0/0 -> 503/0/0 (18 new expectations, zero pre-existing value changed).
This commit is contained in:
@@ -2,9 +2,12 @@
|
||||
-- than harmonized_code alone: aggregate rows carry NO harmonized_code by
|
||||
-- construction (harmonized space is leaf-only), so a plain
|
||||
-- `harmonized_code IS NOT NULL` filter would drop every legacy IG aggregate --
|
||||
-- 6.4e9 of M and 3.1e8 of L in corpus units. COALESCE keeps the one real IG
|
||||
-- collapse rule (M38 -> M36, SB012, year-disjoint 1967-2011 vs 2012+) while
|
||||
-- never dropping a row.
|
||||
-- in the bundled fixture corpus (year 2011; 2012+ all carry a harmonized_code)
|
||||
-- that is $379,016,063k across 25,688 M rows and $2,277,458k across 19,266 L
|
||||
-- rows (`SELECT year, LEFT(item_code,1), SUM(amt), COUNT(*) FROM ig_long
|
||||
-- WHERE harmonized_code IS NULL GROUP BY 1, 2`). COALESCE keeps the one real
|
||||
-- IG collapse rule (M38 -> M36, SB012, year-disjoint 1967-2011 vs 2012+)
|
||||
-- while never dropping a row.
|
||||
CREATE OR REPLACE VIEW ig_long_harmonized AS
|
||||
SELECT * REPLACE (COALESCE(harmonized_code, item_code) AS item_code)
|
||||
FROM long
|
||||
|
||||
Reference in New Issue
Block a user