jared
e2088458e1
Nine review items on the expenditure_concept = direct|total feature: - bool_and(is_aggregate) -> bool_or(is_aggregate) for aggregate_fallback: bool_and silently misreported $5,740,775,000 of aggregate-sourced IG dollars (AL state 2011) as aggregate_fallback = FALSE, because the dense wide-era data puts a $0 leaf row in the same group as the real aggregate row. bool_or is a no-op for Direct/Revenue (verified: 0 mismatched groups across both tables) and correct for the IG leg. - Added a year-disjointness invariant test for the four legacy aggregate/leaf IG pairs (M47/M94, M89/M91-93, L47/L94, L89/L91-93), scoped to the aggregate flag rather than bare code presence (M89/L89 continue past 2011 as independent, non-aggregate leaves). - Extended the real-SQL-text/synthetic-parquet harness in test-views.R to pin ig_long/ig_long_harmonized's predicates directly (aggregate rows retained, NULL harmonized_code coalesced, L-- excluded), rather than relying on one fixture row's incidental shape. - Added a test proving the .harmonization_view_files schema-v5 guard is necessary (not just incidental) against a corpus whose `long` genuinely lacks a harmonized_code column, and rewrote the misleading "v5-only parquet files" comment to name both real reasons a file is gated. - Fixed an NA-fragile subtype filter, extended the expected-view-list test, guarded .verb_spendrev() against total on a non-spending view_base, added a roxygen caveat against summing total across levels of government, and replaced an uncheckable corpus-wide SQL comment figure with a fixture-verifiable one. Full suite: 485/0/0 -> 503/0/0 (18 new expectations, zero pre-existing value changed).
16 lines
902 B
SQL
16 lines
902 B
SQL
-- Harmonized-basis IG rows. Uses COALESCE(harmonized_code, item_code) rather
|
|
-- than harmonized_code alone: aggregate rows carry NO harmonized_code by
|
|
-- construction (harmonized space is leaf-only), so a plain
|
|
-- `harmonized_code IS NOT NULL` filter would drop every legacy IG aggregate --
|
|
-- in the bundled fixture corpus (year 2011; 2012+ all carry a harmonized_code)
|
|
-- that is $379,016,063k across 25,688 M rows and $2,277,458k across 19,266 L
|
|
-- rows (`SELECT year, LEFT(item_code,1), SUM(amt), COUNT(*) FROM ig_long
|
|
-- WHERE harmonized_code IS NULL GROUP BY 1, 2`). COALESCE keeps the one real
|
|
-- IG collapse rule (M38 -> M36, SB012, year-disjoint 1967-2011 vs 2012+)
|
|
-- while never dropping a row.
|
|
CREATE OR REPLACE VIEW ig_long_harmonized AS
|
|
SELECT * REPLACE (COALESCE(harmonized_code, item_code) AS item_code)
|
|
FROM long
|
|
WHERE LEFT(item_code, 1) IN ('M', 'L')
|
|
AND item_code NOT LIKE '%--';
|