fix: regenerate the bundled fixture against the sparsified corpus (#18)
The fixture predated three shipped corpus changes at once: no J rows in summary_categories (it was built before the crosswalk completion), no representation.parquet or code_set.parquet, and a still-dense wide era. Every test in this package and in cog-api runs against it, so both suites were green against a corpus that no longer exists. This is #18's stated prerequisite; it proves nothing about production until it lands. Regenerated from the publish tree at pipeline_commit 83f9715 (schema v6, built 2026-07-29). FY2011 goes from 2,864,212 rows to 496,004 -- 82.7% of the old partition was explicit zeros -- and the fixture now ships all ten publish-tree metadata tables rather than six. The generator's file list is a single constant now, so the copy step and the manifest step cannot drift. Three test repairs, each a real consequence of sparsification rather than a number to bump: test-categories.R "assistance" joined the spending subtype vocabulary with the J-prefix codes. test-spending.R The harmonization block counts rows that exist. Broward's E21/F21/G21 were zero-pads and are gone, so the anchor moves to FL state, whose three NA-mapped rows carry $2.83B -- the amount accounting was previously asserted only against 0 and could not have caught a bug. Broward keeps a test of its own, now asserting the zero-pads are absent. test-expenditure-concept.R Coverage-gap suggestions are presence-based. AL state's only FY2011 B47 cell was an explicit zero, so ig_federal_b47_wide stopped being a candidate there; FL state carries a real amount, so the counterpart guard is exercised against a suggestion that fires. test-fixture-vintage.R pins the structural facts that separate this vintage from its predecessor -- the ten metadata tables, the dense/sparse representation contract, zero explicit zeros in FY2011, code_set coverage, and J19's category. Checked against the old fixture: FY2011 carried 2,368,208 explicit zeros, so the assertion discriminates rather than merely passing. Suites: uscogdata 594 pass / 0 fail / 6 skip (was 576/0/6). cog-api 357 pass / 0 fail / 8 skip against the regenerated fixture, unchanged from its baseline.
This commit is contained in:
@@ -244,24 +244,42 @@ test_that("basis defaults to 'harmonized' when not passed", {
|
||||
test_that("provenance carries basis + harmonization block with na_rows_excluded", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
r <- cog_spending("121011212191", 2011:2012, "Corrections")
|
||||
# FL state government. The harmonization block is scoped by government,
|
||||
# year and flow prefix -- NOT by category -- so a Corrections query still
|
||||
# counts every E/F/G-prefixed row the harmonized basis drops for having
|
||||
# no harmonized_code. The three that apply here are E21/F21/G21
|
||||
# (Education NEC, SB184-186, "discontinued_na", wide-era window ending
|
||||
# FY2011); the other discontinued_na rulings live outside E/F/G.
|
||||
# See docs/phase_r_harmonization_review.md § 1.3/1.4 and cog_pipeline
|
||||
# data/harmonization_map.csv.
|
||||
r <- cog_spending("120000226351", 2011:2012, "Corrections")
|
||||
prov <- attr(r, "provenance")
|
||||
expect_equal(prov$basis, "harmonized")
|
||||
expect_true(prov$harmonization$applied)
|
||||
expect_true(prov$harmonization$na_rows_excluded >= 0L)
|
||||
expect_true(prov$harmonization$na_amount_excluded >= 0)
|
||||
# Data-verified for the v6 fixture (corpus 2026-07-22). The Task 18 map
|
||||
# extension added E/F/G-prefix discontinued_na rulings the earlier pin's
|
||||
# comment predated: E21/F21/G21 (Education NEC local, SB184-186,
|
||||
# "trivial; explicit-NA, full wide-era window"). Broward's 2011 legacy
|
||||
# partition zero-pads exactly those three codes, so this query now
|
||||
# excludes 3 NA-harmonized rows -- all with amt = 0, hence the excluded
|
||||
# AMOUNT stays exactly zero. (The other discontinued_na rulings -- S74,
|
||||
# Z61, X04, X06, the debt-detail family, L24 -- remain outside the
|
||||
# E/F/G/K prefixes.) See docs/phase_r_harmonization_review.md § 1.3/1.4
|
||||
# and cog_pipeline data/harmonization_map.csv E21/F21/G21 rows.
|
||||
expect_equal(prov$harmonization$na_rows_excluded, 3L)
|
||||
expect_equal(prov$harmonization$na_amount_excluded, 0)
|
||||
# $2,825,439 thousands of FY2011 E21 + F21 + G21, reported in full USD.
|
||||
# Pinning a non-zero amount is the point: the earlier Broward anchor's
|
||||
# three rows were all explicit zeros, so the AMOUNT accounting was
|
||||
# asserted only against 0 and could not have caught a bug.
|
||||
expect_equal(prov$harmonization$na_amount_excluded, 2825439 * 1000)
|
||||
})
|
||||
})
|
||||
|
||||
test_that("sparsification removed the wide era's zero-pads from the exclusion count", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
# Broward County FY2011 used to carry E21/F21/G21 rows of exactly $0 --
|
||||
# the wide era stored every government x every code, zeros included. The
|
||||
# published corpus no longer does (SB194, cog_pipeline#64), so there is
|
||||
# now nothing for the harmonized basis to exclude. Absence in a
|
||||
# dense_source year means Census published $0; it does not mean the
|
||||
# exclusion machinery stopped working, which the FL state anchor above
|
||||
# proves independently.
|
||||
r <- cog_spending("121011212191", 2011:2012, "Corrections")
|
||||
h <- attr(r, "provenance")$harmonization
|
||||
expect_true(h$applied)
|
||||
expect_equal(h$na_rows_excluded, 0L)
|
||||
expect_equal(h$na_amount_excluded, 0)
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
Reference in New Issue
Block a user