fix: regenerate the bundled fixture against the sparsified corpus (#18) #20

Merged
jared merged 1 commits from fix/regen-fixture-corpus-18 into main 2026-07-30 10:32:40 -04:00
Owner

Closes #18's stated prerequisite (DoD 1). The complete= verb work (DoD 2-5) stays for a follow-on.

Why

inst/extdata/fixture_corpus/ predated three shipped corpus changes at once: no J rows in summary_categories, no representation.parquet or code_set.parquet, and a still-dense wide era. Every test in this package and in cog-api runs against it, so both suites were green against a corpus that no longer exists.

What changed

Regenerated from the publish tree at pipeline_commit 83f9715 (schema v6, built 2026-07-29).

before after
FY2011 partition rows 2,864,212 496,004
FY2011 explicit zeros 2,368,208 (82.7%) 0
metadata tables shipped 6 10

New in the fixture: representation.parquet, code_set.parquet, census_collection_coverage.parquet, lineage_events.parquet. The generator's file list is now a single constant, so its copy step and its manifest step cannot drift apart.

Test repairs — all real consequences, not numbers bumped

  • test-categories.R — assistance joined the spending subtype vocabulary with the J-prefix codes.
  • test-spending.R — the harmonization block counts rows that exist. Broward's E21/F21/G21 were zero-pads and are gone, so the anchor moves to FL state, whose three NA-mapped rows carry $2.83B; the amount accounting was previously asserted only against 0 and could not have caught a bug. Broward keeps a test of its own, now asserting the zero-pads are absent.
  • test-expenditure-concept.R — coverage-gap suggestions are presence-based. AL state's only FY2011 B47 cell was an explicit zero, so ig_federal_b47_wide stopped being a candidate there; FL state carries a real amount, so the counterpart guard is exercised against a suggestion that genuinely fires.

New: test-fixture-vintage.R

Pins the structural facts separating this vintage from its predecessor — the ten metadata tables, the dense/sparse representation contract, zero explicit zeros in FY2011, code_set coverage, J19's category. Checked against the old fixture (2,368,208 explicit zeros), so the assertion discriminates rather than merely passing. A fixture left behind by a future publish now fails loudly.

Verification

suite before after
uscogdata 576 / 0 / 6 594 / 0 / 6
cog-api (same fixture, via installed uscogdata) 357 / 0 / 8 357 / 0 / 8
Closes #18's stated prerequisite (DoD 1). The `complete=` verb work (DoD 2-5) stays for a follow-on. ## Why `inst/extdata/fixture_corpus/` predated three shipped corpus changes at once: no `J` rows in `summary_categories`, no `representation.parquet` or `code_set.parquet`, and a still-dense wide era. Every test in this package **and in cog-api** runs against it, so both suites were green against a corpus that no longer exists. ## What changed Regenerated from the publish tree at `pipeline_commit 83f9715` (schema v6, built 2026-07-29). | | before | after | |---|--:|--:| | FY2011 partition rows | 2,864,212 | 496,004 | | FY2011 explicit zeros | 2,368,208 (82.7%) | 0 | | metadata tables shipped | 6 | 10 | New in the fixture: `representation.parquet`, `code_set.parquet`, `census_collection_coverage.parquet`, `lineage_events.parquet`. The generator's file list is now a single constant, so its copy step and its manifest step cannot drift apart. ## Test repairs — all real consequences, not numbers bumped - **test-categories.R** — `assistance` joined the spending subtype vocabulary with the J-prefix codes. - **test-spending.R** — the harmonization block counts rows that *exist*. Broward's E21/F21/G21 were zero-pads and are gone, so the anchor moves to FL state, whose three NA-mapped rows carry $2.83B; the amount accounting was previously asserted only against 0 and could not have caught a bug. Broward keeps a test of its own, now asserting the zero-pads are absent. - **test-expenditure-concept.R** — coverage-gap suggestions are presence-based. AL state's only FY2011 `B47` cell was an explicit zero, so `ig_federal_b47_wide` stopped being a candidate there; FL state carries a real amount, so the counterpart guard is exercised against a suggestion that genuinely fires. ## New: test-fixture-vintage.R Pins the structural facts separating this vintage from its predecessor — the ten metadata tables, the dense/sparse representation contract, zero explicit zeros in FY2011, `code_set` coverage, J19's category. Checked against the old fixture (2,368,208 explicit zeros), so the assertion discriminates rather than merely passing. A fixture left behind by a future publish now fails loudly. ## Verification | suite | before | after | |---|---|---| | uscogdata | 576 / 0 / 6 | **594 / 0 / 6** | | cog-api (same fixture, via installed uscogdata) | 357 / 0 / 8 | **357 / 0 / 8** |
jared added 1 commit 2026-07-30 10:20:37 -04:00
fix: regenerate the bundled fixture against the sparsified corpus (#18)
R-CMD-check / check (push) Successful in 3m3s
R-CMD-check / check (pull_request) Successful in 2m51s
c375c55da7
The fixture predated three shipped corpus changes at once: no J rows in
summary_categories (it was built before the crosswalk completion), no
representation.parquet or code_set.parquet, and a still-dense wide era.
Every test in this package and in cog-api runs against it, so both suites
were green against a corpus that no longer exists. This is #18's stated
prerequisite; it proves nothing about production until it lands.

Regenerated from the publish tree at pipeline_commit 83f9715 (schema v6,
built 2026-07-29). FY2011 goes from 2,864,212 rows to 496,004 -- 82.7% of
the old partition was explicit zeros -- and the fixture now ships all ten
publish-tree metadata tables rather than six. The generator's file list is
a single constant now, so the copy step and the manifest step cannot drift.

Three test repairs, each a real consequence of sparsification rather than
a number to bump:

  test-categories.R          "assistance" joined the spending subtype
                             vocabulary with the J-prefix codes.

  test-spending.R            The harmonization block counts rows that
                             exist. Broward's E21/F21/G21 were zero-pads
                             and are gone, so the anchor moves to FL state,
                             whose three NA-mapped rows carry $2.83B --
                             the amount accounting was previously asserted
                             only against 0 and could not have caught a
                             bug. Broward keeps a test of its own, now
                             asserting the zero-pads are absent.

  test-expenditure-concept.R Coverage-gap suggestions are presence-based.
                             AL state's only FY2011 B47 cell was an
                             explicit zero, so ig_federal_b47_wide stopped
                             being a candidate there; FL state carries a
                             real amount, so the counterpart guard is
                             exercised against a suggestion that fires.

test-fixture-vintage.R pins the structural facts that separate this vintage
from its predecessor -- the ten metadata tables, the dense/sparse
representation contract, zero explicit zeros in FY2011, code_set coverage,
and J19's category. Checked against the old fixture: FY2011 carried
2,368,208 explicit zeros, so the assertion discriminates rather than
merely passing.

Suites: uscogdata 594 pass / 0 fail / 6 skip (was 576/0/6).
cog-api 357 pass / 0 fail / 8 skip against the regenerated fixture,
unchanged from its baseline.
jared merged commit 47dc08c4b0 into main 2026-07-30 10:32:40 -04:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Civilytics/uscogdata#20