fix: regenerate the bundled fixture against the sparsified corpus (#18)
R-CMD-check / check (push) Successful in 3m3s
R-CMD-check / check (pull_request) Successful in 2m51s

The fixture predated three shipped corpus changes at once: no J rows in
summary_categories (it was built before the crosswalk completion), no
representation.parquet or code_set.parquet, and a still-dense wide era.
Every test in this package and in cog-api runs against it, so both suites
were green against a corpus that no longer exists. This is #18's stated
prerequisite; it proves nothing about production until it lands.

Regenerated from the publish tree at pipeline_commit 83f9715 (schema v6,
built 2026-07-29). FY2011 goes from 2,864,212 rows to 496,004 -- 82.7% of
the old partition was explicit zeros -- and the fixture now ships all ten
publish-tree metadata tables rather than six. The generator's file list is
a single constant now, so the copy step and the manifest step cannot drift.

Three test repairs, each a real consequence of sparsification rather than
a number to bump:

  test-categories.R          "assistance" joined the spending subtype
                             vocabulary with the J-prefix codes.

  test-spending.R            The harmonization block counts rows that
                             exist. Broward's E21/F21/G21 were zero-pads
                             and are gone, so the anchor moves to FL state,
                             whose three NA-mapped rows carry $2.83B --
                             the amount accounting was previously asserted
                             only against 0 and could not have caught a
                             bug. Broward keeps a test of its own, now
                             asserting the zero-pads are absent.

  test-expenditure-concept.R Coverage-gap suggestions are presence-based.
                             AL state's only FY2011 B47 cell was an
                             explicit zero, so ig_federal_b47_wide stopped
                             being a candidate there; FL state carries a
                             real amount, so the counterpart guard is
                             exercised against a suggestion that fires.

test-fixture-vintage.R pins the structural facts that separate this vintage
from its predecessor -- the ten metadata tables, the dense/sparse
representation contract, zero explicit zeros in FY2011, code_set coverage,
and J19's category. Checked against the old fixture: FY2011 carried
2,368,208 explicit zeros, so the assertion discriminates rather than
merely passing.

Suites: uscogdata 594 pass / 0 fail / 6 skip (was 576/0/6).
cog-api 357 pass / 0 fail / 8 skip against the regenerated fixture,
unchanged from its baseline.
This commit is contained in:
2026-07-30 10:20:09 -04:00
parent 82acda6f93
commit c375c55da7
14 changed files with 252 additions and 60 deletions
+38 -34
View File
@@ -11,12 +11,14 @@
# Each partition is a full year (all states/govs) as published, so
# Broward County FL and every other previously-pinned government stay
# covered without any per-gov slicing logic.
# 2. Copies the full canonical_fips_xwalk.parquet, canonical_alias.parquet,
# summary_categories.parquet, harmonization_map.parquet,
# harmonization_recipes.parquet, and series_breaks.parquet metadata
# tables as-is (these are small cross-vintage registries, not
# partitioned by year, so the fixture ships the complete tables rather
# than a year-scoped subset).
# 2. Copies every metadata parquet the publish tree ships (see
# .FIXTURE_METADATA_FILES) as-is. These are small cross-vintage
# registries, not partitioned by year, so the fixture ships the complete
# tables rather than a year-scoped subset. representation.parquet and
# code_set.parquet are what make the sparse wide era interpretable --
# absence means "$0" in a dense_source year and "not reported" in a
# sparse_source one -- so a fixture without them cannot represent the
# published corpus.
# 3. Resyncs the four reference docs (data_dictionary.md,
# reader-specification.md, README.md, series_breaks.md) from the
# publish tree's docs/.
@@ -38,6 +40,22 @@
# source("data-raw/regenerate_fixture_corpus.R")
# regenerate_fixture_corpus(publish_cache_dir = "/path/to/publish_cache")
# Every metadata parquet the publish tree ships, in the order they appear in
# the corpus manifest. Single source of truth for both the copy step and the
# fixture manifest, so the two can never drift apart.
.FIXTURE_METADATA_FILES <- c(
"canonical_alias.parquet",
"canonical_fips_xwalk.parquet",
"census_collection_coverage.parquet",
"code_set.parquet",
"harmonization_map.parquet",
"harmonization_recipes.parquet",
"lineage_events.parquet",
"representation.parquet",
"series_breaks.parquet",
"summary_categories.parquet"
)
regenerate_fixture_corpus <- function(
publish_cache_dir = file.path(
"..", "cog_pipeline", "_targets", "publish_cache"
@@ -100,20 +118,11 @@ regenerate_fixture_corpus <- function(
invisible(NULL)
}
# Copy the full (not year-scoped) canonical_fips_xwalk, canonical_alias,
# summary_categories, and (schema v5+) harmonization_map/
# harmonization_recipes/series_breaks parquet tables.
# Copy the full (not year-scoped) metadata tables listed in
# .FIXTURE_METADATA_FILES.
#' @noRd
.copy_metadata_parquets <- function(publish_cache_dir, fixture_dir) {
files <- c(
"canonical_fips_xwalk.parquet",
"canonical_alias.parquet",
"summary_categories.parquet",
"harmonization_map.parquet",
"harmonization_recipes.parquet",
"series_breaks.parquet"
)
for (f in files) {
for (f in .FIXTURE_METADATA_FILES) {
src <- file.path(publish_cache_dir, "data", f)
dst <- file.path(fixture_dir, "data", f)
if (!file.exists(src)) {
@@ -179,15 +188,7 @@ regenerate_fixture_corpus <- function(
)
})
metadata_files <- c(
"canonical_alias.parquet",
"canonical_fips_xwalk.parquet",
"summary_categories.parquet",
"harmonization_map.parquet",
"harmonization_recipes.parquet",
"series_breaks.parquet"
)
metadata <- lapply(metadata_files, function(f) {
metadata <- lapply(.FIXTURE_METADATA_FILES, function(f) {
rel <- file.path("data", f)
path <- file.path(fixture_dir, rel)
list(
@@ -203,13 +204,16 @@ regenerate_fixture_corpus <- function(
pipeline_commit = source_manifest$pipeline_commit,
fixture_note = paste(
"Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full",
"corpus available via USCOGDATA_URL. Regenerated for Phase R2",
"(schema_version 5, harmonization_map/harmonization_recipes/",
"series_breaks parquet tables added). 2011/2012 straddle the",
"wide-aggregate -> modern-leaf format boundary exercised by basis=",
"\"harmonized\" and recipe= queries; 2019/2020 retain the prior",
"per-capita/CPI regression anchors. Full canonical_fips_xwalk master",
"and canonical_alias lookup table included via",
"corpus available via USCOGDATA_URL. Regenerated from the sparsified",
"schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit",
"zeros, so FY2011 absence means Census published $0 while FY2012+",
"absence means not reported. representation.parquet and",
"code_set.parquet carry that rule and ship in full, as do every other",
"metadata table in the publish tree. 2011/2012 straddle both the",
"wide-aggregate -> modern-leaf format boundary (exercised by",
"basis=\"harmonized\" and recipe= queries) and the dense -> sparse",
"representation boundary (SB194); 2019/2020 retain the prior",
"per-capita/CPI regression anchors. Regenerated via",
"data-raw/regenerate_fixture_corpus.R."
),
data_vintage = source_manifest$data_vintage,