fix: regenerate the bundled fixture against the sparsified corpus (#18)
The fixture predated three shipped corpus changes at once: no J rows in summary_categories (it was built before the crosswalk completion), no representation.parquet or code_set.parquet, and a still-dense wide era. Every test in this package and in cog-api runs against it, so both suites were green against a corpus that no longer exists. This is #18's stated prerequisite; it proves nothing about production until it lands. Regenerated from the publish tree at pipeline_commit 83f9715 (schema v6, built 2026-07-29). FY2011 goes from 2,864,212 rows to 496,004 -- 82.7% of the old partition was explicit zeros -- and the fixture now ships all ten publish-tree metadata tables rather than six. The generator's file list is a single constant now, so the copy step and the manifest step cannot drift. Three test repairs, each a real consequence of sparsification rather than a number to bump: test-categories.R "assistance" joined the spending subtype vocabulary with the J-prefix codes. test-spending.R The harmonization block counts rows that exist. Broward's E21/F21/G21 were zero-pads and are gone, so the anchor moves to FL state, whose three NA-mapped rows carry $2.83B -- the amount accounting was previously asserted only against 0 and could not have caught a bug. Broward keeps a test of its own, now asserting the zero-pads are absent. test-expenditure-concept.R Coverage-gap suggestions are presence-based. AL state's only FY2011 B47 cell was an explicit zero, so ig_federal_b47_wide stopped being a candidate there; FL state carries a real amount, so the counterpart guard is exercised against a suggestion that fires. test-fixture-vintage.R pins the structural facts that separate this vintage from its predecessor -- the ten metadata tables, the dense/sparse representation contract, zero explicit zeros in FY2011, code_set coverage, and J19's category. Checked against the old fixture: FY2011 carried 2,368,208 explicit zeros, so the assertion discriminates rather than merely passing. Suites: uscogdata 594 pass / 0 fail / 6 skip (was 576/0/6). cog-api 357 pass / 0 fail / 8 skip against the regenerated fixture, unchanged from its baseline.
This commit is contained in:
@@ -11,12 +11,14 @@
|
||||
# Each partition is a full year (all states/govs) as published, so
|
||||
# Broward County FL and every other previously-pinned government stay
|
||||
# covered without any per-gov slicing logic.
|
||||
# 2. Copies the full canonical_fips_xwalk.parquet, canonical_alias.parquet,
|
||||
# summary_categories.parquet, harmonization_map.parquet,
|
||||
# harmonization_recipes.parquet, and series_breaks.parquet metadata
|
||||
# tables as-is (these are small cross-vintage registries, not
|
||||
# partitioned by year, so the fixture ships the complete tables rather
|
||||
# than a year-scoped subset).
|
||||
# 2. Copies every metadata parquet the publish tree ships (see
|
||||
# .FIXTURE_METADATA_FILES) as-is. These are small cross-vintage
|
||||
# registries, not partitioned by year, so the fixture ships the complete
|
||||
# tables rather than a year-scoped subset. representation.parquet and
|
||||
# code_set.parquet are what make the sparse wide era interpretable --
|
||||
# absence means "$0" in a dense_source year and "not reported" in a
|
||||
# sparse_source one -- so a fixture without them cannot represent the
|
||||
# published corpus.
|
||||
# 3. Resyncs the four reference docs (data_dictionary.md,
|
||||
# reader-specification.md, README.md, series_breaks.md) from the
|
||||
# publish tree's docs/.
|
||||
@@ -38,6 +40,22 @@
|
||||
# source("data-raw/regenerate_fixture_corpus.R")
|
||||
# regenerate_fixture_corpus(publish_cache_dir = "/path/to/publish_cache")
|
||||
|
||||
# Every metadata parquet the publish tree ships, in the order they appear in
|
||||
# the corpus manifest. Single source of truth for both the copy step and the
|
||||
# fixture manifest, so the two can never drift apart.
|
||||
.FIXTURE_METADATA_FILES <- c(
|
||||
"canonical_alias.parquet",
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"census_collection_coverage.parquet",
|
||||
"code_set.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"lineage_events.parquet",
|
||||
"representation.parquet",
|
||||
"series_breaks.parquet",
|
||||
"summary_categories.parquet"
|
||||
)
|
||||
|
||||
regenerate_fixture_corpus <- function(
|
||||
publish_cache_dir = file.path(
|
||||
"..", "cog_pipeline", "_targets", "publish_cache"
|
||||
@@ -100,20 +118,11 @@ regenerate_fixture_corpus <- function(
|
||||
invisible(NULL)
|
||||
}
|
||||
|
||||
# Copy the full (not year-scoped) canonical_fips_xwalk, canonical_alias,
|
||||
# summary_categories, and (schema v5+) harmonization_map/
|
||||
# harmonization_recipes/series_breaks parquet tables.
|
||||
# Copy the full (not year-scoped) metadata tables listed in
|
||||
# .FIXTURE_METADATA_FILES.
|
||||
#' @noRd
|
||||
.copy_metadata_parquets <- function(publish_cache_dir, fixture_dir) {
|
||||
files <- c(
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"canonical_alias.parquet",
|
||||
"summary_categories.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"series_breaks.parquet"
|
||||
)
|
||||
for (f in files) {
|
||||
for (f in .FIXTURE_METADATA_FILES) {
|
||||
src <- file.path(publish_cache_dir, "data", f)
|
||||
dst <- file.path(fixture_dir, "data", f)
|
||||
if (!file.exists(src)) {
|
||||
@@ -179,15 +188,7 @@ regenerate_fixture_corpus <- function(
|
||||
)
|
||||
})
|
||||
|
||||
metadata_files <- c(
|
||||
"canonical_alias.parquet",
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"summary_categories.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"series_breaks.parquet"
|
||||
)
|
||||
metadata <- lapply(metadata_files, function(f) {
|
||||
metadata <- lapply(.FIXTURE_METADATA_FILES, function(f) {
|
||||
rel <- file.path("data", f)
|
||||
path <- file.path(fixture_dir, rel)
|
||||
list(
|
||||
@@ -203,13 +204,16 @@ regenerate_fixture_corpus <- function(
|
||||
pipeline_commit = source_manifest$pipeline_commit,
|
||||
fixture_note = paste(
|
||||
"Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full",
|
||||
"corpus available via USCOGDATA_URL. Regenerated for Phase R2",
|
||||
"(schema_version 5, harmonization_map/harmonization_recipes/",
|
||||
"series_breaks parquet tables added). 2011/2012 straddle the",
|
||||
"wide-aggregate -> modern-leaf format boundary exercised by basis=",
|
||||
"\"harmonized\" and recipe= queries; 2019/2020 retain the prior",
|
||||
"per-capita/CPI regression anchors. Full canonical_fips_xwalk master",
|
||||
"and canonical_alias lookup table included via",
|
||||
"corpus available via USCOGDATA_URL. Regenerated from the sparsified",
|
||||
"schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit",
|
||||
"zeros, so FY2011 absence means Census published $0 while FY2012+",
|
||||
"absence means not reported. representation.parquet and",
|
||||
"code_set.parquet carry that rule and ship in full, as do every other",
|
||||
"metadata table in the publish tree. 2011/2012 straddle both the",
|
||||
"wide-aggregate -> modern-leaf format boundary (exercised by",
|
||||
"basis=\"harmonized\" and recipe= queries) and the dense -> sparse",
|
||||
"representation boundary (SB194); 2019/2020 retain the prior",
|
||||
"per-capita/CPI regression anchors. Regenerated via",
|
||||
"data-raw/regenerate_fixture_corpus.R."
|
||||
),
|
||||
data_vintage = source_manifest$data_vintage,
|
||||
|
||||
Reference in New Issue
Block a user