feat: basis= harmonized/raw with v4/v5 dual-accept
Adds schema_version 5 support alongside the existing v4 corpus:
.validate_schema() now accepts a supported set (4, 5) instead of a single
expected version, and cog_spending()/cog_revenue() gain basis =
c("harmonized", "raw"). Harmonized basis routes to new
spending_annotated_harmonized / revenue_annotated_harmonized views built on
spending_long_harmonized / revenue_long_harmonized (REPLACE(harmonized_code
AS item_code), excluding aggregate and NA-harmonized rows); raw basis is
byte-identical to the pre-Phase-R2 behavior. On a v4 corpus, an unspecified
basis silently resolves to "raw" with a provenance note; an explicit
basis = "harmonized" aborts with an actionable message.
Provenance gains basis, basis_note, and a harmonization block
(applied/na_rows_excluded/na_amount_excluded). The five new schema-v5-only
SQL views (harmonized long/annotated views, harmonization_map,
harmonization_recipes, series_breaks_pq) are registered conditionally on
manifest$schema_version >= 5, since DuckDB's read_parquet() errors eagerly
at CREATE VIEW time when the backing file doesn't exist on a v4 corpus.
Fixture corpus regenerated to schema_version 5 / years 2011, 2012, 2019,
2020 (2011->2012 spans the wide-aggregate -> modern-leaf format boundary
needed for the harmonization/recipe work), with the harmonization_map /
harmonization_recipes / series_breaks parquet tables bundled alongside the
existing metadata registries.
This commit is contained in:
@@ -3,12 +3,20 @@
|
||||
# Regenerate inst/extdata/fixture_corpus/ from a cog_pipeline publish tree.
|
||||
#
|
||||
# What this does:
|
||||
# 1. Copies the year=2019 and year=2020 long partitions as-is (byte-for-
|
||||
# byte) from <publish_cache>/data/long/ into the fixture.
|
||||
# 1. Copies each requested year's long partition as-is (byte-for-byte)
|
||||
# from <publish_cache>/data/long/ into the fixture. Default years are
|
||||
# c(2011L, 2012L, 2019L, 2020L): 2011/2012 straddle the wide-aggregate
|
||||
# -> modern-leaf format boundary (the harmonization/recipe seam), and
|
||||
# 2019/2020 are the pre-existing per-capita/CPI regression anchors.
|
||||
# Each partition is a full year (all states/govs) as published, so
|
||||
# Broward County FL and every other previously-pinned government stay
|
||||
# covered without any per-gov slicing logic.
|
||||
# 2. Copies the full canonical_fips_xwalk.parquet, canonical_alias.parquet,
|
||||
# and summary_categories.parquet metadata tables as-is (these are small
|
||||
# cross-vintage registries, not partitioned by year, so the fixture
|
||||
# ships the complete tables rather than a year-scoped subset).
|
||||
# summary_categories.parquet, harmonization_map.parquet,
|
||||
# harmonization_recipes.parquet, and series_breaks.parquet metadata
|
||||
# tables as-is (these are small cross-vintage registries, not
|
||||
# partitioned by year, so the fixture ships the complete tables rather
|
||||
# than a year-scoped subset).
|
||||
# 3. Resyncs the four reference docs (data_dictionary.md,
|
||||
# reader-specification.md, README.md, series_breaks.md) from the
|
||||
# publish tree's docs/.
|
||||
@@ -35,7 +43,7 @@ regenerate_fixture_corpus <- function(
|
||||
"..", "cog_pipeline", "_targets", "publish_cache"
|
||||
),
|
||||
fixture_dir = file.path("inst", "extdata", "fixture_corpus"),
|
||||
fixture_years = c(2019L, 2020L)) {
|
||||
fixture_years = c(2011L, 2012L, 2019L, 2020L)) {
|
||||
stopifnot(
|
||||
requireNamespace("digest", quietly = TRUE),
|
||||
requireNamespace("jsonlite", quietly = TRUE),
|
||||
@@ -92,14 +100,18 @@ regenerate_fixture_corpus <- function(
|
||||
invisible(NULL)
|
||||
}
|
||||
|
||||
# Copy the full (not year-scoped) canonical_fips_xwalk, canonical_alias, and
|
||||
# summary_categories parquet tables.
|
||||
# Copy the full (not year-scoped) canonical_fips_xwalk, canonical_alias,
|
||||
# summary_categories, and (schema v5+) harmonization_map/
|
||||
# harmonization_recipes/series_breaks parquet tables.
|
||||
#' @noRd
|
||||
.copy_metadata_parquets <- function(publish_cache_dir, fixture_dir) {
|
||||
files <- c(
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"canonical_alias.parquet",
|
||||
"summary_categories.parquet"
|
||||
"summary_categories.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"series_breaks.parquet"
|
||||
)
|
||||
for (f in files) {
|
||||
src <- file.path(publish_cache_dir, "data", f)
|
||||
@@ -170,7 +182,10 @@ regenerate_fixture_corpus <- function(
|
||||
metadata_files <- c(
|
||||
"canonical_alias.parquet",
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"summary_categories.parquet"
|
||||
"summary_categories.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"series_breaks.parquet"
|
||||
)
|
||||
metadata <- lapply(metadata_files, function(f) {
|
||||
rel <- file.path("data", f)
|
||||
@@ -187,11 +202,15 @@ regenerate_fixture_corpus <- function(
|
||||
built_at = format(Sys.time(), "%Y-%m-%dT%H:%M:%SZ", tz = "UTC"),
|
||||
pipeline_commit = source_manifest$pipeline_commit,
|
||||
fixture_note = paste(
|
||||
"Two-year (2019-2020) fixture for uscogdata tests. Full corpus",
|
||||
"available via USCOGDATA_URL. Regenerated for Phase P",
|
||||
"(schema_version 4, uniformly 12-char canonical_govid) with the full",
|
||||
"canonical_fips_xwalk master and the new canonical_alias lookup",
|
||||
"table via data-raw/regenerate_fixture_corpus.R."
|
||||
"Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full",
|
||||
"corpus available via USCOGDATA_URL. Regenerated for Phase R2",
|
||||
"(schema_version 5, harmonization_map/harmonization_recipes/",
|
||||
"series_breaks parquet tables added). 2011/2012 straddle the",
|
||||
"wide-aggregate -> modern-leaf format boundary exercised by basis=",
|
||||
"\"harmonized\" and recipe= queries; 2019/2020 retain the prior",
|
||||
"per-capita/CPI regression anchors. Full canonical_fips_xwalk master",
|
||||
"and canonical_alias lookup table included via",
|
||||
"data-raw/regenerate_fixture_corpus.R."
|
||||
),
|
||||
data_vintage = source_manifest$data_vintage,
|
||||
scope = source_manifest$scope,
|
||||
|
||||
Reference in New Issue
Block a user