feat: basis= harmonized/raw with v4/v5 dual-accept

Adds schema_version 5 support alongside the existing v4 corpus:
.validate_schema() now accepts a supported set (4, 5) instead of a single
expected version, and cog_spending()/cog_revenue() gain basis =
c("harmonized", "raw"). Harmonized basis routes to new
spending_annotated_harmonized / revenue_annotated_harmonized views built on
spending_long_harmonized / revenue_long_harmonized (REPLACE(harmonized_code
AS item_code), excluding aggregate and NA-harmonized rows); raw basis is
byte-identical to the pre-Phase-R2 behavior. On a v4 corpus, an unspecified
basis silently resolves to "raw" with a provenance note; an explicit
basis = "harmonized" aborts with an actionable message.

Provenance gains basis, basis_note, and a harmonization block
(applied/na_rows_excluded/na_amount_excluded). The five new schema-v5-only
SQL views (harmonized long/annotated views, harmonization_map,
harmonization_recipes, series_breaks_pq) are registered conditionally on
manifest$schema_version >= 5, since DuckDB's read_parquet() errors eagerly
at CREATE VIEW time when the backing file doesn't exist on a v4 corpus.

Fixture corpus regenerated to schema_version 5 / years 2011, 2012, 2019,
2020 (2011->2012 spans the wide-aggregate -> modern-leaf format boundary
needed for the harmonization/recipe work), with the harmonization_map /
harmonization_recipes / series_breaks parquet tables bundled alongside the
existing metadata registries.
This commit is contained in:
2026-07-18 23:19:17 -04:00
parent 3b725770d2
commit 7818cd2b1a
35 changed files with 579 additions and 53 deletions
+39 -1
View File
@@ -107,6 +107,44 @@ test_that("cog_manifest returns the active session's parsed manifest", {
expect_true(m$schema_version >= 4L)
yrs <- vapply(m$files$long_partitions, function(p) as.integer(p$year),
integer(1))
expect_setequal(yrs, c(2019L, 2020L))
expect_setequal(yrs, c(2011L, 2012L, 2019L, 2020L))
})
})
test_that(".validate_schema accepts schema_version 4 and 5, rejects others", {
expect_silent(uscogdata:::.validate_schema(list(schema_version = 4L)))
expect_silent(uscogdata:::.validate_schema(list(schema_version = 5L)))
expect_error(
uscogdata:::.validate_schema(list(schema_version = 3L)),
"schema_version"
)
expect_error(
uscogdata:::.validate_schema(list(schema_version = 6L)),
"schema_version"
)
})
test_that("cog_open succeeds against a doctored schema_version 4 corpus (dual-accept)", {
skip_if_no_corpus()
with_doctored_schema_version(4L, {
con <- cog_open()
expect_true(DBI::dbIsValid(con))
expect_equal(as.integer(cog_manifest()$schema_version), 4L)
# Core (pre-Phase-R2) views must still register on a v4 corpus.
views <- DBI::dbGetQuery(con,
"SELECT table_name FROM information_schema.tables
WHERE table_schema = 'main' AND table_type = 'VIEW'"
)$table_name
expect_true(all(c("spending_annotated", "revenue_annotated") %in% views))
# Schema-v5-only harmonization views must NOT register on a v4 corpus:
# their parquet sources don't exist there and DuckDB's read_parquet()
# errors eagerly at CREATE VIEW time for a missing file/glob, so
# .register_views() gates these on manifest$schema_version >= 5.
expect_false(any(c(
"spending_long_harmonized", "spending_annotated_harmonized",
"harmonization_recipes", "harmonization_map", "series_breaks_pq"
) %in% views))
})
})