jared
7818cd2b1a
Adds schema_version 5 support alongside the existing v4 corpus:
.validate_schema() now accepts a supported set (4, 5) instead of a single
expected version, and cog_spending()/cog_revenue() gain basis =
c("harmonized", "raw"). Harmonized basis routes to new
spending_annotated_harmonized / revenue_annotated_harmonized views built on
spending_long_harmonized / revenue_long_harmonized (REPLACE(harmonized_code
AS item_code), excluding aggregate and NA-harmonized rows); raw basis is
byte-identical to the pre-Phase-R2 behavior. On a v4 corpus, an unspecified
basis silently resolves to "raw" with a provenance note; an explicit
basis = "harmonized" aborts with an actionable message.
Provenance gains basis, basis_note, and a harmonization block
(applied/na_rows_excluded/na_amount_excluded). The five new schema-v5-only
SQL views (harmonized long/annotated views, harmonization_map,
harmonization_recipes, series_breaks_pq) are registered conditionally on
manifest$schema_version >= 5, since DuckDB's read_parquet() errors eagerly
at CREATE VIEW time when the backing file doesn't exist on a v4 corpus.
Fixture corpus regenerated to schema_version 5 / years 2011, 2012, 2019,
2020 (2011->2012 spans the wide-aggregate -> modern-leaf format boundary
needed for the harmonization/recipe work), with the harmonization_map /
harmonization_recipes / series_breaks parquet tables bundled alongside the
existing metadata registries.
36 lines
1.5 KiB
R
36 lines
1.5 KiB
R
# R/views.R
|
|
|
|
# SQL files whose view definitions read schema-v5-only parquet tables
|
|
# (harmonization_map.parquet, harmonization_recipes.parquet,
|
|
# series_breaks.parquet) or select from views built on top of them. DuckDB's
|
|
# read_parquet() resolves the file at CREATE VIEW time (even for a view, it
|
|
# still needs the source schema) and errors immediately -- "IO Error: No
|
|
# files found" -- if the path doesn't exist, so these cannot be registered
|
|
# unconditionally against a v4 corpus the way the rest of inst/sql/ is.
|
|
# Registration is therefore gated on manifest$schema_version >= 5; verb-level
|
|
# *usage* of the resulting views is separately gated by .resolve_basis() /
|
|
# .require_schema_v5().
|
|
.harmonization_view_files <- c(
|
|
"22-spending_long_harmonized.sql",
|
|
"23-revenue_long_harmonized.sql",
|
|
"33-harmonization_map.sql",
|
|
"34-harmonization_recipes.sql",
|
|
"35-series_breaks_pq.sql",
|
|
"42-spending_annotated_harmonized.sql",
|
|
"43-revenue_annotated_harmonized.sql"
|
|
)
|
|
|
|
#' Register DuckDB views from inst/sql/ SQL files
|
|
#' @noRd
|
|
.register_views <- function(con, url, manifest) {
|
|
sql_dir <- system.file("sql", package = "uscogdata")
|
|
files <- sort(list.files(sql_dir, pattern = "\\.sql$", full.names = TRUE))
|
|
schema_version <- suppressWarnings(as.integer(manifest$schema_version %||% 0L))
|
|
for (f in files) {
|
|
if (basename(f) %in% .harmonization_view_files && schema_version < 5L) next
|
|
sql <- paste(readLines(f, warn = FALSE), collapse = "\n")
|
|
sql <- gsub("\\{url\\}", url, sql, fixed = FALSE)
|
|
DBI::dbExecute(con, sql)
|
|
}
|
|
}
|