fix: exercise real harmonized-view SQL in tests; unambiguous recipe provenance
R-CMD-check / check (pull_request) Successful in 3m5s
R-CMD-check / check (push) Successful in 3m1s

test-views.R's harmonized-view test previously ran a hand-rolled REPLACE
query with no WHERE clause, so a regression in any of
inst/sql/22-spending_long_harmonized.sql / 23-revenue_long_harmonized.sql's
three predicates (NOT is_aggregate, harmonized_code IS NOT NULL, the
E/F/G/K or T/A/U/B/C/D prefix filter) would go uncaught. Replaced it with a
test that reads the real SQL files off disk, substitutes {url} exactly as
.register_views() does, and executes them (plus their 10-long.sql
dependency) against a synthetic hive-partitioned parquet tree written via
DuckDB's own COPY ... TO (FORMAT PARQUET) (no arrow dependency, matching
this package's existing convention). Ten rows are crafted so each predicate
is independently falsifiable by a specific row; manually broke each
predicate in turn to confirm the test fails exactly as expected, then
restored the SQL files (see the task report for the RED-phase transcript).

Also fixes a provenance ambiguity: a recipe= query bypasses
spending_annotated(_harmonized)/revenue_annotated(_harmonized) entirely
(.run_recipe() joins `long` directly), so basis= has no effect on it, but
provenance was still reporting basis = "harmonized"/"raw" (whatever the
argument resolved to) with harmonization$applied = FALSE alongside it --
misleading, since it looks like harmonization was evaluated and found
nothing to exclude rather than "not applicable here." Recipe results now
report basis = "recipe" with an inert harmonization block carrying an
explicit note, regardless of what basis= was passed.
This commit is contained in:
2026-07-18 23:55:48 -04:00
parent 4de915b557
commit 77f48047b1
5 changed files with 169 additions and 43 deletions
+36 -11
View File
@@ -28,13 +28,20 @@
#' folding). On a corpus with `schema_version < 5` (no harmonization
#' tables), `basis` silently resolves to `"raw"` when left at its default
#' and the resolution is recorded in the provenance; explicitly passing
#' `basis = "harmonized"` on such a corpus aborts.
#' `basis = "harmonized"` on such a corpus aborts. Ignored when `recipe`
#' is set (see below).
#' @param recipe Optional harmonization recipe id (see [cog_recipes()]) for
#' multi-code cross-vintage series that a 1:1 harmonized_code mapping
#' can't express (e.g. a wide-era aggregate that only splits into leaf
#' codes in the modern era). Mutually exclusive with `category`. The
#' result's subtype column reads `"recipe"` and `category` reads the
#' recipe's label. Requires `schema_version >= 5`.
#' recipe's label. Requires `schema_version >= 5`. A recipe query bypasses
#' `basis` entirely (it joins `long` directly rather than going through
#' the `*_annotated`/`*_annotated_harmonized` views), so the `basis`
#' argument is ignored and the result's provenance reports
#' `basis = "recipe"` with an inert `harmonization` block (`applied =
#' FALSE`, pointing at the `recipe` block instead) rather than a
#' possibly-misleading `"harmonized"`/`"raw"` value.
#' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
#' `spend_subtype`, `category`, `amt_nominal`, optional `amt_real`,
#' optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
@@ -109,14 +116,32 @@ cog_spending <- function(govid, years, category = NULL,
result$notes <- .notes_column(result)
harmonization <- .build_harmonization_block(
con, govid, years, resolved, flow_prefixes
)
suggestions <- if (is.null(recipe)) {
.build_suggestions(con, govid, years, category, result, resolved$basis)
# A recipe result doesn't go through spending_annotated(_harmonized) /
# revenue_annotated(_harmonized) at all -- .run_recipe()'s generic join
# reads `long` directly -- so `basis` and the `harmonization` exclusion
# count (which is itself computed from `long`, independent of which view
# a non-recipe query used) would describe a code path this result never
# took. Rather than report a technically-still-computed but misleading
# basis = "harmonized"/"raw" + harmonization$applied combo, recipe
# results report basis = "recipe" and an explicit, inert harmonization
# block pointing at the `recipe` block instead. Task 12 (cog-api) passes
# provenance through verbatim, so this needs to be unambiguous rather
# than technically-defensible-but-confusing.
if (!is.null(recipe)) {
basis_for_prov <- "recipe"
basis_note_for_prov <- NA_character_
harmonization <- list(
applied = FALSE, na_rows_excluded = 0L, na_amount_excluded = 0,
note = "basis/harmonization not applicable to recipe results; see the recipe block instead"
)
suggestions <- list()
} else {
list()
basis_for_prov <- resolved$basis
basis_note_for_prov <- resolved$note
harmonization <- .build_harmonization_block(
con, govid, years, resolved, flow_prefixes
)
suggestions <- .build_suggestions(con, govid, years, category, result, resolved$basis)
}
prov <- .build_provenance(
@@ -130,8 +155,8 @@ cog_spending <- function(govid, years, category = NULL,
result = result,
sql = sql,
subtype_col = subtype_col,
basis = resolved$basis,
basis_note = resolved$note,
basis = basis_for_prov,
basis_note = basis_note_for_prov,
harmonization = harmonization,
recipe = recipe_block,
suggestions = suggestions