fix: scope total's coverage-gap detection to the Direct leg; require IG category rows (C1, C2)

C1: spending_long/spending_long_harmonized filter NOT is_aggregate but
ig_long deliberately doesn't (legacy IG lives on aggregate rows), so a
legacy aggregate-only family (e.g. Corrections pre-2012) can survive on
the IG leg while Direct is suppressed. expenditure_concept = "total"
then UNIONs an IG-only figure that reads as a plausible Total, and the
coverage-gap suggestion machinery -- fed the UNION'd result -- saw the
surviving IG row as coverage and stayed silent.

  (a) .build_suggestions() is now fed a Direct-leg-only view of the
      result (IG rows filtered out before the gap-years computation),
      so the recipe hints fire for "total" exactly as they do for
      "direct".
  (b) Any row where IG has dollars but Direct has none for the same
      (year, canonical_govid, category) is now flagged: the row's
      `notes` name the recovering recipe (drawn from the Direct-leg
      suggestions), and provenance gains an explicit
      `expenditure_concept_direct_suppressed` boolean plus an appended
      warning on `expenditure_concept_note` -- both cheap for a
      downstream consumer (cog-api passes provenance through verbatim)
      to test, rather than silently asserting Direct + IG when that
      arithmetic didn't happen.

Measured before/after on AL state government, Corrections, 2011:
"total" already correctly returns the corpus's actual IG-only figure
($31,358,000, vs. true Direct of $521,651,000 via recipe =
"corrections_combined"), but before this fix it did so with 0
suggestions and an unqualified "Total = Direct + IG" note; after, it
fires 3 recipe hints and both the row notes and provenance say plainly
that Direct is unavailable through this basis.

C2: the 66 M/L summary_categories rows arrived via cog_pipeline PR #59
with no schema_version bump, so schema_version can't gate "total" --
a pre-#59 corpus can report any supported schema_version and still
have zero M/L category rows, in which case ig_annotated's LEFT JOIN
silently produces NA category/spend_subtype (0 rows for a specific
category, or one invisible NA-subtype group for category = NULL). New
.require_ig_categories() checks summary_categories directly and aborts
with class uscogdata_ig_categories_unsupported, naming PR #59 and
directing the user to a newer corpus.

Reconciles tests/testthat/test-views.R's v4-shaped-corpus test (whose
synthetic summary_categories carries only one E36 row) by asserting
the new guard fires against that same connection, rather than leaving
the two silently contradictory.
This commit is contained in:
2026-07-27 12:06:03 -04:00
parent 54ece11867
commit c7260cb20c
6 changed files with 222 additions and 10 deletions
+34
View File
@@ -57,3 +57,37 @@ with_doctored_schema_version <- function(version, code) {
}, add = TRUE)
force(code)
}
# Copy the bundled fixture to a temp dir with summary_categories.parquet
# rewritten to drop every M/L (intergovernmental) row, then run `code`
# against it with a clean session (mirrors with_fixture_corpus()/
# with_doctored_schema_version()). Models a real pre-cog_pipeline-PR#59
# corpus: the 66 M/L category rows shipped with NO schema_version bump (see
# C2 in the expenditure-concept review), so schema_version is left
# untouched here -- only the category data itself is rolled back.
with_corpus_missing_ig_categories <- function(code) {
src <- fixture_corpus_path()
tmp <- withr::local_tempdir(.local_envir = parent.frame())
file.copy(list.files(src, full.names = TRUE), tmp, recursive = TRUE)
cats_path <- file.path(tmp, "data", "summary_categories.parquet")
filtered_path <- file.path(tmp, "data", "summary_categories_filtered.parquet")
write_con <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
DBI::dbExecute(write_con, sprintf(
"COPY (SELECT * FROM read_parquet(%s) WHERE LEFT(item_code, 1) NOT IN ('M', 'L'))
TO %s (FORMAT PARQUET)",
uscogdata:::.sql_lit_chr(cats_path), uscogdata:::.sql_lit_chr(filtered_path)
))
file.remove(cats_path)
file.rename(filtered_path, cats_path)
old_url <- Sys.getenv("USCOGDATA_URL", unset = NA)
uscogdata:::cog_close()
Sys.setenv(USCOGDATA_URL = paste0(tmp, "/"))
on.exit({
uscogdata:::cog_close()
if (is.na(old_url)) Sys.unsetenv("USCOGDATA_URL") else Sys.setenv(USCOGDATA_URL = old_url)
}, add = TRUE)
force(code)
}
+17
View File
@@ -301,6 +301,23 @@ test_that(".harmonization_view_files guard is necessary: registration against a
on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
DBI::dbExecute(con2, .read_view_sql("10-long.sql"))
expect_error(DBI::dbExecute(con2, .read_view_sql("25-ig_long_harmonized.sql")))
# Reconciling this test with the C2 guard (expenditure-concept review):
# `ig_annotated`/`spending_annotated` registering cleanly above proves
# only that CREATE VIEW binds against a `summary_categories` with no M/L
# rows at all (this synthetic corpus's own summary_categories has a
# single E36 row, see the COPY above) -- a LEFT JOIN never fails to
# resolve regardless of what the joined-to table contains. It does NOT
# mean querying expenditure_concept = "total" against this shape is safe:
# exactly this corpus (schema_version reported as supported, but
# summary_categories predates the M/L rows cog_pipeline PR #59 added) is
# what .require_ig_categories() exists to catch at the *verb* level,
# since PR #59 shipped those rows with no schema_version bump. Confirm
# the new runtime guard actually fires against this same `con`.
expect_error(
uscogdata:::.require_ig_categories(con),
class = "uscogdata_ig_categories_unsupported"
)
})
test_that("spending_long filters to E/F/G/K prefixes and excludes aggregates", {