docs: survey_weight is col 28 under schema v6 (was col 26 in v5)

Rebase onto the v6 main (bd53230) shifted survey_weight from col 26 to
col 28: v5→v6 inserted cog_legacy_state/cog_legacy_county at positions
10-11 (26→28 cols). Position confirmed against the regenerated v6
fixture and both corpus docs (reader-specification.md §3 'Long parquet
schema (28 columns)' row 28; data_dictionary.md '28-column schema v6'
row 28).

.Rbuildignore

@@ -16,4 +16,3 @@
 ^Meta$
 ^\.gitea$
 ^CLAUDE\.md$
-^\.superpowers$

.gitignore

@@ -9,6 +9,3 @@ docs/
 /Meta/
 .DS_Store
 /.quarto/
-
-# SDD working artifacts (ledger, briefs, review packages) — plans/ stays tracked
-.superpowers/sdd/

NEWS.md

@@ -1,117 +1,5 @@
 # uscogdata 0.1.0 (development)
 
-## Multi-government aggregates now disclose their reporting coverage
-
-* The Census of Governments is a **complete census only in years ending in 2
-  and 7**; every other year is a sample, and the sample varies enormously. On
-  the bundled fixture, Wisconsin's 608-city universe rolls up **597**
-  governments in FY2012 and **112** in FY2019 — an 18%-to-98% swing the
-  return value said nothing about, so a statewide total resting on a fifth of
-  the universe looked exactly like one resting on all of it.
-* `cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` gain
-  `coverage`:
-
-  | value | effect |
-  |---|---|
-  | `"all"` (default) | every unit that reported that year — unchanged behaviour |
-  | `"census"` | census years only; aborts if the range holds none rather than returning nothing |
-  | `"consistent"` | only units reporting in *every* requested year — a balanced panel |
-
-* **Regardless of mode**, every result now carries `provenance$coverage` with
-  per-year `n_units_reporting`, `n_units_expected` and `is_census_year`, plus
-  `provenance$coverage_mode`. `cog_explain()` prints a "Reporting coverage"
-  section. So the default mode can no longer mislead silently.
-* `is_census_year` is a statement about the **survey calendar**, never a claim
-  of completeness: FY1967 is a census year in which only 97 of Wisconsin's 608
-  cities report. `n_units_reporting` is the number that tells the truth.
-* On `cog_peer_compare()` the target is exempt from `"consistent"` balancing —
-  it is the subject of the comparison, not a member of the cohort — and the
-  `summary_*` quantiles are computed after the filter, so they describe the
-  cohort actually returned. `n_units_reporting` counts peers only, against the
-  cohort size.
-* On `cog_find_peers()`, `coverage` governs the cohort **vintage** when `year`
-  is `NULL`: `"census"` snaps to the most recent census year with an observed
-  population, so a cohort is not built from a sample year in which most of the
-  candidate universe is absent.
-
-## `complete = TRUE`: absent cells, labelled with why they are absent
-
-* `cog_spending()` and `cog_revenue()` gain `complete`, defaulting to `FALSE`
-  (today's behaviour). With `complete = TRUE` the requested grid is filled
-  from the corpus's `code_set` table and every row carries a new
-  `value_source` column:
-
-  | `value_source` | meaning | `amt_nominal` |
-  |---|---|---|
-  | `reported` | the corpus carries this cell | as published |
-  | `census_zero` | dense-source year (≤ FY2011), cell absent — Census published `$0` | `0` |
-  | `not_reported` | sparse-source year (≥ FY2012), cell absent — unknown | `NA` |
-
-  The `NA` is deliberate and is the whole point: filling a modern absence
-  with `0` would invent data, which is precisely the error the corpus's
-  representation contract exists to prevent.
-* This restores information the reader lost when the corpus was sparsified
-  (`SB194`, cog_pipeline#64) — a wide-era query whose cells were all `$0`
-  had begun returning nothing at all — and improves on what came before it,
-  since the pre-sparsification corpus could not distinguish a published zero
-  from an unreported cell either.
-* The grid is scoped to each government's **own type**, so a county is never
-  filled with cells only a state can report.
-* Needs a corpus published from 2026-07-29 onward (when `representation` and
-  `code_set` began shipping); aborts with class
-  `uscogdata_representation_unavailable` otherwise. Gated on the manifest
-  listing those tables rather than on `schema_version`, which was never
-  bumped for the change. Not available with `recipe` or
-  `expenditure_concept = "total"` — neither draws its cells from `code_set`.
-* `provenance$completion` reports `applied`, `rows_filled`, and the per-year
-  `absence_means` rule; `cog_explain()` prints a "Completion" section.
-
-## Corpus-wide series breaks now reach users (`corpus_break_refs`)
-
-* Four catalogued series breaks carry `fin_code = "ALL"` — caveats about the
-  corpus as a whole rather than about one item code. `series_break_refs` is
-  built by matching `fin_code` against the item codes in the result, and no
-  row's `item_code` is ever the literal `"ALL"`, so **none of them could ever
-  be surfaced**: `SB085` (dollar precision across the 1976/1977 boundary),
-  `SB087` (imputation exclusion from FY2002), `SB194` (the dense → sparse
-  representation change at FY2012) and `SB086` (the government id scheme
-  change at FY2017).
-* Provenance gains `corpus_break_refs`, selected on the break-year window
-  alone and disjoint from `series_break_refs` by construction, so a consumer
-  can tell a whole-result caveat from a break in one series. `cog_explain()`
-  prints them under their own "Corpus-wide caveats" heading. cog-api passes
-  provenance through verbatim, so the field appears there without an API
-  change.
-* `SB194` is the one that made this urgent: a query spanning FY2011 → FY2012
-  crosses the boundary where an absent cell stops meaning "Census published
-  `$0`" and starts meaning "not reported", and until now nothing said so.
-
-## Bundled fixture regenerated against the sparsified corpus
-
-* `inst/extdata/fixture_corpus/` now tracks the corpus published on
-  2026-07-29 (`pipeline_commit 83f9715`, schema v6). The wide era no longer
-  stores explicit zeros: FY2011 fell from 2,864,212 rows to 496,004, of
-  which none are `$0`. **Absence now means two different things** — in a
-  `dense_source` year (≤ FY2011) an absent cell means Census published `$0`;
-  in a `sparse_source` year (≥ FY2012) it means not reported. The corpus
-  carries that rule in two new tables the fixture now ships,
-  `representation.parquet` and `code_set.parquet`, alongside
-  `census_collection_coverage.parquet` and `lineage_events.parquet`
-  (all ten publish-tree metadata tables, up from six). Catalogued upstream
-  as series break `SB194`.
-* `cog_categories()` gains an `assistance` spending subtype: the J-prefix
-  aid/benefit codes (`J19`, `J67`, `J68`, `J85`) are categorised now that
-  the upstream crosswalk covers every flow code carrying dollars.
-* Two consequences worth knowing about, both visible in provenance rather
-  than in returned dollars. The harmonization block's `na_rows_excluded`
-  counts only rows that exist, so wide-era codes that were zero-padded no
-  longer appear there. Coverage-gap `suggestions` are presence-based for the
-  same reason, so a recipe whose component codes were all `$0` for a given
-  government-year is no longer suggested for it.
-* `tests/testthat/test-fixture-vintage.R` pins these structural facts, so a
-  fixture left behind by a future publish fails loudly instead of letting the
-  suite pass against a corpus that no longer exists.
-
 ## Breaking: corpus schema_version 4 (Phase P canonical ids)
 
 * The package now requires corpus `schema_version = 4` (`MinCorpusSchema` /

R/complete.R

@@ -1,148 +0,0 @@
-# R/complete.R
-#
-# `complete = TRUE` on the money verbs. Fills the requested grid so that a
-# cell the corpus does not carry still appears, labelled with WHY it is
-# missing.
-#
-# The corpus stopped storing the wide era's explicit zeros
-# (cog_pipeline#64, series break SB194), which made absence ambiguous:
-#
-#   <= FY2011  dense_source   absent => Census published $0    (census_zero)
-#   >= FY2012  sparse_source  absent => not reported, unknown  (not_reported)
-#
-# Before sparsification a wide-era query whose cells were all $0 came back as
-# explicit $0 rows; afterwards it came back empty, with nothing to say which
-# of the two meanings applied. This restores that -- and improves on it,
-# because the pre-sparsification corpus could not distinguish the two either.
-#
-# `census_zero` fills carry `amt_nominal = 0`; `not_reported` fills carry NA.
-# That difference is the entire point: writing 0 into a modern absence would
-# invent data, which is the error the representation contract exists to stop.
-
-#' @noRd
-.abort_complete_unsupported <- function(reason, alternative) {
-  cli::cli_abort(c(
-    "{.code complete = TRUE} is not supported for this query.",
-    x = reason,
-    i = alternative
-  ), class = "uscogdata_complete_unsupported")
-}
-
-#' @noRd
-.require_representation <- function(con, manifest) {
-  needed <- c("representation.parquet", "code_set.parquet")
-  missing <- needed[!vapply(needed, function(f) .corpus_has_table(manifest, f),
-                            logical(1))]
-  if (length(missing) == 0L) return(invisible(TRUE))
-  cli::cli_abort(c(
-    "This corpus does not publish the representation contract.",
-    x = "Missing: {.file {missing}}.",
-    i = "{.code complete = TRUE} needs those tables to know whether an absent cell means Census published $0 or means the government did not report.",
-    i = "They ship with corpora published from 2026-07-29 onward; re-point {.envvar USCOGDATA_URL} at a current corpus, or omit {.code complete}."
-  ), class = "uscogdata_representation_unavailable")
-}
-
-#' The cells a government-year COULD carry: every code in force for that
-#' government's own type, mapped through `summary_categories`, restricted to
-#' the calling verb's flow prefixes and (when given) its category filter.
-#'
-#' Scoped by `govs_type` deliberately. Filling against the union of all types
-#' would invent cells that the government can never report -- a county row for
-#' "state IG transfer to school districts" -- and those inventions would then
-#' be indistinguishable from real census zeros.
-#'
-#' `NOT cs.is_aggregate` mirrors `spending_long` / `revenue_long`, which drop
-#' aggregate rows. Without it the grid would offer cells the verb structurally
-#' never returns, so every one of them would fill as a phantom $0.
-#' @noRd
-.completion_grid_sql <- function(subtype_col, govid, years, category,
-                                 flow_prefixes) {
-  category_pred <- if (is.null(category)) {
-    ""
-  } else {
-    sprintf("AND c.category IN (%s)", .sql_lit_chr(category))
-  }
-  sprintf(
-    "SELECT DISTINCT
-       cs.year,
-       x.canonical_govid,
-       x.gov_name,
-       c.%1$s AS subtype_value,
-       c.category,
-       r.absence_means
-     FROM code_set cs
-     JOIN canonical_fips_xwalk x ON x.govs_type = cs.type
-     JOIN summary_categories c   ON c.item_code = cs.item_code
-     JOIN representation r       ON r.year = cs.year
-     WHERE x.canonical_govid IN (%2$s)
-       AND cs.year IN (%3$s)
-       AND NOT cs.is_aggregate
-       AND LEFT(cs.item_code, 1) IN (%4$s)
-       AND c.category IS NOT NULL
-       AND c.%1$s IS NOT NULL
-       %5$s",
-    subtype_col, .sql_lit_chr(govid),
-    paste(as.integer(years), collapse = ","),
-    .sql_lit_chr(flow_prefixes), category_pred
-  )
-}
-
-#' Fill `result` out to the full grid, stamping `value_source` on every row.
-#'
-#' Returns the completed tibble with a `.completion` attribute carrying the
-#' provenance block. Reported rows are passed through untouched -- filling
-#' must never alter or drop what the corpus actually published.
-#' @noRd
-.complete_result <- function(result, con, subtype_col, govid, years, category,
-                             flow_prefixes) {
-  grid <- tibble::as_tibble(DBI::dbGetQuery(
-    con, .completion_grid_sql(subtype_col, govid, years, category, flow_prefixes)
-  ))
-
-  result$value_source <- rep("reported", nrow(result))
-  if (nrow(grid) == 0L) {
-    attr(result, ".completion") <- list(
-      applied = TRUE, rows_filled = 0L, absence_means = list()
-    )
-    return(result)
-  }
-
-  names(grid)[names(grid) == "subtype_value"] <- subtype_col
-  key <- function(d) {
-    paste(d$year, d$canonical_govid, d[[subtype_col]], d$category, sep = "\r")
-  }
-  missing <- grid[!key(grid) %in% key(result), , drop = FALSE]
-
-  if (nrow(missing) > 0L) {
-    filled <- tibble::tibble(
-      year            = as.integer(missing$year),
-      canonical_govid = as.character(missing$canonical_govid),
-      gov_name        = as.character(missing$gov_name),
-      category        = as.character(missing$category),
-      # census_zero is a value Census published; not_reported is unknown and
-      # must stay NA. Collapsing the two to 0 is the defect, not the fill.
-      amt_nominal     = ifelse(missing$absence_means == "census_zero",
-                               0, NA_real_),
-      codes_included  = NA_character_,
-      aggregate_fallback = NA,
-      value_source    = as.character(missing$absence_means)
-    )
-    filled[[subtype_col]] <- as.character(missing[[subtype_col]])
-    if ("notes" %in% names(result)) filled$notes <- NA_character_
-
-    result <- dplyr::bind_rows(result, filled)
-    result <- result[order(result$year, result$canonical_govid,
-                           result[[subtype_col]], result$category), ,
-                     drop = FALSE]
-  }
-
-  rules <- unique(grid[, c("year", "absence_means")])
-  attr(result, ".completion") <- list(
-    applied = TRUE,
-    rows_filled = nrow(missing),
-    absence_means = stats::setNames(
-      as.list(as.character(rules$absence_means)), as.character(rules$year)
-    )
-  )
-  result
-}

R/config.R

@@ -21,30 +21,7 @@
   .uscogdata_defaults[[key]]
 }
 
-#' Resolve the corpus URL, guaranteeing the trailing slash the package assumes.
+.resolve_url <- function() .cfg("url")
-#'
-#' Every consumer builds locations by CONCATENATION -- `paste0(url,
-#' "manifest.json")` in manifest.R, `paste0(url, e$path)` in mirror.R, and the
-#' parquet glob in views.R -- and mirror.R:104 documents the invariant outright
-#' ('url ends in "/"'). Nothing enforced it, so a URL entered without the slash
-#' failed silently and misleadingly:
-#'
-#'   HTTPS -> ".../downloadmanifest.json"; the host answers with an HTML 404
-#'            page, which lands in the JSON parser as the lexical error
-#'            reported in issue #3 -- pointing the user at "login page / wrong
-#'            share" when the real cause was one missing character.
-#'   local -> ".../corpusdata/long/**/*.parquet" and a DuckDB "No files found".
-#'
-#' Normalizing here fixes every consumer at once, rather than each call site
-#' re-deriving the same invariant. An empty setting is passed through
-#' untouched so manifest.R's "not configured" guard still fires instead of the
-#' value degrading into a bare "/" filesystem root.
-#' @noRd
-.resolve_url <- function() {
-  url <- .cfg("url")
-  if (is.null(url) || !nzchar(url) || grepl("/$", url)) return(url)
-  paste0(url, "/")
-}
 
 .resolve_cache_dir <- function() {
   v <- .cfg("cache_dir")

R/coverage.R

@@ -1,107 +0,0 @@
-# R/coverage.R
-#
-# Reporting-coverage disclosure for the multi-government verbs (uscogdata#13,
-# findings F-020 and F-023).
-#
-# The Census of Governments is a COMPLETE CENSUS only in years ending in 2 and
-# 7. Every other year is a sample, and the sample varies enormously: on the
-# bundled fixture, Wisconsin's 608-city universe reports 597 governments in
-# FY2012 and 112 in FY2019. Summing "whatever reported" across those years is
-# what the verbs have always done -- correctly -- but the return value said
-# nothing about it, so a statewide total resting on 18% of the universe looked
-# exactly like one resting on 98%.
-#
-# Owner's settled design: a `coverage` argument selecting WHICH units to
-# include, plus always-on metadata saying how many there were either way. The
-# principle behind it: using these verbs correctly must not require the caller
-# to know the survey calendar.
-
-# Years ending in 2 or 7 are full censuses of every government; all others are
-# samples.
-.CENSUS_YEAR_ENDINGS <- c(2L, 7L)
-
-#' @noRd
-.is_census_year <- function(years) {
-  as.integer(years) %% 10L %in% .CENSUS_YEAR_ENDINGS
-}
-
-#' @noRd
-.validate_coverage <- function(coverage) {
-  tryCatch(
-    match.arg(coverage, c("all", "census", "consistent")),
-    error = function(e) {
-      cli::cli_abort(
-        "`coverage` must be one of {.val all}, {.val census} or {.val consistent}.",
-        class = "uscogdata_invalid_coverage", parent = e
-      )
-    }
-  )
-}
-
-#' Restrict `years` to census years for `coverage = "census"`.
-#'
-#' Aborts rather than returning an empty result when the requested range holds
-#' no census year: silently handing back zero rows for a query the caller
-#' believes they made is the failure mode this whole issue is about.
-#' @noRd
-.apply_census_years <- function(years, coverage, verb) {
-  if (!identical(coverage, "census")) return(as.integer(years))
-  keep <- as.integer(years)[.is_census_year(years)]
-  if (length(keep) == 0L) {
-    cli::cli_abort(c(
-      "{.code coverage = \"census\"} leaves no years to query.",
-      x = "None of the requested years end in 2 or 7: {.val {sort(unique(as.integer(years)))}}.",
-      i = "Census of Governments years ending in 2 or 7 are complete censuses; all others are samples.",
-      i = "Use {.code coverage = \"all\"} (the default) to keep every requested year, or request a census year."
-    ), class = "uscogdata_no_census_years")
-  }
-  sort(keep)
-}
-
-#' Keep only units that report in EVERY requested year (a balanced panel).
-#'
-#' `id_col` is the government identifier; `keep_ids` are rows exempt from the
-#' filter (the peer-comparison target, which is the subject of the comparison
-#' rather than a member of the cohort being balanced).
-#' @noRd
-.filter_consistent <- function(result, years, id_col = "canonical_govid",
-                               keep_ids = character(0)) {
-  years <- unique(as.integer(years))
-  if (nrow(result) == 0L || length(years) <= 1L) return(result)
-  ids <- setdiff(unique(result[[id_col]]), c(NA, keep_ids))
-  present <- vapply(ids, function(g) {
-    all(years %in% unique(as.integer(result$year[result[[id_col]] == g])))
-  }, logical(1))
-  consistent <- c(ids[present], keep_ids)
-  result[result[[id_col]] %in% consistent | is.na(result[[id_col]]), ,
-         drop = FALSE]
-}
-
-#' Per-year coverage metadata, always attached regardless of mode.
-#'
-#' Built from the REQUESTED years rather than the years present in the result,
-#' so a year in which nothing reported still appears -- with
-#' `n_units_reporting = 0`, which is precisely the disclosure a silently
-#' missing year fails to make.
-#'
-#' `n_units_reporting` describes the result the caller actually received, so
-#' under `coverage = "consistent"` it reports the balanced count. `is_census_year`
-#' is a statement about the SURVEY CALENDAR, never a claim of completeness:
-#' FY1967 is a census year in which only 97 of Wisconsin's 608 cities report.
-#' `n_units_reporting` is the number that tells the truth.
-#' @noRd
-.coverage_table <- function(result, years, n_expected,
-                            id_col = "canonical_govid", rows = NULL) {
-  years <- sort(unique(as.integer(years)))
-  src <- if (is.null(rows)) result else rows
-  reporting <- vapply(years, function(y) {
-    ids <- src[[id_col]][as.integer(src$year) == y]
-    length(unique(ids[!is.na(ids)]))
-  }, integer(1))
-  tibble::tibble(
-    year              = years,
-    n_units_reporting = as.integer(reporting),
-    n_units_expected  = rep(as.integer(n_expected), length(years)),
-    is_census_year    = .is_census_year(years)
-  )
-}

R/explain.R

@@ -60,21 +60,6 @@ cog_explain <- function(result, format = c("print", "list")) {
     cli::cli_text("Basis: {prov$basis}{note}")
   }
 
-  if (!is.null(prov$expenditure_concept)) {
-    concept_note <- if (!is.null(prov$expenditure_concept_note) &&
-                         !is.na(prov$expenditure_concept_note)) {
-      sprintf(" (%s)", prov$expenditure_concept_note)
-    } else {
-      ""
-    }
-    cli::cli_text("Concept: {prov$expenditure_concept}{concept_note}")
-    if (isTRUE(prov$expenditure_concept_direct_suppressed)) {
-      cli::cli_alert_warning(
-        "Direct leg unavailable for at least one requested (year, category) -- affected rows report intergovernmental dollars alone, not Direct + IG. See each row's notes."
-      )
-    }
-  }
-
   cli::cli_h2("Codes observed")
   codes <- prov$codes_summed$observed
   if (length(codes) == 0L) {
@@ -118,54 +103,11 @@ cog_explain <- function(result, format = c("print", "list")) {
     cli::cli_ul(sugg_lines)
   }
 
-  if (!is.null(prov$coverage) && nrow(prov$coverage) > 0L) {
-    cli::cli_h2("Reporting coverage")
-    cli::cli_text("Mode: {prov$coverage_mode %||% 'all'}")
-    cov <- prov$coverage
-    cli::cli_ul(sprintf(
-      "%d: %d of %d units reporting (%.0f%%) -- %s year",
-      cov$year, cov$n_units_reporting, cov$n_units_expected,
-      100 * cov$n_units_reporting / pmax(cov$n_units_expected, 1L),
-      ifelse(cov$is_census_year, "census", "sample")
-    ))
-    if (any(!cov$is_census_year)) {
-      cli::cli_text(
-        "Note: the Census of Governments is a complete census only in years ending in 2 or 7; every other year is a sample."
-      )
-    }
-  }
-
-  if (isTRUE(prov$completion$applied)) {
-    cli::cli_h2("Completion")
-    cli::cli_text(
-      "Filled {prov$completion$rows_filled} absent cell(s) from the corpus code set."
-    )
-    rules <- prov$completion$absence_means
-    if (length(rules) > 0L) {
-      cli::cli_ul(vapply(names(rules), function(y) {
-        sprintf("%s: an absent cell means %s", y,
-                if (identical(rules[[y]], "census_zero")) {
-                  "Census published $0 (filled as 0)"
-                } else {
-                  "the government did not report (filled as NA, not 0)"
-                })
-      }, character(1)))
-    }
-  }
-
   if (length(prov$series_break_refs) > 0L) {
     cli::cli_h2("Series breaks")
     cli::cli_ul(.series_break_story_lines(prov$series_break_refs))
   }
 
-  # Kept in a section of its own: these qualify the whole result, so folding
-  # them in with the per-code breaks above would invite reading them as a
-  # caveat about one series.
-  if (length(prov$corpus_break_refs) > 0L) {
-    cli::cli_h2("Corpus-wide caveats")
-    cli::cli_ul(.series_break_story_lines(prov$corpus_break_refs))
-  }
-
   cli::cli_h2("Transformations")
   uc <- prov$transformations$units_conversion
   if (isTRUE(uc$applied)) {

R/peers.R

@@ -19,13 +19,6 @@
 #'   target's population at `year` to produce absolute bounds. If `FALSE`,
 #'   `pop_range` is interpreted as absolute population counts.
 #' @param max_peers Integer cap on the number of peers returned.
-#' @param coverage Survey-cycle handling; see [cog_peer_compare()]. Here it
-#'   governs the cohort VINTAGE when `year` is `NULL`: `"census"` snaps to the
-#'   most recent census year with an observed population, so a cohort is not
-#'   built from a sample year in which most of the candidate universe is
-#'   absent. `"consistent"` needs a year range, which cohort selection does not
-#'   have, so it selects like `"all"` and is carried on the result as
-#'   `attr(x, "coverage")` for [cog_peer_compare()].
 #' @return Tibble with columns `canonical_govid`, `gov_name`, `fips_state`,
 #'   `population`, `pop_ratio`, `rank`. The cohort year is attached as
 #'   `attr(x, "cohort_year")`.
@@ -36,9 +29,7 @@ cog_find_peers <- function(target_govid,
                            same_state = FALSE,
                            pop_range = c(0.7, 1.3),
                            is_ratio = TRUE,
-                           max_peers = 10L,
+                           max_peers = 10L) {
-                           coverage = c("all", "census", "consistent")) {
-  coverage <- .validate_coverage(coverage)
   if (!is.character(target_govid) || length(target_govid) != 1L) {
     cli::cli_abort("`target_govid` must be a length-1 character string.")
   }
@@ -68,7 +59,7 @@ cog_find_peers <- function(target_govid,
     ))
   }
 
-  cohort_year <- .resolve_cohort_year(con, target_govid, year, coverage)
+  cohort_year <- .resolve_cohort_year(con, target_govid, year)
 
   pop_sql <- sprintf(
     "SELECT population FROM gov_population_yearly
@@ -116,34 +107,12 @@ cog_find_peers <- function(target_govid,
   attr(peers, "cohort_year") <- as.integer(cohort_year)
   attr(peers, "pop_range")   <- as.numeric(pop_range)
   attr(peers, "is_ratio")    <- isTRUE(is_ratio)
-  attr(peers, "coverage")    <- coverage
-  attr(peers, "is_census_year") <- .is_census_year(cohort_year)
   peers
 }
 
-# `coverage` picks the cohort vintage when the caller did not name one.
-# "census" snaps to the most recent CENSUS year with an observed population,
-# so a cohort is not silently built from a sample year in which most of the
-# candidate universe is absent. "consistent" is a comparison-time concept --
-# it needs a year RANGE, which cohort selection does not have -- so it selects
-# like "all" here and is carried on the result for cog_peer_compare().
 #' @noRd
-.resolve_cohort_year <- function(con, target_govid, year,
+.resolve_cohort_year <- function(con, target_govid, year) {
-                                 coverage = "all") {
   if (!is.null(year)) return(as.integer(year))
-  if (identical(coverage, "census")) {
-    sql <- sprintf(
-      "SELECT MAX(year) AS y FROM gov_population_yearly
-       WHERE canonical_govid = %s AND year %% 10 IN (2, 7)",
-      .sql_lit_chr(target_govid)
-    )
-    y <- DBI::dbGetQuery(con, sql)$y
-    if (length(y) > 0L && !is.na(y)) return(as.integer(y))
-    cli::cli_abort(c(
-      "{.code coverage = \"census\"} found no census year with an observed population for {target_govid}.",
-      i = "Pass an explicit {.arg year}, or use {.code coverage = \"all\"}."
-    ), class = "uscogdata_no_census_years")
-  }
   sql <- sprintf(
     "SELECT MAX(year) AS y FROM gov_population_yearly
      WHERE canonical_govid = %s",
@@ -164,9 +133,7 @@ cog_find_peers <- function(target_govid,
 #' [cog_find_peers()] result or a character vector of `canonical_govid`) and
 #' appends peer-distribution summary rows (`summary_p25`, `summary_p50`,
 #' `summary_p75`) so the result can be faceted by `role` in a single ggplot
-#' call. Those summary rows are quantiles **within each category**, not
+#' call.
-#' quantiles of each peer's total — see the `@return` section before summing
-#' them.
 #'
 #' @param target_govid Character scalar.
 #' @param peers A tibble from [cog_find_peers()] or a character vector of
@@ -176,35 +143,6 @@ cog_find_peers <- function(target_govid,
 #' @param per_capita Default `TRUE` — peer compare usually normalizes by
 #'   population.
 #' @param adjust_to_year Integer base year for CPI-U conversion or `NULL`.
-#' @param expenditure_concept `"direct"` (default) or `"total"`. Currently only
-#'   `"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
-#'   single-government queries but cannot be used here because combining Total
-#'   across peer sets counts intergovernmental transfers twice.
-#' @param coverage How to handle the Census of Governments survey cycle,
-#'   which is a **complete census only in years ending in 2 and 7** -- every
-#'   other year is a sample, and the sample varies enormously (on the bundled
-#'   fixture, Wisconsin's 608-city universe reports 597 governments in FY2012
-#'   and 112 in FY2019).
-#'
-#'   * `"all"` (default) -- every unit that reported that year. Unchanged
-#'     behaviour, so existing code keeps working.
-#'   * `"census"` -- census years only. Aborts if the requested range holds
-#'     none, rather than silently returning nothing.
-#'   * `"consistent"` -- only units reporting in *every* requested year, giving
-#'     a balanced panel.
-#'
-#'   Regardless of mode, `provenance$coverage` always carries per-year
-#'   `n_units_reporting`, `n_units_expected` and `is_census_year`, and
-#'   `provenance$coverage_mode` records the mode. `is_census_year` is a
-#'   statement about the **survey calendar**, never a claim of completeness:
-#'   FY1967 is a census year in which only 97 of Wisconsin's 608 cities
-#'   report. `n_units_reporting` is the number that tells the truth.
-#'
-#'   The comparison target is exempt from `"consistent"` balancing -- it is the
-#'   subject of the comparison, not a member of the cohort -- and the
-#'   `summary_*` quantiles are computed AFTER the filter, so they describe the
-#'   cohort actually returned. `n_units_reporting` counts peers only, against
-#'   the cohort size: "3 of your 15 peers reported in FY2019".
 #' @return Tibble matching [cog_spending()]'s columns, plus a `role`
 #'   column taking values `"target"`, `"peer"`, `"summary_p25"`,
 #'   `"summary_p50"`, or `"summary_p75"`, `target_rank` (target's rank
@@ -213,43 +151,10 @@ cog_find_peers <- function(target_govid,
 #'   `attr(peers, "cohort_year")`; `NA` when `peers` was a bare character
 #'   vector). Provenance reports `verb = "cog_peer_compare"`, `peer_count`,
 #'   `cohort_year`, and `cohort_govids`.
-#'
-#'   **The `summary_*` rows are per-category quantiles: they are not additive.**
-#'   Each one is computed **within each `(year, spend_subtype,
-#'   category)` cell** across the peer set, so a `summary_p50` row is *the
-#'   median peer's value in that one category*, not *the value of the median
-#'   peer's total*. The median peer for Police and the median peer for Fire
-#'   are usually different governments, so summing `summary_*` rows across
-#'   categories does not give any peer's total and misstates the band it
-#'   appears to describe — measured at −32.7% to +251.0% across 24 years on
-#'   one cohort, with a sign flip at FY2012.
-#'
-#'   Facet by `role` **and** `category` (the documented use, and what the
-#'   rows are built for). For a genuine "median peer's total spending" line,
-#'   sum each peer's own categories first and take the quantile of those
-#'   per-government totals:
-#'
-#'   ```r
-#'   library(dplyr)
-#'   cmp |>
-#'     filter(role %in% c("target", "peer")) |>
-#'     group_by(year, role, canonical_govid) |>
-#'     summarise(total = sum(amt_per_capita_real, na.rm = TRUE), .groups = "drop") |>
-#'     filter(role == "peer") |>
-#'     group_by(year) |>
-#'     summarise(p50 = quantile(total, 0.5, na.rm = TRUE))
-#'   ```
 #' @export
 cog_peer_compare <- function(target_govid, peers, category, years,
-                             per_capita = TRUE, adjust_to_year = NULL,
+                             per_capita = TRUE, adjust_to_year = NULL) {
-                             expenditure_concept = c("direct", "total"),
-                             coverage = c("all", "census", "consistent")) {
   call <- match.call()
-  expenditure_concept <- match.arg(expenditure_concept)
-  coverage <- .validate_coverage(coverage)
-  if (identical(expenditure_concept, "total")) {
-    .abort_concept_not_aggregatable("cog_peer_compare")
-  }
   if (!is.character(target_govid) || length(target_govid) != 1L) {
     cli::cli_abort("`target_govid` must be a length-1 character string.")
   }
@@ -269,20 +174,9 @@ cog_peer_compare <- function(target_govid, peers, category, years,
   peer_govids <- peer_govids[!is.na(peer_govids) & nzchar(peer_govids)]
   all_govids <- unique(c(target_govid, peer_govids))
 
-  years <- .apply_census_years(years, coverage, "cog_peer_compare")
-
   r <- cog_spending(all_govids, years, category, per_capita, adjust_to_year)
   r$role <- ifelse(r$canonical_govid == target_govid, "target", "peer")
 
-  # The target is exempt from balancing: it is the subject of the comparison,
-  # not a member of the cohort being balanced, and dropping it would leave a
-  # peer comparison with nothing to compare. Filtering happens BEFORE the
-  # quantiles below, so a "consistent" cohort's summary rows describe that
-  # cohort rather than the unbalanced one.
-  if (identical(coverage, "consistent")) {
-    r <- .filter_consistent(r, years, keep_ids = target_govid)
-  }
-
   value_col <- .peer_value_col(per_capita, adjust_to_year)
 
   summary_rows <- .peer_summary_rows(r, value_col)
@@ -303,14 +197,6 @@ cog_peer_compare <- function(target_govid, peers, category, years,
     canonical_govid = target_govid,
     gov_name        = unique(r$gov_name[r$role == "target"])
   )
-  # Counted over PEER rows only, against the cohort size: "3 of your 15 peers
-  # reported in FY2019". Including the target would inflate every count by one
-  # and make a cohort that has entirely stopped reporting look non-empty.
-  prov$coverage_mode <- coverage
-  prov$coverage <- .coverage_table(
-    out, years, length(peer_govids),
-    rows = r[r$role == "peer", , drop = FALSE]
-  )
   attr(out, "provenance") <- prov
   out
 }

R/provenance.R

@@ -6,12 +6,8 @@
                               per_capita, adjust_to_year, result, sql,
                               subtype_col, basis = NA_character_,
                               basis_note = NA_character_,
-                              expenditure_concept = "direct",
-                              expenditure_concept_note = NA_character_,
-                              expenditure_concept_direct_suppressed = FALSE,
                               harmonization = NULL, recipe = NULL,
-                              suggestions = list(),
+                              suggestions = list()) {
-                              completion = NULL) {
   manifest <- .uscogdata_env$manifest
 
   codes <- result[["codes_included"]]
@@ -38,20 +34,11 @@
 
   schema_version <- suppressWarnings(as.integer(manifest$schema_version %||% 0L))
   con <- .uscogdata_env$con
-  have_con <- !is.null(con) && DBI::dbIsValid(con)
+  break_refs <- if (!is.null(con) && DBI::dbIsValid(con)) {
-  break_refs <- if (have_con) {
     .build_series_break_refs(con, codes_observed, years, schema_version)
   } else {
     character(0)
   }
-  # Corpus-wide caveats travel separately: they qualify the whole result
-  # rather than one series, and they do not depend on codes_observed (see
-  # .build_corpus_break_refs()).
-  corpus_refs <- if (have_con) {
-    .build_corpus_break_refs(con, years, schema_version)
-  } else {
-    character(0)
-  }
 
   list(
     verb = verb,
@@ -64,9 +51,6 @@
     category = category,
     basis = basis,
     basis_note = basis_note,
-    expenditure_concept = expenditure_concept,
-    expenditure_concept_note = expenditure_concept_note,
-    expenditure_concept_direct_suppressed = isTRUE(expenditure_concept_direct_suppressed),
     harmonization = harmonization %||% list(
       applied = FALSE, na_rows_excluded = 0L, na_amount_excluded = 0,
       note = NA_character_
@@ -126,14 +110,6 @@
       )
     ),
     series_break_refs = break_refs,
-    corpus_break_refs = corpus_refs,
-    # What `complete = TRUE` filled, and the rule it filled by. Always
-    # present so a consumer can read `completion$applied` without testing
-    # for the key -- an absent block and applied = FALSE would otherwise be
-    # indistinguishable from an older reader version.
-    completion = completion %||% list(
-      applied = FALSE, rows_filled = 0L, absence_means = list()
-    ),
     manifest = list(
       schema_version = as.integer(manifest$schema_version),
       pipeline_commit = manifest$pipeline_commit %||% NA_character_,

R/revenue.R

@@ -11,13 +11,11 @@
 #' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
 #'   `revenue_subtype`, `category`, `amt_nominal`, optional `amt_real`,
 #'   optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
-#'   optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`,
+#'   optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`.
-#'   and `value_source` when `complete = TRUE`.
 #' @export
 cog_revenue <- function(govid, years, category = NULL,
                         per_capita = FALSE, adjust_to_year = NULL,
-                        basis = c("harmonized", "raw"), recipe = NULL,
+                        basis = c("harmonized", "raw"), recipe = NULL) {
-                        complete = FALSE) {
   .verb_spendrev(
     verb           = "cog_revenue",
     view_base      = "revenue_annotated",
@@ -30,7 +28,6 @@ cog_revenue <- function(govid, years, category = NULL,
     per_capita     = per_capita,
     adjust_to_year = adjust_to_year,
     basis          = basis,
-    recipe         = recipe,
+    recipe         = recipe
-    complete       = complete
   )
 }

R/rollup.R

@@ -25,31 +25,6 @@
 #'   population from `gov_population_yearly`. Govs with missing population
 #'   are excluded from the result.
 #' @param adjust_to_year Integer base year for CPI-U conversion, or `NULL`.
-#' @param expenditure_concept `"direct"` (default) or `"total"`. Currently only
-#'   `"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
-#'   single-government queries but cannot be used here because combining Total
-#'   across multiple layers of government double-counts intergovernmental
-#'   transfers (a state's payment to a school district is the same dollar the
-#'   district reports as its own Direct spending).
-#' @param coverage How to handle the Census of Governments survey cycle,
-#'   which is a **complete census only in years ending in 2 and 7** -- every
-#'   other year is a sample, and the sample varies enormously (on the bundled
-#'   fixture, Wisconsin's 608-city universe reports 597 governments in FY2012
-#'   and 112 in FY2019).
-#'
-#'   * `"all"` (default) -- every unit that reported that year. Unchanged
-#'     behaviour, so existing code keeps working.
-#'   * `"census"` -- census years only. Aborts if the requested range holds
-#'     none, rather than silently returning nothing.
-#'   * `"consistent"` -- only units reporting in *every* requested year, giving
-#'     a balanced panel.
-#'
-#'   Regardless of mode, `provenance$coverage` always carries per-year
-#'   `n_units_reporting`, `n_units_expected` and `is_census_year`, and
-#'   `provenance$coverage_mode` records the mode. `is_census_year` is a
-#'   statement about the **survey calendar**, never a claim of completeness:
-#'   FY1967 is a census year in which only 97 of Wisconsin's 608 cities
-#'   report. `n_units_reporting` is the number that tells the truth.
 #' @return Tibble with columns `year`, `layer`, `canonical_govid`, `gov_name`,
 #'   `spend_subtype`, `category`, `amt_nominal`, optional `amt_real` /
 #'   `amt_per_capita_nominal` / `amt_per_capita_real`, optional `pop_source`,
@@ -58,15 +33,8 @@
 #'   and `rollup$included_govids` / `rollup$excluded_govids`.
 #' @export
 cog_geographic_rollup <- function(govids, category, years,
-                                  per_capita = FALSE, adjust_to_year = NULL,
+                                  per_capita = FALSE, adjust_to_year = NULL) {
-                                  expenditure_concept = c("direct", "total"),
-                                  coverage = c("all", "census", "consistent")) {
   call <- match.call()
-  expenditure_concept <- match.arg(expenditure_concept)
-  coverage <- .validate_coverage(coverage)
-  if (identical(expenditure_concept, "total")) {
-    .abort_concept_not_aggregatable("cog_geographic_rollup")
-  }
   .validate_rollup_layers(govids)
 
   govids <- lapply(govids, .coerce_govid_input, arg = "govids[[layer]]")
@@ -80,20 +48,11 @@ cog_geographic_rollup <- function(govids, category, years,
     layer           = rep(layer_names, lengths(govids))
   )
 
-  # coverage = "census" drops non-census years BEFORE the query rather than
-  # after: a sample year's rows are not wanted at all, and fetching them only
-  # to discard them would also let them into the coverage table.
-  years <- .apply_census_years(years, coverage, "cog_geographic_rollup")
-
   r <- cog_spending(all_govids, years, category, per_capita, adjust_to_year)
   r <- dplyr::left_join(r, layer_map, by = "canonical_govid",
                         relationship = "many-to-many")
   r$scope_note <- .rollup_scope_note(r$layer)
 
-  if (identical(coverage, "consistent")) {
-    r <- .filter_consistent(r, years)
-  }
-
   excluded <- character(0)
   if (isTRUE(per_capita) && "pop_source" %in% names(r)) {
     drop <- r$pop_source == "unavailable"
@@ -112,11 +71,6 @@ cog_geographic_rollup <- function(govids, category, years,
     included_govids = included,
     excluded_govids = excluded
   )
-  # n_units_expected is the universe the CALLER named -- the govids passed in
-  # -- not the national universe. That is what makes the ratio meaningful:
-  # "597 of the 608 Wisconsin cities you asked about reported in FY2012".
-  prov$coverage_mode <- coverage
-  prov$coverage <- .coverage_table(r, years, length(unique(all_govids)))
   attr(r, "provenance") <- prov
 
   r

R/search.R

@@ -6,11 +6,8 @@
 #' the cross-vintage canonical-government registry. Operates in two modes:
 #'
 #' * **Utility mode** (single `name`, the original behavior): returns all
-#'   rows whose `gov_name` contains `name` as a **literal, case-insensitive
+#'   rows whose `gov_name` matches the regex case-insensitively, sorted by
-#'   substring**, sorted by `population_acs` descending. Useful for
+#'   `population_acs` descending. Useful for exploratory lookups.
-#'   exploratory lookups. Regex metacharacters in `name` are escaped, so a
-#'   government is findable by its own complete name even when that name
-#'   contains parentheses or a period.
 #' * **Basket mode** (`length(name) > 1`): resolves each input row to a
 #'   single canonical govid and returns a tibble in input order, suitable
 #'   for piping straight into [cog_spending()] / [cog_revenue()] /
@@ -22,8 +19,7 @@
 #' 1. Filter `canonical_fips_xwalk` by `state` and (if non-NA) `type`.
 #' 2. **Exact pass:** case-insensitive equality against `gov_name`.
 #'    Single hit -> resolved. Multiple -> step 4.
-#' 3. **Substring fallback:** case-insensitive literal substring against
+#' 3. **Substring fallback:** case-insensitive regex against `gov_name`.
-#'    `gov_name` (metacharacters escaped).
 #'    Single hit -> resolved (`match_method = "substring"`). Zero hits ->
 #'    `status = "no_match"`. Multiple hits -> step 4.
 #' 4. **Disambiguation:** if matches share one `govs_type`, pick the
@@ -52,7 +48,7 @@
 #'   [cog_spending()], [cog_revenue()].
 #' @examples
 #' \dontrun{
-#' # Utility mode — exploratory substring lookup
+#' # Utility mode — exploratory regex lookup
 #' cog_gov_search("broward", state = "FL")
 #'
 #' # Basket mode — resolve a known cohort
@@ -102,16 +98,9 @@ cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
     if (!is.character(name) || length(name) != 1L) {
       cli::cli_abort("`name` must be a length-1 character string.")
     }
-    # Escaped, so `name` is a literal case-insensitive substring -- the same
-    # treatment basket mode has always given it. Interpolating it raw made a
-    # government unfindable by its own name whenever that name contains a
-    # metacharacter (FREDONIA (BRISCOE) CITY), turned a bare "." into a
-    # match-everything wildcard, and let malformed pattern text reach the
-    # engine as an error -- which cog-api surfaced as a 500, reachable by
-    # typing a real name one character at a time (uscogdata#16, F-025).
     preds <- c(preds,
                sprintf("regexp_matches(gov_name, %s, 'i')",
-                       .sql_lit_chr(.escape_regex(name))))
+                       .sql_lit_chr(name)))
   }
   if (!is.null(state)) {
     st_fips <- .coerce_state_to_fips(state)
@@ -147,9 +136,8 @@ cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
 #' @noRd
 .escape_regex <- function(x) {
   # Backslash-escape POSIX regex metacharacters so `name` is treated as a
-  # literal substring in the DuckDB regexp_matches call. Used by BOTH modes:
+  # literal substring in the DuckDB regexp_matches call (substring fallback
-  # utility mode used to interpolate raw, which was a defect rather than a
+  # only; utility-mode intentionally preserves regex behavior).
-  # feature -- see the call site and uscogdata#16.
   gsub("([\\^$.|?*+(){}\\[\\]])", "\\\\\\1", x, perl = TRUE)
 }
 

R/series_breaks.R

@@ -14,41 +14,9 @@
   sql <- sprintf(
     "SELECT DISTINCT break_id
      FROM series_breaks_pq
-     WHERE fin_code IN (%s) AND fin_code <> 'ALL'
+     WHERE fin_code IN (%s) AND break_year BETWEEN %d AND %d
-       AND break_year BETWEEN %d AND %d
      ORDER BY break_id",
     .sql_lit_chr(codes_observed), min(as.integer(years)), max(as.integer(years))
   )
   DBI::dbGetQuery(con, sql)$break_id
 }
-
-#' Corpus-wide caveats: catalogued breaks whose `fin_code` is the literal
-#' `"ALL"` rather than an item code. They qualify the whole result, so they
-#' cannot be matched the way `.build_series_break_refs()` matches -- no row's
-#' `item_code` is ever `"ALL"`, which is exactly why they reached no user
-#' before uscogdata#19. Selection is on the break_year window alone: which
-#' codes a result happens to contain is irrelevant to a caveat about the
-#' corpus.
-#'
-#' All four catalogued entries are *boundary* caveats (dollar precision
-#' across 1976/1977, imputation exclusion from 2002, the dense -> sparse
-#' representation change at 2012, the id scheme change at 2017), so the same
-#' `break_year BETWEEN min(years) AND max(years)` rule the code-specific
-#' path uses is the right one -- a request that never crosses the boundary
-#' is not affected by it.
-#'
-#' Returned separately from `series_break_refs` so a consumer can tell a
-#' whole-result caveat from a break in one series; the two are disjoint by
-#' construction.
-#' @noRd
-.build_corpus_break_refs <- function(con, years, schema_version) {
-  if (schema_version < 5L || length(years) == 0L) return(character(0))
-  sql <- sprintf(
-    "SELECT DISTINCT break_id
-     FROM series_breaks_pq
-     WHERE fin_code = 'ALL' AND break_year BETWEEN %d AND %d
-     ORDER BY break_id",
-    min(as.integer(years)), max(as.integer(years))
-  )
-  DBI::dbGetQuery(con, sql)$break_id
-}

R/spending.R

@@ -42,75 +42,20 @@
 #'   `basis = "recipe"` with an inert `harmonization` block (`applied =
 #'   FALSE`, pointing at the `recipe` block instead) rather than a
 #'   possibly-misleading `"harmonized"`/`"raw"` value.
-#' @param expenditure_concept `"direct"` (default) returns only the
-#'   government's own direct spending (item codes `E`/`F`/`G`), unchanged
-#'   from prior releases. `"total"` additionally UNIONs in the
-#'   intergovernmental leg -- payments to local governments (`M` codes) and
-#'   to the state government (`L` codes, excluding the `L--` family-total
-#'   rollup) -- so results gain rows with `spend_subtype ==
-#'   "intergovernmental"`. Requires the active corpus's `summary_categories`
-#'   to carry M/L rows (added by cog_pipeline PR #59); aborts with class
-#'   `uscogdata_ig_categories_unsupported` on an older corpus rather than
-#'   silently under-reporting. Mutually exclusive with `recipe` (a recipe
-#'   already defines its own component codes). **Do not sum `"total"`
-#'   results across levels of government** (e.g. state + county + city):
-#'   a state's `M12` payment to a school district is the same dollar the
-#'   district reports as its own direct `E12`, so summing both double-counts
-#'   it. This matters in particular with [cog_geographic_rollup()], which
-#'   sums across exactly that kind of multi-layer government set.
-#'
-#'   In the legacy wide era (<= FY2011), some functions are published ONLY
-#'   as an aggregate-flagged family total (e.g. Corrections' `E04`/`E05`
-#'   split), which the Direct leg excludes by construction but the IG leg
-#'   deliberately keeps (see `inst/sql/24-ig_long.sql`). For a `"total"`
-#'   query, any (year, category) where this leaves intergovernmental rows
-#'   with NO Direct counterpart is flagged: the affected rows' `notes`
-#'   name the harmonization recipe that recovers the missing Direct
-#'   component (when one exists), and
-#'   `provenance$expenditure_concept_direct_suppressed` is `TRUE` -- the
-#'   figure in those rows is the intergovernmental leg alone, not Direct +
-#'   IG.
-#' @param complete If `TRUE`, fill the requested grid so that a cell the
-#'   corpus does not carry still appears, labelled with **why** it is
-#'   missing, and add a `value_source` column to every row:
-#'
-#'   * `"reported"` — the corpus carries this cell.
-#'   * `"census_zero"` — dense-source year (`<= FY2011`), cell absent:
-#'     Census published `$0`. `amt_nominal` is `0`.
-#'   * `"not_reported"` — sparse-source year (`>= FY2012`), cell absent: the
-#'     government did not report, and the value is unknown. `amt_nominal` is
-#'     `NA`, **not** `0` — writing a zero there would invent data.
-#'
-#'   The grid comes from the corpus's `code_set` table, scoped to each
-#'   government's own type, so a county is never filled with cells only a
-#'   state can report. Reported rows are passed through untouched.
-#'
-#'   Defaults to `FALSE` (the historical behaviour: absent cells simply do
-#'   not appear). Needs a corpus published from 2026-07-29 onward, which is
-#'   when `representation`/`code_set` began shipping; aborts with class
-#'   `uscogdata_representation_unavailable` otherwise. Not available with
-#'   `recipe` or with `expenditure_concept = "total"` (class
-#'   `uscogdata_complete_unsupported`) — neither draws its cells from
-#'   `code_set`.
 #' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
 #'   `spend_subtype`, `category`, `amt_nominal`, optional `amt_real`,
 #'   optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
-#'   optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`,
+#'   optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`.
-#'   and `value_source` when `complete = TRUE`.
+#'   Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`.
-#'   Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`,
-#'   whose `completion` block reports `applied`, `rows_filled`, and the
-#'   per-year `absence_means` rule that was applied.
 #' @export
 cog_spending <- function(govid, years, category = NULL,
                          per_capita = FALSE, adjust_to_year = NULL,
-                         basis = c("harmonized", "raw"), recipe = NULL,
+                         basis = c("harmonized", "raw"), recipe = NULL) {
-                         expenditure_concept = c("direct", "total"),
-                         complete = FALSE) {
   .verb_spendrev(
     verb          = "cog_spending",
     view_base     = "spending_annotated",
     subtype_col   = "spend_subtype",
-    flow_prefixes = c("E", "F", "G"),
+    flow_prefixes = c("E", "F", "G", "K"),
     call          = match.call(),
     govid         = govid,
     years         = years,
@@ -118,102 +63,28 @@ cog_spending <- function(govid, years, category = NULL,
     per_capita    = per_capita,
     adjust_to_year = adjust_to_year,
     basis         = basis,
-    recipe        = recipe,
+    recipe        = recipe
-    expenditure_concept = expenditure_concept,
-    complete      = complete
   )
 }
 
-#' @noRd
-.abort_concept_not_aggregatable <- function(verb) {
-  cli::cli_abort(c(
-    "{.code expenditure_concept = \"total\"} cannot be used in {.fn {verb}}.",
-    "*" = "Use {.code expenditure_concept = \"direct\"} (the default) for any \\
-           comparison or sum that spans more than one government.",
-    "i" = "Why: Census \"Total\" is a government's own Direct spending PLUS the \\
-           money it hands to other governments. The receiving government reports \\
-           that same dollar again as its own Direct when it actually spends it, \\
-           so combining Total across governments double-counts intergovernmental \\
-           transfers.",
-    "i" = "For one government's own Total, use \\
-           {.code cog_spending(expenditure_concept = \"total\")}."
-  ), class = "uscogdata_concept_not_aggregatable")
-}
-
 #' @noRd
 .verb_spendrev <- function(verb, view_base, subtype_col, flow_prefixes, call,
                            govid, years, category,
                            per_capita, adjust_to_year,
-                           basis = c("harmonized", "raw"), recipe = NULL,
+                           basis = c("harmonized", "raw"), recipe = NULL) {
-                           expenditure_concept = c("direct", "total"),
-                           complete = FALSE) {
   basis_explicit <- length(basis) == 1L
   basis <- match.arg(basis, c("harmonized", "raw"))
-  # match.arg() itself throws a base `simpleError`, not an rlang-classed
-  # condition; wrap it so an invalid expenditure_concept aborts consistently
-  # with the rest of this package's validation (cli::cli_abort -> rlang_error).
-  expenditure_concept <- tryCatch(
-    match.arg(expenditure_concept, c("direct", "total")),
-    error = function(e) {
-      cli::cli_abort(
-        "`expenditure_concept` must be one of {.val direct} or {.val total}.",
-        class = "uscogdata_invalid_expenditure_concept",
-        parent = e
-      )
-    }
-  )
 
   govid <- .coerce_govid_input(govid, arg = "govid")
   .validate_verb_inputs(govid, years, category, per_capita, adjust_to_year,
                         recipe)
 
-  if (!is.null(recipe) && identical(expenditure_concept, "total")) {
-    cli::cli_abort(c(
-      "`recipe` and `expenditure_concept = \"total\"` are mutually exclusive.",
-      i = "A recipe defines its own component codes; pass one or the other.",
-      i = "For a recipe's intergovernmental counterpart, use the matching IG recipe (e.g. `corrections_ig_local_combined`)."
-    ), class = "uscogdata_recipe_concept_conflict")
-  }
-
-  # .verb_spendrev() is shared with cog_revenue(), which never exposes
-  # expenditure_concept and always resolves it to "direct" -- so nothing on
-  # the public API can reach this today. But it's a cheap guard against a
-  # future call (direct or via a modified cog_revenue()) that would UNION
-  # the IG leg's expenditure M/L rows into a revenue result, which has no
-  # matching IG view and no sensible meaning.
-  if (identical(expenditure_concept, "total") &&
-      !identical(view_base, "spending_annotated")) {
-    cli::cli_abort(
-      paste0(
-        "`expenditure_concept = \"total\"` is only supported for spending ",
-        "(view_base = \"spending_annotated\"); got view_base = ",
-        "{.val {view_base}}."
-      ),
-      class = "uscogdata_expenditure_concept_unsupported"
-    )
-  }
-
-  complete <- isTRUE(complete)
-  if (complete && !is.null(recipe)) {
-    .abort_complete_unsupported(
-      "A recipe defines its own component codes and never goes through `summary_categories`, so there is no grid to fill from.",
-      "Query the recipe without `complete`, or use a category query with `complete = TRUE`."
-    )
-  }
-  if (complete && identical(expenditure_concept, "total")) {
-    .abort_complete_unsupported(
-      "The intergovernmental leg deliberately keeps aggregate-flagged rows (see `inst/sql/24-ig_long.sql`), so its cells are not the ones `code_set` describes.",
-      "Use `expenditure_concept = \"direct\"` with `complete = TRUE`, or drop `complete`."
-    )
-  }
-
   years   <- as.integer(years)
   if (!is.null(adjust_to_year)) adjust_to_year <- as.integer(adjust_to_year)
 
   con <- .ensure_session()
   manifest <- .uscogdata_env$manifest
   scope <- .check_govids_in_scope(govid)
-  if (complete) .require_representation(con, manifest)
 
   resolved <- .resolve_basis(basis, basis_explicit, manifest)
 
@@ -234,33 +105,17 @@ cog_spending <- function(govid, years, category = NULL,
     category_for_prov <- recipe_label
   } else {
     view <- .select_view(view_base, resolved$basis)
-    ig_view <- if (identical(expenditure_concept, "total")) {
+    sql <- .build_verb_sql(view, subtype_col, govid, years, category)
-      .require_ig_categories(con)
-      .select_ig_view(resolved$basis)
-    } else {
-      NULL
-    }
-    sql <- .build_verb_sql(view, subtype_col, govid, years, category, ig_view)
     result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
   }
 
-  # Fill BEFORE per_capita / inflation so the added cells get the same
-  # treatment as reported ones: a census_zero stays $0 per capita and in real
-  # dollars, and a not_reported stays NA through both rather than becoming a
-  # spurious 0.
-  completion <- list(applied = FALSE, rows_filled = 0L, absence_means = list())
-  if (complete) {
-    result <- .complete_result(result, con, subtype_col, govid, years,
-                               category, flow_prefixes)
-    completion <- attr(result, ".completion")
-    attr(result, ".completion") <- NULL
-  }
-
   if (per_capita) result <- .attach_per_capita(result, con, govid)
   if (!is.null(adjust_to_year)) {
     result <- .attach_real_dollars(result, adjust_to_year, per_capita)
   }
 
+  result$notes <- .notes_column(result)
+
   # A recipe result doesn't go through spending_annotated(_harmonized) /
   # revenue_annotated(_harmonized) at all -- .run_recipe()'s generic join
   # reads `long` directly -- so `basis` and the `harmonization` exclusion
@@ -286,64 +141,7 @@ cog_spending <- function(govid, years, category = NULL,
     harmonization <- .build_harmonization_block(
       con, govid, years, resolved, flow_prefixes
     )
-    # C1(a): gap detection must run against the Direct leg alone. `result`
+    suggestions <- .build_suggestions(con, govid, years, category, resolved$basis)
-    # can also carry UNION'd intergovernmental rows (expenditure_concept =
-    # "total"), and the wide era (<= FY2011) routinely has legacy IG dollars
-    # surviving (ig_long deliberately keeps aggregate rows) for a
-    # (year, category) whose legacy Direct dollars were suppressed (spending_
-    # long/spending_long_harmonized both filter NOT is_aggregate). Passing
-    # the UNION'd result here would let a surviving IG row count as coverage
-    # and silently cancel the recipe-hint suggestion that should fire.
-    direct_leg_result <- if (identical(expenditure_concept, "total")) {
-      result[!(result[[subtype_col]] %in% "intergovernmental"), , drop = FALSE]
-    } else {
-      result
-    }
-    suggestions <- .build_suggestions(con, govid, years, category,
-                                       direct_leg_result,
-                                       resolved$basis, flow_prefixes)
-  }
-
-  # C1(b): when expenditure_concept = "total", flag any row where the IG
-  # leg has dollars but the Direct leg has none for that same (year,
-  # canonical_govid, category) AND a harmonization recipe actually recovers
-  # the missing Direct dollars for that exact triple -- see
-  # .detect_direct_suppressed() for why bare Direct-row absence alone is NOT
-  # sufficient (the dominant real cause is a government that simply has no
-  # direct spending in that category, which is correct, ordinary data). When
-  # a covering recipe is found, both the row-level notes and the provenance
-  # say so rather than pass silently as a plausible Total.
-  direct_suppressed_info <- if (identical(expenditure_concept, "total")) {
-    .detect_direct_suppressed(con, result, subtype_col)
-  } else {
-    list(flag = rep(FALSE, nrow(result)), notes = rep(NA_character_, nrow(result)))
-  }
-  direct_suppressed <- direct_suppressed_info$flag
-  direct_suppressed_flag <- isTRUE(any(direct_suppressed))
-
-  result$notes <- .notes_column(result, direct_suppressed_info$notes)
-
-  # Determine expenditure_concept_note: only non-empty for "total", explains
-  # how the IG leg was assembled from legacy-era aggregates. When the Direct
-  # leg is suppressed for at least one requested (year, category), append an
-  # explicit warning rather than let the base note's "Total = Direct + IG"
-  # framing stand unqualified for rows where that arithmetic didn't happen.
-  expenditure_concept_note_for_prov <- if (identical(expenditure_concept, "total")) {
-    base_note <- "Total = Direct + intergovernmental (M to local govts + L to state govts). Legacy-era IG is assembled from aggregate-flagged rows, which are year-disjoint from their modern leaf components; the L-- family total is excluded."
-    if (direct_suppressed_flag) {
-      paste0(
-        base_note,
-        " NOTE: for at least one requested (year, category) the Direct leg ",
-        "has NO rows in this corpus (a legacy aggregate-only family) -- the ",
-        "affected result rows report the intergovernmental leg alone, not ",
-        "Direct + IG. See `expenditure_concept_direct_suppressed` and each ",
-        "affected row's `notes`."
-      )
-    } else {
-      base_note
-    }
-  } else {
-    NA_character_
   }
 
   prov <- .build_provenance(
@@ -359,13 +157,9 @@ cog_spending <- function(govid, years, category = NULL,
     subtype_col    = subtype_col,
     basis          = basis_for_prov,
     basis_note     = basis_note_for_prov,
-    expenditure_concept = expenditure_concept,
-    expenditure_concept_note = expenditure_concept_note_for_prov,
-    expenditure_concept_direct_suppressed = direct_suppressed_flag,
     harmonization  = harmonization,
     recipe         = recipe_block,
-    suggestions    = suggestions,
+    suggestions    = suggestions
-    completion     = completion
   )
   prov$scope$govids_found   <- scope$found
   prov$scope$govids_missing <- scope$missing
@@ -417,43 +211,6 @@ cog_spending <- function(govid, years, category = NULL,
   if (identical(basis, "harmonized")) paste0(view_base, "_harmonized") else view_base
 }
 
-#' @noRd
-.select_ig_view <- function(basis) {
-  if (identical(basis, "harmonized")) "ig_annotated_harmonized" else "ig_annotated"
-}
-
-#' Abort unless the active corpus's `summary_categories` actually carries
-#' intergovernmental (M/L) rows.
-#'
-#' The 66 M/L category rows arrived via cog_pipeline PR #59 with NO
-#' `schema_version` bump (`DESCRIPTION` still declares `MinCorpusSchema: 4`),
-#' so `schema_version` alone cannot gate `expenditure_concept = "total"` --
-#' a pre-#59 corpus can validly report schema_version 4, 5, or 6 and still
-#' have zero M/L rows in `summary_categories`. Against such a corpus,
-#' `ig_annotated`'s LEFT JOIN to `summary_categories` silently produces NA
-#' `category`/`spend_subtype` for every IG row: with a `category` filter
-#' this returns 0 rows (reads as "no intergovernmental spending" rather than
-#' "can't tell"), and with `category = NULL` every IG dollar collapses into
-#' one NA-subtype group that is invisible to the `spend_subtype ==
-#' "intergovernmental"` filter this package's own tests, roxygen, and
-#' vignette all rely on. Checking the data directly (rather than
-#' schema_version) is the only reliable gate.
-#' @noRd
-.require_ig_categories <- function(con, what = "expenditure_concept = \"total\"") {
-  n <- DBI::dbGetQuery(con,
-    "SELECT COUNT(*) AS n FROM summary_categories WHERE LEFT(item_code, 1) IN ('M', 'L')"
-  )$n
-  if (identical(as.integer(n), 0L)) {
-    cli::cli_abort(c(
-      sprintf("%s requires a corpus with intergovernmental category rows.", what),
-      x = "The active corpus's `summary_categories` has no M/L (intergovernmental) rows.",
-      i = "This corpus predates the intergovernmental category rows added by cog_pipeline PR #59.",
-      i = "Point USCOGDATA_URL at a newer corpus that includes the M/L summary_categories rows."
-    ), class = "uscogdata_ig_categories_unsupported")
-  }
-  invisible(TRUE)
-}
-
 #' @noRd
 .sql_lit_chr <- function(x) {
   safe <- gsub("'", "''", x, fixed = TRUE)
@@ -461,8 +218,7 @@ cog_spending <- function(govid, years, category = NULL,
 }
 
 #' @noRd
-.build_verb_sql <- function(view, subtype_col, govid, years, category,
+.build_verb_sql <- function(view, subtype_col, govid, years, category) {
-                            ig_view = NULL) {
   govid_lit <- .sql_lit_chr(govid)
   years_lit <- paste(as.integer(years), collapse = ",")
   category_pred <- if (is.null(category)) {
@@ -471,26 +227,6 @@ cog_spending <- function(govid, years, category = NULL,
     sprintf("AND category IN (%s)", .sql_lit_chr(category))
   }
 
-  # expenditure_concept = "total" adds the intergovernmental leg. UNION ALL,
-  # never UNION: the two legs are disjoint by item_code prefix (E/F/G vs M/L),
-  # so de-duplication would be pure cost, and a silent row-drop if two
-  # governments ever reported identical values.
-  source_expr <- if (is.null(ig_view)) {
-    view
-  } else {
-    sprintf("(SELECT * FROM %s UNION ALL SELECT * FROM %s)", view, ig_view)
-  }
-
-  # bool_or(), not bool_and(): a no-op for the Direct/revenue legs (those
-  # views filter NOT is_aggregate, so no row in any group is ever aggregate),
-  # but load-bearing for the IG leg, which deliberately keeps aggregate rows
-  # (see inst/sql/24-ig_long.sql). The wide era is dense -- every government
-  # has a row for every code in a family, most of them $0 -- so a $0 leaf
-  # commonly lands in the same (year, gov, subtype, category) group as the
-  # real aggregate row. bool_and() would then read FALSE for that group even
-  # though its dollars came entirely from an aggregate row, silently
-  # suppressing the "Aggregate fallback applied" note on exactly the rows
-  # this feature exists to surface.
   sprintf(
     "SELECT
        year,
@@ -500,14 +236,14 @@ cog_spending <- function(govid, years, category = NULL,
        category,
        SUM(amt) * 1000.0 AS amt_nominal,
        string_agg(DISTINCT item_code, ',' ORDER BY item_code) AS codes_included,
-       bool_or(is_aggregate) AS aggregate_fallback
+       bool_and(is_aggregate) AS aggregate_fallback
      FROM %2$s
      WHERE canonical_govid IN (%3$s)
        AND year IN (%4$s)
        %5$s
      GROUP BY year, canonical_govid, gov_name, xwalk_gov_name, %1$s, category
      ORDER BY year, canonical_govid, %1$s, category",
-    subtype_col, source_expr, govid_lit, years_lit, category_pred
+    subtype_col, view, govid_lit, years_lit, category_pred
   )
 }
 
@@ -560,131 +296,11 @@ cog_spending <- function(govid, years, category = NULL,
   result
 }
 
-#' Detect rows where expenditure_concept = "total" is reporting the
-#' intergovernmental leg with NO Direct counterpart in the same (year,
-#' canonical_govid, category) group AND a harmonization recipe actually
-#' recovers the missing Direct dollars for that exact (year, canonical_govid,
-#' category) triple.
-#'
-#' Bare Direct-row absence is deliberately NOT sufficient on its own: the
-#' dominant real cause of "no Direct sibling row" is a government that simply
-#' has no direct spending in that category (e.g. a state that funds K-12
-#' entirely through school districts), which is correct, ordinary data, not
-#' suppression. Genuine suppression -- a legacy aggregate-only family whose
-#' Direct-leg basis query excludes it by construction (spending_long/
-#' spending_long_harmonized both filter NOT is_aggregate) -- always has a
-#' covering harmonization recipe, because that is exactly what the recipe
-#' catalog exists to recover (see R/suggestions.R and `cog_recipes()`). So
-#' checking "does a recipe actually cover this triple" cleanly separates the
-#' two cases instead of conflating them.
-#'
-#' Returns `list(flag, notes)`, both the same length as `result`: `flag` is
-#' `TRUE` only for the `spend_subtype == "intergovernmental"` row(s) in a
-#' suppressed group, and `notes` names the recovering recipe(s) for those
-#' rows (`NA` everywhere else).
 #' @noRd
-.detect_direct_suppressed <- function(con, result, subtype_col) {
+.notes_column <- function(result) {
-  n <- nrow(result)
-  empty_notes <- rep(NA_character_, n)
-  if (n == 0L) return(list(flag = logical(0), notes = character(0)))
-  is_ig <- result[[subtype_col]] %in% "intergovernmental"
-  if (!any(is_ig)) return(list(flag = rep(FALSE, n), notes = empty_notes))
-
-  key <- paste(result$year, result$canonical_govid, result$category, sep = "\r")
-  has_direct <- key %in% unique(key[!is_ig])
-  candidate <- is_ig & !has_direct
-
-  flag <- rep(FALSE, n)
-  notes <- empty_notes
-  if (!any(candidate)) return(list(flag = flag, notes = notes))
-
-  idx <- which(candidate)
-  rows <- unique(result[idx, c("year", "canonical_govid", "category")])
-  covering <- .covering_recipes(con, rows)
-  cov_key <- paste(covering$year, covering$canonical_govid, covering$category,
-                    sep = "\r")
-
-  for (i in idx) {
-    k <- paste(result$year[i], result$canonical_govid[i], result$category[i],
-               sep = "\r")
-    m <- match(k, cov_key)
-    if (is.na(m)) next
-    ids <- covering$recipe_ids[[m]]
-    if (length(ids) == 0L) next
-    flag[i] <- TRUE
-    notes[i] <- sprintf(
-      "Direct component is unavailable through this basis for this year; recover it via recipe = '%s' (see cog_recipes()).",
-      paste(sort(unique(ids)), collapse = "', '")
-    )
-  }
-  list(flag = flag, notes = notes)
-}
-
-#' For each (year, canonical_govid, category) triple potentially affected by
-#' a suppressed Direct leg, find the harmonization recipe(s) that (a) cover
-#' this `category` (share a component item_code via `summary_categories`,
-#' excluding any recipe that is itself entirely intergovernmental M/L -- the
-#' same exclusion `.build_suggestions()` applies, see I2) and (b) actually
-#' produce a `long` row for this exact (canonical_govid, year) via the same
-#' generic join `.run_recipe()` uses (component year_min/year_max +
-#' gov_type_scope, no is_aggregate filter -- a recipe's whole point is to
-#' recover data that's aggregate-only). Adds a list-column `recipe_ids`
-#' (possibly length-0) to `rows`.
-#' @noRd
-.covering_recipes <- function(con, rows) {
-  rows$recipe_ids <- vector("list", nrow(rows))
-  cats <- unique(rows$category[!is.na(rows$category)])
-  if (length(cats) == 0L) return(rows)
-
-  cand <- DBI::dbGetQuery(con, sprintf(
-    "SELECT DISTINCT sc.category, r.recipe_id
-     FROM harmonization_recipes r
-     JOIN summary_categories sc ON sc.item_code = r.component_code
-     WHERE sc.category IN (%s)
-       AND r.recipe_id NOT IN (
-         SELECT DISTINCT recipe_id FROM harmonization_recipes
-         WHERE LEFT(component_code, 1) IN ('M', 'L')
-       )",
-    .sql_lit_chr(cats)
-  ))
-  if (nrow(cand) == 0L) return(rows)
-
-  recipe_ids_all <- unique(cand$recipe_id)
-  govids <- unique(rows$canonical_govid)
-  years  <- unique(rows$year)
-  covered <- DBI::dbGetQuery(con, sprintf(
-    "SELECT DISTINCT r.recipe_id, l.canonical_govid, l.year
-     FROM long l
-     JOIN harmonization_recipes r
-       ON l.item_code = r.component_code
-      AND l.year BETWEEN r.year_min AND r.year_max
-      AND (r.gov_type_scope = 'all'
-           OR (r.gov_type_scope = 'state' AND l.type = 0)
-           OR (r.gov_type_scope = 'local' AND l.type BETWEEN 1 AND 3))
-     WHERE r.recipe_id IN (%s)
-       AND l.canonical_govid IN (%s)
-       AND l.year IN (%s)",
-    .sql_lit_chr(recipe_ids_all), .sql_lit_chr(govids), paste(years, collapse = ",")
-  ))
-
-  for (i in seq_len(nrow(rows))) {
-    cat_i <- rows$category[i]
-    if (is.na(cat_i)) next
-    cat_recipe_ids <- cand$recipe_id[cand$category == cat_i]
-    if (length(cat_recipe_ids) == 0L) next
-    sub <- covered[covered$canonical_govid == rows$canonical_govid[i] &
-                   covered$year == rows$year[i] &
-                   covered$recipe_id %in% cat_recipe_ids, ]
-    rows$recipe_ids[[i]] <- sort(unique(sub$recipe_id))
-  }
-  rows
-}
-
-#' @noRd
-.notes_column <- function(result, direct_suppressed_notes = NULL) {
   n <- nrow(result)
   if (n == 0L) return(character(0))
-  parts <- vector("list", 3L)
+  parts <- vector("list", 2L)
   agg <- result[["aggregate_fallback"]]
   parts[[1]] <- if (!is.null(agg)) {
     ifelse(agg %in% TRUE,
@@ -701,11 +317,6 @@ cog_spending <- function(govid, years, category = NULL,
   } else {
     rep(NA_character_, n)
   }
-  parts[[3]] <- if (!is.null(direct_suppressed_notes)) {
-    direct_suppressed_notes
-  } else {
-    rep(NA_character_, n)
-  }
   out <- character(n)
   for (i in seq_len(n)) {
     pieces <- vapply(parts, `[[`, character(1), i)

R/suggestions.R

@@ -1,8 +1,9 @@
 # R/suggestions.R
 # Recipe-component-driven signposting: when a basis = "harmonized" query for
-# a category comes back with a coverage gap in some requested years (the
+# a category asks for a code that is itself a harmonization recipe
-# result has no rows at all in that year) that a harmonization recipe would
+# component, and that specific code has no rows in some requested years
-# actually fill for this government, surface that recipe as a suggestion.
+# while the recipe's own generic join would still fill those years for this
+# government, surface that recipe as a suggestion.
 #
 # This is deliberately keyed off the recipe catalog's component codes, not
 # off harmonization_map rows: no live map row carries a non-blank
@@ -12,83 +13,58 @@
 # suggestion off of, just a leaf-code absence a recipe happens to fill).
 # See docs/phase_r_harmonization_review.md § 0.3.
 #
-# Scope is deliberately narrow: signposting only runs when the caller
+# Scope is deliberately narrow in one respect and, as of Phase R3 Task 19c,
+# deliberately WIDE in another: signposting only runs when the caller
 # supplied a `category` (an un-scoped, all-categories query has no single
-# coverage question to answer) and only flags a recipe when the ACTUAL
+# coverage question to answer), but within that category it now checks
-# result has zero rows in a requested year AND the candidate recipe's own
+# EACH recipe component that is itself a category member individually,
-# generic join (same join .run_recipe() uses, including its wide-era
+# rather than asking whether the whole category *result* has zero rows
-# aggregate rows) produces at least one row for this government in that
+# that year. A recipe fires when one of its own components has zero rows
-# year. Checking presence per-government (not corpus-wide) avoids false
+# for this government in a requested year, AND SOME OTHER component of that
-# positives from ordinary reporting variance -- most governments don't use
+# SAME recipe -- excluding the gapped one itself -- has a row (same join
-# every sibling code in a multi-code category every year, and that is not
+# .run_recipe() uses, aggregate rows included) for that year. This is the
-# a format-boundary gap worth signposting.
+# literal review-doc § 0.3 criterion: "...has no rows ... but other
-#
+# components do." A component's OWN aggregate-only row does not satisfy
-# C1(a): for expenditure_concept = "total" callers, `result` here must
+# its own gap (self-coverage is not "other components"); only a genuinely
-# already be the Direct-leg subset (the caller filters out
+# different sibling component can. This fires even if OTHER, unrelated
-# spend_subtype == "intergovernmental" rows before calling in). A gap year
+# codes in the same category have full data that year and the overall
-# is "the requested year has no Direct rows", never "no rows at all" --
+# result looks complete. That is a deliberate narrowing of the R2-era
-# an IG row surviving on a legacy aggregate that Direct excludes must not
+# false-positive guard: most governments don't use every sibling code in a
-# read as coverage and cancel the very suggestion that would recover it.
+# multi-code category every year, and per-code detection WILL flag some of
+# that as a "gap" even though it's really just a government not having
+# that particular sub-type of spending, not a format-boundary artifact.
+# The remaining guard against ordinary reporting variance is the
+# per-government, per-OTHER-component `covered` check below (a component
+# is only flagged when a DIFFERENT component of the SAME recipe -- not
+# some unrelated code, and not the gapped component's own aggregate row --
+# actually has something to offer in that year); it no longer tries to
+# avoid noise from sibling *codes*, only from a recipe with genuinely
+# nothing else to contribute. The acceptable noise level this trade
+# produces is a product decision, measured (not tuned here) by
+# data-raw/measure_signposting_rate.R and ruled on at Checkpoint R3.
 
-#' Build the `prov$suggestions` list for a (non-recipe) basis = "harmonized"
+#' Recipe components that are classified under the requested category --
-#' verb call: recipes whose generic join would fill a real gap in `result`.
+#' the codes a category-scoped query actually "requests". A recipe can
-#'
+#' have components outside the category (e.g. general_gov_e89_wide's E85
-#' @param con Active DuckDB connection.
+#' leg has no category assignment); those never trigger on their own, they
-#' @param govid Character vector of canonical_govid values (the verb's raw
+#' just were never part of what this query asked for.
-#'   `govid`).
-#' @param years Integer vector of requested years.
-#' @param category `category` argument as passed to the verb (character
-#'   vector or `NULL`; suggestions are only computed when non-NULL).
-#' @param result The verb's already-computed result tibble (post basis
-#'   query, pre per_capita/adjust_to_year), pre-filtered to the Direct leg
-#'   only when the caller's `expenditure_concept = "total"` (see C1(a)).
-#' @param basis The *resolved* basis (`"harmonized"` or `"raw"`).
-#' @param flow_prefixes The calling verb's own flow-type prefixes (e.g.
-#'   `c("E", "F", "G")` for `cog_spending()`, `c("T", "A", "U", "B", "C",
-#'   "D")` for `cog_revenue()` -- see `.verb_spendrev()`). Passed through to
-#'   `.attach_ig_counterparts()` to keep the intergovernmental-counterpart
-#'   lookup scoped to the calling verb's own flow family.
-#' @return List of `list(recipe_id, label, available_years, hint,
-#'   ig_recipe_id)`, possibly empty.
 #' @noRd
-.build_suggestions <- function(con, govid, years, category, result, basis,
+.category_recipe_components <- function(con, category) {
-                                flow_prefixes) {
+  DBI::dbGetQuery(con, sprintf(
-  if (!identical(basis, "harmonized") || is.null(category)) return(list())
+    "SELECT DISTINCT r.recipe_id, r.component_code, r.year_min, r.year_max,
-
+            r.gov_type_scope
-  # Exclude any recipe that is ITSELF an intergovernmental (M/L) recipe --
+     FROM harmonization_recipes r
-  # i.e. every one of its own component codes is M/L-prefixed. Without this,
+     JOIN summary_categories sc
-  # a category whose summary_categories rows span both a Direct family
+       ON sc.item_code = r.component_code AND sc.category IN (%s)",
-  # (e.g. E04/E05, "Corrections") and its M/L counterpart (M04/M05, same
-  # category since Task 1) makes the M/L recipe itself (e.g.
-  # `corrections_ig_local_combined`) a raw top-level candidate for a plain
-  # (Direct) cog_spending() call -- following that hint would silently
-  # return intergovernmental dollars under `expenditure_concept = "direct"`
-  # provenance. This is a stronger, unconditional exclusion than the
-  # flow-prefix gate below/in `.attach_ig_counterparts()`: an M/L recipe
-  # should never be suggested as a coverage-gap filler for EITHER verb, not
-  # just kept from being named as the *counterpart* of another suggestion.
-  candidates <- DBI::dbGetQuery(con, sprintf(
-    "SELECT DISTINCT recipe_id FROM harmonization_recipes
-     WHERE component_code IN (
-       SELECT DISTINCT item_code FROM summary_categories WHERE category IN (%s)
-     )
-     AND recipe_id NOT IN (
-       SELECT DISTINCT recipe_id FROM harmonization_recipes
-       WHERE LEFT(component_code, 1) IN ('M', 'L')
-     )",
     .sql_lit_chr(category)
-  ))$recipe_id
+  ))
-  if (length(candidates) == 0L) return(list())
+}
 
-  result_years <- if (is.null(result) || nrow(result) == 0L) {
+#' Label + overall year coverage for a set of recipe ids (the suggestion's
-    integer(0)
+#' `label`/`available_years`).
-  } else {
+#' @noRd
-    unique(as.integer(result$year))
+.recipe_meta <- function(con, candidates) {
-  }
+  tibble::as_tibble(DBI::dbGetQuery(con, sprintf(
-  gap_years <- setdiff(as.integer(years), result_years)
-  if (length(gap_years) == 0L) return(list())
-
-  meta <- tibble::as_tibble(DBI::dbGetQuery(con, sprintf(
     "SELECT recipe_id, any_value(label) AS label,
             MIN(year_min) AS year_min, MAX(year_max) AS year_max
      FROM harmonization_recipes
@@ -96,13 +72,48 @@
      GROUP BY recipe_id",
     .sql_lit_chr(candidates)
   )))
+}
 
-  # Which (recipe_id, year) pairs the recipe's own generic join actually
+#' Which (recipe_id, component_code, year) triples have at least one
-  # covers for this government, restricted to the gap years -- the same
+#' NOT-aggregate row for these governments -- i.e. that specific requested
-  # join .run_recipe() uses (component year_min/year_max + gov_type_scope,
+#' code itself has data, scoped exactly like .run_recipe()'s join
-  # no is_aggregate filter), just checking existence instead of summing.
+#' (component year_min/year_max + gov_type_scope). NOT-aggregate mirrors
-  covered <- DBI::dbGetQuery(con, sprintf(
+#' what basis = "harmonized" itself excludes: an aggregate-only year is a
-    "SELECT DISTINCT r.recipe_id, l.year
+#' gap for that code exactly as it would be in a plain category query.
+#' @noRd
+.component_presence <- function(con, candidates, govid, years_lit) {
+  DBI::dbGetQuery(con, sprintf(
+    "SELECT DISTINCT r.recipe_id, r.component_code, l.year
+     FROM long l
+     JOIN harmonization_recipes r
+       ON l.item_code = r.component_code
+      AND l.year BETWEEN r.year_min AND r.year_max
+      AND (r.gov_type_scope = 'all'
+           OR (r.gov_type_scope = 'state' AND l.type = 0)
+           OR (r.gov_type_scope = 'local' AND l.type BETWEEN 1 AND 3))
+     WHERE NOT l.is_aggregate
+       AND r.recipe_id IN (%s)
+       AND l.canonical_govid IN (%s)
+       AND l.year IN (%s)",
+    .sql_lit_chr(candidates), .sql_lit_chr(govid), years_lit
+  ))
+}
+
+#' Which (recipe_id, component_code, year) triples have at least one row
+#' (aggregate rows included) for these governments -- the same scoping
+#' .run_recipe()'s join uses (component year_min/year_max + gov_type_scope),
+#' just checking existence instead of summing. Kept at per-component grain
+#' (not unioned across the whole recipe, unlike the R2/R3-pre-fix version of
+#' this function) so a gap check can require the covering evidence to come
+#' from a DIFFERENT component -- review-doc § 0.3's "other components", not
+#' the gapped component's own aggregate row. This is the per-government
+#' guard against ordinary reporting variance: a recipe with genuinely
+#' nothing to offer from any OTHER component (aggregate or leaf) never
+#' fires.
+#' @noRd
+.recipe_coverage <- function(con, candidates, govid, years_lit) {
+  DBI::dbGetQuery(con, sprintf(
+    "SELECT DISTINCT r.recipe_id, r.component_code, l.year
      FROM long l
      JOIN harmonization_recipes r
        ON l.item_code = r.component_code
@@ -113,13 +124,68 @@
      WHERE r.recipe_id IN (%s)
        AND l.canonical_govid IN (%s)
        AND l.year IN (%s)",
-    .sql_lit_chr(candidates), .sql_lit_chr(govid),
+    .sql_lit_chr(candidates), .sql_lit_chr(govid), years_lit
-    paste(gap_years, collapse = ",")
   ))
+}
+
+#' TRUE if recipe `rid` has at least one requested component with an
+#' in-scope requested year that has no data (`present`), in a year some
+#' OTHER component of the same recipe is otherwise fillable (`covered`,
+#' excluding the component under test) -- the per-code gap the R2
+#' whole-result check couldn't see, covered by another component the way
+#' review-doc § 0.3 specifies (not by the gapped component's own aggregate
+#' row -- that is self-coverage, not "other components", and must not
+#' count).
+#' @noRd
+.recipe_component_gapped <- function(rid, requested, present, covered, years) {
+  comps <- requested[requested$recipe_id == rid, , drop = FALSE]
+  for (i in seq_len(nrow(comps))) {
+    this_code <- comps$component_code[i]
+    in_scope <- years[years >= comps$year_min[i] & years <= comps$year_max[i]]
+    if (length(in_scope) == 0L) next
+    has_data <- present$year[
+      present$recipe_id == rid & present$component_code == this_code
+    ]
+    gap_years <- setdiff(in_scope, has_data)
+    if (length(gap_years) == 0L) next
+    other_covered_years <- covered$year[
+      covered$recipe_id == rid & covered$component_code != this_code
+    ]
+    if (any(gap_years %in% other_covered_years)) return(TRUE)
+  }
+  FALSE
+}
+
+#' Build the `prov$suggestions` list for a (non-recipe) basis = "harmonized"
+#' verb call: recipes whose generic join would fill a real per-code gap for
+#' the requested category.
+#'
+#' @param con Active DuckDB connection.
+#' @param govid Character vector of canonical_govid values (the verb's raw
+#'   `govid`).
+#' @param years Integer vector of requested years.
+#' @param category `category` argument as passed to the verb (character
+#'   vector or `NULL`; suggestions are only computed when non-NULL).
+#' @param basis The *resolved* basis (`"harmonized"` or `"raw"`).
+#' @return List of `list(recipe_id, label, available_years, hint)`, possibly
+#'   empty.
+#' @noRd
+.build_suggestions <- function(con, govid, years, category, basis) {
+  if (!identical(basis, "harmonized") || is.null(category)) return(list())
+
+  requested <- .category_recipe_components(con, category)
+  if (nrow(requested) == 0L) return(list())
+  candidates <- unique(requested$recipe_id)
+  years_int  <- as.integer(years)
+  years_lit  <- paste(years_int, collapse = ",")
+
+  meta    <- .recipe_meta(con, candidates)
+  present <- .component_presence(con, candidates, govid, years_lit)
+  covered <- .recipe_coverage(con, candidates, govid, years_lit)
 
   suggestions <- list()
   for (rid in candidates) {
-    if (!rid %in% covered$recipe_id) next
+    if (!.recipe_component_gapped(rid, requested, present, covered, years_int)) next
     m <- meta[meta$recipe_id == rid, ]
     suggestions[[length(suggestions) + 1L]] <- list(
       recipe_id = rid,
@@ -128,121 +194,19 @@
       hint = sprintf("re-run with recipe = '%s'", rid)
     )
   }
-  .attach_ig_counterparts(con, suggestions, flow_prefixes)
+  suggestions
-}
-
-#' Attach `ig_recipe_id` to each suggestion: the intergovernmental-expenditure
-#' recipe (an M-to-local or L-to-state recipe) whose component codes cover
-#' exactly the same set of function suffixes as the firing recipe's own
-#' components, e.g. `corrections_combined`'s {E04, E05} -> suffixes {"04",
-#' "05"} matches `corrections_ig_local_combined`'s {M04, M05} -> the same
-#' {"04", "05"}. `NULL` when no such recipe exists, which also covers the
-#' case where the firing recipe already IS the IG recipe (self-matches are
-#' excluded, so an IG recipe never names itself as its own counterpart).
-#'
-#' Matching is deliberately an exact set match, not "any suffix in common":
-#' the two-digit suffix only means the same "function" across recipes that
-#' share the underlying Census functional-classification scheme (E/F/G/L/M
-#' all use "04"/"05" for corrections). M/L "combined other" codes (47/89/
-#' 91-94) reuse digits for an unrelated catch-all construct, so e.g.
-#' `general_gov_e89_wide`'s {E85, E89} -> {"85", "89"} must NOT match
-#' `ige_local_m89_wide`'s {"89", "91", "92", "93"} on the shared "89" alone.
-#' Checked by hand against the full harmonization_recipes catalog: only the
-#' corrections family (E/F/G/M, suffixes 04/05) has an exact-set match in
-#' this corpus.
-#'
-#' Exact-set suffix matching is NOT enough on its own, though: the same
-#' reused-digit problem exists ACROSS the revenue-side IG families too.
-#' `ig_local_d47_wide` (D47/D94, suffixes {"47","94"}) is an exact-set match
-#' for `ige_local_m47_wide` (M47/M94, same suffixes) even though one is
-#' intergovernmental REVENUE received from local governments and the other is
-#' intergovernmental EXPENDITURE paid to local governments -- unrelated flows
-#' that happen to reuse "47"/"94" for their own "transit/utilities" and
-#' "other/combined" catch-alls. `ig_federal_b47_wide`, `ig_state_c47_wide`,
-#' and their `*_89` siblings all collide the same way. None of this is
-#' reachable via `cog_revenue()` in the bundled fixture today (its B/C/D
-#' recipes never happen to have a covered gap year for any fixture govid),
-#' but it IS reachable via a mis-scoped `cog_spending()` call on a
-#' revenue-only category, e.g. `cog_spending(gov, category = "IG Federal")`
-#' fires `ig_federal_b47_wide`/`ig_federal_b89_wide` for real in the fixture
-#' -- so this is a live, not merely theoretical, gap.
-#'
-#' Two flow-family checks close this, both required (see
-#' `tests/testthat/test-expenditure-concept.R`, "revenue-flavored ... never
-#' receives an M/L counterpart" tests, for the pairwise verification):
-#'   1. `own_prefix %in% flow_prefixes`: the firing recipe's own component
-#'      codes must belong to the calling verb's own flow family (the same
-#'      `flow_prefixes` `.build_harmonization_block()` uses, see
-#'      `R/basis.R`). This blocks a recipe surfaced through a mis-scoped
-#'      category from ever reaching the M/L search, e.g. `cog_spending()`'s
-#'      flow_prefixes are `c("E","F","G")`, which `ig_federal_b47_wide`'s own
-#'      `"B"` is not part of.
-#'   2. `own_prefix %in% c("E","F","G")`: M/L only ever pairs with the
-#'      DIRECT-expenditure family, never with revenue (`cog_revenue()`'s
-#'      flow_prefixes already fold B/C/D in as ordinary revenue -- there is
-#'      no separate "Total" bolt-on for revenue the way `expenditure_concept`
-#'      adds one for spending) and never with ANOTHER M/L recipe (without
-#'      this check, `ige_local_m47_wide` would wrongly match sibling
-#'      `ige_state_l47_wide` on their shared {"47","94"} suffix set).
-#'      Condition 1 alone does not catch this: under `cog_revenue()`,
-#'      `ig_federal_b47_wide`'s own `"B"` IS inside revenue's own
-#'      `flow_prefixes`, so only this second, family-specific check blocks
-#'      the search.
-#' @noRd
-.attach_ig_counterparts <- function(con, suggestions, flow_prefixes) {
-  if (length(suggestions) == 0L) return(suggestions)
-
-  comp <- DBI::dbGetQuery(con,
-    "SELECT recipe_id, component_code FROM harmonization_recipes")
-  comp$prefix <- substr(comp$component_code, 1L, 1L)
-  comp$suffix <- substr(comp$component_code, 2L, nchar(comp$component_code))
-  suffix_sets <- lapply(split(comp$suffix, comp$recipe_id), function(x) sort(unique(x)))
-  prefix_sets <- lapply(split(comp$prefix, comp$recipe_id), function(x) sort(unique(x)))
-
-  ig_recipe_ids <- unique(comp$recipe_id[comp$prefix %in% c("M", "L")])
-
-  find_counterpart <- function(rid) {
-    own_prefix <- prefix_sets[[rid]]
-    own_suffix <- suffix_sets[[rid]]
-    if (is.null(own_prefix) || is.null(own_suffix)) return(NULL)
-    if (!all(own_prefix %in% flow_prefixes)) return(NULL)
-    if (!all(own_prefix %in% c("E", "F", "G"))) return(NULL)
-    for (cand in ig_recipe_ids) {
-      if (identical(cand, rid)) next
-      if (setequal(suffix_sets[[cand]], own_suffix)) return(cand)
-    }
-    NULL
-  }
-
-  lapply(suggestions, function(s) {
-    # `s$ig_recipe_id <- NULL` would DELETE the element rather than set it
-    # (standard R list-assignment gotcha), leaving no-match entries missing
-    # the key entirely instead of carrying it as NULL. Single-bracket
-    # assignment with a wrapped list preserves a NULL-valued element so the
-    # field is always present, per the brief's "NULL when there is none".
-    s["ig_recipe_id"] <- list(find_counterpart(s$recipe_id))
-    s
-  })
 }
 
 #' Emit the single cli::cli_inform() message summarizing all suggestions
 #' for a verb call (the brief's "one message", not one per suggestion).
 #' Bullet text is pre-formatted plain text (no cli/glue `{}` markup) since
 #' recipe ids/labels are untrusted-ish data values, not literal call-site
-#' expressions. When a suggestion has an `ig_recipe_id`, one indented
+#' expressions.
-#' continuation line is appended naming the intergovernmental counterpart
-#' recipe (embedded `\n` renders as a hanging-indent continuation of the
-#' same bullet under cli, not a new bullet).
 #' @noRd
 .inform_suggestions <- function(suggestions) {
   bullets <- vapply(suggestions, function(s) {
-    bullet <- sprintf("%s (%d-%d): %s", s$recipe_id,
+    sprintf("%s (%d-%d): %s", s$recipe_id,
             s$available_years[1], s$available_years[2], s$hint)
-    if (!is.null(s$ig_recipe_id)) {
-      bullet <- paste0(bullet, sprintf(
-        "\n  intergovernmental counterpart: recipe = '%s'", s$ig_recipe_id))
-    }
-    bullet
   }, character(1))
   cli::cli_inform(c(
     i = "Coverage gap detected for the requested years; a harmonization recipe may fill it:",

R/views.R

@@ -1,60 +1,25 @@
 # R/views.R
 
-# SQL files that cannot be registered unconditionally against a v4 corpus,
+# SQL files whose view definitions read schema-v5-only parquet tables
-# for one of two distinct reasons -- both fail at CREATE VIEW time (DuckDB
+# (harmonization_map.parquet, harmonization_recipes.parquet,
-# resolves a view's source schema eagerly, even though it defers execution),
+# series_breaks.parquet) or select from views built on top of them. DuckDB's
-# so a v4 corpus can't tolerate either unconditionally:
+# read_parquet() resolves the file at CREATE VIEW time (even for a view, it
-#
+# still needs the source schema) and errors immediately -- "IO Error: No
-#  (a) Missing FILE. 33-/34-/35- read_parquet() a v5-only parquet table
+# files found" -- if the path doesn't exist, so these cannot be registered
-#      (harmonization_map.parquet, harmonization_recipes.parquet,
+# unconditionally against a v4 corpus the way the rest of inst/sql/ is.
-#      series_breaks.parquet) that doesn't exist at all on a v4 corpus --
+# Registration is therefore gated on manifest$schema_version >= 5; verb-level
-#      "IO Error: No files found".
+# *usage* of the resulting views is separately gated by .resolve_basis() /
-#
+# .require_schema_v5().
-#  (b) Missing COLUMN. 22-/23-/25- reference `long.harmonized_code`, a
-#      column that does not exist on a v4 corpus's `long` table (harmonized
-#      space was introduced in schema v5) -- "Binder Error: Referenced
-#      column harmonized_code not found". 42-/43-/45- are on this list only
-#      because they SELECT s.* FROM the (a)/(b) views above, so they'd fail
-#      to resolve their own source view if it weren't already skipped.
-#
-# Registration is therefore gated on manifest$schema_version >= 5 for all of
-# them; verb-level *usage* of the resulting views is separately gated by
-# .resolve_basis() / .require_schema_v5().
 .harmonization_view_files <- c(
   "22-spending_long_harmonized.sql",
   "23-revenue_long_harmonized.sql",
-  "25-ig_long_harmonized.sql",
   "33-harmonization_map.sql",
   "34-harmonization_recipes.sql",
   "35-series_breaks_pq.sql",
   "42-spending_annotated_harmonized.sql",
-  "43-revenue_annotated_harmonized.sql",
+  "43-revenue_annotated_harmonized.sql"
-  "45-ig_annotated_harmonized.sql"
 )
 
-# The representation contract (cog_pipeline#64): two parquet tables that say
-# what an ABSENT cell means in a given year. Gated on manifest PRESENCE, not
-# on schema_version, because the sparsification that introduced them did not
-# bump the version -- the pre-sparsification corpus this package shipped
-# against until 2026-07-30 was already schema v6 and carried neither table.
-# Keying off the version number would therefore register a view over a file
-# that does not exist and fail at CREATE VIEW time on exactly the corpora this
-# check exists to tolerate.
-.representation_view_files <- c(
-  "36-representation.sql" = "representation.parquet",
-  "37-code_set.sql"       = "code_set.parquet"
-)
-
-#' Does the mounted corpus publish `file` (e.g. "code_set.parquet")?
-#' Reads the manifest's metadata list rather than stat-ing the URL, so it
-#' works identically for a local fixture and a remote share.
-#' @noRd
-.corpus_has_table <- function(manifest, file) {
-  paths <- vapply(manifest$files$metadata %||% list(),
-                  function(f) as.character(f$path %||% ""), character(1))
-  file %in% basename(paths)
-}
-
 #' Register DuckDB views from inst/sql/ SQL files
 #' @noRd
 .register_views <- function(con, url, manifest) {
@@ -62,10 +27,7 @@
   files   <- sort(list.files(sql_dir, pattern = "\\.sql$", full.names = TRUE))
   schema_version <- suppressWarnings(as.integer(manifest$schema_version %||% 0L))
   for (f in files) {
-    base <- basename(f)
+    if (basename(f) %in% .harmonization_view_files && schema_version < 5L) next
-    if (base %in% .harmonization_view_files && schema_version < 5L) next
-    if (base %in% names(.representation_view_files) &&
-        !.corpus_has_table(manifest, .representation_view_files[[base]])) next
     sql <- paste(readLines(f, warn = FALSE), collapse = "\n")
     sql <- gsub("\\{url\\}", url, sql, fixed = FALSE)
     DBI::dbExecute(con, sql)

README.md

@@ -19,59 +19,36 @@ package implements.
 # pak::pkg_install("gitea.civilytics.org/Civilytics/uscogdata")
 ```
 
-## Amounts are in full US dollars
-
-Every amount column this package returns — `amt_nominal`, `amt_real`,
-`amt_per_capita_nominal`, `amt_per_capita_real` — is in **full US dollars**.
-
-The raw Census source files report **thousands of dollars**, and the corpus's
-own `amt` column preserves that. The verbs multiply by 1000 on the way out, so
-you never have to. The conversion is recorded in every result:
-
-```r
-r <- cog_spending("552025209777", 2020L)
-attr(r, "provenance")$transformations$units_conversion
-#> $applied TRUE  $source_unit "$1,000s (raw Census)"  $target_unit "$USD"  $multiplier 1000
-```
-
-**Do not multiply again.** If you have read elsewhere that COG amounts are in
-`$1,000s` — true of the raw corpus, and of `cog_explorer`'s conventions doc —
-that rule does not apply to anything a `cog_*()` verb hands you. Applying it
-twice overstates every figure by 1000x, and the result looks plausible rather
-than obviously wrong.
-
 ## Configuration
 
 - `USCOGDATA_URL` — corpus root URL (public Nextcloud share, trailing slash)
 - `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
 - `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
 
-## Direct vs Total spending
+## Raw-parquet caveat: `survey_weight` is not an aggregation weight
 
-`cog_spending(..., expenditure_concept = c("direct", "total"))` controls
+Users reading the corpus parquet directly (DuckDB, arrow) will see a
-whose spending a result counts. `"direct"` (the default) is a government's
+`survey_weight` column (schema v6, col 28). It is legacy Census IndFin
-own current operations, capital outlay, and other direct spending. `"total"`
+sample-design **metadata passed through verbatim** — the Census Bureau's own
-additionally adds in the intergovernmental legs — money it hands to other
+source documentation says it "is for informational purposes only and should
-governments to spend on its behalf — which is meaningful for describing one
+not be used to derive any other statistics" (`_ReadMe_First_IndFin.txt`;
-government's own budget over time, but double-counts when summed across
+likewise `UserGuide.xls` Data User Note 8: "Do not use the weight field to
-governments (a state's payment to a county is the same dollar the county
+derive state or national totals"). The raw encoding is also inconsistent
-reports as its own direct spending).
+across vintages (reciprocal scale most years, direct scale in 2003, a `1`
-
+placeholder in 1967/70/71/73/2001, all-`0` in 2007–2012, `NA` for all
-**Rule of thumb: any figure that spans more than one government uses
+modern-source rows), so `sum(amt * survey_weight/10000)`-style expressions
-`direct`.** `cog_geographic_rollup()` and `cog_peer_compare()` enforce this
+produce silently wrong totals — including exact zeros for 2007–2012. Sum
-by refusing `expenditure_concept = "total"`. See
+`amt` unweighted; no uscogdata function reads this column. Full evidence:
-`vignette("total-spending", package = "uscogdata")` for the full
+`cog_pipeline/.superpowers/sdd/weight-semantics-findings.md`.
-explanation with worked examples.
 
 ## Developer notes
 
 ### Testing
 
 The package ships a bundled fixture corpus at `inst/extdata/fixture_corpus/` —
-a 15 MB four-year slice (2011, 2012, 2019, 2020) of the full corpus covering
+a 3.6 MB two-year slice (2019 + 2020) of the full corpus covering all 50
-all 50 states. `tests/testthat/setup.R` automatically points `USCOGDATA_URL`
+states. `tests/testthat/setup.R` automatically points `USCOGDATA_URL` at this
-at this fixture, so the full test suite runs offline with no network
+fixture, so the full test suite runs offline with no network dependency:
-dependency:
 
 ```r
 devtools::test()   # uses bundled fixture, no credentials required

data-raw/measure_signposting_rate.R

@@ -0,0 +1,563 @@
+# data-raw/measure_signposting_rate.R
+#
+# Phase R3 Task 19c: measures the harmonization-signposting suggestion rate
+# under THREE `.build_suggestions()` implementations, over a realistic query
+# battery:
+#   every summary_categories category
+#     x a 3-year pre/post-2012 span (the wide-aggregate -> modern-leaf
+#       format-boundary window; falls back to the widest span the corpus
+#       actually supports if it can't fill a full 3+3 design -- see
+#       .measure_year_span())
+#     x up to N_GOV sampled governments (seeded, deterministic)
+#
+# The three arms, oldest to newest:
+#   - "coarse"  (git ref b0df1ec, the merged R2 tip): a year counts as
+#     gapped only when the WHOLE category result has zero rows that year.
+#   - "selfcov" (git ref da72bf3, Task 19c's first per-code pass, since
+#     amended after review): per-code, but a component's gap could be
+#     satisfied by ANY component of the recipe INCLUDING ITSELF -- so a
+#     code whose only representation in a year was its own wide-era
+#     aggregate row satisfied its own coverage check. Flagged in review as
+#     not matching review-doc S: 0.3's literal criterion ("... has no rows
+#     ... but OTHER components do") and fixed in the next commit.
+#   - "percode" (live code): per-code, requiring a genuinely DIFFERENT
+#     sibling component to supply the covering evidence -- the shipped,
+#     corrected implementation.
+#
+# READ THIS BEFORE QUOTING ANY DELTA FROM THIS SCRIPT
+# -----------------------------------------------------
+# The coarse and per-code checks are PARTLY DISJOINT, not nested. Per-code
+# is NOT a strict widening of coarse: there are queries coarse fires on that
+# per-code does not, so moving coarse -> percode both ADDS and REMOVES
+# signposting. Every `*_delta_pp` figure this script reports -- overall and
+# per category -- is therefore a NET of those two flows and can mask a
+# coverage loss in either direction. A headline "+X pp" can sit on top of
+# categories that lost coverage outright (a NEGATIVE corrected_delta_pp),
+# and a category-level zero can be an add and a loss cancelling. Read
+# `$subset_relation` (printed under "Subset relation" below) alongside any
+# delta; that section is where the two flows are separated.
+#
+# The disjointness is structural, not a sampling artifact. Both arms pair a
+# gap test with a coverage test, and it is the COVERAGE test that differs:
+#   - coarse:  gap = the WHOLE category result has zero rows that year;
+#              covered = the recipe's generic join has ANY row that year
+#              (unioned across all components -- a component's own
+#              aggregate row counts).
+#   - percode: gap = one specific component has no non-aggregate row that
+#              year; covered = a DIFFERENT component of the SAME recipe has
+#              a row that year (self-coverage explicitly excluded, per
+#              review-doc S: 0.3's "...but OTHER components do").
+# So when a whole category is empty in a year -- exactly coarse's trigger --
+# and the only covering evidence is the gapped component's own wide-era
+# aggregate row, coarse fires and per-code CANNOT: there is by construction
+# no other component to supply the evidence. That case is already pinned as
+# intended behaviour in tests/testthat/test-recipes.R ("per-code gap does
+# NOT fire when a code's only coverage is its own aggregate row"). This
+# script's job is to say how often it costs coverage, not to relitigate it.
+#
+# This script MEASURES the deltas; it does not decide whether the resulting
+# signposting trade -- added "noise" in one direction, lost whole-category
+# gap coverage in the other -- is acceptable. That is Jared's ruling at
+# Checkpoint R3 (see
+# cog_pipeline/.superpowers/sdd/phase-r-task-19c-brief.md). The selfcov arm
+# exists purely to answer a narrower, mechanical question for that ruling:
+# how much of the coarse -> percode delta was ever attributable to the
+# self-coverage bug (selfcov -> percode), as opposed to genuine
+# other-component coverage (coarse -> percode directly)?
+#
+# All three arms no longer coexist in R/suggestions.R (each superseded the
+# last in place), so this script pulls each VERBATIM from git history and
+# evaluates it in an isolated environment parented on the uscogdata
+# namespace, so each still resolves the unchanged sibling helpers it
+# depends on (.sql_lit_chr()) exactly as the live package did at that
+# commit. This guarantees every non-live arm is the actual shipped code at
+# that point, not a hand-reconstruction that could silently drift from what
+# really shipped.
+#
+# Usage (from the uscogdata package root; a git checkout, not a tarball):
+#   Rscript data-raw/measure_signposting_rate.R
+#   USCOGDATA_URL=<staged-corpus-url> Rscript data-raw/measure_signposting_rate.R
+#
+# Or from R:
+#   source("data-raw/measure_signposting_rate.R")
+#   res <- measure_signposting_rate(corpus_url = "<url>")
+#   res$summary; res$by_category
+
+#' Pull a historical `.build_suggestions()` (and whatever helpers it uses)
+#' verbatim from git history and evaluate it in an isolated environment
+#' parented on the uscogdata namespace, so it resolves unchanged sibling
+#' helpers (`.sql_lit_chr()`) the same way the live package does.
+#' @noRd
+.measure_load_git_impl <- function(git_ref, git_path = "R/suggestions.R") {
+  old_src <- tryCatch(
+    system2("git", c("show", sprintf("%s:%s", git_ref, git_path)),
+            stdout = TRUE, stderr = TRUE),
+    error = function(e) NULL
+  )
+  status <- attr(old_src, "status")
+  if (is.null(old_src) || (!is.null(status) && status != 0L) ||
+      !any(grepl("^\\.build_suggestions", old_src))) {
+    stop(
+      "Could not retrieve the .build_suggestions() implementation from ",
+      "git ref '", git_ref, "' at '", git_path, "'. Run this script from ",
+      "inside the uscogdata git checkout (not a tarball/installed copy).",
+      call. = FALSE
+    )
+  }
+  env <- new.env(parent = asNamespace("uscogdata"))
+  # eval(parse()) here is safe: `old_src` is not external/untrusted input --
+  # it is this repo's OWN historical R/suggestions.R, fetched via `git show`
+  # from a fixed, hardcoded internal commit ref (overridable only by a
+  # caller who already has R-level code execution in this dev-only
+  # measurement script). No network or user-supplied data reaches this call.
+  eval(parse(text = old_src), envir = env)
+  stopifnot(is.function(env$.build_suggestions))
+  env
+}
+
+#' Resolve the query battery's year span: a `pre_n`-year window immediately
+#' before `boundary_year` unioned with a `post_n`-year window starting at
+#' `boundary_year` (default 3+3 around 2012, the wide-aggregate ->
+#' modern-leaf format boundary). Falls back to every distinct year the
+#' corpus actually has in `long` when it can't fill that full design, and
+#' says so explicitly in `$note` rather than silently padding or
+#' fabricating years.
+#' @noRd
+.measure_year_span <- function(con, boundary_year = 2012L,
+                                pre_n = 3L, post_n = 3L) {
+  available <- sort(as.integer(
+    DBI::dbGetQuery(con, "SELECT DISTINCT year FROM long")$year
+  ))
+  desired_pre  <- (boundary_year - pre_n):(boundary_year - 1L)
+  desired_post <- boundary_year:(boundary_year + post_n - 1L)
+  actual_pre   <- intersect(desired_pre, available)
+  actual_post  <- intersect(desired_post, available)
+  full_design  <- length(actual_pre) == pre_n && length(actual_post) == post_n
+
+  if (full_design) {
+    years <- sort(c(actual_pre, actual_post))
+    note <- sprintf(
+      "Full %d-year pre/%d-year post-%d design available -- using years: %s.",
+      pre_n, post_n, boundary_year, paste(years, collapse = ", ")
+    )
+  } else {
+    years <- available
+    note <- sprintf(paste(
+      "Corpus does NOT support a full %d-year pre/%d-year post-%d span",
+      "(desired pre-window %s -> only %s present; desired post-window %s",
+      "-> only %s present). Falling back to the WIDEST span this corpus",
+      "supports: all %d distinct year(s) actually in `long`: %s.",
+      "This is NOT a 3-year pre/post-%d design -- reported as measured,",
+      "not padded or fabricated."
+    ),
+    pre_n, post_n, boundary_year,
+    paste(desired_pre, collapse = ","),
+    if (length(actual_pre)) paste(actual_pre, collapse = ",") else "none",
+    paste(desired_post, collapse = ","),
+    if (length(actual_post)) paste(actual_post, collapse = ",") else "none",
+    length(available), paste(available, collapse = ", "),
+    boundary_year)
+  }
+  list(years = years, full_design = full_design, note = note,
+       available = available)
+}
+
+#' Deterministically sample up to `n` distinct governments that actually
+#' report *something* in the battery's year span (querying a government
+#' with zero presence in every measured year isn't a realistic query).
+#' @noRd
+.measure_sample_govids <- function(con, years, n = 20L, seed = 19L) {
+  pool <- DBI::dbGetQuery(con, sprintf(
+    "SELECT DISTINCT canonical_govid FROM long WHERE year IN (%s)
+     ORDER BY canonical_govid",
+    paste(years, collapse = ",")
+  ))$canonical_govid
+  if (length(pool) <= n) return(sort(pool))
+  set.seed(seed)
+  sort(sample(pool, n))
+}
+
+#' Comma-join a suggestion list's recipe ids (stable order) for the detail
+#' frame's audit columns; `""` when nothing fired.
+#' @noRd
+.measure_recipe_ids <- function(suggestions) {
+  if (length(suggestions) == 0L) return("")
+  paste(sort(vapply(suggestions, function(s) s$recipe_id, character(1))),
+        collapse = ",")
+}
+
+#' Run one (category, government) query through the coarse (`coarse_env`),
+#' self-coverage-allowed (`selfcov_env`), and live per-code
+#' `.build_suggestions()` and return a one-row summary of what each fired.
+#'
+#' Also records the coarse arm's OWN trigger evidence -- `coarse_gap_years`,
+#' the requested years in which the whole category result has zero rows --
+#' so a coarse-fired/per-code-silent disagreement can be named down to
+#' (category, government, year) instead of just counted. Per-code's gap
+#' years are deliberately NOT re-derived here: that would mean
+#' reimplementing `.recipe_component_gapped()`'s set arithmetic in the
+#' measurement harness, where it could silently drift from the code under
+#' measurement. Per-code rows are identified by the recipe ids they fired.
+#' @noRd
+.measure_one_query <- function(con, coarse_env, selfcov_env, category,
+                                category_type, govid, years) {
+  view        <- if (identical(category_type, "revenue")) {
+    "revenue_annotated_harmonized"
+  } else {
+    "spending_annotated_harmonized"
+  }
+  subtype_col <- if (identical(category_type, "revenue")) {
+    "revenue_subtype"
+  } else {
+    "spend_subtype"
+  }
+  sql    <- .build_verb_sql(view, subtype_col, govid, years, category)
+  result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
+
+  coarse_sugg  <- coarse_env$.build_suggestions(
+    con, govid, years, category, result, "harmonized"
+  )
+  selfcov_sugg <- selfcov_env$.build_suggestions(
+    con, govid, years, category, "harmonized"
+  )
+  percode_sugg <- .build_suggestions(con, govid, years, category, "harmonized")
+
+  result_years <- if (nrow(result) == 0L) integer(0) else unique(as.integer(result$year))
+  gap_years    <- sort(setdiff(as.integer(years), result_years))
+
+  data.frame(
+    category         = category,
+    category_type    = category_type,
+    canonical_govid  = govid,
+    n_result_rows    = nrow(result),
+    coarse_gap_years = paste(gap_years, collapse = ","),
+    n_coarse         = length(coarse_sugg),
+    n_selfcov        = length(selfcov_sugg),
+    n_percode        = length(percode_sugg),
+    fired_coarse     = length(coarse_sugg) > 0L,
+    fired_selfcov    = length(selfcov_sugg) > 0L,
+    fired_percode    = length(percode_sugg) > 0L,
+    coarse_recipes   = .measure_recipe_ids(coarse_sugg),
+    percode_recipes  = .measure_recipe_ids(percode_sugg),
+    stringsAsFactors = FALSE
+  )
+}
+
+#' Columns that identify a disagreeing query well enough for a human to go
+#' and inspect it, in print order. Intersected with what `detail` actually
+#' has, so this works on a minimal hand-built frame too.
+#' @noRd
+.MEASURE_IDENTITY_COLS <- c(
+  "category", "category_type", "canonical_govid", "coarse_gap_years",
+  "n_result_rows", "coarse_recipes", "percode_recipes"
+)
+
+#' Separate the two flows the `*_delta_pp` figures net together.
+#'
+#' Per-code is NOT a widening of coarse (see this file's header): the two
+#' checks pair different gap tests with different coverage tests, so moving
+#' coarse -> percode both adds and removes firings. This splits the
+#' disagreement into:
+#'   - `violations`: coarse fired, per-code did NOT -- signposting coverage
+#'     LOST. These are what a net delta hides. `holds` is FALSE whenever
+#'     this is non-empty, i.e. whenever coarse is not a subset of per-code.
+#'   - `additions`: per-code fired, coarse did NOT -- the expected gain.
+#'
+#' Deliberately returns the offending rows, not just counts, so the
+#' Checkpoint R3 ruling can be made against named (category, government,
+#' year) cases. Deliberately does NOT assert -- the violation set is really
+#' non-empty on the staged corpus, and a hard assertion here would only
+#' break the harness that is supposed to report it.
+#' @noRd
+.measure_subset_relation <- function(detail) {
+  required <- c("category", "canonical_govid", "fired_coarse", "fired_percode")
+  absent   <- if (is.data.frame(detail)) setdiff(required, names(detail)) else required
+  if (!is.data.frame(detail) || length(absent) > 0L) {
+    stop("`detail` must be a data frame with columns ",
+         paste(required, collapse = ", "), " (missing: ",
+         paste(absent, collapse = ", "), ").", call. = FALSE)
+  }
+  fired_coarse  <- as.logical(detail$fired_coarse)
+  fired_percode <- as.logical(detail$fired_percode)
+  if (anyNA(fired_coarse) || anyNA(fired_percode)) {
+    stop("`fired_coarse`/`fired_percode` must be non-NA logicals.", call. = FALSE)
+  }
+
+  keep     <- intersect(.MEASURE_IDENTITY_COLS, names(detail))
+  viol_idx <- which(fired_coarse & !fired_percode)
+  add_idx  <- which(fired_percode & !fired_coarse)
+
+  list(
+    holds          = length(viol_idx) == 0L,
+    n_queries      = nrow(detail),
+    n_coarse_fired = sum(fired_coarse),
+    n_percode_fired = sum(fired_percode),
+    n_both         = sum(fired_coarse & fired_percode),
+    n_violations   = length(viol_idx),
+    n_additions    = length(add_idx),
+    violations     = detail[viol_idx, keep, drop = FALSE],
+    additions      = detail[add_idx, keep, drop = FALSE]
+  )
+}
+
+#' Render a data frame of disagreeing queries as indented report lines,
+#' capped at `max_rows` with an explicit note about what was withheld (the
+#' full set is always in the returned `$subset_relation`).
+#' @noRd
+.measure_fmt_rows <- function(df, max_rows = 50L) {
+  if (nrow(df) == 0L) return("    (none)")
+  shown <- utils::head(df, max_rows)
+  out   <- paste0("    ", utils::capture.output(print(shown, row.names = FALSE)))
+  if (nrow(df) > max_rows) {
+    out <- c(out, sprintf("    ... %d more row(s) not shown; full set in $subset_relation.",
+                          nrow(df) - max_rows))
+  }
+  out
+}
+
+#' Format `.measure_subset_relation()` as a prominent, clearly-labelled
+#' report section. States plainly whether coarse is a subset of per-code
+#' and, when it is not, exactly where it breaks.
+#' @noRd
+.measure_format_subset_report <- function(rel, max_rows = 50L) {
+  lines <- c(
+    "==== Subset relation: is COARSE a subset of PER-CODE? ====",
+    sprintf("Queries: %d  |  coarse fired: %d  |  per-code fired: %d  |  both: %d",
+            rel$n_queries, rel$n_coarse_fired, rel$n_percode_fired, rel$n_both)
+  )
+  if (rel$holds) {
+    lines <- c(lines, sprintf(paste(
+      "HOLDS: coarse IS a subset of per-code -- 0 of %d queries fire under",
+      "coarse but not per-code. On THIS battery the delta is a pure",
+      "addition of %d query/queries, with no coverage lost."
+    ), rel$n_queries, rel$n_additions))
+  } else {
+    lines <- c(lines,
+      "*** VIOLATED: coarse is NOT a subset of per-code. ***",
+      sprintf(paste(
+        "%d of %d queries fire under COARSE but NOT under PER-CODE:",
+        "signposting coverage the move LOSES."
+      ), rel$n_violations, rel$n_queries),
+      sprintf(paste(
+        "Every delta reported above is therefore a NET of %d addition(s)",
+        "MINUS %d loss(es), and understates both. Do not read it as",
+        "'per-code fires wherever coarse did, plus more'."
+      ), rel$n_additions, rel$n_violations),
+      "",
+      paste("  COVERAGE LOST -- coarse fired, per-code silent.",
+            "`coarse_gap_years` is the requested year(s) in which the whole",
+            "category result was empty (coarse's own trigger evidence):"),
+      .measure_fmt_rows(rel$violations, max_rows)
+    )
+  }
+  c(lines, "",
+    sprintf("  COVERAGE ADDED -- per-code fired, coarse silent (%d query/queries):",
+            rel$n_additions),
+    .measure_fmt_rows(rel$additions, max_rows))
+}
+
+#' Measure the coarse-vs-per-code signposting suggestion rate over a
+#' realistic query battery (every category x a pre/post-boundary_year span
+#' x up to n_gov sampled governments).
+#'
+#' @param corpus_url Corpus to measure against. Defaults to
+#'   `Sys.getenv("USCOGDATA_URL")`; if that's unset, falls back to the
+#'   bundled v5 fixture (so the script runs out of the box). Re-run with
+#'   `USCOGDATA_URL` pointed at the staged/full corpus later.
+#' @param n_gov Governments to sample (deterministically). "Up to" -- if
+#'   the corpus has fewer distinct governments in the measured years than
+#'   this, every one of them is used.
+#' @param seed Sampling seed (fixed for reproducibility).
+#' @param boundary_year,pre_years_n,post_years_n Define the desired query
+#'   span: `pre_years_n` years immediately before `boundary_year`, unioned
+#'   with `post_years_n` years starting at `boundary_year`. Falls back to
+#'   the corpus's widest actually-available span when this can't be filled
+#'   (see `.measure_year_span()`).
+#' @param coarse_ref Git ref to pull the R2 coarse `.build_suggestions()`
+#'   from.
+#' @param selfcov_ref Git ref to pull Task 19c's first, self-coverage-
+#'   allowed per-code `.build_suggestions()` from (amended after review).
+#' @param verbose Print progress/notes as the battery runs.
+#' @return Invisibly, a list with `corpus_url`, `years`, `span_note`,
+#'   `full_design`, `govids`, `seed`, `detail` (one row per query),
+#'   `by_category`, and `summary`.
+#' @noRd
+measure_signposting_rate <- function(corpus_url = Sys.getenv("USCOGDATA_URL", unset = NA),
+                                      n_gov = 20L,
+                                      seed = 19L,
+                                      boundary_year = 2012L,
+                                      pre_years_n = 3L,
+                                      post_years_n = 3L,
+                                      coarse_ref = "b0df1ec",
+                                      selfcov_ref = "da72bf3",
+                                      verbose = TRUE) {
+  pkgload::load_all(".", quiet = TRUE)
+
+  if (is.na(corpus_url) || !nzchar(corpus_url)) {
+    corpus_url <- paste0(
+      system.file("extdata/fixture_corpus", package = "uscogdata"), "/"
+    )
+    if (verbose) {
+      message("No USCOGDATA_URL set; defaulting to the bundled v5 fixture: ",
+              corpus_url)
+    }
+  }
+
+  old_url <- Sys.getenv("USCOGDATA_URL", unset = NA)
+  cog_close()
+  Sys.setenv(USCOGDATA_URL = corpus_url)
+  on.exit({
+    cog_close()
+    if (is.na(old_url)) Sys.unsetenv("USCOGDATA_URL") else Sys.setenv(USCOGDATA_URL = old_url)
+  }, add = TRUE)
+
+  con <- cog_open()
+  coarse_env  <- .measure_load_git_impl(git_ref = coarse_ref)
+  selfcov_env <- .measure_load_git_impl(git_ref = selfcov_ref)
+
+  span   <- .measure_year_span(con, boundary_year, pre_years_n, post_years_n)
+  years  <- span$years
+  if (verbose) message(span$note)
+
+  govids <- .measure_sample_govids(con, years, n = n_gov, seed = seed)
+  if (verbose) {
+    message(sprintf("Sampled %d government(s) (seed = %d) from %d present in years %s.",
+                     length(govids), seed,
+                     length(DBI::dbGetQuery(con, sprintf(
+                       "SELECT DISTINCT canonical_govid FROM long WHERE year IN (%s)",
+                       paste(years, collapse = ",")))$canonical_govid),
+                     paste(years, collapse = ", ")))
+  }
+
+  categories <- DBI::dbGetQuery(con,
+    "SELECT DISTINCT category, category_type FROM summary_categories
+     WHERE category IS NOT NULL ORDER BY category_type, category")
+  if (verbose) {
+    message(sprintf("Battery: %d categories x %d governments = %d queries.",
+                     nrow(categories), length(govids),
+                     nrow(categories) * length(govids)))
+  }
+
+  rows <- vector("list", nrow(categories) * length(govids))
+  k <- 0L
+  for (ci in seq_len(nrow(categories))) {
+    for (gv in govids) {
+      k <- k + 1L
+      rows[[k]] <- .measure_one_query(
+        con, coarse_env, selfcov_env,
+        category = categories$category[ci],
+        category_type = categories$category_type[ci],
+        govid = gv, years = years
+      )
+    }
+  }
+  detail <- do.call(rbind, rows)
+  # percode only ever fires where selfcov also fires (percode is a strict
+  # narrowing of selfcov: same gap detection, plus the self-coverage path
+  # removed) -- this is what makes the decomposition below exact rather
+  # than approximate. Checked, not assumed.
+  stopifnot(all(detail$fired_percode <= detail$fired_selfcov))
+  detail$fired_selfcov_only <- detail$fired_selfcov & !detail$fired_percode
+
+  # coarse vs percode is NOT a subset relation the way percode vs selfcov
+  # is (see header). Measured and REPORTED, never asserted: the violation
+  # set is genuinely non-empty on the staged corpus, and a stopifnot() here
+  # would break the harness whose whole job is to surface it.
+  subset_relation <- .measure_subset_relation(detail)
+  detail$coarse_only  <- detail$fired_coarse & !detail$fired_percode
+  detail$percode_only <- detail$fired_percode & !detail$fired_coarse
+
+  by_category <- dplyr::summarise(
+    dplyr::group_by(detail, category, category_type),
+    n_queries      = dplyr::n(),
+    coarse_rate    = mean(fired_coarse),
+    selfcov_rate   = mean(fired_selfcov),
+    percode_rate   = mean(fired_percode),
+    original_delta_pp  = (mean(fired_selfcov) - mean(fired_coarse)) * 100,
+    corrected_delta_pp = (mean(fired_percode) - mean(fired_coarse)) * 100,
+    selfcov_share_pp    = mean(fired_selfcov_only) * 100,
+    # the two flows corrected_delta_pp nets together, per category
+    n_coarse_only  = sum(coarse_only),
+    n_percode_only = sum(percode_only),
+    .groups = "drop"
+  )
+  by_category <- dplyr::arrange(by_category, dplyr::desc(corrected_delta_pp))
+
+  summary_overall <- data.frame(
+    n_queries     = nrow(detail),
+    n_categories  = nrow(categories),
+    n_governments = length(govids),
+    coarse_fired  = sum(detail$fired_coarse),
+    selfcov_fired = sum(detail$fired_selfcov),
+    percode_fired = sum(detail$fired_percode),
+    coarse_rate   = mean(detail$fired_coarse),
+    selfcov_rate  = mean(detail$fired_selfcov),
+    percode_rate  = mean(detail$fired_percode)
+  )
+  summary_overall$original_delta_pp  <- (summary_overall$selfcov_rate - summary_overall$coarse_rate) * 100
+  summary_overall$corrected_delta_pp <- (summary_overall$percode_rate - summary_overall$coarse_rate) * 100
+  summary_overall$selfcov_share_pp   <- mean(detail$fired_selfcov_only) * 100
+  summary_overall$relative_increase  <- if (summary_overall$coarse_rate > 0) {
+    summary_overall$percode_rate / summary_overall$coarse_rate - 1
+  } else {
+    NA_real_
+  }
+
+  if (verbose) {
+    message(sprintf(
+      "Coarse rate: %.4f (%d/%d)  |  Self-cov-allowed rate: %.4f (%d/%d)  |  Corrected per-code rate: %.4f (%d/%d)",
+      summary_overall$coarse_rate, summary_overall$coarse_fired, summary_overall$n_queries,
+      summary_overall$selfcov_rate, summary_overall$selfcov_fired, summary_overall$n_queries,
+      summary_overall$percode_rate, summary_overall$percode_fired, summary_overall$n_queries
+    ))
+    message(sprintf(
+      "Original delta (selfcov - coarse): %+.2f pp  |  Corrected delta (percode - coarse): %+.2f pp  |  Self-coverage share of original delta: %.2f pp (%d/%d queries fired ONLY via self-coverage)",
+      summary_overall$original_delta_pp, summary_overall$corrected_delta_pp,
+      summary_overall$selfcov_share_pp,
+      sum(detail$fired_selfcov_only), summary_overall$n_queries
+    ))
+    message(if (subset_relation$holds) {
+      sprintf("Subset relation coarse <= percode HOLDS (0 coarse-only firings); delta is a pure addition of %d.",
+              subset_relation$n_additions)
+    } else {
+      sprintf("*** Subset relation coarse <= percode VIOLATED: %d coarse-only firing(s) LOST vs %d percode-only added. Deltas above are NETS. ***",
+              subset_relation$n_violations, subset_relation$n_additions)
+    })
+  }
+
+  invisible(list(
+    corpus_url      = corpus_url,
+    years           = years,
+    span_note       = span$note,
+    full_design     = span$full_design,
+    govids          = govids,
+    seed            = seed,
+    n_categories    = nrow(categories),
+    detail          = detail,
+    by_category     = by_category,
+    summary         = summary_overall,
+    subset_relation = subset_relation
+  ))
+}
+
+if (identical(environment(), globalenv()) && sys.nframe() == 0L) {
+  res <- measure_signposting_rate()
+  cat("\n==== Query battery ====\n")
+  cat("Corpus:", res$corpus_url, "\n")
+  cat("Years:", paste(res$years, collapse = ", "), "\n")
+  cat("Full 3-year pre/post-2012 design achieved:", res$full_design, "\n")
+  cat(res$span_note, "\n")
+  cat("Governments sampled:", length(res$govids), "\n\n")
+
+  cat("==== Overall summary (coarse / self-coverage-allowed / corrected per-code) ====\n")
+  print(res$summary)
+
+  cat("\n==== By category (sorted by corrected delta, descending) ====\n")
+  cat("NOTE: corrected_delta_pp is a NET. n_coarse_only = firings LOST going\n")
+  cat("coarse -> percode; n_percode_only = firings ADDED. A category can be\n")
+  cat("negative (net coverage loss) even when the overall figure is positive.\n")
+  print(as.data.frame(res$by_category), row.names = FALSE)
+
+  cat("\n")
+  cat(paste(.measure_format_subset_report(res$subset_relation), collapse = "\n"), "\n")
+}

data-raw/regenerate_fixture_corpus.R

@@ -11,14 +11,12 @@
 #      Each partition is a full year (all states/govs) as published, so
 #      Broward County FL and every other previously-pinned government stay
 #      covered without any per-gov slicing logic.
-#   2. Copies every metadata parquet the publish tree ships (see
+#   2. Copies the full canonical_fips_xwalk.parquet, canonical_alias.parquet,
-#      .FIXTURE_METADATA_FILES) as-is. These are small cross-vintage
+#      summary_categories.parquet, harmonization_map.parquet,
-#      registries, not partitioned by year, so the fixture ships the complete
+#      harmonization_recipes.parquet, and series_breaks.parquet metadata
-#      tables rather than a year-scoped subset. representation.parquet and
+#      tables as-is (these are small cross-vintage registries, not
-#      code_set.parquet are what make the sparse wide era interpretable --
+#      partitioned by year, so the fixture ships the complete tables rather
-#      absence means "$0" in a dense_source year and "not reported" in a
+#      than a year-scoped subset).
-#      sparse_source one -- so a fixture without them cannot represent the
-#      published corpus.
 #   3. Resyncs the four reference docs (data_dictionary.md,
 #      reader-specification.md, README.md, series_breaks.md) from the
 #      publish tree's docs/.
@@ -40,22 +38,6 @@
 #   source("data-raw/regenerate_fixture_corpus.R")
 #   regenerate_fixture_corpus(publish_cache_dir = "/path/to/publish_cache")
 
-# Every metadata parquet the publish tree ships, in the order they appear in
-# the corpus manifest. Single source of truth for both the copy step and the
-# fixture manifest, so the two can never drift apart.
-.FIXTURE_METADATA_FILES <- c(
-  "canonical_alias.parquet",
-  "canonical_fips_xwalk.parquet",
-  "census_collection_coverage.parquet",
-  "code_set.parquet",
-  "harmonization_map.parquet",
-  "harmonization_recipes.parquet",
-  "lineage_events.parquet",
-  "representation.parquet",
-  "series_breaks.parquet",
-  "summary_categories.parquet"
-)
-
 regenerate_fixture_corpus <- function(
     publish_cache_dir = file.path(
       "..", "cog_pipeline", "_targets", "publish_cache"
@@ -118,11 +100,20 @@ regenerate_fixture_corpus <- function(
   invisible(NULL)
 }
 
-# Copy the full (not year-scoped) metadata tables listed in
+# Copy the full (not year-scoped) canonical_fips_xwalk, canonical_alias,
-# .FIXTURE_METADATA_FILES.
+# summary_categories, and (schema v5+) harmonization_map/
+# harmonization_recipes/series_breaks parquet tables.
 #' @noRd
 .copy_metadata_parquets <- function(publish_cache_dir, fixture_dir) {
-  for (f in .FIXTURE_METADATA_FILES) {
+  files <- c(
+    "canonical_fips_xwalk.parquet",
+    "canonical_alias.parquet",
+    "summary_categories.parquet",
+    "harmonization_map.parquet",
+    "harmonization_recipes.parquet",
+    "series_breaks.parquet"
+  )
+  for (f in files) {
     src <- file.path(publish_cache_dir, "data", f)
     dst <- file.path(fixture_dir, "data", f)
     if (!file.exists(src)) {
@@ -188,7 +179,15 @@ regenerate_fixture_corpus <- function(
     )
   })
 
-  metadata <- lapply(.FIXTURE_METADATA_FILES, function(f) {
+  metadata_files <- c(
+    "canonical_alias.parquet",
+    "canonical_fips_xwalk.parquet",
+    "summary_categories.parquet",
+    "harmonization_map.parquet",
+    "harmonization_recipes.parquet",
+    "series_breaks.parquet"
+  )
+  metadata <- lapply(metadata_files, function(f) {
     rel <- file.path("data", f)
     path <- file.path(fixture_dir, rel)
     list(
@@ -204,16 +203,13 @@ regenerate_fixture_corpus <- function(
     pipeline_commit = source_manifest$pipeline_commit,
     fixture_note = paste(
       "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full",
-      "corpus available via USCOGDATA_URL. Regenerated from the sparsified",
+      "corpus available via USCOGDATA_URL. Regenerated for Phase R2",
-      "schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit",
+      "(schema_version 5, harmonization_map/harmonization_recipes/",
-      "zeros, so FY2011 absence means Census published $0 while FY2012+",
+      "series_breaks parquet tables added). 2011/2012 straddle the",
-      "absence means not reported. representation.parquet and",
+      "wide-aggregate -> modern-leaf format boundary exercised by basis=",
-      "code_set.parquet carry that rule and ship in full, as do every other",
+      "\"harmonized\" and recipe= queries; 2019/2020 retain the prior",
-      "metadata table in the publish tree. 2011/2012 straddle both the",
+      "per-capita/CPI regression anchors. Full canonical_fips_xwalk master",
-      "wide-aggregate -> modern-leaf format boundary (exercised by",
+      "and canonical_alias lookup table included via",
-      "basis=\"harmonized\" and recipe= queries) and the dense -> sparse",
-      "representation boundary (SB194); 2019/2020 retain the prior",
-      "per-capita/CPI regression anchors. Regenerated via",
       "data-raw/regenerate_fixture_corpus.R."
     ),
     data_vintage    = source_manifest$data_vintage,

inst/extdata/fixture_corpus/manifest.json

@@ -1,8 +1,8 @@
 {
   "schema_version": 6,
-  "built_at": "2026-07-30T14:07:36Z",
+  "built_at": "2026-07-23T16:14:30Z",
-  "pipeline_commit": "83f9715",
+  "pipeline_commit": "4f992a0",
-  "fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated from the sparsified schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit zeros, so FY2011 absence means Census published $0 while FY2012+ absence means not reported. representation.parquet and code_set.parquet carry that rule and ship in full, as do every other metadata table in the publish tree. 2011/2012 straddle both the wide-aggregate -> modern-leaf format boundary (exercised by basis=\"harmonized\" and recipe= queries) and the dense -> sparse representation boundary (SB194); 2019/2020 retain the prior per-capita/CPI regression anchors. Regenerated via data-raw/regenerate_fixture_corpus.R.",
+  "fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated for Phase R2 (schema_version 5, harmonization_map/harmonization_recipes/ series_breaks parquet tables added). 2011/2012 straddle the wide-aggregate -> modern-leaf format boundary exercised by basis= \"harmonized\" and recipe= queries; 2019/2020 retain the prior per-capita/CPI regression anchors. Full canonical_fips_xwalk master and canonical_alias lookup table included via data-raw/regenerate_fixture_corpus.R.",
   "data_vintage": {
     "source_vintages": {
       "2012": "10162019",
@@ -36,9 +36,9 @@
       {
         "year": 2011,
         "path": "data/long/year=2011/part-0.parquet",
-        "sha256": "7848e18497080c8980a4f89c5b386205b2c5bc90db6773827ea01ab3943d16b1",
+        "sha256": "84302ab364dc9fc3b3fbbc3c3f8b826e3508b4d73ff7c42d094d3863cd1e37b5",
-        "row_count": 496004,
+        "row_count": 2864212,
-        "size_bytes": 2202455
+        "size_bytes": 3845911
       },
       {
         "year": 2012,
@@ -74,14 +74,9 @@
         "description": "canonical_fips_xwalk.parquet"
       },
       {
-        "path": "data/census_collection_coverage.parquet",
+        "path": "data/summary_categories.parquet",
-        "sha256": "143e025616cde684da7c4442bc00d07fbd1556fabb0ea96223931b737e5d10a4",
+        "sha256": "8e6fcd4dd9bb4723841a67233b19388c9762dfc23b4479501183cebf7ea3c1b5",
-        "description": "census_collection_coverage.parquet"
+        "description": "summary_categories.parquet"
-      },
-      {
-        "path": "data/code_set.parquet",
-        "sha256": "4cffcb0198dd51e4ff2b694050bb371a5f9965cdac12f25521cb628fb8e118a9",
-        "description": "code_set.parquet"
       },
       {
         "path": "data/harmonization_map.parquet",
@@ -93,25 +88,10 @@
         "sha256": "1133e9a0b02f8f34f5f936e55c5ecd596bb8a55d8425dcce76767f0f3203581c",
         "description": "harmonization_recipes.parquet"
       },
-      {
-        "path": "data/lineage_events.parquet",
-        "sha256": "36c16acfbe621d61010984767f1c566993b8a5f481a2c1e134c4c0a600e4502f",
-        "description": "lineage_events.parquet"
-      },
-      {
-        "path": "data/representation.parquet",
-        "sha256": "31ec328a7dd505a321b45f97aafff12e53d68a1a986f63509863035b22a4360d",
-        "description": "representation.parquet"
-      },
       {
         "path": "data/series_breaks.parquet",
-        "sha256": "5ae050dd7a76c4d25e5f99e7c2e81c1896482e3504e0443b47ab5d78ba148953",
+        "sha256": "b0b6794b6887a4f300079adfa10029c2a77109faa4952fbff1c5a270793cc02b",
         "description": "series_breaks.parquet"
-      },
-      {
-        "path": "data/summary_categories.parquet",
-        "sha256": "e71d6d70d767c26c983fe56213baf204355f879582aa94841e62d9aea1877f83",
-        "description": "summary_categories.parquet"
       }
     ]
   },

inst/schemas/provenance-v1.json

@@ -12,19 +12,6 @@
     "category":       { "type": ["string", "array", "null"] },
     "basis":          { "type": ["string", "null"] },
     "basis_note":     { "type": ["string", "null"] },
-    "expenditure_concept": {
-      "type": "string",
-      "enum": ["direct", "total"],
-      "description": "Which spending concept produced this result. 'direct' is the government's own E/F/G spending; 'total' adds its intergovernmental payments (M to local governments, L to state governments). Only 'direct' is valid for results combined across governments."
-    },
-    "expenditure_concept_note": {
-      "type": ["string", "null"],
-      "description": "How the intergovernmental leg was assembled; null for 'direct'."
-    },
-    "expenditure_concept_direct_suppressed": {
-      "type": "boolean",
-      "description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'direct'. See the affected rows' `notes` for the recovering recipe, if any."
-    },
     "harmonization":  { "type": "object" },
     "recipe":         { "type": ["object", "null"] },
     "suggestions":    { "type": "array" },
@@ -33,20 +20,6 @@
     "aggregate_fallback": { "type": ["object", "null"] },
     "transformations":{ "type": "object" },
     "series_break_refs": { "type": "array", "items": { "type": "string" } },
-    "completion": {
-      "type": "object",
-      "description": "What `complete = TRUE` filled. `applied` is FALSE on an ordinary query. `rows_filled` counts cells added to the requested grid, and `absence_means` maps each requested year to the meaning of an absent cell there ('census_zero' in a dense_source year, 'not_reported' in a sparse_source one). Filled rows carry `value_source` in the result: 'reported', 'census_zero' (amount 0 -- Census published $0), or 'not_reported' (amount NA -- unknown).",
-      "properties": {
-        "applied": { "type": "boolean" },
-        "rows_filled": { "type": "integer" },
-        "absence_means": { "type": "object" }
-      }
-    },
-    "corpus_break_refs": {
-      "type": "array",
-      "items": { "type": "string" },
-      "description": "Ids of catalogued series breaks whose fin_code is the literal 'ALL' -- caveats about the corpus as a whole (dollar precision across 1976/1977, imputation exclusion from 2002, the dense -> sparse representation change at 2012, the government id scheme change at 2017) rather than about one item code. Selected on the break_year window alone, so they do not depend on which codes a result contains. Disjoint from series_break_refs by construction: an entry qualifies the whole result, not one series."
-    },
     "manifest":       { "type": "object" },
     "sql_query":      { "type": "string" }
   }

inst/sql/20-spending_long.sql

@@ -1,5 +1,5 @@
 CREATE OR REPLACE VIEW spending_long AS
 SELECT *
 FROM long
-WHERE LEFT(item_code, 1) IN ('E', 'F', 'G')
+WHERE LEFT(item_code, 1) IN ('E', 'F', 'G', 'K')
   AND NOT is_aggregate;

inst/sql/22-spending_long_harmonized.sql

@@ -3,4 +3,4 @@ SELECT * REPLACE (harmonized_code AS item_code)
 FROM long
 WHERE NOT is_aggregate
   AND harmonized_code IS NOT NULL
-  AND LEFT(harmonized_code, 1) IN ('E', 'F', 'G');
+  AND LEFT(harmonized_code, 1) IN ('E', 'F', 'G', 'K');

inst/sql/24-ig_long.sql

@@ -1,18 +0,0 @@
--- Intergovernmental expenditure rows (M = to local govts, L = to state govts).
---
--- Deliberately does NOT filter `NOT is_aggregate`, unlike spending_long. In the
--- wide era (<= FY2011) the IG families M05/M12/M47/M89/L47/L89 are published
--- ONLY as aggregate-flagged rows -- filtering them would hide ~70% of legacy IG
--- dollars and make Total silently collapse to Direct. This is safe because the
--- aggregate codes and their modern leaf components are strictly year-disjoint
--- (M47 ends 2011 / M94 starts 2012; M89 is aggregate only <= 2011 and a leaf
--- from 2012 alongside M91-93), so no row is ever counted twice. Same argument
--- the pipeline's recipe joins use.
---
--- `L--` IS excluded: it is the IG-to-state FAMILY TOTAL and genuinely rolls up
--- the L-NN codes, so including it would double-count.
-CREATE OR REPLACE VIEW ig_long AS
-SELECT *
-FROM long
-WHERE LEFT(item_code, 1) IN ('M', 'L')
-  AND item_code NOT LIKE '%--';

inst/sql/25-ig_long_harmonized.sql

@@ -1,15 +0,0 @@
--- Harmonized-basis IG rows. Uses COALESCE(harmonized_code, item_code) rather
--- than harmonized_code alone: aggregate rows carry NO harmonized_code by
--- construction (harmonized space is leaf-only), so a plain
--- `harmonized_code IS NOT NULL` filter would drop every legacy IG aggregate --
--- in the bundled fixture corpus (year 2011; 2012+ all carry a harmonized_code)
--- that is $379,016,063k across 25,688 M rows and $2,277,458k across 19,266 L
--- rows (`SELECT year, LEFT(item_code,1), SUM(amt), COUNT(*) FROM ig_long
--- WHERE harmonized_code IS NULL GROUP BY 1, 2`). COALESCE keeps the one real
--- IG collapse rule (M38 -> M36, SB012, year-disjoint 1967-2011 vs 2012+)
--- while never dropping a row.
-CREATE OR REPLACE VIEW ig_long_harmonized AS
-SELECT * REPLACE (COALESCE(harmonized_code, item_code) AS item_code)
-FROM long
-WHERE LEFT(item_code, 1) IN ('M', 'L')
-  AND item_code NOT LIKE '%--';

inst/sql/36-representation.sql

@@ -1,3 +0,0 @@
-CREATE OR REPLACE VIEW representation AS
-SELECT *
-FROM read_parquet('{url}data/representation.parquet');

inst/sql/37-code_set.sql

@@ -1,3 +0,0 @@
-CREATE OR REPLACE VIEW code_set AS
-SELECT *
-FROM read_parquet('{url}data/code_set.parquet');

inst/sql/44-ig_annotated.sql

@@ -1,16 +0,0 @@
-CREATE OR REPLACE VIEW ig_annotated AS
-SELECT
-  s.*,
-  x.gov_name       AS xwalk_gov_name,
-  x.govs_type,
-  x.type_label,
-  x.fips_state     AS xwalk_fips_state,
-  x.fips_county    AS xwalk_fips_county,
-  x.fips_place,
-  x.population_acs,
-  c.category,
-  c.category_type,
-  c.spend_subtype
-FROM ig_long s
-LEFT JOIN canonical_fips_xwalk x USING (canonical_govid)
-LEFT JOIN summary_categories   c USING (item_code);

inst/sql/45-ig_annotated_harmonized.sql

@@ -1,16 +0,0 @@
-CREATE OR REPLACE VIEW ig_annotated_harmonized AS
-SELECT
-  s.*,
-  x.gov_name       AS xwalk_gov_name,
-  x.govs_type,
-  x.type_label,
-  x.fips_state     AS xwalk_fips_state,
-  x.fips_county    AS xwalk_fips_county,
-  x.fips_place,
-  x.population_acs,
-  c.category,
-  c.category_type,
-  c.spend_subtype
-FROM ig_long_harmonized s
-LEFT JOIN canonical_fips_xwalk x USING (canonical_govid)
-LEFT JOIN summary_categories   c USING (item_code);

man/cog_find_peers.Rd

@@ -11,8 +11,7 @@ cog_find_peers(
   same_state = FALSE,
   pop_range = c(0.7, 1.3),
   is_ratio = TRUE,
-  max_peers = 10L,
+  max_peers = 10L
-  coverage = c("all", "census", "consistent")
 )
 }
 \arguments{
@@ -35,14 +34,6 @@ target's population at `year` to produce absolute bounds. If `FALSE`,
 `pop_range` is interpreted as absolute population counts.}
 
 \item{max_peers}{Integer cap on the number of peers returned.}
-
-\item{coverage}{Survey-cycle handling; see [cog_peer_compare()]. Here it
-governs the cohort VINTAGE when `year` is `NULL`: `"census"` snaps to the
-most recent census year with an observed population, so a cohort is not
-built from a sample year in which most of the candidate universe is
-absent. `"consistent"` needs a year range, which cohort selection does not
-have, so it selects like `"all"` and is carried on the result as
-`attr(x, "coverage")` for [cog_peer_compare()].}
 }
 \value{
 Tibble with columns `canonical_govid`, `gov_name`, `fips_state`,

man/cog_geographic_rollup.Rd

@@ -9,9 +9,7 @@ cog_geographic_rollup(
   category,
   years,
   per_capita = FALSE,
-  adjust_to_year = NULL,
+  adjust_to_year = NULL
-  expenditure_concept = c("direct", "total"),
-  coverage = c("all", "census", "consistent")
 )
 }
 \arguments{
@@ -29,33 +27,6 @@ population from `gov_population_yearly`. Govs with missing population
 are excluded from the result.}
 
 \item{adjust_to_year}{Integer base year for CPI-U conversion, or `NULL`.}
-
-\item{expenditure_concept}{`"direct"` (default) or `"total"`. Currently only
-`"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
-single-government queries but cannot be used here because combining Total
-across multiple layers of government double-counts intergovernmental
-transfers (a state's payment to a school district is the same dollar the
-district reports as its own Direct spending).}
-
-\item{coverage}{How to handle the Census of Governments survey cycle,
-  which is a **complete census only in years ending in 2 and 7** -- every
-  other year is a sample, and the sample varies enormously (on the bundled
-  fixture, Wisconsin's 608-city universe reports 597 governments in FY2012
-  and 112 in FY2019).
-
-  * `"all"` (default) -- every unit that reported that year. Unchanged
-    behaviour, so existing code keeps working.
-  * `"census"` -- census years only. Aborts if the requested range holds
-    none, rather than silently returning nothing.
-  * `"consistent"` -- only units reporting in *every* requested year, giving
-    a balanced panel.
-
-  Regardless of mode, `provenance$coverage` always carries per-year
-  `n_units_reporting`, `n_units_expected` and `is_census_year`, and
-  `provenance$coverage_mode` records the mode. `is_census_year` is a
-  statement about the **survey calendar**, never a claim of completeness:
-  FY1967 is a census year in which only 97 of Wisconsin's 608 cities
-  report. `n_units_reporting` is the number that tells the truth.}
 }
 \value{
 Tibble with columns `year`, `layer`, `canonical_govid`, `gov_name`,

man/cog_gov_search.Rd

@@ -32,11 +32,8 @@ the cross-vintage canonical-government registry. Operates in two modes:
 }
 \details{
 * **Utility mode** (single `name`, the original behavior): returns all
-  rows whose `gov_name` contains `name` as a **literal, case-insensitive
+  rows whose `gov_name` matches the regex case-insensitively, sorted by
-  substring**, sorted by `population_acs` descending. Useful for
+  `population_acs` descending. Useful for exploratory lookups.
-  exploratory lookups. Regex metacharacters in `name` are escaped, so a
-  government is findable by its own complete name even when that name
-  contains parentheses or a period.
 * **Basket mode** (`length(name) > 1`): resolves each input row to a
   single canonical govid and returns a tibble in input order, suitable
   for piping straight into [cog_spending()] / [cog_revenue()] /
@@ -48,8 +45,7 @@ the cross-vintage canonical-government registry. Operates in two modes:
 1. Filter `canonical_fips_xwalk` by `state` and (if non-NA) `type`.
 2. **Exact pass:** case-insensitive equality against `gov_name`.
    Single hit -> resolved. Multiple -> step 4.
-3. **Substring fallback:** case-insensitive literal substring against
+3. **Substring fallback:** case-insensitive regex against `gov_name`.
-   `gov_name` (metacharacters escaped).
    Single hit -> resolved (`match_method = "substring"`). Zero hits ->
    `status = "no_match"`. Multiple hits -> step 4.
 4. **Disambiguation:** if matches share one `govs_type`, pick the
@@ -62,7 +58,7 @@ inputs (`ambiguous` / `no_match`) appear only in the sidecar.
 }
 \examples{
 \dontrun{
-# Utility mode — exploratory substring lookup
+# Utility mode — exploratory regex lookup
 cog_gov_search("broward", state = "FL")
 
 # Basket mode — resolve a known cohort

man/cog_peer_compare.Rd

@@ -10,9 +10,7 @@ cog_peer_compare(
   category,
   years,
   per_capita = TRUE,
-  adjust_to_year = NULL,
+  adjust_to_year = NULL
-  expenditure_concept = c("direct", "total"),
-  coverage = c("all", "census", "consistent")
 )
 }
 \arguments{
@@ -29,37 +27,6 @@ cog_peer_compare(
 population.}
 
 \item{adjust_to_year}{Integer base year for CPI-U conversion or `NULL`.}
-
-\item{expenditure_concept}{`"direct"` (default) or `"total"`. Currently only
-`"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
-single-government queries but cannot be used here because combining Total
-across peer sets counts intergovernmental transfers twice.}
-
-\item{coverage}{How to handle the Census of Governments survey cycle,
-  which is a **complete census only in years ending in 2 and 7** -- every
-  other year is a sample, and the sample varies enormously (on the bundled
-  fixture, Wisconsin's 608-city universe reports 597 governments in FY2012
-  and 112 in FY2019).
-
-  * `"all"` (default) -- every unit that reported that year. Unchanged
-    behaviour, so existing code keeps working.
-  * `"census"` -- census years only. Aborts if the requested range holds
-    none, rather than silently returning nothing.
-  * `"consistent"` -- only units reporting in *every* requested year, giving
-    a balanced panel.
-
-  Regardless of mode, `provenance$coverage` always carries per-year
-  `n_units_reporting`, `n_units_expected` and `is_census_year`, and
-  `provenance$coverage_mode` records the mode. `is_census_year` is a
-  statement about the **survey calendar**, never a claim of completeness:
-  FY1967 is a census year in which only 97 of Wisconsin's 608 cities
-  report. `n_units_reporting` is the number that tells the truth.
-
-  The comparison target is exempt from `"consistent"` balancing -- it is the
-  subject of the comparison, not a member of the cohort -- and the
-  `summary_*` quantiles are computed AFTER the filter, so they describe the
-  cohort actually returned. `n_units_reporting` counts peers only, against
-  the cohort size: "3 of your 15 peers reported in FY2019".}
 }
 \value{
 Tibble matching [cog_spending()]'s columns, plus a `role`
@@ -70,39 +37,11 @@ Tibble matching [cog_spending()]'s columns, plus a `role`
   `attr(peers, "cohort_year")`; `NA` when `peers` was a bare character
   vector). Provenance reports `verb = "cog_peer_compare"`, `peer_count`,
   `cohort_year`, and `cohort_govids`.
-
-  **The `summary_*` rows are per-category quantiles: they are not additive.**
-  Each one is computed **within each `(year, spend_subtype,
-  category)` cell** across the peer set, so a `summary_p50` row is *the
-  median peer's value in that one category*, not *the value of the median
-  peer's total*. The median peer for Police and the median peer for Fire
-  are usually different governments, so summing `summary_*` rows across
-  categories does not give any peer's total and misstates the band it
-  appears to describe — measured at −32.7% to +251.0% across 24 years on
-  one cohort, with a sign flip at FY2012.
-
-  Facet by `role` **and** `category` (the documented use, and what the
-  rows are built for). For a genuine "median peer's total spending" line,
-  sum each peer's own categories first and take the quantile of those
-  per-government totals:
-
-  ```r
-  library(dplyr)
-  cmp |>
-    filter(role %in% c("target", "peer")) |>
-    group_by(year, role, canonical_govid) |>
-    summarise(total = sum(amt_per_capita_real, na.rm = TRUE), .groups = "drop") |>
-    filter(role == "peer") |>
-    group_by(year) |>
-    summarise(p50 = quantile(total, 0.5, na.rm = TRUE))
-  ```
 }
 \description{
 Pulls spending for the target plus a peer set (either a
 [cog_find_peers()] result or a character vector of `canonical_govid`) and
 appends peer-distribution summary rows (`summary_p25`, `summary_p50`,
 `summary_p75`) so the result can be faceted by `role` in a single ggplot
-call. Those summary rows are quantiles **within each category**, not
+call.
-quantiles of each peer's total — see the `@return` section before summing
-them.
 }

man/cog_revenue.Rd

@@ -11,8 +11,7 @@ cog_revenue(
   per_capita = FALSE,
   adjust_to_year = NULL,
   basis = c("harmonized", "raw"),
-  recipe = NULL,
+  recipe = NULL
-  complete = FALSE
 )
 }
 \arguments{
@@ -56,36 +55,12 @@ argument is ignored and the result's provenance reports
 `basis = "recipe"` with an inert `harmonization` block (`applied =
 FALSE`, pointing at the `recipe` block instead) rather than a
 possibly-misleading `"harmonized"`/`"raw"` value.}
-
-\item{complete}{If `TRUE`, fill the requested grid so that a cell the
-  corpus does not carry still appears, labelled with **why** it is
-  missing, and add a `value_source` column to every row:
-
-  * `"reported"` — the corpus carries this cell.
-  * `"census_zero"` — dense-source year (`<= FY2011`), cell absent:
-    Census published `$0`. `amt_nominal` is `0`.
-  * `"not_reported"` — sparse-source year (`>= FY2012`), cell absent: the
-    government did not report, and the value is unknown. `amt_nominal` is
-    `NA`, **not** `0` — writing a zero there would invent data.
-
-  The grid comes from the corpus's `code_set` table, scoped to each
-  government's own type, so a county is never filled with cells only a
-  state can report. Reported rows are passed through untouched.
-
-  Defaults to `FALSE` (the historical behaviour: absent cells simply do
-  not appear). Needs a corpus published from 2026-07-29 onward, which is
-  when `representation`/`code_set` began shipping; aborts with class
-  `uscogdata_representation_unavailable` otherwise. Not available with
-  `recipe` or with `expenditure_concept = "total"` (class
-  `uscogdata_complete_unsupported`) — neither draws its cells from
-  `code_set`.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,
   `revenue_subtype`, `category`, `amt_nominal`, optional `amt_real`,
   optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
-  optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`,
+  optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`.
-  and `value_source` when `complete = TRUE`.
 }
 \description{
 Mirror of [cog_spending()] for revenue categories. One row per

man/cog_spending.Rd

@@ -11,9 +11,7 @@ cog_spending(
   per_capita = FALSE,
   adjust_to_year = NULL,
   basis = c("harmonized", "raw"),
-  recipe = NULL,
+  recipe = NULL
-  expenditure_concept = c("direct", "total"),
-  complete = FALSE
 )
 }
 \arguments{
@@ -57,68 +55,13 @@ argument is ignored and the result's provenance reports
 `basis = "recipe"` with an inert `harmonization` block (`applied =
 FALSE`, pointing at the `recipe` block instead) rather than a
 possibly-misleading `"harmonized"`/`"raw"` value.}
-
-\item{expenditure_concept}{`"direct"` (default) returns only the
-  government's own direct spending (item codes `E`/`F`/`G`), unchanged
-  from prior releases. `"total"` additionally UNIONs in the
-  intergovernmental leg -- payments to local governments (`M` codes) and
-  to the state government (`L` codes, excluding the `L--` family-total
-  rollup) -- so results gain rows with `spend_subtype ==
-  "intergovernmental"`. Requires the active corpus's `summary_categories`
-  to carry M/L rows (added by cog_pipeline PR #59); aborts with class
-  `uscogdata_ig_categories_unsupported` on an older corpus rather than
-  silently under-reporting. Mutually exclusive with `recipe` (a recipe
-  already defines its own component codes). **Do not sum `"total"`
-  results across levels of government** (e.g. state + county + city):
-  a state's `M12` payment to a school district is the same dollar the
-  district reports as its own direct `E12`, so summing both double-counts
-  it. This matters in particular with [cog_geographic_rollup()], which
-  sums across exactly that kind of multi-layer government set.
-
-  In the legacy wide era (<= FY2011), some functions are published ONLY
-  as an aggregate-flagged family total (e.g. Corrections' `E04`/`E05`
-  split), which the Direct leg excludes by construction but the IG leg
-  deliberately keeps (see `inst/sql/24-ig_long.sql`). For a `"total"`
-  query, any (year, category) where this leaves intergovernmental rows
-  with NO Direct counterpart is flagged: the affected rows' `notes`
-  name the harmonization recipe that recovers the missing Direct
-  component (when one exists), and
-  `provenance$expenditure_concept_direct_suppressed` is `TRUE` -- the
-  figure in those rows is the intergovernmental leg alone, not Direct +
-  IG.}
-
-\item{complete}{If `TRUE`, fill the requested grid so that a cell the
-  corpus does not carry still appears, labelled with **why** it is
-  missing, and add a `value_source` column to every row:
-
-  * `"reported"` — the corpus carries this cell.
-  * `"census_zero"` — dense-source year (`<= FY2011`), cell absent:
-    Census published `$0`. `amt_nominal` is `0`.
-  * `"not_reported"` — sparse-source year (`>= FY2012`), cell absent: the
-    government did not report, and the value is unknown. `amt_nominal` is
-    `NA`, **not** `0` — writing a zero there would invent data.
-
-  The grid comes from the corpus's `code_set` table, scoped to each
-  government's own type, so a county is never filled with cells only a
-  state can report. Reported rows are passed through untouched.
-
-  Defaults to `FALSE` (the historical behaviour: absent cells simply do
-  not appear). Needs a corpus published from 2026-07-29 onward, which is
-  when `representation`/`code_set` began shipping; aborts with class
-  `uscogdata_representation_unavailable` otherwise. Not available with
-  `recipe` or with `expenditure_concept = "total"` (class
-  `uscogdata_complete_unsupported`) — neither draws its cells from
-  `code_set`.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,
   `spend_subtype`, `category`, `amt_nominal`, optional `amt_real`,
   optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
-  optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`,
+  optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`.
-  and `value_source` when `complete = TRUE`.
+  Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`.
-  Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`,
-  whose `completion` block reports `applied`, `rows_filled`, and the
-  per-year `absence_means` rule that was applied.
 }
 \description{
 One row per `(year, canonical_govid, spend_subtype, category)`. Amounts are

tests/testthat/helper-fixture.R

@@ -6,33 +6,6 @@ fixture_corpus_path <- function() {
   if (nzchar(p)) paste0(p, "/") else ""
 }
 
-# Path to a file in the SOURCE tree (README.md, man/*.Rd, vignettes/*.Rmd),
-# or "" when it isn't there.
-#
-# Tests that assert on documentation content have to read the sources, and the
-# sources only exist when the suite runs from a checkout. Under R CMD check the
-# suite runs from the INSTALLED package, where man/ and vignettes/ are not
-# shipped and `../../README.md` does not resolve -- so those tests must skip
-# rather than error. CI runs testthat::test_local() from the checkout BEFORE
-# rcmdcheck, so the assertions are still enforced on every push; this only
-# stops them from failing a context that structurally cannot satisfy them.
-source_tree_path <- function(...) {
-  p <- testthat::test_path("..", "..", ...)
-  if (file.exists(p)) p else ""
-}
-
-# Skip unless every named source file is present (see source_tree_path()).
-skip_if_no_source_tree <- function(...) {
-  paths <- vapply(list(...), function(rel) do.call(source_tree_path, as.list(rel)),
-                  character(1))
-  missing <- vapply(paths, function(p) !nzchar(p), logical(1))
-  testthat::skip_if(
-    any(missing),
-    "package source tree not available (running against the installed package)"
-  )
-  invisible(paths)
-}
-
 # Skip a test if no corpus is reachable (bundled fixture or explicit remote URL).
 skip_if_no_corpus <- function() {
   p <- fixture_corpus_path()
@@ -84,73 +57,3 @@ with_doctored_schema_version <- function(version, code) {
   }, add = TRUE)
   force(code)
 }
-
-# Copy the bundled fixture to a temp dir with representation.parquet and
-# code_set.parquet removed (and dropped from the manifest's metadata list),
-# then run `code` against it. Models a corpus published BEFORE sparsification:
-# schema_version is left alone deliberately, because it was never bumped for
-# that change -- the pre-sparsification fixture this package shipped until
-# 2026-07-30 was schema v6 and carried neither table. Presence in the manifest
-# is therefore the only honest signal, and this helper is what proves the
-# package keys off it rather than off the version number.
-with_corpus_missing_representation <- function(code) {
-  src <- fixture_corpus_path()
-  tmp <- withr::local_tempdir(.local_envir = parent.frame())
-  file.copy(list.files(src, full.names = TRUE), tmp, recursive = TRUE)
-
-  dropped <- c("representation.parquet", "code_set.parquet")
-  file.remove(file.path(tmp, "data", dropped))
-
-  manifest_path <- file.path(tmp, "manifest.json")
-  m <- jsonlite::fromJSON(manifest_path, simplifyVector = FALSE)
-  m$files$metadata <- Filter(
-    function(f) !basename(f$path) %in% dropped, m$files$metadata
-  )
-  writeLines(
-    jsonlite::toJSON(m, auto_unbox = TRUE, pretty = TRUE, null = "null"),
-    manifest_path
-  )
-
-  old_url <- Sys.getenv("USCOGDATA_URL", unset = NA)
-  uscogdata:::cog_close()
-  Sys.setenv(USCOGDATA_URL = paste0(tmp, "/"))
-  on.exit({
-    uscogdata:::cog_close()
-    if (is.na(old_url)) Sys.unsetenv("USCOGDATA_URL") else Sys.setenv(USCOGDATA_URL = old_url)
-  }, add = TRUE)
-  force(code)
-}
-
-# Copy the bundled fixture to a temp dir with summary_categories.parquet
-# rewritten to drop every M/L (intergovernmental) row, then run `code`
-# against it with a clean session (mirrors with_fixture_corpus()/
-# with_doctored_schema_version()). Models a real pre-cog_pipeline-PR#59
-# corpus: the 66 M/L category rows shipped with NO schema_version bump (see
-# C2 in the expenditure-concept review), so schema_version is left
-# untouched here -- only the category data itself is rolled back.
-with_corpus_missing_ig_categories <- function(code) {
-  src <- fixture_corpus_path()
-  tmp <- withr::local_tempdir(.local_envir = parent.frame())
-  file.copy(list.files(src, full.names = TRUE), tmp, recursive = TRUE)
-
-  cats_path <- file.path(tmp, "data", "summary_categories.parquet")
-  filtered_path <- file.path(tmp, "data", "summary_categories_filtered.parquet")
-  write_con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
-  DBI::dbExecute(write_con, sprintf(
-    "COPY (SELECT * FROM read_parquet(%s) WHERE LEFT(item_code, 1) NOT IN ('M', 'L'))
-     TO %s (FORMAT PARQUET)",
-    uscogdata:::.sql_lit_chr(cats_path), uscogdata:::.sql_lit_chr(filtered_path)
-  ))
-  file.remove(cats_path)
-  file.rename(filtered_path, cats_path)
-
-  old_url <- Sys.getenv("USCOGDATA_URL", unset = NA)
-  uscogdata:::cog_close()
-  Sys.setenv(USCOGDATA_URL = paste0(tmp, "/"))
-  on.exit({
-    uscogdata:::cog_close()
-    if (is.na(old_url)) Sys.unsetenv("USCOGDATA_URL") else Sys.setenv(USCOGDATA_URL = old_url)
-  }, add = TRUE)
-  force(code)
-}

tests/testthat/helper-walkthrough-raw.R

@@ -1,44 +0,0 @@
-# Helper for the Madison-walkthrough finding tests (uscogdata #11-#16).
-#
-# Those tests all assert something about what a `cog_*` verb includes or
-# excludes. The expected amounts must therefore come from the RAW corpus, never
-# from the verb under test: verifying an absence through the filter that creates
-# it proves nothing. `wt_raw_*()` opens its own DuckDB connection straight onto
-# the corpus's `long` parquet partitions, bypassing uscogdata's SQL views (and
-# therefore its `flow_prefixes` filtering) entirely.
-
-wt_corpus_glob <- function() {
-  url <- Sys.getenv("USCOGDATA_URL")
-  if (!nzchar(url)) testthat::skip("USCOGDATA_URL is not set")
-  paste0(sub("/$", "", url), "/data/long/**/*.parquet")
-}
-
-wt_raw_query <- function(sql) {
-  con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
-  DBI::dbGetQuery(con, sql)
-}
-
-# Sum of `amt` (in $1,000s, as the corpus stores it) for one government-year,
-# restricted either to an explicit set of item codes or to a set of first-letter
-# prefixes. Aggregate rows are excluded, matching every published verb.
-wt_raw_amt <- function(govid, year, codes = NULL, prefixes = NULL) {
-  stopifnot(xor(is.null(codes), is.null(prefixes)))
-  filter_sql <- if (!is.null(codes)) {
-    paste0("item_code IN (", paste0("'", codes, "'", collapse = ", "), ")")
-  } else {
-    paste0("LEFT(item_code, 1) IN (", paste0("'", prefixes, "'", collapse = ", "), ")")
-  }
-  out <- wt_raw_query(paste0(
-    "SELECT COALESCE(SUM(amt), 0) AS amt FROM read_parquet('", wt_corpus_glob(), "') ",
-    "WHERE canonical_govid = '", govid, "' AND year = ", year,
-    " AND NOT is_aggregate AND ", filter_sql
-  ))
-  out$amt[[1]]
-}
-
-# The item codes a verb reports having summed, flattened out of the
-# comma-separated `codes_included` column.
-wt_codes_included <- function(df) {
-  sort(unique(trimws(unlist(strsplit(stats::na.omit(df$codes_included), ",")))))
-}

tests/testthat/test-amount-units-documented.R

@@ -1,52 +0,0 @@
-# Madison walkthrough audit -- finding F-004. Tracked as uscogdata#15.
-# See docs/walkthroughs/FINDINGS.md in cog_explorer.
-#
-# The raw Census files report thousands of dollars; this package multiplies by
-# 1000 and returns full US dollars. That is the friendlier choice and is not
-# wrong -- but cog_explorer's CLAUDE.md states "All raw `amt` values are in
-# $1,000s", so a reader who applies that rule to amt_nominal overstates every
-# figure by 1000x, and gets a plausible-looking number rather than an obvious
-# error. The audit rates this the highest-consequence definitional gap it found.
-#
-# Deliberately NOT asserted here: man/cog_spending.Rd and man/cog_revenue.Rd,
-# which ALREADY carry the statement in their @return sections (verified
-# 2026-07-29), as does cog-api's data-dictionary.md (since 2b71b41). The gap is
-# in the surfaces a reader meets first and in cog_explorer's own conventions
-# doc -- see uscogdata#15 for the full surface-by-surface table and for the two
-# secondary tasks (cog_explorer/CLAUDE.md, which has no git remote, and
-# cog-api's llms.txt, which is silent on units).
-
-test_that("returned amounts are documented as full US dollars where readers meet the package", {
-
-  # README and vignettes ship only in the source tree, not in the installed
-  # package, so these assertions cannot run under R CMD check -- CI's earlier
-  # testthat::test_local() step is what enforces them. See
-  # skip_if_no_source_tree() in helper-fixture.R.
-  docs <- skip_if_no_source_tree(
-    "README.md",
-    c("vignettes", "total-spending.Rmd"),
-    c("vignettes", "population-denominators.Rmd")
-  )
-
-  says_units <- function(path) {
-    txt <- paste(readLines(path, warn = FALSE), collapse = " ")
-    grepl("full US dollars|full U\\.S\\. dollars", txt, ignore.case = TRUE) &&
-      grepl("\\$1,000s|thousands of dollars", txt, ignore.case = TRUE)
-  }
-
-  for (path in docs) expect_true(says_units(path))
-
-  # Pin the documented claim to the actual behaviour, so the two cannot drift.
-  # The expected raw amount is read straight from the corpus's parquet
-  # partitions -- never through cog_spending(), which is the thing being
-  # described. Madison FY2020: E/F/G = 623,347 ($1,000s) -> $623,347,000.
-  raw_thousands <- wt_raw_amt("552025209777", 2020L, prefixes = c("E", "F", "G"))
-  expect_equal(raw_thousands, 623347)
-
-  returned <- cog_spending(govid = "552025209777", years = 2020L)
-  expect_equal(sum(returned$amt_nominal), raw_thousands * 1000)
-
-  units <- attr(returned, "provenance")$transformations$units_conversion
-  expect_true(units$applied)
-  expect_equal(units$multiplier, 1000)
-})

tests/testthat/test-categories.R

@@ -15,22 +15,7 @@ test_that("cog_categories(type = 'spending') returns only expenditure rows", {
   skip_if_no_corpus()
   r <- cog_categories(type = "spending")
   expect_true(all(r$category_type == "expenditure"))
-  # "assistance" (the J-prefix aid/benefit codes) joined the vocabulary with
+  expect_true(all(r$subtype %in% c("operations", "capital")))
-  # the crosswalk completion in cog_pipeline#60/#65 -- every flow code
-  # carrying dollars now maps to a category.
-  expect_true(all(r$subtype %in%
-                  c("operations", "capital", "intergovernmental", "assistance")))
-})
-
-test_that("cog_categories surfaces the intergovernmental spending subtype", {
-  skip_if_no_corpus()
-  r <- cog_categories(type = "spending")
-  expect_true("intergovernmental" %in% r$subtype)
-  # IG rows reuse the existing functional categories -- they add a subtype,
-  # not new category values.
-  ig_cats     <- sort(unique(r$category[r$subtype == "intergovernmental"]))
-  direct_cats <- sort(unique(r$category[r$subtype != "intergovernmental"]))
-  expect_true(all(ig_cats %in% c(direct_cats, "Other Education")))
 })
 
 test_that("cog_categories(type = 'revenue') returns only revenue rows", {

tests/testthat/test-complete.R

@@ -1,188 +0,0 @@
-# tests/testthat/test-complete.R
-#
-# uscogdata#18. The published corpus no longer stores the wide era's explicit
-# zeros (cog_pipeline#64, series break SB194), so absence means two different
-# things:
-#
-#   <= FY2011 (dense_source) : cell absent => Census published $0
-#   >= FY2012 (sparse_source): cell absent => not reported, unknown
-#
-# `complete = TRUE` fills the requested grid from `code_set` and stamps every
-# row's `value_source` so the two are distinguishable. Expected row sets here
-# are built from the corpus parquet directly, never from the verb under test --
-# verifying what a filter does through that same filter proves nothing.
-
-# The (subtype, category) cells that SHOULD exist for one government-year:
-# every code in force for that government's type, mapped through
-# summary_categories, matching the verb's flow prefixes and excluding
-# aggregate-flagged codes (which spending_long/revenue_long drop).
-raw_expected_cells <- function(govid, year, prefixes, subtype_col) {
-  fx <- sub("/$", "", Sys.getenv("USCOGDATA_URL"))
-  q <- function(f) sprintf("read_parquet('%s/data/%s')", fx, f)
-  wt_raw_query(sprintf(
-    "SELECT DISTINCT c.%s AS subtype, c.category
-     FROM %s cs
-     JOIN %s x ON x.govs_type = cs.type
-     JOIN %s c ON c.item_code = cs.item_code
-     WHERE x.canonical_govid = '%s'
-       AND cs.year = %d
-       AND NOT cs.is_aggregate
-       AND LEFT(cs.item_code, 1) IN (%s)
-       AND c.category IS NOT NULL
-       AND c.%s IS NOT NULL",
-    subtype_col, q("code_set.parquet"), q("canonical_fips_xwalk.parquet"),
-    q("summary_categories.parquet"), govid, year,
-    paste0("'", prefixes, "'", collapse = ","), subtype_col
-  ))
-}
-
-test_that("complete = FALSE is the default and changes nothing", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    plain <- cog_spending("121011212191", 2011L)
-    explicit <- cog_spending("121011212191", 2011L, complete = FALSE)
-    expect_equal(nrow(plain), nrow(explicit))
-    expect_false("value_source" %in% names(plain))
-  })
-})
-
-test_that("complete = TRUE round-trips a dense-source year to the pre-sparsification cells", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    # FY2011 is dense_source: before sparsification this government carried a
-    # row for every code in force, most of them $0. complete = TRUE must
-    # reproduce that cell set exactly.
-    r <- cog_spending("121011212191", 2011L, complete = TRUE)
-    expected <- raw_expected_cells("121011212191", 2011L,
-                                   c("E", "F", "G"), "spend_subtype")
-
-    key <- function(sub, cat) paste(sub, cat, sep = "|")
-    expect_setequal(key(r$spend_subtype, r$category),
-                    key(expected$subtype, expected$category))
-    expect_gt(nrow(expected), 0L)
-
-    # Every filled cell in a dense-source year is a Census-published $0 --
-    # never "unknown", which is what the modern era's absences mean.
-    expect_setequal(unique(r$value_source), c("reported", "census_zero"))
-    expect_true(all(r$amt_nominal[r$value_source == "census_zero"] == 0))
-    expect_true(all(r$amt_nominal[r$value_source == "reported"] != 0))
-  })
-})
-
-test_that("complete = TRUE preserves the reported rows and their amounts exactly", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    plain <- cog_spending("121011212191", 2011L)
-    full  <- cog_spending("121011212191", 2011L, complete = TRUE)
-
-    # Filling adds rows; it must never alter or drop one.
-    expect_gt(nrow(full), nrow(plain))
-    reported <- full[full$value_source == "reported", ]
-    expect_equal(nrow(reported), nrow(plain))
-    expect_equal(sum(reported$amt_nominal), sum(plain$amt_nominal))
-    # ... and the total is unchanged, because every added cell is $0.
-    expect_equal(sum(full$amt_nominal, na.rm = TRUE), sum(plain$amt_nominal))
-  })
-})
-
-test_that("a sparse-source year's absences are unknown, not zero", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    # FY2019 is sparse_source: an absent cell means the government did not
-    # report, which is NOT a zero. Filling those with 0 would invent data --
-    # the exact error the representation contract exists to prevent.
-    r <- cog_spending("121011212191", 2019L, complete = TRUE)
-    filled <- r[r$value_source != "reported", ]
-    expect_gt(nrow(filled), 0L)
-    expect_true(all(filled$value_source == "not_reported"))
-    expect_true(all(is.na(filled$amt_nominal)))
-    expect_false(any(r$value_source == "census_zero"))
-  })
-})
-
-test_that("the fill is scoped to each government's own type", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    # Filling against the union of all types would invent cells for codes a
-    # county can never report. Every filled category must be one that
-    # code_set puts in force for type 1 (county) specifically.
-    r <- cog_spending("121011212191", 2011L, complete = TRUE)
-    county_cells <- raw_expected_cells("121011212191", 2011L,
-                                       c("E", "F", "G"), "spend_subtype")
-    expect_true(all(r$category %in% county_cells$category))
-  })
-})
-
-test_that("complete = TRUE respects the category filter", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    r <- cog_spending("121011212191", 2011L, category = "Police",
-                      complete = TRUE)
-    expect_true(all(r$category == "Police"))
-    expect_true("value_source" %in% names(r))
-  })
-})
-
-test_that("cog_revenue() completes on its own flow", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    r <- cog_revenue("121011212191", 2011L, complete = TRUE)
-    expected <- raw_expected_cells("121011212191", 2011L,
-                                   c("T", "A", "U", "B", "C", "D"),
-                                   "revenue_subtype")
-    key <- function(sub, cat) paste(sub, cat, sep = "|")
-    expect_setequal(key(r$revenue_subtype, r$category),
-                    key(expected$subtype, expected$category))
-    expect_setequal(unique(r$value_source), c("reported", "census_zero"))
-  })
-})
-
-test_that("provenance records the completion and its absence rule", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    prov <- attr(cog_spending("121011212191", 2011L, complete = TRUE),
-                 "provenance")
-    expect_true(prov$completion$applied)
-    expect_equal(prov$completion$absence_means$`2011`, "census_zero")
-    expect_gt(prov$completion$rows_filled, 0L)
-
-    off <- attr(cog_spending("121011212191", 2011L), "provenance")
-    expect_false(off$completion$applied)
-    expect_equal(off$completion$rows_filled, 0L)
-  })
-})
-
-test_that("complete = TRUE is refused where the fill would be guesswork", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    # A recipe defines its own component codes and does not go through
-    # summary_categories at all, so there is no grid to fill from.
-    expect_error(
-      cog_spending("121011212191", 2011L, recipe = "corrections_combined",
-                   complete = TRUE),
-      class = "uscogdata_complete_unsupported"
-    )
-    # The intergovernmental leg keeps aggregate rows by design
-    # (inst/sql/24-ig_long.sql), so its grid is not code_set's grid.
-    expect_error(
-      cog_spending("121011212191", 2011L, expenditure_concept = "total",
-                   complete = TRUE),
-      class = "uscogdata_complete_unsupported"
-    )
-  })
-})
-
-test_that("complete = TRUE aborts on a corpus with no representation contract", {
-  skip_if_no_corpus()
-  # A corpus published before sparsification carries neither table, so there
-  # is nothing to fill from and no rule saying what an absence means. That
-  # must abort rather than guess.
-  with_corpus_missing_representation({
-    expect_error(
-      cog_spending("121011212191", 2011L, complete = TRUE),
-      class = "uscogdata_representation_unavailable"
-    )
-    # ... while an ordinary query on the same corpus still works.
-    expect_gt(nrow(cog_spending("121011212191", 2011L)), 0L)
-  })
-})

tests/testthat/test-config.R

@@ -26,41 +26,3 @@ test_that(".resolve_cache_dir falls back to R_user_dir", {
     })
   })
 })
-
-# ---------------------------------------------------------------------------
-# Trailing-slash normalization (uscogdata #3 follow-up).
-#
-# EVERY consumer builds paths by concatenation: paste0(url, "manifest.json")
-# (manifest.R), paste0(url, e$path) (mirror.R), and the parquet glob in
-# views.R. mirror.R:104 even comments 'url ends in "/"' -- an assumption the
-# package documents and relies on but never enforced.
-#
-# A URL missing its trailing slash therefore fails SILENTLY and confusingly:
-#   HTTPS -> ".../downloadmanifest.json" -> the host answers with an HTML 404
-#            page -> the jsonlite lexical error that issue #3 reported;
-#   local -> ".../corpusdata/long/**/*.parquet" -> DuckDB "No files found".
-# Neither message points at the real cause. Normalize once, at resolution.
-# ---------------------------------------------------------------------------
-
-test_that(".resolve_url appends a missing trailing slash", {
-  withr::local_envvar(USCOGDATA_URL = "https://example.org/s/TOKEN/download")
-  expect_equal(.resolve_url(), "https://example.org/s/TOKEN/download/")
-})
-
-test_that(".resolve_url leaves an existing trailing slash alone", {
-  withr::local_envvar(USCOGDATA_URL = "https://example.org/s/TOKEN/download/")
-  expect_equal(.resolve_url(), "https://example.org/s/TOKEN/download/")
-})
-
-test_that(".resolve_url normalizes a local path without a trailing slash", {
-  withr::local_envvar(USCOGDATA_URL = "/tmp/corpus")
-  expect_equal(.resolve_url(), "/tmp/corpus/")
-})
-
-test_that(".resolve_url does not invent a slash for an empty setting", {
-  # An unset/empty URL must stay empty so the "not configured" guard in
-  # manifest.R still fires, rather than degrading into a bare "/" root.
-  withr::local_envvar(USCOGDATA_URL = "")
-  withr::local_options(uscogdata.url = "")
-  expect_equal(.resolve_url(), "")
-})

tests/testthat/test-corpus-breaks.R

@@ -1,94 +0,0 @@
-# tests/testthat/test-corpus-breaks.R
-#
-# uscogdata#19. Four catalogued series breaks carry fin_code = "ALL" -- they
-# are caveats about the corpus itself rather than about one item code:
-#
-#   SB085  1977  dollar precision across the 1976/1977 boundary
-#   SB087  2002  imputation exclusion FY2002-2006
-#   SB194  2012  dense -> sparse representation change
-#   SB086  2017  government ID scheme change
-#
-# .build_series_break_refs() matches `fin_code IN (<codes in the result>)`,
-# and no row's item_code is ever the literal "ALL", so none of them could
-# ever reach a user. They now travel in their own provenance field,
-# `corpus_break_refs`, which keeps them distinguishable from the
-# code-specific `series_break_refs` (an ALL caveat qualifies the whole
-# result, not one series).
-
-test_that("corpus_break_refs surfaces an ALL-scoped break the year range spans", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    # SB194 sits at FY2012 -- the dense/sparse boundary. A query spanning
-    # 2011 -> 2012 straddles it, and this is the case cog_pipeline#64's
-    # DoD 4 intended to reach users.
-    r <- cog_spending("121011212191", 2011:2012, "Police")
-    prov <- attr(r, "provenance")
-    expect_true("SB194" %in% prov$corpus_break_refs)
-  })
-})
-
-test_that("corpus_break_refs stays empty when no ALL break falls in the range", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    # 2019-2020 spans no catalogued corpus-wide break.
-    r <- cog_spending("121011212191", 2019:2020, "Police")
-    expect_equal(attr(r, "provenance")$corpus_break_refs, character(0))
-  })
-})
-
-test_that("corpus_break_refs and series_break_refs stay disjoint", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    r <- cog_spending("121011212191", 2011:2012, "Police")
-    prov <- attr(r, "provenance")
-    expect_type(prov$series_break_refs, "character")
-    expect_type(prov$corpus_break_refs, "character")
-    # An ALL caveat must never masquerade as a break in a specific series.
-    expect_length(intersect(prov$series_break_refs, prov$corpus_break_refs), 0L)
-    expect_false("SB194" %in% prov$series_break_refs)
-  })
-})
-
-test_that(".build_corpus_break_refs matches on the break_year window alone", {
-  skip_if_no_corpus()
-  con <- cog_open()
-  on.exit(cog_close())
-
-  # SB085's boundary is 1976/1977, outside the fixture's partitions -- the
-  # series_breaks table is a full cross-vintage registry, so the matching
-  # logic is testable there even though no long partition covers it.
-  expect_true("SB085" %in% uscogdata:::.build_corpus_break_refs(
-    con, years = 1975:1980, schema_version = 6L
-  ))
-  # ... and does not fire for a range that misses it, unlike a filter keyed
-  # on the era rather than the boundary.
-  expect_false("SB085" %in% uscogdata:::.build_corpus_break_refs(
-    con, years = 1978:1980, schema_version = 6L
-  ))
-
-  # Unlike code-specific refs, these do not depend on which codes a result
-  # happens to contain -- that dependency is the whole defect.
-  expect_setequal(
-    uscogdata:::.build_corpus_break_refs(con, years = 2001:2003, schema_version = 6L),
-    "SB087"
-  )
-
-  # Gated on schema_version >= 5: series_breaks_pq is not registered below it.
-  expect_equal(
-    uscogdata:::.build_corpus_break_refs(con, years = 2011:2012, schema_version = 4L),
-    character(0)
-  )
-})
-
-test_that("cog_explain() prints corpus-wide caveats under their own heading", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    r <- cog_spending("121011212191", 2011:2012, "Police")
-    out <- paste(c(
-      capture.output(cog_explain(r)),
-      capture.output(cog_explain(r), type = "message")
-    ), collapse = "\n")
-    expect_match(out, "Corpus-wide caveats", fixed = TRUE)
-    expect_match(out, "SB194", fixed = TRUE)
-  })
-})

tests/testthat/test-coverage-disclosure.R

@@ -1,103 +0,0 @@
-# Madison walkthrough audit -- findings F-020 and F-023. Tracked as uscogdata#13.
-# See docs/walkthroughs/FINDINGS.md in cog_explorer.
-#
-# The owner's settled design (2026-07-28): a `coverage` argument on
-# cog_geographic_rollup(), cog_find_peers()/cog_peer_compare() and their
-# cog-api equivalents --
-#   "all"        every unit that reported that year (today's behaviour, DEFAULT)
-#   "census"     census years only (years ending 2 or 7)
-#   "consistent" only units reporting in every requested year (balanced panel)
-# -- PLUS always-on coverage metadata on every result regardless of mode:
-# n_units_reporting, n_units_expected, is_census_year.
-#
-# Motivating principle: using these verbs correctly must not require the user to
-# know that the Census of Governments is a complete census only in years ending
-# in 2 and 7.
-#
-# The helper below accepts that metadata either as columns on the returned
-# tibble or as a per-year table in provenance$coverage -- the design fixes the
-# three field names and that they reach the caller, not the container.
-
-wt_coverage <- function(x) {
-  prov <- attr(x, "provenance")
-  cov <- prov$coverage
-  if (is.null(cov)) {
-    needed <- c("year", "n_units_reporting", "n_units_expected", "is_census_year")
-    expect_true(all(needed %in% names(x)))
-    cov <- unique(x[, needed])
-  }
-  cov[order(cov$year), ]
-}
-
-test_that("multi-government aggregates disclose reporting coverage on every result", {
-
-  # -- F-020: geographic rollups -------------------------------------------
-  # Wisconsin's city/village universe is 608 governments. On the bundled
-  # fixture, FY2012 (a census year) has 597 of them reporting while FY2019 and
-  # FY2020 (sample years) have 112 and 114 -- an 18%-98% swing that today's
-  # return value says nothing about. Counts cross-checked against the raw
-  # corpus, not through cog_geographic_rollup(), which is under test.
-  wi <- cog_gov_search(name = NULL, state = "WI", type = "city")
-  expect_equal(nrow(wi), 608L)
-
-  roll <- cog_geographic_rollup(govids = list(city = wi$canonical_govid),
-                                category = NULL, years = c(2011L, 2012L, 2019L, 2020L))
-  cov <- wt_coverage(roll)
-
-  expect_equal(cov$n_units_expected, rep(608L, 4L))
-  expect_equal(cov$n_units_reporting, c(152L, 597L, 112L, 114L))
-  expect_equal(cov$is_census_year, c(FALSE, TRUE, FALSE, FALSE))
-
-  # Cross-check against the raw partitions, scoped to the SAME universe the
-  # rollup was given -- the 608 govids above. Scoping instead on the long
-  # table's own `type`/`fips_state` asks a different question and answers 595:
-  # VERNON VILLAGE and WAUKESHA VILLAGE carry type = 3 there (their as-of-year
-  # identity, when they were townships) while the xwalk lists them as
-  # govs_type = 2 (their present identity, as villages). Schema v6 made the
-  # long table's geography present-harmonized and moved as-of-year to the
-  # *_asof columns, but `type` still reads as-of-year -- see .validate_schema()
-  # in R/manifest.R. n_units_reporting counts against the requested universe,
-  # so 597 is the number that answers "how many of the governments I asked
-  # about reported".
-  raw_2012 <- wt_raw_query(paste0(
-    "SELECT COUNT(DISTINCT canonical_govid) n FROM read_parquet('", wt_corpus_glob(), "') ",
-    "WHERE year = 2012 AND LEFT(item_code, 1) IN ('E','F','G') AND NOT is_aggregate ",
-    "AND canonical_govid IN (",
-    paste0("'", wi$canonical_govid, "'", collapse = ","), ")"))
-  expect_equal(cov$n_units_reporting[cov$year == 2012], as.integer(raw_2012$n[[1]]))
-
-  # -- F-023: peer cohorts --------------------------------------------------
-  # CHILTON CITY, WI (ACS population 4,017): a 15-peer cohort fixed at FY2012
-  # reports 15 of 15 in FY2012 and only 3 of 15 in FY2019 and FY2020. Nothing
-  # in cog_peer_compare()'s return distinguishes those years today.
-  chilton <- "552015177095"
-  peers <- cog_find_peers(chilton, year = 2012L, max_peers = 15L)
-  expect_equal(nrow(peers), 15L)
-
-  cmp <- cog_peer_compare(target_govid = chilton, peers = peers, category = NULL,
-                          years = c(2012L, 2019L, 2020L), per_capita = TRUE)
-  cov_peers <- wt_coverage(cmp)
-  expect_equal(cov_peers$n_units_expected, rep(15L, 3L))
-  expect_equal(cov_peers$n_units_reporting, c(15L, 3L, 3L))
-  expect_equal(cov_peers$is_census_year, c(TRUE, FALSE, FALSE))
-
-  # -- the three coverage modes --------------------------------------------
-  expect_equal(attr(cog_peer_compare(target_govid = chilton, peers = peers,
-                                     category = NULL, years = c(2012L, 2019L, 2020L),
-                                     per_capita = TRUE),
-                    "provenance")$coverage_mode, "all")     # unchanged default
-
-  consistent <- cog_peer_compare(target_govid = chilton, peers = peers,
-                                 category = NULL, years = c(2012L, 2019L, 2020L),
-                                 per_capita = TRUE, coverage = "consistent")
-  n_by_year <- tapply(consistent$canonical_govid[consistent$role == "peer"],
-                      consistent$year[consistent$role == "peer"],
-                      function(g) length(unique(g)))
-  expect_equal(unname(as.integer(n_by_year)), c(3L, 3L, 3L))  # balanced panel
-
-  census_only <- cog_geographic_rollup(govids = list(city = wi$canonical_govid),
-                                       category = NULL,
-                                       years = c(2011L, 2012L, 2019L, 2020L),
-                                       coverage = "census")
-  expect_equal(sort(unique(census_only$year)), 2012)
-})

tests/testthat/test-expenditure-concept.R

@@ -1,544 +0,0 @@
-test_that("the corpus contains no K-prefix rows, so the Direct leg omits K", {
-  con <- .ensure_session()
-  n <- DBI::dbGetQuery(con,
-    "SELECT COUNT(*) AS n FROM long WHERE LEFT(item_code, 1) = 'K'")$n
-  expect_equal(n, 0)
-
-  sql_files <- c("20-spending_long.sql", "22-spending_long_harmonized.sql")
-  for (f in sql_files) {
-    txt <- paste(readLines(system.file("sql", f, package = "uscogdata")),
-                 collapse = " ")
-    expect_false(grepl("'K'", txt, fixed = TRUE),
-                 label = paste(f, "must not reference the inert K prefix"))
-  }
-})
-
-test_that("expenditure_concept defaults to direct and preserves today's numbers", {
-  gov <- "010000226085"                      # Alabama state government
-  base <- cog_spending(gov, years = 2019, category = "Police")
-  expl <- cog_spending(gov, years = 2019, category = "Police",
-                       expenditure_concept = "direct")
-  expect_equal(base$amt_nominal, expl$amt_nominal)
-  expect_false("intergovernmental" %in% base$spend_subtype)
-})
-
-test_that("expenditure_concept = 'total' adds an intergovernmental subtype", {
-  gov <- "010000226085"
-  d <- cog_spending(gov, years = 2019, category = "Police",
-                    expenditure_concept = "direct")
-  t <- cog_spending(gov, years = 2019, category = "Police",
-                    expenditure_concept = "total")
-  expect_true("intergovernmental" %in% t$spend_subtype)
-  # Direct rows are untouched; Total only ever ADDS. Use %in% rather than
-  # != : a category = NULL result can contain a NULL-subtype group (codes
-  # with no summary_categories row, e.g. E16/E21/E85/F16/F85/G16/G21/G85),
-  # and `NA != "intergovernmental"` is NA, not TRUE, which would silently
-  # smuggle an all-NA phantom row into dt.
-  dt <- t[!(t$spend_subtype %in% "intergovernmental"), ]
-  expect_equal(sort(dt$amt_nominal), sort(d$amt_nominal))
-  expect_gt(sum(t$amt_nominal), sum(d$amt_nominal))
-})
-
-test_that("legacy-era Total does not collapse to Direct (the is_aggregate trap)", {
-  # In the wide era the IG dollars live almost entirely on aggregate-flagged
-  # rows. A Total leg that inherited the Direct leg's NOT is_aggregate filter
-  # would silently return Total == Direct here.
-  gov <- "010000226085"
-  d <- cog_spending(gov, years = 2011, category = "Education K-12",
-                    expenditure_concept = "direct")
-  t <- cog_spending(gov, years = 2011, category = "Education K-12",
-                    expenditure_concept = "total")
-  expect_true("intergovernmental" %in% t$spend_subtype)
-  ig <- sum(t$amt_nominal[t$spend_subtype == "intergovernmental"])
-  expect_gt(ig, 0)
-  expect_gt(sum(t$amt_nominal), sum(d$amt_nominal))
-})
-
-test_that("the IG leg never includes the L-- family total", {
-  con <- .ensure_session()
-  codes <- DBI::dbGetQuery(con,
-    "SELECT DISTINCT item_code FROM ig_long")$item_code
-  expect_false(any(grepl("--$", codes)))
-  expect_true(all(substr(codes, 1, 1) %in% c("M", "L")))
-})
-
-test_that("expenditure_concept rejects unknown values", {
-  expect_error(
-    cog_spending("010000226085", years = 2019, expenditure_concept = "gross"),
-    class = "rlang_error"
-  )
-})
-
-test_that("total composes with basis = 'raw' and basis = 'harmonized'", {
-  gov <- "010000226085"
-  h <- cog_spending(gov, years = 2011, category = "Education K-12",
-                    expenditure_concept = "total", basis = "harmonized")
-  r <- cog_spending(gov, years = 2011, category = "Education K-12",
-                    expenditure_concept = "total", basis = "raw")
-  ig_h <- sum(h$amt_nominal[h$spend_subtype == "intergovernmental"])
-  ig_r <- sum(r$amt_nominal[r$spend_subtype == "intergovernmental"])
-  # The only IG harmonization rule is M38 -> M36 (year-disjoint), so the IG
-  # total must agree between bases even though the code labels may differ.
-  expect_equal(ig_h, ig_r)
-})
-
-test_that("recipe = and expenditure_concept = 'total' together aborts", {
-  expect_error(
-    cog_spending("121011212191", 2020L, recipe = "corrections_combined",
-                expenditure_concept = "total"),
-    class = "uscogdata_recipe_concept_conflict"
-  )
-})
-
-test_that("aggregate-sourced IG dollars are flagged aggregate_fallback = TRUE (bool_or, not bool_and)", {
-  # Regression test: .build_verb_sql() originally used bool_and(is_aggregate)
-  # for aggregate_fallback, which is correct for the Direct leg (a group can
-  # never mix aggregate and non-aggregate rows there -- spending_long filters
-  # NOT is_aggregate) but wrong for the IG leg. The wide era is dense -- every
-  # government has a $0 row for every code in a family -- so a $0 leaf sits in
-  # the same (year, gov, subtype, category) group as the real aggregate row
-  # and flips bool_and() to FALSE. Measured: AL state 2011 had $5,740,775,000
-  # of aggregate-sourced IG dollars (Corrections $31,358,000 + Education K-12
-  # $5,152,385,000 + General Government $557,032,000) reporting
-  # aggregate_fallback = FALSE under bool_and(), with the only TRUE row being
-  # Transit Utilities at $0. bool_or() reports all of them correctly.
-  gov <- "010000226085"
-  t <- cog_spending(gov, years = 2011, category = "Education K-12",
-                    expenditure_concept = "total")
-  ig <- t[t$spend_subtype == "intergovernmental", ]
-  expect_equal(nrow(ig), 1L)
-  expect_true(ig$aggregate_fallback)
-  expect_true(nzchar(ig$notes))
-  expect_match(ig$notes, "Aggregate fallback applied", fixed = TRUE)
-})
-
-test_that("legacy aggregate IG codes are year-disjoint from their modern leaf components", {
-  # The safety of ig_long's deliberate omission of `NOT is_aggregate` (see
-  # inst/sql/24-ig_long.sql) rests entirely on each legacy code's AGGREGATE
-  # instance being year-disjoint from the modern leaf codes it rolls up --
-  # if a future corpus rebuild ever back-filled a leaf into a year where the
-  # code is still flagged aggregate, `total` would silently double-count and
-  # this suite would still pass. This test fails loudly if that ever
-  # happens.
-  #
-  # Note the invariant is scoped to the AGGREGATE flag, not bare code
-  # presence: M89/L89 do NOT disappear after the wide era the way M47/L47
-  # do -- they continue past 2011 as their OWN independent leaf line item
-  # (is_aggregate = FALSE) alongside M91-93/L91-93, which is fine because a
-  # non-aggregate M89/L89 no longer represents a rollup of those codes.
-  # (Verified in the fixture: M89/L89 are is_aggregate = TRUE only in 2011,
-  # when M91-93/L91-93 don't exist yet; from 2012 on M89/L89 are
-  # is_aggregate = FALSE leaves coexisting with M91-93/L91-93.)
-  #
-  # Pairs are the M/L-prefixed components (this package's ig_long only
-  # covers M/L; other prefixes in the same rollup, e.g. N/O/P/Q/R, fall
-  # outside its domain and are irrelevant here) enumerated in
-  # cog_pipeline's data/wide_to_long_xwalk.csv `full_desc` column (read
-  # once at authoring time, not at test time -- this test stays offline):
-  #   M47 "To local governments, total (includes N47, O47, P47, R47, and M94)"
-  #   M89 "To local governments, total (incl N89, O89, P89, R89, M91, M92, and M93)"
-  #   L47 "To state government (includes L94)"
-  #   L89 "To state government (includes L91, L92, and L93)"
-  con <- .ensure_session()
-  pairs <- list(
-    list(aggregate = "M47", components = "M94"),
-    list(aggregate = "M89", components = c("M91", "M92", "M93")),
-    list(aggregate = "L47", components = "L94"),
-    list(aggregate = "L89", components = c("L91", "L92", "L93"))
-  )
-  agg_years_by_code <- DBI::dbGetQuery(con,
-    "SELECT DISTINCT year, item_code FROM ig_long WHERE is_aggregate")
-  codes_by_year <- DBI::dbGetQuery(con, "SELECT DISTINCT year, item_code FROM ig_long")
-
-  for (p in pairs) {
-    agg_years <- agg_years_by_code$year[agg_years_by_code$item_code == p$aggregate]
-    for (yr in agg_years) {
-      codes_yr <- codes_by_year$item_code[codes_by_year$year == yr]
-      has_component <- any(p$components %in% codes_yr)
-      expect_false(
-        has_component,
-        label = sprintf(
-          "year %s has aggregate-flagged %s co-occurring with a modern component (%s)",
-          yr, p$aggregate, paste(p$components, collapse = ",")
-        )
-      )
-    }
-  }
-})
-
-test_that(".verb_spendrev rejects expenditure_concept = 'total' for a non-spending view_base", {
-  # cog_revenue() never exposes expenditure_concept and always resolves it
-  # to the "direct" default, so there is no revenue codepath that reaches
-  # this today -- but .verb_spendrev() is shared, and nothing else stops a
-  # future caller from passing expenditure_concept = "total" alongside
-  # view_base = "revenue_annotated", which would UNION expenditure M/L rows
-  # into a revenue result. Exercise the internal helper directly.
-  expect_error(
-    uscogdata:::.verb_spendrev(
-      verb = "cog_revenue_test", view_base = "revenue_annotated",
-      subtype_col = "revenue_subtype",
-      flow_prefixes = c("T", "A", "U", "B", "C", "D"),
-      call = quote(cog_revenue_test()),
-      govid = "010000226085", years = 2019L, category = NULL,
-      per_capita = FALSE, adjust_to_year = NULL, basis = "raw",
-      recipe = NULL, expenditure_concept = "total"
-    ),
-    class = "uscogdata_expenditure_concept_unsupported"
-  )
-})
-
-test_that("cog_geographic_rollup refuses expenditure_concept = 'total'", {
-  expect_error(
-    cog_geographic_rollup(
-      govids = list(state = "010000226085"),
-      category = "Police", years = 2019,
-      expenditure_concept = "total"
-    ),
-    class = "uscogdata_concept_not_aggregatable"
-  )
-})
-
-test_that("cog_peer_compare refuses expenditure_concept = 'total'", {
-  expect_error(
-    cog_peer_compare(
-      target_govid = "010000226085", peers = "010000226085",
-      category = "Police", years = 2019,
-      expenditure_concept = "total"
-    ),
-    class = "uscogdata_concept_not_aggregatable"
-  )
-})
-
-test_that("the refusal message names the fix and the reason", {
-  err <- tryCatch(
-    cog_geographic_rollup(govids = list(state = "010000226085"),
-                          category = "Police", years = 2019,
-                          expenditure_concept = "total"),
-    condition = function(e) e
-  )
-  msg <- paste(conditionMessage(err), collapse = " ")
-  expect_match(msg, "direct")
-  expect_match(msg, "double-count|double count")
-  expect_match(msg, "cog_geographic_rollup")
-
-  # Test that cog_peer_compare's message names its own function
-  err2 <- tryCatch(
-    cog_peer_compare(target_govid = "010000226085", peers = "010000226085",
-                     category = "Police", years = 2019,
-                     expenditure_concept = "total"),
-    condition = function(e) e
-  )
-  msg2 <- paste(conditionMessage(err2), collapse = " ")
-  expect_match(msg2, "direct")
-  expect_match(msg2, "double-count|double count")
-  expect_match(msg2, "cog_peer_compare")
-})
-
-test_that("both cross-government verbs still accept the direct default", {
-  expect_no_error(
-    cog_geographic_rollup(govids = list(state = "010000226085"),
-                          category = "Police", years = 2019)
-  )
-  expect_no_error(
-    cog_peer_compare(target_govid = "010000226085", peers = "010000226085",
-                     category = "Police", years = 2019)
-  )
-})
-
-test_that("provenance always records the expenditure concept", {
-  d <- cog_spending("010000226085", years = 2019, category = "Police")
-  t <- cog_spending("010000226085", years = 2019, category = "Police",
-                    expenditure_concept = "total")
-  expect_equal(attr(d, "provenance")$expenditure_concept, "direct")
-  expect_equal(attr(t, "provenance")$expenditure_concept, "total")
-  # The note explains the non-obvious part: how legacy IG was assembled.
-  expect_true(nzchar(attr(t, "provenance")$expenditure_concept_note))
-  expect_true(is.na(attr(d, "provenance")$expenditure_concept_note) ||
-              !nzchar(attr(d, "provenance")$expenditure_concept_note))
-})
-
-test_that("the provenance schema documents expenditure_concept", {
-  sch <- jsonlite::fromJSON(
-    system.file("schemas", "provenance-v1.json", package = "uscogdata"),
-    simplifyVector = FALSE
-  )
-  expect_true("expenditure_concept" %in% names(sch$properties))
-})
-
-test_that("a firing suggestion names the intergovernmental counterpart recipe", {
-  # Corrections has no legacy leaf rows, so the coverage-gap suggestion fires;
-  # corrections_ig_local_combined is its IG counterpart.
-  r <- suppressMessages(
-    cog_spending("010000226085", years = c(2005, 2011), category = "Corrections")
-  )
-  sugg <- attr(r, "provenance")$suggestions
-  expect_gt(length(sugg), 0L)
-  ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
-  expect_true("corrections_combined" %in% ids)
-  ig <- unlist(lapply(sugg, function(s) s$ig_recipe_id))
-  expect_true("corrections_ig_local_combined" %in% ig)
-})
-
-test_that("no suggestion fires for a healthy query", {
-  r <- cog_spending("010000226085", years = 2019, category = "Police")
-  expect_length(attr(r, "provenance")$suggestions, 0L)
-})
-
-test_that("a mis-scoped cog_spending() call never attaches an M/L counterpart to a revenue-flavored recipe", {
-  # "IG Federal" is a revenue-only category (summary_categories maps it to
-  # B-prefixed component codes only; its recipes are ig_federal_b47_wide /
-  # ig_federal_b89_wide). A cog_spending() call scoped to it returns zero
-  # spending rows for every requested year -- there is no spending
-  # component in this category at all -- so the coverage-gap machinery
-  # fires for real (not hypothetically) even though this isn't the kind of
-  # format-boundary gap the recipe catalog is meant to signpost. This is
-  # exactly the live-corpus risk flagged in review: ig_federal_b47_wide's
-  # own component codes (B47/B94, suffixes {"47","94"}) are an EXACT
-  # suffix-set match for the expenditure recipe ige_local_m47_wide
-  # (M47/M94, same suffixes) -- a coincidence of reused digits, not a real
-  # Direct/Total pairing. The flow-family gate in
-  # .attach_ig_counterparts() must keep ig_recipe_id NULL here.
-  #
-  # Anchored on FL state government, not AL. Coverage is presence-based: a
-  # recipe is only suggested when its component codes have rows for the
-  # requested government-year. AL state's only FY2011 B47 cell was an
-  # explicit zero, which the corpus no longer stores after sparsification
-  # (SB194, cog_pipeline#64), so the recipe stopped being a candidate there.
-  # FL state carries a real FY2011 B47 amount, so this exercises the guard
-  # against a suggestion that genuinely fires.
-  r <- suppressMessages(
-    cog_spending("120000226351", years = c(2005, 2011), category = "IG Federal")
-  )
-  sugg <- attr(r, "provenance")$suggestions
-  expect_gt(length(sugg), 0L)
-  ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
-  expect_true("ig_federal_b47_wide" %in% ids)
-  ig <- unlist(lapply(sugg, function(s) s$ig_recipe_id))
-  expect_length(ig, 0L)
-})
-
-test_that("C1: 'total' on a legacy aggregate-only family reports the IG-only figure honestly, not as Direct + IG", {
-  # AL state government, Corrections, 2011. Measured pre-fix: 'total'
-  # returned $31,358,000 (the IG leg alone, on an aggregate-flagged M04/M05
-  # row) with 0 suggestions (the surviving IG row made the gap-detection
-  # machinery think the Direct leg was covered) and a note asserting
-  # "Total = Direct + intergovernmental" with no caveat. True Direct (via
-  # recipe = "corrections_combined") is $521,651,000 -- the IG-only figure
-  # is ~6% of it.
-  gov <- "010000226085"
-
-  d <- cog_spending(gov, years = 2011, category = "Corrections",
-                    expenditure_concept = "direct")
-  expect_equal(nrow(d), 0L)
-
-  t <- suppressMessages(cog_spending(
-    gov, years = 2011, category = "Corrections", expenditure_concept = "total"
-  ))
-  expect_equal(nrow(t), 1L)
-  expect_equal(t$spend_subtype, "intergovernmental")
-  expect_equal(t$amt_nominal, 31358000)
-
-  r <- cog_spending(gov, years = 2011, recipe = "corrections_combined")
-  expect_equal(r$amt_nominal, 521651000)
-
-  # C1(a): the recipe hints must fire for "total" exactly as they do for
-  # "direct" -- the surviving IG row must not be mistaken for Direct
-  # coverage.
-  prov <- attr(t, "provenance")
-  expect_gt(length(prov$suggestions), 0L)
-  ids <- vapply(prov$suggestions, function(s) s$recipe_id %||% "", character(1))
-  expect_true("corrections_combined" %in% ids)
-
-  # C1(b): the affected row's notes name a recovering recipe rather than
-  # staying silent, and the provenance carries a flag a downstream consumer
-  # (e.g. cog-api, which passes provenance through verbatim) can test.
-  expect_true(nzchar(t$notes))
-  expect_match(t$notes, "unavailable", fixed = TRUE)
-  expect_match(t$notes, "corrections_combined", fixed = TRUE)
-  expect_true(prov$expenditure_concept_direct_suppressed)
-
-  # The base "Total = Direct + IG" note must NOT stand unqualified when that
-  # arithmetic didn't actually happen for this row.
-  expect_match(prov$expenditure_concept_note, "NOTE", fixed = TRUE)
-  expect_match(prov$expenditure_concept_note,
-              "expenditure_concept_direct_suppressed", fixed = TRUE)
-})
-
-test_that("C1(b): expenditure_concept_direct_suppressed is FALSE when the Direct leg is present", {
-  d <- cog_spending("010000226085", years = 2019, category = "Police",
-                    expenditure_concept = "direct")
-  t <- cog_spending("010000226085", years = 2019, category = "Police",
-                    expenditure_concept = "total")
-  expect_false(isTRUE(attr(d, "provenance")$expenditure_concept_direct_suppressed))
-  expect_false(isTRUE(attr(t, "provenance")$expenditure_concept_direct_suppressed))
-  expect_false(any(nzchar(t$notes[t$spend_subtype == "intergovernmental"]) &
-                   grepl("unavailable", t$notes[t$spend_subtype == "intergovernmental"])))
-})
-
-# M/I fix: .detect_direct_suppressed() was equating "no Direct sibling row"
-# with "Direct was suppressed", but the dominant real cause is a government
-# that simply has no direct spending in that category -- correct, ordinary
-# data. The fix gates the flag (and its row note) on a harmonization recipe
-# ACTUALLY covering that exact (year, canonical_govid, category) triple.
-
-test_that("M/I: true positive, category supplied explicitly (unchanged behavior)", {
-  al <- "010000226085"
-  t_cat <- suppressMessages(cog_spending(
-    al, years = 2011, category = "Corrections", expenditure_concept = "total"
-  ))
-  expect_true(attr(t_cat, "provenance")$expenditure_concept_direct_suppressed)
-  expect_match(t_cat$notes, "corrections_combined", fixed = TRUE)
-  expect_match(t_cat$notes, "unavailable", fixed = TRUE)
-})
-
-test_that("M/I: true positive, category = NULL now also names the recipe (was the fallback bug)", {
-  # Root bug: .build_suggestions() short-circuits to list() when category is
-  # NULL, so the note previously always hit its "no covering recipe found"
-  # fallback here even though corrections_combined genuinely covers this row.
-  al <- "010000226085"
-  t_null <- suppressMessages(cog_spending(
-    al, years = 2011, category = NULL, expenditure_concept = "total"
-  ))
-  corr_row <- t_null[t_null$category %in% "Corrections", ]
-  expect_equal(nrow(corr_row), 1L)
-  expect_true(attr(t_null, "provenance")$expenditure_concept_direct_suppressed)
-  expect_match(corr_row$notes, "corrections_combined", fixed = TRUE)
-  expect_match(corr_row$notes, "unavailable", fixed = TRUE)
-  expect_false(grepl("no covering recipe found", corr_row$notes, fixed = TRUE))
-})
-
-test_that("M/I: false positive -- Virginia Education K-12 FY2019 total is NOT flagged", {
-  # States fund K-12 through school districts, so the Direct leg (E12/F12/
-  # G12) is genuinely, correctly zero -- not suppressed. Must not be flagged
-  # and must carry no suppression note.
-  va <- "510000227542"
-  t_va <- suppressMessages(cog_spending(
-    va, years = 2019, category = "Education K-12", expenditure_concept = "total"
-  ))
-  expect_equal(nrow(t_va), 1L)
-  expect_equal(t_va$spend_subtype, "intergovernmental")
-  expect_equal(t_va$amt_nominal, 8028179000)
-  expect_false(isTRUE(attr(t_va, "provenance")$expenditure_concept_direct_suppressed))
-  expect_false(nzchar(t_va$notes) && grepl("unavailable", t_va$notes))
-})
-
-test_that("M/I: false positive by construction -- 'Other Education' has no E/F/G code, never flagged", {
-  # "Other Education" maps only to M21/L21 in summary_categories -- there is
-  # no E/F/G code for it in this corpus at all, so no Direct-recovering
-  # recipe can exist and it must never be flagged, in any fixture year.
-  con <- uscogdata:::.ensure_session()
-  years_all <- DBI::dbGetQuery(con, "SELECT DISTINCT year FROM long ORDER BY year")$year
-  states <- DBI::dbGetQuery(con,
-    "SELECT DISTINCT canonical_govid FROM long WHERE type = 0")$canonical_govid
-  oe <- suppressMessages(cog_spending(
-    states, years = years_all, category = "Other Education",
-    expenditure_concept = "total"
-  ))
-  expect_false(isTRUE(attr(oe, "provenance")$expenditure_concept_direct_suppressed))
-  expect_false(any(nzchar(oe$notes) & grepl("unavailable", oe$notes)))
-})
-
-test_that("M/I: a clean FY2019 category = NULL total query flags far fewer than the pre-fix 32/50 states", {
-  con <- uscogdata:::.ensure_session()
-  states <- DBI::dbGetQuery(con,
-    "SELECT DISTINCT canonical_govid FROM long WHERE type = 0")$canonical_govid
-  r <- suppressMessages(cog_spending(
-    states, years = 2019, category = NULL, expenditure_concept = "total"
-  ))
-  ig <- r[r$spend_subtype == "intergovernmental", ]
-  flagged <- ig[nzchar(ig$notes) & grepl("unavailable", ig$notes), ]
-  expect_lt(length(unique(flagged$canonical_govid)), 32L)
-  # Every remaining flagged row must actually name a covering recipe --
-  # never the old no-recipe-found fallback.
-  expect_true(all(grepl("recipe = '", flagged$notes, fixed = TRUE)))
-  expect_false(any(grepl("no covering recipe found", flagged$notes, fixed = TRUE)))
-})
-
-test_that("C2: expenditure_concept = 'total' aborts on a corpus with no intergovernmental category rows", {
-  with_corpus_missing_ig_categories({
-    con <- uscogdata:::.ensure_session()
-    n <- DBI::dbGetQuery(con,
-      "SELECT COUNT(*) AS n FROM summary_categories WHERE LEFT(item_code, 1) IN ('M', 'L')"
-    )$n
-    expect_equal(n, 0)
-
-    err <- tryCatch(
-      cog_spending("010000226085", years = 2019, category = "Police",
-                  expenditure_concept = "total"),
-      condition = function(e) e
-    )
-    expect_s3_class(err, "uscogdata_ig_categories_unsupported")
-    msg <- conditionMessage(err)
-    expect_match(msg, "PR #59|predates", perl = TRUE)
-  })
-
-  # 'direct' is unaffected on the same corpus -- the guard is scoped to
-  # expenditure_concept = "total" only.
-  with_corpus_missing_ig_categories({
-    expect_no_error(
-      cog_spending("010000226085", years = 2019, category = "Police",
-                  expenditure_concept = "direct")
-    )
-  })
-})
-
-test_that("C2: expenditure_concept = 'total' still works on a corpus that DOES carry M/L category rows", {
-  expect_no_error(
-    cog_spending("010000226085", years = 2019, category = "Police",
-                expenditure_concept = "total")
-  )
-})
-
-test_that("I2: an intergovernmental (M/L) recipe never appears as its own top-level suggestion", {
-  # Task 1's M04/M05 category rows share the "Corrections" summary_categories
-  # category with the Direct-flavored E04/E05, so `corrections_ig_local_
-  # combined` (entirely M-prefixed) becomes a raw *candidate* in
-  # .build_suggestions()'s component_code-driven query. Following a
-  # "re-run with recipe = 'corrections_ig_local_combined'" hint on a plain
-  # cog_spending() call would silently return intergovernmental dollars
-  # under provenance$expenditure_concept = "direct". Task 6's gate
-  # (.attach_ig_counterparts()) already protects the *counterpart* lookup;
-  # this exercises that the candidate list itself is filtered too.
-  r <- suppressMessages(
-    cog_spending("010000226085", years = c(2005, 2011), category = "Corrections")
-  )
-  sugg <- attr(r, "provenance")$suggestions
-  ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
-  expect_true("corrections_combined" %in% ids)
-  expect_false("corrections_ig_local_combined" %in% ids)
-})
-
-test_that(".attach_ig_counterparts() never pairs a revenue-side recipe with its coincidental M/L suffix twin", {
-  # Broader version of the case above, run at the matching-helper level
-  # (the same level code review's pairwise enumeration was done at) rather
-  # than end-to-end: the fixture has no (govid, year) combination where
-  # cog_revenue() itself produces a covered gap for any B/C/D recipe, so an
-  # end-to-end repro for THIS specific set of recipes isn't reachable
-  # today. Each of these six recipes shares an exact suffix set with an
-  # M/L expenditure recipe purely by reused-digit coincidence:
-  #   ig_federal_b47_wide {"47","94"} == ige_local_m47_wide / ige_state_l47_wide
-  #   ig_federal_b89_wide {"89","91","92","93"} == ige_local_m89_wide / ige_state_l89_wide
-  #   ig_state_c47_wide   {"47","94"} == ige_local_m47_wide / ige_state_l47_wide
-  #   ig_state_c89_wide   {"89","91","92","93"} == ige_local_m89_wide / ige_state_l89_wide
-  #   ig_local_d47_wide   {"47","94"} == ige_local_m47_wide / ige_state_l47_wide
-  #   ig_local_d89_wide   {"89","91","92","93"} == ige_local_m89_wide / ige_state_l89_wide
-  # None of them may receive an ig_recipe_id under cog_revenue()'s own
-  # flow_prefixes, since M/L only ever pairs with the direct-expenditure
-  # (E/F/G) family.
-  con <- uscogdata:::.ensure_session()
-  fake_suggestion <- function(rid) {
-    list(recipe_id = rid, label = "x", available_years = c(1967L, 2023L),
-         hint = "h")
-  }
-  fake_suggestions <- lapply(
-    c("ig_federal_b47_wide", "ig_federal_b89_wide",
-      "ig_state_c47_wide", "ig_state_c89_wide",
-      "ig_local_d47_wide", "ig_local_d89_wide"),
-    fake_suggestion
-  )
-  out <- uscogdata:::.attach_ig_counterparts(
-    con, fake_suggestions, c("T", "A", "U", "B", "C", "D")
-  )
-  ig <- unlist(lapply(out, function(s) s$ig_recipe_id))
-  expect_length(ig, 0L)
-})

tests/testthat/test-expenditure-concepts.R

@@ -1,75 +0,0 @@
-# Madison walkthrough audit -- findings F-012, F-017, F-018.
-# Tracked as uscogdata#11. See docs/walkthroughs/FINDINGS.md in cog_explorer.
-#
-# The owner's settled three-concept model (2026-07-28):
-#   total   = primary + interest + intergovernmental transfers
-#   direct  = primary + interest          (Census's published Direct Expenditure)
-#   primary = direct minus debt service   (the NEW DEFAULT)
-# implemented by reclassifying on the crosswalk's `spend_type` column, NOT on
-# item-code first letters -- F-018 shows prefix `Y` carries both revenue
-# (Y01/Y02) and expenditure (Y05/Y06) codes, so no first-letter allowlist can
-# route them correctly.
-#
-# Fixture reproducibility: the finding's headline reconciliation is Madison
-# FY2022, where the corpus carries I89 = 46,609 (thousands) and Census's
-# published Direct Expenditure is $654,893,000 against cog_spending()'s
-# $608,284,000 (-7.1%). FY2022 is outside the bundled fixture's year window
-# (2011/2012/2019/2020), so the same invariant is asserted on FY2020, where the
-# fixture carries I89 = 27,704. Anyone running against the full corpus should
-# also check the FY2022 numbers above.
-
-test_that("expenditure concepts classify on spend_type, not item-code prefix", {
-  testthat::skip("Blocked on uscogdata#11 (findings F-012, F-017, F-018)")
-
-  mad      <- "552025209777"   # MADISON CITY, WI
-  wi_state <- "550000227544"   # WISCONSIN (state government)
-
-  # -- F-012: `primary` is the new default, and equals today's E/F/G figure ---
-  primary <- cog_spending(govid = mad, years = 2020L)
-  expect_equal(attr(primary, "provenance")$expenditure_concept, "primary")
-  expect_equal(sum(primary$amt_nominal), 623347000)
-
-  # -- F-012: `direct` adds interest on long-term debt ------------------------
-  # Expected interest read from the RAW corpus, never through cog_spending(),
-  # which is the filter under test.
-  interest <- wt_raw_amt(mad, 2020L, prefixes = "I")
-  expect_equal(interest, 27704)                       # I89, in $1,000s
-
-  direct <- cog_spending(govid = mad, years = 2020L, expenditure_concept = "direct")
-  expect_equal(sum(direct$amt_nominal), 651051000)    # 623,347 + 27,704 thousands
-  expect_equal(sum(direct$amt_nominal) - sum(primary$amt_nominal), interest * 1000)
-  expect_true("I89" %in% wt_codes_included(direct))
-
-  # -- F-017: `total` carries Q12/Q18, state IG transfers to school districts --
-  # Wisconsin FY2019: Q12 = 6,431,530 and Q18 = 533,391 (thousands). Today
-  # neither verb's flow_prefixes contains "Q", so both are dropped from the one
-  # concept that is supposed to include intergovernmental transfers.
-  ig_expected <- wt_raw_amt(wi_state, 2019L, prefixes = c("M", "L", "Q"))
-  expect_equal(ig_expected, 11609814)                 # M 4,644,893 + Q 6,964,921
-
-  wi_direct <- cog_spending(govid = wi_state, years = 2019L,
-                            expenditure_concept = "direct")
-  wi_total  <- cog_spending(govid = wi_state, years = 2019L,
-                            expenditure_concept = "total")
-
-  # total - direct is exactly the intergovernmental component. Asserted as a
-  # delta rather than a grand total so this stays correct however the J and Y
-  # families land inside `primary`.
-  expect_equal(sum(wi_total$amt_nominal) - sum(wi_direct$amt_nominal),
-               ig_expected * 1000)
-  expect_true(all(c("Q12", "Q18") %in% wt_codes_included(wi_total)))
-
-  # -- F-018: prefix Y splits revenue from expenditure, by spend_type ---------
-  # Y01/Y02 are Insurance Trust revenue; Y05/Y06 are Insurance Trust benefit
-  # payments. All four share the first letter `Y` and the spend_type
-  # "Insurance Trust", so this pair of assertions is the concrete proof that
-  # classification is no longer keyed on the first letter.
-  wi_revenue <- cog_revenue(govid = wi_state, years = 2019L)
-  spend_codes <- wt_codes_included(wi_total)
-  rev_codes   <- wt_codes_included(wi_revenue)
-
-  expect_true("Y05" %in% spend_codes)
-  expect_false("Y05" %in% rev_codes)
-  expect_true("Y01" %in% rev_codes)
-  expect_false("Y01" %in% spend_codes)
-})

tests/testthat/test-explain.R

@@ -70,37 +70,6 @@ test_that("cog_explain prints a Suggestions section when the provenance has one"
   expect_true(grepl("re-run with recipe", txt))
 })
 
-test_that("cog_explain prints the expenditure concept (I1)", {
-  skip_if_no_corpus()
-  d <- cog_spending("010000226085", years = 2019, category = "Police")
-  t <- cog_spending("010000226085", years = 2019, category = "Police",
-                    expenditure_concept = "total")
-  txt_d <- paste(c(
-    capture.output(cog_explain(d)),
-    capture.output(cog_explain(d), type = "message")
-  ), collapse = "\n")
-  txt_t <- paste(c(
-    capture.output(cog_explain(t)),
-    capture.output(cog_explain(t), type = "message")
-  ), collapse = "\n")
-  expect_true(grepl("Concept: direct", txt_d))
-  expect_true(grepl("Concept: total", txt_t))
-})
-
-test_that("cog_explain surfaces the C1(b) direct-suppressed flag as a warning", {
-  skip_if_no_corpus()
-  t <- suppressMessages(cog_spending(
-    "010000226085", years = 2011, category = "Corrections",
-    expenditure_concept = "total"
-  ))
-  expect_true(attr(t, "provenance")$expenditure_concept_direct_suppressed)
-  txt <- paste(c(
-    capture.output(cog_explain(t)),
-    capture.output(cog_explain(t), type = "message")
-  ), collapse = "\n")
-  expect_true(grepl("Direct leg unavailable", txt))
-})
-
 test_that("cog_explain prints denominator + popyear_range + counts", {
   skip_if_no_corpus()
   with_fixture_corpus({

tests/testthat/test-fixture-vintage.R

@@ -1,112 +0,0 @@
-# tests/testthat/test-fixture-vintage.R
-#
-# The bundled fixture is a slice of a real cog_pipeline publish tree, and
-# every test in this package -- plus the whole cog-api suite -- runs against
-# it. When the published corpus changes shape and the fixture does not, both
-# suites stay green against a corpus that no longer exists (uscogdata#18).
-#
-# These tests pin the structural facts that distinguish the current published
-# vintage from its predecessor, so a stale fixture fails loudly instead of
-# passing quietly. They assert shape, never dollar values: re-running
-# data-raw/regenerate_fixture_corpus.R against a newer publish tree should
-# keep them green.
-
-# Open a bare DuckDB connection on the fixture's parquet files. Deliberately
-# not the package session: these assertions are about what the fixture
-# CONTAINS, and routing them through the reader's own views would let a
-# filter hide the very absence being checked.
-fixture_query <- function(sql, ...) {
-  con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
-  path <- function(rel) {
-    sprintf("read_parquet(%s)",
-            DBI::dbQuoteString(con, file.path(fixture_corpus_path(), rel)))
-  }
-  DBI::dbGetQuery(con, do.call(sprintf, c(list(sql), lapply(c(...), path))))
-}
-
-test_that("fixture ships every metadata table the publish tree does", {
-  skip_if_no_corpus()
-  # representation/code_set are what make a sparse corpus interpretable; a
-  # fixture without them predates sparsification (cog_pipeline#64).
-  expected <- c(
-    "canonical_alias.parquet", "canonical_fips_xwalk.parquet",
-    "census_collection_coverage.parquet", "code_set.parquet",
-    "harmonization_map.parquet", "harmonization_recipes.parquet",
-    "lineage_events.parquet", "representation.parquet",
-    "series_breaks.parquet", "summary_categories.parquet"
-  )
-  on_disk <- basename(list.files(
-    file.path(fixture_corpus_path(), "data"), pattern = "\\.parquet$"
-  ))
-  expect_true(all(expected %in% on_disk))
-
-  # The manifest must list them too -- consumers read the manifest, not ls().
-  in_manifest <- with_fixture_corpus(
-    basename(vapply(cog_manifest()$files$metadata, function(f) f$path, character(1)))
-  )
-  expect_true(all(expected %in% in_manifest))
-})
-
-test_that("fixture carries the dense/sparse representation contract", {
-  skip_if_no_corpus()
-  rep <- fixture_query(
-    "SELECT year, representation, absence_means FROM %s
-     WHERE year IN (2011, 2012, 2019, 2020) ORDER BY year",
-    "data/representation.parquet"
-  )
-  expect_equal(nrow(rep), 4L)
-  expect_equal(rep$representation, c("dense_source", rep("sparse_source", 3L)))
-  expect_equal(rep$absence_means, c("census_zero", rep("not_reported", 3L)))
-})
-
-test_that("the fixture's wide era is sparse, not zero-padded", {
-  skip_if_no_corpus()
-  # FY2011 is a dense_source year: the corpus publishes only the cells Census
-  # reported non-zero, and an absent cell means Census published $0. Before
-  # sparsification this partition was 2,864,212 rows, ~83% of them explicit
-  # zeros. A single explicit zero here means the fixture predates the change.
-  zeros_2011 <- fixture_query(
-    "SELECT COUNT(*) AS n FROM %s WHERE amt = 0",
-    "data/long/year=2011/part-0.parquet"
-  )$n
-  expect_equal(zeros_2011, 0L)
-
-  # The modern era is a different regime: a reported zero there is real data
-  # (the government filed $0), so zeros legitimately survive and must not be
-  # asserted away.
-  expect_gt(
-    fixture_query("SELECT COUNT(*) AS n FROM %s", "data/long/year=2012/part-0.parquet")$n,
-    0L
-  )
-})
-
-test_that("code_set covers every fixture year with the reader-spec columns", {
-  skip_if_no_corpus()
-  cs <- fixture_query(
-    "SELECT * FROM %s WHERE year IN (2011, 2012, 2019, 2020)",
-    "data/code_set.parquet"
-  )
-  expect_true(all(
-    c("code_set_id", "year", "type", "item_code", "is_aggregate", "n_units")
-    %in% names(cs)
-  ))
-  expect_setequal(unique(cs$year), c(2011L, 2012L, 2019L, 2020L))
-})
-
-test_that("every flow code carrying dollars has a category, J-prefix included", {
-  skip_if_no_corpus()
-  # The J (assistance/benefit) codes were uncategorised until the crosswalk
-  # completion shipped (cog_pipeline#60/#65, J19 held back until #64's
-  # duplication fix landed). Their absence is how a pre-crosswalk fixture
-  # gives itself away.
-  j <- fixture_query(
-    "SELECT item_code, category, category_type, spend_subtype FROM %s
-     WHERE LEFT(item_code, 1) = 'J' ORDER BY item_code",
-    "data/summary_categories.parquet"
-  )
-  expect_true("J19" %in% j$item_code)
-  expect_true(all(j$category_type == "expenditure"))
-  expect_true(all(j$spend_subtype == "assistance"))
-  expect_false(any(is.na(j$category)))
-})

tests/testthat/test-gov-search-literal-match.R

@@ -1,57 +0,0 @@
-# Madison walkthrough audit -- finding F-025. Tracked as uscogdata#16.
-# See docs/walkthroughs/FINDINGS.md in cog_explorer.
-#
-# cog_gov_search()'s UTILITY mode interpolates `name` into
-#   regexp_matches(gov_name, <name>, 'i')
-# unescaped (R/search.R:102), while BASKET mode in the same file already routes
-# it through .escape_regex() (R/search.R:307) with the comment "so `name` is
-# treated as a literal substring". Two failure modes result:
-#   correctness -- a real government cannot be found by its own exact name, and
-#                  a single "." matches everything (HTTP 200 both ways via the API);
-#   robustness  -- malformed regex reaches the engine and errors, which cog-api
-#                  surfaces as a 500, reachable by typing a real name one
-#                  character at a time.
-#
-# NOT asserted here: the finding's `q=St. Louis` example. Under correct literal
-# matching that search still returns 0 rows, because the stored name is
-# "ST LOUIS CITY" with no period -- it demonstrates today's over-matching
-# semantics, not a row the fix makes findable.
-
-test_that("cog_gov_search() matches name literally, not as an unescaped regex", {
-
-  # -- correctness (1): a government must be findable by its own exact name ---
-  # FREDONIA (BRISCOE) CITY is real; today the parentheses are read as regex
-  # grouping, so its own complete name matches nothing.
-  fredonia <- cog_gov_search(name = "FREDONIA (BRISCOE) CITY")
-  expect_equal(nrow(fredonia), 1L)
-  expect_equal(fredonia$canonical_govid, "052117184386")
-  expect_equal(cog_gov_search(name = "FREDONIA (BRISCOE)")$canonical_govid,
-               "052117184386")
-
-  # -- correctness (2): a metacharacter must not become a wildcard ------------
-  # No Wisconsin city or village name contains a literal period -- established
-  # against the raw registry below, NOT through the verb under test. A literal
-  # search for "." must therefore return nothing; today it returns all 608.
-  con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
-  xwalk <- paste0(sub("/$", "", Sys.getenv("USCOGDATA_URL")),
-                  "/data/canonical_fips_xwalk.parquet")
-  with_dot <- DBI::dbGetQuery(con, paste0(
-    "SELECT COUNT(*) n FROM read_parquet('", xwalk, "') ",
-    "WHERE fips_state = '55' AND govs_type = 2 AND gov_name LIKE '%.%'"))
-  expect_equal(as.integer(with_dot$n[[1]]), 0L)
-
-  expect_equal(nrow(cog_gov_search(name = ".", state = "WI", type = "city")), 0L)
-  expect_equal(nrow(cog_gov_search(name = "M.dison", state = "WI", type = "city")), 0L)
-  expect_equal(nrow(cog_gov_search(name = "Mad(i|o)son", state = "WI", type = "city")), 0L)
-
-  # A metacharacter-free name still resolves exactly as before.
-  expect_equal(nrow(cog_gov_search(name = "Madison", state = "WI", type = "city")), 1L)
-
-  # -- robustness: malformed pattern text returns no rows, and does not error --
-  # "[" alone, and "Athens-Clarke County (bal" -- an in-progress substring of
-  # ATHENS-CLARKE COUNTY (BALANCE), a real government -- both currently raise
-  # (DuckDB: "Invalid Input Error: missing ]").
-  expect_equal(nrow(cog_gov_search(name = "[")), 0L)
-  expect_equal(nrow(cog_gov_search(name = "Athens-Clarke County (bal")), 0L)
-})

tests/testthat/test-peer-summary-scope.R

@@ -1,51 +0,0 @@
-# Madison walkthrough audit -- finding F-021. Tracked as uscogdata#14.
-# See docs/walkthroughs/FINDINGS.md in cog_explorer.
-#
-# .peer_summary_rows() computes stats::quantile() separately INSIDE each
-# (year, spend_subtype, category) cell. A summary_p50 row is therefore "the
-# median peer's value in that one category", not "the value of the median
-# peer's total". Summing those rows across categories -- the obvious move for a
-# caller who wants one peer-median total line and reads only the column names --
-# misstated a total-spending band by -32.7% to +251.0% across the 24 years the
-# audit tested, with a sign flip at FY2012.
-#
-# The verb is not wrong and its documented use (faceting by role AND category)
-# is unaffected, so the fix is documentation: one sentence in @return.
-
-test_that("cog_peer_compare() documents that summary_* rows are per-category quantiles", {
-
-  # man/ ships only in the source tree (the installed package carries a
-  # compiled help database instead), so the prose assertions below cannot run
-  # under R CMD check -- CI's earlier testthat::test_local() step enforces
-  # them. The numeric pin further down needs only the corpus, but it lives in
-  # the same test_that() as the sentence it protects, deliberately: they are
-  # one claim, and splitting them would let the prose drift while a separate
-  # test kept passing.
-  rd_path <- skip_if_no_source_tree(c("man", "cog_peer_compare.Rd"))
-  rd <- paste(readLines(rd_path, warn = FALSE), collapse = " ")
-
-  # The @return section must say the quantile is computed within each cell...
-  expect_match(rd, "within each|per-category|per category", ignore.case = TRUE)
-  # ...and must warn that the rows are not additive across category.
-  expect_match(rd, "not additive|do(es)? not sum|cannot be summed", ignore.case = TRUE)
-  # ...naming the grouping explicitly.
-  expect_match(rd, "spend_subtype", fixed = TRUE)
-
-  # Pin the mechanism numerically so a future refactor that quietly changes the
-  # quantile grouping fails here rather than silently invalidating the sentence
-  # above. Fixture: Madison, 10 peers found at FY2020, category = NULL.
-  peers <- cog_find_peers("552025209777", year = 2020L, max_peers = 10L)
-  cmp <- cog_peer_compare(target_govid = "552025209777", peers = peers,
-                          category = NULL, years = 2020L, per_capita = TRUE)
-
-  naive <- sum(cmp$amt_per_capita_nominal[cmp$role == "summary_p50"], na.rm = TRUE)
-
-  peer_rows <- cmp[cmp$role == "peer", ]
-  per_gov <- tapply(peer_rows$amt_per_capita_nominal, peer_rows$canonical_govid,
-                    sum, na.rm = TRUE)
-  correct <- unname(stats::quantile(per_gov, 0.5, na.rm = TRUE))
-
-  expect_equal(round(naive), 6180)     # summing the built-in summary rows
-  expect_equal(round(correct), 2043)   # quantile of each peer's OWN total
-  expect_gt(naive / correct, 2)        # a +200% misstatement on this cohort
-})

tests/testthat/test-recipes.R

@@ -176,6 +176,16 @@ test_that("recipe = requires schema_version >= 5", {
 })
 
 # --- signposting -------------------------------------------------------
+#
+# Phase R3 / Task 19c: .build_suggestions() was narrowed from a whole-result
+# gap check (R2: does the ENTIRE category result have zero rows in a
+# requested year) to per-code gap detection (does a specific recipe
+# component -- itself a member of the requested category -- have zero rows
+# in a year the recipe's own generic join otherwise covers). See
+# R/suggestions.R's header comment and docs/phase_r_harmonization_review.md
+# § 0.3. The R2 test below ("...across the 2011->2012 gap") is unaffected
+# by the refinement (it already passed under both the coarse and per-code
+# rule). The next few pin cases the coarse rule specifically could NOT see.
 
 test_that("signposting suggests corrections_combined across the 2011->2012 gap", {
   skip_if_no_corpus()
@@ -193,10 +203,96 @@ test_that("signposting suggests corrections_combined across the 2011->2012 gap",
   expect_equal(hit$available_years, c(1967L, 2023L))
 })
 
-test_that("no signposting when the result already has full year coverage", {
+test_that("per-code gap does NOT fire when a code's only coverage is its own aggregate row (self-coverage is not \"other components\")", {
   skip_if_no_corpus()
+  # Broward, FY2011 ONLY (isolating the 2011 half of the query above): E05,
+  # F05, and G05 each report SOLELY as a wide-era AGGREGATE row that year
+  # (216088, 1453, 270 respectively); E04/F04/G04 -- their modern-only
+  # siblings -- don't exist as codes at all before 2012, corpus-wide (zero
+  # rows for any government). Each component's own aggregate row would
+  # trivially satisfy a same-component "covered" check, but review-doc
+  # § 0.3's criterion is explicit that a gap must be covered by "OTHER
+  # components", not the gapped component's own aggregate form. With no
+  # OTHER component present for any of the three Corrections recipes in
+  # 2011, none of them should fire -- this is what the combined
+  # 2011-2012 test above actually relies on 2012 (E05 gapped, E04 -- a
+  # genuinely different component -- covers) to fire, not 2011.
+  r <- cog_spending("121011212191", years = 2011L, category = "Corrections")
+  prov <- attr(r, "provenance")
+  expect_length(prov$suggestions, 0L)
+})
+
+test_that("per-code gap fires even when a sibling code masks the whole-result check (Cleburne County, FY2012)", {
+  skip_if_no_corpus()
+  # Cleburne County, AL (canonical_govid 011029122489), FY2012: E04 ($854)
+  # and E05 ($1) both report ("operations" subtype), and G04 ($14,000,
+  # corrections_other_capital_combined's modern-only leg) also reports
+  # ("capital" subtype) -- so the WHOLE category result is non-empty for
+  # 2012 (2 rows) and the R2 whole-result check would never look further.
+  # But G05 -- G04's OWN recipe sibling, the 1967-2023 wide leg -- has
+  # ZERO rows at all that year: a genuine, per-code gap the recipe exists
+  # to bridge, invisible at the category-result grain because it's masked
+  # by G04's own data, let alone the unrelated E04/E05 pair.
+  r <- cog_spending("011029122489", years = 2012L, category = "Corrections")
+  expect_equal(nrow(r), 2L)  # operations + capital rows: a non-empty result
+
+  prov <- attr(r, "provenance")
+  ids <- vapply(prov$suggestions, function(s) s$recipe_id, character(1))
+  expect_true("corrections_other_capital_combined" %in% ids)
+  hit <- prov$suggestions[[which(ids == "corrections_other_capital_combined")]]
+  expect_equal(hit$hint, "re-run with recipe = 'corrections_other_capital_combined'")
+  expect_equal(hit$available_years, c(1967L, 2023L))
+
+  # corrections_combined must NOT fire: E04 AND E05 both have real 2012
+  # data for this government, so neither of ITS OWN components is gapped.
+  expect_false("corrections_combined" %in% ids)
+})
+
+test_that("per-code gap does not fire when no recipe component has any data at all (ordinary reporting variance, not a format-boundary gap)", {
+  skip_if_no_corpus()
+  # Same government/year as above: F04 and F05 (corrections_capital_combined)
+  # are BOTH completely absent -- Cleburne simply never reported capital
+  # corrections spending under that code family in 2012, wide-era or
+  # modern. The recipe's own generic join (aggregate-inclusive, either
+  # component) has nothing to offer either, so this must stay silent --
+  # the per-government `covered` guard the header comment describes is
+  # unchanged and still does this filtering.
+  r <- cog_spending("011029122489", years = 2012L, category = "Corrections")
+  prov <- attr(r, "provenance")
+  ids <- vapply(prov$suggestions, function(s) s$recipe_id, character(1))
+  expect_false("corrections_capital_combined" %in% ids)
+})
+
+test_that("per-code gap fires for Broward 2019-2020 even though the category result looks complete", {
+  skip_if_no_corpus()
+  # Broward reports E04 + G04 (modern leaf codes) in BOTH 2019 and 2020 but
+  # never reports E05 or G05 (their own recipe siblings) in either year --
+  # a real per-code gap in two of the three Corrections recipes, invisible
+  # under the R2 coarse check because the category *result* is non-empty
+  # both years (this replaces the old R2-era "full year coverage" test,
+  # whose premise -- that a non-empty result implies nothing to signpost --
+  # is exactly what this refinement narrows; see data-raw/
+  # measure_signposting_rate.R for the measured rate change this causes).
+  # corrections_capital_combined correctly stays silent: Broward reports
+  # neither F04 nor F05 in 2019 or 2020, so that recipe's own join has
+  # nothing to offer either (ordinary non-reporting, not a format-boundary
+  # gap) -- the per-government `covered` guard still does its job here too.
   r <- cog_spending("121011212191", years = 2019:2020, category = "Corrections")
   prov <- attr(r, "provenance")
+  ids <- vapply(prov$suggestions, function(s) s$recipe_id, character(1))
+  expect_true("corrections_combined" %in% ids)
+  expect_true("corrections_other_capital_combined" %in% ids)
+  expect_false("corrections_capital_combined" %in% ids)
+})
+
+test_that("no signposting when every recipe component genuinely has data (true full per-code coverage)", {
+  skip_if_no_corpus()
+  # Maricopa County, AZ (canonical_govid 041013160815): all six Corrections
+  # codes (E04, E05, F04, F05, G04, G05) report real, nonzero, non-aggregate
+  # amounts in BOTH 2019 and 2020 -- genuinely nothing for any recipe to
+  # fill, even at the finer per-code grain this refinement now checks.
+  r <- cog_spending("041013160815", years = 2019:2020, category = "Corrections")
+  prov <- attr(r, "provenance")
   expect_length(prov$suggestions, 0L)
 })
 

tests/testthat/test-revenue-concept-insurance-trust.R

@@ -1,55 +0,0 @@
-# Madison walkthrough audit -- finding F-014. Tracked as uscogdata#12.
-# See docs/walkthroughs/FINDINGS.md in cog_explorer.
-#
-# cog_revenue()'s flow_prefixes = c("T","A","U","B","C","D") never returns
-# item-code prefix X (Employee Retirement) or Y (other Insurance Trust). Per
-# Census's standard identity, Total Revenue = General + Utility + Liquor Store +
-# Insurance Trust Revenue, and Employee Retirement System contributions and
-# earnings ARE the Insurance Trust Revenue component -- so prefix X sits inside
-# a published Census revenue concept exactly the way I89 sits inside Census's
-# Direct Expenditure concept (finding F-012).
-#
-# CAVEAT FOR WHOEVER PICKS THIS UP: the argument name below (`revenue_concept =
-# "total"`) is this test's *proposal*, not a settled decision. The owner's
-# 2026-07-28 resolution covers expenditure concepts only; no revenue-side
-# naming has been ruled on. If the eventual argument is named differently,
-# change the two calls here -- the asserted dollar invariants are what matter
-# and are independent of the naming.
-#
-# Fixture reproducibility: Madison's own X-prefix revenue (FY1970-FY1986,
-# $15,098,000 nominal, $0 thereafter) is outside the bundled fixture's year
-# window (2011/2012/2019/2020), so the same invariant is asserted on Wisconsin
-# state government FY2012, where the fixture carries nonzero X01/X05/X08.
-
-test_that("cog_revenue() can return Census Total Revenue including Insurance Trust (prefix X)", {
-  testthat::skip("Blocked on uscogdata#12 (finding F-014)")
-
-  wi_state <- "550000227544"   # WISCONSIN (state government)
-
-  # Revenue-shaped Employee Retirement codes, read from the RAW corpus rather
-  # than through cog_revenue(), which is the filter under test:
-  #   X01 local employee contribution, X04/X05 contributions and transfers from
-  #   other governments, X08 earnings on investments.
-  x_revenue <- wt_raw_amt(wi_state, 2012L, codes = c("X01", "X04", "X05", "X08"))
-  expect_equal(x_revenue, 2038800)   # 615,835 + 0 + 560,382 + 862,583 ($1,000s)
-
-  general <- cog_revenue(govid = wi_state, years = 2012L)
-  expect_equal(sum(general$amt_nominal), 31338293000)
-
-  total <- cog_revenue(govid = wi_state, years = 2012L, revenue_concept = "total")
-  expect_equal(sum(total$amt_nominal) - sum(general$amt_nominal), x_revenue * 1000)
-  expect_equal(sum(total$amt_nominal), 33377093000)
-  expect_true(all(c("X01", "X05", "X08") %in% wt_codes_included(total)))
-
-  # Sibling codes under the SAME first letter must stay out: X11/X12 are
-  # benefit payments (an expenditure) and X21/X30/X47 are cash and securities
-  # holdings (a balance-sheet stock). This is the F-018 point restated on the
-  # revenue side -- the split has to come from the crosswalk's spend_type, not
-  # from the letter X.
-  expect_false(any(c("X11", "X12", "X21", "X30", "X47") %in% wt_codes_included(total)))
-
-  # Every returned row still resolves to a category. summary_categories has
-  # zero rows for prefix X today, so relaxing the prefix filter alone would
-  # produce category = NA rows -- see census_of_governments_finance_pipeline#60.
-  expect_false(any(is.na(total$category)))
-})

tests/testthat/test-rollup.R

@@ -80,11 +80,8 @@ test_that("cog_geographic_rollup provenance reports the outer verb", {
 
 test_that("cog_geographic_rollup accepts data.frames per layer", {
   skip_if_no_corpus()
-  # Unanchored: utility mode matches literally now, so "^...$" would be
+  fl_state  <- cog_gov_search("^FLORIDA$", type = "state")
-  # searched for as characters rather than read as anchors (uscogdata#16).
+  broward   <- cog_gov_search("^BROWARD COUNTY$", state = "FL", type = "county")
-  # Both still resolve to exactly one row once scoped by type/state.
-  fl_state  <- cog_gov_search("FLORIDA", type = "state")
-  broward   <- cog_gov_search("BROWARD COUNTY", state = "FL", type = "county")
   r <- cog_geographic_rollup(
     govids = list(state = fl_state, county = broward),
     category = "Police", years = 2020L

tests/testthat/test-signposting-harness.R

@@ -0,0 +1,164 @@
+# tests/testthat/test-signposting-harness.R
+#
+# Pins the subset-relation REPORTING in data-raw/measure_signposting_rate.R.
+#
+# Phase R3 Task 19c narrowed signposting from a coarse whole-result gap
+# check to per-code gap detection. Those two checks are partly DISJOINT,
+# not nested: a query can fire under coarse and stay silent under per-code,
+# so the harness's `*_delta_pp` figures are NETS that can hide a coverage
+# loss. `.measure_subset_relation()` is what separates the two flows, and
+# `.measure_format_subset_report()` is what puts the loss in front of a
+# human. Both are load-bearing for the Checkpoint R3 ruling, so both are
+# pinned here: if the violation detection is deleted, inverted, or quietly
+# downgraded to a count with no identities, these tests fail.
+#
+# These tests do NOT assert that the violation set is empty -- it is
+# genuinely non-empty, and asserting otherwise would be pinning a bug as a
+# contract. They assert only that a real violation is DETECTED and NAMED.
+
+# The harness lives in data-raw/, which is .Rbuildignore'd, so it is absent
+# from an installed/checked tarball. Source it into an env parented on the
+# namespace so it resolves the package internals it calls (.build_verb_sql,
+# .build_suggestions) exactly as it does when run for real.
+harness_env <- function() {
+  path <- testthat::test_path("..", "..", "data-raw", "measure_signposting_rate.R")
+  skip_if_not(file.exists(path),
+              "data-raw/ is .Rbuildignore'd; harness not present in this tree")
+  env <- new.env(parent = asNamespace("uscogdata"))
+  source(path, local = env)
+  env
+}
+
+# A detail frame in exactly the shape .measure_one_query() emits, covering
+# all four quadrants of the coarse x percode cross-tab.
+fake_detail <- function() {
+  data.frame(
+    category         = c("Corrections", "Other Taxes", "Police", "Fire"),
+    category_type    = c("expenditure", "revenue", "expenditure", "expenditure"),
+    canonical_govid  = c("121011212191", "472155175824", "011029122489",
+                         "041013160815"),
+    n_result_rows    = c(0L, 1L, 2L, 6L),
+    coarse_gap_years = c("2011", "", "", ""),
+    fired_coarse     = c(TRUE, FALSE, TRUE, FALSE),
+    fired_percode    = c(FALSE, TRUE, TRUE, FALSE),
+    coarse_recipes   = c("corrections_combined", "", "police_combined", ""),
+    percode_recipes  = c("", "t29_license_wide", "police_combined", ""),
+    stringsAsFactors = FALSE
+  )
+}
+
+test_that(".measure_subset_relation() separates coverage LOST from coverage ADDED", {
+  e   <- harness_env()
+  rel <- e$.measure_subset_relation(fake_detail())
+
+  # Row 1 (coarse fired, per-code silent) is the violation; row 2 is the
+  # addition; row 3 agrees; row 4 is silent.
+  expect_false(rel$holds)
+  expect_equal(rel$n_violations, 1L)
+  expect_equal(rel$n_additions, 1L)
+  expect_equal(rel$n_coarse_fired, 2L)
+  expect_equal(rel$n_percode_fired, 2L)
+  expect_equal(rel$n_both, 1L)
+  expect_equal(rel$n_queries, 4L)
+
+  # The violation must be NAMED down to (category, government, year), not
+  # merely counted -- that is what makes it inspectable at Checkpoint R3.
+  expect_equal(rel$violations$category, "Corrections")
+  expect_equal(rel$violations$canonical_govid, "121011212191")
+  expect_equal(rel$violations$coarse_gap_years, "2011")
+  expect_equal(rel$violations$coarse_recipes, "corrections_combined")
+
+  # Inversion guard: an implementation that swapped the two directions
+  # would report the addition as a violation and vice versa.
+  expect_false("Other Taxes" %in% rel$violations$category)
+  expect_equal(rel$additions$category, "Other Taxes")
+  expect_false("Corrections" %in% rel$additions$category)
+
+  # Agreeing and silent queries belong to neither set.
+  expect_false("Police" %in% c(rel$violations$category, rel$additions$category))
+  expect_false("Fire" %in% c(rel$violations$category, rel$additions$category))
+})
+
+test_that(".measure_subset_relation() reports holds = TRUE only when nothing fires coarse-only", {
+  e <- harness_env()
+  # Drop the violating row: coarse is now genuinely a subset of per-code.
+  clean <- fake_detail()[-1L, , drop = FALSE]
+  rel   <- e$.measure_subset_relation(clean)
+
+  expect_true(rel$holds)
+  expect_equal(rel$n_violations, 0L)
+  expect_equal(nrow(rel$violations), 0L)
+  expect_equal(rel$n_additions, 1L)
+})
+
+test_that(".measure_subset_relation() validates its input rather than silently mis-reporting", {
+  e <- harness_env()
+  expect_error(e$.measure_subset_relation("not a data frame"), "must be a data frame")
+  expect_error(e$.measure_subset_relation(fake_detail()[, c("category", "canonical_govid")]),
+               "fired_coarse")
+  bad <- fake_detail()
+  bad$fired_percode[1] <- NA
+  expect_error(e$.measure_subset_relation(bad), "non-NA logicals")
+})
+
+test_that("the subset report NAMES a coarse-only firing as a violation", {
+  e   <- harness_env()
+  txt <- paste(e$.measure_format_subset_report(e$.measure_subset_relation(fake_detail())),
+               collapse = "\n")
+
+  # Stated plainly as a violation, not buried.
+  expect_match(txt, "VIOLATED")
+  expect_match(txt, "COVERAGE LOST")
+  expect_no_match(txt, "HOLDS")
+  # ...and the offending query named, so a human can go look at it.
+  expect_match(txt, "Corrections")
+  expect_match(txt, "121011212191")
+  expect_match(txt, "2011")
+  # ...and the delta explicitly flagged as a net of both directions.
+  expect_match(txt, "NET")
+})
+
+test_that("the subset report says HOLDS when coarse really is a subset", {
+  e   <- harness_env()
+  rel <- e$.measure_subset_relation(fake_detail()[-1L, , drop = FALSE])
+  txt <- paste(e$.measure_format_subset_report(rel), collapse = "\n")
+
+  expect_match(txt, "HOLDS")
+  expect_no_match(txt, "VIOLATED")
+  expect_no_match(txt, "COVERAGE LOST")
+})
+
+test_that("a REAL coarse-fires/per-code-silent query is measured and reported as a violation", {
+  skip_if_no_corpus()
+  # Broward County FY2011, Corrections: E05/F05/G05 report SOLELY as
+  # wide-era aggregate rows, which basis = "harmonized" excludes, so the
+  # whole category result is empty -- coarse's trigger. Their modern-only
+  # siblings E04/F04/G04 do not exist as codes at all before 2012, so no
+  # OTHER component can supply per-code's covering evidence and per-code
+  # is structurally unable to fire. This is the disjointness the harness
+  # exists to surface, measured end-to-end through the real git-loaded
+  # coarse arm and the live per-code arm (not a hand-built frame).
+  e   <- harness_env()
+  con <- uscogdata:::cog_open()
+  row <- e$.measure_one_query(
+    con,
+    coarse_env  = e$.measure_load_git_impl("b0df1ec"),
+    selfcov_env = e$.measure_load_git_impl("da72bf3"),
+    category = "Corrections", category_type = "expenditure",
+    govid = "121011212191", years = 2011L
+  )
+
+  expect_true(row$fired_coarse)
+  expect_false(row$fired_percode)
+  expect_equal(row$n_result_rows, 0L)
+  expect_equal(row$coarse_gap_years, "2011")
+
+  rel <- e$.measure_subset_relation(row)
+  expect_false(rel$holds)
+  expect_equal(rel$n_violations, 1L)
+  expect_equal(rel$violations$canonical_govid, "121011212191")
+
+  txt <- paste(e$.measure_format_subset_report(rel), collapse = "\n")
+  expect_match(txt, "VIOLATED")
+  expect_match(txt, "121011212191")
+})

tests/testthat/test-spending.R

@@ -98,11 +98,7 @@ test_that("cog_spending rejects invalid inputs", {
 
 test_that("cog_spending accepts a cog_gov_search result directly", {
   skip_if_no_corpus()
-  # Unanchored: utility mode matches `name` as a literal substring now, so
+  picks <- cog_gov_search("^BROWARD COUNTY$", state = "FL", type = "county")
-  # "^...$" would be searched for as those characters rather than read as
-  # anchors (uscogdata#16). Scoped by state and type, the bare name still
-  # resolves to exactly one row.
-  picks <- cog_gov_search("BROWARD COUNTY", state = "FL", type = "county")
   expect_gt(nrow(picks), 0L)
   r <- cog_spending(picks, 2020L, "Corrections")
   expect_equal(unique(r$canonical_govid), "121011212191")
@@ -248,42 +244,24 @@ test_that("basis defaults to 'harmonized' when not passed", {
 test_that("provenance carries basis + harmonization block with na_rows_excluded", {
   skip_if_no_corpus()
   with_fixture_corpus({
-    # FL state government. The harmonization block is scoped by government,
+    r <- cog_spending("121011212191", 2011:2012, "Corrections")
-    # year and flow prefix -- NOT by category -- so a Corrections query still
-    # counts every E/F/G-prefixed row the harmonized basis drops for having
-    # no harmonized_code. The three that apply here are E21/F21/G21
-    # (Education NEC, SB184-186, "discontinued_na", wide-era window ending
-    # FY2011); the other discontinued_na rulings live outside E/F/G.
-    # See docs/phase_r_harmonization_review.md § 1.3/1.4 and cog_pipeline
-    # data/harmonization_map.csv.
-    r <- cog_spending("120000226351", 2011:2012, "Corrections")
     prov <- attr(r, "provenance")
     expect_equal(prov$basis, "harmonized")
     expect_true(prov$harmonization$applied)
+    expect_true(prov$harmonization$na_rows_excluded >= 0L)
+    expect_true(prov$harmonization$na_amount_excluded >= 0)
+    # Data-verified for the v6 fixture (corpus 2026-07-22). The Task 18 map
+    # extension added E/F/G-prefix discontinued_na rulings the earlier pin's
+    # comment predated: E21/F21/G21 (Education NEC local, SB184-186,
+    # "trivial; explicit-NA, full wide-era window"). Broward's 2011 legacy
+    # partition zero-pads exactly those three codes, so this query now
+    # excludes 3 NA-harmonized rows -- all with amt = 0, hence the excluded
+    # AMOUNT stays exactly zero. (The other discontinued_na rulings -- S74,
+    # Z61, X04, X06, the debt-detail family, L24 -- remain outside the
+    # E/F/G/K prefixes.) See docs/phase_r_harmonization_review.md § 1.3/1.4
+    # and cog_pipeline data/harmonization_map.csv E21/F21/G21 rows.
     expect_equal(prov$harmonization$na_rows_excluded, 3L)
-    # $2,825,439 thousands of FY2011 E21 + F21 + G21, reported in full USD.
+    expect_equal(prov$harmonization$na_amount_excluded, 0)
-    # Pinning a non-zero amount is the point: the earlier Broward anchor's
-    # three rows were all explicit zeros, so the AMOUNT accounting was
-    # asserted only against 0 and could not have caught a bug.
-    expect_equal(prov$harmonization$na_amount_excluded, 2825439 * 1000)
-  })
-})
-
-test_that("sparsification removed the wide era's zero-pads from the exclusion count", {
-  skip_if_no_corpus()
-  with_fixture_corpus({
-    # Broward County FY2011 used to carry E21/F21/G21 rows of exactly $0 --
-    # the wide era stored every government x every code, zeros included. The
-    # published corpus no longer does (SB194, cog_pipeline#64), so there is
-    # now nothing for the harmonized basis to exclude. Absence in a
-    # dense_source year means Census published $0; it does not mean the
-    # exclusion machinery stopped working, which the FL state anchor above
-    # proves independently.
-    r <- cog_spending("121011212191", 2011:2012, "Corrections")
-    h <- attr(r, "provenance")$harmonization
-    expect_true(h$applied)
-    expect_equal(h$na_rows_excluded, 0L)
-    expect_equal(h$na_amount_excluded, 0)
   })
 })
 
@@ -325,13 +303,11 @@ test_that("provenance$series_break_refs is a populated-when-applicable character
     r <- cog_spending("121011212191", 2020L, "Corrections")
     refs <- attr(r, "provenance")$series_break_refs
     expect_type(refs, "character")
-    # No catalogued code-specific series_breaks_pq row falls inside this
+    # No catalogued series_breaks_pq row falls inside this fixture's
-    # fixture's 2011/2012/2019/2020 window for the codes this query touches
+    # 2011/2012/2019/2020 window for the codes this query touches (E04/G04)
-    # (E04/G04) -- data-verified; the mechanism itself is what's under test
+    # -- data-verified; the mechanism itself is what's under test here, via
-    # here, via a query-shaped unit test in test-views.R since the fixture
+    # a query-shaped unit test in test-views.R since the fixture has no
-    # has no positive case to pin against. Corpus-wide ("ALL") entries never
+    # positive case to pin against.
-    # appear in this field by construction -- they travel in
-    # corpus_break_refs; see test-corpus-breaks.R.
     expect_equal(refs, character(0))
   })
 })

tests/testthat/test-views.R

@@ -9,9 +9,7 @@ test_that("all expected views register on session open", {
   expected <- c(
     "long", "spending_long", "revenue_long",
     "canonical_fips_xwalk", "summary_categories",
-    "spending_annotated", "revenue_annotated",
+    "spending_annotated", "revenue_annotated"
-    "ig_long", "ig_annotated",
-    "ig_long_harmonized", "ig_annotated_harmonized"
   )
   expect_true(all(expected %in% views$table_name))
 })
@@ -112,78 +110,12 @@ test_that("inst/sql/22- and 23- harmonized views enforce every WHERE predicate (
   expect_equal(rev$amt, 225)
 })
 
-test_that("inst/sql/24- and 25- IG views retain aggregates, COALESCE NULL harmonized_code, and exclude the L-- family total (real SQL text, synthetic parquet)", {
-  # ig_long / ig_long_harmonized have the subtlest predicates in the package:
-  # a deliberately ABSENT `NOT is_aggregate` (unlike every other *_long view),
-  # and COALESCE(harmonized_code, item_code) instead of a plain
-  # `harmonized_code IS NOT NULL` filter. The only end-to-end guard on this
-  # today is bound to AL state / 2011 / Education K-12, where M12 happens to
-  # be the sole IG code present -- regenerate the fixture without that one
-  # row and the guard would die silently while staying green. As with the
-  # 22-/23- test above, this reads the real inst/sql/24-/25- text off disk
-  # and executes it against a synthetic hive-partitioned parquet tree, so a
-  # regression in either predicate changes which rows survive.
-  skip_if_no_corpus()
-
-  tmp <- withr::local_tempdir()
-  part_dir <- file.path(tmp, "data", "long", "year=2004")
-  dir.create(part_dir, recursive = TRUE)
-  part_path <- file.path(part_dir, "part-0.parquet")
-
-  write_con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
-  DBI::dbExecute(write_con, sprintf("
-    COPY (
-      SELECT * FROM (VALUES
-        ('ig-A', 'M04', 100,   false, 'M04'), -- control: passes through as-is
-        ('ig-B', 'M38', 50,    false, 'M36'), -- fold control: real SB012 rule, renamed to M36 under harmonized basis
-        ('ig-C', 'M47', 99999, true,  NULL),  -- legacy aggregate, NO harmonized_code: must survive BOTH views
-        ('ig-D', 'L--', 55555, false, 'L--'), -- family total: excluded from BOTH views
-        ('ig-E', 'T29', 44444, false, 'T29')  -- wrong prefix (revenue, not M/L): excluded from BOTH views
-      ) AS t(canonical_govid, item_code, amt, is_aggregate, harmonized_code)
-    ) TO %s (FORMAT PARQUET)
-  ", uscogdata:::.sql_lit_chr(part_path)))
-
-  sql_dir <- system.file("sql", package = "uscogdata")
-  .read_view_sql <- function(filename) {
-    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
-  }
-
-  con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
-  DBI::dbExecute(con, .read_view_sql("10-long.sql"))
-  DBI::dbExecute(con, .read_view_sql("24-ig_long.sql"))
-  DBI::dbExecute(con, .read_view_sql("25-ig_long_harmonized.sql"))
-
-  raw <- DBI::dbGetQuery(con,
-    "SELECT item_code, SUM(amt) AS amt FROM ig_long
-     GROUP BY item_code ORDER BY item_code"
-  )
-  # L-- (family total) and T29 (wrong prefix) are gone; the aggregate row
-  # M47 survives -- proof `NOT is_aggregate` is absent from ig_long.
-  expect_equal(raw$item_code, c("M04", "M38", "M47"))
-  expect_equal(raw$amt, c(100, 50, 99999))
-
-  harmonized <- DBI::dbGetQuery(con,
-    "SELECT item_code, SUM(amt) AS amt FROM ig_long_harmonized
-     GROUP BY item_code ORDER BY item_code"
-  )
-  # M38 folds to M36 (real harmonized_code present); M47 keeps its raw code
-  # via COALESCE(NULL, 'M47') -- proof the aggregate row is NOT dropped by
-  # a plain `harmonized_code IS NOT NULL` filter. L-- and T29 stay excluded.
-  expect_equal(harmonized$item_code, c("M04", "M36", "M47"))
-  expect_equal(harmonized$amt, c(100, 50, 99999))
-})
-
 test_that(".build_series_break_refs matches fin_code + break_year window", {
-  # No CODE-SPECIFIC series_breaks_pq row falls inside the bundled fixture's
+  # No series_breaks_pq row falls inside the bundled fixture's 2011-2020
-  # 2011-2020 window (data-verified; see the "series_break_refs" test in
+  # window (data-verified; see the "series_break_refs" test in
   # test-spending.R), so this proves the matching logic itself against the
   # live view + a synthetic year window that DOES hit a cataloged break
-  # (SB075, fin_code E62, break_year 2005). The corpus-wide entries are a
+  # (SB075, fin_code E62, break_year 2005).
-  # separate path with its own coverage -- SB194 does sit at 2012, inside
-  # the fixture window; see test-corpus-breaks.R.
   skip_if_no_corpus()
   con <- cog_open()
   on.exit(cog_close())
@@ -220,108 +152,6 @@ test_that("schema v5 harmonization views register when the corpus supports them"
   expect_true(all(expected_v5 %in% views$table_name))
 })
 
-test_that(".harmonization_view_files guard is necessary: registration against a v4-shaped corpus (no harmonized_code column at all) succeeds only because the harmonized views are skipped", {
-  # with_doctored_schema_version() (used elsewhere in this suite) only
-  # rewrites manifest.json's schema_version -- the underlying `long` parquet
-  # is still the bundled v6 fixture, which DOES have a harmonized_code
-  # column, so it only proves the skip *happens*, not that it is *required*.
-  # This test builds a genuinely v4-shaped corpus: `long` has no
-  # harmonized_code column at all, matching a real pre-Phase-R2 publish
-  # tree, and then shows two things: (1) the real .register_views(), gated
-  # on manifest$schema_version, registers cleanly against it; (2) the exact
-  # SQL text of a gated file (25-ig_long_harmonized.sql), executed directly
-  # against the same corpus without the gate, fails -- proving the gate is
-  # load-bearing, not incidental.
-  tmp <- withr::local_tempdir()
-  part_dir <- file.path(tmp, "data", "long", "year=2004")
-  dir.create(part_dir, recursive = TRUE)
-  part_path <- file.path(part_dir, "part-0.parquet")
-
-  write_con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
-  DBI::dbExecute(write_con, sprintf("
-    COPY (
-      SELECT * FROM (VALUES
-        ('gov-1', 'E36', 100, false, 500000, 2020)
-      ) AS t(canonical_govid, item_code, amt, is_aggregate, population, popyear)
-    ) TO %s (FORMAT PARQUET)
-  ", uscogdata:::.sql_lit_chr(part_path)))
-
-  xwalk_path <- file.path(tmp, "data", "canonical_fips_xwalk.parquet")
-  DBI::dbExecute(write_con, sprintf("
-    COPY (
-      SELECT * FROM (VALUES
-        ('gov-1', 'Test Gov', 1, 'County', '01', '001', NULL, 500000)
-      ) AS t(canonical_govid, gov_name, govs_type, type_label, fips_state,
-             fips_county, fips_place, population_acs)
-    ) TO %s (FORMAT PARQUET)
-  ", uscogdata:::.sql_lit_chr(xwalk_path)))
-
-  cats_path <- file.path(tmp, "data", "summary_categories.parquet")
-  DBI::dbExecute(write_con, sprintf("
-    COPY (
-      SELECT * FROM (VALUES
-        ('E36', 'Test Category', 'expenditure', 'direct', NULL)
-      ) AS t(item_code, category, category_type, spend_subtype, revenue_subtype)
-    ) TO %s (FORMAT PARQUET)
-  ", uscogdata:::.sql_lit_chr(cats_path)))
-
-  # Confirm the synthetic `long` genuinely lacks harmonized_code (not just
-  # NULL values -- the column itself must be absent) before trusting the
-  # rest of this test.
-  cols <- DBI::dbGetQuery(write_con, sprintf(
-    "DESCRIBE SELECT * FROM read_parquet(%s)", uscogdata:::.sql_lit_chr(part_path)
-  ))$column_name
-  expect_false("harmonized_code" %in% cols)
-
-  url <- paste0(tmp, "/")
-
-  # (1) Full .register_views() against this v4-shaped corpus must succeed --
-  # this is the behavior the guard exists to protect.
-  con <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
-  expect_no_error(
-    uscogdata:::.register_views(con, url, manifest = list(schema_version = 4L))
-  )
-  views <- DBI::dbGetQuery(con,
-    "SELECT table_name FROM information_schema.tables
-     WHERE table_schema = 'main' AND table_type = 'VIEW'")$table_name
-  expect_true(all(c("ig_long", "ig_annotated", "spending_annotated") %in% views))
-  expect_false(any(c("ig_long_harmonized", "ig_annotated_harmonized",
-                      "spending_long_harmonized") %in% views))
-
-  # (2) Prove the gate is load-bearing: the exact SQL text of the skipped
-  # file, executed directly (bypassing .register_views()'s schema_version
-  # check) against the SAME corpus, fails because it references
-  # long.harmonized_code, a column this corpus's `long` does not have.
-  sql_dir <- system.file("sql", package = "uscogdata")
-  .read_view_sql <- function(filename) {
-    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", url, txt, fixed = FALSE)
-  }
-  con2 <- DBI::dbConnect(duckdb::duckdb())
-  on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
-  DBI::dbExecute(con2, .read_view_sql("10-long.sql"))
-  expect_error(DBI::dbExecute(con2, .read_view_sql("25-ig_long_harmonized.sql")))
-
-  # Reconciling this test with the C2 guard (expenditure-concept review):
-  # `ig_annotated`/`spending_annotated` registering cleanly above proves
-  # only that CREATE VIEW binds against a `summary_categories` with no M/L
-  # rows at all (this synthetic corpus's own summary_categories has a
-  # single E36 row, see the COPY above) -- a LEFT JOIN never fails to
-  # resolve regardless of what the joined-to table contains. It does NOT
-  # mean querying expenditure_concept = "total" against this shape is safe:
-  # exactly this corpus (schema_version reported as supported, but
-  # summary_categories predates the M/L rows cog_pipeline PR #59 added) is
-  # what .require_ig_categories() exists to catch at the *verb* level,
-  # since PR #59 shipped those rows with no schema_version bump. Confirm
-  # the new runtime guard actually fires against this same `con`.
-  expect_error(
-    uscogdata:::.require_ig_categories(con),
-    class = "uscogdata_ig_categories_unsupported"
-  )
-})
-
 test_that("spending_long filters to E/F/G/K prefixes and excludes aggregates", {
   skip_if_no_corpus()
   con <- cog_open()

vignettes/population-denominators.Rmd

@@ -13,8 +13,6 @@ knitr::opts_chunk$set(eval = FALSE, collapse = TRUE, comment = "#>")
 
 # Why per-year population matters
 
-A note on units first, since every figure below is a rate: the numerator is in **full US dollars**. The raw Census files report **thousands of dollars** and the corpus keeps them that way in its own `amt` column, but `cog_spending()` and `cog_revenue()` multiply by 1000 on the way out, so `amt_per_capita_nominal` is already dollars per person. Do not scale it again.
-
 Per-capita finance numbers divide each year's spending or revenue by a population denominator. The choice of denominator is a research decision, not an implementation detail: a 24-year corpus paired with a single 5-year ACS estimate produces biased per-capita values whose magnitude scales with each government's population change.
 
 `uscogdata` defaults to the **Census F-33 population value Census itself uses to compute its published per-capita tables.** That value is recorded on every COG row as `population`, with `popyear` indicating the vintage. For a city that grew from 200,000 to 300,000 between 2000 and 2023, this default reproduces the per-capita value Census published. A static ACS denominator would have understated 2000 per-capita by ~33%.

vignettes/total-spending.Rmd

@@ -1,236 +0,0 @@
----
-title: "Total spending: Direct, Total, and when each is right"
-output: rmarkdown::html_vignette
-vignette: >
-  %\VignetteIndexEntry{Total spending: Direct, Total, and when each is right}
-  %\VignetteEngine{knitr::rmarkdown}
-  %\VignetteEncoding{UTF-8}
----
-
-```{r setup, include = FALSE}
-knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
-```
-
-# Two questions that sound the same but aren't
-
-"Total spending" means two different things depending on whether the question
-is about one government or several:
-
-1. **"What did my county spend in total, a decade ago vs today?"** — one
-   government, tracked over time. Either `direct` or `total` spending answers
-   this correctly, as long as the same concept is used for both years.
-2. **"How do all the counties in my state compare, a decade ago vs today,
-   against the neighboring state?"** — several governments, summed together.
-   Here only `direct` gives the right answer; summing `total` across
-   governments double-counts money that passes between them.
-
-`cog_spending()`'s `expenditure_concept` argument (`"direct"` or `"total"`)
-controls which of these a query answers. This vignette walks through both
-questions with code that actually runs against the package's bundled fixture
-corpus, then explains why the second question refuses `"total"` outright.
-
-Before any of the numbers below: every amount column here — `amt_nominal`,
-`amt_real`, and their `amt_per_capita_*` counterparts — is in **full US
-dollars**. The raw Census files report **thousands of dollars** and the
-corpus preserves that in its own `amt` column, but the verbs multiply by 1000
-on the way out. So `amt_nominal = 1317000` means $1.317 million, not $1.317
-billion. Do not scale it again.
-
-```{r}
-library(uscogdata)
-
-# Point at the bundled offline fixture (years 2011, 2012, 2019, 2020, all 50
-# states) so this vignette knits without network access. In real use,
-# USCOGDATA_URL is instead set to the published corpus URL -- see README.md.
-Sys.setenv(USCOGDATA_URL = paste0(
-  system.file("extdata/fixture_corpus", package = "uscogdata"), "/"
-))
-```
-
-The fixture doesn't carry 2017 or the present year, so the examples below use
-the closest years it does ship -- **2012 and 2020** -- in place of "2017 vs
-today" / "ten years ago vs today". Point `USCOGDATA_URL` at the published
-corpus and swap in real years; the mechanics are identical.
-
-# Archetype 1: one government's own trend
-
-For a single government, `total` is a legitimate way to describe "everything
-this government spent, including money it handed to other governments to
-spend on its behalf":
-
-```{r}
-al_total <- cog_spending(
-  "010000226085",                 # Alabama, the state government
-  years = c(2012, 2020),
-  category = "Highways",
-  expenditure_concept = "total"
-)
-al_total
-```
-
-The `intergovernmental` rows are what `"total"` adds on top of `"direct"`
-(`capital` + `operations`): Alabama's own payments out to counties and
-cities for highway work. Because this query only ever concerns Alabama,
-including that piece is safe -- there's no other government's number it
-could be double-counted against.
-
-`"direct"` (the default) answers the same trend question just as validly:
-
-```{r}
-al_direct <- cog_spending(
-  "010000226085", years = c(2012, 2020), category = "Highways"
-  # expenditure_concept = "direct" is the default; shown here for contrast
-)
-al_direct
-```
-
-Both are internally consistent series. What breaks the comparison is
-**switching concepts between the two years being compared** -- e.g. `direct`
-for 2012 and `total` for 2020 -- which manufactures a trend that isn't
-really there. Pick one concept for a given question and hold it fixed across
-every year in the series.
-
-# Archetype 2: a cross-government rollup
-
-`cog_geographic_rollup()` sums spending across state/county/city layers for
-a place. Its default -- and, as shown below, its *only* accepted value for
-`expenditure_concept` -- is `"direct"`:
-
-```{r}
-fl_rollup <- cog_geographic_rollup(
-  govids = list(
-    state  = "120000226351",                   # Florida
-    county = c("121011212191", "121099101897") # Broward + Palm Beach
-  ),
-  category = "Highways",
-  years = c(2012, 2020)
-)
-fl_rollup
-```
-
-For the neighboring state, the comparison is a single government, so it's a
-plain `cog_spending()` call rather than a rollup:
-
-```{r}
-ga_state <- cog_spending(
-  "130000226087", years = c(2012, 2020), category = "Highways" # Georgia
-)
-ga_state
-```
-
-Now the same rollup, but asking for `expenditure_concept = "total"`:
-
-```{r, error = TRUE}
-cog_geographic_rollup(
-  govids = list(state = "120000226351", county = "121011212191"),
-  category = "Highways",
-  years = 2020,
-  expenditure_concept = "total"
-)
-```
-
-`cog_geographic_rollup()` (and `cog_peer_compare()`, for the same reason)
-refuses `"total"` outright rather than silently returning an inflated
-number. The next section is why.
-
-# The mechanism
-
-Suppose Alabama gives a county $10M toward a highway project. That $10M
-shows up **twice** in the underlying corpus:
-
-- Once on Alabama's own record, coded `M44` ("to local governments,
-  Highways") -- Alabama's intergovernmental leg.
-- Again on the county's record, coded `E44` / `F44` ("Highways, current
-  operations" / "capital outlay") -- the county's direct spending, because
-  the county is the government that actually lets the contract and pays the
-  paving crew.
-
-`direct` (item codes `E`/`F`/`G`) only ever counts the second of those --
-the government that actually did the spending. `total` (Direct plus the
-`M`/`L` intergovernmental legs) counts the first one *as well*, which is
-exactly right for describing Alabama's own budget: Alabama's `total`
-genuinely includes the $10M it committed to highways, whether it built the
-road itself or paid the county to. But sum `total` across Alabama **and**
-the county, and that $10M is counted twice -- once as Alabama's payment out,
-once as the county's spending in -- reporting $20M of highway work for $10M
-actually spent.
-
-This is exactly the shape of query `cog_geographic_rollup()` exists to run
-(summing across layers of government), so it refuses `"total"` rather than
-silently overstating every multi-layer figure it produces.
-
-# How big is the risk in practice
-
-Intergovernmental transfers aren't evenly distributed by government type.
-Measured against the bundled fixture corpus (all 50 states, each of its
-four years -- 2011, 2012, 2019, 2020), intergovernmental spending as a
-share of a government's own Direct spending is:
-
-| Government type | Intergovernmental / Direct |
-|---|---|
-| State  | 16.7%-48.4% (varies by year; 24.0% pooled across all four) |
-| County | 3.4%-5.1% (varies by year) |
-| City   | 2.6%-3.1% (varies by year) |
-
-So the Direct/Total choice matters overwhelmingly for **state** governments
--- a state's Total genuinely differs from its Direct by a meaningful margin,
-while for a county or city the two are close. The state range is also far
-wider than a single flat figure would suggest: legacy wide-era years (2011:
-48.4%) carry proportionally more intergovernmental spending than the modern
-era (2019-2020: 16.7%-17.0%), so a state's Direct/Total gap can be nearly
-3x larger a decade earlier than it is today. That's also why the mistake
-this vignette warns about is easy to make unnoticed at the county/city level
-and costly at the state level: rolling up every government in a state using
-`total` instead of `direct` overstates the true figure -- measured at 7.6%
-for Alabama in FY2019, and 11.6% nationally.
-
-# Why Total = Direct + M + L, not Direct + M
-
-It's tempting to assume `total` only needs to add `M`. But `M` and `L` are
-both money the queried government itself pays **out** -- they're not two
-different accounts of a receiving government's revenue. `M` is what it
-pays to other **local** governments (e.g. a county paying a city for a
-shared paving contract); `L` is what it pays **up** to its **state**
-government (e.g. a county's contribution to a state-administered program).
-A local government's Total genuinely includes both legs, because both are
-its own spending, just routed to a different kind of recipient. On the
-bundled fixture corpus (all 50 states, 2011/2012/2019/2020), `L` is 0 for
-state governments (a state has no "payments to the state government" leg of
-its own) but is 43%-51% the size of `M` for counties (varies by year) and
-144%-189% the size of `M` for cities (varies by year; 166% pooled across
-all four) -- so a `total` that omitted `L` would silently undercount Total
-specifically for local governments, and for cities `L` is often the
-*larger* of the two legs.
-`cog_spending(expenditure_concept = "total")` includes both legs (excluding
-the `L--` family-total rollup row, which would double-count its own
-components).
-
-# Composition rules
-
-- `expenditure_concept` (whose spending counts -- Direct vs Direct plus
-  intergovernmental) is **orthogonal** to `basis` (which vintage of the
-  item-code space a query resolves against -- `"harmonized"` vs `"raw"`).
-  They combine freely: `expenditure_concept = "total", basis = "raw"` is a
-  valid, meaningful query, and so is every other pairing.
-- `expenditure_concept = "total"` is **mutually exclusive** with `recipe`: a
-  recipe already defines its own component codes (some recipes have their
-  own matching intergovernmental counterpart recipe instead -- see
-  `cog_recipes()` and the "firing suggestion" notes surfaced in
-  `cog_spending()`'s provenance), so layering a second, generic `total`
-  union on top of a recipe query has no well-defined meaning. Passing both
-  together aborts with an error naming the conflict.
-- `expenditure_concept` is a **spending-only** concept: `cog_revenue()`
-  doesn't expose it (revenue's own intergovernmental codes are a different
-  axis -- see `?cog_revenue`).
-
-# Summary
-
-- Comparing one government to itself over time: `"direct"` or `"total"`
-  both work -- pick one and hold it fixed across every year compared.
-- Comparing or summing across governments -- counties within a state, a
-  state against its neighbor, cities against counties: use `"direct"`.
-  `cog_geographic_rollup()` and `cog_peer_compare()` enforce this by
-  refusing `"total"`.
-- `"total"` = Direct (`E`/`F`/`G`) + intergovernmental (`M` to local
-  governments + `L` to the state government, excluding the `L--`
-  family-total row).