Compare commits

...
11 Commits
Author SHA1 Message Date
jaredandClaude Opus 5.5 ba57f9cb27 docs: record session
Mirror to GitHub / mirror (push) Successful in 8s
R-CMD-check / check (push) Successful in 3m21s
Journal entries for the move out of Nextcloud and the 2026-09-09 CI fix,
and the regenerated status board.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:15:56 -04:00
jaredandClaude Opus 5.5 63b421aea2 docs: point CLAUDE.md at the pipeline clone's paths
The repo moved out of Nextcloud today, and its sibling pipeline clone is
now census_of_governments_finance_pipeline, not cog_explorer/cog_pipeline.
The phase N task plan had already moved to docs/archive/ on 2026-07-23
(pipeline acfacf2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:15:00 -04:00
jaredandClaude Sonnet 5 d1165a507e fix(ci): find the google-chrome apt source by content, not filename
Mirror to GitHub / mirror (push) Successful in 7s
R-CMD-check / check (push) Successful in 4m35s
The prior commit's `sudo rm -f /etc/apt/sources.list.d/google-chrome.list`
guessed a filename that doesn't match what's actually on GitHub's
ubuntu-latest image: the step reported success (rm -f swallows
"no such file"), but setup-r@v2's subsequent apt-get update still hit
the same dl.google.com Hash Sum mismatch. Locate the offending source
by grepping for its URL instead, and fail this step explicitly if a
match survives removal, so a future runner-image change is loud here
rather than confusing inside setup-r@v2.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 13:53:40 -04:00
jaredandClaude Sonnet 5 d83301bdbd fix(ci): drop unused google-chrome apt source before setup-r
Mirror to GitHub / mirror (push) Successful in 6s
R-CMD-check / check (push) Successful in 3m30s
r-lib/actions/setup-r@v2 runs `sudo apt-get update` before installing
R, which fails outright if any configured apt source is broken - even
one we never touch. The google-chrome source baked into GitHub's
ubuntu-latest image intermittently serves a stale Packages.gz that
doesn't match its own Release file's hash (a Google CDN sync race),
which took down every Linux leg of the R-CMD-check matrix on
2026-09-09 regardless of R version. Removing the source we don't need
makes this failure class impossible here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 13:47:42 -04:00
jaredandClaude Sonnet 5 9adb921170 docs: record session (#36)
Mirror to GitHub / mirror (push) Successful in 7s
R-CMD-check / check (push) Successful in 3m37s
Journal entries for the #33/#34 split and the #36 coverage counter
(two bugs caught before merge in the latter), plus the vignette
obligation filed as #72. Status board regenerated and republished.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 13:38:25 -04:00
jaredandClaude Sonnet 5 9f5cd988b2 docs(suggestions): restore dropped M/L-counterpart rationale (roborev job 224)
Mirror to GitHub / mirror (push) Successful in 7s
R-CMD-check / check (push) Successful in 3m46s
The #33 decomposition (0c7c7eb) silently dropped four roxygen lines
explaining why .attach_ig_counterparts()'s second flow-family check is
needed: condition 1 alone doesn't block ig_federal_b47_wide under
cog_revenue(), since its own "B" IS inside revenue's own flow_prefixes.
Restored them, plus the backticks around "B" that were also dropped as
an unstated formatting change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 13:35:18 -04:00
jared 9eaa759ccb Merge pull request 'feat(coverage): n_units_collected separates sampling from real zeros (#36)' (#71) from fix/coverage-collected-36 into main
Mirror to GitHub / mirror (push) Successful in 8s
R-CMD-check / check (push) Successful in 3m57s
2026-09-09 11:51:06 -04:00
jaredandClaude Sonnet 5 b41d5ee2aa feat(coverage): n_units_collected separates sampling from real zeros (#36)
R-CMD-check / check (push) Successful in 3m39s
R-CMD-check / check (pull_request) Successful in 3m42s
provenance$coverage's n_units_reporting is category-conditional: it
counts governments with rows for the SPECIFIC requested category, which
conflates two different things -- a government never collected that
year (sampling), and one collected but genuinely spending nothing in
that category (a real zero). FY2012 Georgia Police is the motivating
case from the issue: a complete census year reads as a 69% "response
rate" because most of the gap is cities that contract policing to the
county sheriff, not non-response.

Adds a second counter, n_units_collected: how many of the caller's
expected cohort appear in the corpus that year for ANY category.
n_units_collected / n_units_expected is the true collection rate;
n_units_reporting / n_units_collected is category participation among
collected units. cog_geographic_rollup() and cog_peer_compare() both
carry it; cog_explain() prints it alongside n_units_reporting.

Two real bugs caught and fixed while finishing this (both against the
already-written, previously-uncommitted draft):

- .coverage_table()'s candidate list for the collection query was
  derived from the category-filtered result rows, not the caller's
  full expected cohort. A government with zero rows in the requested
  category across every requested year never appears in that result,
  so it was silently excluded from n_units_collected too -- collapsing
  the new counter back to the old, broken one for exactly the
  governments it exists to count. Fixed by threading an explicit
  `expected_ids` (all_govids / peer_govids) through instead.
- The collection query hardcoded long_view = "spending_long_harmonized",
  which does not exist on a corpus with schema_version < 5 (R/basis.R
  resolves basis = "raw" there; R/views.R only registers the
  harmonized views on v5+). cog_geographic_rollup()/cog_peer_compare()
  would hard-error on a corpus vintage the package otherwise explicitly
  supports. Fixed by deriving long_view from the basis cog_spending()
  actually resolved (prov$basis) via the existing .select_long_view()
  helper, matching how every other basis-aware query in the package
  already does this.

Also: cog_explain()'s general "complete census only in years ending in
2 or 7" footnote was gated on the OLD counter's absence, making it
permanently unreachable now that both callers always supply the new
one -- ungated it, since the explanation is orthogonal to which
counter set is present. Dropped a dead conditional branch, fixed two
stale roxygen blocks in R/peers.R/R/rollup.R still describing the old
two-counter model, fixed the same staleness in README.md, and switched
two `uscogdata:::` self-references to the package's own convention of
calling internal helpers unqualified.

1101 tests pass (2 skipped live-corpus), including new direct
regression tests for both bugs above (one exercising a government
collected-but-absent from a category result, one running the full
rollup/peer-compare path against a doctored schema_version 4 corpus).

Reviewed by an independent code-reviewer pass (1 HIGH, 1 MEDIUM, 3 LOW
-- all addressed above).

Closes #36.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 11:46:17 -04:00
jaredandClaude Sonnet 5 d0d724c4c8 fix(suggestions): guard empty candidates and restore year typing in .query_covered_years()
Mirror to GitHub / mirror (push) Successful in 6s
R-CMD-check / check (push) Successful in 3m15s
Addresses roborev jobs 205/208 (reviewing pre-merge draft/original commits
of #33/#34, now split and merged as #69/#70):

- Restore `res$year <- as.integer(res$year)` in .query_covered_years(),
  dropped during the #33 rebuild as apparently-dead code. Its own roxygen
  promises an integer `year` column; without the coercion the
  implementation no longer matches that documented contract.
- Guard .query_covered_years() against empty `candidates`, mirroring the
  existing `gap_years` guard. Not reachable via .build_suggestions() today
  (candidates is checked non-empty before this is called), but the
  extracted helper is independently callable and previously built a
  malformed `WHERE r.recipe_id IN ()` clause for a hypothetical direct
  caller with no candidates.
- Add a direct fixture-only unit test of .query_candidate_recipes()'s
  category_type filtering (#34) in both flow directions -- queries only
  summary_categories/harmonization_recipes metadata, so it runs without
  skip_if_no_corpus(), unlike the two existing end-to-end tests.

1080 tests pass (2 skipped live-corpus).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 11:28:01 -04:00
jared 56f610ea3f Merge pull request 'fix(suggestions): scope candidate recipes by category_type (#34)' (#70) from fix/category-type-34 into main
Mirror to GitHub / mirror (push) Successful in 6s
R-CMD-check / check (push) Successful in 3m36s
2026-09-09 11:18:52 -04:00
jaredandClaude Sonnet 5 392643bd74 fix(suggestions): scope candidate recipes by category_type (#34)
R-CMD-check / check (push) Successful in 4m15s
R-CMD-check / check (pull_request) Successful in 4m25s
.query_candidate_recipes() (extracted in #33) now filters candidates by
category_type ('expenditure' vs 'revenue'), derived from the calling
verb's own flow_prefixes (E/F/G -> 'expenditure', else 'revenue').

Without this, a category shared across both flow families in
summary_categories leaked cross-family recipes: cog_revenue(category =
"Corrections") surfaced the expenditure-only corrections_combined recipe
(E04/E05) merely because "Corrections" is also a spending category name,
and cog_spending(category = "IG Federal") surfaced the revenue-only
ig_federal_b47_wide recipe. Both are wrong: following either hint would
attribute dollars to the wrong flow, or (IG Federal) fire the
coverage-gap machinery for a category the calling verb structurally
cannot report on at all.

Updates the two tests this changes the expected behavior of:
- "a mis-scoped cog_spending() call never attaches an M/L counterpart to
  a revenue-flavored recipe" (test-expenditure-concept.R): IG Federal is
  revenue-only, so a spending call now finds zero candidates outright
  rather than firing the suggestion and then blocking its M/L
  counterpart as a second-order check.
- "cog_revenue never suggests expenditure-only recipes"
  (test-recipes.R, was "I1: ... never fabricates suppressed dollars"):
  corrections_combined is expenditure-only, so a revenue call now never
  considers it as a candidate, rather than considering it and reporting
  zero suppressed dollars.

All 1078 tests pass (2 skipped live-corpus), measured devtools::test()
against this commit in a clean worktree stacked on the #33 refactor.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 11:11:16 -04:00
16 changed files with 516 additions and 139 deletions
+23
View File
@@ -44,6 +44,29 @@ jobs:
- uses: r-lib/actions/setup-pandoc@v2
- name: Drop the unused google-chrome apt source
if: runner.os == 'Linux'
run: |
set -x
grep -rl 'dl\.google\.com' /etc/apt/sources.list.d/ /etc/apt/sources.list 2>/dev/null || true
sudo sh -c "grep -rl 'dl\.google\.com' /etc/apt/sources.list.d/ /etc/apt/sources.list 2>/dev/null | xargs -r rm -f"
grep -rl 'dl\.google\.com' /etc/apt/sources.list.d/ /etc/apt/sources.list 2>/dev/null && exit 1 || true
# setup-r@v2 runs `sudo apt-get update` before installing R, and that
# command fails outright if ANY configured apt source is broken --
# even one we never use. The google-chrome source baked into GitHub's
# ubuntu-latest image intermittently serves a stale Packages.gz that
# doesn't match its own Release file's hash (a Google CDN sync race,
# not anything about R or this repo), which took down every ubuntu
# leg of this matrix on 2026-09-09. `rm -f` on a guessed filename
# (google-chrome.list) reported success but removed nothing -- the
# file it actually is on this image apparently doesn't match that
# name, since the source kept showing up in setup-r's `apt-get
# update` afterward. Find-by-content instead of guessing the
# filename, and fail loudly here (before setup-r even runs) if a
# matching source is still present, so a future runner-image change
# surfaces as a clear failure in this step instead of a confusing
# one in setup-r.
- uses: r-lib/actions/setup-r@v2
with:
r-version: ${{ matrix.config.r }}
+4 -3
View File
@@ -5,11 +5,12 @@
R package providing a curated reader API for the Civilytics US Census of
Governments finance corpus. Reads Hive-partitioned parquet + `manifest.json`
published by `cog_pipeline` via DuckDB (local path or remote URL). This is a
standalone Gitea repo, sibling to `cog_explorer/cog_pipeline/`.
standalone Gitea repo, sibling to the pipeline repo,
`Civilytics/census_of_governments_finance_pipeline`, cloned beside this one.
**Gitea remote:** `gitea.civilytics.org/Civilytics/uscogdata`
**Full implementation plan:** `../cog_pipeline/docs/plan_phase_n_tasks.md` (Tasks 2.1–2.8 + Phase 3)
**Reader contract spec:** `../cog_pipeline/docs/reader-specification.md`
**Full implementation plan (archived):** `../census_of_governments_finance_pipeline/docs/archive/plan_phase_n_tasks.md` (Tasks 2.1–2.8 + Phase 3)
**Reader contract spec:** `../census_of_governments_finance_pipeline/docs/reader-specification.md`
## Architecture
+76 -6
View File
@@ -84,22 +84,92 @@
#' `n_units_reporting = 0`, which is precisely the disclosure a silently
#' missing year fails to make.
#'
#' `n_units_reporting` describes the result the caller actually received, so
#' under `coverage = "consistent"` it reports the balanced count. `is_census_year`
#' is a statement about the SURVEY CALENDAR, never a claim of completeness:
#' FY1967 is a census year in which only 97 of Wisconsin's 608 cities report.
#' `n_units_reporting` is the number that tells the truth.
#' Three counters are returned, each answering a different question:
#'
#' * `n_units_expected` -- the universe the caller named (govids passed in,
#' or peers for cog_peer_compare). "How many governments did you ask
#' about?"
#' * `n_units_collected` -- how many of those appear in the corpus at all
#' that year, in ANY category. This is a statement about survey collection,
#' independent of what was asked for: "of the governments you named, how
#' many did Census actually collect data from this year?" It separates
#' sampling (not collected) from real zeros (collected but spends nothing
#' in your category).
#' * `n_units_reporting` -- how many of those appear with rows for the
#' SPECIFIC category you requested. This is always <= n_units_collected:
#' a government can be collected but have no rows for "Police" because it
#' contracts policing to the county sheriff, not because it wasn't
#' surveyed.
#'
#' `n_units_reporting` therefore conflates two very different things: a unit
#' that was not collected (sampling) and a unit that was collected but spends
#' nothing in that category. The ratio n_units_collected / n_units_expected is
#' the true collection rate; n_units_reporting / n_units_collected measures
#' category participation among collected units.
#'
#' `is_census_year` is a statement about the SURVEY CALENDAR, never a claim of
#' completeness: FY1967 is a census year in which only 97 of Wisconsin's 608
#' cities report. The counters are what tell the truth.
#'
#' @param con Active DuckDB connection (used to look up n_units_collected).
#' @param long_view The verb's own long view, used for the collection query;
#' NULL skips the lookup and leaves n_units_collected as NA_integer_.
#' @param expected_ids The full EXPECTED cohort (govids the caller named),
#' used as the candidate list for the collection query. Required alongside
#' `con`/`long_view` for a correct count -- see the note below on why it
#' must not be derived from `result`/`rows`. `NULL`, or non-`NULL` but
#' empty after dropping `NA`/`""` entries, skips the lookup and leaves
#' n_units_collected as NA_integer_.
#' @noRd
.coverage_table <- function(result, years, n_expected,
id_col = "canonical_govid", rows = NULL) {
id_col = "canonical_govid", rows = NULL,
con = NULL, long_view = NULL,
expected_ids = NULL) {
years <- sort(unique(as.integer(years)))
src <- if (is.null(rows)) result else rows
reporting <- vapply(years, function(y) {
ids <- src[[id_col]][as.integer(src$year) == y]
length(unique(ids[!is.na(ids)]))
}, integer(1))
# n_units_collected: count EXPECTED cohort members present in the corpus
# for ANY category that year, not just the requested one. This separates
# sampling (not collected at all) from real zeros (collected but no rows
# for this category). Only computed when a connection, long_view, AND
# expected_ids are all provided; otherwise NA_integer_.
#
# The candidate list MUST be expected_ids, not derived from `result`/
# `rows`: a government with zero rows in the requested category across
# EVERY requested year never appears in `result` at all, so deriving
# candidates from it would silently exclude exactly the "collected but
# real zero" governments this counter exists to count -- collapsing
# n_units_collected back to n_units_reporting for precisely the case #36
# was filed over.
if (!is.null(con) && !is.null(long_view) && length(expected_ids) > 0L) {
cohort_chr <- .sql_lit_chr(unique(expected_ids[!is.na(expected_ids) &
nzchar(expected_ids)]))
years_lit <- paste(years, collapse = ",")
collected_q <- sprintf(
"SELECT year, COUNT(DISTINCT canonical_govid) AS n
FROM %s
WHERE canonical_govid IN (%s)
AND year IN (%s)
GROUP BY year",
long_view, cohort_chr, years_lit
)
collected_df <- DBI::dbGetQuery(con, collected_q)
collected_map <- setNames(collected_df$n, as.integer(collected_df$year))
collected <- vapply(years, function(y) {
val <- collected_map[as.character(y)]
if (is.na(val)) 0L else as.integer(val)
}, integer(1))
} else {
collected <- rep(NA_integer_, length(years))
}
tibble::tibble(
year = years,
n_units_collected = collected,
n_units_reporting = as.integer(reporting),
n_units_expected = rep(as.integer(n_expected), length(years)),
is_census_year = .is_census_year(years)
+18
View File
@@ -166,12 +166,30 @@ cog_explain <- function(result, format = c("print", "list")) {
cli::cli_h2("Reporting coverage")
cli::cli_text("Mode: {prov$coverage_mode %||% 'all'}")
cov <- prov$coverage
has_collected <- "n_units_collected" %in% names(cov)
if (has_collected) {
# Three counters: collected separates sampling from real zeros;
# reporting is category-conditional and never a response rate.
cli::cli_ul(sprintf(
"%d: %d of %d units collected, %d reporting in this category -- %s year",
cov$year,
cov$n_units_collected,
cov$n_units_expected,
cov$n_units_reporting,
ifelse(cov$is_census_year, "census", "sample")
))
} else {
cli::cli_ul(sprintf(
"%d: %d of %d units reporting (%.0f%%) -- %s year",
cov$year, cov$n_units_reporting, cov$n_units_expected,
100 * cov$n_units_reporting / pmax(cov$n_units_expected, 1L),
ifelse(cov$is_census_year, "census", "sample")
))
}
# Explains what the per-row "-- sample year" tag means, regardless of
# which branch above rendered it -- not gated on has_collected, which
# would make this permanently unreachable now that both real callers
# (cog_geographic_rollup(), cog_peer_compare()) always supply it.
if (any(!cov$is_census_year)) {
cli::cli_text(
"Note: the Census of Governments is a complete census only in years ending in 2 or 7; every other year is a sample."
+34 -13
View File
@@ -195,11 +195,13 @@ cog_find_peers <- function(target_govid,
#' a balanced panel.
#'
#' Regardless of mode, `provenance$coverage` always carries per-year
#' `n_units_reporting`, `n_units_expected` and `is_census_year`, and
#' `provenance$coverage_mode` records the mode. `is_census_year` is a
#' statement about the **survey calendar**, never a claim of completeness:
#' FY1967 is a census year in which only 97 of Wisconsin's 608 cities
#' report. `n_units_reporting` is the number that tells the truth.
#' `n_units_expected`, `n_units_collected`, `n_units_reporting` and
#' `is_census_year`, and `provenance$coverage_mode` records the mode.
#' `is_census_year` is a statement about the **survey calendar**, never a
#' claim of completeness: FY1967 is a census year in which only 97 of
#' Wisconsin's 608 cities report. `n_units_reporting` is
#' category-conditional and is not a response rate on its own -- see
#' "Reading `coverage`" below for what each counter answers.
#'
#' The comparison target is exempt from `"consistent"` balancing -- it is the
#' subject of the comparison, not a member of the cohort -- and the
@@ -241,14 +243,23 @@ cog_find_peers <- function(target_govid,
#' summarise(p50 = quantile(total, 0.5, na.rm = TRUE))
#' ```
#' @section Reading `coverage`:
#' `provenance$coverage` reports `n_units_reporting` against
#' `n_units_expected` per year. **`n_units_reporting` is category-conditional:
#' it counts cohort members with rows for the category you asked for, not
#' cohort members collected that year.** A government that was surveyed and
#' genuinely spends nothing in that category is indistinguishable here from one
#' that was never surveyed.
#' `provenance$coverage` carries three per-year counters:
#'
#' * `n_units_expected` -- how many governments you asked about.
#' * `n_units_collected` -- how many of those appear in the corpus at all
#' that year (in ANY category), separating sampling from real zeros.
#' * `n_units_reporting` -- how many have rows for the SPECIFIC category you
#' requested. This is always <= n_units_collected: a government can be
#' collected but have no rows for "Police" because it contracts policing
#' to the county sheriff, not because it wasn't surveyed.
#'
#' **`n_units_reporting` is category-conditional** and therefore **not a
#' response rate**: `n_units_reporting / n_units_expected` conflates sampling
#' (never collected) with real zeros (collected but spends nothing in your
#' category). Use `n_units_collected / n_units_expected` for the true
#' collection rate, and `n_units_reporting / n_units_collected` for category
#' participation among collected units.
#'
#' The ratio is therefore **not a response rate** and must not be used as one.
#' In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
#' `category = "Police"`; the 174-city gap is overwhelmingly cities that
#' contract policing to the county sheriff, not non-response.
@@ -325,10 +336,20 @@ cog_peer_compare <- function(target_govid, peers, category, years,
# Counted over PEER rows only, against the cohort size: "3 of your 15 peers
# reported in FY2019". Including the target would inflate every count by one
# and make a cohort that has entirely stopped reporting look non-empty.
# n_units_collected is looked up against the spending long view matching
# whatever basis cog_spending() actually resolved above (prov$basis) --
# NOT hardcoded to spending_long_harmonized, which does not exist on a
# corpus with schema_version < 5 (R/basis.R resolves basis = "raw" there,
# and only *_long, not *_long_harmonized, is registered; see R/views.R).
# The con comes from .ensure_session() already called inside cog_spending().
con <- .ensure_session()
prov$coverage_mode <- coverage
prov$coverage <- .coverage_table(
out, years, length(peer_govids),
rows = r[r$role == "peer", , drop = FALSE]
rows = r[r$role == "peer", , drop = FALSE],
con = con,
long_view = .select_long_view("spending_annotated", prov$basis),
expected_ids = peer_govids
)
attr(out, "provenance") <- prov
out
+36 -13
View File
@@ -49,11 +49,13 @@
#' a balanced panel.
#'
#' Regardless of mode, `provenance$coverage` always carries per-year
#' `n_units_reporting`, `n_units_expected` and `is_census_year`, and
#' `provenance$coverage_mode` records the mode. `is_census_year` is a
#' statement about the **survey calendar**, never a claim of completeness:
#' FY1967 is a census year in which only 97 of Wisconsin's 608 cities
#' report. `n_units_reporting` is the number that tells the truth.
#' `n_units_expected`, `n_units_collected`, `n_units_reporting` and
#' `is_census_year`, and `provenance$coverage_mode` records the mode.
#' `is_census_year` is a statement about the **survey calendar**, never a
#' claim of completeness: FY1967 is a census year in which only 97 of
#' Wisconsin's 608 cities report. `n_units_reporting` is
#' category-conditional and is not a response rate on its own -- see
#' "Reading `coverage`" below for what each counter answers.
#' @return Tibble with columns `year`, `layer`, `canonical_govid`, `gov_name`,
#' `spend_subtype`, `category`, `amt_nominal`, optional `amt_real` /
#' `amt_per_capita_nominal` / `amt_per_capita_real`, optional `pop_source`,
@@ -61,14 +63,23 @@
#' `provenance` attribute with `verb = "cog_geographic_rollup"`, `layers`,
#' and `rollup$included_govids` / `rollup$excluded_govids`.
#' @section Reading `coverage`:
#' `provenance$coverage` reports `n_units_reporting` against
#' `n_units_expected` per year. **`n_units_reporting` is category-conditional:
#' it counts governments with rows for the category you asked for, not
#' governments collected that year.** A government that was surveyed and
#' genuinely spends nothing in that category is indistinguishable here from one
#' that was never surveyed.
#' `provenance$coverage` carries three per-year counters:
#'
#' * `n_units_expected` -- how many governments you asked about.
#' * `n_units_collected` -- how many of those appear in the corpus at all
#' that year (in ANY category), separating sampling from real zeros.
#' * `n_units_reporting` -- how many have rows for the SPECIFIC category you
#' requested. This is always <= n_units_collected: a government can be
#' collected but have no rows for "Police" because it contracts policing
#' to the county sheriff, not because it wasn't surveyed.
#'
#' **`n_units_reporting` is category-conditional** and therefore **not a
#' response rate**: `n_units_reporting / n_units_expected` conflates sampling
#' (never collected) with real zeros (collected but spends nothing in your
#' category). Use `n_units_collected / n_units_expected` for the true
#' collection rate, and `n_units_reporting / n_units_collected` for category
#' participation among collected units.
#'
#' The ratio is therefore **not a response rate** and must not be used as one.
#' In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
#' `category = "Police"`; the 174-city gap is overwhelmingly cities that
#' contract policing to the county sheriff, not non-response.
@@ -137,8 +148,20 @@ cog_geographic_rollup <- function(govids, category, years,
# n_units_expected is the universe the CALLER named -- the govids passed in
# -- not the national universe. That is what makes the ratio meaningful:
# "597 of the 608 Wisconsin cities you asked about reported in FY2012".
# n_units_collected is looked up against the spending long view matching
# whatever basis cog_spending() actually resolved above (prov$basis) --
# NOT hardcoded to spending_long_harmonized, which does not exist on a
# corpus with schema_version < 5 (R/basis.R resolves basis = "raw" there,
# and only *_long, not *_long_harmonized, is registered; see R/views.R).
# The con comes from .ensure_session() already called inside cog_spending().
con <- .ensure_session()
prov$coverage_mode <- coverage
prov$coverage <- .coverage_table(r, years, length(unique(all_govids)))
prov$coverage <- .coverage_table(
r, years, length(unique(all_govids)),
con = con,
long_view = .select_long_view("spending_annotated", prov$basis),
expected_ids = all_govids
)
attr(r, "provenance") <- prov
r
+49 -13
View File
@@ -89,7 +89,7 @@
#' project's "functions under 50 lines" convention:
#' \itemize{
#' \item `.query_candidate_recipes()` -- candidate recipe lookup by
#' category/subtype scope + M/L exclusion.
#' category/subtype scope + `category_type` filter (#34) + M/L exclusion.
#' \item `.query_recipe_meta()` -- metadata (label, year spans).
#' \item `.query_covered_years()` -- Path 1 gap-year coverage via the
#' recipe's own generic join.
@@ -126,8 +126,16 @@
# by `category` (`.ALL_CATEGORIES` is never a row in
# `summary_categories.category`, so a category-keyed sub-select always
# came back empty here). The M/L exclusion below is unchanged either way.
candidates <- .query_candidate_recipes(con, category, all_categories,
subtype_col, subtype_scope)
#
# Issue #34: scope the candidate query by `category_type` ('expenditure'
# vs 'revenue') to prevent cross-flow-family leakage -- e.g.
# `cog_revenue(category = "Corrections")` must not surface
# expenditure-only recipes (E04/E05) merely because they share the same
# category name in summary_categories. The type is derived from
# flow_prefixes: E/F/G -> 'expenditure', anything else -> 'revenue'.
candidates <- .query_candidate_recipes(con, category, flow_prefixes,
all_categories, subtype_col,
subtype_scope)
if (length(candidates) == 0L) return(list())
result_years <- if (is.null(result) || nrow(result) == 0L) {
@@ -217,8 +225,17 @@
#' `summary_categories.category`, so a category-keyed sub-select always
#' returns zero candidates and silently disables signposting.
#'
#' Scope is also by `category_type` ('expenditure' vs 'revenue', Issue #34)
#' to prevent cross-flow-family leakage: `cog_revenue(category =
#' "Corrections")` must not surface expenditure-only recipes (E04/E05)
#' merely because they share the same category name in summary_categories.
#' The type is derived from flow_prefixes: E/F/G -> 'expenditure', anything
#' else -> 'revenue'.
#'
#' @param con Active DuckDB connection.
#' @param category Category name, or `NULL`.
#' @param flow_prefixes The calling verb's own flow-type prefixes (see
#' `.build_suggestions()`). Used to derive `category_type` (#34).
#' @param all_categories `TRUE` when the caller used `.ALL_CATEGORIES`.
#' @param subtype_col Name of the summary_categories subtype column to
#' scope by when `all_categories = TRUE`; ignored otherwise.
@@ -226,18 +243,31 @@
#' when `all_categories = TRUE`; ignored otherwise.
#' @return Character vector of recipe IDs (possibly empty).
#' @noRd
.query_candidate_recipes <- function(con, category, all_categories = FALSE,
.query_candidate_recipes <- function(con, category, flow_prefixes,
all_categories = FALSE,
subtype_col = NULL,
subtype_scope = NULL) {
# Issue #34: derive category_type from flow_prefixes to prevent
# cross-flow-family leakage -- e.g. cog_revenue(category = "Corrections")
# must not surface expenditure-only recipes merely because they share the
# same category name in summary_categories.
category_type <- if (all(flow_prefixes %in% c("E", "F", "G"))) {
"expenditure"
} else {
"revenue"
}
candidate_scope_sql <- if (isTRUE(all_categories)) {
sprintf(
"SELECT DISTINCT item_code FROM summary_categories WHERE %s IN (%s)",
subtype_col, .sql_lit_chr(subtype_scope)
"SELECT DISTINCT item_code FROM summary_categories
WHERE %s IN (%s) AND category_type = '%s'",
subtype_col, .sql_lit_chr(subtype_scope), category_type
)
} else {
sprintf(
"SELECT DISTINCT item_code FROM summary_categories WHERE category IN (%s)",
.sql_lit_chr(category)
"SELECT DISTINCT item_code FROM summary_categories
WHERE category IN (%s) AND category_type = '%s'",
.sql_lit_chr(category), category_type
)
}
@@ -270,14 +300,14 @@
#' result.
#' @return Data frame with columns `recipe_id` (character) and `year`
#' (integer). Returns an empty data frame (`recipe_id = character(0)`,
#' `year = integer(0)`) when `gap_years` is empty, so callers can safely
#' reference `$recipe_id`.
#' `year = integer(0)`) when `gap_years` or `candidates` is empty, so
#' callers can safely reference `$recipe_id`.
#' @noRd
.query_covered_years <- function(con, candidates, cohort, gap_years) {
if (length(gap_years) == 0L) {
if (length(gap_years) == 0L || length(candidates) == 0L) {
return(data.frame(recipe_id = character(0), year = integer(0)))
}
DBI::dbGetQuery(con, sprintf(
res <- DBI::dbGetQuery(con, sprintf(
"SELECT DISTINCT r.recipe_id, l.year
FROM long l
JOIN harmonization_recipes r
@@ -292,6 +322,8 @@
.sql_lit_chr(candidates), .cohort_sql(cohort, "l.canonical_govid"),
paste(gap_years, collapse = ",")
))
res$year <- as.integer(res$year)
res
}
#' Query recipe metadata: labels and year spans for a set of candidate
@@ -358,7 +390,7 @@
#' `R/basis.R`). This blocks a recipe surfaced through a mis-scoped
#' category from ever reaching the M/L search, e.g. `cog_spending()`'s
#' flow_prefixes are `c("E","F","G")`, which `ig_federal_b47_wide`'s own
#' "B" is not part of.
#' `"B"` is not part of.
#' 2. `own_prefix %in% c("E","F","G")`: M/L only ever pairs with the
#' DIRECT-expenditure family, never with revenue (`cog_revenue()`'s
#' flow_prefixes already fold B/C/D in as ordinary revenue -- there is
@@ -366,6 +398,10 @@
#' adds one for spending) and never with ANOTHER M/L recipe (without
#' this check, `ige_local_m47_wide` would wrongly match sibling
#' `ige_state_l47_wide` on their shared {"47","94"} suffix set).
#' Condition 1 alone does not catch this: under `cog_revenue()`,
#' `ig_federal_b47_wide`'s own `"B"` IS inside revenue's own
#' `flow_prefixes`, so only this second, family-specific check blocks
#' the search.
#' @noRd
.attach_ig_counterparts <- function(con, suggestions, flow_prefixes) {
if (length(suggestions) == 0L) return(suggestions)
+10 -3
View File
@@ -227,7 +227,8 @@ A statewide total resting on a fifth of the universe looks exactly like one
resting on all of it, so every multi-government result now says which it is:
```r
attr(rollup, "provenance")$coverage # per-year n_units_reporting, is_census_year
attr(rollup, "provenance")$coverage
# per-year n_units_expected, n_units_collected, n_units_reporting, is_census_year
```
`cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` take a
@@ -235,8 +236,14 @@ attr(rollup, "provenance")$coverage # per-year n_units_reporting, is_census_ye
`"consistent"` (only units reporting in every requested year, a balanced
panel).
`n_units_reporting` is **category-conditional**, and it is not a response rate. A government that was surveyed and genuinely spends
nothing in the requested category is indistinguishable from one never surveyed.
`n_units_reporting` is **category-conditional**: it counts governments with
rows for the *specific* category you asked for, so a government that was
surveyed and genuinely spends nothing in that category is indistinguishable
from one never surveyed — it is not a response rate on its own.
`n_units_collected` is the number that separates them: governments present in
the corpus that year for *any* category. `n_units_collected / n_units_expected`
is the true collection rate; `n_units_reporting / n_units_collected` is
category participation among collected units.
### Absent cells mean two different things
+23 -12
View File
@@ -55,11 +55,13 @@ Direct spending); `"primary"` and `"direct"` combine safely.}
a balanced panel.
Regardless of mode, `provenance$coverage` always carries per-year
`n_units_reporting`, `n_units_expected` and `is_census_year`, and
`provenance$coverage_mode` records the mode. `is_census_year` is a
statement about the **survey calendar**, never a claim of completeness:
FY1967 is a census year in which only 97 of Wisconsin's 608 cities
report. `n_units_reporting` is the number that tells the truth.}
`n_units_expected`, `n_units_collected`, `n_units_reporting` and
`is_census_year`, and `provenance$coverage_mode` records the mode.
`is_census_year` is a statement about the **survey calendar**, never a
claim of completeness: FY1967 is a census year in which only 97 of
Wisconsin's 608 cities report. `n_units_reporting` is
category-conditional and is not a response rate on its own -- see
"Reading `coverage`" below for what each counter answers.}
}
\value{
Tibble with columns `year`, `layer`, `canonical_govid`, `gov_name`,
@@ -86,14 +88,23 @@ by design — see `vignette('population-denominators')`.
}
\section{Reading `coverage`}{
`provenance$coverage` reports `n_units_reporting` against
`n_units_expected` per year. **`n_units_reporting` is category-conditional:
it counts governments with rows for the category you asked for, not
governments collected that year.** A government that was surveyed and
genuinely spends nothing in that category is indistinguishable here from one
that was never surveyed.
`provenance$coverage` carries three per-year counters:
* `n_units_expected` -- how many governments you asked about.
* `n_units_collected` -- how many of those appear in the corpus at all
that year (in ANY category), separating sampling from real zeros.
* `n_units_reporting` -- how many have rows for the SPECIFIC category you
requested. This is always <= n_units_collected: a government can be
collected but have no rows for "Police" because it contracts policing
to the county sheriff, not because it wasn't surveyed.
**`n_units_reporting` is category-conditional** and therefore **not a
response rate**: `n_units_reporting / n_units_expected` conflates sampling
(never collected) with real zeros (collected but spends nothing in your
category). Use `n_units_collected / n_units_expected` for the true
collection rate, and `n_units_reporting / n_units_collected` for category
participation among collected units.
The ratio is therefore **not a response rate** and must not be used as one.
In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
`category = "Police"`; the 174-city gap is overwhelmingly cities that
contract policing to the county sheriff, not non-response.
+23 -12
View File
@@ -50,11 +50,13 @@ safely.}
a balanced panel.
Regardless of mode, `provenance$coverage` always carries per-year
`n_units_reporting`, `n_units_expected` and `is_census_year`, and
`provenance$coverage_mode` records the mode. `is_census_year` is a
statement about the **survey calendar**, never a claim of completeness:
FY1967 is a census year in which only 97 of Wisconsin's 608 cities
report. `n_units_reporting` is the number that tells the truth.
`n_units_expected`, `n_units_collected`, `n_units_reporting` and
`is_census_year`, and `provenance$coverage_mode` records the mode.
`is_census_year` is a statement about the **survey calendar**, never a
claim of completeness: FY1967 is a census year in which only 97 of
Wisconsin's 608 cities report. `n_units_reporting` is
category-conditional and is not a response rate on its own -- see
"Reading `coverage`" below for what each counter answers.
The comparison target is exempt from `"consistent"` balancing -- it is the
subject of the comparison, not a member of the cohort -- and the
@@ -109,14 +111,23 @@ them.
}
\section{Reading `coverage`}{
`provenance$coverage` reports `n_units_reporting` against
`n_units_expected` per year. **`n_units_reporting` is category-conditional:
it counts cohort members with rows for the category you asked for, not
cohort members collected that year.** A government that was surveyed and
genuinely spends nothing in that category is indistinguishable here from one
that was never surveyed.
`provenance$coverage` carries three per-year counters:
* `n_units_expected` -- how many governments you asked about.
* `n_units_collected` -- how many of those appear in the corpus at all
that year (in ANY category), separating sampling from real zeros.
* `n_units_reporting` -- how many have rows for the SPECIFIC category you
requested. This is always <= n_units_collected: a government can be
collected but have no rows for "Police" because it contracts policing
to the county sheriff, not because it wasn't surveyed.
**`n_units_reporting` is category-conditional** and therefore **not a
response rate**: `n_units_reporting / n_units_expected` conflates sampling
(never collected) with real zeros (collected but spends nothing in your
category). Use `n_units_collected / n_units_expected` for the true
collection rate, and `n_units_reporting / n_units_collected` for category
participation among collected units.
The ratio is therefore **not a response rate** and must not be used as one.
In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
`category = "Police"`; the 174-city gap is overwhelmingly cities that
contract policing to the county sheriff, not non-response.
+37
View File
@@ -12,3 +12,40 @@ file carries the reasoning and the pointers.
- **Refs** — commits, issues, decision records.
---
## 2026-09-30 · repo · moved out of Nextcloud; code syncs only through Gitea
**Why:** Nextcloud was syncing this repo's `.git`, which risks conflict copies inside
it and stalls the client's cold scan across dozens of repos. Nothing untracked is
data, so nothing is linked (`--no-data`); the move changed no test result (1,101
pass, 2 skip before and after).
**Obligates:** #73, #74
**Refs:** 4bedf85 (ci/apt-https pushed to clear the move's preflight), 63b421a
## 2026-09-09 · ci · Linux R-CMD-check legs unblocked (recorded 2026-09-30)
**Why:** GitHub's ubuntu-latest image carries a google-chrome apt source that
intermittently fails its own hash check, and setup-r's `apt-get update` died on it for
every R version; the source is now found by URL and removed, and the step fails
loudly if it survives.
**Obligates:** (none)
**Refs:** d83301b, d1165a5
## 2026-09-09 · model · #36 coverage counter finished, two bugs caught before merge
**Why:** the drafted n_units_collected fix (uncommitted) derived its candidate
cohort from category-filtered results instead of the caller's full expected
cohort, and hardcoded a view name absent below schema_version 5 -- both
silent on the fixture, both would have shipped without an independent review
pass before merge.
**Obligates:** #72
**Refs:** #36, b41d5ee, PR#71
## 2026-09-09 · model · #33/#34 split into independently-tested commits
**Why:** #33's own branch had its tests sitting uncommitted, and quietly bundled
a behavior change (#34) into what its commit message called a pure refactor;
splitting them let each pass CI with its own tests instead of merging on a
false "tests pass" claim.
**Obligates:** (none)
**Refs:** #33, #34, 0c7c7eb, 392643b, d0d724c, 9f5cd98, PR#69, PR#70
+16 -32
View File
@@ -7,46 +7,30 @@
## Where this stands
uscogdata is at 0.4.0 and its public surface is settled: the query verbs, the cohort
predicates added in this release, and the provenance contract every verb returns.
The six open issues split cleanly. Two are API work carried out of the #9 review pass
and deliberately deferred there rather than fixed in that branch. Three concern the
corpus layer, and the largest of them, partition-level caching, was named the single
highest-leverage change on the remote path before being deferred. One, the
data-correction intake (#52), is a decision rather than a task: it was parked during
the 0.3.0 design, and the API announcement waits on it, because without it the corpus
cannot make the "traceable and correctable" claim that most distinguishes it from
Census's own files.
Nothing here is blocked on anything else, so the ordering is a judgement about value
rather than a dependency graph.
Compass's own files moved out of `docs/` this session. They were sitting inside
pkgdown's output directory, and `pkgdown::clean_site()` deletes every top-level entry
there except `CNAME` and `dev` — asked directly, it listed `docs/pm` and
`docs/decisions` among the 28 it would remove, with the guard that would have stopped
it satisfied by `docs/pkgdown.yml`. They are in `pm/` now. Nothing was lost: the
journal had no entries and there were no decision records yet, which made this the
cheapest moment to move. The `.gitignore` workaround that re-included two children of
an excluded `docs/` is gone with it.
uscogdata's code now lives only in Gitea; the old Nextcloud folder keeps just build
output and working files. The move changed no test result: 1,101 pass and the two
opt-in live-corpus tests skip, before and after. It turned up one defect, #73: the
fixture's reference docs have never been in git, because a `.gitignore` rule matches
too broadly. Next in line are #73 and #72 (a vignette for the coverage counters), with
#52 (data-correction intake) and #74 (the stale state section in `CLAUDE.md`) waiting
on a decision.
## Ready to work on next
- **#34** cog_revenue() offers expenditure recipes as suggestions: scope the candidate query by category_type · `ws/api` — nothing is blocking it; something is currently wrong
- **#36** n_units_reporting is category-conditional and cannot be read as a response rate · `ws/corpus` — nothing is blocking it; owed work from an earlier change
- **#73** The fixture corpus's docs/ never reaches git: .gitignore's docs/ rule is unanchored · `ws/corpus` — nothing is blocking it; something is currently wrong
- **#72** docs: add a vignette explaining provenance$coverage counters · `ws/docs` — nothing is blocking it; owed work from an earlier change
- **#2** Extend population data to be households as an alternate spending denominator · `ws/corpus` — nothing is blocking it
- **#33** Decompose .build_suggestions() (106 lines) into named helpers · `ws/api` — nothing is blocking it
- **#52** Release 11/11: design the data-correction intake (deferred; gates the API announcement) · `ws/corpus` — nothing is blocking it
- **#64** Partition-level caching: R/cache.R is still a stub, and the remote path pays for it every session · `ws/corpus` — nothing is blocking it
- **#74** CLAUDE.md's Current State section is frozen at 2026-08-03 · `ws/docs` — waiting on a person, not on other work
- **#52** Release 11/11: design the data-correction intake (deferred; gates the API announcement) · `ws/corpus` — waiting on a person, not on other work
## Workstreams
| Stream | Commits since | Open | Debt | Owes docs |
|---|---|---|---|---|
| Query verbs and results | 77 | 2 | 0 | no |
| Corpus, mirror, provenance | 39 | 4 | 1 | no |
| Vignettes and guides | 34 | 0 | 0 | **yes** |
| Query verbs and results | 0 | 0 | 0 | no |
| Corpus, mirror, provenance | 0 | 4 | 0 | no |
| Vignettes and guides | 0 | 2 | 2 | no |
## CI
@@ -58,8 +42,8 @@ an excluded `docs/` is gone with it.
_Nothing blocks anything else, so there is no graph to draw._
- Marker: `none` (no journal entry yet)
- Commits since: 165
- Marker: `9adb9211` (2026-09-09)
- Commits since: 3
- Open issues: 6
</details>
+108
View File
@@ -48,6 +48,13 @@ test_that("multi-government aggregates disclose reporting coverage on every resu
expect_equal(cov$n_units_reporting, c(152L, 597L, 112L, 114L))
expect_equal(cov$is_census_year, c(FALSE, TRUE, FALSE, FALSE))
# uscogdata#36: with category = NULL (no category scope), "reported at
# all" and "collected" are the same question, so n_units_collected must
# equal n_units_reporting exactly here. This case alone cannot catch a
# regression in HOW n_units_collected is computed, though: see the
# category-scoped test below for that.
expect_equal(cov$n_units_collected, cov$n_units_reporting)
# Cross-check against the raw partitions, scoped to the SAME universe the
# rollup was given -- the 608 govids above. Scoping instead on the long
# table's own `type`/`fips_state` asks a different question and answers 595:
@@ -80,6 +87,8 @@ test_that("multi-government aggregates disclose reporting coverage on every resu
expect_equal(cov_peers$n_units_expected, rep(15L, 3L))
expect_equal(cov_peers$n_units_reporting, c(15L, 3L, 3L))
expect_equal(cov_peers$is_census_year, c(TRUE, FALSE, FALSE))
# uscogdata#36: same identity as the rollup case above, category = NULL.
expect_equal(cov_peers$n_units_collected, cov_peers$n_units_reporting)
# -- the three coverage modes --------------------------------------------
expect_equal(attr(cog_peer_compare(target_govid = chilton, peers = peers,
@@ -101,3 +110,102 @@ test_that("multi-government aggregates disclose reporting coverage on every resu
coverage = "census")
expect_equal(sort(unique(census_only$year)), 2012)
})
test_that("n_units_collected separates sampling from real zeros, category-scoped (uscogdata#36)", {
# The motivating case from the issue: Wisconsin cities, category = "Police".
# FY2012 is a complete census year -- collection is not partial -- yet a
# category-conditional n_units_reporting alone reads like a sampling gap.
# n_units_collected must diverge from n_units_reporting here, unlike the
# category = NULL cases above, because most of the FY2012 gap is cities
# that contract policing to the county sheriff (collected, real zero), not
# cities Census never surveyed.
wi <- cog_gov_search(name = NULL, state = "WI", type = "city")
roll <- suppressMessages(cog_geographic_rollup(
govids = list(city = wi$canonical_govid), category = "Police",
years = c(2011L, 2012L, 2019L, 2020L)))
cov <- wt_coverage(roll)
expect_equal(cov$n_units_expected, rep(608L, 4L))
expect_equal(cov$n_units_collected, c(152L, 597L, 112L, 114L))
expect_equal(cov$n_units_reporting, c(152L, 485L, 109L, 111L))
# The pair the issue actually wants: collected/expected is the true
# collection rate (98% in the FY2012 census year, matching the raw
# cross-check above); reporting/collected is category participation among
# collected units (81% -- most of the gap is real, not sampling).
expect_equal(round(cov$n_units_collected[cov$year == 2012] /
cov$n_units_expected[cov$year == 2012], 2), 0.98)
expect_equal(round(cov$n_units_reporting[cov$year == 2012] /
cov$n_units_collected[cov$year == 2012], 2), 0.81)
# Every year: collected is bounded between reporting and expected.
expect_true(all(cov$n_units_collected >= cov$n_units_reporting))
expect_true(all(cov$n_units_collected <= cov$n_units_expected))
})
test_that(".coverage_table() candidates a government collected-but-absent from the category result (uscogdata#36)", {
# Direct regression test for the mechanism itself: n_units_collected's
# candidate list must be the caller's full expected cohort (expected_ids),
# never derived from `result`/`rows`. A government with zero rows in the
# requested category across every requested year never appears in
# `result` at all, so deriving candidates from `result` would silently
# drop exactly the "collected but real zero" governments this counter
# exists to count -- collapsing it back to n_units_reporting.
con <- uscogdata:::.ensure_session()
# A real fixture govid, present in spending_long_harmonized for 2019 (in
# SOME category), but absent from this fake category-specific `result`.
govid <- "011021100004"
fake_result <- data.frame(canonical_govid = character(0), year = integer(0))
cov <- uscogdata:::.coverage_table(
fake_result, years = 2019L, n_expected = 1L,
con = con, long_view = "spending_long_harmonized",
expected_ids = govid
)
expect_equal(cov$n_units_collected, 1L)
expect_equal(cov$n_units_reporting, 0L)
# Without a connection, long_view, or expected_ids, the lookup is skipped
# rather than silently wrong.
no_con <- uscogdata:::.coverage_table(fake_result, years = 2019L, n_expected = 1L)
expect_true(is.na(no_con$n_units_collected))
no_ids <- uscogdata:::.coverage_table(
fake_result, years = 2019L, n_expected = 1L,
con = con, long_view = "spending_long_harmonized"
)
expect_true(is.na(no_ids$n_units_collected))
})
test_that("n_units_collected uses the resolved basis's long view, not a hardcoded harmonized one (uscogdata#36)", {
# spending_long_harmonized only exists when schema_version >= 5 (R/views.R
# gates the harmonization views on it); on an older corpus cog_spending()
# resolves basis = "raw" and queries spending_long instead. The coverage
# lookup must follow the SAME resolved basis, not a literal
# "spending_long_harmonized", or it hard-errors with a DuckDB catalog
# error on every schema_version < 5 corpus -- a vintage the package
# otherwise explicitly still supports (see test-manifest.R's dual-accept
# tests).
skip_if_no_corpus()
with_doctored_schema_version(4L, {
con <- cog_open()
ids <- DBI::dbGetQuery(con,
"SELECT DISTINCT canonical_govid FROM spending_long WHERE year = 2011 LIMIT 3"
)$canonical_govid
expect_gte(length(ids), 3L)
roll <- suppressMessages(cog_geographic_rollup(
list(city = ids), category = NULL, years = 2011L))
expect_equal(attr(roll, "provenance")$basis, "raw")
cov <- attr(roll, "provenance")$coverage
expect_false(is.na(cov$n_units_collected))
expect_equal(cov$n_units_collected, length(ids))
cmp <- suppressMessages(cog_peer_compare(
target_govid = ids[1], peers = ids[-1], category = NULL, years = 2011L))
expect_equal(attr(cmp, "provenance")$basis, "raw")
cov_peers <- attr(cmp, "provenance")$coverage
expect_false(is.na(cov_peers$n_units_collected))
})
})
+7 -5
View File
@@ -315,15 +315,17 @@ test_that("a mis-scoped cog_spending() call never attaches an M/L counterpart to
# (SB194, cog_pipeline#64), so the recipe stopped being a candidate there.
# FL state carries a real FY2011 B47 amount, so this exercises the guard
# against a suggestion that genuinely fires.
#
# Issue #34: "IG Federal" maps to B-prefixed codes in summary_categories
# with category_type = 'revenue'. A spending verb (flow_prefixes E/F/G)
# now scopes its candidate query by category_type = 'expenditure', so it
# correctly finds NO candidates for this revenue-only category -- the
# suggestion machinery cannot fire, and no M/L counterpart is attached.
r <- suppressMessages(
cog_spending("120000226351", years = c(2005, 2011), category = "IG Federal")
)
sugg <- attr(r, "provenance")$suggestions
expect_gt(length(sugg), 0L)
ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
expect_true("ig_federal_b47_wide" %in% ids)
ig <- unlist(lapply(sugg, function(s) s$ig_recipe_id))
expect_length(ig, 0L)
expect_length(sugg, 0L)
})
test_that("C1: 'total' on a legacy aggregate-only family reports the IG-only figure honestly, not as Direct + IG", {
+14
View File
@@ -118,3 +118,17 @@ test_that("cog_explain prints denominator + popyear_range + counts", {
expect_false(grepl("popyear range: 19-20", out, fixed = TRUE))
})
})
test_that("cog_explain reports units collected alongside units reporting (uscogdata#36)", {
skip_if_no_corpus()
wi <- cog_gov_search(name = NULL, state = "WI", type = "city")
roll <- suppressMessages(cog_geographic_rollup(
govids = list(city = wi$canonical_govid), category = "Police",
years = 2012L))
out <- paste(c(
capture.output(cog_explain(roll)),
capture.output(cog_explain(roll), type = "message")
), collapse = "\n")
expect_true(grepl("597 of 608 units collected", out, fixed = TRUE))
expect_true(grepl("485 reporting in this category", out, fixed = TRUE))
})
+32 -21
View File
@@ -371,34 +371,45 @@ test_that("uscogdata#9: the revenue verb inherits the same trigger", {
expect_null(sugg[[1]]$ig_recipe_id)
})
test_that("I1: cog_revenue never fabricates suppressed dollars for an expenditure-only recipe", {
test_that("I1 + #34: cog_revenue never suggests expenditure-only recipes", {
# uscogdata#9 review, finding I1: Corrections is an expenditure-only
# category (E04/E05). cog_revenue() naturally returns zero rows for it, so
# corrections_combined still fires as an empty_year suggestion (its own
# generic join finds real E04/E05 data for this government) -- but before
# the flow_prefixes fix, .suppressed_components() measured E04/E05 against
# cog_revenue()'s OWN view (which can never contain an E-coded row by
# construction) and reported the full $3,631,945,000 as "suppressed",
# when cog_spending() for the same gov/years/category actually returns
# $3,691,029,000 -- nothing was suppressed at all.
# category (E04/E05). Before the flow_prefixes fix (#9), .suppressed_components()
# measured E04/E05 against cog_revenue()'s OWN view and reported $3.6B as
# "suppressed" -- nothing was suppressed at all.
#
# Issue #34 builds on that: the candidate query now also filters by
# category_type ('revenue'), so expenditure-only recipes like corrections_combined
# (whose components E04/E05 are classified as 'expenditure' in summary_categories)
# are never even considered for a revenue verb. This is stronger than just
# suppressing the dollar claim -- it prevents the suggestion from firing at all.
skip_if_no_corpus()
r <- suppressMessages(
cog_revenue("061037123085", years = 2019:2020, category = "Corrections"))
sugg <- attr(r, "provenance")$suggestions
ids <- vapply(sugg, function(s) s$recipe_id, character(1))
expect_true("corrections_combined" %in% ids)
ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
hit <- sugg[[which(ids == "corrections_combined")]]
expect_equal(hit$suppressed_amount, 0)
expect_equal(hit$suppressed_years, integer(0))
expect_equal(hit$suppressed_codes, character(0))
# corrections_combined should NOT appear -- its components are expenditure-only.
expect_false("corrections_combined" %in% ids)
})
# And cog_spending() for the identical gov/years/category is unaffected --
# it actually finds the E04/E05 dollars the buggy measurement claimed were
# excluded.
sp <- suppressMessages(
cog_spending("061037123085", years = 2019:2020, category = "Corrections"))
expect_equal(sum(sp$amt_nominal), 3691029000)
test_that(".query_candidate_recipes() scopes candidates by category_type (#34)", {
# Direct assertion on the mechanism the two tests above exercise
# end-to-end: corrections_combined's own components (E04/E05) are
# category_type = 'expenditure' in summary_categories, so an
# expenditure-flavored flow_prefixes call must surface it and a
# revenue-flavored one must not. This queries only summary_categories/
# harmonization_recipes (no government data), so it runs against the
# bundled fixture with no skip_if_no_corpus() needed.
con <- uscogdata:::.ensure_session()
expenditure <- uscogdata:::.query_candidate_recipes(
con, category = "Corrections", flow_prefixes = c("E", "F", "G"))
expect_true("corrections_combined" %in% expenditure)
revenue <- uscogdata:::.query_candidate_recipes(
con, category = "Corrections",
flow_prefixes = c("T", "A", "U", "B", "C", "D"))
expect_false("corrections_combined" %in% revenue)
})
test_that("uscogdata#9: no partial-coverage fire in a modern year", {