Files
uscogdata/man/cog_geographic_rollup.Rd
T
jaredandClaude Sonnet 5 b41d5ee2aa
R-CMD-check / check (push) Successful in 3m39s
R-CMD-check / check (pull_request) Successful in 3m42s
feat(coverage): n_units_collected separates sampling from real zeros (#36)
provenance$coverage's n_units_reporting is category-conditional: it
counts governments with rows for the SPECIFIC requested category, which
conflates two different things -- a government never collected that
year (sampling), and one collected but genuinely spending nothing in
that category (a real zero). FY2012 Georgia Police is the motivating
case from the issue: a complete census year reads as a 69% "response
rate" because most of the gap is cities that contract policing to the
county sheriff, not non-response.

Adds a second counter, n_units_collected: how many of the caller's
expected cohort appear in the corpus that year for ANY category.
n_units_collected / n_units_expected is the true collection rate;
n_units_reporting / n_units_collected is category participation among
collected units. cog_geographic_rollup() and cog_peer_compare() both
carry it; cog_explain() prints it alongside n_units_reporting.

Two real bugs caught and fixed while finishing this (both against the
already-written, previously-uncommitted draft):

- .coverage_table()'s candidate list for the collection query was
  derived from the category-filtered result rows, not the caller's
  full expected cohort. A government with zero rows in the requested
  category across every requested year never appears in that result,
  so it was silently excluded from n_units_collected too -- collapsing
  the new counter back to the old, broken one for exactly the
  governments it exists to count. Fixed by threading an explicit
  `expected_ids` (all_govids / peer_govids) through instead.
- The collection query hardcoded long_view = "spending_long_harmonized",
  which does not exist on a corpus with schema_version < 5 (R/basis.R
  resolves basis = "raw" there; R/views.R only registers the
  harmonized views on v5+). cog_geographic_rollup()/cog_peer_compare()
  would hard-error on a corpus vintage the package otherwise explicitly
  supports. Fixed by deriving long_view from the basis cog_spending()
  actually resolved (prov$basis) via the existing .select_long_view()
  helper, matching how every other basis-aware query in the package
  already does this.

Also: cog_explain()'s general "complete census only in years ending in
2 or 7" footnote was gated on the OLD counter's absence, making it
permanently unreachable now that both callers always supply the new
one -- ungated it, since the explanation is orthogonal to which
counter set is present. Dropped a dead conditional branch, fixed two
stale roxygen blocks in R/peers.R/R/rollup.R still describing the old
two-counter model, fixed the same staleness in README.md, and switched
two `uscogdata:::` self-references to the package's own convention of
calling internal helpers unqualified.

1101 tests pass (2 skipped live-corpus), including new direct
regression tests for both bugs above (one exercising a government
collected-but-absent from a category result, one running the full
rollup/peer-compare path against a doctored schema_version 4 corpus).

Reviewed by an independent code-reviewer pass (1 HIGH, 1 MEDIUM, 3 LOW
-- all addressed above).

Closes #36.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 11:46:17 -04:00

118 lines
5.4 KiB
R

% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/rollup.R
\name{cog_geographic_rollup}
\alias{cog_geographic_rollup}
\title{Aggregate spending across state/county/city layers for a place}
\usage{
cog_geographic_rollup(
govids,
category,
years,
per_capita = FALSE,
adjust_to_year = NULL,
expenditure_concept = c("primary", "direct", "total"),
coverage = c("all", "census", "consistent")
)
}
\arguments{
\item{govids}{Named list with any non-empty subset of elements named
`state`, `county`, `city`. Each element is a character vector of
`canonical_govid` values. At least one layer required.}
\item{category}{Single category name or character vector (passed through
to [cog_spending()]), or the reserved `"All Categories"` for one summed
row per `(year, canonical_govid, subtype)` covering every category in the
concept's scope. `"All Categories"` is the efficient way to build a
geographic total: without it a caller must issue one rollup per category
and sum the results themselves.}
\item{years}{Integer vector of years.}
\item{per_capita}{If `TRUE`, per-capita uses each gov's own per-year
population from `gov_population_yearly`. Govs with missing population
are excluded from the result.}
\item{adjust_to_year}{Integer base year for CPI-U conversion, or `NULL`.}
\item{expenditure_concept}{`"primary"` (default), `"direct"`, or
`"total"` -- see [cog_spending()] for the three concepts. `"total"` is
refused here because combining Total across multiple layers of
government double-counts intergovernmental transfers (a state's payment
to a school district is the same dollar the district reports as its own
Direct spending); `"primary"` and `"direct"` combine safely.}
\item{coverage}{How to handle the Census of Governments survey cycle,
which is a **complete census only in years ending in 2 and 7** -- every
other year is a sample, and the sample varies enormously (on the bundled
fixture, Wisconsin's 608-city universe reports 597 governments in FY2012
and 112 in FY2019).
* `"all"` (default) -- every unit that reported that year. Unchanged
behaviour, so existing code keeps working.
* `"census"` -- census years only. Aborts if the requested range holds
none, rather than silently returning nothing.
* `"consistent"` -- only units reporting in *every* requested year, giving
a balanced panel.
Regardless of mode, `provenance$coverage` always carries per-year
`n_units_expected`, `n_units_collected`, `n_units_reporting` and
`is_census_year`, and `provenance$coverage_mode` records the mode.
`is_census_year` is a statement about the **survey calendar**, never a
claim of completeness: FY1967 is a census year in which only 97 of
Wisconsin's 608 cities report. `n_units_reporting` is
category-conditional and is not a response rate on its own -- see
"Reading `coverage`" below for what each counter answers.}
}
\value{
Tibble with columns `year`, `layer`, `canonical_govid`, `gov_name`,
`spend_subtype`, `category`, `amt_nominal`, optional `amt_real` /
`amt_per_capita_nominal` / `amt_per_capita_real`, optional `pop_source`,
`codes_included`, `aggregate_fallback`, `scope_note`, `notes`. Carries a
`provenance` attribute with `verb = "cog_geographic_rollup"`, `layers`,
and `rollup$included_govids` / `rollup$excluded_govids`.
}
\description{
Wraps [cog_spending()], tags each row with its layer, and attaches a
human-readable `scope_note` documenting geographic-scope caveats (e.g.
"county totals include areas outside the listed city"). Useful for
"place portraits" that compare a city to the surrounding county and
containing state on one set of axes.
}
\details{
When `per_capita = TRUE`, rows whose government has no observed
population in that year (`pop_source == "unavailable"`) are dropped from
the result. The dropped govids are recorded in
`provenance$rollup$excluded_govids`. This excludes special districts
(gov type 4) and school districts (gov type 5) from per-capita rollups
by design — see `vignette('population-denominators')`.
}
\section{Reading `coverage`}{
`provenance$coverage` carries three per-year counters:
* `n_units_expected` -- how many governments you asked about.
* `n_units_collected` -- how many of those appear in the corpus at all
that year (in ANY category), separating sampling from real zeros.
* `n_units_reporting` -- how many have rows for the SPECIFIC category you
requested. This is always <= n_units_collected: a government can be
collected but have no rows for "Police" because it contracts policing
to the county sheriff, not because it wasn't surveyed.
**`n_units_reporting` is category-conditional** and therefore **not a
response rate**: `n_units_reporting / n_units_expected` conflates sampling
(never collected) with real zeros (collected but spends nothing in your
category). Use `n_units_collected / n_units_expected` for the true
collection rate, and `n_units_reporting / n_units_collected` for category
participation among collected units.
In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
`category = "Police"`; the 174-city gap is overwhelmingly cities that
contract policing to the county sheriff, not non-response.
The comparison that *is* valid is the same category across a census year
(ending in 2 or 7) and a sample year, where the real-zero component is
roughly constant and the difference reflects the survey cycle. `is_census_year`
marks which is which.
}