uscogdata/specs/2026-04-29-per-year-population-denominator-design.md

Per-year population denominators in uscogdata

Date: 2026-04-29 Status: Design — pending implementation Scope: uscogdata 0.1 (pre-release; no version bump) Related: cog_pipeline (data dictionary updates)

Problem

uscogdata::cog_spending(per_capita = TRUE) and cog_revenue(per_capita = TRUE) currently divide every year's nominal amount by a single static population value — canonical_fips_xwalk.population_acs, the ACS 2018-2022 5-year estimate.

For a 24-year corpus (2000–2023) this introduces a systematic bias proportional to each government's population change over that span. Fast-growing places have their early-year per-capita numbers understated; shrinking places have theirs overstated. The bias commonly exceeds 20% and can exceed 50% for cities like Detroit. Provenance currently advertises this denominator explicitly, so the error is visible to careful users — but the default behavior produces wrong numbers.

cog_geographic_rollup() has the same bug. cog_find_peers() / cog_peer_compare() use the same static value to define peer cohorts, which is defensible for matching but is no longer necessary now that per-year population is available.

Background — population sources

Source	What it is	Where it lives
Census F-33 population	Population value Census uses on each COG row to compute its own per-capita tables. Almost always a Population Estimates Program (PEP) estimate; sometimes lagged a year for fiscal-year alignment, recorded in popyear	long.population, long.popyear (per row)
PEP (raw)	Census Bureau's official annual intercensal estimates. Distinct from F-33 because F-33 sometimes uses a lagged vintage	Not in corpus; available via tidycensus
ACS 5-year	American Community Survey 5-year rolling average. Different methodology, includes margin of error, only available 2005-2009 onward	canonical_fips_xwalk.population_acs (one fixed vintage)
Decennial	Actual count, every 10 years	Not in corpus

F-33 population is the right default: it's what Census itself uses, so per- capita results published by uscogdata reconcile with Census's own published tables.

Approach

Use the per-row population already present in long, joined on (canonical_govid, year). No new external data dependency. Coverage:

• Types 0–3 (state, county, city, township): observed every year by design
• Types 4–5 (special districts, schools): always NA — masked in cog_pipeline/R/read_modern.R because the F-33 schema does not carry a population value for these gov types

Type-4 and type-5 govs return NA per-capita with a pop_source = "unavailable" flag and a note. No silent substitution.

The architecture leaves the door open for future denominators (PEP, ACS, decennial) by surfacing pop_source as a first-class result column. Adding a new source later is a join change, not an API change.

Detailed design

New view: gov_population_yearly

-- inst/sql/32-gov_population_yearly.sql
CREATE OR REPLACE VIEW gov_population_yearly AS
SELECT DISTINCT
  year,
  canonical_govid,
  population,
  popyear
FROM long
WHERE population IS NOT NULL;

SELECT DISTINCT collapses the metadata column duplicated across each gov-year's item rows. A test asserts (year, canonical_govid) is unique to catch any future source-data divergence.

cog_spending() and cog_revenue()

.attach_per_capita() (in R/spending.R) is rewritten to:

1. Query gov_population_yearly for the requested govids and years.
2. LEFT JOIN on (canonical_govid, year) so missing rows produce NA.
3. Compute amt_per_capita_nominal = amt_nominal / population. NA when population is NA.
4. Drop population from the returned tibble (keep pop_source instead).

Result tibble gains one new column when per_capita = TRUE:

• pop_source: "census_f33" when a denominator was found, "unavailable" when NA.

notes is extended: when pop_source == "unavailable", append "No population denominator available for this gov type". The notes column is updated to concatenate multiple notes with "; " (it currently holds at most one).

amt_per_capita_real is NA whenever amt_per_capita_nominal is NA.

cog_geographic_rollup()

The current implementation does not sum amounts within a layer — it returns one row per (year, canonical_govid, subtype, category) tagged with its layer, intended for side-by-side "place portrait" comparisons (a city, the county containing it, the state containing both). That semantics is preserved.

The only behavior change in this work is per-row exclusion when per_capita = TRUE:

1. After cog_spending() returns with the per-row per-year denominator from Task 3, drop rows where pop_source == "unavailable" so the result never contains NA per-capita rows.
2. Record the dropped canonical_govids in provenance$rollup$excluded_govids and the kept ones in provenance$rollup$included_govids.

Documentation states explicitly: Per-capita rollups include only governments observed in both the finance and population panels for the given year. Special districts and school districts (gov types 4 and 5) are therefore excluded from per-capita rollups by design.

Provenance gains:

• rollup.included_govids — canonical_govids present in the result
• rollup.excluded_govids — canonical_govids dropped for missing pop

cog_find_peers()

Signature: cog_find_peers(target_govid, year = NULL, pop_range = c(0.5, 2), ...)

• year is a single integer. When NULL, defaults to the most recent year present in gov_population_yearly for the target.
• Looks up target's population at year. Errors if NA, with a message listing nearby years where target is observed.
• Filters candidates by gov_population_yearly.population at the same year, within pop_range[1] * target_pop and pop_range[2] * target_pop.
• Orders by |log(pop_ratio)| ascending.

Returned columns: canonical_govid, gov_name, govs_type, fips_state, population, pop_ratio, rank. The column previously named population_acs is renamed to population.

The cohort year is attached as a tibble attribute: attr(x, "cohort_year").

cog_peer_compare()

Existing signature unchanged: cog_peer_compare(target_govid, peers, category, years, per_capita = TRUE, adjust_to_year = NULL). The caller supplies peers (either a cog_find_peers() result tibble or a character vector of canonical_govid). The cohort year is implicit in whichever year the caller used to call cog_find_peers().

Behavior changes:

• When peers is a tibble carrying attr(peers, "cohort_year"), cog_peer_compare() reads it and stamps every result row with a constant cohort_year column.
• When peers is a bare character vector, cohort_year in the result is NA.
• Provenance gets cohort_year (scalar or NA) and the cohort govids list.

Users who want time-varying cohorts call cog_find_peers() per year and stitch the cog_peer_compare() results themselves — documented in the vignette with a worked example.

Provenance updates

provenance$transformations$per_capita becomes:

list(
  applied = TRUE,
  denominator_source = "Census F-33 population (per-year, from long.population)",
  popyear_range = c(<min>, <max>),
  pop_source_counts = list(census_f33 = N1, unavailable = N2)
)

For peer compare results, additional provenance:

list(
  cohort_year = <int>,
  cohort_govids = <character>,
  pop_range = c(<lo>, <hi>)
)

For rollup results, additional provenance:

list(
  rollup = list(
    included_govids = <character>,
    excluded_govids = <character>
  )
)

R/explain.R is updated to render the new fields.

Documentation

New vignette vignettes/population-denominators.Rmd:

1. The four population sources explained
2. Why F-33 is the default — and how it reconciles with Census's own per-capita tables
3. The popyear quirk: Census sometimes uses a lagged estimate for fiscal-year alignment. Recorded in provenance, not in the result.
4. Worked example showing the bias from the old static-ACS approach versus per-year F-33 (e.g., Detroit 2003 vs. 2023)
5. Worked example of a rolling-cohort peer comparison built by looping cog_peer_compare() per year
6. Future direction: pop_source is structured so PEP, ACS time-series, or decennial denominators can be added later without API changes

cog_pipeline/docs/data_dictionary.md entry for long.population and long.popyear: definition, source (F-33 fixed-width files, byte ranges), type-4/5 masking rule, relationship to PEP.

Tests

• gov_population_yearly returns one row per (year, canonical_govid) (uniqueness)
• cog_spending(per_capita = TRUE) returns different denominators for different years for a known gov in the fixture (use any gov whose population changes between 2019 and 2020)
• Type-4 and type-5 govids in the fixture return pop_source = "unavailable" and NA per-capita with the expected note
• cog_geographic_rollup(per_capita = TRUE) excludes missing-pop govs and records them in provenance
• cog_find_peers() defaults year to the most recent year for a target with known population history
• cog_find_peers() errors with a helpful message when target has no observed population in the requested year
• cog_peer_compare() defaults cohort_year and produces a result with a constant cohort_year column
• Provenance carries denominator_source, popyear_range, and pop_source_counts
• Regression test against a fixed govid+year showing the new per-capita value differs from the old (static-ACS) by exactly the ratio of population_acs to long.population for that gov-year

Migration

Pre-release; no version bump. NEWS.md Unreleased entry:

Per-capita denominators now use per-year Census F-33 population. Previously, cog_spending() and cog_revenue() divided all years' amounts by a single ACS 2018-2022 population, producing biased per-capita values for time-series. They now divide by the F-33 population recorded for each gov-year. Type-4 (special districts) and type-5 (school districts) govs return NA per-capita with pop_source = "unavailable".

Peer matching now uses per-year population. cog_find_peers() gains a year argument (defaults to most recent observed year). cog_peer_compare() gains cohort_year. Cohorts are still fixed for a single peer-compare call; users wanting moving cohorts loop themselves.

Rollups exclude govs with missing population. cog_geographic_rollup() per-capita totals include only govs where both the finance variable and population are observed in that year; excluded govids are recorded in provenance.

Returned column population_acs from cog_find_peers() is renamed to population and reflects the cohort-year vintage.

File impact

File	Change
inst/sql/32-gov_population_yearly.sql	New
R/spending.R (.attach_per_capita, .notes_column)	Per-year join, pop_source, multi-note concat
R/peers.R (cog_find_peers, cog_peer_compare)	year / cohort_year args, query new view, column rename
R/rollup.R	Skip-with-record for missing-pop govs
R/provenance.R	New denominator/cohort/rollup fields
R/explain.R	Render new fields
vignettes/population-denominators.Rmd	New
tests/testthat/	Per-year denominator, type-4/5, rollup exclusion, peer cohort, provenance
cog_pipeline/docs/data_dictionary.md	Document long.population, long.popyear, masking
NEWS.md	Unreleased entry

Out of scope

• PEP/ACS/decennial denominators — architected for, not implemented
• per_pupil denominator using long.enrollment for type-5 — deferred
• Covering-county fallback for type-4 — deliberately not done
• Backfilling population for type-4/5 from any external source
• Changes to cog_explorer callers — separate follow-up, after this lands