uscogdata/specs/2026-04-29-per-year-population-denominator-design.md

# Per-year population denominators in uscogdata

**Date:** 2026-04-29
**Status:** Design — pending implementation
**Scope:** uscogdata 0.1 (pre-release; no version bump)
**Related:** cog_pipeline (data dictionary updates)

## Problem

`uscogdata::cog_spending(per_capita = TRUE)` and `cog_revenue(per_capita = TRUE)`
currently divide every year's nominal amount by a single static population value
— `canonical_fips_xwalk.population_acs`, the ACS 2018-2022 5-year estimate.

For a 24-year corpus (2000–2023) this introduces a systematic bias proportional
to each government's population change over that span. Fast-growing places have
their early-year per-capita numbers understated; shrinking places have theirs
overstated. The bias commonly exceeds 20% and can exceed 50% for cities like
Detroit. Provenance currently advertises this denominator explicitly, so the
error is visible to careful users — but the default behavior produces wrong
numbers.

`cog_geographic_rollup()` has the same bug. `cog_find_peers()` /
`cog_peer_compare()` use the same static value to define peer cohorts, which
is defensible for matching but is no longer necessary now that per-year
population is available.

## Background — population sources

| Source | What it is | Where it lives |
|---|---|---|
| **Census F-33 `population`** | Population value Census uses on each COG row to compute its own per-capita tables. Almost always a Population Estimates Program (PEP) estimate; sometimes lagged a year for fiscal-year alignment, recorded in `popyear` | `long.population`, `long.popyear` (per row) |
| **PEP** (raw) | Census Bureau's official annual intercensal estimates. Distinct from F-33 because F-33 sometimes uses a lagged vintage | Not in corpus; available via tidycensus |
| **ACS 5-year** | American Community Survey 5-year rolling average. Different methodology, includes margin of error, only available 2005-2009 onward | `canonical_fips_xwalk.population_acs` (one fixed vintage) |
| **Decennial** | Actual count, every 10 years | Not in corpus |

F-33 `population` is the right default: it's what Census itself uses, so per-
capita results published by uscogdata reconcile with Census's own published
tables.

## Approach

Use the per-row `population` already present in `long`, joined on
`(canonical_govid, year)`. No new external data dependency. Coverage:

- **Types 0–3** (state, county, city, township): observed every year by design
- **Types 4–5** (special districts, schools): always NA — masked in
  `cog_pipeline/R/read_modern.R` because the F-33 schema does not carry a
  population value for these gov types

Type-4 and type-5 govs return `NA` per-capita with a `pop_source = "unavailable"`
flag and a note. No silent substitution.

The architecture leaves the door open for future denominators (PEP, ACS,
decennial) by surfacing `pop_source` as a first-class result column. Adding a
new source later is a join change, not an API change.

## Detailed design

### New view: `gov_population_yearly`

```sql
-- inst/sql/32-gov_population_yearly.sql
CREATE OR REPLACE VIEW gov_population_yearly AS
SELECT DISTINCT
  year,
  canonical_govid,
  population,
  popyear
FROM long
WHERE population IS NOT NULL;
```

`SELECT DISTINCT` collapses the metadata column duplicated across each gov-year's
item rows. A test asserts `(year, canonical_govid)` is unique to catch any
future source-data divergence.

### `cog_spending()` and `cog_revenue()`

`.attach_per_capita()` (in `R/spending.R`) is rewritten to:

1. Query `gov_population_yearly` for the requested govids and years.
2. `LEFT JOIN` on `(canonical_govid, year)` so missing rows produce NA.
3. Compute `amt_per_capita_nominal = amt_nominal / population`. NA when
   population is NA.
4. Drop `population` from the returned tibble (keep `pop_source` instead).

Result tibble gains one new column when `per_capita = TRUE`:

- `pop_source`: `"census_f33"` when a denominator was found, `"unavailable"`
  when NA.

`notes` is extended: when `pop_source == "unavailable"`, append
`"No population denominator available for this gov type"`. The `notes` column
is updated to concatenate multiple notes with `"; "` (it currently holds at
most one).

`amt_per_capita_real` is NA whenever `amt_per_capita_nominal` is NA.

### `cog_geographic_rollup()`

The current implementation does **not** sum amounts within a layer — it returns
one row per `(year, canonical_govid, subtype, category)` tagged with its
layer, intended for side-by-side "place portrait" comparisons (a city, the
county containing it, the state containing both). That semantics is preserved.

The only behavior change in this work is per-row exclusion when `per_capita = TRUE`:

1. After `cog_spending()` returns with the per-row per-year denominator from
   Task 3, drop rows where `pop_source == "unavailable"` so the result never
   contains NA per-capita rows.
2. Record the dropped `canonical_govid`s in `provenance$rollup$excluded_govids`
   and the kept ones in `provenance$rollup$included_govids`.

Documentation states explicitly: *Per-capita rollups include only governments
observed in both the finance and population panels for the given year. Special
districts and school districts (gov types 4 and 5) are therefore excluded from
per-capita rollups by design.*

Provenance gains:

- `rollup.included_govids` — `canonical_govid`s present in the result
- `rollup.excluded_govids` — `canonical_govid`s dropped for missing pop

### `cog_find_peers()`

Signature: `cog_find_peers(target_govid, year = NULL, pop_range = c(0.5, 2), ...)`

- `year` is a single integer. When `NULL`, defaults to the most recent year
  present in `gov_population_yearly` for the target.
- Looks up target's `population` at `year`. Errors if NA, with a message
  listing nearby years where target *is* observed.
- Filters candidates by `gov_population_yearly.population` at the same `year`,
  within `pop_range[1] * target_pop` and `pop_range[2] * target_pop`.
- Orders by `|log(pop_ratio)|` ascending.

Returned columns: `canonical_govid`, `gov_name`, `govs_type`, `fips_state`,
`population`, `pop_ratio`, `rank`. The column previously named `population_acs`
is renamed to `population`.

The cohort year is attached as a tibble attribute: `attr(x, "cohort_year")`.

### `cog_peer_compare()`

Existing signature unchanged:
`cog_peer_compare(target_govid, peers, category, years, per_capita = TRUE, adjust_to_year = NULL)`.
The caller supplies `peers` (either a `cog_find_peers()` result tibble or a
character vector of `canonical_govid`). The cohort year is implicit in
whichever year the caller used to call `cog_find_peers()`.

Behavior changes:

- When `peers` is a tibble carrying `attr(peers, "cohort_year")`,
  `cog_peer_compare()` reads it and stamps every result row with a constant
  `cohort_year` column.
- When `peers` is a bare character vector, `cohort_year` in the result is `NA`.
- Provenance gets `cohort_year` (scalar or NA) and the cohort govids list.

Users who want time-varying cohorts call `cog_find_peers()` per year and
stitch the `cog_peer_compare()` results themselves — documented in the
vignette with a worked example.

### Provenance updates

`provenance$transformations$per_capita` becomes:

```r
list(
  applied = TRUE,
  denominator_source = "Census F-33 population (per-year, from long.population)",
  popyear_range = c(<min>, <max>),
  pop_source_counts = list(census_f33 = N1, unavailable = N2)
)
```

For peer compare results, additional provenance:

```r
list(
  cohort_year = <int>,
  cohort_govids = <character>,
  pop_range = c(<lo>, <hi>)
)
```

For rollup results, additional provenance:

```r
list(
  rollup = list(
    included_govids = <character>,
    excluded_govids = <character>
  )
)
```

`R/explain.R` is updated to render the new fields.

### Documentation

**New vignette** `vignettes/population-denominators.Rmd`:

1. The four population sources explained
2. Why F-33 is the default — and how it reconciles with Census's own per-capita
   tables
3. The `popyear` quirk: Census sometimes uses a lagged estimate for fiscal-year
   alignment. Recorded in provenance, not in the result.
4. Worked example showing the bias from the old static-ACS approach versus
   per-year F-33 (e.g., Detroit 2003 vs. 2023)
5. Worked example of a rolling-cohort peer comparison built by looping
   `cog_peer_compare()` per year
6. Future direction: `pop_source` is structured so PEP, ACS time-series, or
   decennial denominators can be added later without API changes

**`cog_pipeline/docs/data_dictionary.md`** entry for `long.population` and
`long.popyear`: definition, source (F-33 fixed-width files, byte ranges),
type-4/5 masking rule, relationship to PEP.

### Tests

- `gov_population_yearly` returns one row per `(year, canonical_govid)` (uniqueness)
- `cog_spending(per_capita = TRUE)` returns different denominators for
  different years for a known gov in the fixture (use any gov whose population
  changes between 2019 and 2020)
- Type-4 and type-5 govids in the fixture return `pop_source = "unavailable"`
  and `NA` per-capita with the expected note
- `cog_geographic_rollup(per_capita = TRUE)` excludes missing-pop govs and
  records them in provenance
- `cog_find_peers()` defaults `year` to the most recent year for a target
  with known population history
- `cog_find_peers()` errors with a helpful message when target has no observed
  population in the requested year
- `cog_peer_compare()` defaults `cohort_year` and produces a result with a
  constant `cohort_year` column
- Provenance carries `denominator_source`, `popyear_range`, and
  `pop_source_counts`
- Regression test against a fixed govid+year showing the new per-capita value
  differs from the old (static-ACS) by exactly the ratio of `population_acs`
  to `long.population` for that gov-year

### Migration

Pre-release; no version bump. `NEWS.md` Unreleased entry:

> **Per-capita denominators now use per-year Census F-33 population.**
> Previously, `cog_spending()` and `cog_revenue()` divided all years' amounts
> by a single ACS 2018-2022 population, producing biased per-capita values
> for time-series. They now divide by the F-33 `population` recorded for each
> gov-year. Type-4 (special districts) and type-5 (school districts) govs
> return `NA` per-capita with `pop_source = "unavailable"`.
>
> **Peer matching now uses per-year population.** `cog_find_peers()` gains a
> `year` argument (defaults to most recent observed year). `cog_peer_compare()`
> gains `cohort_year`. Cohorts are still fixed for a single peer-compare call;
> users wanting moving cohorts loop themselves.
>
> **Rollups exclude govs with missing population.** `cog_geographic_rollup()`
> per-capita totals include only govs where both the finance variable and
> population are observed in that year; excluded govids are recorded in
> provenance.
>
> Returned column `population_acs` from `cog_find_peers()` is renamed to
> `population` and reflects the cohort-year vintage.

### File impact

| File | Change |
|---|---|
| `inst/sql/32-gov_population_yearly.sql` | New |
| `R/spending.R` (`.attach_per_capita`, `.notes_column`) | Per-year join, `pop_source`, multi-note concat |
| `R/peers.R` (`cog_find_peers`, `cog_peer_compare`) | `year` / `cohort_year` args, query new view, column rename |
| `R/rollup.R` | Skip-with-record for missing-pop govs |
| `R/provenance.R` | New denominator/cohort/rollup fields |
| `R/explain.R` | Render new fields |
| `vignettes/population-denominators.Rmd` | New |
| `tests/testthat/` | Per-year denominator, type-4/5, rollup exclusion, peer cohort, provenance |
| `cog_pipeline/docs/data_dictionary.md` | Document `long.population`, `long.popyear`, masking |
| `NEWS.md` | Unreleased entry |

## Out of scope

- PEP/ACS/decennial denominators — architected for, not implemented
- `per_pupil` denominator using `long.enrollment` for type-5 — deferred
- Covering-county fallback for type-4 — deliberately not done
- Backfilling population for type-4/5 from any external source
- Changes to `cog_explorer` callers — separate follow-up, after this lands