Closes#13 (findings F-020, F-023). Stacked on #23 — retarget to main once that merges. Unblocks cog-api#8.
The problem
The Census of Governments is a complete census only in years ending in 2 and 7. Every other year is a sample, and the sample varies enormously. Neither verb had any concept of "the universe" — each summed or labelled whichever govids happened to have rows and returned that with nothing distinguishing "every government reported" from "a fifth of them did".
On the bundled fixture, Wisconsin's 608-city universe:
year
reporting
census year?
2011
152
no
2012
597
yes
2019
112
no
2020
114
no
The peer side is worse exposure, not better: a Madison-scale cohort looks stable because Madison is large, while governments matched to a small target sit in exactly the population band the sample cycle hits hardest. Chilton's 15-peer cohort reports 15 of 15 in FY2012 and 3 of 15 in FY2019.
The fix
coverage = c("all", "census", "consistent") on all three verbs, defaulting to "all" so nothing currently calling them changes — plus always-onprovenance$coverage (per-year n_units_reporting, n_units_expected, is_census_year) and provenance$coverage_mode. cog_explain() prints a "Reporting coverage" section.
The default mode can no longer mislead silently, which is the point: using these verbs correctly must not require knowing the survey calendar.
"consistent" on that Wisconsin rollup yields a proper balanced panel — 85 governments in all four years.
Decisions worth stating
n_units_expected is the universe the caller named, not the national one — that is what makes the ratio mean something ("597 of the 608 you asked about"). For peers it is the cohort size, counted over peer rows only; including the target would inflate every count by one and make a cohort that has entirely stopped reporting look non-empty.
The table is built from the requested years, not the years present in the result, so a year in which nothing reported still appears with n_units_reporting = 0. A year that vanishes silently is precisely the disclosure failure at issue.
"census" filters years before the query and aborts when the range holds none, rather than returning an empty result for a query the caller believes they made.
"consistent" exempts the peer target — subject of the comparison, not a member of the cohort being balanced — and the summary_* quantiles are computed after the filter so they describe the cohort actually returned.
is_census_year is about the survey calendar, never completeness (DoD 3): FY1967 is a census year in which only 97 of Wisconsin's 608 cities report.
One fix to the committed test — it was internally inconsistent
It pinned n_units_reporting == 597 for FY2012 and asserted that number equals a raw cross-check which answers 595. Both are right, for different questions:
VERNON VILLAGE and WAUKESHA VILLAGE carry type = 3 in long (their as-of-year identity, as townships) while the xwalk lists them as govs_type = 2 (their present identity, as villages). Schema v6 made the long table's geography present-harmonized but type still reads as-of-year.
The rollup counts against the requested govid set, so 597 answers "how many of the governments I asked about reported". The cross-check now scopes to that same universe instead of to long.type/long.fips_state — still reading raw parquet rather than going through the verb under test.
Closes #13 (findings F-020, F-023). **Stacked on #23** — retarget to `main` once that merges. Unblocks **cog-api#8**.
## The problem
The Census of Governments is a complete census **only in years ending in 2 and 7**. Every other year is a sample, and the sample varies enormously. Neither verb had any concept of "the universe" — each summed or labelled whichever govids happened to have rows and returned that with nothing distinguishing *"every government reported"* from *"a fifth of them did"*.
On the bundled fixture, Wisconsin's 608-city universe:
| year | reporting | census year? |
|---:|---:|---|
| 2011 | 152 | no |
| 2012 | **597** | **yes** |
| 2019 | 112 | no |
| 2020 | 114 | no |
The peer side is worse exposure, not better: a Madison-scale cohort looks stable *because Madison is large*, while governments matched to a small target sit in exactly the population band the sample cycle hits hardest. Chilton's 15-peer cohort reports 15 of 15 in FY2012 and **3 of 15** in FY2019.
## The fix
`coverage = c("all", "census", "consistent")` on all three verbs, defaulting to `"all"` so nothing currently calling them changes — **plus always-on** `provenance$coverage` (per-year `n_units_reporting`, `n_units_expected`, `is_census_year`) and `provenance$coverage_mode`. `cog_explain()` prints a "Reporting coverage" section.
The default mode can no longer mislead silently, which is the point: using these verbs correctly must not require knowing the survey calendar.
`"consistent"` on that Wisconsin rollup yields a proper balanced panel — 85 governments in all four years.
## Decisions worth stating
- **`n_units_expected` is the universe the caller named**, not the national one — that is what makes the ratio mean something ("597 of the 608 you asked about"). For peers it is the cohort size, counted over peer rows only; including the target would inflate every count by one and make a cohort that has entirely stopped reporting look non-empty.
- **The table is built from the requested years**, not the years present in the result, so a year in which nothing reported still appears with `n_units_reporting = 0`. A year that vanishes silently is precisely the disclosure failure at issue.
- **`"census"` filters years before the query** and aborts when the range holds none, rather than returning an empty result for a query the caller believes they made.
- **`"consistent"` exempts the peer target** — subject of the comparison, not a member of the cohort being balanced — and the `summary_*` quantiles are computed *after* the filter so they describe the cohort actually returned.
- **`is_census_year` is about the survey calendar, never completeness** (DoD 3): FY1967 is a census year in which only 97 of Wisconsin's 608 cities report.
## One fix to the committed test — it was internally inconsistent
It pinned `n_units_reporting == 597` for FY2012 **and** asserted that number equals a raw cross-check which answers **595**. Both are right, for different questions:
`VERNON VILLAGE` and `WAUKESHA VILLAGE` carry `type = 3` in `long` (their as-of-year identity, as townships) while the xwalk lists them as `govs_type = 2` (their present identity, as villages). Schema v6 made the long table's geography present-harmonized but `type` still reads as-of-year.
The rollup counts against the **requested govid set**, so 597 answers "how many of the governments I asked about reported". The cross-check now scopes to that same universe instead of to `long.type`/`long.fips_state` — still reading raw parquet rather than going through the verb under test.
## Verification
| | before | after |
|---|---|---|
| suite | 658 / 0 / 3 | **670 / 0 / 2** |
| `rcmdcheck` | clean | **0 / 0 / 0** |
The two remaining skips are #11 and #12.
jared
changed target branch from feat/complete-argument-18 to main2026-07-30 12:06:42 -04:00
The Census of Governments is a complete census only in years ending in 2 and
7. Every other year is a sample, and the sample varies enormously. Neither
cog_geographic_rollup() nor cog_peer_compare()/cog_find_peers() had any
concept of "the universe": each summed or labelled whichever govids happened
to have rows and returned that with nothing distinguishing "every government
reported" from "a fifth of them did".
On the bundled fixture, Wisconsin's 608-city universe rolls up 597
governments in FY2012 and 112 in FY2019. The peer side is worse exposure, not
better: a Madison-scale cohort looks stable because Madison is large, while
governments matched to a small target sit in exactly the population band the
sample cycle hits hardest. Chilton's 15-peer cohort reports 15 of 15 in
FY2012 and 3 of 15 in FY2019.
Implements the owner's settled design: coverage = c("all", "census",
"consistent") on all three verbs, defaulting to "all" so nothing currently
calling them changes, PLUS always-on provenance$coverage carrying per-year
n_units_reporting / n_units_expected / is_census_year and
provenance$coverage_mode. cog_explain() prints a "Reporting coverage"
section. The default mode can no longer mislead silently, which is the point
-- using these verbs correctly must not require knowing the survey calendar.
Decisions worth stating:
- n_units_expected is the universe the CALLER named, not the national one.
That is what makes the ratio mean something: "597 of the 608 Wisconsin
cities you asked about". For peers it is the cohort size, counted over
peer rows only -- including the target would inflate every count by one
and make a cohort that has entirely stopped reporting look non-empty.
- The coverage table is built from the REQUESTED years, not the years
present in the result, so a year in which nothing reported still appears
with n_units_reporting = 0. A year that vanishes silently is precisely
the disclosure failure at issue.
- "census" filters years BEFORE the query, and aborts when the range holds
no census year rather than returning an empty result for a query the
caller believes they made.
- "consistent" exempts the peer-comparison target: it is the subject of the
comparison, not a member of the cohort being balanced, and dropping it
would leave nothing to compare. The summary_* quantiles are computed
AFTER the filter so they describe the cohort actually returned.
- is_census_year is documented as a statement about the survey CALENDAR,
never a claim of completeness -- FY1967 is a census year in which only 97
of Wisconsin's 608 cities report (DoD 3). n_units_reporting is the number
that tells the truth.
On cog_find_peers(), where there is no year range, coverage governs the
cohort VINTAGE: "census" snaps to the most recent census year with an
observed population, so a cohort is not built from a sample year in which
most of the candidate universe is absent. "consistent" is a comparison-time
concept and selects like "all" there, carried on the result for
cog_peer_compare().
One fix to the committed test, which was internally inconsistent. It pinned
n_units_reporting == 597 for FY2012 AND asserted that number equals a raw
cross-check that answers 595. Both numbers are right for different questions:
VERNON VILLAGE and WAUKESHA VILLAGE carry type = 3 in `long` (their
as-of-year identity, as townships) while the xwalk lists them as govs_type =
2 (their present identity, as villages) -- schema v6 made the long table's
geography present-harmonized but `type` still reads as-of-year. The rollup
counts against the requested govid set, so 597 answers "how many of the
governments I asked about reported". The cross-check now scopes to that same
universe instead of to long.type/long.fips_state; it still reads raw parquet
rather than going through the verb under test.
Suite: 670 pass / 0 fail / 2 skip (was 658/0/3). rcmdcheck clean.
The two remaining skips are #11 and #12.
jared
merged commit 915a4d0678 into main2026-07-30 12:06:53 -04:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #13 (findings F-020, F-023). Stacked on #23 — retarget to
mainonce that merges. Unblocks cog-api#8.The problem
The Census of Governments is a complete census only in years ending in 2 and 7. Every other year is a sample, and the sample varies enormously. Neither verb had any concept of "the universe" — each summed or labelled whichever govids happened to have rows and returned that with nothing distinguishing "every government reported" from "a fifth of them did".
On the bundled fixture, Wisconsin's 608-city universe:
The peer side is worse exposure, not better: a Madison-scale cohort looks stable because Madison is large, while governments matched to a small target sit in exactly the population band the sample cycle hits hardest. Chilton's 15-peer cohort reports 15 of 15 in FY2012 and 3 of 15 in FY2019.
The fix
coverage = c("all", "census", "consistent")on all three verbs, defaulting to"all"so nothing currently calling them changes — plus always-onprovenance$coverage(per-yearn_units_reporting,n_units_expected,is_census_year) andprovenance$coverage_mode.cog_explain()prints a "Reporting coverage" section.The default mode can no longer mislead silently, which is the point: using these verbs correctly must not require knowing the survey calendar.
"consistent"on that Wisconsin rollup yields a proper balanced panel — 85 governments in all four years.Decisions worth stating
n_units_expectedis the universe the caller named, not the national one — that is what makes the ratio mean something ("597 of the 608 you asked about"). For peers it is the cohort size, counted over peer rows only; including the target would inflate every count by one and make a cohort that has entirely stopped reporting look non-empty.n_units_reporting = 0. A year that vanishes silently is precisely the disclosure failure at issue."census"filters years before the query and aborts when the range holds none, rather than returning an empty result for a query the caller believes they made."consistent"exempts the peer target — subject of the comparison, not a member of the cohort being balanced — and thesummary_*quantiles are computed after the filter so they describe the cohort actually returned.is_census_yearis about the survey calendar, never completeness (DoD 3): FY1967 is a census year in which only 97 of Wisconsin's 608 cities report.One fix to the committed test — it was internally inconsistent
It pinned
n_units_reporting == 597for FY2012 and asserted that number equals a raw cross-check which answers 595. Both are right, for different questions:VERNON VILLAGEandWAUKESHA VILLAGEcarrytype = 3inlong(their as-of-year identity, as townships) while the xwalk lists them asgovs_type = 2(their present identity, as villages). Schema v6 made the long table's geography present-harmonized buttypestill reads as-of-year.The rollup counts against the requested govid set, so 597 answers "how many of the governments I asked about reported". The cross-check now scopes to that same universe instead of to
long.type/long.fips_state— still reading raw parquet rather than going through the verb under test.Verification
rcmdcheckThe two remaining skips are #11 and #12.
The Census of Governments is a complete census only in years ending in 2 and 7. Every other year is a sample, and the sample varies enormously. Neither cog_geographic_rollup() nor cog_peer_compare()/cog_find_peers() had any concept of "the universe": each summed or labelled whichever govids happened to have rows and returned that with nothing distinguishing "every government reported" from "a fifth of them did". On the bundled fixture, Wisconsin's 608-city universe rolls up 597 governments in FY2012 and 112 in FY2019. The peer side is worse exposure, not better: a Madison-scale cohort looks stable because Madison is large, while governments matched to a small target sit in exactly the population band the sample cycle hits hardest. Chilton's 15-peer cohort reports 15 of 15 in FY2012 and 3 of 15 in FY2019. Implements the owner's settled design: coverage = c("all", "census", "consistent") on all three verbs, defaulting to "all" so nothing currently calling them changes, PLUS always-on provenance$coverage carrying per-year n_units_reporting / n_units_expected / is_census_year and provenance$coverage_mode. cog_explain() prints a "Reporting coverage" section. The default mode can no longer mislead silently, which is the point -- using these verbs correctly must not require knowing the survey calendar. Decisions worth stating: - n_units_expected is the universe the CALLER named, not the national one. That is what makes the ratio mean something: "597 of the 608 Wisconsin cities you asked about". For peers it is the cohort size, counted over peer rows only -- including the target would inflate every count by one and make a cohort that has entirely stopped reporting look non-empty. - The coverage table is built from the REQUESTED years, not the years present in the result, so a year in which nothing reported still appears with n_units_reporting = 0. A year that vanishes silently is precisely the disclosure failure at issue. - "census" filters years BEFORE the query, and aborts when the range holds no census year rather than returning an empty result for a query the caller believes they made. - "consistent" exempts the peer-comparison target: it is the subject of the comparison, not a member of the cohort being balanced, and dropping it would leave nothing to compare. The summary_* quantiles are computed AFTER the filter so they describe the cohort actually returned. - is_census_year is documented as a statement about the survey CALENDAR, never a claim of completeness -- FY1967 is a census year in which only 97 of Wisconsin's 608 cities report (DoD 3). n_units_reporting is the number that tells the truth. On cog_find_peers(), where there is no year range, coverage governs the cohort VINTAGE: "census" snaps to the most recent census year with an observed population, so a cohort is not built from a sample year in which most of the candidate universe is absent. "consistent" is a comparison-time concept and selects like "all" there, carried on the result for cog_peer_compare(). One fix to the committed test, which was internally inconsistent. It pinned n_units_reporting == 597 for FY2012 AND asserted that number equals a raw cross-check that answers 595. Both numbers are right for different questions: VERNON VILLAGE and WAUKESHA VILLAGE carry type = 3 in `long` (their as-of-year identity, as townships) while the xwalk lists them as govs_type = 2 (their present identity, as villages) -- schema v6 made the long table's geography present-harmonized but `type` still reads as-of-year. The rollup counts against the requested govid set, so 597 answers "how many of the governments I asked about reported". The cross-check now scopes to that same universe instead of to long.type/long.fips_state; it still reads raw parquet rather than going through the verb under test. Suite: 670 pass / 0 fail / 2 skip (was 658/0/3). rcmdcheck clean. The two remaining skips are #11 and #12.