2 Commits

Include renames

Author	SHA1	Message	Date
jaredandClaude Sonnet 5	b41d5ee2aa	feat(coverage): n_units_collected separates sampling from real zeros (#36) ... provenance$coverage's n_units_reporting is category-conditional: it counts governments with rows for the SPECIFIC requested category, which conflates two different things -- a government never collected that year (sampling), and one collected but genuinely spending nothing in that category (a real zero). FY2012 Georgia Police is the motivating case from the issue: a complete census year reads as a 69% "response rate" because most of the gap is cities that contract policing to the county sheriff, not non-response. Adds a second counter, n_units_collected: how many of the caller's expected cohort appear in the corpus that year for ANY category. n_units_collected / n_units_expected is the true collection rate; n_units_reporting / n_units_collected is category participation among collected units. cog_geographic_rollup() and cog_peer_compare() both carry it; cog_explain() prints it alongside n_units_reporting. Two real bugs caught and fixed while finishing this (both against the already-written, previously-uncommitted draft): - .coverage_table()'s candidate list for the collection query was derived from the category-filtered result rows, not the caller's full expected cohort. A government with zero rows in the requested category across every requested year never appears in that result, so it was silently excluded from n_units_collected too -- collapsing the new counter back to the old, broken one for exactly the governments it exists to count. Fixed by threading an explicit `expected_ids` (all_govids / peer_govids) through instead. - The collection query hardcoded long_view = "spending_long_harmonized", which does not exist on a corpus with schema_version < 5 (R/basis.R resolves basis = "raw" there; R/views.R only registers the harmonized views on v5+). cog_geographic_rollup()/cog_peer_compare() would hard-error on a corpus vintage the package otherwise explicitly supports. Fixed by deriving long_view from the basis cog_spending() actually resolved (prov$basis) via the existing .select_long_view() helper, matching how every other basis-aware query in the package already does this. Also: cog_explain()'s general "complete census only in years ending in 2 or 7" footnote was gated on the OLD counter's absence, making it permanently unreachable now that both callers always supply the new one -- ungated it, since the explanation is orthogonal to which counter set is present. Dropped a dead conditional branch, fixed two stale roxygen blocks in R/peers.R/R/rollup.R still describing the old two-counter model, fixed the same staleness in README.md, and switched two `uscogdata:::` self-references to the package's own convention of calling internal helpers unqualified. 1101 tests pass (2 skipped live-corpus), including new direct regression tests for both bugs above (one exercising a government collected-but-absent from a category result, one running the full rollup/peer-compare path against a doctored schema_version 4 corpus). Reviewed by an independent code-reviewer pass (1 HIGH, 1 MEDIUM, 3 LOW -- all addressed above). Closes #36. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 11:46:17 -04:00
jared	d95c9032c5	feat: coverage argument + always-on reporting-coverage metadata (#13) ... The Census of Governments is a complete census only in years ending in 2 and 7. Every other year is a sample, and the sample varies enormously. Neither cog_geographic_rollup() nor cog_peer_compare()/cog_find_peers() had any concept of "the universe": each summed or labelled whichever govids happened to have rows and returned that with nothing distinguishing "every government reported" from "a fifth of them did". On the bundled fixture, Wisconsin's 608-city universe rolls up 597 governments in FY2012 and 112 in FY2019. The peer side is worse exposure, not better: a Madison-scale cohort looks stable because Madison is large, while governments matched to a small target sit in exactly the population band the sample cycle hits hardest. Chilton's 15-peer cohort reports 15 of 15 in FY2012 and 3 of 15 in FY2019. Implements the owner's settled design: coverage = c("all", "census", "consistent") on all three verbs, defaulting to "all" so nothing currently calling them changes, PLUS always-on provenance$coverage carrying per-year n_units_reporting / n_units_expected / is_census_year and provenance$coverage_mode. cog_explain() prints a "Reporting coverage" section. The default mode can no longer mislead silently, which is the point -- using these verbs correctly must not require knowing the survey calendar. Decisions worth stating: - n_units_expected is the universe the CALLER named, not the national one. That is what makes the ratio mean something: "597 of the 608 Wisconsin cities you asked about". For peers it is the cohort size, counted over peer rows only -- including the target would inflate every count by one and make a cohort that has entirely stopped reporting look non-empty. - The coverage table is built from the REQUESTED years, not the years present in the result, so a year in which nothing reported still appears with n_units_reporting = 0. A year that vanishes silently is precisely the disclosure failure at issue. - "census" filters years BEFORE the query, and aborts when the range holds no census year rather than returning an empty result for a query the caller believes they made. - "consistent" exempts the peer-comparison target: it is the subject of the comparison, not a member of the cohort being balanced, and dropping it would leave nothing to compare. The summary_* quantiles are computed AFTER the filter so they describe the cohort actually returned. - is_census_year is documented as a statement about the survey CALENDAR, never a claim of completeness -- FY1967 is a census year in which only 97 of Wisconsin's 608 cities report (DoD 3). n_units_reporting is the number that tells the truth. On cog_find_peers(), where there is no year range, coverage governs the cohort VINTAGE: "census" snaps to the most recent census year with an observed population, so a cohort is not built from a sample year in which most of the candidate universe is absent. "consistent" is a comparison-time concept and selects like "all" there, carried on the result for cog_peer_compare(). One fix to the committed test, which was internally inconsistent. It pinned n_units_reporting == 597 for FY2012 AND asserted that number equals a raw cross-check that answers 595. Both numbers are right for different questions: VERNON VILLAGE and WAUKESHA VILLAGE carry type = 3 in `long` (their as-of-year identity, as townships) while the xwalk lists them as govs_type = 2 (their present identity, as villages) -- schema v6 made the long table's geography present-harmonized but `type` still reads as-of-year. The rollup counts against the requested govid set, so 597 answers "how many of the governments I asked about reported". The cross-check now scopes to that same universe instead of to long.type/long.fips_state; it still reads raw parquet rather than going through the verb under test. Suite: 670 pass / 0 fail / 2 skip (was 658/0/3). rcmdcheck clean. The two remaining skips are #11 and #12.	2026-07-30 11:57:11 -04:00