feat: 'All Categories' pseudo-category + n_units_reporting semantics #37

2026-08-05T12:04:52-04:00

jared commented

2026-08-05 12:04:52 -04:00

Closes cog-api#37 (reader half) and uscogdata#36.

Adds a reserved category = "All Categories" that returns one summed row
per (year, govid, subtype) across every category in the requested concept's
subtype scope. The concept boundary in this package is subtype, not category,
so the sum is defined by the subtype allowlist rather than by category
membership — which is what makes it a guarantee rather than a convenience.

Named 'All Categories' rather than 'Total' because category = "Total"
would sit one argument away from expenditure_concept = "total" and mean
something different.

Also documents that n_units_reporting is category-conditional and is not a
response rate (uscogdata#36).

Verified on the full 1967-2024 corpus across FY1972/2002/2022/2024, spanning
both the census-id and FIPS-id vintages:

FY1972  by_category 196,747,000    all_categories 196,747,000    MATCH
FY2002  by_category 1,471,819,000  all_categories 1,471,819,000  MATCH
FY2022  by_category 1,763,918,000  all_categories 1,763,918,000  MATCH
FY2024  by_category 2,708,239,000  all_categories 2,708,239,000  MATCH

Revenue verified too (Atlanta FY2022, expenditure_concept = "general"):
1,627,842,000 both ways.

Full test suite: 880 PASS / 0 FAIL / 0 SKIP / 0 WARN.

Version bumped to 0.2.0 (minor — adds public surface without breaking any
existing call). This is load-bearing: cog-api installs this package with
install_local(), which no-ops when the version already matches, so Phase 1
needs the bump to actually exercise the new reader instead of silently
testing against 0.1.0.

Closes cog-api#37 (reader half) and uscogdata#36. Adds a reserved `category = "All Categories"` that returns one summed row per (year, govid, subtype) across every category in the requested concept's subtype scope. The concept boundary in this package is subtype, not category, so the sum is defined by the subtype allowlist rather than by category membership — which is what makes it a guarantee rather than a convenience. Named 'All Categories' rather than 'Total' because `category = "Total"` would sit one argument away from `expenditure_concept = "total"` and mean something different. Also documents that `n_units_reporting` is category-conditional and is not a response rate (uscogdata#36). Verified on the full 1967-2024 corpus across FY1972/2002/2022/2024, spanning both the census-id and FIPS-id vintages: ``` FY1972 by_category 196,747,000 all_categories 196,747,000 MATCH FY2002 by_category 1,471,819,000 all_categories 1,471,819,000 MATCH FY2022 by_category 1,763,918,000 all_categories 1,763,918,000 MATCH FY2024 by_category 2,708,239,000 all_categories 2,708,239,000 MATCH ``` Revenue verified too (Atlanta FY2022, `expenditure_concept = "general"`): 1,627,842,000 both ways. Full test suite: 880 PASS / 0 FAIL / 0 SKIP / 0 WARN. Version bumped to 0.2.0 (minor — adds public surface without breaking any existing call). This is load-bearing: `cog-api` installs this package with `install_local()`, which no-ops when the version already matches, so Phase 1 needs the bump to actually exercise the new reader instead of silently testing against 0.1.0.

jared added 7 commits 2026-08-05 12:04:52 -04:00

feat: all-categories mode in .build_verb_sql() 5e22e940e7

The concept boundary in this package is subtype, not category, so a
total is the existing query with the category dimension collapsed and
no category predicate applied. subtype is deliberately kept in the
grouping: subtype=operations plus all-categories is 'operating
expenditure', which is the measure a fiscal comparison wants.

Named 'All Categories' rather than 'Total' because category='Total'
would sit one argument from expenditure_concept='total' and mean
something different.

feat: accept category = 'All Categories' on cog_spending/cog_revenue 11ae99c382

Returns one summed row per (year, govid, subtype) across every
category in the requested concept's subtype scope, so a caller never
sums categories client-side and cannot sum the wrong scope.

Combining it with other category names is an error rather than a
silent partial sum.

docs: fix false subtype= claim in All Categories roxygen (F1) 503fa6562f

The @param category text on cog_spending()/cog_revenue() told users to
"Combine with subtype = ..." but neither verb has a subtype argument.
Replace with accurate guidance: filter the returned frame's
spend_subtype/revenue_subtype column.

feat: advertise 'All Categories' from cog_categories() 12a9be110f

A reserved value nobody can discover is a trap, and this is the view
the API's /categories endpoint is built from. Emitted for the two flow
vocabularies only -- cog_balances() returns a stock and has no concept
to sum within.

Also fix test-categories.R to exclude pseudo-category rows from
crosswalk-specific assertions (one row per (category, subtype) pair,
non-empty item_codes, valid subtypes).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

test: prove 'All Categories' passes through the geographic rollup f1e9aa383a

Geographic totals are the expensive case cog-api#37 was filed about --
without this a caller issues one rollup per category and sums them.
The pass-through was expected to work by construction; this asserts it
rather than assuming it, including under per_capita and inflation
adjustment.

docs: n_units_reporting is category-conditional, not a response rate 44e9b40b86

Closes uscogdata#36. It counts governments with rows for the requested
category, so a surveyed government that genuinely spends nothing there
is indistinguishable from one never surveyed. In FY2022, a complete
census year, Georgia reports 393 of 567 cities for Police -- the gap is
cities that contract to the sheriff.

Documents the comparison that IS valid: same category, census year vs
sample year.

chore: release 0.2.0

R-CMD-check / check (push) Successful in 3m34s

Details

R-CMD-check / check (pull_request) Successful in 6m30s

Details

61b9c95731

Bumps the minor version because 'All Categories' adds public surface
without breaking any existing call.

The bump is load-bearing, not cosmetic: cog-api installs this package
with install_local(), which no-ops when the version already matches.
Without it, Phase 1 would silently test against the 0.1.0 reader and
pass while proving nothing.

jared referenced this pull request

2026-08-05 12:36:40 -04:00

revenue_concept='general' silently returns own-source only for all 35,040 local governments: IG revenue is aggregate-only and dropped by revenue_long #38

jared added 2 commits 2026-08-05 12:44:20 -04:00

fix: close five final-review gaps in all-categories mode a5f86d87b3

- .detect_direct_suppressed() keys on (year, canonical_govid, category);
  all-categories mode collapses category to one literal value, so the key
  collides and the detector silently reports FALSE instead of "unknown".
  Report NA there instead, and stop isTRUE() in .build_provenance() from
  collapsing that NA back to FALSE. Schema widened to allow null.
- Refuse complete = TRUE + category = "All Categories": the completion grid
  has no per-category cells left to fill once categories are collapsed,
  so the prior silent 0-rows-filled result was never actually checked.
- cog_balances(category = "All Categories") returned zero rows with no
  error. .validate_verb_inputs() gains allow_all_categories (default
  FALSE); .verb_spendrev() passes TRUE, cog_balances() does not, so the
  three verbs share one place to reject it instead of drifting again.
- Fix the false `subtype = "operations"` argument claim (no such argument
  exists) in NEWS.md and an internal spending.R comment.

Adds three covering tests to test-all-categories.R for the three
behaviour changes above.

fix: scope all-categories suggestion candidates by subtype, not category (finding 6)

R-CMD-check / check (push) Successful in 3m41s

Details

R-CMD-check / check (pull_request) Successful in 4m52s

Details

498950afa6

.build_suggestions()'s recipe-candidate sub-select was keyed on
`WHERE category IN (<category>)`. The reserved pseudo-category
"All Categories" is never itself a row in summary_categories.category, so
in all-categories mode `candidates` always came back empty and coverage
signposting (uscogdata#9) was structurally impossible for the one mode
whose entire premise is "you cannot sum the wrong scope" -- measured on
Los Angeles County FY2011: category = "Public Welfare" reports 2
suggestions (incl. $271,589,000 excluded E68), category = "All Categories"
reported 0, silently losing that same signal.

Apply the branch's own design principle: the concept boundary is subtype,
not category. .build_suggestions() now accepts all_categories/subtype_col/
subtype_scope (all optional, default off, so no other caller's behaviour
changes) and, when all-categories mode is active, scopes the candidate
sub-select by `<subtype_col> IN (<subtype_scope>)` instead -- symmetric
with .build_verb_sql()'s own WHERE predicate. The M/L recipe exclusion and
the is.null(category) early return are unchanged.

After the fix, LA County FY2011 "All Categories" reports 5 suggestions,
including welfare_cash_e68_wide for the exact $271,589,000 gap.

Adds two covering tests to test-all-categories.R using the bundled fixture
(AL state gov, FY2011, "Corrections"): one end-to-end (per-category and
all-categories both signpost the same recipe) and one direct on
.build_suggestions() proving the subtype-vs-category branch is what
changes the query. Updates the 0.2.0 NEWS entry.

jared merged commit e3ab26c3e6 into main

2026-08-05 12:50:46 -04:00

jared referenced this pull request

2026-08-05 12:52:50 -04:00

revenue_concept='general' silently returns own-source only for all 35,040 local governments: IG revenue is aggregate-only and dropped by revenue_long #38

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Civilytics/uscogdata#37