Tracks the corpus published 2026-07-30 at pipeline_commit e64a046. Groundwork for #11; does not implement it — #11's skip() stays in place.
What changed
data-raw/regenerate_fixture_corpus.R re-run against the new publish tree. The fixture crosswalk goes 291 → 324 rows and gains a balance_subtype column:
Only three fixture files move (series_breaks.parquet, summary_categories.parquet, manifest.json) — no long partition changes, because the published change was metadata-only.
test-categories.R's vocabulary assertions are extended for the new values. cog_categories() is a catalogue verb, so it correctly surfaces every category_type the corpus carries; the stock/flow guard belongs on cog_spending()/cog_revenue(), which must never return a balance row (tracked as #25).
#11 could not be implemented before this: it names the gap itself — "summary_categories currently has zero rows for prefixes I, Q, X, and Y, so admitting them here without a category mapping would surface rows with category = NA." All four prefixes now map, and prefix Y spans all three category_types, which is F-018 made concrete in the crosswalk.
Both are in the comment on #11 with full measurements; summarising here because they change the spec:
direct must include insurance benefits, not just interest. The 2006 manual §5.2.2.1 defines Direct as "all expenditure other than intergovernmental", explicitly naming payments to retirees; social insurance trust is a sector, not a character. Reconciled against Census's own published state aggregates (20statetypepu.txt, independent of our corpus): total reproduces Census's published expenditure sum to the dollar for Wisconsin ($40,460,441k) and California ($375,553,177k) FY2020, with 100.0000% crosswalk coverage of their published expenditure codes. Omitting insurance benefits would understate California's Direct Expenditure by 10.9% — larger than the 7.1% F-012 was filed for. Owner confirmed.
#11's DoD item 4 conflicts with #12. It requires Y01 in the defaultcog_revenue(), while #12 requires the default to be general ($31,338,293,000 for WI FY2012 — which the new insurance_trust subtype reproduces exactly). Y01 and X01 are both Insurance Trust Revenue, so a default holding one but not the other matches no Census concept (manual §4.3). Resolved by keeping the default general and moving #11's proof off the default call.
Test plan
Fixture regenerated via the committed script, not by hand.
test-fixture-vintage.R passes, so the fixture is not stale.
Full suite: 0 failures, 2 skips — the #11 and #12 blocks, unchanged.
Next
Reader change only: rewrite inst/sql/20-spending_long.sql off LEFT(item_code,1) IN ('E','F','G') onto a join to summary_categories, add expenditure_concept = c("primary","direct","total") with primary as default and direct = primary + interest + insurance_benefits, then delete the skip().
Tracks the corpus published 2026-07-30 at `pipeline_commit e64a046`. Groundwork for #11; **does not implement it** — #11's `skip()` stays in place.
## What changed
`data-raw/regenerate_fixture_corpus.R` re-run against the new publish tree. The fixture crosswalk goes **291 → 324 rows** and gains a `balance_subtype` column:
| what | source |
|---|---|
| `category_type = 'balance'` + `balance_subtype`, 14 cash-and-security holding codes | pipeline#76 / PR #77 |
| `I`/`Q`/`Y` flow codes + `interest`, `insurance_benefits`, `insurance_trust` subtypes | pipeline PR #78 |
| `SB195`/`SB196`, the X40/X41 book→market basis change | pipeline#76 |
Only three fixture files move (`series_breaks.parquet`, `summary_categories.parquet`, `manifest.json`) — no `long` partition changes, because the published change was metadata-only.
`test-categories.R`'s vocabulary assertions are extended for the new values. `cog_categories()` is a **catalogue** verb, so it correctly surfaces every `category_type` the corpus carries; the stock/flow guard belongs on `cog_spending()`/`cog_revenue()`, which must never return a `balance` row (tracked as #25).
## Why this unblocks #11
#11 could not be implemented before this: it names the gap itself — *"`summary_categories` currently has zero rows for prefixes `I`, `Q`, `X`, and `Y`, so admitting them here without a category mapping would surface rows with `category = NA`."* All four prefixes now map, and prefix `Y` spans all three `category_type`s, which is F-018 made concrete in the crosswalk.
## Two findings recorded on #11 while doing this
Both are in [the comment on #11](https://gitea.civilytics.org/Civilytics/uscogdata/issues/11) with full measurements; summarising here because they change the spec:
1. **`direct` must include insurance benefits, not just interest.** The 2006 manual §5.2.2.1 defines Direct as *"all expenditure other than intergovernmental"*, explicitly naming payments to retirees; social insurance trust is a *sector*, not a character. Reconciled against Census's own published state aggregates (`20statetypepu.txt`, independent of our corpus): `total` reproduces Census's published expenditure sum **to the dollar** for Wisconsin (`$40,460,441k`) and California (`$375,553,177k`) FY2020, with **100.0000%** crosswalk coverage of their published expenditure codes. Omitting insurance benefits would understate California's Direct Expenditure by **10.9%** — larger than the 7.1% F-012 was filed for. Owner confirmed.
2. **#11's DoD item 4 conflicts with #12.** It requires `Y01` in the *default* `cog_revenue()`, while #12 requires the default to be `general` (`$31,338,293,000` for WI FY2012 — which the new `insurance_trust` subtype reproduces exactly). `Y01` and `X01` are both Insurance Trust Revenue, so a default holding one but not the other matches no Census concept (manual §4.3). Resolved by keeping the default `general` and moving #11's proof off the default call.
## Test plan
- [x] Fixture regenerated via the committed script, not by hand.
- [x] `test-fixture-vintage.R` passes, so the fixture is not stale.
- [x] Full suite: **0 failures, 2 skips** — the #11 and #12 blocks, unchanged.
## Next
Reader change only: rewrite `inst/sql/20-spending_long.sql` off `LEFT(item_code,1) IN ('E','F','G')` onto a join to `summary_categories`, add `expenditure_concept = c("primary","direct","total")` with `primary` as default and `direct = primary + interest + insurance_benefits`, then delete the `skip()`.
Tracks the corpus published 2026-07-30, which adds category_type = 'balance'
(pipeline#76) and the I/Q/Y flow codes (pipeline#78) -- the crosswalk
prerequisite for #11's three-concept expenditure model.
Fixture crosswalk goes 291 -> 324 rows and gains balance_subtype. Only three
files change (series_breaks, summary_categories, manifest); no long partition
moves, because the published change was metadata-only.
test-categories.R's vocabulary assertions extended for the new values:
category_type gains 'balance', spending subtypes gain 'interest' and
'insurance_benefits', revenue subtypes gain 'insurance_trust'.
cog_categories() is a catalogue verb so it surfaces every category_type the
corpus carries; the stock/flow guard belongs on the money verbs.
Suite: 0 failures, 2 skips (the #11 and #12 blocks).
jared
merged commit 5d77d39711 into main2026-07-30 16:22:19 -04:00
This PR now carries the full #11 implementation on top of the fixture regen (commit 93300ae).
What changed
Classification is crosswalk membership, never item-code first letters. All six flow views (20-25) now select on summary_categories membership; summary_categories moved to 11- so it registers before them (DuckDB binds view sources eagerly).
expenditure_concept = c("primary", "direct", "total"), primary is the new default: primary = operations + capital + assistance; direct = primary + interest + insurance_benefits (Census Direct, manual 5.2.2.1 incl. retirees); total = direct + intergovernmental. The IG leg gains Q11/Q12/Q18 (F-017) — state IG is ~36% of Direct pooled on the fixture once Q is counted, vs ~24% under the old M/L-only measurement.
cog_geographic_rollup()/cog_peer_compare() accept primary/direct (still refuse total) and now actually pass the concept through to cog_spending() — previously the argument was validated but silently ignored.
Per the 2026-07-30 rulings: cog_revenue()'s default stays General Revenue (crosswalk scope revenue_subtype <> 'insurance_trust', behavior-identical to the old prefix list — verified zero rows differ on the fixture); the F-018 Y01 proof moved off the default call onto the crosswalk itself; revenue provenance tracks the resolved default ("primary").
uscogdata#25 reader-side guarantee is now asserted at view level (no balance member in spending_long/revenue_long) and verb level (WI FY2019, which carries real Y-balance rows). #25 can likely be closed when this merges — your call.
Docs: README + total-spending vignette rewritten for three concepts; vignette measurements recomputed under the new definitions (FY2019 total-vs-direct rollup overstatement: 24.1% Alabama, 23.2% national).
Done criteria met: the skip() at tests/testthat/test-expenditure-concepts.R:22 is deleted and the test passes. Suite: 696 pass / 0 fail / 1 skip (the #12 skip, expected). Vignette re-knit clean against the fixture.
This PR now carries the full #11 implementation on top of the fixture regen (commit 93300ae).
**What changed**
- Classification is crosswalk membership, never item-code first letters. All six flow views (`20`-`25`) now select on `summary_categories` membership; `summary_categories` moved to `11-` so it registers before them (DuckDB binds view sources eagerly).
- `expenditure_concept = c("primary", "direct", "total")`, **`primary` is the new default**: primary = operations + capital + assistance; direct = primary + interest + insurance_benefits (Census Direct, manual 5.2.2.1 incl. retirees); total = direct + intergovernmental. The IG leg gains `Q11`/`Q12`/`Q18` (F-017) — state IG is ~36% of Direct pooled on the fixture once Q is counted, vs ~24% under the old M/L-only measurement.
- `cog_geographic_rollup()`/`cog_peer_compare()` accept `primary`/`direct` (still refuse `total`) and now actually pass the concept through to `cog_spending()` — previously the argument was validated but silently ignored.
- Per the 2026-07-30 rulings: `cog_revenue()`'s default stays General Revenue (crosswalk scope `revenue_subtype <> 'insurance_trust'`, behavior-identical to the old prefix list — verified zero rows differ on the fixture); the F-018 Y01 proof moved off the default call onto the crosswalk itself; revenue provenance tracks the resolved default (`"primary"`).
- **uscogdata#25 reader-side guarantee is now asserted** at view level (no balance member in `spending_long`/`revenue_long`) and verb level (WI FY2019, which carries real Y-balance rows). #25 can likely be closed when this merges — your call.
- Docs: README + `total-spending` vignette rewritten for three concepts; vignette measurements recomputed under the new definitions (FY2019 total-vs-direct rollup overstatement: 24.1% Alabama, 23.2% national).
**Done criteria met**: the `skip()` at `tests/testthat/test-expenditure-concepts.R:22` is deleted and the test passes. Suite: 696 pass / 0 fail / 1 skip (the #12 skip, expected). Vignette re-knit clean against the fixture.
Left alone deliberately: `DESCRIPTION` roxygen pin (7.3.3; regen reflow reverted), #12 (revenue concepts), #9 re-measurement.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Tracks the corpus published 2026-07-30 at
pipeline_commit e64a046. Groundwork for #11; does not implement it — #11'sskip()stays in place.What changed
data-raw/regenerate_fixture_corpus.Rre-run against the new publish tree. The fixture crosswalk goes 291 → 324 rows and gains abalance_subtypecolumn:category_type = 'balance'+balance_subtype, 14 cash-and-security holding codesI/Q/Yflow codes +interest,insurance_benefits,insurance_trustsubtypesSB195/SB196, the X40/X41 book→market basis changeOnly three fixture files move (
series_breaks.parquet,summary_categories.parquet,manifest.json) — nolongpartition changes, because the published change was metadata-only.test-categories.R's vocabulary assertions are extended for the new values.cog_categories()is a catalogue verb, so it correctly surfaces everycategory_typethe corpus carries; the stock/flow guard belongs oncog_spending()/cog_revenue(), which must never return abalancerow (tracked as #25).Why this unblocks #11
#11 could not be implemented before this: it names the gap itself — "
summary_categoriescurrently has zero rows for prefixesI,Q,X, andY, so admitting them here without a category mapping would surface rows withcategory = NA." All four prefixes now map, and prefixYspans all threecategory_types, which is F-018 made concrete in the crosswalk.Two findings recorded on #11 while doing this
Both are in the comment on #11 with full measurements; summarising here because they change the spec:
directmust include insurance benefits, not just interest. The 2006 manual §5.2.2.1 defines Direct as "all expenditure other than intergovernmental", explicitly naming payments to retirees; social insurance trust is a sector, not a character. Reconciled against Census's own published state aggregates (20statetypepu.txt, independent of our corpus):totalreproduces Census's published expenditure sum to the dollar for Wisconsin ($40,460,441k) and California ($375,553,177k) FY2020, with 100.0000% crosswalk coverage of their published expenditure codes. Omitting insurance benefits would understate California's Direct Expenditure by 10.9% — larger than the 7.1% F-012 was filed for. Owner confirmed.#11's DoD item 4 conflicts with #12. It requires
Y01in the defaultcog_revenue(), while #12 requires the default to begeneral($31,338,293,000for WI FY2012 — which the newinsurance_trustsubtype reproduces exactly).Y01andX01are both Insurance Trust Revenue, so a default holding one but not the other matches no Census concept (manual §4.3). Resolved by keeping the defaultgeneraland moving #11's proof off the default call.Test plan
test-fixture-vintage.Rpasses, so the fixture is not stale.Next
Reader change only: rewrite
inst/sql/20-spending_long.sqloffLEFT(item_code,1) IN ('E','F','G')onto a join tosummary_categories, addexpenditure_concept = c("primary","direct","total")withprimaryas default anddirect = primary + interest + insurance_benefits, then delete theskip().This PR now carries the full #11 implementation on top of the fixture regen (commit
93300ae).What changed
20-25) now select onsummary_categoriesmembership;summary_categoriesmoved to11-so it registers before them (DuckDB binds view sources eagerly).expenditure_concept = c("primary", "direct", "total"),primaryis the new default: primary = operations + capital + assistance; direct = primary + interest + insurance_benefits (Census Direct, manual 5.2.2.1 incl. retirees); total = direct + intergovernmental. The IG leg gainsQ11/Q12/Q18(F-017) — state IG is ~36% of Direct pooled on the fixture once Q is counted, vs ~24% under the old M/L-only measurement.cog_geographic_rollup()/cog_peer_compare()acceptprimary/direct(still refusetotal) and now actually pass the concept through tocog_spending()— previously the argument was validated but silently ignored.cog_revenue()'s default stays General Revenue (crosswalk scoperevenue_subtype <> 'insurance_trust', behavior-identical to the old prefix list — verified zero rows differ on the fixture); the F-018 Y01 proof moved off the default call onto the crosswalk itself; revenue provenance tracks the resolved default ("primary").spending_long/revenue_long) and verb level (WI FY2019, which carries real Y-balance rows). #25 can likely be closed when this merges — your call.total-spendingvignette rewritten for three concepts; vignette measurements recomputed under the new definitions (FY2019 total-vs-direct rollup overstatement: 24.1% Alabama, 23.2% national).Done criteria met: the
skip()attests/testthat/test-expenditure-concepts.R:22is deleted and the test passes. Suite: 696 pass / 0 fail / 1 skip (the #12 skip, expected). Vignette re-knit clean against the fixture.Left alone deliberately:
DESCRIPTIONroxygen pin (7.3.3; regen reflow reverted), #12 (revenue concepts), #9 re-measurement.