Closes the last blocked test in the package. Suite: 716 pass / 0 fail / 0 skip — the first time uscogdata has had no skipped tests.
Depends on census_of_governments_finance_pipeline#79 (the crosswalk rows) and on republishing the corpus; the bundled fixture is already regenerated at pipeline_commit aadb46b.
What this adds
cog_revenue() gains revenue_concept, mirroring expenditure_concept, with Census's two published concepts defined as crosswalk revenue_subtype sets rather than item-code prefixes:
general = own_source + federal + state + local_aid (the default)
total = general + utility + liquor_store + insurance_trust
The manual defines the first by subtracting the other three from the second (§4.3), so both are computable only once all four families are named — which is what pipeline#79 does. Insurance trust now includes the employee-retirement X codes (X01/X02/X05/X08) alongside the Y codes.
Mechanically this reuses the subtype_scope machinery #11 introduced: revenue_long/revenue_long_harmonized carry every revenue subtype and the concept narrows in R, exactly as expenditure_concept narrows spending_long. So the concept automatically scopes the verb SQL, the harmonization exclusion count, and the complete = TRUE grid.
Two corrections to the blocked test, both verified against Census
The test had been written before the Y codes were classified, and its expectations were stale in two ways. Both were caught by reconciling against Census's own computed concept fields (IndFin FY2012, Wisconsin state) rather than by re-deriving from our corpus:
It asserted X01+X04+X05+X08 and omitted X02.X02 is state-employee contributions, which applies to state governments and is nonzero for Wisconsin ($245,083K). X04 is an exhibit code for an intragovernmental transfer that X05's own definition excludes by name. Census's computed "Total Emp Ret Rev" is $2,283,883K — exactly X01+X02+X05+X08, confirming both the inclusion and the exclusion.
Its expected total of $33,377,093K predated the Y classification. Complete Total Revenue for WI FY2012 is $34,881,961K (general 31,338,293 + Y 1,259,785 + X 2,283,883). The test now asserts the delta as X + Y as well as the level, so it stays correct however the utility/liquor families land.
Behaviour change worth flagging
general is now strict Census General Revenue, so utility and liquor store revenue leave the default. Measured on the fixture, that is:
gov type
utility + liquor as % of what cog_revenue() returned before
State
1.2%
County
1.7%
City
15.9%
Township
4.8%
This is the "will matter for cities" case the open question predicted. Anyone wanting the old inclusive number asks for revenue_concept = "total" (which also adds insurance trust) or filters on subtype directly.
One fix found along the way
cog_explain() printed expenditure_concept unconditionally, so a cog_revenue() caller was told "Concept: primary" — a spending concept their result has nothing to do with. It now prints each verb's own concept.
Also
Series breaks SB197–SB202 (from pipeline#79) document that the X family stops at FY2016, so a total series steps at the FY2016/FY2017 seam for collection-scope reasons. Documented in ?cog_revenue and the README.
uscogdata#25 restated for the widest revenue concept: total must not admit balance stocks. Asserted.
Provenance gains revenue_concept (additive; cog-api passes provenance through verbatim, so no API change is required).
Not done here
The fixture is built from pipeline#79's branch commit aadb46b. Once #79 merges it should be regenerated against the main commit so the published provenance matches — cheap, and I'll do it as a follow-up commit on this branch before merge.
Closes the last blocked test in the package. **Suite: 716 pass / 0 fail / 0 skip** — the first time `uscogdata` has had no skipped tests.
Depends on `census_of_governments_finance_pipeline#79` (the crosswalk rows) and on republishing the corpus; the bundled fixture is already regenerated at `pipeline_commit aadb46b`.
## What this adds
`cog_revenue()` gains `revenue_concept`, mirroring `expenditure_concept`, with Census's two published concepts defined as crosswalk `revenue_subtype` sets rather than item-code prefixes:
```
general = own_source + federal + state + local_aid (the default)
total = general + utility + liquor_store + insurance_trust
```
The manual defines the first by *subtracting* the other three from the second (§4.3), so both are computable only once all four families are named — which is what pipeline#79 does. Insurance trust now includes the employee-retirement `X` codes (`X01`/`X02`/`X05`/`X08`) alongside the `Y` codes.
Mechanically this reuses the `subtype_scope` machinery `#11` introduced: `revenue_long`/`revenue_long_harmonized` carry **every** revenue subtype and the concept narrows in R, exactly as `expenditure_concept` narrows `spending_long`. So the concept automatically scopes the verb SQL, the harmonization exclusion count, and the `complete = TRUE` grid.
## Two corrections to the blocked test, both verified against Census
The test had been written before the `Y` codes were classified, and its expectations were stale in two ways. Both were caught by reconciling against Census's own **computed concept fields** (IndFin FY2012, Wisconsin state) rather than by re-deriving from our corpus:
1. **It asserted `X01+X04+X05+X08` and omitted `X02`.** `X02` is state-employee contributions, which applies to state governments and is nonzero for Wisconsin ($245,083K). `X04` is an exhibit code for an *intragovernmental* transfer that `X05`'s own definition excludes by name. Census's computed "Total Emp Ret Rev" is **$2,283,883K** — exactly `X01+X02+X05+X08`, confirming both the inclusion and the exclusion.
2. **Its expected `total` of $33,377,093K predated the `Y` classification.** Complete Total Revenue for WI FY2012 is **$34,881,961K** (general 31,338,293 + Y 1,259,785 + X 2,283,883). The test now asserts the delta as `X + Y` as well as the level, so it stays correct however the utility/liquor families land.
## Behaviour change worth flagging
`general` is now **strict** Census General Revenue, so utility and liquor store revenue leave the default. Measured on the fixture, that is:
| gov type | utility + liquor as % of what `cog_revenue()` returned before |
|---|---|
| State | 1.2% |
| County | 1.7% |
| **City** | **15.9%** |
| Township | 4.8% |
This is the "will matter for cities" case the open question predicted. Anyone wanting the old inclusive number asks for `revenue_concept = "total"` (which also adds insurance trust) or filters on subtype directly.
## One fix found along the way
`cog_explain()` printed `expenditure_concept` unconditionally, so a `cog_revenue()` caller was told **"Concept: primary"** — a spending concept their result has nothing to do with. It now prints each verb's own concept.
## Also
- Series breaks `SB197`–`SB202` (from pipeline#79) document that the `X` family stops at FY2016, so a `total` series steps at the FY2016/FY2017 seam for collection-scope reasons. Documented in `?cog_revenue` and the README.
- `uscogdata#25` restated for the widest revenue concept: `total` must not admit balance stocks. Asserted.
- Provenance gains `revenue_concept` (additive; `cog-api` passes provenance through verbatim, so no API change is required).
## Not done here
The fixture is built from pipeline#79's **branch** commit `aadb46b`. Once #79 merges it should be regenerated against the `main` commit so the published provenance matches — cheap, and I'll do it as a follow-up commit on this branch before merge.
Rewrites expenditure/revenue classification off item-code first-letter
prefixes and onto summary_categories membership (F-018: prefix Y spans
revenue, expenditure, and balance codes), and exposes
expenditure_concept = c("primary", "direct", "total") with primary as
the new default:
primary = operations + capital + assistance
direct = primary + interest + insurance_benefits (Census Direct)
total = direct + intergovernmental (M/L/Q via ig views)
- inst/sql: flow views (20-25) select by crosswalk membership;
summary_categories moves to 11- so it registers before them (DuckDB
binds view sources eagerly). The IG leg gains Q11/Q12/Q18 state
school-system payments (F-017).
- R: one subtype scope per verb call drives the verb SQL, the
harmonization exclusion count, and the complete = TRUE grid;
flow_prefixes survives only to scope recipe suggestions.
cog_geographic_rollup/cog_peer_compare accept primary|direct, still
refuse total, and now actually pass the concept through.
- Balance codes can never reach a spending or revenue result
(uscogdata#25), asserted at both view and verb level.
- Deletes the #11 skip; per the 2026-07-30 owner ruling the F-018 Y01
proof is asserted against the crosswalk, not the default
cog_revenue() call (which stays General Revenue pending #12).
Suite: 696 pass / 0 fail / 1 skip (#12, expected).
Closes#11
Closes the last blocked test in the suite. Owner ruled both halves of the
open question yes on 2026-07-30.
`cog_revenue()` gains `revenue_concept`, mirroring `expenditure_concept`,
with Census's two published concepts defined as crosswalk
`revenue_subtype` sets rather than item-code prefixes:
general = own_source + federal + state + local_aid (the default)
total = general + utility + liquor_store + insurance_trust
The manual defines the first by subtracting the other three from the
second (4.3), so both are computable only once all four families are
named -- which cog_pipeline#79 does. Insurance trust now includes the
employee-retirement X codes (X01/X02/X05/X08) alongside the Y codes.
- inst/sql: revenue_long / revenue_long_harmonized carry EVERY revenue
subtype; the concept narrows in R via the existing subtype_scope
machinery, exactly as expenditure_concept narrows spending_long.
- cog_explain() now prints each verb's OWN concept. It previously
printed `expenditure_concept` unconditionally, so a cog_revenue()
caller was told "Concept: primary" -- a spending concept their result
has nothing to do with.
- Fixture regenerated at pipeline_commit aadb46b (330 crosswalk rows).
Corrected two stale expectations in the blocked test while un-skipping
it. It asserted X01+X04+X05+X08 and omitted X02, which applies to state
governments and is nonzero for Wisconsin; X04 is an exhibit code for an
INTRAgovernmental transfer that Census's own "Total Emp Ret Rev"
excludes. Verified against that Census field: the right set is
X01+X02+X05+X08 = $2,283,883k, exactly. And its expected `total` of
$33,377,093k predated the Y codes being classified -- complete Total
Revenue for WI FY2012 is $34,881,961k (general 31,338,293 + Y 1,259,785
+ X 2,283,883).
Behaviour change worth knowing: `general` is now STRICT Census General
Revenue, so utility and liquor store revenue leave the default. Measured
on the fixture that is 15.9% of what cog_revenue() returned for cities,
vs 1.2% for states and 1.7% for counties.
Suite: 716 pass / 0 fail / 0 skip -- the first time this package has had
no skipped tests.
Closes#12
The "Not done here" caveat above is resolved — no fixture regeneration is needed. The merge was a normal merge commit (93b3264), so aadb46b is now an ancestor of origin/main:
The fixture's pipeline_commit: aadb46b is therefore a real, reachable commit on main whose tree produced exactly this corpus — arguably more precise provenance than the merge commit would be. Rebuilding publish_tree against main confirmed it: 852 skipped, 0 built, because nothing about the content changed.
This PR is ready to merge. The corpus republish (scripts/publish_all.R --execute) is the remaining step and is still awaiting explicit approval, since it mirrors to Hugging Face and restarts the production API.
**Update after pipeline#79 merged (2026-07-31).**
The "Not done here" caveat above is resolved — **no fixture regeneration is needed**. The merge was a normal merge commit (`93b3264`), so `aadb46b` is now an ancestor of `origin/main`:
```
git merge-base --is-ancestor aadb46b origin/main -> true
git diff --stat aadb46b origin/main -- data/ R/ -> empty
```
The fixture's `pipeline_commit: aadb46b` is therefore a real, reachable commit on `main` whose tree produced exactly this corpus — arguably more precise provenance than the merge commit would be. Rebuilding `publish_tree` against `main` confirmed it: `852 skipped, 0 built`, because nothing about the content changed.
Re-verified on `main`:
- publish tree: 71 files, 330 crosswalk rows, schema v6, all gates passing
- reader suite: **716 pass / 0 fail / 0 skip**
- publish dry run: clean, nothing published
This PR is ready to merge. The corpus republish (`scripts/publish_all.R --execute`) is the remaining step and is still awaiting explicit approval, since it mirrors to Hugging Face and restarts the production API.
jared
merged commit de7ccbebc7 into main2026-07-30 22:09:45 -04:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes the last blocked test in the package. Suite: 716 pass / 0 fail / 0 skip — the first time
uscogdatahas had no skipped tests.Depends on
census_of_governments_finance_pipeline#79(the crosswalk rows) and on republishing the corpus; the bundled fixture is already regenerated atpipeline_commit aadb46b.What this adds
cog_revenue()gainsrevenue_concept, mirroringexpenditure_concept, with Census's two published concepts defined as crosswalkrevenue_subtypesets rather than item-code prefixes:The manual defines the first by subtracting the other three from the second (§4.3), so both are computable only once all four families are named — which is what pipeline#79 does. Insurance trust now includes the employee-retirement
Xcodes (X01/X02/X05/X08) alongside theYcodes.Mechanically this reuses the
subtype_scopemachinery#11introduced:revenue_long/revenue_long_harmonizedcarry every revenue subtype and the concept narrows in R, exactly asexpenditure_conceptnarrowsspending_long. So the concept automatically scopes the verb SQL, the harmonization exclusion count, and thecomplete = TRUEgrid.Two corrections to the blocked test, both verified against Census
The test had been written before the
Ycodes were classified, and its expectations were stale in two ways. Both were caught by reconciling against Census's own computed concept fields (IndFin FY2012, Wisconsin state) rather than by re-deriving from our corpus:X01+X04+X05+X08and omittedX02.X02is state-employee contributions, which applies to state governments and is nonzero for Wisconsin ($245,083K).X04is an exhibit code for an intragovernmental transfer thatX05's own definition excludes by name. Census's computed "Total Emp Ret Rev" is $2,283,883K — exactlyX01+X02+X05+X08, confirming both the inclusion and the exclusion.totalof $33,377,093K predated theYclassification. Complete Total Revenue for WI FY2012 is $34,881,961K (general 31,338,293 + Y 1,259,785 + X 2,283,883). The test now asserts the delta asX + Yas well as the level, so it stays correct however the utility/liquor families land.Behaviour change worth flagging
generalis now strict Census General Revenue, so utility and liquor store revenue leave the default. Measured on the fixture, that is:cog_revenue()returned beforeThis is the "will matter for cities" case the open question predicted. Anyone wanting the old inclusive number asks for
revenue_concept = "total"(which also adds insurance trust) or filters on subtype directly.One fix found along the way
cog_explain()printedexpenditure_conceptunconditionally, so acog_revenue()caller was told "Concept: primary" — a spending concept their result has nothing to do with. It now prints each verb's own concept.Also
SB197–SB202(from pipeline#79) document that theXfamily stops at FY2016, so atotalseries steps at the FY2016/FY2017 seam for collection-scope reasons. Documented in?cog_revenueand the README.uscogdata#25restated for the widest revenue concept:totalmust not admit balance stocks. Asserted.revenue_concept(additive;cog-apipasses provenance through verbatim, so no API change is required).Not done here
The fixture is built from pipeline#79's branch commit
aadb46b. Once #79 merges it should be regenerated against themaincommit so the published provenance matches — cheap, and I'll do it as a follow-up commit on this branch before merge.Rewrites expenditure/revenue classification off item-code first-letter prefixes and onto summary_categories membership (F-018: prefix Y spans revenue, expenditure, and balance codes), and exposes expenditure_concept = c("primary", "direct", "total") with primary as the new default: primary = operations + capital + assistance direct = primary + interest + insurance_benefits (Census Direct) total = direct + intergovernmental (M/L/Q via ig views) - inst/sql: flow views (20-25) select by crosswalk membership; summary_categories moves to 11- so it registers before them (DuckDB binds view sources eagerly). The IG leg gains Q11/Q12/Q18 state school-system payments (F-017). - R: one subtype scope per verb call drives the verb SQL, the harmonization exclusion count, and the complete = TRUE grid; flow_prefixes survives only to scope recipe suggestions. cog_geographic_rollup/cog_peer_compare accept primary|direct, still refuse total, and now actually pass the concept through. - Balance codes can never reach a spending or revenue result (uscogdata#25), asserted at both view and verb level. - Deletes the #11 skip; per the 2026-07-30 owner ruling the F-018 Y01 proof is asserted against the crosswalk, not the default cog_revenue() call (which stays General Revenue pending #12). Suite: 696 pass / 0 fail / 1 skip (#12, expected). Closes #11Update after pipeline#79 merged (2026-07-31).
The "Not done here" caveat above is resolved — no fixture regeneration is needed. The merge was a normal merge commit (
93b3264), soaadb46bis now an ancestor oforigin/main:The fixture's
pipeline_commit: aadb46bis therefore a real, reachable commit onmainwhose tree produced exactly this corpus — arguably more precise provenance than the merge commit would be. Rebuildingpublish_treeagainstmainconfirmed it:852 skipped, 0 built, because nothing about the content changed.Re-verified on
main:This PR is ready to merge. The corpus republish (
scripts/publish_all.R --execute) is the remaining step and is still awaiting explicit approval, since it mirrors to Hugging Face and restarts the production API.