feat: revenue_concept = c("general", "total") off the crosswalk (#12)
Closes the last blocked test in the suite. Owner ruled both halves of the open question yes on 2026-07-30. `cog_revenue()` gains `revenue_concept`, mirroring `expenditure_concept`, with Census's two published concepts defined as crosswalk `revenue_subtype` sets rather than item-code prefixes: general = own_source + federal + state + local_aid (the default) total = general + utility + liquor_store + insurance_trust The manual defines the first by subtracting the other three from the second (4.3), so both are computable only once all four families are named -- which cog_pipeline#79 does. Insurance trust now includes the employee-retirement X codes (X01/X02/X05/X08) alongside the Y codes. - inst/sql: revenue_long / revenue_long_harmonized carry EVERY revenue subtype; the concept narrows in R via the existing subtype_scope machinery, exactly as expenditure_concept narrows spending_long. - cog_explain() now prints each verb's OWN concept. It previously printed `expenditure_concept` unconditionally, so a cog_revenue() caller was told "Concept: primary" -- a spending concept their result has nothing to do with. - Fixture regenerated at pipeline_commit aadb46b (330 crosswalk rows). Corrected two stale expectations in the blocked test while un-skipping it. It asserted X01+X04+X05+X08 and omitted X02, which applies to state governments and is nonzero for Wisconsin; X04 is an exhibit code for an INTRAgovernmental transfer that Census's own "Total Emp Ret Rev" excludes. Verified against that Census field: the right set is X01+X02+X05+X08 = $2,283,883k, exactly. And its expected `total` of $33,377,093k predated the Y codes being classified -- complete Total Revenue for WI FY2012 is $34,881,961k (general 31,338,293 + Y 1,259,785 + X 2,283,883). Behaviour change worth knowing: `general` is now STRICT Census General Revenue, so utility and liquor store revenue leave the default. Measured on the fixture that is 15.9% of what cog_revenue() returned for cities, vs 1.2% for states and 1.7% for counties. Suite: 716 pass / 0 fail / 0 skip -- the first time this package has had no skipped tests. Closes #12
This commit is contained in:
Binary file not shown.
Binary file not shown.
+4
-4
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"schema_version": 6,
|
||||
"built_at": "2026-07-30T20:01:56Z",
|
||||
"pipeline_commit": "e64a046",
|
||||
"built_at": "2026-07-31T00:47:27Z",
|
||||
"pipeline_commit": "aadb46b",
|
||||
"fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated from the sparsified schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit zeros, so FY2011 absence means Census published $0 while FY2012+ absence means not reported. representation.parquet and code_set.parquet carry that rule and ship in full, as do every other metadata table in the publish tree. 2011/2012 straddle both the wide-aggregate -> modern-leaf format boundary (exercised by basis=\"harmonized\" and recipe= queries) and the dense -> sparse representation boundary (SB194); 2019/2020 retain the prior per-capita/CPI regression anchors. Regenerated via data-raw/regenerate_fixture_corpus.R.",
|
||||
"data_vintage": {
|
||||
"source_vintages": {
|
||||
@@ -105,12 +105,12 @@
|
||||
},
|
||||
{
|
||||
"path": "data/series_breaks.parquet",
|
||||
"sha256": "381090660c8b8a1bee852e7f63d29b9ecaf10f71870f41c63de92017c83b6f1f",
|
||||
"sha256": "06dcc995ff533e57cc65fa25086cc9bf83ba592c58bf7cc99269dc2576f69944",
|
||||
"description": "series_breaks.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/summary_categories.parquet",
|
||||
"sha256": "e4918abf8e9e6d1372d7ddc255dc199f9c303e250f4106449241b48c68abee67",
|
||||
"sha256": "e3b0efa00ce713b8f45829b89cfde24b55333f26101f0495df82d85997d18d8e",
|
||||
"description": "summary_categories.parquet"
|
||||
}
|
||||
]
|
||||
|
||||
@@ -25,6 +25,12 @@
|
||||
"type": "boolean",
|
||||
"description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'primary' or 'direct'. See the affected rows' `notes` for the recovering recipe, if any."
|
||||
},
|
||||
"revenue_concept": {
|
||||
"type": "string",
|
||||
"enum": ["general", "total"],
|
||||
"description": "Which revenue concept produced this result, defined as crosswalk revenue_subtype sets (never item-code prefixes). 'general' (the default) is Census General Revenue: own_source + federal + state + local_aid. 'total' is Census Total Revenue: general plus utility, liquor store and insurance trust revenue. Census defines the first by subtracting the other three from the second (manual section 4.3). Meaningful for cog_revenue() results; spending results carry the default.",
|
||||
"$comment": "The employee-retirement (X) codes inside insurance_trust stop at FY2016, so a 'total' series steps at the FY2016/FY2017 seam for collection-scope reasons (series breaks SB197-SB202)."
|
||||
},
|
||||
"harmonization": { "type": "object" },
|
||||
"recipe": { "type": ["object", "null"] },
|
||||
"suggestions": { "type": "array" },
|
||||
|
||||
@@ -1,15 +1,18 @@
|
||||
-- Revenue rows, classified by crosswalk MEMBERSHIP rather than item-code
|
||||
-- first letter (see 20-spending_long.sql for why prefixes cannot work).
|
||||
-- Scope is Census General Revenue: every crosswalk revenue subtype EXCEPT
|
||||
-- insurance_trust (Y01/Y02/Y04/Y11/Y12/Y51/Y52). Owner ruling 2026-07-30:
|
||||
-- the default revenue concept stays general; surfacing insurance-trust
|
||||
-- revenue through an explicit concept argument is uscogdata#12.
|
||||
--
|
||||
-- Carries EVERY revenue subtype. Which of Census's two published concepts a
|
||||
-- query actually returns is decided per revenue_concept in R
|
||||
-- (.verb_spendrev), exactly as expenditure_concept narrows spending_long:
|
||||
-- general = own_source + federal + state + local_aid (the default)
|
||||
-- total = general + utility + liquor_store + insurance_trust
|
||||
-- Census defines the first by subtracting the other three from the second
|
||||
-- (manual section 4.3), so both concepts need all four families present here.
|
||||
CREATE OR REPLACE VIEW revenue_long AS
|
||||
SELECT *
|
||||
FROM long
|
||||
WHERE item_code IN (
|
||||
SELECT item_code FROM summary_categories
|
||||
WHERE category_type = 'revenue'
|
||||
AND revenue_subtype <> 'insurance_trust'
|
||||
)
|
||||
AND NOT is_aggregate;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
-- Harmonized-basis twin of 21-revenue_long.sql: same crosswalk-membership
|
||||
-- classification (General Revenue = revenue minus insurance_trust), applied
|
||||
-- classification (every revenue subtype; the concept narrows in R), applied
|
||||
-- to harmonized_code rather than the published item_code.
|
||||
CREATE OR REPLACE VIEW revenue_long_harmonized AS
|
||||
SELECT * REPLACE (harmonized_code AS item_code)
|
||||
@@ -9,5 +9,4 @@ WHERE NOT is_aggregate
|
||||
AND harmonized_code IN (
|
||||
SELECT item_code FROM summary_categories
|
||||
WHERE category_type = 'revenue'
|
||||
AND revenue_subtype <> 'insurance_trust'
|
||||
);
|
||||
|
||||
Reference in New Issue
Block a user