jared
4b23dbd9f4
Closes the last blocked test in the suite. Owner ruled both halves of the open question yes on 2026-07-30. `cog_revenue()` gains `revenue_concept`, mirroring `expenditure_concept`, with Census's two published concepts defined as crosswalk `revenue_subtype` sets rather than item-code prefixes: general = own_source + federal + state + local_aid (the default) total = general + utility + liquor_store + insurance_trust The manual defines the first by subtracting the other three from the second (4.3), so both are computable only once all four families are named -- which cog_pipeline#79 does. Insurance trust now includes the employee-retirement X codes (X01/X02/X05/X08) alongside the Y codes. - inst/sql: revenue_long / revenue_long_harmonized carry EVERY revenue subtype; the concept narrows in R via the existing subtype_scope machinery, exactly as expenditure_concept narrows spending_long. - cog_explain() now prints each verb's OWN concept. It previously printed `expenditure_concept` unconditionally, so a cog_revenue() caller was told "Concept: primary" -- a spending concept their result has nothing to do with. - Fixture regenerated at pipeline_commit aadb46b (330 crosswalk rows). Corrected two stale expectations in the blocked test while un-skipping it. It asserted X01+X04+X05+X08 and omitted X02, which applies to state governments and is nonzero for Wisconsin; X04 is an exhibit code for an INTRAgovernmental transfer that Census's own "Total Emp Ret Rev" excludes. Verified against that Census field: the right set is X01+X02+X05+X08 = $2,283,883k, exactly. And its expected `total` of $33,377,093k predated the Y codes being classified -- complete Total Revenue for WI FY2012 is $34,881,961k (general 31,338,293 + Y 1,259,785 + X 2,283,883). Behaviour change worth knowing: `general` is now STRICT Census General Revenue, so utility and liquor store revenue leave the default. Measured on the fixture that is 15.9% of what cog_revenue() returned for cities, vs 1.2% for states and 1.7% for counties. Suite: 716 pass / 0 fail / 0 skip -- the first time this package has had no skipped tests. Closes #12
60 lines
4.5 KiB
JSON
60 lines
4.5 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"$id": "https://civilytics.org/schemas/uscogdata/provenance-v1.json",
|
|
"title": "uscogdata provenance v1",
|
|
"type": "object",
|
|
"required": ["verb", "target", "years", "scope", "manifest", "sql_query"],
|
|
"properties": {
|
|
"verb": { "type": "string" },
|
|
"call": { "type": "string" },
|
|
"target": { "type": "object" },
|
|
"years": { "type": "array", "items": { "type": "integer" } },
|
|
"category": { "type": ["string", "array", "null"] },
|
|
"basis": { "type": ["string", "null"] },
|
|
"basis_note": { "type": ["string", "null"] },
|
|
"expenditure_concept": {
|
|
"type": "string",
|
|
"enum": ["primary", "direct", "total"],
|
|
"description": "Which spending concept produced this result, defined as crosswalk spend_subtype sets (never item-code prefixes). 'primary' (the default) is the government's own service provision: operations + capital + assistance. 'direct' adds interest on debt and insurance trust benefit payments (Census's published Direct Expenditure). 'total' adds intergovernmental payments (M to local governments, L to state government, Q11/Q12/Q18 to school systems). Only 'primary' and 'direct' are valid for results combined across governments."
|
|
},
|
|
"expenditure_concept_note": {
|
|
"type": ["string", "null"],
|
|
"description": "How the intergovernmental leg was assembled; null for 'primary' and 'direct'."
|
|
},
|
|
"expenditure_concept_direct_suppressed": {
|
|
"type": "boolean",
|
|
"description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'primary' or 'direct'. See the affected rows' `notes` for the recovering recipe, if any."
|
|
},
|
|
"revenue_concept": {
|
|
"type": "string",
|
|
"enum": ["general", "total"],
|
|
"description": "Which revenue concept produced this result, defined as crosswalk revenue_subtype sets (never item-code prefixes). 'general' (the default) is Census General Revenue: own_source + federal + state + local_aid. 'total' is Census Total Revenue: general plus utility, liquor store and insurance trust revenue. Census defines the first by subtracting the other three from the second (manual section 4.3). Meaningful for cog_revenue() results; spending results carry the default.",
|
|
"$comment": "The employee-retirement (X) codes inside insurance_trust stop at FY2016, so a 'total' series steps at the FY2016/FY2017 seam for collection-scope reasons (series breaks SB197-SB202)."
|
|
},
|
|
"harmonization": { "type": "object" },
|
|
"recipe": { "type": ["object", "null"] },
|
|
"suggestions": { "type": "array" },
|
|
"scope": { "type": "object" },
|
|
"codes_summed": { "type": "object" },
|
|
"aggregate_fallback": { "type": ["object", "null"] },
|
|
"transformations":{ "type": "object" },
|
|
"series_break_refs": { "type": "array", "items": { "type": "string" } },
|
|
"completion": {
|
|
"type": "object",
|
|
"description": "What `complete = TRUE` filled. `applied` is FALSE on an ordinary query. `rows_filled` counts cells added to the requested grid, and `absence_means` maps each requested year to the meaning of an absent cell there ('census_zero' in a dense_source year, 'not_reported' in a sparse_source one). Filled rows carry `value_source` in the result: 'reported', 'census_zero' (amount 0 -- Census published $0), or 'not_reported' (amount NA -- unknown).",
|
|
"properties": {
|
|
"applied": { "type": "boolean" },
|
|
"rows_filled": { "type": "integer" },
|
|
"absence_means": { "type": "object" }
|
|
}
|
|
},
|
|
"corpus_break_refs": {
|
|
"type": "array",
|
|
"items": { "type": "string" },
|
|
"description": "Ids of catalogued series breaks whose fin_code is the literal 'ALL' -- caveats about the corpus as a whole (dollar precision across 1976/1977, imputation exclusion from 2002, the dense -> sparse representation change at 2012, the government id scheme change at 2017) rather than about one item code. Selected on the break_year window alone, so they do not depend on which codes a result contains. Disjoint from series_break_refs by construction: an entry qualifies the whole result, not one series."
|
|
},
|
|
"manifest": { "type": "object" },
|
|
"sql_query": { "type": "string" }
|
|
}
|
|
}
|