jaredandClaude Opus 5
4b749205a5
Final whole-branch review fix wave for the partial-coverage signposting feature: - I1: .suppressed_components() now filters measured recipe components to the calling verb's own flow_prefixes. Without this, a candidate recipe from the OTHER flow family was always absent from the verb's own view by construction and so was always reported as "suppressed" -- fabricating a dollar claim across flow families (cog_revenue(category = "Corrections") claimed $3.63B excluded that cog_spending() actually reports in full). - I2: reworded provenance-v1.json's trigger/suppressed_amount descriptions to describe what the code actually measures (the verb's underlying long view, not "the result"), and to note suppressed_amount can be negative. Added ig_recipe_id to the suggestions items' required list, matching the key's always-set/nullable runtime behavior. - I3(a): restated the year/govid literals inside .suppressed_components()'s NOT EXISTS subquery so DuckDB can partition-prune that side too (verified via EXPLAIN: Scanning Files 1/4 instead of an unfiltered full scan; all.equal(old, new) results confirmed unchanged). - I3(b): added .needs_suppression_query(), a free, exact pre-check reusing the verb's own already-computed result$codes_included to skip the anti- join round trip on the common fully-covered path, without weakening the "suppression can fire with zero gap years" guarantee. - M4: corrected the overbroad "confines every fire to 2011" scope claim in R/suggestions.R and NEWS.md -- the suppressed-dollar measurement is now flow-scoped (post-I1), but the empty_year trigger itself is not, and can still fire in modern years for a mis-scoped cross-flow-family category. Added a regression test for I1 plus direct unit-test coverage for the new flow-family filter and the I3(b) pre-check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
111 lines
8.6 KiB
JSON
111 lines
8.6 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"$id": "https://civilytics.org/schemas/uscogdata/provenance-v1.json",
|
|
"title": "uscogdata provenance v1",
|
|
"type": "object",
|
|
"required": ["verb", "target", "years", "scope", "manifest", "sql_query"],
|
|
"properties": {
|
|
"verb": { "type": "string" },
|
|
"call": { "type": "string" },
|
|
"target": { "type": "object" },
|
|
"years": { "type": "array", "items": { "type": "integer" } },
|
|
"category": { "type": ["string", "array", "null"] },
|
|
"basis": { "type": ["string", "null"] },
|
|
"basis_note": { "type": ["string", "null"] },
|
|
"expenditure_concept": {
|
|
"type": "string",
|
|
"enum": ["primary", "direct", "total"],
|
|
"description": "Which spending concept produced this result, defined as crosswalk spend_subtype sets (never item-code prefixes). 'primary' (the default) is the government's own service provision: operations + capital + assistance. 'direct' adds interest on debt and insurance trust benefit payments (Census's published Direct Expenditure). 'total' adds intergovernmental payments (M to local governments, L to state government, Q11/Q12/Q18 to school systems). Only 'primary' and 'direct' are valid for results combined across governments."
|
|
},
|
|
"expenditure_concept_note": {
|
|
"type": ["string", "null"],
|
|
"description": "How the intergovernmental leg was assembled; null for 'primary' and 'direct'."
|
|
},
|
|
"expenditure_concept_direct_suppressed": {
|
|
"type": "boolean",
|
|
"description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'primary' or 'direct'. See the affected rows' `notes` for the recovering recipe, if any."
|
|
},
|
|
"revenue_concept": {
|
|
"type": "string",
|
|
"enum": ["general", "total"],
|
|
"description": "Which revenue concept produced this result, defined as crosswalk revenue_subtype sets (never item-code prefixes). 'general' (the default) is Census General Revenue: own_source + federal + state + local_aid. 'total' is Census Total Revenue: general plus utility, liquor store and insurance trust revenue. Census defines the first by subtracting the other three from the second (manual section 4.3). Meaningful for cog_revenue() results; spending results carry the default.",
|
|
"$comment": "The employee-retirement (X) codes inside insurance_trust stop at FY2016, so a 'total' series steps at the FY2016/FY2017 seam for collection-scope reasons (series breaks SB197-SB202)."
|
|
},
|
|
"harmonization": { "type": "object" },
|
|
"recipe": { "type": ["object", "null"] },
|
|
"suggestions": {
|
|
"type": "array",
|
|
"description": "Harmonization recipes that would fill incomplete coverage in the requested years for this government. Empty on a healthy query, on an un-scoped (category = NULL) query, on basis = 'raw', and on a recipe = query (which resolves its own coverage).",
|
|
"items": {
|
|
"type": "object",
|
|
"required": ["recipe_id", "label", "available_years", "hint", "ig_recipe_id",
|
|
"trigger", "suppressed_amount", "suppressed_years", "suppressed_codes"],
|
|
"properties": {
|
|
"recipe_id": { "type": "string" },
|
|
"label": { "type": "string" },
|
|
"available_years": {
|
|
"type": "array",
|
|
"items": { "type": "integer" },
|
|
"description": "[year_min, year_max] of the recipe's component coverage."
|
|
},
|
|
"hint": { "type": "string" },
|
|
"ig_recipe_id": {
|
|
"type": ["string", "null"],
|
|
"description": "The intergovernmental (M/L) counterpart recipe covering the same function suffixes, or null. Never set for revenue recipes."
|
|
},
|
|
"trigger": {
|
|
"type": "string",
|
|
"enum": ["empty_year", "suppressed_component"],
|
|
"description": "Why this fired. 'empty_year': the result has no rows at all in a requested year. 'suppressed_component': the result HAS rows, but a component code carries dollars this government reports in the requested years that the verb's underlying long view structurally excludes -- aggregate-published, carrying no harmonized code, or absent from summary_categories. This is NOT the same thing as 'excluded from the result': a component present in the view under a different category (a scoping choice, e.g. a different `category` or a narrower `expenditure_concept`) contributes 0 and never fires. 'empty_year' wins when both apply, being the stronger claim; the suppressed_* fields are populated either way, using the same underlying-view measurement, and can be 0 even on an 'empty_year' fire."
|
|
},
|
|
"suppressed_amount": {
|
|
"type": "number",
|
|
"description": "Full US dollars this government reports, in the recipe's component codes, in the requested years, that the verb's underlying long view structurally excludes (aggregate-published, carrying no harmonized code, or absent from summary_categories) -- summed across those years. This is NOT the same quantity as 'what the result excludes': a component present in the view under a different category or a narrower `expenditure_concept` is scoped out on purpose, counts as 0 here, and is not suppression. 0 does not always mean full coverage -- see 'trigger' and 'empty_year'. May be negative where Census publishes a negative `amt` for the excluded rows."
|
|
},
|
|
"suppressed_years": {
|
|
"type": "array",
|
|
"items": { "type": "integer" },
|
|
"description": "The requested years contributing to suppressed_amount."
|
|
},
|
|
"suppressed_codes": {
|
|
"type": "array",
|
|
"items": { "type": "string" },
|
|
"description": "The excluded component item codes, sorted."
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"scope": { "type": "object" },
|
|
"codes_summed": { "type": "object" },
|
|
"aggregate_fallback": { "type": ["object", "null"] },
|
|
"transformations":{ "type": "object" },
|
|
"series_break_refs": { "type": "array", "items": { "type": "string" } },
|
|
"completion": {
|
|
"type": "object",
|
|
"description": "What `complete = TRUE` filled. `applied` is FALSE on an ordinary query. `rows_filled` counts cells added to the requested grid, and `absence_means` maps each requested year to the meaning of an absent cell there ('census_zero' in a dense_source year, 'not_reported' in a sparse_source one). Filled rows carry `value_source` in the result: 'reported', 'census_zero' (amount 0 -- Census published $0), or 'not_reported' (amount NA -- unknown).",
|
|
"properties": {
|
|
"applied": { "type": "boolean" },
|
|
"rows_filled": { "type": "integer" },
|
|
"absence_means": { "type": "object" }
|
|
}
|
|
},
|
|
"corpus_break_refs": {
|
|
"type": "array",
|
|
"items": { "type": "string" },
|
|
"description": "Ids of catalogued series breaks whose fin_code is the literal 'ALL' -- caveats about the corpus as a whole (dollar precision across 1976/1977, imputation exclusion from 2002, the dense -> sparse representation change at 2012, the government id scheme change at 2017) rather than about one item code. Selected on the break_year window alone, so they do not depend on which codes a result contains. Disjoint from series_break_refs by construction: an entry qualifies the whole result, not one series."
|
|
},
|
|
"balance_caveats": {
|
|
"type": ["object", "null"],
|
|
"description": "Present only on cog_balances() results (null/absent for cog_spending()/cog_revenue()). `not_gaap` is always TRUE and `not_gaap_note` explains that Census holdings are gross -- no liabilities are netted -- so they are NOT comparable to a GAAP fund balance. `coverage_window` maps EVERY balance_subtype present in the mounted corpus -- not only the ones this query observed -- to its measured [min year, max year] there (never hardcoded), so a caller can see which families exist and over what span before deciding they missed one. `truncated` is the query-scoped field: it lists only the subtypes this result actually observed whose coverage_window does not fully span the requested years.",
|
|
"properties": {
|
|
"not_gaap": { "type": "boolean" },
|
|
"not_gaap_note": { "type": "string" },
|
|
"coverage_window": { "type": "object" },
|
|
"truncated": { "type": "array", "items": { "type": "string" } }
|
|
}
|
|
},
|
|
"manifest": { "type": "object" },
|
|
"sql_query": { "type": "string" }
|
|
}
|
|
}
|