fix: scope suppressed-dollar measurement to the calling verb's own flow family (#9)
Final whole-branch review fix wave for the partial-coverage signposting feature: - I1: .suppressed_components() now filters measured recipe components to the calling verb's own flow_prefixes. Without this, a candidate recipe from the OTHER flow family was always absent from the verb's own view by construction and so was always reported as "suppressed" -- fabricating a dollar claim across flow families (cog_revenue(category = "Corrections") claimed $3.63B excluded that cog_spending() actually reports in full). - I2: reworded provenance-v1.json's trigger/suppressed_amount descriptions to describe what the code actually measures (the verb's underlying long view, not "the result"), and to note suppressed_amount can be negative. Added ig_recipe_id to the suggestions items' required list, matching the key's always-set/nullable runtime behavior. - I3(a): restated the year/govid literals inside .suppressed_components()'s NOT EXISTS subquery so DuckDB can partition-prune that side too (verified via EXPLAIN: Scanning Files 1/4 instead of an unfiltered full scan; all.equal(old, new) results confirmed unchanged). - I3(b): added .needs_suppression_query(), a free, exact pre-check reusing the verb's own already-computed result$codes_included to skip the anti- join round trip on the common fully-covered path, without weakening the "suppression can fire with zero gap years" guarantee. - M4: corrected the overbroad "confines every fire to 2011" scope claim in R/suggestions.R and NEWS.md -- the suppressed-dollar measurement is now flow-scoped (post-I1), but the empty_year trigger itself is not, and can still fire in modern years for a mis-scoped cross-flow-family category. Added a regression test for I1 plus direct unit-test coverage for the new flow-family filter and the I3(b) pre-check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -38,8 +38,8 @@
|
||||
"description": "Harmonization recipes that would fill incomplete coverage in the requested years for this government. Empty on a healthy query, on an un-scoped (category = NULL) query, on basis = 'raw', and on a recipe = query (which resolves its own coverage).",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["recipe_id", "label", "available_years", "hint", "trigger",
|
||||
"suppressed_amount", "suppressed_years", "suppressed_codes"],
|
||||
"required": ["recipe_id", "label", "available_years", "hint", "ig_recipe_id",
|
||||
"trigger", "suppressed_amount", "suppressed_years", "suppressed_codes"],
|
||||
"properties": {
|
||||
"recipe_id": { "type": "string" },
|
||||
"label": { "type": "string" },
|
||||
@@ -56,11 +56,11 @@
|
||||
"trigger": {
|
||||
"type": "string",
|
||||
"enum": ["empty_year", "suppressed_component"],
|
||||
"description": "Why this fired. 'empty_year': the result has no rows at all in a requested year. 'suppressed_component': the result HAS rows, but a component code carries dollars the verb's long view structurally excludes -- aggregate-published, or absent from summary_categories. 'empty_year' wins when both apply, being the stronger claim; the suppressed_* fields are populated either way."
|
||||
"description": "Why this fired. 'empty_year': the result has no rows at all in a requested year. 'suppressed_component': the result HAS rows, but a component code carries dollars this government reports in the requested years that the verb's underlying long view structurally excludes -- aggregate-published, carrying no harmonized code, or absent from summary_categories. This is NOT the same thing as 'excluded from the result': a component present in the view under a different category (a scoping choice, e.g. a different `category` or a narrower `expenditure_concept`) contributes 0 and never fires. 'empty_year' wins when both apply, being the stronger claim; the suppressed_* fields are populated either way, using the same underlying-view measurement, and can be 0 even on an 'empty_year' fire."
|
||||
},
|
||||
"suppressed_amount": {
|
||||
"type": "number",
|
||||
"description": "Full US dollars this government holds in the recipe's component codes that the result excludes, summed across the requested years. 0 when nothing is suppressed."
|
||||
"description": "Full US dollars this government reports, in the recipe's component codes, in the requested years, that the verb's underlying long view structurally excludes (aggregate-published, carrying no harmonized code, or absent from summary_categories) -- summed across those years. This is NOT the same quantity as 'what the result excludes': a component present in the view under a different category or a narrower `expenditure_concept` is scoped out on purpose, counts as 0 here, and is not suppression. 0 does not always mean full coverage -- see 'trigger' and 'empty_year'. May be negative where Census publishes a negative `amt` for the excluded rows."
|
||||
},
|
||||
"suppressed_years": {
|
||||
"type": "array",
|
||||
|
||||
Reference in New Issue
Block a user