fix: scope all-categories suggestion candidates by subtype, not category (finding 6)
R-CMD-check / check (push) Successful in 3m41s
R-CMD-check / check (pull_request) Successful in 4m52s

.build_suggestions()'s recipe-candidate sub-select was keyed on
`WHERE category IN (<category>)`. The reserved pseudo-category
"All Categories" is never itself a row in summary_categories.category, so
in all-categories mode `candidates` always came back empty and coverage
signposting (uscogdata#9) was structurally impossible for the one mode
whose entire premise is "you cannot sum the wrong scope" -- measured on
Los Angeles County FY2011: category = "Public Welfare" reports 2
suggestions (incl. $271,589,000 excluded E68), category = "All Categories"
reported 0, silently losing that same signal.

Apply the branch's own design principle: the concept boundary is subtype,
not category. .build_suggestions() now accepts all_categories/subtype_col/
subtype_scope (all optional, default off, so no other caller's behaviour
changes) and, when all-categories mode is active, scopes the candidate
sub-select by `<subtype_col> IN (<subtype_scope>)` instead -- symmetric
with .build_verb_sql()'s own WHERE predicate. The M/L recipe exclusion and
the is.null(category) early return are unchanged.

After the fix, LA County FY2011 "All Categories" reports 5 suggestions,
including welfare_cash_e68_wide for the exact $271,589,000 gap.

Adds two covering tests to test-all-categories.R using the bundled fixture
(AL state gov, FY2011, "Corrections"): one end-to-end (per-category and
all-categories both signpost the same recipe) and one direct on
.build_suggestions() proving the subtype-vs-category branch is what
changes the query. Updates the 0.2.0 NEWS entry.
This commit is contained in:
2026-08-05 12:38:12 -04:00
parent a5f86d87b3
commit 498950afa6
4 changed files with 139 additions and 4 deletions
+4 -1
View File
@@ -442,7 +442,10 @@ cog_spending <- function(govid, years, category = NULL,
suggestions <- .build_suggestions(con, govid, years, category,
direct_leg_result,
resolved$basis, flow_prefixes,
.select_long_view(view_base, resolved$basis))
.select_long_view(view_base, resolved$basis),
all_categories = all_categories,
subtype_col = subtype_col,
subtype_scope = subtype_scope)
}
# C1(b): when expenditure_concept = "total", flag any row where the IG
+46 -3
View File
@@ -59,12 +59,36 @@
#' @param long_view Name of the verb's own long view (from
#' `.select_long_view()`), passed through to `.suppressed_components()` to
#' measure the second qualifying path (uscogdata#9).
#' @param all_categories `TRUE` when the caller's `category` is the reserved
#' pseudo-category (`.ALL_CATEGORIES`). Defaults to `FALSE` so no other
#' caller's behaviour changes. When `TRUE`, the candidate-recipe sub-select
#' is scoped by `subtype_col`/`subtype_scope` instead of by `category` --
#' symmetric with `.build_verb_sql()`'s own all-categories branch (see
#' R/spending.R): the concept's subtype allowlist is the real scope
#' boundary, not any literal category value, and
#' `.ALL_CATEGORIES` ("All Categories") is never itself a row in
#' `summary_categories.category`, so leaving the category-keyed sub-select
#' in place here always returned zero candidates and silently disabled
#' signposting in all-categories mode (final whole-branch review, finding
#' 6).
#' @param subtype_col Name of the `summary_categories` subtype column to
#' scope by when `all_categories = TRUE` (`"spend_subtype"` or
#' `"revenue_subtype"` -- the same value `.build_verb_sql()` already
#' receives as its own `subtype_col`). Ignored when `all_categories =
#' FALSE`. `NULL` by default.
#' @param subtype_scope Character vector of subtype values to scope by when
#' `all_categories = TRUE` (the same value `.build_verb_sql()` already
#' receives as its own `subtype_scope` -- the concept's subtype allowlist,
#' e.g. `.expenditure_concept_subtypes(expenditure_concept)`). Ignored when
#' `all_categories = FALSE`. `NULL` by default.
#' @return List of `list(recipe_id, label, available_years, hint,
#' ig_recipe_id, trigger, suppressed_amount, suppressed_years,
#' suppressed_codes)`, possibly empty.
#' @noRd
.build_suggestions <- function(con, govid, years, category, result, basis,
flow_prefixes, long_view) {
flow_prefixes, long_view,
all_categories = FALSE,
subtype_col = NULL, subtype_scope = NULL) {
if (!identical(basis, "harmonized") || is.null(category)) return(list())
# Exclude any recipe that is ITSELF an intergovernmental (M/L) recipe --
@@ -79,16 +103,35 @@
# flow-prefix gate below/in `.attach_ig_counterparts()`: an M/L recipe
# should never be suggested as a coverage-gap filler for EITHER verb, not
# just kept from being named as the *counterpart* of another suggestion.
#
# The inner sub-select is the concept boundary (finding 6, final
# whole-branch review): in all-categories mode it is scoped by
# `subtype_col`/`subtype_scope` -- the same allowlist `.build_verb_sql()`
# applies as a WHERE predicate to make the summed result a *concept*, not
# by `category` (`.ALL_CATEGORIES` is never a row in
# `summary_categories.category`, so a category-keyed sub-select always
# came back empty here). The M/L exclusion below is unchanged either way.
candidate_scope_sql <- if (isTRUE(all_categories)) {
sprintf(
"SELECT DISTINCT item_code FROM summary_categories WHERE %s IN (%s)",
subtype_col, .sql_lit_chr(subtype_scope)
)
} else {
sprintf(
"SELECT DISTINCT item_code FROM summary_categories WHERE category IN (%s)",
.sql_lit_chr(category)
)
}
candidates <- DBI::dbGetQuery(con, sprintf(
"SELECT DISTINCT recipe_id FROM harmonization_recipes
WHERE component_code IN (
SELECT DISTINCT item_code FROM summary_categories WHERE category IN (%s)
%s
)
AND recipe_id NOT IN (
SELECT DISTINCT recipe_id FROM harmonization_recipes
WHERE LEFT(component_code, 1) IN ('M', 'L')
)",
.sql_lit_chr(category)
candidate_scope_sql
))$recipe_id
if (length(candidates) == 0L) return(list())