feat: name a cohort by state/type predicate instead of a 40k-id IN list
R-CMD-check / check (push) Successful in 4m29s
R-CMD-check / check (pull_request) Successful in 4m19s

cog_spending(), cog_revenue() and cog_balances() gain optional state/type
arguments. Both default to NULL, so every existing govid-based call is
unchanged.

The verbs took a cohort only as a govid vector, which .sql_lit_chr()
rendered into a quoted IN list and .verb_spendrev() embedded into 5-8
separate statements per call: the scope check, the main aggregate, the
per-capita join, the harmonization block, and the suggestion and
suppression queries. For type = "city" that list is 301,589 characters,
parsed and planned from scratch every time it appears.

Passing state/type instead expresses the cohort as a subquery against
canonical_fips_xwalk, so its size never enters the SQL string at all.

Measured on the production corpus, same FY2022 aggregate over the
20,106-government city cohort, DUCKDB_THREADS=2, median of 5:

  IN (20,106 literals) -- 0.3.0            432 ms
  join against a temp cohort table         132 ms
  predicate on canonical_fips_xwalk         102 ms
  no cohort filter at all (the floor)      105 ms

The predicate reaches the no-filter floor: the cohort restriction is
now free. End to end through cog_spending(category = "Police"),
1080 ms -> 271 ms, 3.99x -- larger than the single-query saving,
because the repetition across statements is what actually cost.

Design decisions, both made explicitly rather than left implicit:

  - govid AND state/type INTERSECT. "These ids, narrowed to that
    state/type" is a real query, and an error here could never be
    relaxed later without breaking callers.
  - A predicate cohort has no id list to report, so
    provenance$scope$govids_found/govids_missing stay empty and a new
    scope$cohort block carries state, type and n_governments. Resolving
    the ids just to report them would put 20,000 govids in every
    fleet-scale response body -- the cost this change removes. A
    govid-named cohort's provenance is untouched.

state/type are coerced with .coerce_state_to_fips()/.coerce_type(), the
same helpers cog_gov_search() uses. That is load-bearing: the argument
is a postal abbreviation ("WI") while fips_state holds a FIPS code
("55"), and a predicate on the raw parameter matches nothing and returns
an empty result indistinguishable from "reported nothing". cog-api hit
exactly this trap optimizing the same path.

.attach_per_capita() now keys its population lookup on the govids present
in the result rather than the requested cohort. Those are the only ones
its LEFT JOIN can match, so the output is identical -- but it needs no id
list, and on a paginated call it looks up one page instead of the fleet.

Fixes uscogdata#58.
This commit is contained in:
2026-08-09 14:15:21 -04:00
parent 698812a25c
commit 0fbae00e27
19 changed files with 714 additions and 75 deletions
+127
View File
@@ -0,0 +1,127 @@
# How a verb names the set of governments it queries.
#
# Historically there was one way: a `govid` character vector, rendered by
# .sql_lit_chr() into a quoted IN list. That is fine for a handful of
# governments and pathological for a fleet. Measured against the production
# corpus, the same FY2022 aggregate over the 20,106-government `type = "city"`
# cohort:
#
# cohort expressed as time
# IN (20,106 literals) 449 ms
# join against a temp cohort table 99 ms
# predicate on canonical_fips_xwalk 94 ms
# no cohort filter at all (the floor) 88 ms
#
# 4.8x, and within 7% of the no-filter floor. The rendered IN list is 301,591
# characters and .verb_spendrev() embeds it in 5-8 separate statements per
# call, so the parse-and-plan cost is paid over and over (uscogdata#58).
#
# A cohort therefore has two independent halves, and a query can carry either
# or both:
#
# ids an explicit canonical_govid vector -> literal IN list
# predicate state/type over canonical_fips_xwalk -> IN (SELECT ...)
#
# Both together is an INTERSECTION -- "these ids, narrowed to that state/type"
# -- never a precedence rule where one silently wins.
#' Build the internal cohort object shared by every query verb.
#'
#' `state` and `type` are coerced with the SAME helpers `cog_gov_search()`
#' uses. That is load-bearing, not tidiness: the public argument is a postal
#' abbreviation (`"WI"`) while `canonical_fips_xwalk.fips_state` holds a FIPS
#' code (`"55"`), and `type` is a label (`"city"`) against an integer
#' `govs_type`. A predicate written against the raw parameter matches nothing
#' and returns an empty result indistinguishable from "this government
#' reported nothing" -- cog-api hit exactly that trap optimizing this path.
#' One definition of the translation, not two.
#'
#' @param govid Already-coerced character vector of canonical_govids, or NULL.
#' @param state Postal abbreviation or FIPS code, or NULL.
#' @param type Type label or integer code, or NULL.
#' @noRd
.make_cohort <- function(govid = NULL, state = NULL, type = NULL) {
if (is.null(govid) && is.null(state) && is.null(type)) {
cli::cli_abort(c(
"A cohort must be named.",
"*" = "Pass {.arg govid} for specific governments, or {.arg state}/{.arg type} for every government matching a predicate.",
"i" = "Passing both intersects them: the governments in {.arg govid} that also match {.arg state}/{.arg type}."
), class = "uscogdata_no_cohort")
}
structure(
list(
ids = govid,
state = state,
type = type,
state_fips = if (is.null(state)) NULL else .coerce_state_to_fips(state),
type_int = if (is.null(type)) NULL else .coerce_type(type)
),
class = "uscogdata_cohort"
)
}
#' Is any part of this cohort expressed as an xwalk predicate?
#' @noRd
.cohort_by_predicate <- function(cohort) {
!is.null(cohort$state_fips) || !is.null(cohort$type_int)
}
#' Render the cohort as a SQL boolean expression over `col`.
#'
#' `col` may be qualified (`"l.canonical_govid"`, `"x.canonical_govid"`) --
#' several call sites join the xwalk under an alias. The subquery's own
#' projected column stays unqualified: it selects from canonical_fips_xwalk,
#' not from the outer relation.
#' @noRd
.cohort_sql <- function(cohort, col = "canonical_govid") {
preds <- character(0)
if (!is.null(cohort$ids)) {
preds <- c(preds, sprintf("%s IN (%s)", col, .sql_lit_chr(cohort$ids)))
}
if (.cohort_by_predicate(cohort)) {
xwalk_preds <- character(0)
if (!is.null(cohort$state_fips)) {
xwalk_preds <- c(xwalk_preds,
sprintf("fips_state = %s", .sql_lit_chr(cohort$state_fips)))
}
if (!is.null(cohort$type_int)) {
xwalk_preds <- c(xwalk_preds, sprintf("govs_type = %d", cohort$type_int))
}
preds <- c(preds, sprintf(
"%s IN (SELECT canonical_govid FROM canonical_fips_xwalk WHERE %s)",
col, paste(xwalk_preds, collapse = " AND ")
))
}
paste(preds, collapse = " AND ")
}
#' How many governments the cohort covers.
#'
#' One COUNT against the crosswalk, used only to populate the provenance
#' `scope$cohort` block. Deliberately a count rather than the id list: a
#' fleet-scale cohort would otherwise put 20,000 ids into every response body,
#' which is the cost this issue exists to remove.
#' @noRd
.cohort_count <- function(con, cohort) {
sql <- sprintf(
"SELECT COUNT(*) AS n FROM canonical_fips_xwalk WHERE %s",
.cohort_sql(cohort)
)
as.integer(DBI::dbGetQuery(con, sql)$n[[1]])
}
#' The provenance `scope$cohort` block for a predicate cohort, or NULL when
#' the cohort was named by id alone (in which case `govids_found`/
#' `govids_missing` already describe it exactly).
#' @noRd
.cohort_provenance <- function(con, cohort) {
if (!.cohort_by_predicate(cohort)) return(NULL)
list(
state = if (is.null(cohort$state)) NA_character_ else as.character(cohort$state),
type = if (is.null(cohort$type)) NA_character_ else as.character(cohort$type),
n_governments = .cohort_count(con, cohort)
)
}