feat: limit/offset on cog_gov_search() and cog_balances() (#57)
R-CMD-check / check (pull_request) Successful in 4m22s
R-CMD-check / check (push) Successful in 4m28s

verbs were left materializing everything and slicing in R -- the pattern behind
the 2026-08-06 production incident. cog_gov_search() had no LIMIT at all, so an
unfiltered call returns the entire 40,336-row crosswalk.

Extracted the #39 machinery into R/pagination.R first (.validate_pagination(),
.paginate_sql(), .take_pagination_total()) rather than growing a third inline
copy: three definitions of what total_rows means is three places for it to
drift. Conflict refusals stay at the call sites because each verb's conflict
set differs. .verb_spendrev() now uses the shared helpers and is unchanged in
behaviour.

The empty-page fallback query is now passed as a thunk, so the unpaginated SQL
is only BUILT when an offset actually lands past the end instead of on every
paged call.

Two things #57 did not anticipate:

- cog_gov_search()'s ORDER BY was not a total order. population_acs DESC NULLS
  LAST leaves ties -- and the whole NULL block -- in scan order, so two requests
  can order them differently and a paged sweep duplicates one row while dropping
  another. Added canonical_govid as tiebreaker. Unpaginated output changes only
  in the relative order of already-tied rows.

- Basket mode returns one resolved row per requested name plus a sidecar
  covering all of them, so a page of it is not a page of anything the caller
  asked for. Refused with uscogdata_basket_pagination_conflict rather than
  silently ignoring the arguments.

Both default to NULL, so cog-api adopts them behind its existing formals()
probe with no lockstep deploy.

Suite: 1067 passed, 0 failed, 0 warnings (2 pre-existing live-corpus skips).
This commit is contained in:
2026-08-10 19:27:11 -04:00
parent 700ae93c9c
commit 2bb9d41d72
8 changed files with 433 additions and 58 deletions
+15 -1
View File
@@ -13,7 +13,9 @@ cog_balances(
basis = c("harmonized", "raw"),
recipe = NULL,
state = NULL,
type = NULL
type = NULL,
limit = NULL,
offset = NULL
)
}
\arguments{
@@ -76,6 +78,14 @@ wide era to the modern one.}
and `govids_missing` are empty -- there is no id list to report against --
and `provenance$scope$cohort` carries `state`, `type` and
`n_governments` instead. A `govid`-named cohort reports exactly as before.}
\item{limit}{Maximum number of result rows to return, pushed into the SQL
rather than applied after materializing every row. `NULL` (default)
returns everything. Cannot be combined with `recipe` -- see `offset` and
`total_rows`.}
\item{offset}{Rows to skip before `limit` starts counting (0-based).
Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
}
\value{
Tibble with columns `year`, `canonical_govid`, `gov_name`,
@@ -94,6 +104,10 @@ Tibble with columns `year`, `canonical_govid`, `gov_name`,
and `truncated` (the observed subtypes whose coverage falls short of the
requested years). `expenditure_concept`/`revenue_concept` are `NA` --
holdings are a stock, not a flow, so neither concept vocabulary applies.
When `limit` is set, also carries a `total_rows` attribute: the full
unpaginated row count, computed by the same query (`COUNT(*) OVER()`)
rather than a second scan.
}
\description{
Returns Census cash-and-security holdings (`category_type = "balance"`):
+24 -4
View File
@@ -4,7 +4,13 @@
\alias{cog_gov_search}
\title{Search for governments by name, state, and/or type}
\usage{
cog_gov_search(name = NULL, state = NULL, type = NULL)
cog_gov_search(
name = NULL,
state = NULL,
type = NULL,
limit = NULL,
offset = NULL
)
}
\arguments{
\item{name}{Character vector of place name(s). Length 1 = utility mode;
@@ -19,12 +25,26 @@ all entries; otherwise must match `length(name)`.}
in basket mode (recycles from length 1). Excluded types `4`/`5` (or
`"special_district"` / `"school_district"`) trigger an explanatory
message and an empty result.}
\item{limit}{Maximum number of rows to return, applied in SQL. `NULL`
(default) returns every match -- which, with no other filter, is the
entire crosswalk. Utility mode only: pagination has no meaning in basket
mode, where the result is one resolved row per requested name in input
order, and is refused there with class
`uscogdata_basket_pagination_conflict`.}
\item{offset}{Rows to skip before `limit` starts counting (0-based).
Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
}
\value{
A tibble of `canonical_fips_xwalk` rows. In utility mode, all
matches sorted by `population_acs` desc. In basket mode, resolved
rows in input order, with `attr(., "resolution")` set to the
sidecar tibble.
matches sorted by `population_acs` desc, ties broken by
`canonical_govid`. In basket mode, resolved rows in input order, with
`attr(., "resolution")` set to the sidecar tibble.
When `limit` is set, carries a `total_rows` attribute: the full
unpaginated match count, computed by the same query (`COUNT(*) OVER()`)
rather than a second scan.
}
\description{
Resolves human-readable place names into rows of `canonical_fips_xwalk`,