feat: limit/offset on cog_gov_search() and cog_balances() (#57)
verbs were left materializing everything and slicing in R -- the pattern behind the 2026-08-06 production incident. cog_gov_search() had no LIMIT at all, so an unfiltered call returns the entire 40,336-row crosswalk. Extracted the #39 machinery into R/pagination.R first (.validate_pagination(), .paginate_sql(), .take_pagination_total()) rather than growing a third inline copy: three definitions of what total_rows means is three places for it to drift. Conflict refusals stay at the call sites because each verb's conflict set differs. .verb_spendrev() now uses the shared helpers and is unchanged in behaviour. The empty-page fallback query is now passed as a thunk, so the unpaginated SQL is only BUILT when an offset actually lands past the end instead of on every paged call. Two things #57 did not anticipate: - cog_gov_search()'s ORDER BY was not a total order. population_acs DESC NULLS LAST leaves ties -- and the whole NULL block -- in scan order, so two requests can order them differently and a paged sweep duplicates one row while dropping another. Added canonical_govid as tiebreaker. Unpaginated output changes only in the relative order of already-tied rows. - Basket mode returns one resolved row per requested name plus a sidecar covering all of them, so a page of it is not a page of anything the caller asked for. Refused with uscogdata_basket_pagination_conflict rather than silently ignoring the arguments. Both default to NULL, so cog-api adopts them behind its existing formals() probe with no lockstep deploy. Suite: 1067 passed, 0 failed, 0 warnings (2 pre-existing live-corpus skips).
This commit is contained in:
+15
-1
@@ -13,7 +13,9 @@ cog_balances(
|
||||
basis = c("harmonized", "raw"),
|
||||
recipe = NULL,
|
||||
state = NULL,
|
||||
type = NULL
|
||||
type = NULL,
|
||||
limit = NULL,
|
||||
offset = NULL
|
||||
)
|
||||
}
|
||||
\arguments{
|
||||
@@ -76,6 +78,14 @@ wide era to the modern one.}
|
||||
and `govids_missing` are empty -- there is no id list to report against --
|
||||
and `provenance$scope$cohort` carries `state`, `type` and
|
||||
`n_governments` instead. A `govid`-named cohort reports exactly as before.}
|
||||
|
||||
\item{limit}{Maximum number of result rows to return, pushed into the SQL
|
||||
rather than applied after materializing every row. `NULL` (default)
|
||||
returns everything. Cannot be combined with `recipe` -- see `offset` and
|
||||
`total_rows`.}
|
||||
|
||||
\item{offset}{Rows to skip before `limit` starts counting (0-based).
|
||||
Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
|
||||
}
|
||||
\value{
|
||||
Tibble with columns `year`, `canonical_govid`, `gov_name`,
|
||||
@@ -94,6 +104,10 @@ Tibble with columns `year`, `canonical_govid`, `gov_name`,
|
||||
and `truncated` (the observed subtypes whose coverage falls short of the
|
||||
requested years). `expenditure_concept`/`revenue_concept` are `NA` --
|
||||
holdings are a stock, not a flow, so neither concept vocabulary applies.
|
||||
|
||||
When `limit` is set, also carries a `total_rows` attribute: the full
|
||||
unpaginated row count, computed by the same query (`COUNT(*) OVER()`)
|
||||
rather than a second scan.
|
||||
}
|
||||
\description{
|
||||
Returns Census cash-and-security holdings (`category_type = "balance"`):
|
||||
|
||||
Reference in New Issue
Block a user