#39 pushed pagination into SQL for cog_spending()/cog_revenue(); the other two verbs
were left materializing everything and slicing in R — the pattern behind the 2026-08-06
production incident. cog_gov_search() had no LIMIT at all, so an unfiltered call
returns the entire 40,336-row crosswalk.
Extracted the machinery first
R/pagination.R now owns .validate_pagination(), .paginate_sql() and .take_pagination_total(). Growing a third inline copy would have been three definitions
of what total_rows means and three places for it to drift. .verb_spendrev() now uses
the shared helpers and is unchanged in behaviour; .build_verb_sql() keeps the outer SELECT * wrapping that makes COUNT(*) OVER() see the post-GROUP BY count, which is
the subtle part worth not duplicating.
Conflict refusals deliberately stay at the call sites — each verb's conflict set differs
(complete/recipe for the money verbs, recipe alone for balances, basket mode for
search), and a shared function taking a list of conflict flags would read worse than
three explicit aborts.
One small improvement while extracting: the empty-page fallback query is passed as a
thunk, so the unpaginated SQL is only built when an offset actually lands past the
end, instead of being constructed and discarded on every paged call.
cog_gov_search()'s ORDER BY was not a total order. The issue says the page goes
"behind the existing deterministic ORDER BY" — it isn't one. ORDER BY population_acs DESC NULLS LAST leaves ties, and the entire NULLS LAST block,
in whatever order the scan produced. Invisible while every call returned the full result
set; unsound the moment you page it, because two requests can order tied rows differently
and a row is duplicated on one page while another is dropped. This is the same class of
bug as cog-api#6. Added canonical_govid as tiebreaker.
Unpaginated output changes only in the relative order of rows that were already tied, but
that is a behaviour change, so it has its own NEWS entry rather than riding along silently.
Basket mode.length(name) > 1 returns one resolved row per requested name in input
order, plus a resolution sidecar covering all of them. A page of that is not a page of
anything the caller asked for — the sidecar would still describe every name — so it is
refused with uscogdata_basket_pagination_conflict rather than silently ignoring the
arguments.
NULL default on both, so every existing call site is unchanged and cog-api adopts this
behind its existing verb_supports_pagination() formals probe with no lockstep deploy.
The page applied in SQL behind a deterministic ORDER BY.
The unpaginated count returned as total_rows via COUNT(*) OVER()in the same
scan, with a fallback second COUNT(*) only when the page is empty — otherwise an
offset past the end would report a wrong zero.
Classed refusals where the combination cannot work.
Verification
30 new tests in test-search-balances-pagination.R: walking every page reconstructs the
unpaginated result exactly, total_rows matches the unpaginated nrow(), an offset past
the end returns zero rows with a correct total, the tiebreaker makes repeated calls
reproducible, pagination_total_rows never reaches the caller's data frame, and both
refusals fire.
R CMD check --as-cran: 0 errors, 0 warnings, 1 NOTE (pre-existing — custom
DESCRIPTION fields, and the r-universe URL 404ing because registration is #47 itself).
Closes #57.
#39 pushed pagination into SQL for `cog_spending()`/`cog_revenue()`; the other two verbs
were left materializing everything and slicing in R — the pattern behind the 2026-08-06
production incident. `cog_gov_search()` had no `LIMIT` at all, so an unfiltered call
returns the entire 40,336-row crosswalk.
## Extracted the machinery first
`R/pagination.R` now owns `.validate_pagination()`, `.paginate_sql()` and
`.take_pagination_total()`. Growing a third inline copy would have been three definitions
of what `total_rows` means and three places for it to drift. `.verb_spendrev()` now uses
the shared helpers and is unchanged in behaviour; `.build_verb_sql()` keeps the outer
`SELECT *` wrapping that makes `COUNT(*) OVER()` see the post-`GROUP BY` count, which is
the subtle part worth not duplicating.
Conflict refusals deliberately stay at the call sites — each verb's conflict set differs
(`complete`/`recipe` for the money verbs, `recipe` alone for balances, basket mode for
search), and a shared function taking a list of conflict flags would read worse than
three explicit aborts.
One small improvement while extracting: the empty-page fallback query is passed as a
thunk, so the unpaginated SQL is only **built** when an offset actually lands past the
end, instead of being constructed and discarded on every paged call.
## Two things #57 did not anticipate
**`cog_gov_search()`'s `ORDER BY` was not a total order.** The issue says the page goes
"behind the existing deterministic `ORDER BY`" — it isn't one.
`ORDER BY population_acs DESC NULLS LAST` leaves ties, and the entire `NULLS LAST` block,
in whatever order the scan produced. Invisible while every call returned the full result
set; unsound the moment you page it, because two requests can order tied rows differently
and a row is duplicated on one page while another is dropped. This is the same class of
bug as cog-api#6. Added `canonical_govid` as tiebreaker.
Unpaginated output changes only in the relative order of rows that were already tied, but
that is a behaviour change, so it has its own NEWS entry rather than riding along silently.
**Basket mode.** `length(name) > 1` returns one resolved row per requested name in input
order, plus a resolution sidecar covering all of them. A page of that is not a page of
anything the caller asked for — the sidecar would still describe every name — so it is
refused with `uscogdata_basket_pagination_conflict` rather than silently ignoring the
arguments.
## Semantics, matching #39 exactly
- `NULL` default on both, so every existing call site is unchanged and cog-api adopts this
behind its existing `verb_supports_pagination()` formals probe with no lockstep deploy.
- The page applied in SQL behind a deterministic `ORDER BY`.
- The unpaginated count returned as `total_rows` via `COUNT(*) OVER()` **in the same
scan**, with a fallback second `COUNT(*)` only when the page is empty — otherwise an
offset past the end would report a wrong zero.
- Classed refusals where the combination cannot work.
## Verification
- 30 new tests in `test-search-balances-pagination.R`: walking every page reconstructs the
unpaginated result exactly, `total_rows` matches the unpaginated `nrow()`, an offset past
the end returns zero rows with a correct total, the tiebreaker makes repeated calls
reproducible, `pagination_total_rows` never reaches the caller's data frame, and both
refusals fire.
- Full suite: **1067 passed, 0 failed, 0 warnings**, 2 pre-existing `test-live-corpus.R`
skips.
- `R CMD check --as-cran`: **0 errors, 0 warnings, 1 NOTE** (pre-existing — custom
DESCRIPTION fields, and the r-universe URL 404ing because registration is #47 itself).
verbs were left materializing everything and slicing in R -- the pattern behind
the 2026-08-06 production incident. cog_gov_search() had no LIMIT at all, so an
unfiltered call returns the entire 40,336-row crosswalk.
Extracted the #39 machinery into R/pagination.R first (.validate_pagination(),
.paginate_sql(), .take_pagination_total()) rather than growing a third inline
copy: three definitions of what total_rows means is three places for it to
drift. Conflict refusals stay at the call sites because each verb's conflict
set differs. .verb_spendrev() now uses the shared helpers and is unchanged in
behaviour.
The empty-page fallback query is now passed as a thunk, so the unpaginated SQL
is only BUILT when an offset actually lands past the end instead of on every
paged call.
Two things #57 did not anticipate:
- cog_gov_search()'s ORDER BY was not a total order. population_acs DESC NULLS
LAST leaves ties -- and the whole NULL block -- in scan order, so two requests
can order them differently and a paged sweep duplicates one row while dropping
another. Added canonical_govid as tiebreaker. Unpaginated output changes only
in the relative order of already-tied rows.
- Basket mode returns one resolved row per requested name plus a sidecar
covering all of them, so a page of it is not a page of anything the caller
asked for. Refused with uscogdata_basket_pagination_conflict rather than
silently ignoring the arguments.
Both default to NULL, so cog-api adopts them behind its existing formals()
probe with no lockstep deploy.
Suite: 1067 passed, 0 failed, 0 warnings (2 pre-existing live-corpus skips).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #57.
#39 pushed pagination into SQL for
cog_spending()/cog_revenue(); the other two verbswere left materializing everything and slicing in R — the pattern behind the 2026-08-06
production incident.
cog_gov_search()had noLIMITat all, so an unfiltered callreturns the entire 40,336-row crosswalk.
Extracted the machinery first
R/pagination.Rnow owns.validate_pagination(),.paginate_sql()and.take_pagination_total(). Growing a third inline copy would have been three definitionsof what
total_rowsmeans and three places for it to drift..verb_spendrev()now usesthe shared helpers and is unchanged in behaviour;
.build_verb_sql()keeps the outerSELECT *wrapping that makesCOUNT(*) OVER()see the post-GROUP BYcount, which isthe subtle part worth not duplicating.
Conflict refusals deliberately stay at the call sites — each verb's conflict set differs
(
complete/recipefor the money verbs,recipealone for balances, basket mode forsearch), and a shared function taking a list of conflict flags would read worse than
three explicit aborts.
One small improvement while extracting: the empty-page fallback query is passed as a
thunk, so the unpaginated SQL is only built when an offset actually lands past the
end, instead of being constructed and discarded on every paged call.
Two things #57 did not anticipate
cog_gov_search()'sORDER BYwas not a total order. The issue says the page goes"behind the existing deterministic
ORDER BY" — it isn't one.ORDER BY population_acs DESC NULLS LASTleaves ties, and the entireNULLS LASTblock,in whatever order the scan produced. Invisible while every call returned the full result
set; unsound the moment you page it, because two requests can order tied rows differently
and a row is duplicated on one page while another is dropped. This is the same class of
bug as cog-api#6. Added
canonical_govidas tiebreaker.Unpaginated output changes only in the relative order of rows that were already tied, but
that is a behaviour change, so it has its own NEWS entry rather than riding along silently.
Basket mode.
length(name) > 1returns one resolved row per requested name in inputorder, plus a resolution sidecar covering all of them. A page of that is not a page of
anything the caller asked for — the sidecar would still describe every name — so it is
refused with
uscogdata_basket_pagination_conflictrather than silently ignoring thearguments.
Semantics, matching #39 exactly
NULLdefault on both, so every existing call site is unchanged and cog-api adopts thisbehind its existing
verb_supports_pagination()formals probe with no lockstep deploy.ORDER BY.total_rowsviaCOUNT(*) OVER()in the samescan, with a fallback second
COUNT(*)only when the page is empty — otherwise anoffset past the end would report a wrong zero.
Verification
test-search-balances-pagination.R: walking every page reconstructs theunpaginated result exactly,
total_rowsmatches the unpaginatednrow(), an offset pastthe end returns zero rows with a correct total, the tiebreaker makes repeated calls
reproducible,
pagination_total_rowsnever reaches the caller's data frame, and bothrefusals fire.
test-live-corpus.Rskips.
R CMD check --as-cran: 0 errors, 0 warnings, 1 NOTE (pre-existing — customDESCRIPTION fields, and the r-universe URL 404ing because registration is #47 itself).
98bf0d9720to2bb9d41d72