feat: limit/offset on cog_gov_search() and cog_balances() (#57)

verbs were left materializing everything and slicing in R -- the pattern behind
the 2026-08-06 production incident. cog_gov_search() had no LIMIT at all, so an
unfiltered call returns the entire 40,336-row crosswalk.

Extracted the #39 machinery into R/pagination.R first (.validate_pagination(),
.paginate_sql(), .take_pagination_total()) rather than growing a third inline
copy: three definitions of what total_rows means is three places for it to
drift. Conflict refusals stay at the call sites because each verb's conflict
set differs. .verb_spendrev() now uses the shared helpers and is unchanged in
behaviour.

The empty-page fallback query is now passed as a thunk, so the unpaginated SQL
is only BUILT when an offset actually lands past the end instead of on every
paged call.

Two things #57 did not anticipate:

- cog_gov_search()'s ORDER BY was not a total order. population_acs DESC NULLS
  LAST leaves ties -- and the whole NULL block -- in scan order, so two requests
  can order them differently and a paged sweep duplicates one row while dropping
  another. Added canonical_govid as tiebreaker. Unpaginated output changes only
  in the relative order of already-tied rows.

- Basket mode returns one resolved row per requested name plus a sidecar
  covering all of them, so a page of it is not a page of anything the caller
  asked for. Refused with uscogdata_basket_pagination_conflict rather than
  silently ignoring the arguments.

Both default to NULL, so cog-api adopts them behind its existing formals()
probe with no lockstep deploy.

Suite: 1067 passed, 0 failed, 0 warnings (2 pre-existing live-corpus skips).

.Rbuildignore

@@ -3,17 +3,17 @@
 ^\.Rproj\.user$
 ^_pkgdown\.yml$
 ^docs$
+^Meta$
+^doc$
 ^pkgdown$
 ^\.github$
 ^LICENSE\.md$
 ^\.git$
 ^\.gitignore$
 \.gitkeep$
-^vignettes$
 ^specs$
 ^plans$
-^doc$
-^Meta$
 ^\.gitea$
 ^CLAUDE\.md$
 ^\.superpowers$
+^CONTRIBUTING\.md$

.gitea/workflows/mirror-github.yml

@@ -0,0 +1,43 @@
+# Mirror the canonical Gitea repo to the public GitHub mirror.
+#
+# Deliberately a plain `git push`, NOT Gitea's built-in push mirror. A push
+# mirror force-updates the refs it owns: if anyone ever clicks Merge on a
+# GitHub PR, the next sync silently overwrites main, the PR still displays
+# "Merged", the commit becomes unreachable, and nothing anywhere says so.
+# A non-force push is REJECTED as non-fast-forward the moment that happens,
+# turning a silent data-loss trap into a red CI run in a place we already look.
+#
+# Do NOT add --force here, and do NOT add GitHub branch protection to the
+# mirror: protection rules block the mirror's legitimate pushes too, breaking
+# normal syncing to catch an abnormal case.
+#
+# PAT_GH is a GitHub personal access token (repo + workflow scope; workflow is
+# required because this pushes .github/workflows/). It is stored as a Gitea
+# Actions secret. The name cannot begin with GITHUB_ or GITEA_ -- Gitea
+# reserves both prefixes for its own injected variables and rejects the secret.
+name: Mirror to GitHub
+
+on:
+  push:
+    branches: [main]
+    tags: ['v*']
+
+jobs:
+  mirror:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Push main and tags to the GitHub mirror
+        env:
+          PAT_GH: ${{ secrets.PAT_GH }}
+        run: |
+          set -eu
+          if [ -z "${PAT_GH:-}" ]; then
+            echo "PAT_GH is unset -- add it under Settings > Actions > Secrets." >&2
+            exit 1
+          fi
+          git push "https://x-access-token:${PAT_GH}@github.com/civilytics/uscogdata.git" \
+            HEAD:refs/heads/main --tags

.github/workflows/R-CMD-check.yaml

@@ -0,0 +1,61 @@
+# Multi-platform R CMD check, running on the GitHub mirror.
+#
+# This exists because the canonical Gitea runner is Linux-only, and this
+# package hard-depends on duckdb and httr2 -- both compiled, both with real
+# platform variance -- while having never been checked on Windows or macOS.
+# A large share of the audience is on Windows.
+#
+# Gitea reads .gitea/workflows and GitHub reads .github/workflows, so this
+# file is inert on the canonical repo and coexists with the Gitea CI that
+# remains authoritative for deploys.
+#
+# The suite needs NO credentials: tests/testthat/setup.R points USCOGDATA_URL
+# at the bundled fixture corpus. That is exactly why inst/extdata/fixture_corpus
+# must never be added to .Rbuildignore.
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+name: R-CMD-check
+
+permissions: read-all
+
+jobs:
+  R-CMD-check:
+    runs-on: ${{ matrix.config.os }}
+    name: ${{ matrix.config.os }} (${{ matrix.config.r }})
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - {os: macos-latest,   r: 'release'}
+          - {os: windows-latest, r: 'release'}
+          - {os: ubuntu-latest,  r: 'devel', http-user-agent: 'release'}
+          - {os: ubuntu-latest,  r: 'release'}
+
+    env:
+      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
+      R_KEEP_PKG_SOURCE: yes
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: r-lib/actions/setup-pandoc@v2
+
+      - uses: r-lib/actions/setup-r@v2
+        with:
+          r-version: ${{ matrix.config.r }}
+          http-user-agent: ${{ matrix.config.http-user-agent }}
+          use-public-rspm: true
+
+      - uses: r-lib/actions/setup-r-dependencies@v2
+        with:
+          extra-packages: any::rcmdcheck
+          needs: check
+
+      - uses: r-lib/actions/check-r-package@v2
+        with:
+          upload-snapshots: true
+          build_args: 'c("--no-manual")'

CLAUDE.md

@@ -124,3 +124,10 @@ devtools::test()
   a numbered `.sql` file; build a query in R.
 - No arrow dependency — DuckDB reads parquet natively
 - `withr` is a Suggests-only dep; only used in tests
+
+## Domain context — read this first
+
+**Before doing any work in this repo, read `~/.claude/memory/values/civilytics.md`.**
+It carries the purpose, direction, and constraints for this domain. It is not optional
+context — read it before planning or writing code, not after. (An `@` import will not
+work here; project-level imports don't preload. The read is the mechanism.)

CONTRIBUTING.md

@@ -0,0 +1,105 @@
+# Contributing to uscogdata
+
+Thanks for reading this — a package like this gets better mostly through people
+noticing that a number looks wrong.
+
+## Where the code lives
+
+Development happens on **Gitea**, at
+`gitea.civilytics.org/Civilytics/uscogdata`. The repository at
+`github.com/civilytics/uscogdata` is a **mirror** that accepts issues and pull
+requests.
+
+## What happens to a GitHub pull request
+
+Open it normally. Behind the scenes it is fetched and landed on the canonical
+Gitea repository, then syncs back:
+
+```sh
+git fetch github refs/pull/42/head:pr-42
+git switch main && git merge --no-ff pr-42
+git push origin main            # Gitea -> mirror -> GitHub
+```
+
+Because the merge preserves your commits at their original SHAs, **GitHub marks
+your PR merged on its own** as soon as the mirror syncs. So:
+
+> If your pull request closes as "Merged" without anyone visibly clicking
+> Merge, that is the normal, successful outcome — not a rejection.
+
+Substantial contributions get a `ctb` entry in `DESCRIPTION`, which surfaces in
+`citation("uscogdata")`.
+
+There is no CLA and no DCO sign-off requirement.
+
+## Running the tests
+
+```r
+devtools::test()   # bundled fixture; no network, no credentials
+```
+
+`tests/testthat/setup.R` points `USCOGDATA_URL` at
+`inst/extdata/fixture_corpus/` automatically — a four-year slice (2011, 2012,
+2019, 2020) covering all 50 states. That is the whole data setup.
+
+## Testing against the live corpus
+
+```sh
+USCOGDATA_LIVE_TEST=true Rscript -e 'devtools::test(filter = "live-corpus")'
+```
+
+This is worth understanding rather than skipping. Until 0.3.0 the package
+**could not read a remote corpus at all** — the partitioned view used a glob,
+and DuckDB cannot expand a glob over generic HTTP. It went unnoticed for months
+because every test path used a local corpus (the bundled fixture), and so did
+the production API (a host mount). Nothing exercised the package the way a new
+user does.
+
+`test-live-corpus.R` is the only test that runs with no `USCOGDATA_URL`, no
+option, and no fixture. If you change anything touching view registration,
+manifest handling, or configuration, run it.
+
+## Do not exclude the fixture from the build
+
+There is a temptation to add `^inst/extdata/fixture_corpus$` to
+`.Rbuildignore` because 15 MB feels large for a package. Don't:
+
+- `vignette("total-spending")` reads from it and would fail to build.
+- `R CMD check` on r-universe and GitHub Actions would have no corpus, so the
+  suite could not run without credentials.
+
+This package is not going to CRAN, so its 5 MB guidance does not apply. A
+package-size NOTE in `R CMD check` is expected and acceptable.
+
+## Downstream consumers
+
+`cog-api` depends on this package and its CI clones uscogdata at
+`USCOGDATA_REF`, **defaulting to `main`**. There is no pin. Anything merged
+here reaches the API's next build, so before merging a change to the reader,
+run the API suite against your branch:
+
+```sh
+Rscript -e "remotes::install_local('/path/to/uscogdata', upgrade = 'never')"
+cd /path/to/cog-api/api/tests/testthat
+Rscript -e 'testthat::test_dir(".", stop_on_failure = TRUE)'
+```
+
+The API calls only exported verbs, so internal refactors are usually safe —
+but "usually" is not a release gate.
+
+## Release checklist
+
+1. `devtools::test()` — green against the bundled fixture, offline.
+2. `USCOGDATA_LIVE_TEST=true devtools::test()` — green against the live corpus.
+3. cog-api suite green against this branch (above).
+4. `devtools::check(args = "--as-cran")` — 0 errors, 0 warnings.
+5. `pkgdown::build_site()` completes.
+6. Vignettes resolve from an installed copy:
+   `vignette("total-spending", package = "uscogdata")`.
+7. **Cold-start check**: on a machine that has never had this package,
+   install it and run the README quickstart verbatim with no environment
+   variables set. This is the only check that catches a
+   corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
+8. Bump `Version` and add a `NEWS.md` section.
+9. Tag, then update the r-universe registry pin at
+   `github.com/civilytics/civilytics.r-universe.dev`.

DESCRIPTION

@@ -1,14 +1,21 @@
 Package: uscogdata
 Type: Package
 Title: Curated Reader for the Civilytics US Census of Governments Finance Corpus
-Version: 0.2.0
+Version: 0.4.0
-Authors@R:
+Authors@R: c(
-    person("Civilytics", , , "jknowles@gmail.com", role = c("aut", "cre"))
+    person(c("Jared", "E."), "Knowles",
+           email = "jared@civilytics.com",
+           role = c("aut", "cre"),
+           comment = c(ORCID = "0000-0003-0005-9478")),
+    person("Civilytics Consulting LLC", role = c("cph", "fnd")))
 Description: Curated R verbs over the Civilytics US Census of Governments
     finance corpus. Provides unit-level financial profiles, geographic
     rollups, and peer comparisons with auditable provenance and built-in
     cross-vintage correctness.
 License: MIT + file LICENSE
+URL: https://github.com/civilytics/uscogdata,
+    https://civilytics.r-universe.dev/uscogdata
+BugReports: https://github.com/civilytics/uscogdata/issues
 Encoding: UTF-8
 LazyData: false
 Depends: R (>= 4.1)
@@ -32,4 +39,4 @@ Config/testthat/edition: 3
 VignetteBuilder: knitr
 RoxygenNote: 7.3.3
 MinCorpusSchema: 4
-MaxCorpusSchema: 5
+MaxCorpusSchema: 7

LICENSE

@@ -1,2 +1,2 @@
 YEAR: 2026
-COPYRIGHT HOLDER: Civilytics
+COPYRIGHT HOLDER: Civilytics Consulting LLC

LICENSE.md

@@ -0,0 +1,21 @@
+# MIT License
+
+Copyright (c) 2026 Civilytics Consulting LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

NEWS.md

@@ -1,3 +1,168 @@
+# uscogdata 0.4.0
+
+## DuckDB's resource budget is configurable
+
+`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
+`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
+spellings) cap the DuckDB connection the package opens. Both follow the same
+env-var > option > default precedence as `USCOGDATA_URL`.
+
+Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
+before -- every visible core. That is right for one interactive session on a
+dedicated machine and wrong for a server: where several readers share a host, each
+otherwise claims the whole machine and they contend. Capping measured ~5% on a
+single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
+cores), which is cheap enough that a server should always cap.
+
+This replaces a workaround in which a consumer reached into the package namespace
+at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
+`SET threads` -- depending both on a private name and on the session already being
+open.
+
+## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
+
+Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
+verbs were left materializing everything and slicing in R. Both now take
+`limit`/`offset` with the same semantics: `NULL` default, the page applied in
+SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
+`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
+than a second query.
+
+`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
+returns the entire 40,336-row crosswalk when called with no filter.
+
+Two refusals rather than silent surprises:
+
+* `cog_balances(recipe = , limit = )` aborts with class
+  `uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
+  separate query that pagination is not wired into.
+* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
+  `uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
+  per requested name with a sidecar covering all of them; a page of that is not
+  a page of anything the caller asked for.
+
+## Cohorts can be named by predicate, not just by id
+
+`cog_spending()`, `cog_revenue()` and `cog_balances()` gain optional `state`
+and `type` arguments. Both default to `NULL`, so every existing call behaves
+exactly as before.
+
+Passing them expresses the cohort as a subquery against `canonical_fips_xwalk`
+inside each statement, instead of round-tripping the ids through R and
+rendering them back into a literal `IN` list:
+
+```r
+# before: resolve 20,106 ids in R, then embed them in every statement
+ids <- cog_gov_search(NULL, state = "CA", type = "city")$canonical_govid
+cog_spending(ids, years = 2022)
+
+# now: the cohort never leaves the database
+cog_spending(years = 2022, state = "CA", type = "city")
+```
+
+Measured against the production corpus, same FY2022 aggregate over the
+20,106-government `type = "city"` cohort:
+
+| cohort expressed as | time |
+|---|---:|
+| `IN (20,106 literals)` | 449 ms |
+| join against a temp cohort table | 99 ms |
+| predicate on `canonical_fips_xwalk` | **94 ms** |
+| no cohort filter at all (the floor) | 88 ms |
+
+**4.8x, within 7% of the floor.** The rendered `IN` list was 301,591
+characters and was re-parsed in 5-8 separate statements per call, so the cost
+was paid repeatedly; the predicate's size is constant in the cohort.
+
+`state` and `type` use the same vocabulary and the same internal coercion as
+`cog_gov_search()` -- `state` is a postal abbreviation (`"WI"`) even though the
+crosswalk column holds a FIPS code (`"55"`).
+
+Supplying `govid` **and** `state`/`type` intersects them: the governments in
+`govid` that also match the predicate. Naming no cohort at all now aborts with
+class `uscogdata_no_cohort` rather than R's "argument is missing" error.
+
+When the cohort is named by predicate there is no id list to report, so
+`provenance$scope$govids_found`/`govids_missing` are empty and
+`provenance$scope$cohort` carries `state`, `type` and `n_governments` instead.
+A `govid`-named cohort's provenance is unchanged.
+
+## Fixes
+
+* `cog_gov_search()` now orders by `population_acs DESC NULLS LAST,
+  canonical_govid`. **`population_acs` alone is not a total order** -- ties, and
+  the entire `NULLS LAST` block, came back in whatever order the scan produced.
+  That was invisible while every call returned the full result set, but it makes
+  a paged sweep unsound: two requests can order tied rows differently, so a row
+  is duplicated on one page and missing from the next. Unpaginated results are
+  unchanged except for the relative order of rows that were already tied.
+
+* An unknown `state` abbreviation now aborts with "Unknown state abbreviation"
+  (class `uscogdata_unknown_state`) instead of base R's "subscript out of
+  bounds". `.state_abbrev_to_fips` is a named character vector, so `[[` on an
+  absent name threw before the curated message could be reached -- making that
+  message unreachable dead code in every verb that takes a `state`.
+
+# uscogdata 0.3.0
+
+First public release.
+
+`uscogdata` provides curated R verbs over the Civilytics US Census of
+Governments finance corpus: unit-level financial profiles, geographic rollups
+and peer comparisons, with auditable provenance on every result.
+
+## What it covers
+
+Government types 0-3 (state, county, municipality, township), FY1967-FY2024 --
+56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
+or FY1969. Special districts (type 4) and school districts (type 5) are out of
+scope pending validation.
+
+## The verbs
+
+`cog_spending()`, `cog_revenue()` and `cog_balances()` for flows and holdings;
+`cog_gov_search()` to resolve place names (including basket mode for many at
+once); `cog_find_peers()` and `cog_peer_compare()` for cohorts;
+`cog_geographic_rollup()` for aggregates; `cog_categories()`, `cog_recipes()`,
+`cog_manifest()` and `cog_explain()` for metadata and provenance; and
+`cog_mirror()` for a local copy of the corpus.
+
+## Reading the corpus now works out of the box
+
+* The package reads the published corpus over HTTPS **with no configuration**.
+  Previously the default was a placeholder sentinel and no document in the
+  package supplied a working URL, so a new user had no path to a session.
+* Remote reads work at all. The partitioned view used a glob, and DuckDB
+  cannot expand a glob over generic HTTP -- there is no directory listing to
+  expand against. Partition paths are now enumerated from the corpus manifest,
+  which is host-agnostic: an HTTPS mirror, a Nextcloud share and a local
+  `cog_mirror()` copy all take the same path.
+* Nothing is written to disk in remote mode; DuckDB fetches only the row
+  groups a query needs.
+
+## Four things to know before your first query
+
+* **Amounts are in full US dollars.** The raw Census files report thousands;
+  the verbs multiply by 1000 on the way out. Do not multiply again.
+* **Multi-government aggregates disclose their coverage.** The Census is a
+  complete enumeration only in years ending in 2 and 7; every other year is a
+  sample. Every such result carries `provenance$coverage` with per-year
+  `n_units_reporting`.
+* **Absence means two different things.** Before FY2012 an absent cell means
+  Census published $0; from FY2012 it means not reported. `complete = TRUE`
+  labels which.
+* **Series breaks reach you unasked.** Catalogued breaks intersecting your
+  query appear in provenance and in `cog_explain()`.
+
+## Known limits
+
+* Special districts (type 4) and school districts (type 5) are out of scope.
+* Per-capita rollups exclude governments with no F-33 population, which is by
+  design but does silently narrow a rollup.
+* `n_units_reporting` is category-conditional and is not a response rate.
+* Employee-retirement (`X`) codes stop at FY2016, when those systems moved to
+  the Annual Survey of Public Pensions.
+
 # uscogdata 0.2.0
 
 ## New features
@@ -39,262 +204,3 @@
   is not a response rate: a government that was surveyed and genuinely spends
   nothing in the requested category is indistinguishable from one never
   surveyed (uscogdata#36).
-
-# uscogdata 0.1.0 (development)
-
-## Signposting now catches partially-suppressed categories
-
-* A coverage suggestion used to fire only when a category returned **no rows
-  at all** in a requested year. That missed the more dangerous case: a
-  category that still returns rows while silently dropping component codes
-  the wide era publishes only as aggregates (#9). `cog_spending(category =
-  "Public Welfare")` for FY2011 returned a plausible figure that omitted
-  `E67`/`E68` entirely -- for Los Angeles County, $2,075,461,000 of a true
-  $5,261,404,000, a 39% understatement, with `provenance$suggestions` empty.
-* Suggestions now also fire on **partial** coverage, and every suggestion
-  carries `trigger` (`"empty_year"` or `"suppressed_component"`),
-  `suppressed_amount`, `suppressed_years` and `suppressed_codes`, so a caller
-  can see how much is missing and decide whether to re-run with the recipe.
-* `cog_revenue()` gets the same fix through the shared verb path. Alaska's
-  FY2011 `Miscellaneous Revenue` reported $943,842,000 while dropping
-  $1,899,995,000 of aggregate-published `U4-` rents and royalties.
-* The trigger stays recipe-driven, so it only fires where a harmonization
-  recipe actually exists to name the fix. `higher_ed_e18_wide` and
-  `general_gov_e89_wide` stay silent in every year measured on the bundled
-  fixture, because their components are ordinary classified leaves even
-  pre-2012.
-* The `suppressed_component` trigger (and any `suppressed_amount`/
-  `suppressed_codes` an `empty_year` fire also carries) is scoped to the
-  calling verb's own flow family: `cog_spending()` only ever measures E/F/G
-  component dollars, `cog_revenue()` only T/A/U/B/C/D. A component from the
-  OTHER flow family reports `suppressed_amount = 0` rather than a fabricated
-  claim. The `empty_year` trigger itself is not flow-scoped -- a category
-  belonging to the other flow (e.g. `cog_spending(category = "IG Local")`)
-  still returns zero rows and can still fire, in any year including modern
-  ones, naming the recipe whose own generic join finds real data for this
-  government. That is a mis-scoped query, not a corpus-format gap, so its
-  `suppressed_amount` is correctly 0.
-
-## New: `cog_balances()` for cash-and-security holdings
-
-* New `cog_balances()` exposes the 14 cash-and-security holding codes
-  (`category_type = "balance"`): fund balances, retirement system holdings and
-  insurance trust balances (#25). Holdings are a stock, not a flow, so the verb
-  has no `expenditure_concept` / `revenue_concept` / `complete` arguments, and
-  no `subtype` argument either -- for holdings, `category` is a strict
-  coarsening of `balance_subtype`, so `category = "Fund Balances"` is exactly
-  the `general` family (`W01`/`W31`/`W61`).
-* `cog_balances()` results carry `provenance$balance_caveats`, recording that
-  Census holdings are gross rather than GAAP fund balance, and the measured
-  coverage window of each subtype family.
-
-## Multi-government aggregates now disclose their reporting coverage
-
-* The Census of Governments is a **complete census only in years ending in 2
-  and 7**; every other year is a sample, and the sample varies enormously. On
-  the bundled fixture, Wisconsin's 608-city universe rolls up **597**
-  governments in FY2012 and **112** in FY2019 — an 18%-to-98% swing the
-  return value said nothing about, so a statewide total resting on a fifth of
-  the universe looked exactly like one resting on all of it.
-* `cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` gain
-  `coverage`:
-
-  | value | effect |
-  |---|---|
-  | `"all"` (default) | every unit that reported that year — unchanged behaviour |
-  | `"census"` | census years only; aborts if the range holds none rather than returning nothing |
-  | `"consistent"` | only units reporting in *every* requested year — a balanced panel |
-
-* **Regardless of mode**, every result now carries `provenance$coverage` with
-  per-year `n_units_reporting`, `n_units_expected` and `is_census_year`, plus
-  `provenance$coverage_mode`. `cog_explain()` prints a "Reporting coverage"
-  section. So the default mode can no longer mislead silently.
-* `is_census_year` is a statement about the **survey calendar**, never a claim
-  of completeness: FY1967 is a census year in which only 97 of Wisconsin's 608
-  cities report. `n_units_reporting` is the number that tells the truth.
-* On `cog_peer_compare()` the target is exempt from `"consistent"` balancing —
-  it is the subject of the comparison, not a member of the cohort — and the
-  `summary_*` quantiles are computed after the filter, so they describe the
-  cohort actually returned. `n_units_reporting` counts peers only, against the
-  cohort size.
-* On `cog_find_peers()`, `coverage` governs the cohort **vintage** when `year`
-  is `NULL`: `"census"` snaps to the most recent census year with an observed
-  population, so a cohort is not built from a sample year in which most of the
-  candidate universe is absent.
-
-## `complete = TRUE`: absent cells, labelled with why they are absent
-
-* `cog_spending()` and `cog_revenue()` gain `complete`, defaulting to `FALSE`
-  (today's behaviour). With `complete = TRUE` the requested grid is filled
-  from the corpus's `code_set` table and every row carries a new
-  `value_source` column:
-
-  | `value_source` | meaning | `amt_nominal` |
-  |---|---|---|
-  | `reported` | the corpus carries this cell | as published |
-  | `census_zero` | dense-source year (≤ FY2011), cell absent — Census published `$0` | `0` |
-  | `not_reported` | sparse-source year (≥ FY2012), cell absent — unknown | `NA` |
-
-  The `NA` is deliberate and is the whole point: filling a modern absence
-  with `0` would invent data, which is precisely the error the corpus's
-  representation contract exists to prevent.
-* This restores information the reader lost when the corpus was sparsified
-  (`SB194`, cog_pipeline#64) — a wide-era query whose cells were all `$0`
-  had begun returning nothing at all — and improves on what came before it,
-  since the pre-sparsification corpus could not distinguish a published zero
-  from an unreported cell either.
-* The grid is scoped to each government's **own type**, so a county is never
-  filled with cells only a state can report.
-* Needs a corpus published from 2026-07-29 onward (when `representation` and
-  `code_set` began shipping); aborts with class
-  `uscogdata_representation_unavailable` otherwise. Gated on the manifest
-  listing those tables rather than on `schema_version`, which was never
-  bumped for the change. Not available with `recipe` or
-  `expenditure_concept = "total"` — neither draws its cells from `code_set`.
-* `provenance$completion` reports `applied`, `rows_filled`, and the per-year
-  `absence_means` rule; `cog_explain()` prints a "Completion" section.
-
-## Corpus-wide series breaks now reach users (`corpus_break_refs`)
-
-* Four catalogued series breaks carry `fin_code = "ALL"` — caveats about the
-  corpus as a whole rather than about one item code. `series_break_refs` is
-  built by matching `fin_code` against the item codes in the result, and no
-  row's `item_code` is ever the literal `"ALL"`, so **none of them could ever
-  be surfaced**: `SB085` (dollar precision across the 1976/1977 boundary),
-  `SB087` (imputation exclusion from FY2002), `SB194` (the dense → sparse
-  representation change at FY2012) and `SB086` (the government id scheme
-  change at FY2017).
-* Provenance gains `corpus_break_refs`, selected on the break-year window
-  alone and disjoint from `series_break_refs` by construction, so a consumer
-  can tell a whole-result caveat from a break in one series. `cog_explain()`
-  prints them under their own "Corpus-wide caveats" heading. cog-api passes
-  provenance through verbatim, so the field appears there without an API
-  change.
-* `SB194` is the one that made this urgent: a query spanning FY2011 → FY2012
-  crosses the boundary where an absent cell stops meaning "Census published
-  `$0`" and starts meaning "not reported", and until now nothing said so.
-
-## Bundled fixture regenerated against the sparsified corpus
-
-* `inst/extdata/fixture_corpus/` now tracks the corpus published on
-  2026-07-29 (`pipeline_commit 83f9715`, schema v6). The wide era no longer
-  stores explicit zeros: FY2011 fell from 2,864,212 rows to 496,004, of
-  which none are `$0`. **Absence now means two different things** — in a
-  `dense_source` year (≤ FY2011) an absent cell means Census published `$0`;
-  in a `sparse_source` year (≥ FY2012) it means not reported. The corpus
-  carries that rule in two new tables the fixture now ships,
-  `representation.parquet` and `code_set.parquet`, alongside
-  `census_collection_coverage.parquet` and `lineage_events.parquet`
-  (all ten publish-tree metadata tables, up from six). Catalogued upstream
-  as series break `SB194`.
-* `cog_categories()` gains an `assistance` spending subtype: the J-prefix
-  aid/benefit codes (`J19`, `J67`, `J68`, `J85`) are categorised now that
-  the upstream crosswalk covers every flow code carrying dollars.
-* Two consequences worth knowing about, both visible in provenance rather
-  than in returned dollars. The harmonization block's `na_rows_excluded`
-  counts only rows that exist, so wide-era codes that were zero-padded no
-  longer appear there. Coverage-gap `suggestions` are presence-based for the
-  same reason, so a recipe whose component codes were all `$0` for a given
-  government-year is no longer suggested for it.
-* `tests/testthat/test-fixture-vintage.R` pins these structural facts, so a
-  fixture left behind by a future publish fails loudly instead of letting the
-  suite pass against a corpus that no longer exists.
-
-## Breaking: corpus schema_version 4 (Phase P canonical ids)
-
-* The package now requires corpus `schema_version = 4` (`MinCorpusSchema` /
-  `MaxCorpusSchema` in `DESCRIPTION` are both `4`); older corpora built
-  against schema 3 are rejected by `cog_open()` with a clear version-mismatch
-  error. `canonical_govid` is now uniformly 12 characters across every
-  vintage the corpus covers (previously a mix of 9-char legacy ids and
-  12-char FIPS ids depending on source year) — **every hardcoded
-  `canonical_govid` literal from a pre-Phase-P corpus is now invalid** and
-  must be re-resolved via `cog_gov_search()` or the new `canonical_alias`
-  lookup table. `canonical_fips_xwalk` gains four columns
-  (`legacy_govs_id`, `census_geoid`, `id_source`; `confidence` is renamed to
-  `pop_confidence`) and a companion `canonical_alias` table ships in the
-  corpus for mapping legacy/alternate ids onto the current canonical
-  namespace. The bundled fixture corpus (`inst/extdata/fixture_corpus/`) has
-  been regenerated against the Phase P publish tree, now ships the full
-  `canonical_fips_xwalk` and `canonical_alias` master tables alongside the
-  2019-2020 long partitions, and is reproducible via
-  `data-raw/regenerate_fixture_corpus.R`.
-
-## Clearer errors when `USCOGDATA_URL` is unconfigured or returns non-JSON
-
-* `cog_open()` now aborts with the `uscogdata_url_not_configured` error
-  class when the resolved corpus URL still contains the placeholder
-  `REPLACE_WITH_SHARE_TOKEN` sentinel (or is empty). The message lists both
-  remediation paths (`Sys.setenv(USCOGDATA_URL = ...)` and
-  `options(uscogdata.url = ...)`) and points at the bundled fixture for
-  offline testing. Previously the package proceeded to fetch the placeholder
-  URL, cached the resulting HTML welcome page, and failed downstream with a
-  cryptic `jsonlite` lexical-error.
-* `.fetch_or_cache_manifest()` now parses the HTTP response body before
-  persisting it. Non-JSON responses (login pages, 404 HTML) raise
-  `uscogdata_invalid_manifest` with the URL, Content-Type, and underlying
-  parse error — and never write to the on-disk cache.
-* Manifest cache writes are now atomic (write to `manifest.json.tmp.<pid>`
-  in `cache_dir`, then `file.rename` over the target), so an interrupted
-  fetch cannot replace a previously-good cache.
-* Existing caches with non-JSON content (poisoned by the prior code path)
-  are silently refetched instead of returning a parse error to the caller.
-* Local `USCOGDATA_URL` paths whose `manifest.json` is not valid JSON now
-  surface the same `uscogdata_invalid_manifest` class with file context.
-
-## Per-capita denominators now use per-year Census F-33 population
-
-* `cog_spending()` and `cog_revenue()` previously divided all years' amounts
-  by a single ACS 2018-2022 estimate (`canonical_fips_xwalk.population_acs`),
-  producing biased per-capita values for time-series analysis. They now
-  divide by the F-33 `population` recorded on each gov-year via the new
-  `gov_population_yearly` view. Result tibbles gain a `pop_source` column
-  with values `"census_f33"` or `"unavailable"`. `notes` is updated to
-  concatenate multiple notes with `"; "`.
-
-## Peer cohorts can be set to a chosen year
-
-* `cog_find_peers()` adds a `year` argument (default: most recent year for
-  which the target has an observed population in `gov_population_yearly`).
-  The returned column previously named `population_acs` is now `population`
-  and reflects the cohort year's vintage. The cohort year is attached to the
-  returned tibble as `attr(x, "cohort_year")`.
-* `cog_peer_compare()` now stamps a `cohort_year` column on its result (read
-  from the peers tibble's attribute) and records `cohort_year` plus
-  `cohort_govids` in provenance. When the caller supplies a bare character
-  vector instead of a `cog_find_peers()` result, `cohort_year` is `NA`.
-
-## Rollups exclude govs missing population
-
-* `cog_geographic_rollup(per_capita = TRUE)` drops rows whose government has
-  `pop_source == "unavailable"` and records the dropped govids in
-  `provenance$rollup$excluded_govids`. This excludes special districts
-  (type 4) and school districts (type 5) from per-capita rollups by design.
-
-## New: vignette and provenance metadata
-
-* New vignette `population-denominators` covers the four population sources,
-  the type-4/5 coverage gap, the popyear quirk, and how to build moving-window
-  peer cohorts manually.
-* Provenance gains `transformations$per_capita$popyear_range` and
-  `pop_source_counts`. `cog_explain()` renders both.
-
-## New features
-
-* `cog_gov_search()` gains a **basket mode**: passing vector `name`
-  / `state` / `type` arguments resolves multiple place names in one
-  call and returns a tibble of canonical rows in input order, ready
-  to pipe into `cog_spending()` / `cog_revenue()`. Per-row resolution
-  follows an exact-then-substring matching algorithm with deterministic
-  disambiguation; ambiguous and missing entries are surfaced via a
-  sidecar audit tibble plus a single console summary message.
-* New exports `cog_basket_resolution()` and `cog_basket_unresolved()`
-  expose the basket sidecar for iterative query refinement.
-
-## Breaking changes
-
-* The first formal of `cog_gov_search()` was renamed from `pattern`
-  to `name`. All existing call sites in `cog_explorer/` and the
-  package itself use positional first-arg, so this rename is
-  non-breaking in practice. Callers that pass `pattern = ...` by name
-  must update to `name = ...`.

R/balances.R

@@ -18,7 +18,9 @@
 #' comparable to a GAAP fund balance from an ACFR.
 #'
 #' @param govid Canonical govid(s): a character vector, or a data frame with a
-#'   `canonical_govid` column (e.g. from [cog_gov_search()]).
+#'   `canonical_govid` column (e.g. from [cog_gov_search()]). `NULL` to name
+#'   the cohort by `state`/`type` instead.
+#' @inheritParams cog_spending
 #' @param years Integer vector of fiscal years.
 #' @param category Optional character vector of categories to keep. One of
 #'   `"Fund Balances"`, `"Insurance Trust Balances"`,
@@ -45,6 +47,12 @@
 #' @param recipe Optional harmonization recipe id (see [cog_recipes()]).
 #'   `"cash_securities_z77_wide"` and `"cash_securities_z78_wide"` bridge the
 #'   wide era to the modern one.
+#' @param limit Maximum number of result rows to return, pushed into the SQL
+#'   rather than applied after materializing every row. `NULL` (default)
+#'   returns everything. Cannot be combined with `recipe` -- see `offset` and
+#'   `total_rows`.
+#' @param offset Rows to skip before `limit` starts counting (0-based).
+#'   Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.
 #'
 #' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
 #'   `balance_subtype`, `category`, `amt_nominal`, `codes_included`,
@@ -62,16 +70,22 @@
 #'   and `truncated` (the observed subtypes whose coverage falls short of the
 #'   requested years). `expenditure_concept`/`revenue_concept` are `NA` --
 #'   holdings are a stock, not a flow, so neither concept vocabulary applies.
+#'
+#'   When `limit` is set, also carries a `total_rows` attribute: the full
+#'   unpaginated row count, computed by the same query (`COUNT(*) OVER()`)
+#'   rather than a second scan.
 #' @export
-cog_balances <- function(govid, years, category = NULL,
+cog_balances <- function(govid = NULL, years, category = NULL,
                          per_capita = FALSE, adjust_to_year = NULL,
-                         basis = c("harmonized", "raw"), recipe = NULL) {
+                         basis = c("harmonized", "raw"), recipe = NULL,
+                         state = NULL, type = NULL,
+                         limit = NULL, offset = NULL) {
   call <- match.call()
   basis <- match.arg(basis, c("harmonized", "raw"))
   # Coerce FIRST, validate second: .validate_verb_inputs() asserts
   # is.character(govid), and a data-frame govid (cog_gov_search() output) has
   # not been unwrapped yet at this point.
-  govid <- .coerce_govid_input(govid)
+  govid <- if (is.null(govid)) NULL else .coerce_govid_input(govid)
   # The money verbs' validator, reused rather than re-implemented (R/spending.R).
   # It covers the exact superset cog_balances() needs -- including the
   # recipe/category mutual-exclusivity guard -- so a second local copy would
@@ -88,9 +102,27 @@ cog_balances <- function(govid, years, category = NULL,
   # validator's own doc comment for the incident that made that matter.
   .validate_verb_inputs(govid, years, category, per_capita, adjust_to_year,
                         recipe)
+
+  # Same semantics as the money verbs (R/pagination.R). Only the `recipe`
+  # conflict applies here: cog_balances() has no `complete` argument, and a
+  # recipe's result comes from .run_recipe()'s own query, which pagination is
+  # not wired into.
+  paging <- .validate_pagination(limit, offset)
+  limit  <- paging$limit
+  offset <- paging$offset
+  if (!is.null(limit) && !is.null(recipe)) {
+    cli::cli_abort(c(
+      "`limit`/`offset` cannot be combined with `recipe`.",
+      "i" = "A recipe's result comes from a separate query (`.run_recipe()`) that pagination is not wired into yet.",
+      "*" = "Drop `limit`/`offset`, or drop `recipe`."
+    ), class = "uscogdata_recipe_pagination_conflict")
+  }
+
   years <- as.integer(years)
   if (!is.null(adjust_to_year)) adjust_to_year <- as.integer(adjust_to_year)
 
+  cohort <- .make_cohort(govid, state, type)
+
   con <- .ensure_session()
   .require_balance_support(con)
   scope <- .check_govids_in_scope(govid)
@@ -103,13 +135,14 @@ cog_balances <- function(govid, years, category = NULL,
   manifest <- .uscogdata_env$manifest
   recipe_block <- NULL
   category_for_prov <- category
+  total_rows <- NULL  # set below only when limit is non-NULL (non-recipe path)
 
   if (!is.null(recipe)) {
     .require_schema_v5(con, manifest, "recipe =")
     .validate_recipe_id(con, recipe)
     comps <- .recipe_components(con, recipe)
     recipe_label <- comps$label[[1]]
-    result <- .run_recipe(con, recipe, govid, years)
+    result <- .run_recipe(con, recipe, cohort, years)
     sql <- attr(result, "sql_query")
     result <- .shape_recipe_result(result, "balance_subtype", recipe_label)
     recipe_block <- list(
@@ -119,15 +152,25 @@ cog_balances <- function(govid, years, category = NULL,
     category_for_prov <- recipe_label
   } else {
     sql <- .build_verb_sql("balance_annotated", "balance_subtype",
-                           govid, years, category,
+                           cohort, years, category,
-                           ig_view = NULL, subtype_scope = NULL)
+                           ig_view = NULL, subtype_scope = NULL,
+                           limit = limit, offset = offset)
     result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
+    if (!is.null(limit)) {
+      paged <- .take_pagination_total(result, con, function() {
+        .build_verb_sql("balance_annotated", "balance_subtype",
+                        cohort, years, category,
+                        ig_view = NULL, subtype_scope = NULL)
+      })
+      result     <- paged$result
+      total_rows <- paged$total_rows
+    }
   }
 
   # Order matters (matches .verb_spendrev()): per-capita first, so
   # .attach_real_dollars() deflates the nominal per-capita column into
   # amt_per_capita_real rather than needing amt_per_capita_nominal recomputed.
-  if (isTRUE(per_capita)) result <- .attach_per_capita(result, con, govid)
+  if (isTRUE(per_capita)) result <- .attach_per_capita(result, con)
   if (!is.null(adjust_to_year)) {
     result <- .attach_real_dollars(result, adjust_to_year, per_capita)
   }
@@ -145,6 +188,7 @@ cog_balances <- function(govid, years, category = NULL,
   )
   prov$scope$govids_found   <- scope$found
   prov$scope$govids_missing <- scope$missing
+  prov$scope$cohort <- .cohort_provenance(con, cohort)
 
   prov$balance_caveats <- .balance_caveats(
     con, prov$codes_summed$observed, years
@@ -152,6 +196,7 @@ cog_balances <- function(govid, years, category = NULL,
   .emit_balance_caveats(prov$balance_caveats)
 
   attr(result, "provenance") <- prov
+  if (!is.null(limit)) attr(result, "total_rows") <- total_rows
   result
 }
 

R/basis.R

@@ -54,7 +54,7 @@
 #' expenditure_concept = "total": ig_long_harmonized COALESCEs rather than
 #' drops NULL-harmonized rows, so harmonization never excludes an IG row.
 #' @noRd
-.build_harmonization_block <- function(con, govid, years, resolved,
+.build_harmonization_block <- function(con, cohort, years, resolved,
                                        subtype_col, subtype_scope) {
   if (!identical(resolved$basis, "harmonized")) {
     return(list(
@@ -68,12 +68,12 @@
   sql <- sprintf(
     "SELECT COUNT(*) AS n, COALESCE(SUM(amt), 0) * 1000.0 AS amt
      FROM long
-     WHERE canonical_govid IN (%s) AND year IN (%s)
+     WHERE %s AND year IN (%s)
        AND NOT is_aggregate AND harmonized_code IS NULL
        AND item_code IN (
          SELECT item_code FROM summary_categories WHERE %s IN (%s)
        )",
-    .sql_lit_chr(govid), paste(as.integer(years), collapse = ","),
+    .cohort_sql(cohort), paste(as.integer(years), collapse = ","),
     subtype_col, .sql_lit_chr(subtype_scope)
   )
   na <- DBI::dbGetQuery(con, sql)

R/cohort.R

@@ -0,0 +1,127 @@
+# How a verb names the set of governments it queries.
+#
+# Historically there was one way: a `govid` character vector, rendered by
+# .sql_lit_chr() into a quoted IN list. That is fine for a handful of
+# governments and pathological for a fleet. Measured against the production
+# corpus, the same FY2022 aggregate over the 20,106-government `type = "city"`
+# cohort:
+#
+#   cohort expressed as                            time
+#   IN (20,106 literals)                           449 ms
+#   join against a temp cohort table                99 ms
+#   predicate on canonical_fips_xwalk               94 ms
+#   no cohort filter at all (the floor)             88 ms
+#
+# 4.8x, and within 7% of the no-filter floor. The rendered IN list is 301,591
+# characters and .verb_spendrev() embeds it in 5-8 separate statements per
+# call, so the parse-and-plan cost is paid over and over (uscogdata#58).
+#
+# A cohort therefore has two independent halves, and a query can carry either
+# or both:
+#
+#   ids        an explicit canonical_govid vector      -> literal IN list
+#   predicate  state/type over canonical_fips_xwalk    -> IN (SELECT ...)
+#
+# Both together is an INTERSECTION -- "these ids, narrowed to that state/type"
+# -- never a precedence rule where one silently wins.
+
+#' Build the internal cohort object shared by every query verb.
+#'
+#' `state` and `type` are coerced with the SAME helpers `cog_gov_search()`
+#' uses. That is load-bearing, not tidiness: the public argument is a postal
+#' abbreviation (`"WI"`) while `canonical_fips_xwalk.fips_state` holds a FIPS
+#' code (`"55"`), and `type` is a label (`"city"`) against an integer
+#' `govs_type`. A predicate written against the raw parameter matches nothing
+#' and returns an empty result indistinguishable from "this government
+#' reported nothing" -- cog-api hit exactly that trap optimizing this path.
+#' One definition of the translation, not two.
+#'
+#' @param govid Already-coerced character vector of canonical_govids, or NULL.
+#' @param state Postal abbreviation or FIPS code, or NULL.
+#' @param type Type label or integer code, or NULL.
+#' @noRd
+.make_cohort <- function(govid = NULL, state = NULL, type = NULL) {
+  if (is.null(govid) && is.null(state) && is.null(type)) {
+    cli::cli_abort(c(
+      "A cohort must be named.",
+      "*" = "Pass {.arg govid} for specific governments, or {.arg state}/{.arg type} for every government matching a predicate.",
+      "i" = "Passing both intersects them: the governments in {.arg govid} that also match {.arg state}/{.arg type}."
+    ), class = "uscogdata_no_cohort")
+  }
+  structure(
+    list(
+      ids        = govid,
+      state      = state,
+      type       = type,
+      state_fips = if (is.null(state)) NULL else .coerce_state_to_fips(state),
+      type_int   = if (is.null(type)) NULL else .coerce_type(type)
+    ),
+    class = "uscogdata_cohort"
+  )
+}
+
+#' Is any part of this cohort expressed as an xwalk predicate?
+#' @noRd
+.cohort_by_predicate <- function(cohort) {
+  !is.null(cohort$state_fips) || !is.null(cohort$type_int)
+}
+
+#' Render the cohort as a SQL boolean expression over `col`.
+#'
+#' `col` may be qualified (`"l.canonical_govid"`, `"x.canonical_govid"`) --
+#' several call sites join the xwalk under an alias. The subquery's own
+#' projected column stays unqualified: it selects from canonical_fips_xwalk,
+#' not from the outer relation.
+#' @noRd
+.cohort_sql <- function(cohort, col = "canonical_govid") {
+  preds <- character(0)
+
+  if (!is.null(cohort$ids)) {
+    preds <- c(preds, sprintf("%s IN (%s)", col, .sql_lit_chr(cohort$ids)))
+  }
+
+  if (.cohort_by_predicate(cohort)) {
+    xwalk_preds <- character(0)
+    if (!is.null(cohort$state_fips)) {
+      xwalk_preds <- c(xwalk_preds,
+                       sprintf("fips_state = %s", .sql_lit_chr(cohort$state_fips)))
+    }
+    if (!is.null(cohort$type_int)) {
+      xwalk_preds <- c(xwalk_preds, sprintf("govs_type = %d", cohort$type_int))
+    }
+    preds <- c(preds, sprintf(
+      "%s IN (SELECT canonical_govid FROM canonical_fips_xwalk WHERE %s)",
+      col, paste(xwalk_preds, collapse = " AND ")
+    ))
+  }
+
+  paste(preds, collapse = " AND ")
+}
+
+#' How many governments the cohort covers.
+#'
+#' One COUNT against the crosswalk, used only to populate the provenance
+#' `scope$cohort` block. Deliberately a count rather than the id list: a
+#' fleet-scale cohort would otherwise put 20,000 ids into every response body,
+#' which is the cost this issue exists to remove.
+#' @noRd
+.cohort_count <- function(con, cohort) {
+  sql <- sprintf(
+    "SELECT COUNT(*) AS n FROM canonical_fips_xwalk WHERE %s",
+    .cohort_sql(cohort)
+  )
+  as.integer(DBI::dbGetQuery(con, sql)$n[[1]])
+}
+
+#' The provenance `scope$cohort` block for a predicate cohort, or NULL when
+#' the cohort was named by id alone (in which case `govids_found`/
+#' `govids_missing` already describe it exactly).
+#' @noRd
+.cohort_provenance <- function(con, cohort) {
+  if (!.cohort_by_predicate(cohort)) return(NULL)
+  list(
+    state         = if (is.null(cohort$state)) NA_character_ else as.character(cohort$state),
+    type          = if (is.null(cohort$type)) NA_character_ else as.character(cohort$type),
+    n_governments = .cohort_count(con, cohort)
+  )
+}

R/complete.R

@@ -57,7 +57,7 @@
 #' aggregate rows. Without it the grid would offer cells the verb structurally
 #' never returns, so every one of them would fill as a phantom $0.
 #' @noRd
-.completion_grid_sql <- function(subtype_col, govid, years, category,
+.completion_grid_sql <- function(subtype_col, cohort, years, category,
                                  subtype_scope) {
   category_pred <- if (is.null(category)) {
     ""
@@ -76,13 +76,13 @@
      JOIN canonical_fips_xwalk x ON x.govs_type = cs.type
      JOIN summary_categories c   ON c.item_code = cs.item_code
      JOIN representation r       ON r.year = cs.year
-     WHERE x.canonical_govid IN (%2$s)
+     WHERE %2$s
        AND cs.year IN (%3$s)
        AND NOT cs.is_aggregate
        AND c.category IS NOT NULL
        AND c.%1$s IN (%4$s)
        %5$s",
-    subtype_col, .sql_lit_chr(govid),
+    subtype_col, .cohort_sql(cohort, "x.canonical_govid"),
     paste(as.integer(years), collapse = ","),
     .sql_lit_chr(subtype_scope), category_pred
   )
@@ -94,10 +94,10 @@
 #' provenance block. Reported rows are passed through untouched -- filling
 #' must never alter or drop what the corpus actually published.
 #' @noRd
-.complete_result <- function(result, con, subtype_col, govid, years, category,
+.complete_result <- function(result, con, subtype_col, cohort, years, category,
                              subtype_scope) {
   grid <- tibble::as_tibble(DBI::dbGetQuery(
-    con, .completion_grid_sql(subtype_col, govid, years, category, subtype_scope)
+    con, .completion_grid_sql(subtype_col, cohort, years, category, subtype_scope)
   ))
 
   result$value_source <- rep("reported", nrow(result))

R/config.R

@@ -5,9 +5,28 @@
 .uscogdata_env <- new.env(parent = emptyenv())
 
 .uscogdata_defaults <- list(
-  url               = "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/",
+  # Public HuggingFace mirror of the published corpus: CC-BY-4.0, no
+  # credential, CDN-backed. This is the default so `library(uscogdata)`
+  # followed by a verb works with zero configuration -- previously the
+  # default was a REPLACE_WITH_SHARE_TOKEN sentinel and no document in the
+  # package supplied a working URL, so a new user had no path to a session.
+  #
+  # The trailing slash is required: every consumer concatenates onto this
+  # (see .resolve_url(), which enforces it anyway).
+  #
+  # Override with USCOGDATA_URL or options(uscogdata.url=) to read a
+  # Nextcloud share or a local copy made by cog_mirror().
+  url               = "https://huggingface.co/datasets/civilytics/us-cog-finance/resolve/main/",
   cache_dir         = NULL,
-  manifest_ttl_secs = 3600L
+  manifest_ttl_secs = 3600L,
+  # NULL means "emit no pragma", which leaves DuckDB's own defaults intact:
+  # every visible core, and 80% of RAM. That is right for one interactive
+  # session on a dedicated machine and wrong for a server, where several
+  # readers share a box and each would otherwise claim all of it. See
+  # .resolve_duckdb_threads() for why this is a supported option rather than
+  # something a consumer reaches into the namespace to set.
+  duckdb_threads      = NULL,
+  duckdb_memory_limit = NULL
 )
 
 #' Resolve a config value: env var > option > default
@@ -50,3 +69,54 @@
   v <- .cfg("cache_dir")
   if (is.null(v)) tools::R_user_dir("uscogdata", "cache") else v
 }
+
+#' Resolve the DuckDB thread cap, or NULL to leave DuckDB's default alone.
+#'
+#' `cog_open()` used to connect with a bare `dbConnect()` and set no `threads`
+#' pragma, so DuckDB claimed every core it could see. cog-api works around that
+#' by reaching into this namespace at boot --
+#' `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
+#' `SET threads` -- which depends on a private name and on the session already
+#' being open. Making it a resolved option removes the reason to do that.
+#'
+#' `.cfg()` returns an environment variable as CHARACTER, so this coerces
+#' rather than trusting the type: `USCOGDATA_DUCKDB_THREADS=4` arrives as "4",
+#' and `sprintf("SET threads TO %d", "4")` would abort inside the connection
+#' path with an error about the pragma rather than about the setting.
+#' @noRd
+.resolve_duckdb_threads <- function() {
+  v <- .cfg("duckdb_threads")
+  if (is.null(v) || (is.character(v) && !nzchar(v))) return(NULL)
+  n <- suppressWarnings(as.integer(v))
+  if (length(n) != 1L || is.na(n) || n < 1L) {
+    cli::cli_abort(c(
+      "{.envvar USCOGDATA_DUCKDB_THREADS} must be a single positive integer.",
+      x = "Got {.val {v}}.",
+      i = "Unset it (or {.code options(uscogdata.duckdb_threads = NULL)}) to use DuckDB's default of every visible core."
+    ), class = "uscogdata_invalid_duckdb_threads")
+  }
+  n
+}
+
+#' Resolve the DuckDB memory limit, or NULL to leave DuckDB's default alone.
+#'
+#' The value is a DuckDB size string (`"4GB"`, `"512MB"`). Only its SHAPE is
+#' checked here -- DuckDB owns the unit vocabulary, and re-implementing that
+#' parse would be a second definition free to drift from the engine's. An
+#' unrecognised unit therefore surfaces as DuckDB's own error at `SET` time,
+#' which names the setting correctly; the check here exists to reject the
+#' inputs that would otherwise reach the connection as a SQL fragment.
+#' @noRd
+.resolve_duckdb_memory_limit <- function() {
+  v <- .cfg("duckdb_memory_limit")
+  if (is.null(v) || (is.character(v) && !nzchar(v))) return(NULL)
+  if (length(v) != 1L || !is.character(v) ||
+      !grepl("^[0-9]+(\\.[0-9]+)?\\s*[A-Za-z]{0,3}$", v)) {
+    cli::cli_abort(c(
+      "{.envvar USCOGDATA_DUCKDB_MEMORY_LIMIT} must be a single DuckDB size string.",
+      x = "Got {.val {v}}.",
+      i = "Examples: {.val 4GB}, {.val 512MB}, {.val 1.5GB}."
+    ), class = "uscogdata_invalid_duckdb_memory_limit")
+  }
+  trimws(v)
+}

R/explain.R

@@ -11,6 +11,30 @@
 #'   returns `result` invisibly for chaining. `"list"` returns the raw
 #'   provenance list (identical to `attr(result, "provenance")`).
 #' @return Either `result` (invisibly) or the provenance list.
+#' @section Two kinds of series break:
+#' Catalogued breaks reach you without being asked for, in two disjoint
+#' fields, because a caveat about one series and a caveat about the whole
+#' corpus are different claims:
+#'
+#' * **`series_break_refs`** — breaks matched against the item codes actually
+#'   present in this result. A break in one code you queried.
+#' * **`corpus_break_refs`** — breaks catalogued with `fin_code = "ALL"`,
+#'   which are statements about the corpus rather than about any one code:
+#'   dollar precision across the 1976/1977 boundary (`SB085`), imputation
+#'   exclusion from FY2002 (`SB087`), the FY2012 dense-to-sparse
+#'   representation change (`SB194`), and the FY2017 government-identifier
+#'   change (`SB086`). These are selected on the break-year window alone.
+#'
+#' `SB194` is the one most likely to matter: a query spanning FY2011 to FY2012
+#' crosses the boundary where an absent cell stops meaning "Census published
+#' $0" and starts meaning "not reported".
+#' @section Other provenance blocks:
+#' `transformations$units_conversion` records the `$1,000s`-to-dollars
+#' multiply that every amount column has already had applied.
+#' `transformations$per_capita` records the population denominator and its
+#' year range. `coverage` and `coverage_mode` appear on multi-government
+#' results (see [cog_geographic_rollup()]). `completion` appears when
+#' `complete = TRUE`. `balance_caveats` appears on [cog_balances()] results.
 #' @export
 cog_explain <- function(result, format = c("print", "list")) {
   format <- match.arg(format)

R/manifest.R

@@ -28,7 +28,7 @@
       "*" = "{.code Sys.setenv(USCOGDATA_URL = \"<url-or-local-path>/\")}",
       "*" = "{.code options(uscogdata.url = \"<url-or-local-path>/\")}",
       i = "For an offline smoke test, use the bundled fixture: {.code system.file(\"extdata/fixture_corpus\", package = \"uscogdata\")}.",
-      i = "For the live Civilytics corpus, request the Nextcloud share URL from the package maintainer."
+      i = "The public corpus is the default: unset USCOGDATA_URL to use it, or point it at a local copy made by {.code cog_mirror()}."
     ), class = "uscogdata_url_not_configured")
   }
   invisible(url)

R/pagination.R

@@ -0,0 +1,81 @@
+# R/pagination.R
+#
+# Shared limit/offset machinery. #39 established the semantics inside
+# .verb_spendrev(); #57 extends them to cog_gov_search() and cog_balances(),
+# which is what made a single definition worth having: three inline copies of
+# "coerce, refuse, unwrap the count" would be three places for the meaning of
+# `total_rows` to drift.
+#
+# The SQL side stays in .build_verb_sql() (R/spending.R) -- it already wraps
+# the aggregate in an outer SELECT so COUNT(*) OVER() sees the post-GROUP-BY
+# row count rather than the pre-aggregation one, and that is the subtle part
+# worth not duplicating either.
+
+#' Coerce and check a limit/offset pair.
+#'
+#' Returns the coerced pair, or NULL for `limit` when no page was requested.
+#' `offset` defaults to 0 whenever `limit` is set, so a caller can supply just
+#' `limit` and get the first page.
+#'
+#' Conflicts with other arguments are deliberately NOT checked here: they
+#' differ per verb (`complete`/`recipe` for the money verbs, basket mode for
+#' `cog_gov_search()`, `recipe` alone for `cog_balances()`), and a shared
+#' function taking a list of conflict flags would be harder to read than the
+#' three explicit refusals at the call sites.
+#' @noRd
+.validate_pagination <- function(limit, offset) {
+  if (is.null(limit)) {
+    return(list(limit = NULL, offset = NULL))
+  }
+  limit <- as.integer(limit)
+  if (length(limit) != 1L || is.na(limit) || limit < 0L) {
+    cli::cli_abort("`limit` must be a single non-negative integer.",
+                   class = "uscogdata_invalid_pagination")
+  }
+  offset <- if (is.null(offset)) 0L else as.integer(offset)
+  if (length(offset) != 1L || is.na(offset) || offset < 0L) {
+    cli::cli_abort("`offset` must be a single non-negative integer.",
+                   class = "uscogdata_invalid_pagination")
+  }
+  list(limit = limit, offset = offset)
+}
+
+#' Wrap a query so one page comes back carrying the unpaginated total.
+#'
+#' `COUNT(*) OVER()` rides along as an ordinary column, so the caller gets the
+#' true total from the SAME scan instead of a second round trip. The outer
+#' `SELECT *` matters: appending LIMIT/OFFSET directly to a grouped query would
+#' have the window function count pre-aggregation rows.
+#' @noRd
+.paginate_sql <- function(base_sql, limit, offset) {
+  if (is.null(limit)) return(base_sql)
+  sprintf(
+    "SELECT *, COUNT(*) OVER() AS pagination_total_rows
+     FROM (%s) AS _paged
+     LIMIT %d OFFSET %d",
+    base_sql, limit, offset
+  )
+}
+
+#' Strip the count column back out and report the unpaginated total.
+#'
+#' Returns `list(result = , total_rows = )`.
+#'
+#' An empty page -- an offset past the end -- carries no row to read the window
+#' function off, so that one case falls back to a second, unpaginated
+#' `COUNT(*)` rather than reporting a wrong zero. `unpaged_sql` is passed as a
+#' function so the fallback query is only BUILT when it is actually needed;
+#' every caller's unpaginated SQL is otherwise constructed on every paged call
+#' and thrown away.
+#' @noRd
+.take_pagination_total <- function(result, con, unpaged_sql) {
+  if (nrow(result) > 0L) {
+    total <- result$pagination_total_rows[[1]]
+    result$pagination_total_rows <- NULL
+    return(list(result = result, total_rows = as.integer(total)))
+  }
+  count_sql <- sprintf("SELECT COUNT(*) AS n FROM (%s) AS _uncounted",
+                       if (is.function(unpaged_sql)) unpaged_sql() else unpaged_sql)
+  list(result = result,
+       total_rows = as.integer(DBI::dbGetQuery(con, count_sql)$n[[1]]))
+}

R/recipes.R

@@ -113,7 +113,7 @@ cog_recipes <- function(pattern = NULL) {
 #' `year_min`/`year_max` -- so there is no double-counting. (Checkpoint
 #' review docs/phase_r_harmonization_review.md § 0.2.)
 #' @noRd
-.run_recipe <- function(con, recipe_id, govid, years) {
+.run_recipe <- function(con, recipe_id, cohort, years) {
   sql <- sprintf(
     "SELECT l.year, l.canonical_govid,
             COALESCE(x.gov_name, l.gov_name) AS gov_name,
@@ -128,11 +128,11 @@ cog_recipes <- function(pattern = NULL) {
            OR (r.gov_type_scope = 'local' AND l.type BETWEEN 1 AND 3))
      LEFT JOIN canonical_fips_xwalk x USING (canonical_govid)
      WHERE r.recipe_id = %1$s
-       AND l.canonical_govid IN (%2$s)
+       AND %2$s
        AND l.year IN (%3$s)
      GROUP BY 1, 2, 3
      ORDER BY 1, 2",
-    .sql_lit_chr(recipe_id), .sql_lit_chr(govid),
+    .sql_lit_chr(recipe_id), .cohort_sql(cohort, "l.canonical_govid"),
     paste(as.integer(years), collapse = ",")
   )
   result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))

R/revenue.R

@@ -50,11 +50,12 @@
 #'   optional `pop_source`, `codes_included`, `aggregate_fallback`, `notes`,
 #'   and `value_source` when `complete = TRUE`.
 #' @export
-cog_revenue <- function(govid, years, category = NULL,
+cog_revenue <- function(govid = NULL, years, category = NULL,
                         per_capita = FALSE, adjust_to_year = NULL,
                         basis = c("harmonized", "raw"), recipe = NULL,
                         revenue_concept = c("general", "total"),
-                        complete = FALSE, limit = NULL, offset = NULL) {
+                        complete = FALSE, limit = NULL, offset = NULL,
+                        state = NULL, type = NULL) {
   # flow_prefixes no longer classifies rows (crosswalk revenue_subtype
   # membership does -- General Revenue, i.e. everything except
   # insurance_trust) -- it only scopes the recipe-suggestion machinery to
@@ -75,6 +76,8 @@ cog_revenue <- function(govid, years, category = NULL,
     revenue_concept = revenue_concept,
     complete       = complete,
     limit          = limit,
-    offset         = offset
+    offset         = offset,
+    state          = state,
+    type           = type
   )
 }

R/search.R

@@ -44,10 +44,22 @@
 #'   in basket mode (recycles from length 1). Excluded types `4`/`5` (or
 #'   `"special_district"` / `"school_district"`) trigger an explanatory
 #'   message and an empty result.
+#' @param limit Maximum number of rows to return, applied in SQL. `NULL`
+#'   (default) returns every match -- which, with no other filter, is the
+#'   entire crosswalk. Utility mode only: pagination has no meaning in basket
+#'   mode, where the result is one resolved row per requested name in input
+#'   order, and is refused there with class
+#'   `uscogdata_basket_pagination_conflict`.
+#' @param offset Rows to skip before `limit` starts counting (0-based).
+#'   Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.
 #' @return A tibble of `canonical_fips_xwalk` rows. In utility mode, all
-#'   matches sorted by `population_acs` desc. In basket mode, resolved
+#'   matches sorted by `population_acs` desc, ties broken by
-#'   rows in input order, with `attr(., "resolution")` set to the
+#'   `canonical_govid`. In basket mode, resolved rows in input order, with
-#'   sidecar tibble.
+#'   `attr(., "resolution")` set to the sidecar tibble.
+#'
+#'   When `limit` is set, carries a `total_rows` attribute: the full
+#'   unpaginated match count, computed by the same query (`COUNT(*) OVER()`)
+#'   rather than a second scan.
 #' @seealso [cog_basket_resolution()], [cog_basket_unresolved()],
 #'   [cog_spending()], [cog_revenue()].
 #' @examples
@@ -82,7 +94,12 @@
 #' )
 #' }
 #' @export
-cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
+cog_gov_search <- function(name = NULL, state = NULL, type = NULL,
+                           limit = NULL, offset = NULL) {
+  paging <- .validate_pagination(limit, offset)
+  limit  <- paging$limit
+  offset <- paging$offset
+
   if (!is.null(type) && length(type) == 1L && .is_excluded_type(type)) {
     cli::cli_inform(c(
       i = "v0.1 covers gov_types 0-3 (state/county/city/township) only.",
@@ -94,6 +111,18 @@ cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
   con <- .ensure_session()
 
   if (length(name) > 1L) {
+    # Basket mode returns one resolved row per requested name, in input order,
+    # with a resolution sidecar describing how each was matched. A page of that
+    # is not a page of anything the caller asked for -- the sidecar would still
+    # describe every name -- so refuse rather than silently ignoring the
+    # arguments. Same shape as the recipe/complete refusals in .verb_spendrev().
+    if (!is.null(limit)) {
+      cli::cli_abort(c(
+        "`limit`/`offset` cannot be combined with basket mode.",
+        "i" = "Basket mode ({.code length(name) > 1}) returns one resolved row per requested name, in input order, with a resolution sidecar covering all of them.",
+        "*" = "Drop `limit`/`offset`, or search one name at a time."
+      ), class = "uscogdata_basket_pagination_conflict")
+    }
     return(.resolve_basket(name = name, state = state, type = type, con = con))
   }
 
@@ -123,12 +152,26 @@ cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
   }
 
   where <- if (length(preds) == 0L) "" else paste("WHERE", paste(preds, collapse = " AND "))
-  sql <- paste(
+  # canonical_govid breaks ties. population_acs alone is NOT a total order --
+  # governments sharing a population, and the whole NULLS LAST block, came back
+  # in whatever order the scan produced. That was invisible while every call
+  # returned the full result set, but it makes a paged sweep unsound: two
+  # requests can order the tied rows differently, so a row is duplicated on one
+  # page and missing from the next. Any pagination has to sit on a total order.
+  base_sql <- paste(
     "SELECT * FROM canonical_fips_xwalk",
     where,
-    "ORDER BY population_acs DESC NULLS LAST"
+    "ORDER BY population_acs DESC NULLS LAST, canonical_govid"
   )
-  tibble::as_tibble(DBI::dbGetQuery(con, sql))
+  result <- tibble::as_tibble(
+    DBI::dbGetQuery(con, .paginate_sql(base_sql, limit, offset))
+  )
+  if (is.null(limit)) return(result)
+
+  paged <- .take_pagination_total(result, con, base_sql)
+  out <- paged$result
+  attr(out, "total_rows") <- paged$total_rows
+  out
 }
 
 #' @noRd
@@ -184,11 +227,18 @@ cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
   if (!is.character(state) || length(state) != 1L) {
     cli::cli_abort("`state` must be a 2-letter USPS abbrev or a FIPS integer.")
   }
-  fips <- .state_abbrev_to_fips[[toupper(state)]]
+  # Membership tested before the lookup, not after: `.state_abbrev_to_fips` is
-  if (is.null(fips)) {
+  # a named CHARACTER vector, and `[[` on a name it does not carry throws
-    cli::cli_abort("Unknown state abbreviation: {state}.")
+  # base R's "subscript out of bounds" rather than returning NULL -- which
+  # made the curated message below unreachable dead code. Reported as a bare
+  # subscript error, `cog_gov_search(state = "ZZ")` gave no hint that the
+  # argument wants a postal abbreviation.
+  key <- toupper(state)
+  if (!key %in% names(.state_abbrev_to_fips)) {
+    cli::cli_abort("Unknown state abbreviation: {state}.",
+                   class = "uscogdata_unknown_state")
   }
-  fips
+  .state_abbrev_to_fips[[key]]
 }
 
 # USPS state / territory abbreviation -> 2-digit FIPS code.

R/session.R

@@ -2,13 +2,23 @@
 
 #' Internal: open session, register views, cache manifest.
 #' Not exported. Called lazily by verbs via .ensure_session().
+#'
+#' `threads` and `memory_limit` default to the resolved configuration and are
+#' applied as pragmas on the new connection. When both resolve to NULL -- which
+#' is the case unless the operator sets one -- NO pragma is issued at all, so an
+#' unconfigured session connects exactly as it did before this argument existed.
 #' @noRd
 cog_open <- function(url = .resolve_url(),
-                     cache_dir = .resolve_cache_dir()) {
+                     cache_dir = .resolve_cache_dir(),
+                     threads = .resolve_duckdb_threads(),
+                     memory_limit = .resolve_duckdb_memory_limit()) {
   .check_url_configured(url)
   if (!dir.exists(cache_dir)) dir.create(cache_dir, recursive = TRUE)
 
   con <- DBI::dbConnect(duckdb::duckdb())
+  # Before anything else touches the connection: httpfs reads the corpus, and
+  # a remote read should already be bound by whatever budget the operator set.
+  .apply_duckdb_limits(con, threads, memory_limit)
   DBI::dbExecute(con, "INSTALL httpfs; LOAD httpfs;")
 
   manifest <- .fetch_or_cache_manifest(url, cache_dir)
@@ -25,6 +35,26 @@ cog_open <- function(url = .resolve_url(),
   invisible(con)
 }
 
+#' Apply the operator's DuckDB resource budget to a fresh connection.
+#'
+#' Split out from cog_open() so the "unset changes nothing" property is one
+#' readable branch rather than two conditionals buried in the connection path.
+#' Both settings are session-scoped in DuckDB, so this must run per connection;
+#' cog_close() discards the connection and the next cog_open() re-resolves,
+#' which is what makes a changed option take effect on the next session.
+#' @noRd
+.apply_duckdb_limits <- function(con, threads, memory_limit) {
+  if (!is.null(threads)) {
+    DBI::dbExecute(con, sprintf("SET threads TO %d", threads))
+  }
+  if (!is.null(memory_limit)) {
+    # Quoted as a string literal: DuckDB's memory_limit takes '4GB', not 4GB.
+    DBI::dbExecute(con, sprintf("SET memory_limit TO %s",
+                                .sql_lit_chr(memory_limit)))
+  }
+  invisible(con)
+}
+
 #' @noRd
 .ensure_session <- function() {
   if (is.null(.uscogdata_env$con) ||

R/spending.R

@@ -68,7 +68,9 @@
 #' millions/billions). The conversion is recorded in the provenance attribute
 #' under `transformations$units_conversion`.
 #'
-#' @param govid Character vector of `canonical_govid` values.
+#' @param govid Character vector of `canonical_govid` values, or `NULL` to name
+#'   the cohort by `state`/`type` instead. One of `govid`, `state`, or `type`
+#'   is required.
 #' @param years Integer vector of years.
 #' @param category Character vector of category names (from
 #'   `summary_categories.category`), or `NULL` for all categories broken out
@@ -185,6 +187,29 @@
 #'   `recipe` and with `complete = TRUE` -- see `offset` and `total_rows`.
 #' @param offset Rows to skip before `limit` starts counting (0-based).
 #'   Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.
+#' @param state,type Name the cohort by predicate instead of by id: `state` is
+#'   a 2-letter USPS abbreviation (or a FIPS code) and `type` is one of
+#'   `"state"`, `"county"`, `"city"`, `"township"` (or the integer `0:3`) --
+#'   the same vocabulary, and the same internal coercion, as
+#'   [cog_gov_search()]. Both default to `NULL`.
+#'
+#'   The cohort is then expressed as a subquery against `canonical_fips_xwalk`
+#'   inside each statement rather than round-tripped through R as a literal id
+#'   list. For a fleet-scale cohort that is the difference between a
+#'   301,591-character `IN` list re-parsed in 5--8 statements per call and a
+#'   constant-size predicate: measured at **94 ms versus 449 ms** for the same
+#'   FY2022 aggregate over the 20,106-government `type = "city"` cohort, within
+#'   7% of the no-filter floor.
+#'
+#'   Supplying `govid` **and** `state`/`type` INTERSECTS them -- the
+#'   governments in `govid` that also match the predicate -- rather than one
+#'   silently taking precedence. Naming no cohort at all (`govid`, `state` and
+#'   `type` all `NULL`) aborts with class `uscogdata_no_cohort`.
+#'
+#'   When the cohort is named by predicate, `provenance$scope$govids_found`
+#'   and `govids_missing` are empty -- there is no id list to report against --
+#'   and `provenance$scope$cohort` carries `state`, `type` and
+#'   `n_governments` instead. A `govid`-named cohort reports exactly as before.
 #' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
 #'   `spend_subtype`, `category`, `amt_nominal`, optional `amt_real`,
 #'   optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
@@ -198,11 +223,12 @@
 #'   round trip -- so a caller walking pages never has to ask "how many are
 #'   there" separately.
 #' @export
-cog_spending <- function(govid, years, category = NULL,
+cog_spending <- function(govid = NULL, years, category = NULL,
                          per_capita = FALSE, adjust_to_year = NULL,
                          basis = c("harmonized", "raw"), recipe = NULL,
                          expenditure_concept = c("primary", "direct", "total"),
-                         complete = FALSE, limit = NULL, offset = NULL) {
+                         complete = FALSE, limit = NULL, offset = NULL,
+                         state = NULL, type = NULL) {
   # flow_prefixes no longer classifies rows (crosswalk subtype membership
   # does, per expenditure_concept) -- it only scopes the recipe-suggestion
   # machinery to this verb's recipe families (see R/suggestions.R; the
@@ -223,7 +249,9 @@ cog_spending <- function(govid, years, category = NULL,
     expenditure_concept = expenditure_concept,
     complete      = complete,
     limit         = limit,
-    offset        = offset
+    offset        = offset,
+    state         = state,
+    type          = type
   )
 }
 
@@ -251,7 +279,8 @@ cog_spending <- function(govid, years, category = NULL,
                            basis = c("harmonized", "raw"), recipe = NULL,
                            expenditure_concept = c("primary", "direct", "total"),
                            revenue_concept = c("general", "total"),
-                           complete = FALSE, limit = NULL, offset = NULL) {
+                           complete = FALSE, limit = NULL, offset = NULL,
+                           state = NULL, type = NULL) {
   basis_explicit <- length(basis) == 1L
   basis <- match.arg(basis, c("harmonized", "raw"))
   # match.arg() itself throws a base `simpleError`, not an rlang-classed
@@ -291,7 +320,10 @@ cog_spending <- function(govid, years, category = NULL,
     .revenue_concept_subtypes(revenue_concept)
   }
 
-  govid <- .coerce_govid_input(govid, arg = "govid")
+  # NULL `govid` means "the cohort is named by predicate"; anything else is
+  # coerced and validated exactly as before, so an empty or wrong-typed vector
+  # still fails with its original message rather than being read as absent.
+  govid <- if (is.null(govid)) NULL else .coerce_govid_input(govid, arg = "govid")
   # allow_all_categories = TRUE: cog_spending()/cog_revenue() are the two
   # verbs the reserved pseudo-category is defined for. cog_balances() shares
   # this validator but leaves the argument at its FALSE default, so it
@@ -300,6 +332,10 @@ cog_spending <- function(govid, years, category = NULL,
   .validate_verb_inputs(govid, years, category, per_capita, adjust_to_year,
                         recipe, allow_all_categories = TRUE)
 
+  # Built after validation so the argument-shape errors above keep firing
+  # first, and before .ensure_session() so a bad state/type costs no I/O.
+  cohort <- .make_cohort(govid, state, type)
+
   # Recognize the reserved pseudo-category. Detected after type validation so a
   # non-character `category` still fails with the ordinary type error.
   all_categories <- !is.null(category) && .ALL_CATEGORIES %in% category
@@ -362,17 +398,10 @@ cog_spending <- function(govid, years, category = NULL,
   # up front rather than silently ignored: complete = TRUE fills a grid over
   # the FULL requested (year, category) space, and a recipe's result comes
   # from .run_recipe()'s own query, which this function does not touch.
+  paging <- .validate_pagination(limit, offset)
+  limit  <- paging$limit
+  offset <- paging$offset
   if (!is.null(limit)) {
-    limit <- as.integer(limit)
-    if (length(limit) != 1L || is.na(limit) || limit < 0L) {
-      cli::cli_abort("`limit` must be a single non-negative integer.",
-                     class = "uscogdata_invalid_pagination")
-    }
-    offset <- if (is.null(offset)) 0L else as.integer(offset)
-    if (length(offset) != 1L || is.na(offset) || offset < 0L) {
-      cli::cli_abort("`offset` must be a single non-negative integer.",
-                     class = "uscogdata_invalid_pagination")
-    }
     if (complete) {
       cli::cli_abort(c(
         "`limit`/`offset` cannot be combined with `complete = TRUE`.",
@@ -407,7 +436,7 @@ cog_spending <- function(govid, years, category = NULL,
     .validate_recipe_id(con, recipe)
     comps <- .recipe_components(con, recipe)
     recipe_label <- comps$label[[1]]
-    result <- .run_recipe(con, recipe, govid, years)
+    result <- .run_recipe(con, recipe, cohort, years)
     sql <- attr(result, "sql_query")
     result <- .shape_recipe_result(result, subtype_col, recipe_label)
     recipe_block <- list(
@@ -423,31 +452,21 @@ cog_spending <- function(govid, years, category = NULL,
     } else {
       NULL
     }
-    sql <- .build_verb_sql(view, subtype_col, govid, years,
+    sql <- .build_verb_sql(view, subtype_col, cohort, years,
                            if (all_categories) NULL else category,
                            ig_view, subtype_scope,
                            all_categories = all_categories,
                            limit = limit, offset = offset)
     result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
     if (!is.null(limit)) {
-      # COUNT(*) OVER() rides along as an ordinary column so the total comes
+      paged <- .take_pagination_total(result, con, function() {
-      # from the same scan when this page has any rows -- see
+        .build_verb_sql(view, subtype_col, cohort, years,
-      # .build_verb_sql(). An empty page (offset past the end) carries no
+                        if (all_categories) NULL else category,
-      # such row to read it from, so that one case falls back to a second,
+                        ig_view, subtype_scope,
-      # unpaginated COUNT(*) query rather than reporting a wrong zero.
+                        all_categories = all_categories)
-      if (nrow(result) > 0L) {
+      })
-        total_rows <- result$pagination_total_rows[[1]]
+      result     <- paged$result
-        result$pagination_total_rows <- NULL
+      total_rows <- paged$total_rows
-      } else {
-        count_sql <- sprintf(
-          "SELECT COUNT(*) AS n FROM (%s) AS _uncounted",
-          .build_verb_sql(view, subtype_col, govid, years,
-                          if (all_categories) NULL else category,
-                          ig_view, subtype_scope,
-                          all_categories = all_categories)
-        )
-        total_rows <- as.integer(DBI::dbGetQuery(con, count_sql)$n[[1]])
-      }
     }
   }
 
@@ -457,13 +476,13 @@ cog_spending <- function(govid, years, category = NULL,
   # spurious 0.
   completion <- list(applied = FALSE, rows_filled = 0L, absence_means = list())
   if (complete) {
-    result <- .complete_result(result, con, subtype_col, govid, years,
+    result <- .complete_result(result, con, subtype_col, cohort, years,
                                category, subtype_scope)
     completion <- attr(result, ".completion")
     attr(result, ".completion") <- NULL
   }
 
-  if (per_capita) result <- .attach_per_capita(result, con, govid)
+  if (per_capita) result <- .attach_per_capita(result, con)
   if (!is.null(adjust_to_year)) {
     result <- .attach_real_dollars(result, adjust_to_year, per_capita)
   }
@@ -491,7 +510,7 @@ cog_spending <- function(govid, years, category = NULL,
     basis_for_prov <- resolved$basis
     basis_note_for_prov <- resolved$note
     harmonization <- .build_harmonization_block(
-      con, govid, years, resolved, subtype_col, subtype_scope
+      con, cohort, years, resolved, subtype_col, subtype_scope
     )
     # C1(a): gap detection must run against the Direct leg alone. `result`
     # can also carry UNION'd intergovernmental rows (expenditure_concept =
@@ -506,7 +525,7 @@ cog_spending <- function(govid, years, category = NULL,
     } else {
       result
     }
-    suggestions <- .build_suggestions(con, govid, years, category,
+    suggestions <- .build_suggestions(con, cohort, years, category,
                                        direct_leg_result,
                                        resolved$basis, flow_prefixes,
                                        .select_long_view(view_base, resolved$basis),
@@ -611,6 +630,12 @@ cog_spending <- function(govid, years, category = NULL,
   )
   prov$scope$govids_found   <- scope$found
   prov$scope$govids_missing <- scope$missing
+  # A predicate-named cohort has no id list to report found/missing against
+  # (both stay empty), so it describes itself instead. Deliberately a COUNT
+  # rather than the resolved ids: enumerating them would put 20,000 govids in
+  # every fleet-scale response body, which is the cost this path exists to
+  # remove. NULL for a govid-named cohort, so that output is untouched.
+  prov$scope$cohort <- .cohort_provenance(con, cohort)
   attr(result, "provenance") <- prov
   attr(result, ".popyear_range") <- NULL
   # Attached here, after every downstream transform (per_capita/real-dollar
@@ -639,7 +664,11 @@ cog_spending <- function(govid, years, category = NULL,
 .validate_verb_inputs <- function(govid, years, category,
                                   per_capita, adjust_to_year, recipe = NULL,
                                   allow_all_categories = FALSE) {
-  if (!is.character(govid) || length(govid) == 0L) {
+  # NULL is allowed only because the caller has already established that the
+  # cohort is named some other way (`state`/`type`); .make_cohort() is what
+  # refuses a call that names no cohort at all. A supplied-but-empty `govid`
+  # still fails here, exactly as before.
+  if (!is.null(govid) && (!is.character(govid) || length(govid) == 0L)) {
     cli::cli_abort("`govid` must be a non-empty character vector.")
   }
   if (!(is.integer(years) || is.numeric(years)) || length(years) == 0L) {
@@ -740,10 +769,10 @@ cog_spending <- function(govid, years, category = NULL,
 }
 
 #' @noRd
-.build_verb_sql <- function(view, subtype_col, govid, years, category,
+.build_verb_sql <- function(view, subtype_col, cohort, years, category,
                             ig_view = NULL, subtype_scope = NULL,
                             all_categories = FALSE, limit = NULL, offset = NULL) {
-  govid_lit <- .sql_lit_chr(govid)
+  cohort_pred <- .cohort_sql(cohort)
   years_lit <- paste(as.integer(years), collapse = ",")
   # In all-categories mode there is no category filter: the sum is defined by
   # the concept's SUBTYPE allowlist (subtype_pred below), which is the real
@@ -813,13 +842,13 @@ cog_spending <- function(govid, years, category = NULL,
        string_agg(DISTINCT item_code, ',' ORDER BY item_code) AS codes_included,
        bool_or(is_aggregate) AS aggregate_fallback
      FROM %2$s
-     WHERE canonical_govid IN (%3$s)
+     WHERE %3$s
        AND year IN (%4$s)
        %5$s
        %6$s
      GROUP BY year, canonical_govid, gov_name, xwalk_gov_name, %1$s%8$s
      ORDER BY year, canonical_govid, %1$s%8$s",
-    subtype_col, source_expr, govid_lit, years_lit, category_pred, subtype_pred,
+    subtype_col, source_expr, cohort_pred, years_lit, category_pred, subtype_pred,
     category_select, category_group
   )
 
@@ -827,26 +856,21 @@ cog_spending <- function(govid, years, category = NULL,
   # matching row across the network only to slice and discard most of it
   # afterward (the pattern behind the 2026-08-06 production incident: a
   # 193,105-row/194-page sweep re-ran the full query and re-listified every
-  # row on EVERY page). COUNT(*) OVER() rides along as an ordinary column so
+  # row on EVERY page). See .paginate_sql() in R/pagination.R for why the
-  # the caller gets the true total from this same scan -- see the call site
+  # wrapping is an outer SELECT rather than a bare LIMIT on base_sql.
-  # in .verb_spendrev(), which reads it off row 1 and strips it back out.
+  .paginate_sql(base_sql, limit, offset)
-  # The outer SELECT * wrapping (rather than appending LIMIT/OFFSET directly
-  # to base_sql) is what makes COUNT(*) OVER() see the post-GROUP-BY row
-  # count, not the pre-aggregation one.
-  if (is.null(limit)) {
-    base_sql
-  } else {
-    sprintf(
-      "SELECT *, COUNT(*) OVER() AS pagination_total_rows
-       FROM (%s) AS _paged
-       LIMIT %d OFFSET %d",
-      base_sql, limit, offset
-    )
-  }
 }
 
+#' Join population onto a result and derive the per-capita columns.
+#'
+#' The population lookup is keyed on the govids PRESENT IN `result`, not on the
+#' cohort that produced it. Those are the only ones the LEFT JOIN below can
+#' match, so the joined output is identical either way -- but it means this
+#' works unchanged for a cohort named by predicate (where no id list exists in
+#' R at all), and on a paginated call it looks up one page's governments
+#' instead of the whole fleet's.
 #' @noRd
-.attach_per_capita <- function(result, con, govid) {
+.attach_per_capita <- function(result, con) {
   if (nrow(result) == 0L) {
     result$amt_per_capita_nominal <- numeric(0)
     result$pop_source <- character(0)
@@ -859,7 +883,7 @@ cog_spending <- function(govid, years, category = NULL,
      FROM gov_population_yearly
      WHERE canonical_govid IN (%s)
        AND year IN (%s)",
-    .sql_lit_chr(govid), years_lit
+    .sql_lit_chr(unique(result$canonical_govid)), years_lit
   )
   pops <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
   result <- dplyr::left_join(result, pops,

R/suggestions.R

@@ -42,8 +42,8 @@
 #' verb call: recipes whose generic join would fill a real gap in `result`.
 #'
 #' @param con Active DuckDB connection.
-#' @param govid Character vector of canonical_govid values (the verb's raw
+#' @param cohort The verb's cohort object (see `.make_cohort()`), naming the
-#'   `govid`).
+#'   governments by id, by state/type predicate, or both.
 #' @param years Integer vector of requested years.
 #' @param category `category` argument as passed to the verb (character
 #'   vector or `NULL`; suggestions are only computed when non-NULL).
@@ -85,7 +85,7 @@
 #'   ig_recipe_id, trigger, suppressed_amount, suppressed_years,
 #'   suppressed_codes)`, possibly empty.
 #' @noRd
-.build_suggestions <- function(con, govid, years, category, result, basis,
+.build_suggestions <- function(con, cohort, years, category, result, basis,
                                 flow_prefixes, long_view,
                                 all_categories = FALSE,
                                 subtype_col = NULL, subtype_scope = NULL) {
@@ -160,7 +160,7 @@
   # violation would kill signposting, the exact failure class uscogdata#9
   # exists to prevent. Owner's call: keep this simple; a batch-aware
   # optimization, if one is worth building, is a separate issue.
-  supp <- .suppressed_components(con, candidates, govid, years, long_view, flow_prefixes)
+  supp <- .suppressed_components(con, candidates, cohort, years, long_view, flow_prefixes)
 
   if (length(gap_years) == 0L && nrow(supp) == 0L) return(list())
 
@@ -188,9 +188,9 @@
              OR (r.gov_type_scope = 'state' AND l.type = 0)
              OR (r.gov_type_scope = 'local' AND l.type BETWEEN 1 AND 3))
        WHERE r.recipe_id IN (%s)
-         AND l.canonical_govid IN (%s)
+         AND %s
          AND l.year IN (%s)",
-      .sql_lit_chr(candidates), .sql_lit_chr(govid),
+      .sql_lit_chr(candidates), .cohort_sql(cohort, "l.canonical_govid"),
       paste(gap_years, collapse = ",")
     ))
   }

R/suppression.R

@@ -50,7 +50,9 @@
 #'
 #' @param con Active DuckDB connection.
 #' @param candidates Character vector of recipe ids to measure.
-#' @param govid Character vector of canonical_govid values.
+#' @param cohort The verb's cohort object (see `.make_cohort()`), rendered
+#'   into the govid predicate on both the outer scan and the restated
+#'   NOT EXISTS filter.
 #' @param years Integer vector of requested years.
 #' @param long_view Name of the verb's long view, from `.select_long_view()`.
 #' @param flow_prefixes The calling verb's own flow-type prefixes (see
@@ -60,7 +62,7 @@
 #'   dollars), `suppressed_codes` (comma-joined, sorted). Zero rows when
 #'   nothing is suppressed.
 #' @noRd
-.suppressed_components <- function(con, candidates, govid, years, long_view,
+.suppressed_components <- function(con, candidates, cohort, years, long_view,
                                     flow_prefixes) {
   empty <- tibble::tibble(
     recipe_id = character(0), year = numeric(0),
@@ -93,7 +95,7 @@
            OR (r.gov_type_scope = 'state' AND l.type = 0)
            OR (r.gov_type_scope = 'local' AND l.type BETWEEN 1 AND 3))
      WHERE r.recipe_id IN (%1$s)
-       AND l.canonical_govid IN (%2$s)
+       AND %2$s
        AND l.year IN (%3$s)
        AND l.amt <> 0
        AND LEFT(r.component_code, 1) IN (%5$s)
@@ -103,13 +105,13 @@
            AND v.year = l.year
            AND v.item_code = l.item_code
            AND v.year IN (%3$s)              -- restated: enables partition pruning (I3a)
-           AND v.canonical_govid IN (%2$s)   -- restated: pushes the govid filter (I3a)
+           AND %6$s   -- restated: pushes the cohort filter (I3a)
        )
      GROUP BY 1, 2
      ORDER BY 1, 2",
-    .sql_lit_chr(candidates), .sql_lit_chr(govid),
+    .sql_lit_chr(candidates), .cohort_sql(cohort, "l.canonical_govid"),
     paste(as.integer(years), collapse = ","), long_view,
-    .sql_lit_chr(flow_prefixes)
+    .sql_lit_chr(flow_prefixes), .cohort_sql(cohort, "v.canonical_govid")
   )
   tibble::as_tibble(DBI::dbGetQuery(con, sql))
 }

R/views.R

@@ -78,6 +78,71 @@
   file %in% basename(paths)
 }
 
+#' Build the SQL path expression for the partitioned `long` table.
+#'
+#' DuckDB cannot expand a glob over generic HTTP: there is no directory
+#' listing to expand against, and `allow_asterisks_in_http_paths` only
+#' forwards the literal `**/*` as a filename, which 404s. Measured against
+#' the published corpus on 2026-08-08, an explicit file list returns the
+#' same 46,148,034 rows the (working) `hf://` glob does, and
+#' `hive_partitioning = true` still recovers `year` from the paths.
+#'
+#' The manifest already enumerates every partition, so we build the list
+#' from it. This is host-agnostic -- Nextcloud, HuggingFace and a local
+#' fixture take the same path -- where an `hf://` URL would tie the reader
+#' to one vendor's protocol and still need special-casing, since manifest
+#' fetching goes through httr2, which cannot speak `hf://`.
+#'
+#' Falls back to the glob when the manifest carries no partition list: a
+#' hand-built manifest in a test (see test-views.R) or a corpus predating
+#' the field. Both are local, where globbing works.
+#' @noRd
+.long_files_sql <- function(url, manifest) {
+  parts <- manifest$files$long_partitions %||% list()
+  if (length(parts) == 0L) {
+    return(.sql_lit_chr(paste0(url, "data/long/**/*.parquet")))
+  }
+  paths <- vapply(parts, function(p) as.character(p$path), character(1))
+  paste0("[", .sql_lit_chr(paste0(url, paths)), "]")
+}
+
+#' Substitute the corpus-location tokens in a view's SQL text.
+#'
+#' One place knows the token vocabulary. `.register_views()` and the tests
+#' that execute a view file directly both route through here. This exists
+#' because four test sites had hand-rolled the `{url}` substitution -- one
+#' of them commented as doing it "exactly as .register_views() does" -- and
+#' every one of them broke the moment a second token was introduced.
+#'
+#' `{long_files}` must be substituted BEFORE `{url}`: it expands to a string
+#' that itself contains the url, so the reverse order leaves the token in
+#' place and DuckDB's parser fails on the brace.
+#'
+#' `manifest` defaults to empty, which routes `.long_files_sql()` to its glob
+#' fallback -- correct for the local temp corpora the direct-execution tests
+#' build.
+#' @noRd
+#' `fixed = TRUE` is load-bearing, not a style choice.
+#'
+#' In regex mode, `gsub()` interprets backslashes in the REPLACEMENT string as
+#' escape sequences and silently drops them. A Windows corpus path is full of
+#' them, so `C:\Users\RUNNER\AppData\...` was substituted in as
+#' `C:UsersRUNNERAppData...` and every DuckDB read failed with "No files found
+#' that match the pattern". `fixed = TRUE` treats pattern and replacement as
+#' literal text, which is what a filesystem path needs.
+#'
+#' This is why the package could not read a LOCAL corpus on Windows at all --
+#' including the test fixture, hence the entire suite, and any `cog_mirror()`
+#' copy. Remote https URLs were unaffected, having no backslashes, which is
+#' part of why it stayed hidden: the bug predates the `{long_files}` token and
+#' lived in the original `{url}` substitution, unnoticed because nothing ever
+#' ran on Windows until the mirror's check matrix existed.
+#' @noRd
+.render_view_sql <- function(sql, url, manifest = list()) {
+  sql <- gsub("{long_files}", .long_files_sql(url, manifest), sql, fixed = TRUE)
+  gsub("{url}", url, sql, fixed = TRUE)
+}
+
 #' Register DuckDB views from inst/sql/ SQL files
 #' @noRd
 .register_views <- function(con, url, manifest) {
@@ -91,7 +156,7 @@
         !.corpus_has_table(manifest, .representation_view_files[[base]])) next
     if (base %in% .balance_view_files && !.corpus_has_balance_subtype(con)) next
     sql <- paste(readLines(f, warn = FALSE), collapse = "\n")
-    sql <- gsub("\\{url\\}", url, sql, fixed = FALSE)
+    sql <- .render_view_sql(sql, url, manifest)
     DBI::dbExecute(con, sql)
   }
 }

README.md

@@ -1,24 +1,127 @@
 # uscogdata
 
-Curated R reader for the Civilytics US Census of Governments finance corpus.
+<!-- badges: start -->
+[![r-universe](https://civilytics.r-universe.dev/badges/uscogdata)](https://civilytics.r-universe.dev/uscogdata)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md)
+<!-- badges: end -->
 
-Provides unit-level financial profiles, geographic rollups, and peer comparisons
+A curated R reader for the Civilytics US Census of Governments finance corpus —
-with auditable provenance and built-in cross-vintage correctness. Reads the
+every dollar that US state, county, municipal and township governments reported
-published corpus (Hive-partitioned parquet + manifest.json) directly from
+raising and spending, from **FY1967 to FY2024**, in one queryable place.
-Nextcloud via DuckDB httpfs — no local bulk downloads required.
 
-## Status
+The Census of Governments is the only nationwide source for local government
+finance, and it is hard to use: item codes change meaning across vintages,
+government identifiers were renumbered in 2017, and an absent value means
+"published zero" in one era and "not reported" in the next. This package
+handles each of those problems, and it tells you when it has — every result
+carries provenance describing what was converted, what was aggregated, and
+which known series breaks intersect your query.
 
-Under active development (Phase 2 of the cog_pipeline project). See
+**Scope:** government types 0–3 (state, county, municipality, township).
-`../cog_pipeline/docs/reader-specification.md` for the reader contract this
+56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
-package implements.
+or FY1969. Special districts (type 4) and school districts (type 5) are
+excluded pending validation.
 
-## Installation
+## Where the data comes from
+
+The corpus is published and documented at the **[US Census of Governments
+Finance API](https://pages.civilytics.org/cog-api/)**. Start there for how the
+data was built, how the identifier and item-code reconciliation works, and what
+the corpus does and does not cover.
+
+- **[API documentation and walkthroughs](https://pages.civilytics.org/cog-api/)**
+  — reference, data dictionary, and worked examples such as the
+  [Southern states guide](https://pages.civilytics.org/cog-api/cog-api-south-guide.html)
+- **[Live API](https://cog-api.civilytics.org/api/v1/)** — the same corpus over
+  HTTP, for Tableau, Python, or anything that isn't R
+- **[Bulk corpus on Hugging Face](https://huggingface.co/datasets/civilytics/us-cog-finance)**
+  — CC-BY-4.0; the same parquet files this package reads
+- **[Census Bureau source data](https://www.census.gov/programs-surveys/gov-finances.html)**
+  — the underlying public files
+
+## Install
 
 ```r
-# pak::pkg_install("gitea.civilytics.org/Civilytics/uscogdata")
+install.packages("uscogdata",
+  repos = c("https://civilytics.r-universe.dev",
+            "https://cloud.r-project.org"))
 ```
 
+Or from source:
+
+```r
+pak::pkg_install("git::https://gitea.civilytics.org/Civilytics/uscogdata.git")
+```
+
+## Quickstart
+
+No configuration, no credentials, no download. The package reads the published
+corpus over HTTPS by default.
+
+```r
+library(uscogdata)
+
+# Resolve a place name to a canonical government id
+madison <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+madison$canonical_govid
+#> [1] "552025209777"
+
+# Police spending, inflation-adjusted and per capita
+spend <- cog_spending(
+  madison$canonical_govid,
+  years          = 2012:2022,
+  category       = "Police",
+  per_capita     = TRUE,
+  adjust_to_year = 2023
+)
+
+# What did that result do to the numbers, and what should you know about them?
+cog_explain(spend)
+```
+
+`years` is required — there is no implicit full-history default.
+
+## Two ways to read the corpus
+
+| | Remote (default) | Mirrored |
+|---|---|---|
+| Setup | none | `cog_mirror(dest)`, 190.6 MB once |
+| Disk used | **0 MB** — HTTP range requests only | 190.6 MB |
+| Per query | ~4 s (one government, one year)<br>~6 s (one government, 23 years) | local speed |
+| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
+
+Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
+works out which row groups it needs, and reads only those. Nothing is cached
+between sessions either, so every query goes back to the network.
+
+The default points at a public HuggingFace mirror of the corpus. If you would
+rather not depend on a third party — for reproducibility, for an air-gapped
+environment, or on principle — **the escape hatch is one function call**:
+
+```r
+cog_mirror("~/cog-corpus")
+Sys.setenv(USCOGDATA_URL = "~/cog-corpus/")
+```
+
+After that, nothing in your analysis touches an external service.
+
+### Configuration
+
+- `USCOGDATA_URL` — corpus root: an HTTPS URL or a local path, **trailing slash required**
+- `USCOGDATA_CACHE_DIR` — where the manifest is cached (default: user cache dir)
+- `USCOGDATA_MANIFEST_TTL_SECS` — manifest re-fetch interval (default 3600)
+- `USCOGDATA_DUCKDB_THREADS` — cap DuckDB's thread count (default: every visible core)
+- `USCOGDATA_DUCKDB_MEMORY_LIMIT` — cap DuckDB's memory, e.g. `"4GB"` (default: DuckDB's own)
+
+Each also has an `options()` spelling — `uscogdata.url`, `uscogdata.duckdb_threads`,
+and so on — and the environment variable wins where both are set.
+
+The two DuckDB caps exist for **servers**, not laptops. Unset, DuckDB claims every
+core it can see, which is right for one interactive session on your own machine and
+wrong when several readers share a box: each claims the whole machine and they fight.
+Capping costs roughly 5% on a single query and is worth it anywhere the process is
+sharing hardware.
+
 ## Amounts are in full US dollars
 
 Every amount column this package returns — `amt_nominal`, `amt_real`,
@@ -29,111 +132,127 @@ own `amt` column preserves that. The verbs multiply by 1000 on the way out, so
 you never have to. The conversion is recorded in every result:
 
 ```r
-r <- cog_spending("552025209777", 2020L)
+attr(spend, "provenance")$transformations$units_conversion
-attr(r, "provenance")$transformations$units_conversion
+#> $applied     TRUE
-#> $applied TRUE  $source_unit "$1,000s (raw Census)"  $target_unit "$USD"  $multiplier 1000
+#> $source_unit "$1,000s (raw Census)"
+#> $target_unit "$USD"
+#> $multiplier  1000
 ```
 
 **Do not multiply again.** If you have read elsewhere that COG amounts are in
-`$1,000s` — true of the raw corpus, and of `cog_explorer`'s conventions doc —
+`$1,000s` — which is true of the raw Census files and of the corpus's own `amt`
-that rule does not apply to anything a `cog_*()` verb hands you. Applying it
+column — that rule does not apply to anything a `cog_*()` verb hands you.
-twice overstates every figure by 1000x, and the result looks plausible rather
+Applying it twice overstates every figure by 1000x, and the result looks
-than obviously wrong.
+plausible rather than obviously wrong.
 
-## Configuration
+## Concepts worth understanding before you publish a number
 
-- `USCOGDATA_URL` — corpus root URL (public Nextcloud share, trailing slash)
+### Primary vs Direct vs Total spending
-- `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
-- `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
-
-## Primary vs Direct vs Total spending
 
 `cog_spending(..., expenditure_concept = c("primary", "direct", "total"))`
-controls whose spending a result counts. Concepts are defined as sets of the
+controls *whose* spending a result counts. Concepts are defined as sets of the
-crosswalk's `spend_subtype` values — never item-code first letters, which
+crosswalk's `spend_subtype` values, never item-code first letters — the letter
-cannot classify correctly (the letter `Y` alone spans revenue, expenditure,
+`Y` alone spans revenue, expenditure and balance codes.
-and balance codes):
 
-- `"primary"` (the default) is the government's own service provision:
+- **`"primary"`** (default) — the government's own service provision: current
-  current operations, capital outlay, and assistance payments.
+  operations, capital outlay, assistance payments.
-- `"direct"` is Census's published Direct Expenditure: `primary` plus
+- **`"direct"`** — Census's published Direct Expenditure: `primary` plus
-  interest on debt and insurance trust benefit payments (e.g. pensions).
+  interest on debt and insurance trust benefits (e.g. pensions).
-- `"total"` additionally adds the intergovernmental leg — money handed to
+- **`"total"`** — adds the intergovernmental leg, money handed to other
-  other governments to spend (`M`/`L` codes plus `Q11`/`Q12`/`Q18` state
+  governments to spend. Meaningful for one government's own budget over time,
-  payments to school systems) — which is meaningful for describing one
+  but it double-counts when summed across governments: a state's payment to a
-  government's own budget over time, but double-counts when summed across
+  county is the same dollar the county reports as its own direct spending.
-  governments (a state's payment to a county is the same dollar the county
-  reports as its own direct spending).
 
-**Rule of thumb: any figure that spans more than one government uses
+**Rule of thumb: any figure spanning more than one government uses `primary`
-`primary` or `direct`.** `cog_geographic_rollup()` and `cog_peer_compare()`
+or `direct`.** `cog_geographic_rollup()` and `cog_peer_compare()` enforce that
-enforce this by refusing `expenditure_concept = "total"`. See
+by refusing `"total"` outright. Worked examples in
-`vignette("total-spending", package = "uscogdata")` for the full
+`vignette("total-spending", package = "uscogdata")`.
-explanation with worked examples.
 
-## General vs Total revenue
+### General vs Total revenue
 
-`cog_revenue(..., revenue_concept = c("general", "total"))` selects between
+`cog_revenue(..., revenue_concept = c("general", "total"))`:
-Census's two published revenue concepts, again defined as crosswalk
-`revenue_subtype` sets rather than item-code prefixes:
 
-- `"general"` (the default) is Census **General Revenue**: own-source
+- **`"general"`** (default) — Census General Revenue: own-source taxes,
-  (taxes, charges, miscellaneous) plus federal, state and local
+  charges and miscellaneous, plus federal, state and local aid.
-  intergovernmental aid.
+- **`"total"`** — General plus utility revenue (`A91`–`A94`), liquor store
-- `"total"` is Census **Total Revenue**: `general` plus utility revenue
+  revenue (`A90`), and insurance trust revenue.
-  (`A91`–`A94`), liquor store revenue (`A90`), and insurance trust revenue
-  (unemployment and workers' compensation `Y` codes plus the
-  employee-retirement `X` codes).
 
-The manual defines the first by subtracting the other three from the second,
+Census defines these by its own identity:
-so the two are related by Census's own identity:
 
 ```
 Total Revenue = General + Utility + Liquor Store + Insurance Trust
 ```
 
-Two things worth knowing before switching to `"total"`:
+Two things to know before switching to `"total"`. **Utility revenue is large
+for cities** — measured on the bundled fixture, utility plus liquor store is
+15.9% of city revenue, against 1.2% for states and 1.7% for counties. And the
+**employee-retirement (`X`) codes stop at FY2016**, when those systems moved to
+the separate Annual Survey of Public Pensions, so a `"total"` series steps down
+at the FY2016/FY2017 boundary for reasons of collection scope, not revenue
+(series breaks `SB197`–`SB209`).
 
-- **Utility revenue is large for cities.** Measured on the bundled fixture,
+### Reporting coverage: the Census is only sometimes a census
-  utility plus liquor store revenue is 15.9% of city (type 2) revenue, versus
-  1.2% for states and 1.7% for counties. `general` excludes it by definition.
-- **The employee-retirement (`X`) codes stop at FY2016**, when those systems
-  moved out of the annual finance file into the separate Annual Survey of
-  Public Pensions. A `"total"` series therefore steps down at the
-  FY2016/FY2017 seam for reasons of collection scope, not revenue (series
-  breaks `SB197`–`SB202`, in the corpus's `series_breaks` table).
 
-## Developer notes
+**The Census of Governments is a complete enumeration only in years ending in
+2 and 7.** Every other year is a sample, and the sample varies enormously —
+measured on the bundled fixture, Wisconsin's 608-city universe rolls up 597
+governments in FY2012 and 112 in FY2019.
 
-### Testing
+A statewide total resting on a fifth of the universe looks exactly like one
-
+resting on all of it, so every multi-government result now says which it is:
-The package ships a bundled fixture corpus at `inst/extdata/fixture_corpus/` —
-a 15 MB four-year slice (2011, 2012, 2019, 2020) of the full corpus covering
-all 50 states. `tests/testthat/setup.R` automatically points `USCOGDATA_URL`
-at this fixture, so the full test suite runs offline with no network
-dependency:
 
 ```r
-devtools::test()   # uses bundled fixture, no credentials required
+attr(rollup, "provenance")$coverage   # per-year n_units_reporting, is_census_year
 ```
 
-### Releasing against the live corpus
+`cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` take a
+`coverage` argument — `"all"` (default), `"census"` (census years only), or
+`"consistent"` (only units reporting in every requested year, a balanced
+panel).
 
-Before cutting a release, run the test suite against the published corpus to
+`n_units_reporting` is **category-conditional**, and it is not a response rate. A government that was surveyed and genuinely spends
-catch any drift between the fixture and the real data:
+nothing in the requested category is indistinguishable from one never surveyed.
+
+### Absent cells mean two different things
+
+Before FY2012, an absent cell means Census published `$0`. From FY2012 on, it
+means not reported. `cog_spending(..., complete = TRUE)` fills the requested
+grid and labels every row with which it is, via `value_source`:
+
+| `value_source` | meaning | `amt_nominal` |
+|---|---|---|
+| `reported` | the corpus carries this cell | as published |
+| `census_zero` | dense-source year (≤ FY2011), absent — Census published `$0` | `0` |
+| `not_reported` | sparse-source year (≥ FY2012), absent — unknown | `NA` |
+
+That `NA` is deliberate. Filling a modern absence with `0` would invent data.
+
+### Series breaks surface on their own
+
+Catalogued breaks that intersect your query appear in provenance whether or not
+you went looking for them — `series_break_refs` for breaks in a specific item code, and
+`corpus_break_refs` for caveats about the corpus as a whole (dollar precision
+across the 1976/1977 boundary, the FY2017 identifier change, the FY2012
+dense→sparse representation change). `cog_explain()` prints both.
+
+## How to cite
 
 ```r
-Sys.setenv(USCOGDATA_URL = "<published-corpus-url-with-trailing-slash>")
+citation("uscogdata")
-devtools::test()
 ```
 
-When the live-corpus run is clean, strip the fixture from the built package by
+The corpus itself is published under CC-BY-4.0. Cite it as:
-adding this line to `.Rbuildignore`:
 
-```
+> Civilytics Consulting. US Census of Governments finance corpus.
-^inst/extdata/fixture_corpus$
+> https://huggingface.co/datasets/civilytics/us-cog-finance
-```
 
-The test suite is URL-agnostic — `setup.R` falls back to `USCOGDATA_URL` when
+## Contributing
-the bundled fixture is absent, so no test code changes are needed for the
+
-release run or after stripping the fixture.
+Development happens on [Gitea](https://gitea.civilytics.org/Civilytics/uscogdata);
+[GitHub](https://github.com/civilytics/uscogdata) is a mirror that accepts
+issues and pull requests. See [CONTRIBUTING.md](CONTRIBUTING.md) for how a
+patch gets from there to here.
+
+## License
+
+MIT © Civilytics Consulting LLC. See [LICENSE.md](LICENSE.md).

_pkgdown.yml

@@ -1,4 +1,5 @@
-url: ~
+url: https://civilytics.r-universe.dev/uscogdata
+
 template:
   bootstrap: 5
 
@@ -15,11 +16,26 @@ reference:
       - cog_gov_search
       - cog_basket_resolution
       - cog_basket_unresolved
-  - title: Session
+  - title: Comparison & aggregation
+    desc: Peer cohorts and geographic aggregates.
     contents:
-      - has_keyword("internal")
+      - cog_find_peers
+      - cog_peer_compare
+      - cog_geographic_rollup
+  - title: Corpus metadata
+    desc: >
+      What the corpus contains, where a given result came from, and how to
+      hold a local copy of it.
+    contents:
+      - cog_categories
+      - cog_recipes
+      - cog_manifest
+      - cog_explain
+      - cog_mirror
 
 articles:
-  - title: Getting started
+  - title: Concepts
     navbar: ~
-    contents: []
+    contents:
+      - total-spending
+      - population-denominators

inst/extdata/fixture_corpus/manifest.json

@@ -1,7 +1,7 @@
 {
   "schema_version": 6,
-  "built_at": "2026-07-31T00:47:27Z",
+  "built_at": "2026-08-03T16:51:32Z",
-  "pipeline_commit": "aadb46b",
+  "pipeline_commit": "e7394a4",
   "fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated from the sparsified schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit zeros, so FY2011 absence means Census published $0 while FY2012+ absence means not reported. representation.parquet and code_set.parquet carry that rule and ship in full, as do every other metadata table in the publish tree. 2011/2012 straddle both the wide-aggregate -> modern-leaf format boundary (exercised by basis=\"harmonized\" and recipe= queries) and the dense -> sparse representation boundary (SB194); 2019/2020 retain the prior per-capita/CPI regression anchors. Regenerated via data-raw/regenerate_fixture_corpus.R.",
   "data_vintage": {
     "source_vintages": {
@@ -105,7 +105,7 @@
       },
       {
         "path": "data/series_breaks.parquet",
-        "sha256": "06dcc995ff533e57cc65fa25086cc9bf83ba592c58bf7cc99269dc2576f69944",
+        "sha256": "731998516cd802f63fcf7fb66053c7a62b7be955ab0794cad4a4979cb7628b87",
         "description": "series_breaks.parquet"
       },
       {

inst/sql/10-long.sql

@@ -1,3 +1,7 @@
 CREATE OR REPLACE VIEW long AS
 SELECT *
-FROM read_parquet('{url}data/long/**/*.parquet', hive_partitioning = true);
+-- {long_files} carries its own quoting: a bracketed list of every partition
+-- the manifest enumerates, or a single quoted glob on fallback. Do NOT wrap
+-- it in quotes. See .long_files_sql() in R/views.R for why a glob alone
+-- cannot work over HTTP.
+FROM read_parquet({long_files}, hive_partitioning = true);

man/cog_balances.Rd

@@ -5,18 +5,23 @@
 \title{Cash and security holdings for one or more governments}
 \usage{
 cog_balances(
-  govid,
+  govid = NULL,
   years,
   category = NULL,
   per_capita = FALSE,
   adjust_to_year = NULL,
   basis = c("harmonized", "raw"),
-  recipe = NULL
+  recipe = NULL,
+  state = NULL,
+  type = NULL,
+  limit = NULL,
+  offset = NULL
 )
 }
 \arguments{
 \item{govid}{Canonical govid(s): a character vector, or a data frame with a
-`canonical_govid` column (e.g. from [cog_gov_search()]).}
+`canonical_govid` column (e.g. from [cog_gov_search()]). `NULL` to name
+the cohort by `state`/`type` instead.}
 
 \item{years}{Integer vector of fiscal years.}
 
@@ -49,6 +54,38 @@ and raw space are identical for holdings. Reported in
 \item{recipe}{Optional harmonization recipe id (see [cog_recipes()]).
 `"cash_securities_z77_wide"` and `"cash_securities_z78_wide"` bridge the
 wide era to the modern one.}
+
+\item{state, type}{Name the cohort by predicate instead of by id: `state` is
+  a 2-letter USPS abbreviation (or a FIPS code) and `type` is one of
+  `"state"`, `"county"`, `"city"`, `"township"` (or the integer `0:3`) --
+  the same vocabulary, and the same internal coercion, as
+  [cog_gov_search()]. Both default to `NULL`.
+
+  The cohort is then expressed as a subquery against `canonical_fips_xwalk`
+  inside each statement rather than round-tripped through R as a literal id
+  list. For a fleet-scale cohort that is the difference between a
+  301,591-character `IN` list re-parsed in 5--8 statements per call and a
+  constant-size predicate: measured at **94 ms versus 449 ms** for the same
+  FY2022 aggregate over the 20,106-government `type = "city"` cohort, within
+  7% of the no-filter floor.
+
+  Supplying `govid` **and** `state`/`type` INTERSECTS them -- the
+  governments in `govid` that also match the predicate -- rather than one
+  silently taking precedence. Naming no cohort at all (`govid`, `state` and
+  `type` all `NULL`) aborts with class `uscogdata_no_cohort`.
+
+  When the cohort is named by predicate, `provenance$scope$govids_found`
+  and `govids_missing` are empty -- there is no id list to report against --
+  and `provenance$scope$cohort` carries `state`, `type` and
+  `n_governments` instead. A `govid`-named cohort reports exactly as before.}
+
+\item{limit}{Maximum number of result rows to return, pushed into the SQL
+rather than applied after materializing every row. `NULL` (default)
+returns everything. Cannot be combined with `recipe` -- see `offset` and
+`total_rows`.}
+
+\item{offset}{Rows to skip before `limit` starts counting (0-based).
+Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,
@@ -67,6 +104,10 @@ Tibble with columns `year`, `canonical_govid`, `gov_name`,
   and `truncated` (the observed subtypes whose coverage falls short of the
   requested years). `expenditure_concept`/`revenue_concept` are `NA` --
   holdings are a stock, not a flow, so neither concept vocabulary applies.
+
+  When `limit` is set, also carries a `total_rows` attribute: the full
+  unpaginated row count, computed by the same query (`COUNT(*) OVER()`)
+  rather than a second scan.
 }
 \description{
 Returns Census cash-and-security holdings (`category_type = "balance"`):

man/cog_explain.Rd

@@ -21,3 +21,33 @@ Prints the structured provenance attached to a tibble returned by any
 `cog_*` verb, or returns it as a list for downstream use (MCP tools,
 dashboards, JSON export).
 }
+\section{Two kinds of series break}{
+
+Catalogued breaks reach you without being asked for, in two disjoint
+fields, because a caveat about one series and a caveat about the whole
+corpus are different claims:
+
+* **`series_break_refs`** — breaks matched against the item codes actually
+  present in this result. A break in one code you queried.
+* **`corpus_break_refs`** — breaks catalogued with `fin_code = "ALL"`,
+  which are statements about the corpus rather than about any one code:
+  dollar precision across the 1976/1977 boundary (`SB085`), imputation
+  exclusion from FY2002 (`SB087`), the FY2012 dense-to-sparse
+  representation change (`SB194`), and the FY2017 government-identifier
+  change (`SB086`). These are selected on the break-year window alone.
+
+`SB194` is the one most likely to matter: a query spanning FY2011 to FY2012
+crosses the boundary where an absent cell stops meaning "Census published
+$0" and starts meaning "not reported".
+}
+
+\section{Other provenance blocks}{
+
+`transformations$units_conversion` records the `$1,000s`-to-dollars
+multiply that every amount column has already had applied.
+`transformations$per_capita` records the population denominator and its
+year range. `coverage` and `coverage_mode` appear on multi-government
+results (see [cog_geographic_rollup()]). `completion` appears when
+`complete = TRUE`. `balance_caveats` appears on [cog_balances()] results.
+}
+

man/cog_gov_search.Rd

@@ -4,7 +4,13 @@
 \alias{cog_gov_search}
 \title{Search for governments by name, state, and/or type}
 \usage{
-cog_gov_search(name = NULL, state = NULL, type = NULL)
+cog_gov_search(
+  name = NULL,
+  state = NULL,
+  type = NULL,
+  limit = NULL,
+  offset = NULL
+)
 }
 \arguments{
 \item{name}{Character vector of place name(s). Length 1 = utility mode;
@@ -19,12 +25,26 @@ all entries; otherwise must match `length(name)`.}
 in basket mode (recycles from length 1). Excluded types `4`/`5` (or
 `"special_district"` / `"school_district"`) trigger an explanatory
 message and an empty result.}
+
+\item{limit}{Maximum number of rows to return, applied in SQL. `NULL`
+(default) returns every match -- which, with no other filter, is the
+entire crosswalk. Utility mode only: pagination has no meaning in basket
+mode, where the result is one resolved row per requested name in input
+order, and is refused there with class
+`uscogdata_basket_pagination_conflict`.}
+
+\item{offset}{Rows to skip before `limit` starts counting (0-based).
+Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
 }
 \value{
 A tibble of `canonical_fips_xwalk` rows. In utility mode, all
-  matches sorted by `population_acs` desc. In basket mode, resolved
+  matches sorted by `population_acs` desc, ties broken by
-  rows in input order, with `attr(., "resolution")` set to the
+  `canonical_govid`. In basket mode, resolved rows in input order, with
-  sidecar tibble.
+  `attr(., "resolution")` set to the sidecar tibble.
+
+  When `limit` is set, carries a `total_rows` attribute: the full
+  unpaginated match count, computed by the same query (`COUNT(*) OVER()`)
+  rather than a second scan.
 }
 \description{
 Resolves human-readable place names into rows of `canonical_fips_xwalk`,

man/cog_revenue.Rd

@@ -5,7 +5,7 @@
 \title{Summarized revenue by category}
 \usage{
 cog_revenue(
-  govid,
+  govid = NULL,
   years,
   category = NULL,
   per_capita = FALSE,
@@ -15,11 +15,15 @@ cog_revenue(
   revenue_concept = c("general", "total"),
   complete = FALSE,
   limit = NULL,
-  offset = NULL
+  offset = NULL,
+  state = NULL,
+  type = NULL
 )
 }
 \arguments{
-\item{govid}{Character vector of `canonical_govid` values.}
+\item{govid}{Character vector of `canonical_govid` values, or `NULL` to name
+the cohort by `state`/`type` instead. One of `govid`, `state`, or `type`
+is required.}
 
 \item{years}{Integer vector of years.}
 
@@ -126,6 +130,30 @@ exactly as before this parameter existed. Mutually exclusive with
 
 \item{offset}{Rows to skip before `limit` starts counting (0-based).
 Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
+
+\item{state, type}{Name the cohort by predicate instead of by id: `state` is
+  a 2-letter USPS abbreviation (or a FIPS code) and `type` is one of
+  `"state"`, `"county"`, `"city"`, `"township"` (or the integer `0:3`) --
+  the same vocabulary, and the same internal coercion, as
+  [cog_gov_search()]. Both default to `NULL`.
+
+  The cohort is then expressed as a subquery against `canonical_fips_xwalk`
+  inside each statement rather than round-tripped through R as a literal id
+  list. For a fleet-scale cohort that is the difference between a
+  301,591-character `IN` list re-parsed in 5--8 statements per call and a
+  constant-size predicate: measured at **94 ms versus 449 ms** for the same
+  FY2022 aggregate over the 20,106-government `type = "city"` cohort, within
+  7% of the no-filter floor.
+
+  Supplying `govid` **and** `state`/`type` INTERSECTS them -- the
+  governments in `govid` that also match the predicate -- rather than one
+  silently taking precedence. Naming no cohort at all (`govid`, `state` and
+  `type` all `NULL`) aborts with class `uscogdata_no_cohort`.
+
+  When the cohort is named by predicate, `provenance$scope$govids_found`
+  and `govids_missing` are empty -- there is no id list to report against --
+  and `provenance$scope$cohort` carries `state`, `type` and
+  `n_governments` instead. A `govid`-named cohort reports exactly as before.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,

man/cog_spending.Rd

@@ -5,7 +5,7 @@
 \title{Summarized spending by category}
 \usage{
 cog_spending(
-  govid,
+  govid = NULL,
   years,
   category = NULL,
   per_capita = FALSE,
@@ -15,11 +15,15 @@ cog_spending(
   expenditure_concept = c("primary", "direct", "total"),
   complete = FALSE,
   limit = NULL,
-  offset = NULL
+  offset = NULL,
+  state = NULL,
+  type = NULL
 )
 }
 \arguments{
-\item{govid}{Character vector of `canonical_govid` values.}
+\item{govid}{Character vector of `canonical_govid` values, or `NULL` to name
+the cohort by `state`/`type` instead. One of `govid`, `state`, or `type`
+is required.}
 
 \item{years}{Integer vector of years.}
 
@@ -146,6 +150,30 @@ exactly as before this parameter existed. Mutually exclusive with
 
 \item{offset}{Rows to skip before `limit` starts counting (0-based).
 Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
+
+\item{state, type}{Name the cohort by predicate instead of by id: `state` is
+  a 2-letter USPS abbreviation (or a FIPS code) and `type` is one of
+  `"state"`, `"county"`, `"city"`, `"township"` (or the integer `0:3`) --
+  the same vocabulary, and the same internal coercion, as
+  [cog_gov_search()]. Both default to `NULL`.
+
+  The cohort is then expressed as a subquery against `canonical_fips_xwalk`
+  inside each statement rather than round-tripped through R as a literal id
+  list. For a fleet-scale cohort that is the difference between a
+  301,591-character `IN` list re-parsed in 5--8 statements per call and a
+  constant-size predicate: measured at **94 ms versus 449 ms** for the same
+  FY2022 aggregate over the 20,106-government `type = "city"` cohort, within
+  7% of the no-filter floor.
+
+  Supplying `govid` **and** `state`/`type` INTERSECTS them -- the
+  governments in `govid` that also match the predicate -- rather than one
+  silently taking precedence. Naming no cohort at all (`govid`, `state` and
+  `type` all `NULL`) aborts with class `uscogdata_no_cohort`.
+
+  When the cohort is named by predicate, `provenance$scope$govids_found`
+  and `govids_missing` are empty -- there is no id list to report against --
+  and `provenance$scope$cohort` carries `state`, `type` and
+  `n_governments` instead. A `govid`-named cohort reports exactly as before.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,

specs/2026-08-08-public-release-design.md

@@ -0,0 +1,296 @@
+# `uscogdata` 0.3.0 — public release
+
+**Date:** 2026-08-08 · **Status:** design, awaiting approval
+**Scope:** release-readiness, README, NEWS. Distribution mechanics recorded here as
+decided, sequenced after the package is clean.
+
+`uscogdata` is feature-complete and the corpus it reads has been public on
+HuggingFace since 2026-08-07 (294 downloads as of this writing). The API built on
+it is live. What does not exist is a public *package*: the repo is private, there
+is no install path, and — measured, not assumed — **a stranger who installed it
+today could not read the corpus at all.**
+
+This spec covers making that untrue.
+
+## Decisions locked
+
+| Decision | Choice |
+|---|---|
+| Canonical source | `gitea.civilytics.org/Civilytics/uscogdata`, flipped public |
+| Public mirror | `github.com/civilytics/uscogdata` — issues, PRs, multi-OS check, CDN |
+| Mirror mechanism | Gitea Actions non-force `git push` (not a push mirror) |
+| Binaries | `civilytics.r-universe.dev`, registry pinned to a release tag |
+| Author of record | Jared E. Knowles `<jared@civilytics.com>`, ORCID `0000-0003-0005-9478` |
+| Copyright | Civilytics Consulting LLC (`cph`, `fnd`) |
+| License | MIT (package) · CC-BY-4.0 (corpus) |
+| Corrections intake | Deferred — see *Out of scope* |
+| Other packages | Parked until this one walks the path end to end |
+
+## P0 — the corpus is unreachable
+
+Two independent faults, either of which alone is fatal.
+
+**No corpus URL exists.** `R/config.R` defaults to the literal
+`REPLACE_WITH_SHARE_TOKEN` sentinel, and no file in the repo supplies a working
+one. A new user calling any verb gets `uscogdata_url_not_configured` with no path
+to resolution.
+
+**Remote reads are broken regardless.** Every partitioned view globs:
+
+```sql
+FROM read_parquet('{url}data/long/**/*.parquet', hive_partitioning = true)
+```
+
+DuckDB 1.5.5 refuses globs over generic HTTP. Its suggested
+`allow_asterisks_in_http_paths` escape hatch does not help — it forwards the
+literal `**/*` as a filename and 404s, because plain HTTP exposes no directory
+listing to expand against.
+
+The package therefore works only against a **local path**. That is how the API
+runs it (`CORPUS_HOST_PATH` is a host mount on maxwell) and how the tests run
+(bundled fixture), which is why the fault went unnoticed. The README's headline
+claim — *"Reads the published corpus directly from Nextcloud via DuckDB httpfs —
+no local bulk downloads required"* — is currently false.
+
+### Fix: enumerate from the manifest, do not glob
+
+`manifest.json` already lists every partition under `files.long_partitions[]`
+with `path`, `year`, `sha256`, `row_count` and `size_bytes` — 56 of them.
+Substituting an explicit file list for the glob was measured against the
+published corpus on 2026-08-08:
+
+| Path | Result |
+|---|---|
+| `https://…/data/long/**/*.parquet` (default) | error — globs unsupported over HTTP |
+| same, `allow_asterisks_in_http_paths = true` | error — literal `**/*` 404s |
+| `hf://datasets/civilytics/us-cog-finance/…` glob | 46,148,034 rows |
+| **explicit list over plain https** | **46,148,034 rows** |
+
+`hive_partitioning = true` still recovers `year` from the paths under
+enumeration, so no downstream view or verb changes.
+
+Enumeration is preferred over `hf://` deliberately. It is **host-agnostic** —
+Nextcloud, HuggingFace, or any static server take the same code path — where
+`hf://` would tie the default to one vendor's protocol and still need
+special-casing, since manifest fetching goes through `httr2`, which cannot speak
+`hf://`. Enumeration also *removes* a dependency (globbing) rather than adding
+one, and the manifest's per-file `sha256` becomes available for integrity
+checking later.
+
+Views are registered from `inst/sql/` with `{url}` substitution in
+`R/views.R:.register_views()`. The list must be built once per session from the
+already-fetched manifest and substituted the same way, so the change is confined
+to view registration and does not touch verb code.
+
+### Fix: ship a working default
+
+`R/config.R`'s default becomes the public HuggingFace `resolve/main/` URL:
+CC-BY-4.0, no token to publish, CDN-backed, and it keeps maxwell's uplink out of
+the path — the same reasoning behind the GitHub mirror and r-universe.
+
+This means `library(uscogdata)` followed by a verb works with **zero
+configuration**, which is what makes the package demonstrable in a README and
+later in a post. `USCOGDATA_URL` and `options(uscogdata.url=)` continue to
+override, so the Nextcloud copy and local mirrors are unaffected.
+
+The `uscogdata_url_not_configured` error class stays — it still fires for an
+explicitly-set empty or placeholder URL — but ceases to be the default
+experience.
+
+### Consequence: `cog_mirror()` is promoted
+
+Measured cost of the remote default, from efron on a good connection:
+
+| | |
+|---|---|
+| Whole corpus | **190.6 MB**, 56 partitions, 46,148,034 rows, FY1967–FY2024 |
+| One government, one year | 1.5 s |
+| One government, all 56 years | 2.8 s |
+| Disk written | **0.00 MB** — range requests only; `external_file_cache` is in-memory |
+
+Nothing persists locally beyond the shared `httpfs` extension in `~/.duckdb` (a
+few MB, once per machine, across all DuckDB use). Costs are RAM and per-query
+bandwidth, since nothing caches between sessions.
+
+Those timings are raw scans. Real verbs additionally join crosswalks, resolve
+categories and assemble provenance, so end-to-end verb latency will be higher and
+**must be re-measured once the fix lands** — it cannot be measured today.
+
+The corpus being only 190.6 MB makes `cog_mirror()` a first-class option rather
+than a developer footnote. The README presents **both paths**:
+
+- **Remote (default, zero setup)** — trying it out, teaching, one-off questions.
+- **Mirrored (`cog_mirror()`, 190 MB once)** — repeated or heavy analysis,
+  offline work, reproducibility, or preferring not to depend on HuggingFace.
+
+The second is also the honest answer to the vendor-dependency question raised by
+defaulting to HuggingFace: **the escape hatch is one function call and 190 MB**,
+after which no analysis touches an external service. The README says so
+explicitly. That is the difference between a convenience default and lock-in.
+
+## Release-readiness fixes
+
+| # | Issue | Fix |
+|---|---|---|
+| 1 | `MaxCorpusSchema: 5` in DESCRIPTION; `.validate_schema()` accepts `4,5,6,7`; published corpus is **7** | `MaxCorpusSchema: 7` |
+| 2 | `^vignettes$` in `.Rbuildignore` — both vignettes absent from the installed package, while README tells users to run `vignette("total-spending")` | Remove `^vignettes$`, `^doc$`, `^Meta$`. Both vignettes build offline (`total-spending` reads the bundled fixture; `population-denominators` is `eval = FALSE`) |
+| 3 | `_pkgdown.yml` reference index covers 6 of 14 exports — pkgdown errors on missing topics | Add `cog_categories`, `cog_explain`, `cog_find_peers`, `cog_geographic_rollup`, `cog_manifest`, `cog_mirror`, `cog_peer_compare`, `cog_recipes`; set `url:` |
+| 4 | No `URL:` / `BugReports:` in DESCRIPTION | Add both, pointing at the GitHub mirror |
+| 5 | No `LICENSE.md`; `LICENSE` holder reads `Civilytics` | `usethis::use_mit_license("Civilytics Consulting LLC")` |
+| 6 | README instructs stripping the fixture at release | Delete that section — see below |
+| 7 | `Authors@R` is an org with no human | Jared E. Knowles `aut`/`cre` + ORCID; Civilytics Consulting LLC `cph`/`fnd` |
+
+**On #6.** The advice to add `^inst/extdata/fixture_corpus$` to `.Rbuildignore`
+is CRAN-sized thinking (5 MB limit) and this package is not going to CRAN.
+Stripping the 15 MB fixture would break `total-spending.Rmd`, which reads from
+it, and would leave r-universe and GitHub Actions unable to run the 28 test files
+without a corpus credential. **The fixture is what lets `R CMD check` pass
+anywhere with zero secrets** — precisely what public CI needs. It ships.
+
+## README
+
+The current README addresses someone standing inside the repo tree: status reads
+"Under active development (Phase 2 of the cog_pipeline project)", it points at
+`../cog_pipeline/docs/reader-specification.md`, the install line is commented
+out, and developer, testing and release sections sit above anything a user needs.
+
+Restructured around a stranger, in this order:
+
+1. **What this is** — one paragraph, and what the corpus covers (types 0–3,
+   FY1967–FY2024, 46M rows, 190.6 MB).
+2. **Install** — r-universe first (binaries), git second.
+3. **Quickstart that actually runs** — resolve a government, get its history,
+   print provenance. No configuration step.
+4. **Two ways to read the corpus** — remote default vs `cog_mirror()`, with the
+   measured numbers and the independence note.
+5. **Amounts are in full US dollars** — kept near the top. This is the errata
+   most likely to produce a wrong answer that looks plausible.
+6. **Concepts** — primary/direct/total spending, general/total revenue,
+   coverage. Condensed, linking to the vignettes for the full treatment.
+7. **How to cite** — `citation("uscogdata")`, corpus CC-BY-4.0 attribution.
+8. **Contributing** — canonical-on-Gitea PR flow.
+
+Developer notes, testing instructions and release procedure move to
+`CONTRIBUTING.md`. Every path reference to a sibling repo is removed or replaced
+with a URL that resolves for someone who has only this repo.
+
+## NEWS.md
+
+`NEWS.md` currently holds two sections. `0.2.0` is a legitimate changelog — the
+`"All Categories"` reserved value, the coverage-signposting fix, the
+`n_units_reporting` documentation — and it stays. Beneath it,
+`0.1.0 (development)` is a pre-release churn log: changes described relative to
+states no user has ever seen ("Breaking: corpus schema_version 4", "the package
+now requires…"), spanning the package's entire pre-release development. To a
+newcomer deciding whether to depend on this, that section reads as instability.
+
+**A new `0.3.0` section is added at the top, framed as the first public
+release**: what the package does, what the corpus covers, and the caveats that
+are genuinely load-bearing. **`0.2.0` is kept verbatim.** **`0.1.0 (development)`
+is dropped** — that history stays in git, where it belongs.
+
+The version is `0.3.0` rather than `0.2.0` because this release changes
+user-visible behaviour: remote corpus reads go from broken to working, and the
+default URL from a dead placeholder to a live corpus. It is also not `1.0.0` —
+the corpus still excludes government types 4 and 5 pending validation, so a
+stability promise would overclaim. No git tag exists for any prior version;
+`chore: release 0.2.0` bumped `DESCRIPTION` and `NEWS` only.
+
+The substantive content is migrated, not deleted. These are hard-won and belong
+in documentation rather than buried in a changelog:
+
+| Content | Destination |
+|---|---|
+| Coverage disclosure on multi-government aggregates (census vs sample years) | README concepts + `cog_geographic_rollup()` docs |
+| `complete = TRUE` three-way absence semantics (`reported` / `census_zero` / `not_reported`) | `cog_spending()` / `cog_revenue()` docs |
+| Series-break and corpus-break surfacing | README + `cog_explain()` docs |
+| $1,000s → full dollars conversion | README, already prominent |
+| Per-year F-33 population denominators | `population-denominators` vignette, already there |
+
+This also makes NEWS reusable as raw material for the release announcement,
+which is the stated downstream purpose.
+
+## Distribution mechanics
+
+Recorded as decided; executed after the package is clean and checks are green.
+
+**Sequence matters.** r-universe publishes check results the moment a package is
+registered. Registering before the fixes above land means a red badge on day one,
+which is a worse first impression than a week's delay.
+
+1. `gitleaks` over full history. A coarse grep found nothing across 140 commits
+   and the default corpus URL is still the placeholder sentinel, but a proper
+   scan is the gate on an irreversible action.
+2. Flip the Gitea repo public. Disable Gitea issues on it, so there is exactly
+   one inbox.
+3. Create `github.com/civilytics/uscogdata`. Add `.github/workflows/` for the
+   Windows/macOS/Linux `R CMD check` matrix — the platforms the Gitea runner
+   cannot provide, and which this package has never been tested on despite
+   depending on duckdb and httr2. Gitea reads `.gitea/workflows`, GitHub reads
+   `.github/workflows`; both live in one tree without colliding.
+4. Gitea Actions workflow pushing to GitHub **without `--force`**, so divergence
+   fails loudly in CI rather than silently overwriting.
+5. Add `jared@civilytics.com` as a verified secondary email on the GitHub
+   account — r-universe links maintainer identity by matching DESCRIPTION's email
+   against registered GitHub emails, and the association only takes effect on the
+   next build.
+6. Tag `v0.3.0`. Create `github.com/civilytics/civilytics.r-universe.dev` with a
+   `packages.json` pinned to the tag, pointing at the GitHub mirror rather than
+   Gitea so clone traffic stays off maxwell. Install the r-universe app.
+
+### PR flow
+
+Never press Merge on GitHub. A merge there is overwritten by the next sync, the
+PR still displays "Merged", and nothing says otherwise.
+
+```sh
+git remote add github https://github.com/civilytics/uscogdata.git
+git config --add remote.github.fetch '+refs/pull/*/head:refs/remotes/github/pr/*'
+git fetch github
+git switch -c pr-42 github/pr/42      # test
+git switch main && git merge --no-ff pr-42
+git push origin main                   # Gitea -> mirror -> GitHub
+```
+
+GitHub auto-closes a PR as merged once its head commit becomes an ancestor of the
+base branch, so `--no-ff` — which preserves the contributor's SHAs — makes the PR
+close itself when the mirror pushes. **For external PRs, merge; do not squash or
+rebase.** Squashing rewrites the SHAs, the auto-close never fires, and closing by
+hand reads to a first-time contributor as rejection.
+
+`CONTRIBUTING.md` states this, and a GitHub Action comments it on incoming PRs.
+No CLA; no DCO.
+
+## Verification
+
+The release is not done until all of these pass:
+
+1. `R CMD check --as-cran` clean on Linux, and on Windows and macOS via the
+   GitHub matrix. This package has never been checked on the latter two.
+2. Full test suite (28 files) green against the **bundled fixture**, offline,
+   with no credentials — the property public CI depends on.
+3. Full test suite green against the **live corpus**, which additionally
+   exercises the enumeration fix that the fixture's local path cannot.
+4. `pkgdown::build_site()` completes.
+5. Both vignettes present in the built tarball and
+   `vignette("total-spending", package = "uscogdata")` resolves from an
+   installed copy.
+6. **Cold-start check on a machine that has never seen this package:** install
+   from r-universe, `library(uscogdata)`, run the README quickstart verbatim with
+   no environment variables set. This is the only test that catches the P0 class
+   of fault, and its absence is why the fault survived.
+7. End-to-end verb latency re-measured against the live corpus and the README's
+   numbers updated if they moved.
+
+## Out of scope
+
+- **Corrections intake.** Deferred by decision. Consequence: the release cannot
+  invite data-error reports or make the "traceable and correctable" claim that
+  most distinguishes this corpus from Census's own files. `BugReports:` points at
+  package issues only. A verified correction should eventually terminate as a
+  `lineage_event` or `series_break` row so it propagates through provenance to
+  every consumer — that design is unstarted.
+- **Announcement posts.** Deferred. The API announcement is gated on corrections
+  landing and merits a Civic Pulse edition.
+- **The rest of the R package backlog.** Parked until this one completes the path.
+- **`cog_pipeline` publication.** Stays private.

tests/testthat/test-all-categories.R

@@ -4,7 +4,7 @@ test_that(".build_verb_sql emits a literal category and no category filter in al
   sql <- uscogdata:::.build_verb_sql(
     view         = "spending_annotated",
     subtype_col  = "spend_subtype",
-    govid        = "552025209777",
+    cohort = uscogdata:::.make_cohort("552025209777"),
     years        = 2019L,
     category     = NULL,
     subtype_scope = c("operations", "capital"),
@@ -24,7 +24,7 @@ test_that(".build_verb_sql emits a literal category and no category filter in al
 test_that(".build_verb_sql is unchanged when all_categories is FALSE", {
   args <- list(
     view = "spending_annotated", subtype_col = "spend_subtype",
-    govid = "552025209777", years = 2019L, category = NULL,
+    cohort = uscogdata:::.make_cohort("552025209777"), years = 2019L, category = NULL,
     subtype_scope = c("operations", "capital")
   )
   old <- do.call(uscogdata:::.build_verb_sql, args)
@@ -244,7 +244,7 @@ test_that('"All Categories" candidate scoping is symmetric with .build_verb_sql(
   con <- uscogdata:::.ensure_session()
 
   none <- uscogdata:::.build_suggestions(
-    con, govid = "010000226085", years = 2011L,
+    con, cohort = uscogdata:::.make_cohort("010000226085"), years = 2011L,
     category = "All Categories", result = NULL, basis = "harmonized",
     flow_prefixes = c("E", "F", "G"),
     long_view = "spending_long_harmonized",
@@ -255,7 +255,7 @@ test_that('"All Categories" candidate scoping is symmetric with .build_verb_sql(
   expect_length(none, 0L)
 
   scoped <- uscogdata:::.build_suggestions(
-    con, govid = "010000226085", years = 2011L,
+    con, cohort = uscogdata:::.make_cohort("010000226085"), years = 2011L,
     category = "All Categories", result = NULL, basis = "harmonized",
     flow_prefixes = c("E", "F", "G"),
     long_view = "spending_long_harmonized",

tests/testthat/test-balances.R

@@ -66,7 +66,7 @@ test_that("inst/sql/26-balance_long.sql enforces NOT is_aggregate (real SQL text
   sql_dir <- system.file("sql", package = "uscogdata")
   .read_view_sql <- function(filename) {
     txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
   }
 
   con <- DBI::dbConnect(duckdb::duckdb())

tests/testthat/test-cohort-verbs.R

@@ -0,0 +1,218 @@
+# The money verbs accepting a cohort by predicate (uscogdata#58).
+#
+# The load-bearing property is EQUIVALENCE: naming the same set of governments
+# by id and by state/type must return the same rows. Everything else here is
+# about the ways that equivalence could silently break -- the postal/FIPS
+# translation, the intersection rule, and provenance no longer having an id
+# list to describe.
+#
+# Fixture cohorts used: RI (fips 44) cities = 8 governments, DE (fips 10)
+# counties = 3. Small on purpose; the size of the win is measured against the
+# production corpus, not here.
+
+# --- equivalence -----------------------------------------------------------
+
+test_that("a predicate cohort returns exactly what the same ids return", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    ids <- cog_gov_search(NULL, state = "RI", type = "city")$canonical_govid
+    expect_gt(length(ids), 1L)
+
+    by_id   <- cog_spending(govid = ids, years = 2019)
+    by_pred <- cog_spending(years = 2019, state = "RI", type = "city")
+
+    # Compare the data itself, ignoring the provenance attribute -- which is
+    # SUPPOSED to differ (see the scope tests below).
+    expect_equal(
+      as.data.frame(by_id[order(by_id$canonical_govid, by_id$category), ]),
+      as.data.frame(by_pred[order(by_pred$canonical_govid, by_pred$category), ]),
+      ignore_attr = TRUE
+    )
+    expect_gt(nrow(by_pred), 0L)
+  })
+})
+
+test_that("equivalence holds for cog_revenue()", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    ids <- cog_gov_search(NULL, state = "DE", type = "county")$canonical_govid
+    by_id   <- cog_revenue(govid = ids, years = 2019)
+    by_pred <- cog_revenue(years = 2019, state = "DE", type = "county")
+    expect_equal(nrow(by_id), nrow(by_pred))
+    expect_equal(sum(by_id$amt_nominal), sum(by_pred$amt_nominal))
+  })
+})
+
+test_that("equivalence holds for cog_balances()", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    ids <- cog_gov_search(NULL, state = "DE", type = "county")$canonical_govid
+    by_id   <- cog_balances(govid = ids, years = 2019)
+    by_pred <- cog_balances(years = 2019, state = "DE", type = "county")
+    expect_equal(nrow(by_id), nrow(by_pred))
+    expect_equal(sum(by_id$amt_nominal), sum(by_pred$amt_nominal))
+  })
+})
+
+test_that("equivalence survives per_capita, adjust_to_year and pagination", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    ids <- cog_gov_search(NULL, state = "RI", type = "city")$canonical_govid
+
+    # per_capita now keys its population lookup on the rows in the result
+    # rather than on the requested cohort; these must stay identical.
+    by_id   <- cog_spending(govid = ids, years = 2019, per_capita = TRUE,
+                            adjust_to_year = 2020)
+    by_pred <- cog_spending(years = 2019, state = "RI", type = "city",
+                            per_capita = TRUE, adjust_to_year = 2020)
+    expect_equal(by_id$amt_per_capita_nominal, by_pred$amt_per_capita_nominal)
+    expect_equal(by_id$amt_per_capita_real, by_pred$amt_per_capita_real)
+    expect_equal(by_id$pop_source, by_pred$pop_source)
+
+    paged_id   <- cog_spending(govid = ids, years = 2019, limit = 5, offset = 5)
+    paged_pred <- cog_spending(years = 2019, state = "RI", type = "city",
+                               limit = 5, offset = 5)
+    expect_equal(as.data.frame(paged_id), as.data.frame(paged_pred),
+                 ignore_attr = TRUE)
+    expect_identical(attr(paged_id, "total_rows"), attr(paged_pred, "total_rows"))
+  })
+})
+
+test_that("a state-only predicate spans every type in that state", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    ids <- cog_gov_search(NULL, state = "DE", type = NULL)$canonical_govid
+    by_id   <- cog_spending(govid = ids, years = 2019)
+    by_pred <- cog_spending(years = 2019, state = "DE")
+    expect_equal(nrow(by_id), nrow(by_pred))
+  })
+})
+
+# --- the postal/FIPS trap --------------------------------------------------
+
+test_that("a postal abbreviation resolves to rows, not to silence", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    # canonical_fips_xwalk.fips_state holds "44", not "RI". A predicate built
+    # from the raw parameter matches nothing and returns an empty result that
+    # reads as "these governments reported nothing" -- the exact trap cog-api
+    # hit. A zero-row result here is the regression.
+    r <- cog_spending(years = 2019, state = "RI", type = "city")
+    expect_gt(nrow(r), 0L)
+
+    # And the FIPS form is accepted as the same cohort.
+    expect_equal(nrow(cog_spending(years = 2019, state = "44", type = "city")),
+                 nrow(r))
+  })
+})
+
+test_that("an unknown state abbreviation aborts with a message that names the problem", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    # Regression: `.state_abbrev_to_fips` is a named character vector, so `[[`
+    # on an absent name threw base R's "subscript out of bounds" and the
+    # curated message was unreachable.
+    expect_error(cog_spending(years = 2019, state = "ZZ"),
+                 class = "uscogdata_unknown_state")
+    expect_error(cog_spending(years = 2019, state = "ZZ"),
+                 "Unknown state abbreviation")
+  })
+})
+
+# --- naming the cohort -----------------------------------------------------
+
+test_that("naming no cohort at all is refused", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    expect_error(cog_spending(years = 2019), class = "uscogdata_no_cohort")
+    expect_error(cog_revenue(years = 2019), class = "uscogdata_no_cohort")
+    expect_error(cog_balances(years = 2019), class = "uscogdata_no_cohort")
+  })
+})
+
+test_that("an empty govid vector still fails as it always did", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    # Supplied-but-empty is a caller error, not "cohort named some other way".
+    expect_error(cog_spending(character(0), 2019),
+                 "must be a non-empty character vector")
+  })
+})
+
+test_that("govid and state/type together intersect", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    cities   <- cog_gov_search(NULL, state = "RI", type = "city")$canonical_govid
+    counties <- cog_gov_search(NULL, state = "DE", type = "county")$canonical_govid
+
+    # The documented rule: the governments in `govid` that ALSO match the
+    # predicate -- never one silently taking precedence over the other.
+    both <- cog_spending(govid = c(cities, counties), years = 2019,
+                         state = "RI", type = "city")
+    only <- cog_spending(govid = cities, years = 2019)
+    expect_equal(as.data.frame(both), as.data.frame(only), ignore_attr = TRUE)
+
+    # A disjoint intersection is empty, not "whichever one won".
+    none <- cog_spending(govid = counties, years = 2019,
+                         state = "RI", type = "city")
+    expect_equal(nrow(none), 0L)
+  })
+})
+
+# --- provenance ------------------------------------------------------------
+
+test_that("a govid cohort's provenance scope is untouched", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    ids <- cog_gov_search(NULL, state = "DE", type = "county")$canonical_govid
+    prov <- attr(cog_spending(govid = ids, years = 2019), "provenance")
+    expect_setequal(prov$scope$govids_found, ids)
+    expect_length(prov$scope$govids_missing, 0L)
+    # No cohort block: govids_found already describes this cohort exactly.
+    expect_null(prov$scope$cohort)
+  })
+})
+
+test_that("a predicate cohort describes itself instead of listing ids", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    prov <- attr(cog_spending(years = 2019, state = "DE", type = "county"),
+                 "provenance")
+    # Deliberately NOT the resolved id list: a fleet-scale cohort would put
+    # 20,000 govids into every response body.
+    expect_length(prov$scope$govids_found, 0L)
+    expect_length(prov$scope$govids_missing, 0L)
+    expect_identical(prov$scope$cohort$state, "DE")
+    expect_identical(prov$scope$cohort$type, "county")
+    expect_identical(prov$scope$cohort$n_governments, 3L)
+  })
+})
+
+test_that("the cohort block counts the intersection, not the predicate alone", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    counties <- cog_gov_search(NULL, state = "DE", type = "county")$canonical_govid
+    prov <- attr(
+      cog_spending(govid = counties[1], years = 2019, state = "DE", type = "county"),
+      "provenance"
+    )
+    expect_identical(prov$scope$cohort$n_governments, 1L)
+  })
+})
+
+# --- the SQL actually changed ----------------------------------------------
+
+test_that("a predicate cohort never renders the ids into the query", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    ids <- cog_gov_search(NULL, state = "RI", type = "city")$canonical_govid
+    prov <- attr(cog_spending(years = 2019, state = "RI", type = "city"),
+                 "provenance")
+    sql <- prov$sql_query %||% prov$sql
+    skip_if(is.null(sql), "provenance carries no SQL for this verb")
+    # The whole point: cohort size does not enter the SQL string.
+    for (id in ids) expect_false(grepl(id, sql, fixed = TRUE))
+    expect_match(sql, "SELECT canonical_govid FROM canonical_fips_xwalk",
+                 fixed = TRUE)
+  })
+})

tests/testthat/test-cohort.R

@@ -0,0 +1,83 @@
+# A cohort is how every verb names the set of governments it queries. It can be
+# named by explicit id, by a predicate over canonical_fips_xwalk, or by both
+# (intersection). These tests cover the SQL construction itself -- pure string
+# building, no corpus needed -- because that is where the postal/FIPS trap and
+# the 40k-literal blowup both live.
+
+test_that(".make_cohort() keeps an explicit id vector as ids", {
+  ch <- .make_cohort(govid = c("550000227544", "060000000001"))
+  expect_identical(ch$ids, c("550000227544", "060000000001"))
+  expect_null(ch$state_fips)
+  expect_null(ch$type_int)
+  expect_false(.cohort_by_predicate(ch))
+})
+
+test_that(".make_cohort() translates a postal abbreviation to FIPS", {
+  # The trap this whole issue exists to avoid: canonical_fips_xwalk.fips_state
+  # holds "55", not "WI". A predicate written against the raw parameter matches
+  # nothing and returns an empty result that reads as "reported nothing".
+  ch <- .make_cohort(state = "WI")
+  expect_identical(ch$state_fips, "55")
+  expect_true(.cohort_by_predicate(ch))
+})
+
+test_that(".make_cohort() translates a type label to its integer code", {
+  expect_identical(.make_cohort(type = "city")$type_int, 2L)
+  expect_identical(.make_cohort(type = "state")$type_int, 0L)
+  expect_identical(.make_cohort(type = 1)$type_int, 1L)
+})
+
+test_that(".make_cohort() reuses the search verb's coercers for invalid input", {
+  expect_error(.make_cohort(state = "ZZ"), "Unknown state abbreviation")
+  expect_error(.make_cohort(type = "special_district"), "Unknown type")
+  # Out-of-scope types (4 = special district, 5 = school district) are refused
+  # by the numeric branch, with the v0.1-scope message.
+  expect_error(.make_cohort(type = 4), "type must be 0, 1, 2, or 3")
+})
+
+test_that(".make_cohort() rejects naming no cohort at all", {
+  expect_error(.make_cohort(), class = "uscogdata_no_cohort")
+})
+
+test_that(".cohort_sql() renders an id cohort as a literal IN list", {
+  sql <- .cohort_sql(.make_cohort(govid = c("a", "b")))
+  expect_identical(sql, "canonical_govid IN ('a','b')")
+})
+
+test_that(".cohort_sql() renders a predicate cohort as an xwalk subquery", {
+  # The point of the issue: the cohort never becomes a literal list, so its
+  # size does not enter the SQL string at all.
+  sql <- .cohort_sql(.make_cohort(state = "WI", type = "city"))
+  expect_match(sql, "SELECT canonical_govid FROM canonical_fips_xwalk", fixed = TRUE)
+  expect_match(sql, "fips_state = '55'", fixed = TRUE)
+  expect_match(sql, "govs_type = 2", fixed = TRUE)
+  expect_false(grepl("'WI'", sql, fixed = TRUE))
+})
+
+test_that(".cohort_sql() renders ids and a predicate as an intersection", {
+  sql <- .cohort_sql(.make_cohort(govid = c("a", "b"), type = "city"))
+  expect_match(sql, "canonical_govid IN ('a','b')", fixed = TRUE)
+  expect_match(sql, "AND canonical_govid IN (SELECT", fixed = TRUE)
+})
+
+test_that(".cohort_sql() honours a column alias", {
+  # Several call sites join the xwalk under an alias (`l.`, `x.`, `v.`), so the
+  # predicate has to be able to name the qualified column.
+  sql <- .cohort_sql(.make_cohort(state = "WI"), col = "l.canonical_govid")
+  expect_match(sql, "l.canonical_govid IN (SELECT", fixed = TRUE)
+  # The subquery's own column stays unqualified -- it selects from the xwalk,
+  # not from the outer relation.
+  expect_match(sql, "SELECT canonical_govid FROM", fixed = TRUE)
+})
+
+test_that(".cohort_sql() escapes quotes in ids", {
+  sql <- .cohort_sql(.make_cohort(govid = "o'brien"))
+  expect_match(sql, "'o''brien'", fixed = TRUE)
+})
+
+test_that("a predicate cohort's SQL does not grow with cohort size", {
+  # The regression this guards: 20,106 ids rendered to a 301,591-character
+  # IN list, embedded in 5-8 statements per call.
+  wide <- .cohort_sql(.make_cohort(state = "CA", type = "city"))
+  expect_lt(nchar(wide), 200L)
+})

tests/testthat/test-config.R

@@ -64,3 +64,109 @@ test_that(".resolve_url does not invent a slash for an empty setting", {
   withr::local_options(uscogdata.url = "")
   expect_equal(.resolve_url(), "")
 })
+
+test_that("the default corpus URL is real, not a placeholder", {
+  # setup.R points USCOGDATA_URL at the bundled fixture for the whole suite,
+  # so both the env var and the option have to be cleared to see the default.
+  withr::local_envvar(USCOGDATA_URL = NA)
+  withr::local_options(uscogdata.url = NULL)
+  url <- .resolve_url()
+  expect_false(grepl("REPLACE_WITH", url, fixed = TRUE))
+  expect_match(url, "^https://")
+  expect_match(url, "/$")
+})
+
+test_that("an explicitly-set sentinel URL still aborts", {
+  # The guard must survive the default change: a user who half-edited a
+  # copied config still gets the actionable error.
+  withr::local_envvar(
+    USCOGDATA_URL = "https://other.example/s/REPLACE_WITH_SHARE_TOKEN/x/"
+  )
+  expect_error(
+    .check_url_configured(.resolve_url()),
+    class = "uscogdata_url_not_configured"
+  )
+})
+
+test_that("DESCRIPTION carries release metadata", {
+  skip_if_no_source_tree("DESCRIPTION")
+  d <- read.dcf(source_tree_path("DESCRIPTION"))
+  fields <- colnames(d)
+
+  expect_true(all(c("URL", "BugReports") %in% fields))
+  expect_match(d[1, "Authors@R"], "Knowles", fixed = TRUE)
+  expect_match(d[1, "Authors@R"], "0000-0003-0005-9478", fixed = TRUE)
+  expect_match(d[1, "Authors@R"], "Civilytics Consulting LLC", fixed = TRUE)
+
+  # The gate in .validate_schema() accepts up to 7 and the published corpus
+  # IS 7; DESCRIPTION must not claim otherwise.
+  expect_equal(as.integer(d[1, "MaxCorpusSchema"]), 7L)
+
+  # Authors@R must actually parse -- a malformed person() call is only
+  # caught at citation()/build time otherwise.
+  people <- eval(parse(text = d[1, "Authors@R"]))
+  expect_s3_class(people, "person")
+  expect_true("cre" %in% unlist(lapply(people, function(p) p$role)))
+})
+
+test_that("LICENSE and LICENSE.md name the same copyright holder", {
+  skip_if_no_source_tree("LICENSE", "LICENSE.md")
+  holder <- sub("^COPYRIGHT HOLDER:\\s*", "",
+                grep("^COPYRIGHT HOLDER:", readLines(source_tree_path("LICENSE"),
+                                                     warn = FALSE), value = TRUE))
+  full <- paste(readLines(source_tree_path("LICENSE.md"), warn = FALSE), collapse = "\n")
+
+  expect_equal(holder, "Civilytics Consulting LLC")
+  expect_match(full, holder, fixed = TRUE)
+  # usethis::use_mit_license() writes LICENSE.md but leaves an existing
+  # LICENSE alone, which is how the two came to disagree in the first place.
+  expect_match(full, "MIT License", fixed = TRUE)
+})
+
+test_that("vignettes are not excluded from the build", {
+  skip_if_no_source_tree(".Rbuildignore")
+  ignore <- readLines(source_tree_path(".Rbuildignore"), warn = FALSE)
+  expect_false(any(grepl("^\\^vignettes\\$$", ignore)))
+  # The fixture is what lets R CMD check run offline with no credentials on
+  # r-universe and GitHub Actions. It must never be excluded.
+  expect_false(any(grepl("fixture_corpus", ignore, fixed = TRUE)))
+  # doc/ and Meta/ ARE build artefacts of devtools::build_vignettes() and must
+  # stay excluded -- R CMD build regenerates inst/doc/ from vignettes/ on its
+  # own, and leaving them in earns a "non-standard file at top level" NOTE.
+  expect_true(any(grepl("^\\^doc\\$$", ignore)))
+  expect_true(any(grepl("^\\^Meta\\$$", ignore)))
+})
+
+test_that("_pkgdown.yml indexes every exported topic", {
+  skip_if_no_source_tree("_pkgdown.yml", "NAMESPACE")
+  exports <- grep("^export\\(", readLines(source_tree_path("NAMESPACE"), warn = FALSE),
+                  value = TRUE)
+  exports <- sub("^export\\((.*)\\)$", "\\1", exports)
+  yml <- paste(readLines(source_tree_path("_pkgdown.yml"), warn = FALSE), collapse = "\n")
+  missing <- exports[!vapply(exports,
+                             function(e) grepl(paste0("\\b", e, "\\b"), yml),
+                             logical(1))]
+  # pkgdown errors on topics missing from the index, so an unlisted export
+  # means the docs site does not build at all.
+  expect_equal(missing, character(0))
+})
+
+test_that("README is written for a stranger, not a repo insider", {
+  skip_if_no_source_tree("README.md")
+  r <- paste(readLines(source_tree_path("README.md"), warn = FALSE), collapse = "\n")
+
+  # No paths that only resolve inside a maintainer's checkout.
+  expect_false(grepl("../cog_pipeline", r, fixed = TRUE))
+  # A real, uncommented install line.
+  expect_match(r, "install.packages", fixed = TRUE)
+  expect_false(grepl("# pak::pkg_install", r, fixed = TRUE))
+  # The errata most likely to produce a plausible-looking wrong answer.
+  expect_match(r, "full US dollars", fixed = TRUE)
+  # The release advice that conflicts with public CI is gone.
+  expect_false(grepl("Rbuildignore", r, fixed = TRUE))
+  # Both read paths documented.
+  expect_match(r, "cog_mirror", fixed = TRUE)
+  # cog_spending() has no default for `years`; a quickstart that omits it
+  # errors on the reader's first call.
+  expect_match(r, "years\\s*=", perl = TRUE)
+})

tests/testthat/test-duckdb-limits.R

@@ -0,0 +1,134 @@
+# tests/testthat/test-duckdb-limits.R
+#
+# uscogdata#60. cog_open() used to connect with a bare dbConnect() and set no
+# resource pragmas, so DuckDB claimed every visible core. That is right for one
+# interactive session on a dedicated machine and wrong for a server: cog-api
+# runs two replicas on an 8-core host budgeted 4, and without a cap each
+# replica independently claims all 8 and they fight.
+#
+# The consumer-side workaround this replaces reached into the namespace at
+# boot -- getFromNamespace(".ensure_session", "uscogdata")() followed by a
+# manual SET threads -- which depends on a private name AND on the session
+# already being open.
+#
+# The load-bearing property is the NEGATIVE one: unset must emit no pragma at
+# all, so an unconfigured session is byte-identical to pre-#60 behaviour.
+
+# Open a session under a given configuration and read a DuckDB setting back.
+# Each call closes first, because both settings are session-scoped: an
+# already-open connection would be reused by .ensure_session() and report the
+# PREVIOUS test's value, which is exactly the false pass to avoid here.
+setting_under <- function(setting, envvars = character(0), opts = list()) {
+  uscogdata:::cog_close()
+  on.exit(uscogdata:::cog_close(), add = TRUE)
+  withr::with_envvar(envvars, {
+    withr::with_options(opts, {
+      con <- uscogdata:::cog_open()
+      DBI::dbGetQuery(
+        con, sprintf("SELECT current_setting('%s') AS v", setting)
+      )$v[[1]]
+    })
+  })
+}
+
+test_that("USCOGDATA_DUCKDB_THREADS caps the connection's thread count", {
+  skip_if_no_corpus()
+  expect_equal(
+    as.integer(setting_under("threads", c(USCOGDATA_DUCKDB_THREADS = "2"))),
+    2L
+  )
+})
+
+test_that("the option spelling works, and the env var beats it", {
+  skip_if_no_corpus()
+  expect_equal(
+    as.integer(setting_under("threads",
+                             c(USCOGDATA_DUCKDB_THREADS = NA),
+                             list(uscogdata.duckdb_threads = 3L))),
+    3L
+  )
+  # Same precedence .cfg() gives every other setting: env var > option.
+  expect_equal(
+    as.integer(setting_under("threads",
+                             c(USCOGDATA_DUCKDB_THREADS = "1"),
+                             list(uscogdata.duckdb_threads = 3L))),
+    1L
+  )
+})
+
+test_that("unset leaves DuckDB's own default in place", {
+  skip_if_no_corpus()
+  # Not asserting a specific number -- the default is core-count-dependent and
+  # a literal would fail on a different machine. The claim is that NO pragma
+  # was issued, so the session sees whatever DuckDB would have chosen on its
+  # own. Compared against a plain connection opened the pre-#60 way.
+  unset <- setting_under("threads",
+                         c(USCOGDATA_DUCKDB_THREADS = NA),
+                         list(uscogdata.duckdb_threads = NULL))
+  bare <- local({
+    con <- DBI::dbConnect(duckdb::duckdb())
+    on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
+    DBI::dbGetQuery(con, "SELECT current_setting('threads') AS v")$v[[1]]
+  })
+  expect_equal(as.integer(unset), as.integer(bare))
+})
+
+test_that("USCOGDATA_DUCKDB_MEMORY_LIMIT is applied", {
+  skip_if_no_corpus()
+  v <- setting_under("memory_limit", c(USCOGDATA_DUCKDB_MEMORY_LIMIT = "2GB"))
+  # DuckDB does not echo back the string it was given: it stores bytes and
+  # reports BINARY units, so "2GB" (2e9 bytes) comes back as "1.8 GiB". Assert
+  # the magnitude it actually means rather than the spelling this package sent
+  # -- matching on "2" passes for the wrong reason and fails on the right one.
+  expect_match(as.character(v), "GiB", fixed = TRUE)
+  # DuckDB also truncates the display to one decimal ("1.8 GiB" for 1.863), so
+  # the tolerance covers rounding, not slack in the setting itself.
+  gib <- as.numeric(sub("\\s*GiB$", "", as.character(v)))
+  expect_equal(gib, 2e9 / 1024^3, tolerance = 0.05)
+})
+
+# --- Validation -------------------------------------------------------------
+# .cfg() returns an env var as CHARACTER. Without coercion here,
+# sprintf("SET threads TO %d", "4") aborts inside the connection path with an
+# error about the pragma rather than about the setting the operator got wrong.
+
+test_that(".resolve_duckdb_threads coerces a character env var to integer", {
+  withr::local_envvar(USCOGDATA_DUCKDB_THREADS = "4")
+  expect_identical(uscogdata:::.resolve_duckdb_threads(), 4L)
+})
+
+test_that(".resolve_duckdb_threads returns NULL when unset or empty", {
+  withr::local_options(uscogdata.duckdb_threads = NULL)
+  withr::local_envvar(USCOGDATA_DUCKDB_THREADS = NA)
+  expect_null(uscogdata:::.resolve_duckdb_threads())
+
+  withr::local_envvar(USCOGDATA_DUCKDB_THREADS = "")
+  expect_null(uscogdata:::.resolve_duckdb_threads())
+})
+
+test_that(".resolve_duckdb_threads rejects values that are not positive integers", {
+  for (bad in c("0", "-1", "two", "1.5.2")) {
+    withr::local_envvar(USCOGDATA_DUCKDB_THREADS = bad)
+    expect_error(uscogdata:::.resolve_duckdb_threads(),
+                 class = "uscogdata_invalid_duckdb_threads")
+  }
+})
+
+test_that(".resolve_duckdb_memory_limit accepts size strings and rejects junk", {
+  withr::local_envvar(USCOGDATA_DUCKDB_MEMORY_LIMIT = "4GB")
+  expect_identical(uscogdata:::.resolve_duckdb_memory_limit(), "4GB")
+
+  withr::local_envvar(USCOGDATA_DUCKDB_MEMORY_LIMIT = "1.5GB")
+  expect_identical(uscogdata:::.resolve_duckdb_memory_limit(), "1.5GB")
+
+  # A SQL fragment must not reach the connection as one.
+  withr::local_envvar(USCOGDATA_DUCKDB_MEMORY_LIMIT = "4GB'; DROP TABLE x; --")
+  expect_error(uscogdata:::.resolve_duckdb_memory_limit(),
+               class = "uscogdata_invalid_duckdb_memory_limit")
+})
+
+test_that(".apply_duckdb_limits issues no statement when both are NULL", {
+  # The negative property, asserted directly rather than inferred: a connection
+  # that would ERROR on any statement proves none was sent.
+  expect_silent(uscogdata:::.apply_duckdb_limits(NULL, NULL, NULL))
+})

tests/testthat/test-live-corpus.R

@@ -0,0 +1,62 @@
+# Network-gated. Set USCOGDATA_LIVE_TEST=true to run.
+#
+# This file exists because the defect fixed for 0.3.0 -- no remote corpus was
+# readable at all, because DuckDB cannot expand a glob over generic HTTP --
+# survived precisely because every other test path used a LOCAL corpus (the
+# bundled fixture), and so did the API in production (a host mount). Nothing
+# ever exercised the package the way a new user does.
+skip_live <- function() {
+  testthat::skip_if_not(
+    identical(tolower(Sys.getenv("USCOGDATA_LIVE_TEST", "")), "true"),
+    "live-corpus test: set USCOGDATA_LIVE_TEST=true to run"
+  )
+}
+
+# The suite's setup.R pins USCOGDATA_URL to the bundled fixture, so reaching
+# the default requires clearing both the env var and the option.
+with_default_corpus <- function(code) {
+  withr::local_envvar(
+    USCOGDATA_URL = NA, USCOGDATA_FIXTURE_URL = NA,
+    .local_envir = parent.frame()
+  )
+  withr::local_options(uscogdata.url = NULL, .local_envir = parent.frame())
+  cog_close()
+  withr::defer(cog_close(), envir = parent.frame())
+  force(code)
+}
+
+test_that("the package reads the public corpus with no configuration at all", {
+  skip_live()
+  with_default_corpus({
+    g <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+    expect_gt(nrow(g), 0)
+
+    s <- cog_spending(g$canonical_govid[1], years = 2022)
+    expect_gt(nrow(s), 0)
+    expect_true(all(c("amt_nominal", "year", "category") %in% names(s)))
+
+    # Amounts are full dollars, already x1000. A city's annual spending is
+    # millions, not thousands -- this catches a regression that dropped or
+    # doubled the conversion.
+    expect_gt(sum(s$amt_nominal, na.rm = TRUE), 1e6)
+
+    p <- attr(s, "provenance")
+    expect_true(isTRUE(p$transformations$units_conversion$applied))
+    expect_equal(p$transformations$units_conversion$multiplier, 1000)
+  })
+})
+
+test_that("a multi-decade query reads across many partitions", {
+  skip_live()
+  with_default_corpus({
+    g <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+    # `years` is required on cog_spending() -- there is no full-history
+    # default at the reader level (the API's /profile route supplies one).
+    s <- cog_spending(g$canonical_govid[1], years = 2000:2022)
+    # Enumeration builds one read_parquet() path per requested partition. If
+    # the list were truncated, or silently collapsed to a single file, the
+    # returned span is what catches it.
+    expect_gt(diff(range(s$year)), 10)
+    expect_gt(length(unique(s$year)), 5)
+  })
+})

tests/testthat/test-long-files.R

@@ -0,0 +1,97 @@
+test_that(".long_files_sql enumerates every partition the manifest lists", {
+  manifest <- list(files = list(long_partitions = list(
+    list(year = 2011L, path = "data/long/year=2011/part-0.parquet"),
+    list(year = 2012L, path = "data/long/year=2012/part-0.parquet")
+  )))
+  expect_equal(
+    uscogdata:::.long_files_sql("https://example.org/corpus/", manifest),
+    paste0(
+      "['https://example.org/corpus/data/long/year=2011/part-0.parquet',",
+      "'https://example.org/corpus/data/long/year=2012/part-0.parquet']"
+    )
+  )
+})
+
+test_that(".long_files_sql falls back to the glob when no partition list is present", {
+  # test-views.R registers views with a hand-built manifest that has no
+  # `files` element. That must keep working: the glob is valid for the
+  # local paths such a manifest is used with.
+  expect_equal(
+    uscogdata:::.long_files_sql("/tmp/corpus/", list(schema_version = 4L)),
+    "'/tmp/corpus/data/long/**/*.parquet'"
+  )
+  expect_equal(
+    uscogdata:::.long_files_sql("/tmp/corpus/", list(files = list(long_partitions = list()))),
+    "'/tmp/corpus/data/long/**/*.parquet'"
+  )
+})
+
+test_that("the enumerated list matches the bundled fixture's partition count", {
+  skip_if_no_corpus()
+  m <- jsonlite::fromJSON(
+    file.path(fixture_corpus_path(), "manifest.json"), simplifyVector = FALSE
+  )
+  out <- uscogdata:::.long_files_sql(fixture_corpus_path(), m)
+  expect_equal(
+    lengths(regmatches(out, gregexpr("part-0\\.parquet", out))),
+    length(m$files$long_partitions)
+  )
+})
+
+test_that("no view SQL survives rendering with an unsubstituted token", {
+  # Introducing {long_files} broke four test sites that had hand-rolled the
+  # {url} substitution -- each failed with a DuckDB parser error on the
+  # surviving brace. This asserts the whole SQL directory renders clean, so
+  # a future token cannot reintroduce that silently.
+  sql_dir <- system.file("sql", package = "uscogdata")
+  for (f in list.files(sql_dir, pattern = "\\.sql$", full.names = TRUE)) {
+    rendered <- uscogdata:::.render_view_sql(
+      paste(readLines(f, warn = FALSE), collapse = "\n"), "/tmp/corpus/"
+    )
+    expect_false(grepl("\\{[a-z_]+\\}", rendered), label = basename(f))
+  }
+})
+
+test_that("registered `long` view reads through the enumerated list", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    con <- uscogdata:::.ensure_session()
+    n <- DBI::dbGetQuery(con, "SELECT count(*) AS n FROM long")$n
+    expect_gt(n, 0)
+    yrs <- DBI::dbGetQuery(con, "SELECT DISTINCT year FROM long ORDER BY year")$year
+    expect_true(all(c(2011, 2012, 2019, 2020) %in% yrs))
+  })
+})
+
+test_that("a Windows-style corpus path survives token substitution", {
+  # gsub() in regex mode treats backslashes in the REPLACEMENT as escape
+  # sequences and silently drops them, so a Windows path went in as
+  # C:\Users\RUNNER\... and came out as C:UsersRUNNER..., after which every
+  # DuckDB read failed with "No files found that match the pattern".
+  #
+  # That made a LOCAL corpus unreadable on Windows -- the bundled fixture
+  # included, so the whole suite failed there -- while remote https URLs
+  # worked fine, having no backslashes. It went unnoticed for the life of the
+  # package because nothing ever ran on Windows.
+  #
+  # Reproducible on any platform: this is string handling, not a filesystem
+  # behaviour, so it does not need a Windows runner to catch.
+  win <- "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\Rtmp123/"
+
+  out <- uscogdata:::.render_view_sql(
+    "FROM read_parquet('{url}data/summary_categories.parquet')", win
+  )
+  expect_true(grepl("C:\\Users\\RUNNER~1\\AppData", out, fixed = TRUE))
+  expect_false(grepl("C:Users", out, fixed = TRUE))
+
+  # The same must hold through the {long_files} path, which embeds the url
+  # once per enumerated partition.
+  manifest <- list(files = list(long_partitions = list(
+    list(year = 2011L, path = "data/long/year=2011/part-0.parquet")
+  )))
+  out2 <- uscogdata:::.render_view_sql(
+    "FROM read_parquet({long_files}, hive_partitioning = true)", win, manifest
+  )
+  expect_true(grepl("C:\\Users\\RUNNER~1\\AppData", out2, fixed = TRUE))
+  expect_false(grepl("C:Users", out2, fixed = TRUE))
+})

tests/testthat/test-manifest.R

@@ -4,12 +4,14 @@
 # protect users from silent failures when USCOGDATA_URL is misconfigured
 # or returns non-JSON content.
 
-test_that("cog_open aborts with actionable error when URL is the placeholder default", {
+test_that("cog_open aborts with actionable error when URL contains the sentinel", {
   uscogdata:::cog_close()
   on.exit(uscogdata:::cog_close(), add = TRUE)
 
-  placeholder <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  # No longer the package default (that is the public HF corpus). This is a
-  withr::with_envvar(c(USCOGDATA_URL = placeholder), {
+  # user who copied a config template and did not finish editing it.
+  sentinel_url <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  withr::with_envvar(c(USCOGDATA_URL = sentinel_url), {
     expect_error(
       uscogdata:::cog_open(),
       class = "uscogdata_url_not_configured"
@@ -35,8 +37,10 @@ test_that("placeholder guard error names both env var and option as remediation"
   uscogdata:::cog_close()
   on.exit(uscogdata:::cog_close(), add = TRUE)
 
-  placeholder <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  # No longer the package default (that is the public HF corpus). This is a
-  withr::with_envvar(c(USCOGDATA_URL = placeholder), {
+  # user who copied a config template and did not finish editing it.
+  sentinel_url <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  withr::with_envvar(c(USCOGDATA_URL = sentinel_url), {
     msg <- tryCatch(uscogdata:::cog_open(), error = conditionMessage)
     expect_match(msg, "USCOGDATA_URL", fixed = TRUE)
     expect_match(msg, "uscogdata.url", fixed = TRUE)

tests/testthat/test-recipes.R

@@ -251,7 +251,7 @@ test_that(".suppressed_components measures the E67/E68 dollars Public Welfare dr
   s <- uscogdata:::.suppressed_components(
     con,
     candidates = c("welfare_cash_e67_wide", "welfare_cash_e68_wide"),
-    govid = "061037123085", years = 2011L,
+    cohort = uscogdata:::.make_cohort("061037123085"), years = 2011L,
     long_view = "spending_long_harmonized",
     flow_prefixes = c("E", "F", "G"))
 
@@ -269,7 +269,7 @@ test_that(".suppressed_components finds nothing in a modern year", {
   s <- uscogdata:::.suppressed_components(
     con,
     candidates = c("welfare_cash_e67_wide", "welfare_cash_e68_wide"),
-    govid = "061037123085", years = 2019L,
+    cohort = uscogdata:::.make_cohort("061037123085"), years = 2019L,
     long_view = "spending_long_harmonized",
     flow_prefixes = c("E", "F", "G"))
   expect_equal(nrow(s), 0L)
@@ -280,7 +280,7 @@ test_that(".suppressed_components rejects a long_view outside the allowlist", {
   con <- uscogdata:::.ensure_session()
   expect_error(
     uscogdata:::.suppressed_components(
-      con, candidates = "welfare_cash_e67_wide", govid = "061037123085",
+      con, candidates = "welfare_cash_e67_wide", cohort = uscogdata:::.make_cohort("061037123085"),
       years = 2011L, long_view = "long; DROP TABLE x",
       flow_prefixes = c("E", "F", "G")),
     class = "uscogdata_internal_error")
@@ -298,7 +298,7 @@ test_that(".suppressed_components never measures a component from the other flow
   s <- uscogdata:::.suppressed_components(
     con,
     candidates = c("welfare_cash_e67_wide", "welfare_cash_e68_wide"),
-    govid = "061037123085", years = 2011L,
+    cohort = uscogdata:::.make_cohort("061037123085"), years = 2011L,
     long_view = "revenue_long_harmonized",
     flow_prefixes = c("T", "A", "U", "B", "C", "D"))
   expect_equal(nrow(s), 0L)

tests/testthat/test-search-balances-pagination.R

@@ -0,0 +1,178 @@
+# tests/testthat/test-search-balances-pagination.R
+#
+# uscogdata#57. cog_spending()/cog_revenue() gained limit/offset in #39;
+# cog_gov_search() and cog_balances() did not, so every consumer of those two
+# was back to materialize-then-slice -- the exact pattern that wedged the
+# production API for hours on 2026-08-06.
+#
+# cog_gov_search() was also the one verb with no LIMIT at all, so an
+# unfiltered call returns the entire 40,336-row crosswalk by accident.
+
+# --- cog_gov_search() -------------------------------------------------------
+
+test_that("cog_gov_search() limit returns the first page of the unpaginated result", {
+  skip_if_no_corpus()
+  full <- cog_gov_search(state = "WI", type = "city")
+  skip_if(nrow(full) < 12L, "fixture has too few WI cities to page")
+
+  page <- cog_gov_search(state = "WI", type = "city", limit = 5L)
+  expect_equal(nrow(page), 5L)
+  expect_equal(page$canonical_govid, full$canonical_govid[1:5])
+})
+
+test_that("cog_gov_search() offset skips ahead without gaps or overlap", {
+  skip_if_no_corpus()
+  full <- cog_gov_search(state = "WI", type = "city")
+  skip_if(nrow(full) < 12L, "fixture has too few WI cities to page")
+
+  p1 <- cog_gov_search(state = "WI", type = "city", limit = 5L)
+  p2 <- cog_gov_search(state = "WI", type = "city", limit = 5L, offset = 5L)
+  expect_equal(p2$canonical_govid, full$canonical_govid[6:10])
+  expect_length(intersect(p1$canonical_govid, p2$canonical_govid), 0L)
+})
+
+test_that("walking every page reconstructs the unpaginated search exactly", {
+  skip_if_no_corpus()
+  full <- cog_gov_search(state = "WI", type = "city")
+  n <- nrow(full)
+  limit <- 7L
+  pages <- list()
+  offset <- 0L
+  repeat {
+    p <- cog_gov_search(state = "WI", type = "city", limit = limit, offset = offset)
+    if (nrow(p) == 0L) break
+    pages[[length(pages) + 1L]] <- p
+    offset <- offset + limit
+    if (offset > n + limit) stop("test runaway: paging did not terminate")
+  }
+  walked <- dplyr::bind_rows(pages)
+  expect_equal(nrow(walked), n)
+  expect_equal(walked$canonical_govid, full$canonical_govid)
+})
+
+test_that("cog_gov_search() total_rows reports the full unpaginated count", {
+  skip_if_no_corpus()
+  full <- cog_gov_search(state = "WI", type = "city")
+  page <- cog_gov_search(state = "WI", type = "city", limit = 3L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("cog_gov_search() offset past the end reports the true total, not zero", {
+  skip_if_no_corpus()
+  full <- cog_gov_search(state = "WI", type = "city")
+  # No row survives to carry COUNT(*) OVER(), so this is the branch that has
+  # to fall back to a second count rather than reporting 0 rows out of 0.
+  page <- cog_gov_search(state = "WI", type = "city",
+                         limit = 5L, offset = nrow(full) + 50L)
+  expect_equal(nrow(page), 0L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("cog_gov_search() bounds an otherwise-unfiltered crosswalk sweep", {
+  skip_if_no_corpus()
+  # The reason this verb needed a limit most: with no filter it returns the
+  # whole crosswalk.
+  page <- cog_gov_search(limit = 10L)
+  expect_equal(nrow(page), 10L)
+  expect_gt(attr(page, "total_rows"), 10L)
+})
+
+test_that("cog_gov_search() orders by a total order, not population alone", {
+  skip_if_no_corpus()
+  # population_acs is not unique -- NA in particular repeats across many rows
+  # -- so paging on it alone can duplicate a row on one page and drop it from
+  # the next. The tiebreaker is what makes the sequence reproducible.
+  full <- cog_gov_search(state = "WI")
+  skip_if(nrow(full) < 5L, "fixture has too few WI governments")
+  expect_equal(cog_gov_search(state = "WI")$canonical_govid,
+               full$canonical_govid)
+
+  ties <- full[is.na(full$population_acs), ]
+  skip_if(nrow(ties) < 2L, "no tied rows in the fixture to order")
+  expect_false(is.unsorted(ties$canonical_govid))
+})
+
+test_that("cog_gov_search() refuses pagination in basket mode", {
+  skip_if_no_corpus()
+  expect_error(
+    cog_gov_search(name = c("MADISON CITY", "MILWAUKEE CITY"),
+                   state = c("WI", "WI"), limit = 1L),
+    class = "uscogdata_basket_pagination_conflict"
+  )
+})
+
+test_that("cog_gov_search() rejects a malformed limit or offset", {
+  skip_if_no_corpus()
+  expect_error(cog_gov_search(state = "WI", limit = -1L),
+               class = "uscogdata_invalid_pagination")
+  expect_error(cog_gov_search(state = "WI", limit = 5L, offset = -1L),
+               class = "uscogdata_invalid_pagination")
+})
+
+# --- cog_balances() ---------------------------------------------------------
+
+test_that("cog_balances() limit/offset walk the unpaginated result exactly", {
+  skip_if_no_corpus()
+  full <- cog_balances(years = 2019:2020, state = "WI", type = "city")
+  skip_if(nrow(full) < 6L, "fixture has too few WI city balance rows to page")
+
+  key <- c("year", "canonical_govid", "balance_subtype", "amt_nominal")
+  p1 <- cog_balances(years = 2019:2020, state = "WI", type = "city", limit = 3L)
+  p2 <- cog_balances(years = 2019:2020, state = "WI", type = "city",
+                     limit = 3L, offset = 3L)
+
+  expect_equal(nrow(p1), 3L)
+  expect_equal(p1[key], full[1:3, key], ignore_attr = TRUE)
+  expect_equal(p2[key], full[4:6, key], ignore_attr = TRUE)
+  # The window-function column is an implementation detail and must not reach
+  # the caller's data frame.
+  expect_false("pagination_total_rows" %in% names(p1))
+})
+
+test_that("cog_balances() total_rows reports the full unpaginated count", {
+  skip_if_no_corpus()
+  full <- cog_balances(years = 2019:2020, state = "WI", type = "city")
+  page <- cog_balances(years = 2019:2020, state = "WI", type = "city", limit = 2L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("cog_balances() offset past the end reports the true total", {
+  skip_if_no_corpus()
+  full <- cog_balances(years = 2019:2020, state = "WI", type = "city")
+  page <- cog_balances(years = 2019:2020, state = "WI", type = "city",
+                       limit = 5L, offset = nrow(full) + 50L)
+  expect_equal(nrow(page), 0L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("cog_balances() refuses pagination alongside a recipe", {
+  skip_if_no_corpus()
+  expect_error(
+    cog_balances(years = 2011, state = "WI", type = "city",
+                 recipe = "cash_securities_z77_wide", limit = 5L),
+    class = "uscogdata_recipe_pagination_conflict"
+  )
+})
+
+test_that("cog_balances() rejects a malformed limit or offset", {
+  skip_if_no_corpus()
+  expect_error(cog_balances(years = 2019, state = "WI", type = "city", limit = -1L),
+               class = "uscogdata_invalid_pagination")
+  expect_error(cog_balances(years = 2019, state = "WI", type = "city",
+                            limit = 5L, offset = -1L),
+               class = "uscogdata_invalid_pagination")
+})
+
+# --- Unchanged without the arguments ----------------------------------------
+
+test_that("both verbs are unchanged when limit is not supplied", {
+  skip_if_no_corpus()
+  # The adoption contract for cog-api: NULL default, so a formals() probe can
+  # feature-detect without any call site changing behaviour.
+  s <- cog_gov_search(state = "WI", type = "city")
+  b <- cog_balances(years = 2019, state = "WI", type = "city")
+  expect_null(attr(s, "total_rows"))
+  expect_null(attr(b, "total_rows"))
+  expect_true(all(c("limit", "offset") %in% names(formals(cog_gov_search))))
+  expect_true(all(c("limit", "offset") %in% names(formals(cog_balances))))
+})

tests/testthat/test-views.R

@@ -103,7 +103,7 @@ test_that("inst/sql/22- and 23- harmonized views enforce every WHERE predicate (
   sql_dir <- system.file("sql", package = "uscogdata")
   .read_view_sql <- function(filename) {
     txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
   }
 
   con <- DBI::dbConnect(duckdb::duckdb())
@@ -183,7 +183,7 @@ test_that("inst/sql/24- and 25- IG views retain aggregates, COALESCE NULL harmon
   sql_dir <- system.file("sql", package = "uscogdata")
   .read_view_sql <- function(filename) {
     txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
   }
 
   con <- DBI::dbConnect(duckdb::duckdb())
@@ -335,7 +335,7 @@ test_that(".harmonization_view_files guard is necessary: registration against a
   sql_dir <- system.file("sql", package = "uscogdata")
   .read_view_sql <- function(filename) {
     txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", url, txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, url)
   }
   con2 <- DBI::dbConnect(duckdb::duckdb())
   on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)