fix: keep doc/ and Meta/ out of the build

Removing ^vignettes$ was right; removing ^doc$ and ^Meta$ with it was not. Those are devtools::build_vignettes() artefacts, not sources -- R CMD build regenerates inst/doc/ from vignettes/ by itself, and shipping the local copies earned a 'non-standard file/directory found at top level' NOTE. R CMD check --as-cran is now 0 errors, 0 warnings, 0 notes.
docs: add CONTRIBUTING with the canonical-on-Gitea PR flow
2026-08-08 18:25:00 -04:00 · 2026-08-08 18:20:42 -04:00 · 2026-08-08 18:20:33 -04:00 · 2026-08-08 18:20:24 -04:00 · 2026-08-08 17:49:10 -04:00 · 2026-08-08 17:48:07 -04:00
43 changed files with 3155 additions and 397 deletions
@@ -3,17 +3,17 @@
 ^\.Rproj\.user$
 ^_pkgdown\.yml$
 ^docs$
+^Meta$
+^doc$
 ^pkgdown$
 ^\.github$
 ^LICENSE\.md$
 ^\.git$
 ^\.gitignore$
 \.gitkeep$
-^vignettes$
 ^specs$
 ^plans$
-^doc$
-^Meta$
 ^\.gitea$
 ^CLAUDE\.md$
 ^\.superpowers$
+^CONTRIBUTING\.md$
@@ -124,3 +124,10 @@ devtools::test()
  a numbered `.sql` file; build a query in R.
 - No arrow dependency — DuckDB reads parquet natively
 - `withr` is a Suggests-only dep; only used in tests
+
+## Domain context — read this first
+
+**Before doing any work in this repo, read `~/.claude/memory/values/civilytics.md`.**
+It carries the purpose, direction, and constraints for this domain. It is not optional
+context — read it before planning or writing code, not after. (An `@` import will not
+work here; project-level imports don't preload. The read is the mechanism.)
@@ -0,0 +1,105 @@
+# Contributing to uscogdata
+
+Thanks for reading this — a package like this gets better mostly through people
+noticing that a number looks wrong.
+
+## Where the code lives
+
+Development happens on **Gitea**, at
+`gitea.civilytics.org/Civilytics/uscogdata`. The repository at
+`github.com/civilytics/uscogdata` is a **mirror** that accepts issues and pull
+requests.
+
+## What happens to a GitHub pull request
+
+Open it normally. Behind the scenes it is fetched and landed on the canonical
+Gitea repository, then syncs back:
+
+```sh
+git fetch github refs/pull/42/head:pr-42
+git switch main && git merge --no-ff pr-42
+git push origin main            # Gitea -> mirror -> GitHub
+```
+
+Because the merge preserves your commits at their original SHAs, **GitHub marks
+your PR merged on its own** as soon as the mirror syncs. So:
+
+> If your pull request closes as "Merged" without anyone visibly clicking
+> Merge, that is the normal, successful outcome — not a rejection.
+
+Substantial contributions get a `ctb` entry in `DESCRIPTION`, which surfaces in
+`citation("uscogdata")`.
+
+There is no CLA and no DCO sign-off requirement.
+
+## Running the tests
+
+```r
+devtools::test()   # bundled fixture; no network, no credentials
+```
+
+`tests/testthat/setup.R` points `USCOGDATA_URL` at
+`inst/extdata/fixture_corpus/` automatically — a four-year slice (2011, 2012,
+2019, 2020) covering all 50 states. That is the whole data setup.
+
+## Testing against the live corpus
+
+```sh
+USCOGDATA_LIVE_TEST=true Rscript -e 'devtools::test(filter = "live-corpus")'
+```
+
+This is worth understanding rather than skipping. Until 0.3.0 the package
+**could not read a remote corpus at all** — the partitioned view used a glob,
+and DuckDB cannot expand a glob over generic HTTP. It went unnoticed for months
+because every test path used a local corpus (the bundled fixture), and so did
+the production API (a host mount). Nothing exercised the package the way a new
+user does.
+
+`test-live-corpus.R` is the only test that runs with no `USCOGDATA_URL`, no
+option, and no fixture. If you change anything touching view registration,
+manifest handling, or configuration, run it.
+
+## Do not exclude the fixture from the build
+
+There is a temptation to add `^inst/extdata/fixture_corpus$` to
+`.Rbuildignore` because 15 MB feels large for a package. Don't:
+
+- `vignette("total-spending")` reads from it and would fail to build.
+- `R CMD check` on r-universe and GitHub Actions would have no corpus, so the
+  suite could not run without credentials.
+
+This package is not going to CRAN, so its 5 MB guidance does not apply. A
+package-size NOTE in `R CMD check` is expected and acceptable.
+
+## Downstream consumers
+
+`cog-api` depends on this package and its CI clones uscogdata at
+`USCOGDATA_REF`, **defaulting to `main`**. There is no pin. Anything merged
+here reaches the API's next build, so before merging a change to the reader,
+run the API suite against your branch:
+
+```sh
+Rscript -e "remotes::install_local('/path/to/uscogdata', upgrade = 'never')"
+cd /path/to/cog-api/api/tests/testthat
+Rscript -e 'testthat::test_dir(".", stop_on_failure = TRUE)'
+```
+
+The API calls only exported verbs, so internal refactors are usually safe —
+but "usually" is not a release gate.
+
+## Release checklist
+
+1. `devtools::test()` — green against the bundled fixture, offline.
+2. `USCOGDATA_LIVE_TEST=true devtools::test()` — green against the live corpus.
+3. cog-api suite green against this branch (above).
+4. `devtools::check(args = "--as-cran")` — 0 errors, 0 warnings.
+5. `pkgdown::build_site()` completes.
+6. Vignettes resolve from an installed copy:
+   `vignette("total-spending", package = "uscogdata")`.
+7. **Cold-start check**: on a machine that has never had this package,
+   install it and run the README quickstart verbatim with no environment
+   variables set. This is the only check that catches a
+   corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
+8. Bump `Version` and add a `NEWS.md` section.
+9. Tag, then update the r-universe registry pin at
+   `github.com/civilytics/civilytics.r-universe.dev`.
@@ -1,14 +1,21 @@
 Package: uscogdata
 Type: Package
 Title: Curated Reader for the Civilytics US Census of Governments Finance Corpus
-Version: 0.1.0
-Authors@R:
-    person("Civilytics", , , "jknowles@gmail.com", role = c("aut", "cre"))
+Version: 0.3.0
+Authors@R: c(
+    person(c("Jared", "E."), "Knowles",
+           email = "jared@civilytics.com",
+           role = c("aut", "cre"),
+           comment = c(ORCID = "0000-0003-0005-9478")),
+    person("Civilytics Consulting LLC", role = c("cph", "fnd")))
 Description: Curated R verbs over the Civilytics US Census of Governments
    finance corpus. Provides unit-level financial profiles, geographic
    rollups, and peer comparisons with auditable provenance and built-in
    cross-vintage correctness.
 License: MIT + file LICENSE
+URL: https://github.com/civilytics/uscogdata,
+    https://civilytics.r-universe.dev/uscogdata
+BugReports: https://github.com/civilytics/uscogdata/issues
 Encoding: UTF-8
 LazyData: false
 Depends: R (>= 4.1)
@@ -32,4 +39,4 @@ Config/testthat/edition: 3
 VignetteBuilder: knitr
 RoxygenNote: 7.3.3
 MinCorpusSchema: 4
-MaxCorpusSchema: 5
+MaxCorpusSchema: 7
@@ -1,2 +1,2 @@
 YEAR: 2026
-COPYRIGHT HOLDER: Civilytics
+COPYRIGHT HOLDER: Civilytics Consulting LLC
@@ -0,0 +1,21 @@
+# MIT License
+
+Copyright (c) 2026 Civilytics Consulting LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -1,258 +1,101 @@
-# uscogdata 0.1.0 (development)
+# uscogdata 0.3.0

-## Signposting now catches partially-suppressed categories
+First public release.

-* A coverage suggestion used to fire only when a category returned **no rows
-  at all** in a requested year. That missed the more dangerous case: a
-  category that still returns rows while silently dropping component codes
-  the wide era publishes only as aggregates (#9). `cog_spending(category =
-  "Public Welfare")` for FY2011 returned a plausible figure that omitted
-  `E67`/`E68` entirely -- for Los Angeles County, $2,075,461,000 of a true
-  $5,261,404,000, a 39% understatement, with `provenance$suggestions` empty.
-* Suggestions now also fire on **partial** coverage, and every suggestion
-  carries `trigger` (`"empty_year"` or `"suppressed_component"`),
-  `suppressed_amount`, `suppressed_years` and `suppressed_codes`, so a caller
-  can see how much is missing and decide whether to re-run with the recipe.
-* `cog_revenue()` gets the same fix through the shared verb path. Alaska's
-  FY2011 `Miscellaneous Revenue` reported $943,842,000 while dropping
-  $1,899,995,000 of aggregate-published `U4-` rents and royalties.
-* The trigger stays recipe-driven, so it only fires where a harmonization
-  recipe actually exists to name the fix. `higher_ed_e18_wide` and
-  `general_gov_e89_wide` stay silent in every year measured on the bundled
-  fixture, because their components are ordinary classified leaves even
-  pre-2012.
-* The `suppressed_component` trigger (and any `suppressed_amount`/
-  `suppressed_codes` an `empty_year` fire also carries) is scoped to the
-  calling verb's own flow family: `cog_spending()` only ever measures E/F/G
-  component dollars, `cog_revenue()` only T/A/U/B/C/D. A component from the
-  OTHER flow family reports `suppressed_amount = 0` rather than a fabricated
-  claim. The `empty_year` trigger itself is not flow-scoped -- a category
-  belonging to the other flow (e.g. `cog_spending(category = "IG Local")`)
-  still returns zero rows and can still fire, in any year including modern
-  ones, naming the recipe whose own generic join finds real data for this
-  government. That is a mis-scoped query, not a corpus-format gap, so its
-  `suppressed_amount` is correctly 0.
+`uscogdata` provides curated R verbs over the Civilytics US Census of
+Governments finance corpus: unit-level financial profiles, geographic rollups
+and peer comparisons, with auditable provenance on every result.

-## New: `cog_balances()` for cash-and-security holdings
+## What it covers

-* New `cog_balances()` exposes the 14 cash-and-security holding codes
-  (`category_type = "balance"`): fund balances, retirement system holdings and
-  insurance trust balances (#25). Holdings are a stock, not a flow, so the verb
-  has no `expenditure_concept` / `revenue_concept` / `complete` arguments, and
-  no `subtype` argument either -- for holdings, `category` is a strict
-  coarsening of `balance_subtype`, so `category = "Fund Balances"` is exactly
-  the `general` family (`W01`/`W31`/`W61`).
-* `cog_balances()` results carry `provenance$balance_caveats`, recording that
-  Census holdings are gross rather than GAAP fund balance, and the measured
-  coverage window of each subtype family.
+Government types 0-3 (state, county, municipality, township), FY1967-FY2024 --
+56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
+or FY1969. Special districts (type 4) and school districts (type 5) are out of
+scope pending validation.

-## Multi-government aggregates now disclose their reporting coverage
+## The verbs

-* The Census of Governments is a **complete census only in years ending in 2
-  and 7**; every other year is a sample, and the sample varies enormously. On
-  the bundled fixture, Wisconsin's 608-city universe rolls up **597**
-  governments in FY2012 and **112** in FY2019 — an 18%-to-98% swing the
-  return value said nothing about, so a statewide total resting on a fifth of
-  the universe looked exactly like one resting on all of it.
-* `cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` gain
-  `coverage`:
+`cog_spending()`, `cog_revenue()` and `cog_balances()` for flows and holdings;
+`cog_gov_search()` to resolve place names (including basket mode for many at
+once); `cog_find_peers()` and `cog_peer_compare()` for cohorts;
+`cog_geographic_rollup()` for aggregates; `cog_categories()`, `cog_recipes()`,
+`cog_manifest()` and `cog_explain()` for metadata and provenance; and
+`cog_mirror()` for a local copy of the corpus.

-  | value | effect |
-  |---|---|
-  | `"all"` (default) | every unit that reported that year — unchanged behaviour |
-  | `"census"` | census years only; aborts if the range holds none rather than returning nothing |
-  | `"consistent"` | only units reporting in *every* requested year — a balanced panel |
+## Reading the corpus now works out of the box

-* **Regardless of mode**, every result now carries `provenance$coverage` with
-  per-year `n_units_reporting`, `n_units_expected` and `is_census_year`, plus
-  `provenance$coverage_mode`. `cog_explain()` prints a "Reporting coverage"
-  section. So the default mode can no longer mislead silently.
-* `is_census_year` is a statement about the **survey calendar**, never a claim
-  of completeness: FY1967 is a census year in which only 97 of Wisconsin's 608
-  cities report. `n_units_reporting` is the number that tells the truth.
-* On `cog_peer_compare()` the target is exempt from `"consistent"` balancing —
-  it is the subject of the comparison, not a member of the cohort — and the
-  `summary_*` quantiles are computed after the filter, so they describe the
-  cohort actually returned. `n_units_reporting` counts peers only, against the
-  cohort size.
-* On `cog_find_peers()`, `coverage` governs the cohort **vintage** when `year`
-  is `NULL`: `"census"` snaps to the most recent census year with an observed
-  population, so a cohort is not built from a sample year in which most of the
-  candidate universe is absent.
+* The package reads the published corpus over HTTPS **with no configuration**.
+  Previously the default was a placeholder sentinel and no document in the
+  package supplied a working URL, so a new user had no path to a session.
+* Remote reads work at all. The partitioned view used a glob, and DuckDB
+  cannot expand a glob over generic HTTP -- there is no directory listing to
+  expand against. Partition paths are now enumerated from the corpus manifest,
+  which is host-agnostic: an HTTPS mirror, a Nextcloud share and a local
+  `cog_mirror()` copy all take the same path.
+* Nothing is written to disk in remote mode; DuckDB fetches only the row
+  groups a query needs.

-## `complete = TRUE`: absent cells, labelled with why they are absent
+## Four things to know before your first query

-* `cog_spending()` and `cog_revenue()` gain `complete`, defaulting to `FALSE`
-  (today's behaviour). With `complete = TRUE` the requested grid is filled
-  from the corpus's `code_set` table and every row carries a new
-  `value_source` column:
+* **Amounts are in full US dollars.** The raw Census files report thousands;
+  the verbs multiply by 1000 on the way out. Do not multiply again.
+* **Multi-government aggregates disclose their coverage.** The Census is a
+  complete enumeration only in years ending in 2 and 7; every other year is a
+  sample. Every such result carries `provenance$coverage` with per-year
+  `n_units_reporting`.
+* **Absence means two different things.** Before FY2012 an absent cell means
+  Census published $0; from FY2012 it means not reported. `complete = TRUE`
+  labels which.
+* **Series breaks reach you unasked.** Catalogued breaks intersecting your
+  query appear in provenance and in `cog_explain()`.

-  | `value_source` | meaning | `amt_nominal` |
-  |---|---|---|
-  | `reported` | the corpus carries this cell | as published |
-  | `census_zero` | dense-source year (≤ FY2011), cell absent — Census published `$0` | `0` |
-  | `not_reported` | sparse-source year (≥ FY2012), cell absent — unknown | `NA` |
+## Known limits

-  The `NA` is deliberate and is the whole point: filling a modern absence
-  with `0` would invent data, which is precisely the error the corpus's
-  representation contract exists to prevent.
-* This restores information the reader lost when the corpus was sparsified
-  (`SB194`, cog_pipeline#64) — a wide-era query whose cells were all `$0`
-  had begun returning nothing at all — and improves on what came before it,
-  since the pre-sparsification corpus could not distinguish a published zero
-  from an unreported cell either.
-* The grid is scoped to each government's **own type**, so a county is never
-  filled with cells only a state can report.
-* Needs a corpus published from 2026-07-29 onward (when `representation` and
-  `code_set` began shipping); aborts with class
-  `uscogdata_representation_unavailable` otherwise. Gated on the manifest
-  listing those tables rather than on `schema_version`, which was never
-  bumped for the change. Not available with `recipe` or
-  `expenditure_concept = "total"` — neither draws its cells from `code_set`.
-* `provenance$completion` reports `applied`, `rows_filled`, and the per-year
-  `absence_means` rule; `cog_explain()` prints a "Completion" section.
+* Special districts (type 4) and school districts (type 5) are out of scope.
+* Per-capita rollups exclude governments with no F-33 population, which is by
+  design but does silently narrow a rollup.
+* `n_units_reporting` is category-conditional and is not a response rate.
+* Employee-retirement (`X`) codes stop at FY2016, when those systems moved to
+  the Annual Survey of Public Pensions.

-## Corpus-wide series breaks now reach users (`corpus_break_refs`)
-
-* Four catalogued series breaks carry `fin_code = "ALL"` — caveats about the
-  corpus as a whole rather than about one item code. `series_break_refs` is
-  built by matching `fin_code` against the item codes in the result, and no
-  row's `item_code` is ever the literal `"ALL"`, so **none of them could ever
-  be surfaced**: `SB085` (dollar precision across the 1976/1977 boundary),
-  `SB087` (imputation exclusion from FY2002), `SB194` (the dense → sparse
-  representation change at FY2012) and `SB086` (the government id scheme
-  change at FY2017).
-* Provenance gains `corpus_break_refs`, selected on the break-year window
-  alone and disjoint from `series_break_refs` by construction, so a consumer
-  can tell a whole-result caveat from a break in one series. `cog_explain()`
-  prints them under their own "Corpus-wide caveats" heading. cog-api passes
-  provenance through verbatim, so the field appears there without an API
-  change.
-* `SB194` is the one that made this urgent: a query spanning FY2011 → FY2012
-  crosses the boundary where an absent cell stops meaning "Census published
-  `$0`" and starts meaning "not reported", and until now nothing said so.
-
-## Bundled fixture regenerated against the sparsified corpus
-
-* `inst/extdata/fixture_corpus/` now tracks the corpus published on
-  2026-07-29 (`pipeline_commit 83f9715`, schema v6). The wide era no longer
-  stores explicit zeros: FY2011 fell from 2,864,212 rows to 496,004, of
-  which none are `$0`. **Absence now means two different things** — in a
-  `dense_source` year (≤ FY2011) an absent cell means Census published `$0`;
-  in a `sparse_source` year (≥ FY2012) it means not reported. The corpus
-  carries that rule in two new tables the fixture now ships,
-  `representation.parquet` and `code_set.parquet`, alongside
-  `census_collection_coverage.parquet` and `lineage_events.parquet`
-  (all ten publish-tree metadata tables, up from six). Catalogued upstream
-  as series break `SB194`.
-* `cog_categories()` gains an `assistance` spending subtype: the J-prefix
-  aid/benefit codes (`J19`, `J67`, `J68`, `J85`) are categorised now that
-  the upstream crosswalk covers every flow code carrying dollars.
-* Two consequences worth knowing about, both visible in provenance rather
-  than in returned dollars. The harmonization block's `na_rows_excluded`
-  counts only rows that exist, so wide-era codes that were zero-padded no
-  longer appear there. Coverage-gap `suggestions` are presence-based for the
-  same reason, so a recipe whose component codes were all `$0` for a given
-  government-year is no longer suggested for it.
-* `tests/testthat/test-fixture-vintage.R` pins these structural facts, so a
-  fixture left behind by a future publish fails loudly instead of letting the
-  suite pass against a corpus that no longer exists.
-
-## Breaking: corpus schema_version 4 (Phase P canonical ids)
-
-* The package now requires corpus `schema_version = 4` (`MinCorpusSchema` /
-  `MaxCorpusSchema` in `DESCRIPTION` are both `4`); older corpora built
-  against schema 3 are rejected by `cog_open()` with a clear version-mismatch
-  error. `canonical_govid` is now uniformly 12 characters across every
-  vintage the corpus covers (previously a mix of 9-char legacy ids and
-  12-char FIPS ids depending on source year) — **every hardcoded
-  `canonical_govid` literal from a pre-Phase-P corpus is now invalid** and
-  must be re-resolved via `cog_gov_search()` or the new `canonical_alias`
-  lookup table. `canonical_fips_xwalk` gains four columns
-  (`legacy_govs_id`, `census_geoid`, `id_source`; `confidence` is renamed to
-  `pop_confidence`) and a companion `canonical_alias` table ships in the
-  corpus for mapping legacy/alternate ids onto the current canonical
-  namespace. The bundled fixture corpus (`inst/extdata/fixture_corpus/`) has
-  been regenerated against the Phase P publish tree, now ships the full
-  `canonical_fips_xwalk` and `canonical_alias` master tables alongside the
-  2019-2020 long partitions, and is reproducible via
-  `data-raw/regenerate_fixture_corpus.R`.
-
-## Clearer errors when `USCOGDATA_URL` is unconfigured or returns non-JSON
-
-* `cog_open()` now aborts with the `uscogdata_url_not_configured` error
-  class when the resolved corpus URL still contains the placeholder
-  `REPLACE_WITH_SHARE_TOKEN` sentinel (or is empty). The message lists both
-  remediation paths (`Sys.setenv(USCOGDATA_URL = ...)` and
-  `options(uscogdata.url = ...)`) and points at the bundled fixture for
-  offline testing. Previously the package proceeded to fetch the placeholder
-  URL, cached the resulting HTML welcome page, and failed downstream with a
-  cryptic `jsonlite` lexical-error.
-* `.fetch_or_cache_manifest()` now parses the HTTP response body before
-  persisting it. Non-JSON responses (login pages, 404 HTML) raise
-  `uscogdata_invalid_manifest` with the URL, Content-Type, and underlying
-  parse error — and never write to the on-disk cache.
-* Manifest cache writes are now atomic (write to `manifest.json.tmp.<pid>`
-  in `cache_dir`, then `file.rename` over the target), so an interrupted
-  fetch cannot replace a previously-good cache.
-* Existing caches with non-JSON content (poisoned by the prior code path)
-  are silently refetched instead of returning a parse error to the caller.
-* Local `USCOGDATA_URL` paths whose `manifest.json` is not valid JSON now
-  surface the same `uscogdata_invalid_manifest` class with file context.
-
-## Per-capita denominators now use per-year Census F-33 population
-
-* `cog_spending()` and `cog_revenue()` previously divided all years' amounts
-  by a single ACS 2018-2022 estimate (`canonical_fips_xwalk.population_acs`),
-  producing biased per-capita values for time-series analysis. They now
-  divide by the F-33 `population` recorded on each gov-year via the new
-  `gov_population_yearly` view. Result tibbles gain a `pop_source` column
-  with values `"census_f33"` or `"unavailable"`. `notes` is updated to
-  concatenate multiple notes with `"; "`.
-
-## Peer cohorts can be set to a chosen year
-
-* `cog_find_peers()` adds a `year` argument (default: most recent year for
-  which the target has an observed population in `gov_population_yearly`).
-  The returned column previously named `population_acs` is now `population`
-  and reflects the cohort year's vintage. The cohort year is attached to the
-  returned tibble as `attr(x, "cohort_year")`.
-* `cog_peer_compare()` now stamps a `cohort_year` column on its result (read
-  from the peers tibble's attribute) and records `cohort_year` plus
-  `cohort_govids` in provenance. When the caller supplies a bare character
-  vector instead of a `cog_find_peers()` result, `cohort_year` is `NA`.
-
-## Rollups exclude govs missing population
-
-* `cog_geographic_rollup(per_capita = TRUE)` drops rows whose government has
-  `pop_source == "unavailable"` and records the dropped govids in
-  `provenance$rollup$excluded_govids`. This excludes special districts
-  (type 4) and school districts (type 5) from per-capita rollups by design.
-
-## New: vignette and provenance metadata
-
-* New vignette `population-denominators` covers the four population sources,
-  the type-4/5 coverage gap, the popyear quirk, and how to build moving-window
-  peer cohorts manually.
-* Provenance gains `transformations$per_capita$popyear_range` and
-  `pop_source_counts`. `cog_explain()` renders both.
+# uscogdata 0.2.0

 ## New features

-* `cog_gov_search()` gains a **basket mode**: passing vector `name`
-  / `state` / `type` arguments resolves multiple place names in one
-  call and returns a tibble of canonical rows in input order, ready
-  to pipe into `cog_spending()` / `cog_revenue()`. Per-row resolution
-  follows an exact-then-substring matching algorithm with deterministic
-  disambiguation; ambiguous and missing entries are surfaced via a
-  sidecar audit tibble plus a single console summary message.
-* New exports `cog_basket_resolution()` and `cog_basket_unresolved()`
-  expose the basket sidecar for iterative query refinement.
+* `cog_spending()` and `cog_revenue()` accept the reserved category
+  `"All Categories"`, returning one summed row per
+  `(year, canonical_govid, subtype)` across every category inside the
+  requested concept's subtype scope. Filtering the result to
+  `spend_subtype == "operations"` gives an operating-expenditure total.
+  `cog_geographic_rollup()` inherits it,
+  which is the efficient way to build a geographic total — previously a
+  caller had to issue one rollup per category and sum the results
+  (cog-api#37).

-## Breaking changes
+  `"All Categories"` is not the same thing as `expenditure_concept = "total"`.
+  The concept chooses which subtypes are in scope; `"All Categories"` chooses
+  whether the rows inside that scope are broken out or summed.

-* The first formal of `cog_gov_search()` was renamed from `pattern`
-  to `name`. All existing call sites in `cog_explorer/` and the
-  package itself use positional first-arg, so this rename is
-  non-breaking in practice. Callers that pass `pattern = ...` by name
-  must update to `name = ...`.
+* `cog_categories()` advertises `"All Categories"` for the expenditure and
+  revenue vocabularies, so the reserved value is discoverable.
+
+* Coverage signposting (see "Signposting now catches partially-suppressed
+  categories" below) now also works in `category = "All Categories"` mode.
+  The recipe-suggestion candidate query used to be scoped by `category`,
+  which is never a match for the reserved `"All Categories"` value, so
+  `provenance$suggestions` always came back empty there — the one mode whose
+  whole point is "you cannot sum the wrong scope" was silently unable to
+  signal a wrong scope. The candidate query is now scoped by the concept's
+  subtype allowlist instead, symmetric with how `.build_verb_sql()` itself
+  scopes the summed total: Los Angeles County FY2011, `category = "All
+  Categories"` still excludes $271,589,000 of aggregate-published Public
+  Welfare (`E68`), but now names `recipe = "welfare_cash_e68_wide"` to
+  recover it instead of reporting zero suggestions.
+
+## Documentation
+
+* `cog_geographic_rollup()` and `cog_peer_compare()` now document that
+  `provenance$coverage`'s `n_units_reporting` is **category-conditional** and
+  is not a response rate: a government that was surveyed and genuinely spends
+  nothing in the requested category is indistinguishable from one never
+  surveyed (uscogdata#36).
@@ -28,7 +28,12 @@
 #'   every combination would be either redundant or empty.
 #'   `category = "Fund Balances"` is exactly the `general` family
 #'   (`W01`/`W31`/`W61`). `balance_subtype` is returned, so a finer split is
-#'   one `dplyr::filter()` away.
+#'   one `dplyr::filter()` away. The reserved pseudo-category
+#'   `"All Categories"` (see [cog_spending()]) is **not** supported here and
+#'   errors with class `uscogdata_all_categories_unsupported`: it sums a
+#'   concept's subtype scope, and holdings are a stock with no concept
+#'   vocabulary to sum across. Omit `category` to get every category broken
+#'   out instead.
 #' @param per_capita Divide holdings by population. Note this is a **stock per
 #'   resident** (reserves per person), which is *not* comparable to
 #'   [cog_spending()]'s per-capita figures -- those are a flow per person.
@@ -74,6 +79,13 @@ cog_balances <- function(govid, years, category = NULL,
  # helper reuse as .build_verb_sql()/.attach_per_capita() below; it does NOT
  # route the verb through .verb_spendrev(), which stays deliberately unused
  # here because its flow vocabulary is meaningless for a stock.
+  #
+  # allow_all_categories is left at its FALSE default (contrast
+  # .verb_spendrev(), which passes TRUE): the all-categories mode's "sum"
+  # only means something in terms of a concept's subtype scope, and holdings
+  # have no concept vocabulary. The reuse above is exactly why this can be a
+  # one-line default rather than a second bespoke check -- see the
+  # validator's own doc comment for the incident that made that matter.
  .validate_verb_inputs(govid, years, category, per_capita, adjust_to_year,
                        recipe)
  years <- as.integer(years)
@@ -22,7 +22,10 @@
 #'   `category` column (e.g. `"Police"` or `"Tax"`).
 #' @return Tibble with columns `category`, `category_type`, `subtype`,
 #'   `n_codes`, `item_codes` (comma-separated, alphabetical). Sorted by
-#'   `category_type`, `category`, `subtype`.
+#'   `category_type`, `category`, `subtype`. Includes one row per flow for the
+#'   reserved pseudo-category `"All Categories"`, which carries `NA` for
+#'   `subtype`, `n_codes` and `item_codes` because it is a query mode rather
+#'   than a crosswalk entry — see [cog_spending()]'s `category` argument.
 #' @export
 cog_categories <- function(type = NULL, pattern = NULL) {
  if (!is.null(type)) {
@@ -63,5 +66,28 @@ cog_categories <- function(type = NULL, pattern = NULL) {
    "GROUP BY category, category_type, subtype
     ORDER BY category_type, category, subtype"
  )
-  tibble::as_tibble(DBI::dbGetQuery(con, sql))
+  out <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
+
+  # The reserved pseudo-category is a query mode, not a crosswalk row, so it
+  # has no item codes to report -- hence NA rather than 0 for n_codes. It is
+  # emitted for the two FLOW vocabularies only: cog_balances() returns a stock
+  # and has no concept argument to sum within.
+  pseudo <- tibble::tibble(
+    category      = .ALL_CATEGORIES,
+    category_type = c("expenditure", "revenue"),
+    subtype       = NA_character_,
+    n_codes       = NA_integer_,
+    item_codes    = NA_character_
+  )
+  if (!is.null(type)) {
+    db_type <- if (type == "spending") "expenditure" else type
+    pseudo <- pseudo[pseudo$category_type == db_type, , drop = FALSE]
+  }
+  if (!is.null(pattern) && nrow(pseudo) > 0L) {
+    keep <- grepl(pattern, pseudo$category, ignore.case = TRUE)
+    pseudo <- pseudo[keep, , drop = FALSE]
+  }
+  if (nrow(pseudo) == 0L) return(out)
+  out <- rbind(out, pseudo)
+  out[order(out$category_type, out$category, out$subtype), , drop = FALSE]
 }
@@ -5,7 +5,18 @@
 .uscogdata_env <- new.env(parent = emptyenv())

 .uscogdata_defaults <- list(
-  url               = "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/",
+  # Public HuggingFace mirror of the published corpus: CC-BY-4.0, no
+  # credential, CDN-backed. This is the default so `library(uscogdata)`
+  # followed by a verb works with zero configuration -- previously the
+  # default was a REPLACE_WITH_SHARE_TOKEN sentinel and no document in the
+  # package supplied a working URL, so a new user had no path to a session.
+  #
+  # The trailing slash is required: every consumer concatenates onto this
+  # (see .resolve_url(), which enforces it anyway).
+  #
+  # Override with USCOGDATA_URL or options(uscogdata.url=) to read a
+  # Nextcloud share or a local copy made by cog_mirror().
+  url               = "https://huggingface.co/datasets/civilytics/us-cog-finance/resolve/main/",
  cache_dir         = NULL,
  manifest_ttl_secs = 3600L
 )
@@ -11,6 +11,30 @@
 #'   returns `result` invisibly for chaining. `"list"` returns the raw
 #'   provenance list (identical to `attr(result, "provenance")`).
 #' @return Either `result` (invisibly) or the provenance list.
+#' @section Two kinds of series break:
+#' Catalogued breaks reach you without being asked for, in two disjoint
+#' fields, because a caveat about one series and a caveat about the whole
+#' corpus are different claims:
+#'
+#' * **`series_break_refs`** — breaks matched against the item codes actually
+#'   present in this result. A break in one code you queried.
+#' * **`corpus_break_refs`** — breaks catalogued with `fin_code = "ALL"`,
+#'   which are statements about the corpus rather than about any one code:
+#'   dollar precision across the 1976/1977 boundary (`SB085`), imputation
+#'   exclusion from FY2002 (`SB087`), the FY2012 dense-to-sparse
+#'   representation change (`SB194`), and the FY2017 government-identifier
+#'   change (`SB086`). These are selected on the break-year window alone.
+#'
+#' `SB194` is the one most likely to matter: a query spanning FY2011 to FY2012
+#' crosses the boundary where an absent cell stops meaning "Census published
+#' $0" and starts meaning "not reported".
+#' @section Other provenance blocks:
+#' `transformations$units_conversion` records the `$1,000s`-to-dollars
+#' multiply that every amount column has already had applied.
+#' `transformations$per_capita` records the population denominator and its
+#' year range. `coverage` and `coverage_mode` appear on multi-government
+#' results (see [cog_geographic_rollup()]). `completion` appears when
+#' `complete = TRUE`. `balance_caveats` appears on [cog_balances()] results.
 #' @export
 cog_explain <- function(result, format = c("print", "list")) {
  format <- match.arg(format)
@@ -28,7 +28,7 @@
      "*" = "{.code Sys.setenv(USCOGDATA_URL = \"<url-or-local-path>/\")}",
      "*" = "{.code options(uscogdata.url = \"<url-or-local-path>/\")}",
      i = "For an offline smoke test, use the bundled fixture: {.code system.file(\"extdata/fixture_corpus\", package = \"uscogdata\")}.",
-      i = "For the live Civilytics corpus, request the Nextcloud share URL from the package maintainer."
+      i = "The public corpus is the default: unset USCOGDATA_URL to use it, or point it at a local copy made by {.code cog_mirror()}."
    ), class = "uscogdata_url_not_configured")
  }
  invisible(url)
@@ -240,6 +240,23 @@ cog_find_peers <- function(target_govid,
 #'     group_by(year) |>
 #'     summarise(p50 = quantile(total, 0.5, na.rm = TRUE))
 #'   ```
+#' @section Reading `coverage`:
+#' `provenance$coverage` reports `n_units_reporting` against
+#' `n_units_expected` per year. **`n_units_reporting` is category-conditional:
+#' it counts cohort members with rows for the category you asked for, not
+#' cohort members collected that year.** A government that was surveyed and
+#' genuinely spends nothing in that category is indistinguishable here from one
+#' that was never surveyed.
+#'
+#' The ratio is therefore **not a response rate** and must not be used as one.
+#' In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
+#' `category = "Police"`; the 174-city gap is overwhelmingly cities that
+#' contract policing to the county sheriff, not non-response.
+#'
+#' The comparison that *is* valid is the same category across a census year
+#' (ending in 2 or 7) and a sample year, where the real-zero component is
+#' roughly constant and the difference reflects the survey cycle. `is_census_year`
+#' marks which is which.
 #' @export
 cog_peer_compare <- function(target_govid, peers, category, years,
                             per_capita = TRUE, adjust_to_year = NULL,
@@ -67,7 +67,15 @@
    basis_note = basis_note,
    expenditure_concept = expenditure_concept,
    expenditure_concept_note = expenditure_concept_note,
-    expenditure_concept_direct_suppressed = isTRUE(expenditure_concept_direct_suppressed),
+    # isTRUE() alone would collapse a deliberate NA (all-categories mode,
+    # where suppression detection cannot run -- see .verb_spendrev()) down to
+    # FALSE, turning "we don't know" back into the false claim this field
+    # exists to avoid. Preserve NA; otherwise normalize to a strict logical.
+    expenditure_concept_direct_suppressed = if (isTRUE(is.na(expenditure_concept_direct_suppressed))) {
+      NA
+    } else {
+      isTRUE(expenditure_concept_direct_suppressed)
+    },
    revenue_concept = revenue_concept,
    harmonization = harmonization %||% list(
      applied = FALSE, na_rows_excluded = 0L, na_amount_excluded = 0,
@@ -8,6 +8,17 @@
 #' multiplies by 1000 and records the conversion in `provenance`).
 #'
 #' @inheritParams cog_spending
+#' @param category Character vector of category names (from
+#'   `summary_categories.category`), or `NULL` for all categories broken out
+#'   one row each. The reserved value `"All Categories"` instead returns a
+#'   single summed row per `(year, canonical_govid, subtype)`, covering every
+#'   category inside the requested concept's subtype scope. It cannot be
+#'   combined with other category names, and it is not the same thing as
+#'   `revenue_concept = "total"`: the concept chooses which subtypes are in
+#'   scope, `"All Categories"` chooses whether rows inside that scope are
+#'   broken out or summed. Because the result keeps one row per
+#'   `revenue_subtype`, filtering the returned frame to
+#'   `revenue_subtype == "own_source"` gives an own-source revenue total.
 #' @param revenue_concept Which of Census's two published revenue concepts to
 #'   return. Concepts are defined as sets of the crosswalk's `revenue_subtype`
 #'   values -- never as item-code first letters, which cannot classify
@@ -43,7 +54,7 @@ cog_revenue <- function(govid, years, category = NULL,
                        per_capita = FALSE, adjust_to_year = NULL,
                        basis = c("harmonized", "raw"), recipe = NULL,
                        revenue_concept = c("general", "total"),
-                        complete = FALSE) {
+                        complete = FALSE, limit = NULL, offset = NULL) {
  # flow_prefixes no longer classifies rows (crosswalk revenue_subtype
  # membership does -- General Revenue, i.e. everything except
  # insurance_trust) -- it only scopes the recipe-suggestion machinery to
@@ -62,6 +73,8 @@ cog_revenue <- function(govid, years, category = NULL,
    basis          = basis,
    recipe         = recipe,
    revenue_concept = revenue_concept,
-    complete       = complete
+    complete       = complete,
+    limit          = limit,
+    offset         = offset
  )
 }
@@ -19,7 +19,11 @@
 #'   `state`, `county`, `city`. Each element is a character vector of
 #'   `canonical_govid` values. At least one layer required.
 #' @param category Single category name or character vector (passed through
-#'   to [cog_spending()]).
+#'   to [cog_spending()]), or the reserved `"All Categories"` for one summed
+#'   row per `(year, canonical_govid, subtype)` covering every category in the
+#'   concept's scope. `"All Categories"` is the efficient way to build a
+#'   geographic total: without it a caller must issue one rollup per category
+#'   and sum the results themselves.
 #' @param years Integer vector of years.
 #' @param per_capita If `TRUE`, per-capita uses each gov's own per-year
 #'   population from `gov_population_yearly`. Govs with missing population
@@ -56,6 +60,23 @@
 #'   `codes_included`, `aggregate_fallback`, `scope_note`, `notes`. Carries a
 #'   `provenance` attribute with `verb = "cog_geographic_rollup"`, `layers`,
 #'   and `rollup$included_govids` / `rollup$excluded_govids`.
+#' @section Reading `coverage`:
+#' `provenance$coverage` reports `n_units_reporting` against
+#' `n_units_expected` per year. **`n_units_reporting` is category-conditional:
+#' it counts governments with rows for the category you asked for, not
+#' governments collected that year.** A government that was surveyed and
+#' genuinely spends nothing in that category is indistinguishable here from one
+#' that was never surveyed.
+#'
+#' The ratio is therefore **not a response rate** and must not be used as one.
+#' In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
+#' `category = "Police"`; the 174-city gap is overwhelmingly cities that
+#' contract policing to the county sheriff, not non-response.
+#'
+#' The comparison that *is* valid is the same category across a census year
+#' (ending in 2 or 7) and a sample year, where the real-zero component is
+#' roughly constant and the difference reflects the survey cycle. `is_census_year`
+#' marks which is which.
 #' @export
 cog_geographic_rollup <- function(govids, category, years,
                                  per_capita = FALSE, adjust_to_year = NULL,
@@ -19,6 +19,14 @@
 .spend_subtypes_primary <- c("operations", "capital", "assistance")
 .spend_subtypes_direct  <- c(.spend_subtypes_primary, "interest", "insurance_benefits")

+# The reserved pseudo-category. Deliberately NOT "Total": `category = "Total"`
+# would sit one argument away from `expenditure_concept = "total"` and mean
+# something different -- the concept selects WHICH SUBTYPES are in scope, this
+# selects whether the rows inside that scope are broken out by category or
+# summed. "All Categories" states the operation and cannot be misread as the
+# concept.
+.ALL_CATEGORIES <- "All Categories"
+
 #' @noRd
 .expenditure_concept_subtypes <- function(concept) {
  switch(concept,
@@ -63,7 +71,16 @@
 #' @param govid Character vector of `canonical_govid` values.
 #' @param years Integer vector of years.
 #' @param category Character vector of category names (from
-#'   `summary_categories.category`), or `NULL` for all categories.
+#'   `summary_categories.category`), or `NULL` for all categories broken out
+#'   one row each. The reserved value `"All Categories"` instead returns a
+#'   single summed row per `(year, canonical_govid, subtype)`, covering every
+#'   category inside the requested concept's subtype scope. It cannot be
+#'   combined with other category names, and it is not the same thing as
+#'   `expenditure_concept = "total"`: the concept chooses which subtypes are in
+#'   scope, `"All Categories"` chooses whether rows inside that scope are
+#'   broken out or summed. Because the result keeps one row per
+#'   `spend_subtype`, filtering the returned frame to
+#'   `spend_subtype == "operations"` gives an operating-expenditure total.
 #' @param per_capita If `TRUE`, adds `amt_per_capita_nominal` (and
 #'   `amt_per_capita_real` when `adjust_to_year` is set) using the per-year
 #'   Census F-33 population from `gov_population_yearly`. Result also gains
@@ -133,7 +150,12 @@
 #'   component (when one exists), and
 #'   `provenance$expenditure_concept_direct_suppressed` is `TRUE` -- the
 #'   figure in those rows is the intergovernmental leg alone, not Direct +
-#'   IG.
+#'   IG. When `category = "All Categories"` is combined with
+#'   `expenditure_concept = "total"`, this detection cannot run (it keys on
+#'   per-category rows, which all-categories mode collapses to one literal
+#'   value), so `expenditure_concept_direct_suppressed` is `NA` rather than a
+#'   possibly-false `FALSE`; query an explicit `category` to get a real
+#'   answer.
 #' @param complete If `TRUE`, fill the requested grid so that a cell the
 #'   corpus does not carry still appears, labelled with **why** it is
 #'   missing, and add a `value_source` column to every row:
@@ -156,6 +178,13 @@
 #'   `recipe` or with `expenditure_concept = "total"` (class
 #'   `uscogdata_complete_unsupported`) — neither draws its cells from
 #'   `code_set`.
+#' @param limit Maximum number of result rows to return, pushed into the SQL
+#'   query itself (`LIMIT`/`OFFSET`) rather than applied after the full
+#'   result is materialized. `NULL` (the default) returns every matching row,
+#'   exactly as before this parameter existed. Mutually exclusive with
+#'   `recipe` and with `complete = TRUE` -- see `offset` and `total_rows`.
+#' @param offset Rows to skip before `limit` starts counting (0-based).
+#'   Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.
 #' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
 #'   `spend_subtype`, `category`, `amt_nominal`, optional `amt_real`,
 #'   optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
@@ -163,13 +192,17 @@
 #'   and `value_source` when `complete = TRUE`.
 #'   Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`,
 #'   whose `completion` block reports `applied`, `rows_filled`, and the
-#'   per-year `absence_means` rule that was applied.
+#'   per-year `absence_means` rule that was applied. When `limit` is set,
+#'   also carries a `total_rows` attribute: the full unpaginated row count,
+#'   computed by the same query (`COUNT(*) OVER()`) rather than a second
+#'   round trip -- so a caller walking pages never has to ask "how many are
+#'   there" separately.
 #' @export
 cog_spending <- function(govid, years, category = NULL,
                         per_capita = FALSE, adjust_to_year = NULL,
                         basis = c("harmonized", "raw"), recipe = NULL,
                         expenditure_concept = c("primary", "direct", "total"),
-                         complete = FALSE) {
+                         complete = FALSE, limit = NULL, offset = NULL) {
  # flow_prefixes no longer classifies rows (crosswalk subtype membership
  # does, per expenditure_concept) -- it only scopes the recipe-suggestion
  # machinery to this verb's recipe families (see R/suggestions.R; the
@@ -188,7 +221,9 @@ cog_spending <- function(govid, years, category = NULL,
    basis         = basis,
    recipe        = recipe,
    expenditure_concept = expenditure_concept,
-    complete      = complete
+    complete      = complete,
+    limit         = limit,
+    offset        = offset
  )
 }

@@ -216,7 +251,7 @@ cog_spending <- function(govid, years, category = NULL,
                           basis = c("harmonized", "raw"), recipe = NULL,
                           expenditure_concept = c("primary", "direct", "total"),
                           revenue_concept = c("general", "total"),
-                           complete = FALSE) {
+                           complete = FALSE, limit = NULL, offset = NULL) {
  basis_explicit <- length(basis) == 1L
  basis <- match.arg(basis, c("harmonized", "raw"))
  # match.arg() itself throws a base `simpleError`, not an rlang-classed
@@ -257,8 +292,24 @@ cog_spending <- function(govid, years, category = NULL,
  }

  govid <- .coerce_govid_input(govid, arg = "govid")
+  # allow_all_categories = TRUE: cog_spending()/cog_revenue() are the two
+  # verbs the reserved pseudo-category is defined for. cog_balances() shares
+  # this validator but leaves the argument at its FALSE default, so it
+  # rejects "All Categories" instead of silently returning zero rows
+  # (finding 3, all-categories review).
  .validate_verb_inputs(govid, years, category, per_capita, adjust_to_year,
-                        recipe)
+                        recipe, allow_all_categories = TRUE)
+
+  # Recognize the reserved pseudo-category. Detected after type validation so a
+  # non-character `category` still fails with the ordinary type error.
+  all_categories <- !is.null(category) && .ALL_CATEGORIES %in% category
+  if (all_categories && length(category) > 1L) {
+    cli::cli_abort(c(
+      "{.val {(.ALL_CATEGORIES)}} cannot be combined with other categories.",
+      "i" = "It already sums every category in the requested concept's scope.",
+      "*" = "Ask for it alone, or list the specific categories you want."
+    ), class = "uscogdata_all_categories_not_combinable")
+  }

  if (!is.null(recipe) && identical(expenditure_concept, "total")) {
    cli::cli_abort(c(
@@ -299,6 +350,44 @@ cog_spending <- function(govid, years, category = NULL,
      "Use `expenditure_concept = \"direct\"` with `complete = TRUE`, or drop `complete`."
    )
  }
+  if (complete && all_categories) {
+    .abort_complete_unsupported(
+      "`category = \"All Categories\"` collapses the category dimension that `code_set` grids over (see `.completion_grid_sql()`), so there is no per-category grid left to fill -- filling a summed row has no defined semantics.",
+      "Drop `complete`, or use `complete = TRUE` with an explicit `category` (or `category = NULL` for every category)."
+    )
+  }
+
+  # limit/offset push the page into the SQL itself (see .build_verb_sql()),
+  # so the two things that would make "a page of what" ambiguous are refused
+  # up front rather than silently ignored: complete = TRUE fills a grid over
+  # the FULL requested (year, category) space, and a recipe's result comes
+  # from .run_recipe()'s own query, which this function does not touch.
+  if (!is.null(limit)) {
+    limit <- as.integer(limit)
+    if (length(limit) != 1L || is.na(limit) || limit < 0L) {
+      cli::cli_abort("`limit` must be a single non-negative integer.",
+                     class = "uscogdata_invalid_pagination")
+    }
+    offset <- if (is.null(offset)) 0L else as.integer(offset)
+    if (length(offset) != 1L || is.na(offset) || offset < 0L) {
+      cli::cli_abort("`offset` must be a single non-negative integer.",
+                     class = "uscogdata_invalid_pagination")
+    }
+    if (complete) {
+      cli::cli_abort(c(
+        "`limit`/`offset` cannot be combined with `complete = TRUE`.",
+        "i" = "`complete` fills a grid over the FULL requested (year, category) space; paginating a slice of already-grouped rows has no defined meaning for the cells it would fill.",
+        "*" = "Drop `limit`/`offset`, or drop `complete`."
+      ), class = "uscogdata_complete_pagination_conflict")
+    }
+    if (!is.null(recipe)) {
+      cli::cli_abort(c(
+        "`limit`/`offset` cannot be combined with `recipe`.",
+        "i" = "A recipe's result comes from a separate query (`.run_recipe()`) that pagination is not wired into yet.",
+        "*" = "Drop `limit`/`offset`, or drop `recipe`."
+      ), class = "uscogdata_recipe_pagination_conflict")
+    }
+  }

  years   <- as.integer(years)
  if (!is.null(adjust_to_year)) adjust_to_year <- as.integer(adjust_to_year)
@@ -312,6 +401,7 @@ cog_spending <- function(govid, years, category = NULL,

  recipe_block <- NULL
  category_for_prov <- category
+  total_rows <- NULL  # set below only when limit is non-NULL (non-recipe path)
  if (!is.null(recipe)) {
    .require_schema_v5(con, manifest, "recipe =")
    .validate_recipe_id(con, recipe)
@@ -333,9 +423,32 @@ cog_spending <- function(govid, years, category = NULL,
    } else {
      NULL
    }
-    sql <- .build_verb_sql(view, subtype_col, govid, years, category, ig_view,
-                           subtype_scope)
+    sql <- .build_verb_sql(view, subtype_col, govid, years,
+                           if (all_categories) NULL else category,
+                           ig_view, subtype_scope,
+                           all_categories = all_categories,
+                           limit = limit, offset = offset)
    result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
+    if (!is.null(limit)) {
+      # COUNT(*) OVER() rides along as an ordinary column so the total comes
+      # from the same scan when this page has any rows -- see
+      # .build_verb_sql(). An empty page (offset past the end) carries no
+      # such row to read it from, so that one case falls back to a second,
+      # unpaginated COUNT(*) query rather than reporting a wrong zero.
+      if (nrow(result) > 0L) {
+        total_rows <- result$pagination_total_rows[[1]]
+        result$pagination_total_rows <- NULL
+      } else {
+        count_sql <- sprintf(
+          "SELECT COUNT(*) AS n FROM (%s) AS _uncounted",
+          .build_verb_sql(view, subtype_col, govid, years,
+                          if (all_categories) NULL else category,
+                          ig_view, subtype_scope,
+                          all_categories = all_categories)
+        )
+        total_rows <- as.integer(DBI::dbGetQuery(con, count_sql)$n[[1]])
+      }
+    }
  }

  # Fill BEFORE per_capita / inflation so the added cells get the same
@@ -396,7 +509,10 @@ cog_spending <- function(govid, years, category = NULL,
    suggestions <- .build_suggestions(con, govid, years, category,
                                       direct_leg_result,
                                       resolved$basis, flow_prefixes,
-                                       .select_long_view(view_base, resolved$basis))
+                                       .select_long_view(view_base, resolved$basis),
+                                       all_categories = all_categories,
+                                       subtype_col = subtype_col,
+                                       subtype_scope = subtype_scope)
  }

  # C1(b): when expenditure_concept = "total", flag any row where the IG
@@ -408,13 +524,32 @@ cog_spending <- function(govid, years, category = NULL,
  # direct spending in that category, which is correct, ordinary data). When
  # a covering recipe is found, both the row-level notes and the provenance
  # say so rather than pass silently as a plausible Total.
-  direct_suppressed_info <- if (identical(expenditure_concept, "total")) {
+  #
+  # In all-categories mode this cannot run at all: .detect_direct_suppressed()
+  # keys on (year, canonical_govid, category), and every row shares the same
+  # literal "All Categories" value, so the key collides across every real
+  # category for that (year, govid) -- an IG-only row for a suppressed
+  # category becomes indistinguishable from one sharing a key with an
+  # unrelated category's ordinary Direct row. `has_direct` would then read
+  # TRUE whenever the government has ANY direct spending at all, and the
+  # detector could never fire. Rather than run it and report a false FALSE,
+  # skip it and record NA -- the provenance must stop making a claim it
+  # cannot support (finding 1, all-categories review).
+  suppression_unavailable <- all_categories &&
+    identical(expenditure_concept, "total")
+  direct_suppressed_info <- if (suppression_unavailable) {
+    list(flag = rep(NA, nrow(result)), notes = rep(NA_character_, nrow(result)))
+  } else if (identical(expenditure_concept, "total")) {
    .detect_direct_suppressed(con, result, subtype_col)
  } else {
    list(flag = rep(FALSE, nrow(result)), notes = rep(NA_character_, nrow(result)))
  }
  direct_suppressed <- direct_suppressed_info$flag
-  direct_suppressed_flag <- isTRUE(any(direct_suppressed))
+  direct_suppressed_flag <- if (suppression_unavailable) {
+    NA
+  } else {
+    isTRUE(any(direct_suppressed))
+  }

  result$notes <- .notes_column(result, direct_suppressed_info$notes)

@@ -423,9 +558,20 @@ cog_spending <- function(govid, years, category = NULL,
  # leg is suppressed for at least one requested (year, category), append an
  # explicit warning rather than let the base note's "Total = Direct + IG"
  # framing stand unqualified for rows where that arithmetic didn't happen.
+  # When suppression detection itself is unavailable (all-categories mode),
+  # say so instead of silently reusing the unqualified base note.
  expenditure_concept_note_for_prov <- if (identical(expenditure_concept, "total")) {
    base_note <- "Total = Direct + intergovernmental (M to local govts + L to state govts). Legacy-era IG is assembled from aggregate-flagged rows, which are year-disjoint from their modern leaf components; the L-- family total is excluded."
-    if (direct_suppressed_flag) {
+    if (suppression_unavailable) {
+      paste0(
+        base_note,
+        " NOTE: direct-leg-suppression detection is unavailable when ",
+        "`category = \"All Categories\"` -- it keys on per-category rows, ",
+        "which this mode collapses. `expenditure_concept_direct_suppressed` ",
+        "is NA here rather than a possibly-false FALSE; query an explicit ",
+        "`category` (or `category = NULL`) to get a real answer."
+      )
+    } else if (isTRUE(direct_suppressed_flag)) {
      paste0(
        base_note,
        " NOTE: for at least one requested (year, category) the Direct leg ",
@@ -467,15 +613,32 @@ cog_spending <- function(govid, years, category = NULL,
  prov$scope$govids_missing <- scope$missing
  attr(result, "provenance") <- prov
  attr(result, ".popyear_range") <- NULL
+  # Attached here, after every downstream transform (per_capita/real-dollar
+  # joins, notes, subtype filtering), the same way provenance is -- an
+  # attribute set before those runs is not guaranteed to survive them.
+  if (!is.null(limit)) attr(result, "total_rows") <- total_rows

  if (length(suggestions) > 0L) .inform_suggestions(suggestions)

  result
 }

+#' Shared input validation for the money/holdings verbs.
+#'
+#' `allow_all_categories` gates the reserved pseudo-category
+#' `.ALL_CATEGORIES` ("All Categories"). It is meaningful only where a
+#' concept's subtype scope defines what "all" sums over --
+#' `cog_spending()`/`cog_revenue()`, via `.verb_spendrev()`, pass `TRUE`.
+#' `cog_balances()` leaves it at the `FALSE` default: holdings are a stock
+#' with no concept vocabulary to sum across (see R/balances.R), and before
+#' this guard existed `cog_balances(category = "All Categories")` silently
+#' matched zero crosswalk rows and returned an empty result with no error
+#' (finding 3, all-categories review). This validator is shared specifically
+#' so the three verbs cannot drift apart on this again.
 #' @noRd
 .validate_verb_inputs <- function(govid, years, category,
-                                  per_capita, adjust_to_year, recipe = NULL) {
+                                  per_capita, adjust_to_year, recipe = NULL,
+                                  allow_all_categories = FALSE) {
  if (!is.character(govid) || length(govid) == 0L) {
    cli::cli_abort("`govid` must be a non-empty character vector.")
  }
@@ -485,6 +648,14 @@ cog_spending <- function(govid, years, category = NULL,
  if (!is.null(category) && !is.character(category)) {
    cli::cli_abort("`category` must be character or NULL.")
  }
+  if (!allow_all_categories && !is.null(category) &&
+      .ALL_CATEGORIES %in% category) {
+    cli::cli_abort(c(
+      "{.val {(.ALL_CATEGORIES)}} is not supported here.",
+      i = "It sums a spending or revenue concept's subtype scope; this verb has no concept vocabulary to sum across.",
+      i = "Use {.fn cog_spending} or {.fn cog_revenue} for an all-categories total."
+    ), class = "uscogdata_all_categories_unsupported")
+  }
  if (!is.logical(per_capita) || length(per_capita) != 1L) {
    cli::cli_abort("`per_capita` must be a length-1 logical.")
  }
@@ -570,10 +741,16 @@ cog_spending <- function(govid, years, category = NULL,

 #' @noRd
 .build_verb_sql <- function(view, subtype_col, govid, years, category,
-                            ig_view = NULL, subtype_scope = NULL) {
+                            ig_view = NULL, subtype_scope = NULL,
+                            all_categories = FALSE, limit = NULL, offset = NULL) {
  govid_lit <- .sql_lit_chr(govid)
  years_lit <- paste(as.integer(years), collapse = ",")
-  category_pred <- if (is.null(category)) {
+  # In all-categories mode there is no category filter: the sum is defined by
+  # the concept's SUBTYPE allowlist (subtype_pred below), which is the real
+  # concept boundary. Filtering by category as well would be a no-op at best
+  # and, if the crosswalk ever gained an uncategorized code, a silent
+  # under-count of the very total this mode exists to guarantee.
+  category_pred <- if (all_categories || is.null(category)) {
    ""
  } else {
    sprintf("AND category IN (%s)", .sql_lit_chr(category))
@@ -612,13 +789,26 @@ cog_spending <- function(govid, years, category = NULL,
  # though its dollars came entirely from an aggregate row, silently
  # suppressing the "Aggregate fallback applied" note on exactly the rows
  # this feature exists to surface.
-  sprintf(
+
+  # Collapse the category dimension. subtype is deliberately KEPT: it is what
+  # lets a caller filter the result to `spend_subtype == "operations"` and
+  # get an operating-expenditure total, the measure a fiscal comparison
+  # actually wants. (There is no `subtype` argument -- this is a post-hoc
+  # filter on the returned column, not a query parameter.)
+  category_select <- if (all_categories) {
+    sprintf("%s AS category", .sql_lit_chr(.ALL_CATEGORIES))
+  } else {
+    "category"
+  }
+  category_group <- if (all_categories) "" else ", category"
+
+  base_sql <- sprintf(
    "SELECT
       year,
       canonical_govid,
       COALESCE(xwalk_gov_name, gov_name) AS gov_name,
       %1$s,
-       category,
+       %7$s,
       SUM(amt) * 1000.0 AS amt_nominal,
       string_agg(DISTINCT item_code, ',' ORDER BY item_code) AS codes_included,
       bool_or(is_aggregate) AS aggregate_fallback
@@ -627,10 +817,32 @@ cog_spending <- function(govid, years, category = NULL,
       AND year IN (%4$s)
       %5$s
       %6$s
-     GROUP BY year, canonical_govid, gov_name, xwalk_gov_name, %1$s, category
-     ORDER BY year, canonical_govid, %1$s, category",
-    subtype_col, source_expr, govid_lit, years_lit, category_pred, subtype_pred
+     GROUP BY year, canonical_govid, gov_name, xwalk_gov_name, %1$s%8$s
+     ORDER BY year, canonical_govid, %1$s%8$s",
+    subtype_col, source_expr, govid_lit, years_lit, category_pred, subtype_pred,
+    category_select, category_group
  )
+
+  # limit/offset push the page into the query itself instead of pulling every
+  # matching row across the network only to slice and discard most of it
+  # afterward (the pattern behind the 2026-08-06 production incident: a
+  # 193,105-row/194-page sweep re-ran the full query and re-listified every
+  # row on EVERY page). COUNT(*) OVER() rides along as an ordinary column so
+  # the caller gets the true total from this same scan -- see the call site
+  # in .verb_spendrev(), which reads it off row 1 and strips it back out.
+  # The outer SELECT * wrapping (rather than appending LIMIT/OFFSET directly
+  # to base_sql) is what makes COUNT(*) OVER() see the post-GROUP-BY row
+  # count, not the pre-aggregation one.
+  if (is.null(limit)) {
+    base_sql
+  } else {
+    sprintf(
+      "SELECT *, COUNT(*) OVER() AS pagination_total_rows
+       FROM (%s) AS _paged
+       LIMIT %d OFFSET %d",
+      base_sql, limit, offset
+    )
+  }
 }

 #' @noRd
@@ -59,12 +59,36 @@
 #' @param long_view Name of the verb's own long view (from
 #'   `.select_long_view()`), passed through to `.suppressed_components()` to
 #'   measure the second qualifying path (uscogdata#9).
+#' @param all_categories `TRUE` when the caller's `category` is the reserved
+#'   pseudo-category (`.ALL_CATEGORIES`). Defaults to `FALSE` so no other
+#'   caller's behaviour changes. When `TRUE`, the candidate-recipe sub-select
+#'   is scoped by `subtype_col`/`subtype_scope` instead of by `category` --
+#'   symmetric with `.build_verb_sql()`'s own all-categories branch (see
+#'   R/spending.R): the concept's subtype allowlist is the real scope
+#'   boundary, not any literal category value, and
+#'   `.ALL_CATEGORIES` ("All Categories") is never itself a row in
+#'   `summary_categories.category`, so leaving the category-keyed sub-select
+#'   in place here always returned zero candidates and silently disabled
+#'   signposting in all-categories mode (final whole-branch review, finding
+#'   6).
+#' @param subtype_col Name of the `summary_categories` subtype column to
+#'   scope by when `all_categories = TRUE` (`"spend_subtype"` or
+#'   `"revenue_subtype"` -- the same value `.build_verb_sql()` already
+#'   receives as its own `subtype_col`). Ignored when `all_categories =
+#'   FALSE`. `NULL` by default.
+#' @param subtype_scope Character vector of subtype values to scope by when
+#'   `all_categories = TRUE` (the same value `.build_verb_sql()` already
+#'   receives as its own `subtype_scope` -- the concept's subtype allowlist,
+#'   e.g. `.expenditure_concept_subtypes(expenditure_concept)`). Ignored when
+#'   `all_categories = FALSE`. `NULL` by default.
 #' @return List of `list(recipe_id, label, available_years, hint,
 #'   ig_recipe_id, trigger, suppressed_amount, suppressed_years,
 #'   suppressed_codes)`, possibly empty.
 #' @noRd
 .build_suggestions <- function(con, govid, years, category, result, basis,
-                                flow_prefixes, long_view) {
+                                flow_prefixes, long_view,
+                                all_categories = FALSE,
+                                subtype_col = NULL, subtype_scope = NULL) {
  if (!identical(basis, "harmonized") || is.null(category)) return(list())

  # Exclude any recipe that is ITSELF an intergovernmental (M/L) recipe --
@@ -79,16 +103,35 @@
  # flow-prefix gate below/in `.attach_ig_counterparts()`: an M/L recipe
  # should never be suggested as a coverage-gap filler for EITHER verb, not
  # just kept from being named as the *counterpart* of another suggestion.
+  #
+  # The inner sub-select is the concept boundary (finding 6, final
+  # whole-branch review): in all-categories mode it is scoped by
+  # `subtype_col`/`subtype_scope` -- the same allowlist `.build_verb_sql()`
+  # applies as a WHERE predicate to make the summed result a *concept*, not
+  # by `category` (`.ALL_CATEGORIES` is never a row in
+  # `summary_categories.category`, so a category-keyed sub-select always
+  # came back empty here). The M/L exclusion below is unchanged either way.
+  candidate_scope_sql <- if (isTRUE(all_categories)) {
+    sprintf(
+      "SELECT DISTINCT item_code FROM summary_categories WHERE %s IN (%s)",
+      subtype_col, .sql_lit_chr(subtype_scope)
+    )
+  } else {
+    sprintf(
+      "SELECT DISTINCT item_code FROM summary_categories WHERE category IN (%s)",
+      .sql_lit_chr(category)
+    )
+  }
  candidates <- DBI::dbGetQuery(con, sprintf(
    "SELECT DISTINCT recipe_id FROM harmonization_recipes
     WHERE component_code IN (
-       SELECT DISTINCT item_code FROM summary_categories WHERE category IN (%s)
+       %s
     )
     AND recipe_id NOT IN (
       SELECT DISTINCT recipe_id FROM harmonization_recipes
       WHERE LEFT(component_code, 1) IN ('M', 'L')
     )",
-    .sql_lit_chr(category)
+    candidate_scope_sql
  ))$recipe_id
  if (length(candidates) == 0L) return(list())

@@ -78,6 +78,55 @@
  file %in% basename(paths)
 }

+#' Build the SQL path expression for the partitioned `long` table.
+#'
+#' DuckDB cannot expand a glob over generic HTTP: there is no directory
+#' listing to expand against, and `allow_asterisks_in_http_paths` only
+#' forwards the literal `**/*` as a filename, which 404s. Measured against
+#' the published corpus on 2026-08-08, an explicit file list returns the
+#' same 46,148,034 rows the (working) `hf://` glob does, and
+#' `hive_partitioning = true` still recovers `year` from the paths.
+#'
+#' The manifest already enumerates every partition, so we build the list
+#' from it. This is host-agnostic -- Nextcloud, HuggingFace and a local
+#' fixture take the same path -- where an `hf://` URL would tie the reader
+#' to one vendor's protocol and still need special-casing, since manifest
+#' fetching goes through httr2, which cannot speak `hf://`.
+#'
+#' Falls back to the glob when the manifest carries no partition list: a
+#' hand-built manifest in a test (see test-views.R) or a corpus predating
+#' the field. Both are local, where globbing works.
+#' @noRd
+.long_files_sql <- function(url, manifest) {
+  parts <- manifest$files$long_partitions %||% list()
+  if (length(parts) == 0L) {
+    return(.sql_lit_chr(paste0(url, "data/long/**/*.parquet")))
+  }
+  paths <- vapply(parts, function(p) as.character(p$path), character(1))
+  paste0("[", .sql_lit_chr(paste0(url, paths)), "]")
+}
+
+#' Substitute the corpus-location tokens in a view's SQL text.
+#'
+#' One place knows the token vocabulary. `.register_views()` and the tests
+#' that execute a view file directly both route through here. This exists
+#' because four test sites had hand-rolled the `{url}` substitution -- one
+#' of them commented as doing it "exactly as .register_views() does" -- and
+#' every one of them broke the moment a second token was introduced.
+#'
+#' `{long_files}` must be substituted BEFORE `{url}`: it expands to a string
+#' that itself contains the url, so the reverse order leaves the token in
+#' place and DuckDB's parser fails on the brace.
+#'
+#' `manifest` defaults to empty, which routes `.long_files_sql()` to its glob
+#' fallback -- correct for the local temp corpora the direct-execution tests
+#' build.
+#' @noRd
+.render_view_sql <- function(sql, url, manifest = list()) {
+  sql <- gsub("\\{long_files\\}", .long_files_sql(url, manifest), sql, fixed = FALSE)
+  gsub("\\{url\\}", url, sql, fixed = FALSE)
+}
+
 #' Register DuckDB views from inst/sql/ SQL files
 #' @noRd
 .register_views <- function(con, url, manifest) {
@@ -91,7 +140,7 @@
        !.corpus_has_table(manifest, .representation_view_files[[base]])) next
    if (base %in% .balance_view_files && !.corpus_has_balance_subtype(con)) next
    sql <- paste(readLines(f, warn = FALSE), collapse = "\n")
-    sql <- gsub("\\{url\\}", url, sql, fixed = FALSE)
+    sql <- .render_view_sql(sql, url, manifest)
    DBI::dbExecute(con, sql)
  }
 }
@@ -1,24 +1,116 @@
 # uscogdata

-Curated R reader for the Civilytics US Census of Governments finance corpus.
+<!-- badges: start -->
+[![r-universe](https://civilytics.r-universe.dev/badges/uscogdata)](https://civilytics.r-universe.dev/uscogdata)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md)
+<!-- badges: end -->

-Provides unit-level financial profiles, geographic rollups, and peer comparisons
-with auditable provenance and built-in cross-vintage correctness. Reads the
-published corpus (Hive-partitioned parquet + manifest.json) directly from
-Nextcloud via DuckDB httpfs — no local bulk downloads required.
+A curated R reader for the Civilytics US Census of Governments finance corpus —
+every dollar that US state, county, municipal and township governments reported
+raising and spending, from **FY1967 to FY2024**, in one queryable place.

-## Status
+The Census of Governments is the only nationwide source for local government
+finance, and it is hard to use: item codes change meaning across vintages,
+government identifiers were renumbered in 2017, and an absent value means
+"published zero" in one era and "not reported" in the next. This package
+handles each of those problems, and it tells you when it has — every result
+carries provenance describing what was converted, what was aggregated, and
+which known series breaks intersect your query.

-Under active development (Phase 2 of the cog_pipeline project). See
-`../cog_pipeline/docs/reader-specification.md` for the reader contract this
-package implements.
+**Scope:** government types 0–3 (state, county, municipality, township).
+56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
+or FY1969. Special districts (type 4) and school districts (type 5) are
+excluded pending validation.

-## Installation
+## Where the data comes from
+
+The corpus is published and documented at the **[US Census of Governments
+Finance API](https://pages.civilytics.org/cog-api/)**. Start there for how the
+data was built, how the identifier and item-code reconciliation works, and what
+the corpus does and does not cover.
+
+- **[API documentation and walkthroughs](https://pages.civilytics.org/cog-api/)**
+  — reference, data dictionary, and worked examples such as the
+  [Southern states guide](https://pages.civilytics.org/cog-api/cog-api-south-guide.html)
+- **[Live API](https://cog-api.civilytics.org/api/v1/)** — the same corpus over
+  HTTP, for Tableau, Python, or anything that isn't R
+- **[Bulk corpus on Hugging Face](https://huggingface.co/datasets/civilytics/us-cog-finance)**
+  — CC-BY-4.0; the same parquet files this package reads
+- **[Census Bureau source data](https://www.census.gov/programs-surveys/gov-finances.html)**
+  — the underlying public files
+
+## Install

 ```r
-# pak::pkg_install("gitea.civilytics.org/Civilytics/uscogdata")
+install.packages("uscogdata",
+  repos = c("https://civilytics.r-universe.dev",
+            "https://cloud.r-project.org"))
 ```

+Or from source:
+
+```r
+pak::pkg_install("git::https://gitea.civilytics.org/Civilytics/uscogdata.git")
+```
+
+## Quickstart
+
+No configuration, no credentials, no download. The package reads the published
+corpus over HTTPS by default.
+
+```r
+library(uscogdata)
+
+# Resolve a place name to a canonical government id
+madison <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+madison$canonical_govid
+#> [1] "552025209777"
+
+# Police spending, inflation-adjusted and per capita
+spend <- cog_spending(
+  madison$canonical_govid,
+  years          = 2012:2022,
+  category       = "Police",
+  per_capita     = TRUE,
+  adjust_to_year = 2023
+)
+
+# What did that result do to the numbers, and what should you know about them?
+cog_explain(spend)
+```
+
+`years` is required — there is no implicit full-history default.
+
+## Two ways to read the corpus
+
+| | Remote (default) | Mirrored |
+|---|---|---|
+| Setup | none | `cog_mirror(dest)`, 190.6 MB once |
+| Disk used | **0 MB** — HTTP range requests only | 190.6 MB |
+| Per query | ~4 s (one government, one year)<br>~6 s (one government, 23 years) | local speed |
+| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
+
+Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
+works out which row groups it needs, and reads only those. Nothing is cached
+between sessions either, so every query goes back to the network.
+
+The default points at a public HuggingFace mirror of the corpus. If you would
+rather not depend on a third party — for reproducibility, for an air-gapped
+environment, or on principle — **the escape hatch is one function call**:
+
+```r
+cog_mirror("~/cog-corpus")
+Sys.setenv(USCOGDATA_URL = "~/cog-corpus/")
+```
+
+After that, nothing in your analysis touches an external service.
+
+### Configuration
+
+- `USCOGDATA_URL` — corpus root: an HTTPS URL or a local path, **trailing slash required**
+- `USCOGDATA_CACHE_DIR` — where the manifest is cached (default: user cache dir)
+- `USCOGDATA_MANIFEST_TTL_SECS` — manifest re-fetch interval (default 3600)
+
 ## Amounts are in full US dollars

 Every amount column this package returns — `amt_nominal`, `amt_real`,
@@ -29,111 +121,127 @@ own `amt` column preserves that. The verbs multiply by 1000 on the way out, so
 you never have to. The conversion is recorded in every result:

 ```r
-r <- cog_spending("552025209777", 2020L)
-attr(r, "provenance")$transformations$units_conversion
-#> $applied TRUE  $source_unit "$1,000s (raw Census)"  $target_unit "$USD"  $multiplier 1000
+attr(spend, "provenance")$transformations$units_conversion
+#> $applied     TRUE
+#> $source_unit "$1,000s (raw Census)"
+#> $target_unit "$USD"
+#> $multiplier  1000
 ```

 **Do not multiply again.** If you have read elsewhere that COG amounts are in
-`$1,000s` — true of the raw corpus, and of `cog_explorer`'s conventions doc —
-that rule does not apply to anything a `cog_*()` verb hands you. Applying it
-twice overstates every figure by 1000x, and the result looks plausible rather
-than obviously wrong.
+`$1,000s` — which is true of the raw Census files and of the corpus's own `amt`
+column — that rule does not apply to anything a `cog_*()` verb hands you.
+Applying it twice overstates every figure by 1000x, and the result looks
+plausible rather than obviously wrong.

-## Configuration
+## Concepts worth understanding before you publish a number

- `USCOGDATA_URL` — corpus root URL (public Nextcloud share, trailing slash)
- `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
- `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
-
-## Primary vs Direct vs Total spending
+### Primary vs Direct vs Total spending

 `cog_spending(..., expenditure_concept = c("primary", "direct", "total"))`
-controls whose spending a result counts. Concepts are defined as sets of the
-crosswalk's `spend_subtype` values — never item-code first letters, which
-cannot classify correctly (the letter `Y` alone spans revenue, expenditure,
-and balance codes):
+controls *whose* spending a result counts. Concepts are defined as sets of the
+crosswalk's `spend_subtype` values, never item-code first letters — the letter
+`Y` alone spans revenue, expenditure and balance codes.

- `"primary"` (the default) is the government's own service provision:
-  current operations, capital outlay, and assistance payments.
- `"direct"` is Census's published Direct Expenditure: `primary` plus
-  interest on debt and insurance trust benefit payments (e.g. pensions).
- `"total"` additionally adds the intergovernmental leg — money handed to
-  other governments to spend (`M`/`L` codes plus `Q11`/`Q12`/`Q18` state
-  payments to school systems) — which is meaningful for describing one
-  government's own budget over time, but double-counts when summed across
-  governments (a state's payment to a county is the same dollar the county
-  reports as its own direct spending).
+- **`"primary"`** (default) — the government's own service provision: current
+  operations, capital outlay, assistance payments.
+- **`"direct"`** — Census's published Direct Expenditure: `primary` plus
+  interest on debt and insurance trust benefits (e.g. pensions).
+- **`"total"`** — adds the intergovernmental leg, money handed to other
+  governments to spend. Meaningful for one government's own budget over time,
+  but it double-counts when summed across governments: a state's payment to a
+  county is the same dollar the county reports as its own direct spending.

-**Rule of thumb: any figure that spans more than one government uses
-`primary` or `direct`.** `cog_geographic_rollup()` and `cog_peer_compare()`
-enforce this by refusing `expenditure_concept = "total"`. See
-`vignette("total-spending", package = "uscogdata")` for the full
-explanation with worked examples.
+**Rule of thumb: any figure spanning more than one government uses `primary`
+or `direct`.** `cog_geographic_rollup()` and `cog_peer_compare()` enforce that
+by refusing `"total"` outright. Worked examples in
+`vignette("total-spending", package = "uscogdata")`.

-## General vs Total revenue
+### General vs Total revenue

-`cog_revenue(..., revenue_concept = c("general", "total"))` selects between
-Census's two published revenue concepts, again defined as crosswalk
-`revenue_subtype` sets rather than item-code prefixes:
+`cog_revenue(..., revenue_concept = c("general", "total"))`:

- `"general"` (the default) is Census **General Revenue**: own-source
-  (taxes, charges, miscellaneous) plus federal, state and local
-  intergovernmental aid.
- `"total"` is Census **Total Revenue**: `general` plus utility revenue
-  (`A91`–`A94`), liquor store revenue (`A90`), and insurance trust revenue
-  (unemployment and workers' compensation `Y` codes plus the
-  employee-retirement `X` codes).
+- **`"general"`** (default) — Census General Revenue: own-source taxes,
+  charges and miscellaneous, plus federal, state and local aid.
+- **`"total"`** — General plus utility revenue (`A91`–`A94`), liquor store
+  revenue (`A90`), and insurance trust revenue.

-The manual defines the first by subtracting the other three from the second,
-so the two are related by Census's own identity:
+Census defines these by its own identity:

 ```
 Total Revenue = General + Utility + Liquor Store + Insurance Trust
 ```

-Two things worth knowing before switching to `"total"`:
+Two things to know before switching to `"total"`. **Utility revenue is large
+for cities** — measured on the bundled fixture, utility plus liquor store is
+15.9% of city revenue, against 1.2% for states and 1.7% for counties. And the
+**employee-retirement (`X`) codes stop at FY2016**, when those systems moved to
+the separate Annual Survey of Public Pensions, so a `"total"` series steps down
+at the FY2016/FY2017 boundary for reasons of collection scope, not revenue
+(series breaks `SB197`–`SB209`).

- **Utility revenue is large for cities.** Measured on the bundled fixture,
-  utility plus liquor store revenue is 15.9% of city (type 2) revenue, versus
-  1.2% for states and 1.7% for counties. `general` excludes it by definition.
- **The employee-retirement (`X`) codes stop at FY2016**, when those systems
-  moved out of the annual finance file into the separate Annual Survey of
-  Public Pensions. A `"total"` series therefore steps down at the
-  FY2016/FY2017 seam for reasons of collection scope, not revenue (series
-  breaks `SB197`–`SB202`, in the corpus's `series_breaks` table).
+### Reporting coverage: the Census is only sometimes a census

-## Developer notes
+**The Census of Governments is a complete enumeration only in years ending in
+2 and 7.** Every other year is a sample, and the sample varies enormously —
+measured on the bundled fixture, Wisconsin's 608-city universe rolls up 597
+governments in FY2012 and 112 in FY2019.

-### Testing
-
-The package ships a bundled fixture corpus at `inst/extdata/fixture_corpus/` —
-a 15 MB four-year slice (2011, 2012, 2019, 2020) of the full corpus covering
-all 50 states. `tests/testthat/setup.R` automatically points `USCOGDATA_URL`
-at this fixture, so the full test suite runs offline with no network
-dependency:
+A statewide total resting on a fifth of the universe looks exactly like one
+resting on all of it, so every multi-government result now says which it is:

 ```r
-devtools::test()   # uses bundled fixture, no credentials required
+attr(rollup, "provenance")$coverage   # per-year n_units_reporting, is_census_year
 ```

-### Releasing against the live corpus
+`cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` take a
+`coverage` argument — `"all"` (default), `"census"` (census years only), or
+`"consistent"` (only units reporting in every requested year, a balanced
+panel).

-Before cutting a release, run the test suite against the published corpus to
-catch any drift between the fixture and the real data:
+`n_units_reporting` is **category-conditional**, and it is not a response rate. A government that was surveyed and genuinely spends
+nothing in the requested category is indistinguishable from one never surveyed.
+
+### Absent cells mean two different things
+
+Before FY2012, an absent cell means Census published `$0`. From FY2012 on, it
+means not reported. `cog_spending(..., complete = TRUE)` fills the requested
+grid and labels every row with which it is, via `value_source`:
+
+| `value_source` | meaning | `amt_nominal` |
+|---|---|---|
+| `reported` | the corpus carries this cell | as published |
+| `census_zero` | dense-source year (≤ FY2011), absent — Census published `$0` | `0` |
+| `not_reported` | sparse-source year (≥ FY2012), absent — unknown | `NA` |
+
+That `NA` is deliberate. Filling a modern absence with `0` would invent data.
+
+### Series breaks surface on their own
+
+Catalogued breaks that intersect your query appear in provenance whether or not
+you went looking for them — `series_break_refs` for breaks in a specific item code, and
+`corpus_break_refs` for caveats about the corpus as a whole (dollar precision
+across the 1976/1977 boundary, the FY2017 identifier change, the FY2012
+dense→sparse representation change). `cog_explain()` prints both.
+
+## How to cite

 ```r
-Sys.setenv(USCOGDATA_URL = "<published-corpus-url-with-trailing-slash>")
-devtools::test()
+citation("uscogdata")
 ```

-When the live-corpus run is clean, strip the fixture from the built package by
-adding this line to `.Rbuildignore`:
+The corpus itself is published under CC-BY-4.0. Cite it as:

-```
-^inst/extdata/fixture_corpus$
-```
+> Civilytics Consulting. US Census of Governments finance corpus.
+> https://huggingface.co/datasets/civilytics/us-cog-finance

-The test suite is URL-agnostic — `setup.R` falls back to `USCOGDATA_URL` when
-the bundled fixture is absent, so no test code changes are needed for the
-release run or after stripping the fixture.
+## Contributing
+
+Development happens on [Gitea](https://gitea.civilytics.org/Civilytics/uscogdata);
+[GitHub](https://github.com/civilytics/uscogdata) is a mirror that accepts
+issues and pull requests. See [CONTRIBUTING.md](CONTRIBUTING.md) for how a
+patch gets from there to here.
+
+## License
+
+MIT © Civilytics Consulting LLC. See [LICENSE.md](LICENSE.md).
@@ -1,4 +1,5 @@
-url: ~
+url: https://civilytics.r-universe.dev/uscogdata
+
 template:
  bootstrap: 5

@@ -15,11 +16,26 @@ reference:
      - cog_gov_search
      - cog_basket_resolution
      - cog_basket_unresolved
-  - title: Session
+  - title: Comparison & aggregation
+    desc: Peer cohorts and geographic aggregates.
    contents:
-      - has_keyword("internal")
+      - cog_find_peers
+      - cog_peer_compare
+      - cog_geographic_rollup
+  - title: Corpus metadata
+    desc: >
+      What the corpus contains, where a given result came from, and how to
+      hold a local copy of it.
+    contents:
+      - cog_categories
+      - cog_recipes
+      - cog_manifest
+      - cog_explain
+      - cog_mirror

 articles:
-  - title: Getting started
+  - title: Concepts
    navbar: ~
-    contents: []
+    contents:
+      - total-spending
+      - population-denominators
@@ -22,8 +22,8 @@
      "description": "How the intergovernmental leg was assembled; null for 'primary' and 'direct'."
    },
    "expenditure_concept_direct_suppressed": {
-      "type": "boolean",
-      "description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'primary' or 'direct'. See the affected rows' `notes` for the recovering recipe, if any."
+      "type": ["boolean", "null"],
+      "description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'primary' or 'direct'. null (NA) when expenditure_concept = 'total' AND category = 'All Categories': the detector keys on per-category rows, which that mode collapses, so suppression cannot be computed -- see `expenditure_concept_note`. See the affected rows' `notes` for the recovering recipe, if any."
    },
    "revenue_concept": {
      "type": "string",
@@ -1,3 +1,7 @@
 CREATE OR REPLACE VIEW long AS
 SELECT *
-FROM read_parquet('{url}data/long/**/*.parquet', hive_partitioning = true);
+-- {long_files} carries its own quoting: a bracketed list of every partition
+-- the manifest enumerates, or a single quoted glob on fallback. Do NOT wrap
+-- it in quotes. See .long_files_sql() in R/views.R for why a glob alone
+-- cannot work over HTTP.
+FROM read_parquet({long_files}, hive_partitioning = true);
@@ -28,7 +28,12 @@ argument: for holdings, `category` is a strict coarsening of
 every combination would be either redundant or empty.
 `category = "Fund Balances"` is exactly the `general` family
 (`W01`/`W31`/`W61`). `balance_subtype` is returned, so a finer split is
-one `dplyr::filter()` away.}
+one `dplyr::filter()` away. The reserved pseudo-category
+`"All Categories"` (see [cog_spending()]) is **not** supported here and
+errors with class `uscogdata_all_categories_unsupported`: it sums a
+concept's subtype scope, and holdings are a stock with no concept
+vocabulary to sum across. Omit `category` to get every category broken
+out instead.}

 \item{per_capita}{Divide holdings by population. Note this is a **stock per
 resident** (reserves per person), which is *not* comparable to
@@ -16,7 +16,10 @@ balance), `"spending"`, `"revenue"`, or `"balance"`.}
 \value{
 Tibble with columns `category`, `category_type`, `subtype`,
  `n_codes`, `item_codes` (comma-separated, alphabetical). Sorted by
-  `category_type`, `category`, `subtype`.
+  `category_type`, `category`, `subtype`. Includes one row per flow for the
+  reserved pseudo-category `"All Categories"`, which carries `NA` for
+  `subtype`, `n_codes` and `item_codes` because it is a query mode rather
+  than a crosswalk entry — see [cog_spending()]'s `category` argument.
 }
 \description{
 Returns the category taxonomy exposed by the corpus's
@@ -21,3 +21,33 @@ Prints the structured provenance attached to a tibble returned by any
 `cog_*` verb, or returns it as a list for downstream use (MCP tools,
 dashboards, JSON export).
 }
+\section{Two kinds of series break}{
+
+Catalogued breaks reach you without being asked for, in two disjoint
+fields, because a caveat about one series and a caveat about the whole
+corpus are different claims:
+
+* **`series_break_refs`** — breaks matched against the item codes actually
+  present in this result. A break in one code you queried.
+* **`corpus_break_refs`** — breaks catalogued with `fin_code = "ALL"`,
+  which are statements about the corpus rather than about any one code:
+  dollar precision across the 1976/1977 boundary (`SB085`), imputation
+  exclusion from FY2002 (`SB087`), the FY2012 dense-to-sparse
+  representation change (`SB194`), and the FY2017 government-identifier
+  change (`SB086`). These are selected on the break-year window alone.
+
+`SB194` is the one most likely to matter: a query spanning FY2011 to FY2012
+crosses the boundary where an absent cell stops meaning "Census published
+$0" and starts meaning "not reported".
+}
+
+\section{Other provenance blocks}{
+
+`transformations$units_conversion` records the `$1,000s`-to-dollars
+multiply that every amount column has already had applied.
+`transformations$per_capita` records the population denominator and its
+year range. `coverage` and `coverage_mode` appear on multi-government
+results (see [cog_geographic_rollup()]). `completion` appears when
+`complete = TRUE`. `balance_caveats` appears on [cog_balances()] results.
+}
+
@@ -20,7 +20,11 @@ cog_geographic_rollup(
 `canonical_govid` values. At least one layer required.}

 \item{category}{Single category name or character vector (passed through
-to [cog_spending()]).}
+to [cog_spending()]), or the reserved `"All Categories"` for one summed
+row per `(year, canonical_govid, subtype)` covering every category in the
+concept's scope. `"All Categories"` is the efficient way to build a
+geographic total: without it a caller must issue one rollup per category
+and sum the results themselves.}

 \item{years}{Integer vector of years.}

@@ -80,3 +84,23 @@ the result. The dropped govids are recorded in
 (gov type 4) and school districts (gov type 5) from per-capita rollups
 by design — see `vignette('population-denominators')`.
 }
+\section{Reading `coverage`}{
+
+`provenance$coverage` reports `n_units_reporting` against
+`n_units_expected` per year. **`n_units_reporting` is category-conditional:
+it counts governments with rows for the category you asked for, not
+governments collected that year.** A government that was surveyed and
+genuinely spends nothing in that category is indistinguishable here from one
+that was never surveyed.
+
+The ratio is therefore **not a response rate** and must not be used as one.
+In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
+`category = "Police"`; the 174-city gap is overwhelmingly cities that
+contract policing to the county sheriff, not non-response.
+
+The comparison that *is* valid is the same category across a census year
+(ending in 2 or 7) and a sample year, where the real-zero component is
+roughly constant and the difference reflects the survey cycle. `is_census_year`
+marks which is which.
+}
+
@@ -107,3 +107,23 @@ call. Those summary rows are quantiles **within each category**, not
 quantiles of each peer's total — see the `@return` section before summing
 them.
 }
+\section{Reading `coverage`}{
+
+`provenance$coverage` reports `n_units_reporting` against
+`n_units_expected` per year. **`n_units_reporting` is category-conditional:
+it counts cohort members with rows for the category you asked for, not
+cohort members collected that year.** A government that was surveyed and
+genuinely spends nothing in that category is indistinguishable here from one
+that was never surveyed.
+
+The ratio is therefore **not a response rate** and must not be used as one.
+In FY2022 — a complete census year — Georgia reports 393 of 567 cities for
+`category = "Police"`; the 174-city gap is overwhelmingly cities that
+contract policing to the county sheriff, not non-response.
+
+The comparison that *is* valid is the same category across a census year
+(ending in 2 or 7) and a sample year, where the real-zero component is
+roughly constant and the difference reflects the survey cycle. `is_census_year`
+marks which is which.
+}
+
@@ -13,7 +13,9 @@ cog_revenue(
  basis = c("harmonized", "raw"),
  recipe = NULL,
  revenue_concept = c("general", "total"),
-  complete = FALSE
+  complete = FALSE,
+  limit = NULL,
+  offset = NULL
 )
 }
 \arguments{
@@ -22,7 +24,16 @@ cog_revenue(
 \item{years}{Integer vector of years.}

 \item{category}{Character vector of category names (from
-`summary_categories.category`), or `NULL` for all categories.}
+`summary_categories.category`), or `NULL` for all categories broken out
+one row each. The reserved value `"All Categories"` instead returns a
+single summed row per `(year, canonical_govid, subtype)`, covering every
+category inside the requested concept's subtype scope. It cannot be
+combined with other category names, and it is not the same thing as
+`revenue_concept = "total"`: the concept chooses which subtypes are in
+scope, `"All Categories"` chooses whether rows inside that scope are
+broken out or summed. Because the result keeps one row per
+`revenue_subtype`, filtering the returned frame to
+`revenue_subtype == "own_source"` gives an own-source revenue total.}

 \item{per_capita}{If `TRUE`, adds `amt_per_capita_nominal` (and
 `amt_per_capita_real` when `adjust_to_year` is set) using the per-year
@@ -106,6 +117,15 @@ possibly-misleading `"harmonized"`/`"raw"` value.}
  `recipe` or with `expenditure_concept = "total"` (class
  `uscogdata_complete_unsupported`) — neither draws its cells from
  `code_set`.}
+
+\item{limit}{Maximum number of result rows to return, pushed into the SQL
+query itself (`LIMIT`/`OFFSET`) rather than applied after the full
+result is materialized. `NULL` (the default) returns every matching row,
+exactly as before this parameter existed. Mutually exclusive with
+`recipe` and with `complete = TRUE` -- see `offset` and `total_rows`.}
+
+\item{offset}{Rows to skip before `limit` starts counting (0-based).
+Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,
@@ -13,7 +13,9 @@ cog_spending(
  basis = c("harmonized", "raw"),
  recipe = NULL,
  expenditure_concept = c("primary", "direct", "total"),
-  complete = FALSE
+  complete = FALSE,
+  limit = NULL,
+  offset = NULL
 )
 }
 \arguments{
@@ -22,7 +24,16 @@ cog_spending(
 \item{years}{Integer vector of years.}

 \item{category}{Character vector of category names (from
-`summary_categories.category`), or `NULL` for all categories.}
+`summary_categories.category`), or `NULL` for all categories broken out
+one row each. The reserved value `"All Categories"` instead returns a
+single summed row per `(year, canonical_govid, subtype)`, covering every
+category inside the requested concept's subtype scope. It cannot be
+combined with other category names, and it is not the same thing as
+`expenditure_concept = "total"`: the concept chooses which subtypes are in
+scope, `"All Categories"` chooses whether rows inside that scope are
+broken out or summed. Because the result keeps one row per
+`spend_subtype`, filtering the returned frame to
+`spend_subtype == "operations"` gives an operating-expenditure total.}

 \item{per_capita}{If `TRUE`, adds `amt_per_capita_nominal` (and
 `amt_per_capita_real` when `adjust_to_year` is set) using the per-year
@@ -97,7 +108,12 @@ possibly-misleading `"harmonized"`/`"raw"` value.}
  component (when one exists), and
  `provenance$expenditure_concept_direct_suppressed` is `TRUE` -- the
  figure in those rows is the intergovernmental leg alone, not Direct +
-  IG.}
+  IG. When `category = "All Categories"` is combined with
+  `expenditure_concept = "total"`, this detection cannot run (it keys on
+  per-category rows, which all-categories mode collapses to one literal
+  value), so `expenditure_concept_direct_suppressed` is `NA` rather than a
+  possibly-false `FALSE`; query an explicit `category` to get a real
+  answer.}

 \item{complete}{If `TRUE`, fill the requested grid so that a cell the
  corpus does not carry still appears, labelled with **why** it is
@@ -121,6 +137,15 @@ possibly-misleading `"harmonized"`/`"raw"` value.}
  `recipe` or with `expenditure_concept = "total"` (class
  `uscogdata_complete_unsupported`) — neither draws its cells from
  `code_set`.}
+
+\item{limit}{Maximum number of result rows to return, pushed into the SQL
+query itself (`LIMIT`/`OFFSET`) rather than applied after the full
+result is materialized. `NULL` (the default) returns every matching row,
+exactly as before this parameter existed. Mutually exclusive with
+`recipe` and with `complete = TRUE` -- see `offset` and `total_rows`.}
+
+\item{offset}{Rows to skip before `limit` starts counting (0-based).
+Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,
@@ -130,7 +155,11 @@ Tibble with columns `year`, `canonical_govid`, `gov_name`,
  and `value_source` when `complete = TRUE`.
  Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`,
  whose `completion` block reports `applied`, `rows_filled`, and the
-  per-year `absence_means` rule that was applied.
+  per-year `absence_means` rule that was applied. When `limit` is set,
+  also carries a `total_rows` attribute: the full unpaginated row count,
+  computed by the same query (`COUNT(*) OVER()`) rather than a second
+  round trip -- so a caller walking pages never has to ask "how many are
+  there" separately.
 }
 \description{
 One row per `(year, canonical_govid, spend_subtype, category)`. Amounts are
@@ -0,0 +1,296 @@
+# `uscogdata` 0.3.0 — public release
+
+**Date:** 2026-08-08 · **Status:** design, awaiting approval
+**Scope:** release-readiness, README, NEWS. Distribution mechanics recorded here as
+decided, sequenced after the package is clean.
+
+`uscogdata` is feature-complete and the corpus it reads has been public on
+HuggingFace since 2026-08-07 (294 downloads as of this writing). The API built on
+it is live. What does not exist is a public *package*: the repo is private, there
+is no install path, and — measured, not assumed — **a stranger who installed it
+today could not read the corpus at all.**
+
+This spec covers making that untrue.
+
+## Decisions locked
+
+| Decision | Choice |
+|---|---|
+| Canonical source | `gitea.civilytics.org/Civilytics/uscogdata`, flipped public |
+| Public mirror | `github.com/civilytics/uscogdata` — issues, PRs, multi-OS check, CDN |
+| Mirror mechanism | Gitea Actions non-force `git push` (not a push mirror) |
+| Binaries | `civilytics.r-universe.dev`, registry pinned to a release tag |
+| Author of record | Jared E. Knowles `<jared@civilytics.com>`, ORCID `0000-0003-0005-9478` |
+| Copyright | Civilytics Consulting LLC (`cph`, `fnd`) |
+| License | MIT (package) · CC-BY-4.0 (corpus) |
+| Corrections intake | Deferred — see *Out of scope* |
+| Other packages | Parked until this one walks the path end to end |
+
+## P0 — the corpus is unreachable
+
+Two independent faults, either of which alone is fatal.
+
+**No corpus URL exists.** `R/config.R` defaults to the literal
+`REPLACE_WITH_SHARE_TOKEN` sentinel, and no file in the repo supplies a working
+one. A new user calling any verb gets `uscogdata_url_not_configured` with no path
+to resolution.
+
+**Remote reads are broken regardless.** Every partitioned view globs:
+
+```sql
+FROM read_parquet('{url}data/long/**/*.parquet', hive_partitioning = true)
+```
+
+DuckDB 1.5.5 refuses globs over generic HTTP. Its suggested
+`allow_asterisks_in_http_paths` escape hatch does not help — it forwards the
+literal `**/*` as a filename and 404s, because plain HTTP exposes no directory
+listing to expand against.
+
+The package therefore works only against a **local path**. That is how the API
+runs it (`CORPUS_HOST_PATH` is a host mount on maxwell) and how the tests run
+(bundled fixture), which is why the fault went unnoticed. The README's headline
+claim — *"Reads the published corpus directly from Nextcloud via DuckDB httpfs —
+no local bulk downloads required"* — is currently false.
+
+### Fix: enumerate from the manifest, do not glob
+
+`manifest.json` already lists every partition under `files.long_partitions[]`
+with `path`, `year`, `sha256`, `row_count` and `size_bytes` — 56 of them.
+Substituting an explicit file list for the glob was measured against the
+published corpus on 2026-08-08:
+
+| Path | Result |
+|---|---|
+| `https://…/data/long/**/*.parquet` (default) | error — globs unsupported over HTTP |
+| same, `allow_asterisks_in_http_paths = true` | error — literal `**/*` 404s |
+| `hf://datasets/civilytics/us-cog-finance/…` glob | 46,148,034 rows |
+| **explicit list over plain https** | **46,148,034 rows** |
+
+`hive_partitioning = true` still recovers `year` from the paths under
+enumeration, so no downstream view or verb changes.
+
+Enumeration is preferred over `hf://` deliberately. It is **host-agnostic** —
+Nextcloud, HuggingFace, or any static server take the same code path — where
+`hf://` would tie the default to one vendor's protocol and still need
+special-casing, since manifest fetching goes through `httr2`, which cannot speak
+`hf://`. Enumeration also *removes* a dependency (globbing) rather than adding
+one, and the manifest's per-file `sha256` becomes available for integrity
+checking later.
+
+Views are registered from `inst/sql/` with `{url}` substitution in
+`R/views.R:.register_views()`. The list must be built once per session from the
+already-fetched manifest and substituted the same way, so the change is confined
+to view registration and does not touch verb code.
+
+### Fix: ship a working default
+
+`R/config.R`'s default becomes the public HuggingFace `resolve/main/` URL:
+CC-BY-4.0, no token to publish, CDN-backed, and it keeps maxwell's uplink out of
+the path — the same reasoning behind the GitHub mirror and r-universe.
+
+This means `library(uscogdata)` followed by a verb works with **zero
+configuration**, which is what makes the package demonstrable in a README and
+later in a post. `USCOGDATA_URL` and `options(uscogdata.url=)` continue to
+override, so the Nextcloud copy and local mirrors are unaffected.
+
+The `uscogdata_url_not_configured` error class stays — it still fires for an
+explicitly-set empty or placeholder URL — but ceases to be the default
+experience.
+
+### Consequence: `cog_mirror()` is promoted
+
+Measured cost of the remote default, from efron on a good connection:
+
+| | |
+|---|---|
+| Whole corpus | **190.6 MB**, 56 partitions, 46,148,034 rows, FY1967–FY2024 |
+| One government, one year | 1.5 s |
+| One government, all 56 years | 2.8 s |
+| Disk written | **0.00 MB** — range requests only; `external_file_cache` is in-memory |
+
+Nothing persists locally beyond the shared `httpfs` extension in `~/.duckdb` (a
+few MB, once per machine, across all DuckDB use). Costs are RAM and per-query
+bandwidth, since nothing caches between sessions.
+
+Those timings are raw scans. Real verbs additionally join crosswalks, resolve
+categories and assemble provenance, so end-to-end verb latency will be higher and
+**must be re-measured once the fix lands** — it cannot be measured today.
+
+The corpus being only 190.6 MB makes `cog_mirror()` a first-class option rather
+than a developer footnote. The README presents **both paths**:
+
+- **Remote (default, zero setup)** — trying it out, teaching, one-off questions.
+- **Mirrored (`cog_mirror()`, 190 MB once)** — repeated or heavy analysis,
+  offline work, reproducibility, or preferring not to depend on HuggingFace.
+
+The second is also the honest answer to the vendor-dependency question raised by
+defaulting to HuggingFace: **the escape hatch is one function call and 190 MB**,
+after which no analysis touches an external service. The README says so
+explicitly. That is the difference between a convenience default and lock-in.
+
+## Release-readiness fixes
+
+| # | Issue | Fix |
+|---|---|---|
+| 1 | `MaxCorpusSchema: 5` in DESCRIPTION; `.validate_schema()` accepts `4,5,6,7`; published corpus is **7** | `MaxCorpusSchema: 7` |
+| 2 | `^vignettes$` in `.Rbuildignore` — both vignettes absent from the installed package, while README tells users to run `vignette("total-spending")` | Remove `^vignettes$`, `^doc$`, `^Meta$`. Both vignettes build offline (`total-spending` reads the bundled fixture; `population-denominators` is `eval = FALSE`) |
+| 3 | `_pkgdown.yml` reference index covers 6 of 14 exports — pkgdown errors on missing topics | Add `cog_categories`, `cog_explain`, `cog_find_peers`, `cog_geographic_rollup`, `cog_manifest`, `cog_mirror`, `cog_peer_compare`, `cog_recipes`; set `url:` |
+| 4 | No `URL:` / `BugReports:` in DESCRIPTION | Add both, pointing at the GitHub mirror |
+| 5 | No `LICENSE.md`; `LICENSE` holder reads `Civilytics` | `usethis::use_mit_license("Civilytics Consulting LLC")` |
+| 6 | README instructs stripping the fixture at release | Delete that section — see below |
+| 7 | `Authors@R` is an org with no human | Jared E. Knowles `aut`/`cre` + ORCID; Civilytics Consulting LLC `cph`/`fnd` |
+
+**On #6.** The advice to add `^inst/extdata/fixture_corpus$` to `.Rbuildignore`
+is CRAN-sized thinking (5 MB limit) and this package is not going to CRAN.
+Stripping the 15 MB fixture would break `total-spending.Rmd`, which reads from
+it, and would leave r-universe and GitHub Actions unable to run the 28 test files
+without a corpus credential. **The fixture is what lets `R CMD check` pass
+anywhere with zero secrets** — precisely what public CI needs. It ships.
+
+## README
+
+The current README addresses someone standing inside the repo tree: status reads
+"Under active development (Phase 2 of the cog_pipeline project)", it points at
+`../cog_pipeline/docs/reader-specification.md`, the install line is commented
+out, and developer, testing and release sections sit above anything a user needs.
+
+Restructured around a stranger, in this order:
+
+1. **What this is** — one paragraph, and what the corpus covers (types 0–3,
+   FY1967–FY2024, 46M rows, 190.6 MB).
+2. **Install** — r-universe first (binaries), git second.
+3. **Quickstart that actually runs** — resolve a government, get its history,
+   print provenance. No configuration step.
+4. **Two ways to read the corpus** — remote default vs `cog_mirror()`, with the
+   measured numbers and the independence note.
+5. **Amounts are in full US dollars** — kept near the top. This is the errata
+   most likely to produce a wrong answer that looks plausible.
+6. **Concepts** — primary/direct/total spending, general/total revenue,
+   coverage. Condensed, linking to the vignettes for the full treatment.
+7. **How to cite** — `citation("uscogdata")`, corpus CC-BY-4.0 attribution.
+8. **Contributing** — canonical-on-Gitea PR flow.
+
+Developer notes, testing instructions and release procedure move to
+`CONTRIBUTING.md`. Every path reference to a sibling repo is removed or replaced
+with a URL that resolves for someone who has only this repo.
+
+## NEWS.md
+
+`NEWS.md` currently holds two sections. `0.2.0` is a legitimate changelog — the
+`"All Categories"` reserved value, the coverage-signposting fix, the
+`n_units_reporting` documentation — and it stays. Beneath it,
+`0.1.0 (development)` is a pre-release churn log: changes described relative to
+states no user has ever seen ("Breaking: corpus schema_version 4", "the package
+now requires…"), spanning the package's entire pre-release development. To a
+newcomer deciding whether to depend on this, that section reads as instability.
+
+**A new `0.3.0` section is added at the top, framed as the first public
+release**: what the package does, what the corpus covers, and the caveats that
+are genuinely load-bearing. **`0.2.0` is kept verbatim.** **`0.1.0 (development)`
+is dropped** — that history stays in git, where it belongs.
+
+The version is `0.3.0` rather than `0.2.0` because this release changes
+user-visible behaviour: remote corpus reads go from broken to working, and the
+default URL from a dead placeholder to a live corpus. It is also not `1.0.0` —
+the corpus still excludes government types 4 and 5 pending validation, so a
+stability promise would overclaim. No git tag exists for any prior version;
+`chore: release 0.2.0` bumped `DESCRIPTION` and `NEWS` only.
+
+The substantive content is migrated, not deleted. These are hard-won and belong
+in documentation rather than buried in a changelog:
+
+| Content | Destination |
+|---|---|
+| Coverage disclosure on multi-government aggregates (census vs sample years) | README concepts + `cog_geographic_rollup()` docs |
+| `complete = TRUE` three-way absence semantics (`reported` / `census_zero` / `not_reported`) | `cog_spending()` / `cog_revenue()` docs |
+| Series-break and corpus-break surfacing | README + `cog_explain()` docs |
+| $1,000s → full dollars conversion | README, already prominent |
+| Per-year F-33 population denominators | `population-denominators` vignette, already there |
+
+This also makes NEWS reusable as raw material for the release announcement,
+which is the stated downstream purpose.
+
+## Distribution mechanics
+
+Recorded as decided; executed after the package is clean and checks are green.
+
+**Sequence matters.** r-universe publishes check results the moment a package is
+registered. Registering before the fixes above land means a red badge on day one,
+which is a worse first impression than a week's delay.
+
+1. `gitleaks` over full history. A coarse grep found nothing across 140 commits
+   and the default corpus URL is still the placeholder sentinel, but a proper
+   scan is the gate on an irreversible action.
+2. Flip the Gitea repo public. Disable Gitea issues on it, so there is exactly
+   one inbox.
+3. Create `github.com/civilytics/uscogdata`. Add `.github/workflows/` for the
+   Windows/macOS/Linux `R CMD check` matrix — the platforms the Gitea runner
+   cannot provide, and which this package has never been tested on despite
+   depending on duckdb and httr2. Gitea reads `.gitea/workflows`, GitHub reads
+   `.github/workflows`; both live in one tree without colliding.
+4. Gitea Actions workflow pushing to GitHub **without `--force`**, so divergence
+   fails loudly in CI rather than silently overwriting.
+5. Add `jared@civilytics.com` as a verified secondary email on the GitHub
+   account — r-universe links maintainer identity by matching DESCRIPTION's email
+   against registered GitHub emails, and the association only takes effect on the
+   next build.
+6. Tag `v0.3.0`. Create `github.com/civilytics/civilytics.r-universe.dev` with a
+   `packages.json` pinned to the tag, pointing at the GitHub mirror rather than
+   Gitea so clone traffic stays off maxwell. Install the r-universe app.
+
+### PR flow
+
+Never press Merge on GitHub. A merge there is overwritten by the next sync, the
+PR still displays "Merged", and nothing says otherwise.
+
+```sh
+git remote add github https://github.com/civilytics/uscogdata.git
+git config --add remote.github.fetch '+refs/pull/*/head:refs/remotes/github/pr/*'
+git fetch github
+git switch -c pr-42 github/pr/42      # test
+git switch main && git merge --no-ff pr-42
+git push origin main                   # Gitea -> mirror -> GitHub
+```
+
+GitHub auto-closes a PR as merged once its head commit becomes an ancestor of the
+base branch, so `--no-ff` — which preserves the contributor's SHAs — makes the PR
+close itself when the mirror pushes. **For external PRs, merge; do not squash or
+rebase.** Squashing rewrites the SHAs, the auto-close never fires, and closing by
+hand reads to a first-time contributor as rejection.
+
+`CONTRIBUTING.md` states this, and a GitHub Action comments it on incoming PRs.
+No CLA; no DCO.
+
+## Verification
+
+The release is not done until all of these pass:
+
+1. `R CMD check --as-cran` clean on Linux, and on Windows and macOS via the
+   GitHub matrix. This package has never been checked on the latter two.
+2. Full test suite (28 files) green against the **bundled fixture**, offline,
+   with no credentials — the property public CI depends on.
+3. Full test suite green against the **live corpus**, which additionally
+   exercises the enumeration fix that the fixture's local path cannot.
+4. `pkgdown::build_site()` completes.
+5. Both vignettes present in the built tarball and
+   `vignette("total-spending", package = "uscogdata")` resolves from an
+   installed copy.
+6. **Cold-start check on a machine that has never seen this package:** install
+   from r-universe, `library(uscogdata)`, run the README quickstart verbatim with
+   no environment variables set. This is the only test that catches the P0 class
+   of fault, and its absence is why the fault survived.
+7. End-to-end verb latency re-measured against the live corpus and the README's
+   numbers updated if they moved.
+
+## Out of scope
+
+- **Corrections intake.** Deferred by decision. Consequence: the release cannot
+  invite data-error reports or make the "traceable and correctable" claim that
+  most distinguishes this corpus from Census's own files. `BugReports:` points at
+  package issues only. A verified correction should eventually terminate as a
+  `lineage_event` or `series_break` row so it propagates through provenance to
+  every consumer — that design is unstarted.
+- **Announcement posts.** Deferred. The API announcement is gated on corrections
+  landing and merits a Civic Pulse edition.
+- **The rest of the R package backlog.** Parked until this one completes the path.
+- **`cog_pipeline` publication.** Stays private.
@@ -0,0 +1,58 @@
+test_that('cog_geographic_rollup() accepts "All Categories" and agrees with per-category sums', {
+  skip_if_no_corpus()
+  govs <- cog_gov_search(name = NULL, state = "WI", type = 2L)
+  expect_gt(nrow(govs), 1L)
+  ids <- list(city = utils::head(govs$canonical_govid, 25L))
+
+  by_cat <- cog_geographic_rollup(ids, category = NULL, years = 2019L)
+  total  <- cog_geographic_rollup(ids, category = "All Categories", years = 2019L)
+
+  expect_setequal(unique(total$category), "All Categories")
+  # one row per (govid, subtype) that appears in the per-category result
+  key_by_cat <- unique(paste(by_cat$canonical_govid, by_cat$spend_subtype))
+  key_total  <- paste(total$canonical_govid, total$spend_subtype)
+  expect_setequal(key_total, key_by_cat)
+
+  lhs <- tapply(by_cat$amt_nominal, paste(by_cat$canonical_govid, by_cat$spend_subtype), sum)
+  rhs <- tapply(total$amt_nominal,  key_total, sum)
+  expect_equal(as.numeric(rhs[names(lhs)]), as.numeric(lhs), tolerance = 1e-8)
+})
+
+test_that('"All Categories" survives per_capita and inflation adjustment through the rollup', {
+  skip_if_no_corpus()
+  govs <- cog_gov_search(name = NULL, state = "WI", type = 2L)
+  ids  <- list(city = utils::head(govs$canonical_govid, 10L))
+  r <- cog_geographic_rollup(ids, category = "All Categories", years = 2019L,
+                             per_capita = TRUE, adjust_to_year = 2020L)
+  expect_true(all(c("amt_per_capita_nominal", "amt_real", "amt_per_capita_real") %in% names(r)))
+  expect_setequal(unique(r$category), "All Categories")
+  expect_true(all(is.finite(r$amt_real)))
+})
+
+test_that('cog_geographic_rollup() still refuses expenditure_concept = "total" with "All Categories"', {
+  skip_if_no_corpus()
+  govs <- cog_gov_search(name = NULL, state = "WI", type = 2L)
+  ids  <- list(city = utils::head(govs$canonical_govid, 5L))
+  expect_error(
+    cog_geographic_rollup(ids, category = "All Categories", years = 2019L,
+                          expenditure_concept = "total")
+  )
+})
+
+test_that("n_units_reporting is category-conditional, not a response rate", {
+  skip_if_no_corpus()
+  govs <- cog_gov_search(name = NULL, state = "WI", type = 2L)
+  ids  <- list(city = govs$canonical_govid)
+
+  police <- cog_geographic_rollup(ids, category = "Police", years = 2012L)
+  allcat <- cog_geographic_rollup(ids, category = "All Categories", years = 2012L)
+
+  cov_police <- cog_explain(police, format = "list")$coverage
+  cov_all    <- cog_explain(allcat, format = "list")$coverage
+
+  # Same year, same requested govids, same collection -- yet a single category
+  # reports fewer units than the all-categories query. That gap is real zeros,
+  # not non-response, which is exactly why the ratio is not a response rate.
+  expect_lte(cov_police$n_units_reporting, cov_all$n_units_reporting)
+  expect_identical(cov_police$n_units_expected, cov_all$n_units_expected)
+})
@@ -0,0 +1,267 @@
+# Baseline at branch point: 843 PASS / 0 FAIL / 0 SKIP / 0 WARN (2026-08-05, origin/main 2fc9e75)
+
+test_that(".build_verb_sql emits a literal category and no category filter in all-categories mode", {
+  sql <- uscogdata:::.build_verb_sql(
+    view         = "spending_annotated",
+    subtype_col  = "spend_subtype",
+    govid        = "552025209777",
+    years        = 2019L,
+    category     = NULL,
+    subtype_scope = c("operations", "capital"),
+    all_categories = TRUE
+  )
+
+  expect_match(sql, "'All Categories' AS category", fixed = TRUE)
+  # no category filter of any kind
+  expect_false(grepl("AND category IN", sql, fixed = TRUE))
+  # category is not a grouping key
+  expect_false(grepl("GROUP BY year, canonical_govid, gov_name, xwalk_gov_name, spend_subtype, category",
+                     sql, fixed = TRUE))
+  # the subtype allowlist still applies -- this is what makes the sum a concept
+  expect_match(sql, "AND spend_subtype IN ('operations','capital')", fixed = TRUE)
+})
+
+test_that(".build_verb_sql is unchanged when all_categories is FALSE", {
+  args <- list(
+    view = "spending_annotated", subtype_col = "spend_subtype",
+    govid = "552025209777", years = 2019L, category = NULL,
+    subtype_scope = c("operations", "capital")
+  )
+  old <- do.call(uscogdata:::.build_verb_sql, args)
+  new <- do.call(uscogdata:::.build_verb_sql, c(args, list(all_categories = FALSE)))
+  expect_identical(old, new)
+  expect_match(new, "GROUP BY year, canonical_govid, gov_name, xwalk_gov_name, spend_subtype, category",
+               fixed = TRUE)
+})
+
+test_that(".ALL_CATEGORIES is the exact reserved string", {
+  expect_identical(uscogdata:::.ALL_CATEGORIES, "All Categories")
+})
+
+test_that('cog_spending(category = "All Categories") sums to the per-category total', {
+  gov <- "552025209777"
+  by_cat <- cog_spending(gov, 2019L)
+  total  <- cog_spending(gov, 2019L, category = "All Categories")
+
+  expect_true(nrow(total) > 0L)
+  expect_setequal(unique(total$category), "All Categories")
+  # one row per subtype present in the by-category result
+  expect_setequal(unique(total$spend_subtype), unique(by_cat$spend_subtype))
+  expect_equal(nrow(total), length(unique(by_cat$spend_subtype)))
+
+  # the dollars agree, per subtype
+  lhs <- tapply(by_cat$amt_nominal, by_cat$spend_subtype, sum)
+  rhs <- tapply(total$amt_nominal,  total$spend_subtype,  sum)
+  expect_equal(as.numeric(rhs[names(lhs)]), as.numeric(lhs), tolerance = 1e-8)
+})
+
+test_that('"All Categories" respects expenditure_concept', {
+  gov <- "552025209777"
+  prim <- cog_spending(gov, 2019L, category = "All Categories",
+                       expenditure_concept = "primary")
+  dir  <- cog_spending(gov, 2019L, category = "All Categories",
+                       expenditure_concept = "direct")
+  # direct = primary plus interest and insurance benefits, so it is never smaller
+  expect_gte(sum(dir$amt_nominal), sum(prim$amt_nominal))
+})
+
+test_that('"All Categories" works on revenue and respects revenue_concept', {
+  gov <- "552025209777"
+  gen <- cog_revenue(gov, 2019L, category = "All Categories",
+                     revenue_concept = "general")
+  tot <- cog_revenue(gov, 2019L, category = "All Categories",
+                     revenue_concept = "total")
+  expect_setequal(unique(gen$category), "All Categories")
+  expect_gte(sum(tot$amt_nominal), sum(gen$amt_nominal))
+})
+
+test_that('"All Categories" cannot be combined with another category', {
+  expect_error(
+    cog_spending("552025209777", 2019L, category = c("All Categories", "Police")),
+    class = "uscogdata_all_categories_not_combinable"
+  )
+})
+
+test_that('"All Categories" is recorded in provenance', {
+  r <- cog_spending("552025209777", 2019L, category = "All Categories")
+  expect_identical(cog_explain(r, format = "list")$category, "All Categories")
+})
+
+test_that('"All Categories" combines with subtype to give operating totals', {
+  gov <- "552025209777"
+  ops_by_cat <- cog_spending(gov, 2019L)
+  ops_by_cat <- ops_by_cat[ops_by_cat$spend_subtype == "operations", ]
+  ops_total  <- cog_spending(gov, 2019L, category = "All Categories")
+  ops_total  <- ops_total[ops_total$spend_subtype == "operations", ]
+  expect_equal(sum(ops_total$amt_nominal), sum(ops_by_cat$amt_nominal),
+               tolerance = 1e-8)
+})
+
+test_that('cog_categories() advertises "All Categories" for both flows', {
+  all <- cog_categories()
+  rows <- all[all$category == "All Categories", ]
+  expect_setequal(rows$category_type, c("expenditure", "revenue"))
+  expect_true(all(is.na(rows$subtype)))
+  expect_true(all(is.na(rows$n_codes)))
+})
+
+test_that('cog_categories(type=) still scopes, including the pseudo-category', {
+  sp <- cog_categories(type = "spending")
+  expect_setequal(unique(sp$category_type), "expenditure")
+  expect_true("All Categories" %in% sp$category)
+
+  rev <- cog_categories(type = "revenue")
+  expect_setequal(unique(rev$category_type), "revenue")
+  expect_true("All Categories" %in% rev$category)
+
+  # balances have no concept vocabulary, so no pseudo-category
+  bal <- cog_categories(type = "balance")
+  expect_false("All Categories" %in% bal$category)
+})
+
+test_that('cog_categories(pattern=) matches the pseudo-category', {
+  hit <- cog_categories(pattern = "^All Categories$")
+  expect_equal(nrow(hit), 2L)
+})
+
+# --- final whole-branch review fixes ---------------------------------------
+
+test_that('complete = TRUE is refused when combined with "All Categories"', {
+  # .completion_grid_sql() would emit `AND c.category IN ('All Categories')`,
+  # match zero crosswalk rows, and the early return in .complete_result()
+  # would stamp completion$applied = TRUE, rows_filled = 0 -- reading as "the
+  # grid was checked and nothing was missing" when nothing was actually
+  # checked. Filling a summed row has no defined semantics, so the verb must
+  # refuse the combination outright (finding 2).
+  expect_error(
+    cog_spending("552025209777", 2019L, category = "All Categories",
+                complete = TRUE),
+    class = "uscogdata_complete_unsupported"
+  )
+  expect_error(
+    cog_revenue("552025209777", 2019L, category = "All Categories",
+               complete = TRUE),
+    class = "uscogdata_complete_unsupported"
+  )
+})
+
+test_that('cog_balances() rejects "All Categories" instead of silently returning zero rows', {
+  # cog_balances() reuses .validate_verb_inputs() but did not pass
+  # allow_all_categories = TRUE, so "All Categories" used to become
+  # `AND category IN ('All Categories')` against balance_annotated -- 0
+  # matching crosswalk rows, 0 rows back, no error (finding 3). Holdings are
+  # a stock with no concept vocabulary to sum across, so the honest answer is
+  # to refuse, the same way cog_spending()/cog_revenue() refuse other
+  # nonsensical combinations.
+  expect_error(
+    cog_balances("552025209777", 2019L, category = "All Categories"),
+    class = "uscogdata_all_categories_unsupported"
+  )
+  # An ordinary category still works -- this is not a blanket regression.
+  r <- suppressMessages(
+    cog_balances("552025209777", 2019L, category = "Fund Balances")
+  )
+  expect_gt(nrow(r), 0L)
+})
+
+test_that('expenditure_concept_direct_suppressed is NA, not FALSE, when categories are collapsed', {
+  # .detect_direct_suppressed() keys on
+  # paste(year, canonical_govid, category, sep = "\r"). In all-categories
+  # mode every row carries the literal "All Categories" value, so an IG-only
+  # row's key collides with any ordinary Direct row for the same
+  # (year, govid) -- has_direct reads TRUE whenever the government has ANY
+  # direct spending at all, candidate is always empty, and the detector can
+  # never fire. Before the fix this silently reported FALSE, an affirmative
+  # claim the code did not actually compute (finding 1). NA is the honest
+  # answer: cog_explain(x, format = "list") is required here, since without
+  # format = "list" it returns the result tibble, not the provenance list.
+  gov <- "552025209777"
+  t <- cog_spending(gov, 2019L, category = "All Categories",
+                    expenditure_concept = "total")
+  prov <- cog_explain(t, format = "list")
+  expect_true(is.na(prov$expenditure_concept_direct_suppressed))
+  expect_false(isTRUE(prov$expenditure_concept_direct_suppressed))
+  expect_match(prov$expenditure_concept_note, "unavailable", fixed = TRUE)
+
+  # A per-category "total" query on the same government/year is unaffected --
+  # the detector can still key correctly and reports a strict logical.
+  t_by_cat <- cog_spending(gov, 2019L, expenditure_concept = "total")
+  prov_by_cat <- cog_explain(t_by_cat, format = "list")
+  expect_false(is.na(prov_by_cat$expenditure_concept_direct_suppressed))
+})
+
+test_that('"All Categories" still signposts coverage gaps (finding 6, final whole-branch review)', {
+  # .build_suggestions()'s candidate sub-select used to be keyed on
+  # `category`, e.g. `WHERE category IN ('All Categories')`. Since
+  # .ALL_CATEGORIES is never itself a row in summary_categories.category,
+  # that sub-select always came back empty in all-categories mode, so
+  # `candidates` was empty and .build_suggestions() short-circuited to
+  # list() -- coverage signposting was structurally impossible for the one
+  # mode whose whole selling point is "you cannot sum the wrong scope"
+  # (uscogdata#9's entire point, silently defeated).
+  #
+  # AL state government, FY2011, category = "Corrections": this category has
+  # no legacy leaf rows in FY2011 (aggregate-flagged E04/E05 family), so the
+  # per-category query returns 0 rows and 3 recipe-hint suggestions fire
+  # (empty_year path). All-categories mode does not have an empty year --
+  # the government has other primary spending in FY2011 -- but the same
+  # suppressed Corrections dollars are still excluded from the summed total,
+  # so the fix (scoping the candidate sub-select by subtype_col/subtype_scope
+  # instead of by category, symmetric with .build_verb_sql()) must still
+  # surface them via the suppressed_component path.
+  gov <- "010000226085"
+
+  by_cat <- suppressMessages(cog_spending(gov, 2011L, category = "Corrections"))
+  sugg_by_cat <- cog_explain(by_cat, format = "list")$suggestions
+  expect_gt(length(sugg_by_cat), 0L)
+
+  all_cat <- suppressMessages(cog_spending(gov, 2011L, category = "All Categories"))
+  sugg_all_cat <- cog_explain(all_cat, format = "list")$suggestions
+  expect_gt(length(sugg_all_cat), 0L)
+
+  # The same Corrections recipe that fired per-category must also fire in
+  # all-categories mode -- not just some unrelated recipe.
+  ids_by_cat <- vapply(sugg_by_cat, function(s) s$recipe_id %||% "", character(1))
+  ids_all_cat <- vapply(sugg_all_cat, function(s) s$recipe_id %||% "", character(1))
+  expect_true("corrections_combined" %in% ids_by_cat)
+  expect_true("corrections_combined" %in% ids_all_cat)
+
+  # In all-categories mode the government DOES have other primary spending
+  # in FY2011 (the year itself is not a gap), so the suggestion can only have
+  # fired via the suppressed_component path, not empty_year.
+  corr_all <- sugg_all_cat[[which(ids_all_cat == "corrections_combined")]]
+  expect_identical(corr_all$trigger, "suppressed_component")
+  expect_gt(corr_all$suppressed_amount, 0)
+})
+
+test_that('"All Categories" candidate scoping is symmetric with .build_verb_sql() -- subtype, not category', {
+  # Direct assertion on the mechanism itself (finding 6): in all-categories
+  # mode .build_suggestions() must scope its candidate recipe sub-select by
+  # subtype_col/subtype_scope, not by the literal "All Categories" value.
+  # Passing all_categories = FALSE with the identical category value proves
+  # the branch -- not merely the subtype_col/subtype_scope arguments' mere
+  # presence -- is what changes the query.
+  con <- uscogdata:::.ensure_session()
+
+  none <- uscogdata:::.build_suggestions(
+    con, govid = "010000226085", years = 2011L,
+    category = "All Categories", result = NULL, basis = "harmonized",
+    flow_prefixes = c("E", "F", "G"),
+    long_view = "spending_long_harmonized",
+    all_categories = FALSE,
+    subtype_col = "spend_subtype",
+    subtype_scope = c("operations", "capital", "assistance")
+  )
+  expect_length(none, 0L)
+
+  scoped <- uscogdata:::.build_suggestions(
+    con, govid = "010000226085", years = 2011L,
+    category = "All Categories", result = NULL, basis = "harmonized",
+    flow_prefixes = c("E", "F", "G"),
+    long_view = "spending_long_harmonized",
+    all_categories = TRUE,
+    subtype_col = "spend_subtype",
+    subtype_scope = c("operations", "capital", "assistance")
+  )
+  expect_gt(length(scoped), 0L)
+})
@@ -66,7 +66,7 @@ test_that("inst/sql/26-balance_long.sql enforces NOT is_aggregate (real SQL text
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
  }

  con <- DBI::dbConnect(duckdb::duckdb())
@@ -29,7 +29,9 @@ test_that("cog_categories(type = 'spending') returns only expenditure rows", {
  # joined with the I/Q/Y flow batch -- the last two characters of Census's
  # expenditure taxonomy. `interest` is what makes the three-concept model
  # computable: primary = direct minus debt service.
-  expect_true(all(r$subtype %in%
+  # Exclude pseudo-category which has NA for subtype
+  r_crosswalk <- r[r$category != "All Categories", ]
+  expect_true(all(r_crosswalk$subtype %in%
                  c("operations", "capital", "intergovernmental", "assistance",
                    "interest", "insurance_benefits")))
 })
@@ -54,7 +56,9 @@ test_that("cog_categories(type = 'revenue') returns only revenue rows", {
  # plus the employee-retirement X codes), utility (A91-A94) and liquor store
  # (A90) revenue by definition, which is what makes both of its published
  # revenue concepts computable -- see `revenue_concept` in `?cog_revenue`.
-  expect_true(all(r$subtype %in%
+  # Exclude pseudo-category which has NA for subtype
+  r_crosswalk <- r[r$category != "All Categories", ]
+  expect_true(all(r_crosswalk$subtype %in%
                  c("own_source", "federal", "state", "local_aid",
                    "insurance_trust", "utility", "liquor_store")))
 })
@@ -69,6 +73,8 @@ test_that("cog_categories(pattern = ...) filters case-insensitively", {
 test_that("cog_categories has one row per (category, subtype)", {
  skip_if_no_corpus()
  r <- cog_categories()
+  # Exclude pseudo-category which is not a crosswalk entry
+  r <- r[r$category != "All Categories", ]
  key <- paste(r$category, r$subtype, sep = "|")
  expect_equal(length(key), length(unique(key)))
 })
@@ -76,6 +82,8 @@ test_that("cog_categories has one row per (category, subtype)", {
 test_that("cog_categories item_codes is non-empty comma-separated string", {
  skip_if_no_corpus()
  r <- cog_categories()
+  # Exclude pseudo-category which has NA for n_codes and item_codes
+  r <- r[r$category != "All Categories", ]
  expect_true(all(nzchar(r$item_codes)))
  expect_true(all(r$n_codes >= 1L))
  # n_codes should equal count of commas + 1
@@ -64,3 +64,109 @@ test_that(".resolve_url does not invent a slash for an empty setting", {
  withr::local_options(uscogdata.url = "")
  expect_equal(.resolve_url(), "")
 })
+
+test_that("the default corpus URL is real, not a placeholder", {
+  # setup.R points USCOGDATA_URL at the bundled fixture for the whole suite,
+  # so both the env var and the option have to be cleared to see the default.
+  withr::local_envvar(USCOGDATA_URL = NA)
+  withr::local_options(uscogdata.url = NULL)
+  url <- .resolve_url()
+  expect_false(grepl("REPLACE_WITH", url, fixed = TRUE))
+  expect_match(url, "^https://")
+  expect_match(url, "/$")
+})
+
+test_that("an explicitly-set sentinel URL still aborts", {
+  # The guard must survive the default change: a user who half-edited a
+  # copied config still gets the actionable error.
+  withr::local_envvar(
+    USCOGDATA_URL = "https://other.example/s/REPLACE_WITH_SHARE_TOKEN/x/"
+  )
+  expect_error(
+    .check_url_configured(.resolve_url()),
+    class = "uscogdata_url_not_configured"
+  )
+})
+
+test_that("DESCRIPTION carries release metadata", {
+  skip_if_no_source_tree("DESCRIPTION")
+  d <- read.dcf(source_tree_path("DESCRIPTION"))
+  fields <- colnames(d)
+
+  expect_true(all(c("URL", "BugReports") %in% fields))
+  expect_match(d[1, "Authors@R"], "Knowles", fixed = TRUE)
+  expect_match(d[1, "Authors@R"], "0000-0003-0005-9478", fixed = TRUE)
+  expect_match(d[1, "Authors@R"], "Civilytics Consulting LLC", fixed = TRUE)
+
+  # The gate in .validate_schema() accepts up to 7 and the published corpus
+  # IS 7; DESCRIPTION must not claim otherwise.
+  expect_equal(as.integer(d[1, "MaxCorpusSchema"]), 7L)
+
+  # Authors@R must actually parse -- a malformed person() call is only
+  # caught at citation()/build time otherwise.
+  people <- eval(parse(text = d[1, "Authors@R"]))
+  expect_s3_class(people, "person")
+  expect_true("cre" %in% unlist(lapply(people, function(p) p$role)))
+})
+
+test_that("LICENSE and LICENSE.md name the same copyright holder", {
+  skip_if_no_source_tree("LICENSE", "LICENSE.md")
+  holder <- sub("^COPYRIGHT HOLDER:\\s*", "",
+                grep("^COPYRIGHT HOLDER:", readLines(source_tree_path("LICENSE"),
+                                                     warn = FALSE), value = TRUE))
+  full <- paste(readLines(source_tree_path("LICENSE.md"), warn = FALSE), collapse = "\n")
+
+  expect_equal(holder, "Civilytics Consulting LLC")
+  expect_match(full, holder, fixed = TRUE)
+  # usethis::use_mit_license() writes LICENSE.md but leaves an existing
+  # LICENSE alone, which is how the two came to disagree in the first place.
+  expect_match(full, "MIT License", fixed = TRUE)
+})
+
+test_that("vignettes are not excluded from the build", {
+  skip_if_no_source_tree(".Rbuildignore")
+  ignore <- readLines(source_tree_path(".Rbuildignore"), warn = FALSE)
+  expect_false(any(grepl("^\\^vignettes\\$$", ignore)))
+  # The fixture is what lets R CMD check run offline with no credentials on
+  # r-universe and GitHub Actions. It must never be excluded.
+  expect_false(any(grepl("fixture_corpus", ignore, fixed = TRUE)))
+  # doc/ and Meta/ ARE build artefacts of devtools::build_vignettes() and must
+  # stay excluded -- R CMD build regenerates inst/doc/ from vignettes/ on its
+  # own, and leaving them in earns a "non-standard file at top level" NOTE.
+  expect_true(any(grepl("^\\^doc\\$$", ignore)))
+  expect_true(any(grepl("^\\^Meta\\$$", ignore)))
+})
+
+test_that("_pkgdown.yml indexes every exported topic", {
+  skip_if_no_source_tree("_pkgdown.yml", "NAMESPACE")
+  exports <- grep("^export\\(", readLines(source_tree_path("NAMESPACE"), warn = FALSE),
+                  value = TRUE)
+  exports <- sub("^export\\((.*)\\)$", "\\1", exports)
+  yml <- paste(readLines(source_tree_path("_pkgdown.yml"), warn = FALSE), collapse = "\n")
+  missing <- exports[!vapply(exports,
+                             function(e) grepl(paste0("\\b", e, "\\b"), yml),
+                             logical(1))]
+  # pkgdown errors on topics missing from the index, so an unlisted export
+  # means the docs site does not build at all.
+  expect_equal(missing, character(0))
+})
+
+test_that("README is written for a stranger, not a repo insider", {
+  skip_if_no_source_tree("README.md")
+  r <- paste(readLines(source_tree_path("README.md"), warn = FALSE), collapse = "\n")
+
+  # No paths that only resolve inside a maintainer's checkout.
+  expect_false(grepl("../cog_pipeline", r, fixed = TRUE))
+  # A real, uncommented install line.
+  expect_match(r, "install.packages", fixed = TRUE)
+  expect_false(grepl("# pak::pkg_install", r, fixed = TRUE))
+  # The errata most likely to produce a plausible-looking wrong answer.
+  expect_match(r, "full US dollars", fixed = TRUE)
+  # The release advice that conflicts with public CI is gone.
+  expect_false(grepl("Rbuildignore", r, fixed = TRUE))
+  # Both read paths documented.
+  expect_match(r, "cog_mirror", fixed = TRUE)
+  # cog_spending() has no default for `years`; a quickstart that omits it
+  # errors on the reader's first call.
+  expect_match(r, "years\\s*=", perl = TRUE)
+})
@@ -0,0 +1,62 @@
+# Network-gated. Set USCOGDATA_LIVE_TEST=true to run.
+#
+# This file exists because the defect fixed for 0.3.0 -- no remote corpus was
+# readable at all, because DuckDB cannot expand a glob over generic HTTP --
+# survived precisely because every other test path used a LOCAL corpus (the
+# bundled fixture), and so did the API in production (a host mount). Nothing
+# ever exercised the package the way a new user does.
+skip_live <- function() {
+  testthat::skip_if_not(
+    identical(tolower(Sys.getenv("USCOGDATA_LIVE_TEST", "")), "true"),
+    "live-corpus test: set USCOGDATA_LIVE_TEST=true to run"
+  )
+}
+
+# The suite's setup.R pins USCOGDATA_URL to the bundled fixture, so reaching
+# the default requires clearing both the env var and the option.
+with_default_corpus <- function(code) {
+  withr::local_envvar(
+    USCOGDATA_URL = NA, USCOGDATA_FIXTURE_URL = NA,
+    .local_envir = parent.frame()
+  )
+  withr::local_options(uscogdata.url = NULL, .local_envir = parent.frame())
+  cog_close()
+  withr::defer(cog_close(), envir = parent.frame())
+  force(code)
+}
+
+test_that("the package reads the public corpus with no configuration at all", {
+  skip_live()
+  with_default_corpus({
+    g <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+    expect_gt(nrow(g), 0)
+
+    s <- cog_spending(g$canonical_govid[1], years = 2022)
+    expect_gt(nrow(s), 0)
+    expect_true(all(c("amt_nominal", "year", "category") %in% names(s)))
+
+    # Amounts are full dollars, already x1000. A city's annual spending is
+    # millions, not thousands -- this catches a regression that dropped or
+    # doubled the conversion.
+    expect_gt(sum(s$amt_nominal, na.rm = TRUE), 1e6)
+
+    p <- attr(s, "provenance")
+    expect_true(isTRUE(p$transformations$units_conversion$applied))
+    expect_equal(p$transformations$units_conversion$multiplier, 1000)
+  })
+})
+
+test_that("a multi-decade query reads across many partitions", {
+  skip_live()
+  with_default_corpus({
+    g <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+    # `years` is required on cog_spending() -- there is no full-history
+    # default at the reader level (the API's /profile route supplies one).
+    s <- cog_spending(g$canonical_govid[1], years = 2000:2022)
+    # Enumeration builds one read_parquet() path per requested partition. If
+    # the list were truncated, or silently collapsed to a single file, the
+    # returned span is what catches it.
+    expect_gt(diff(range(s$year)), 10)
+    expect_gt(length(unique(s$year)), 5)
+  })
+})
@@ -0,0 +1,64 @@
+test_that(".long_files_sql enumerates every partition the manifest lists", {
+  manifest <- list(files = list(long_partitions = list(
+    list(year = 2011L, path = "data/long/year=2011/part-0.parquet"),
+    list(year = 2012L, path = "data/long/year=2012/part-0.parquet")
+  )))
+  expect_equal(
+    uscogdata:::.long_files_sql("https://example.org/corpus/", manifest),
+    paste0(
+      "['https://example.org/corpus/data/long/year=2011/part-0.parquet',",
+      "'https://example.org/corpus/data/long/year=2012/part-0.parquet']"
+    )
+  )
+})
+
+test_that(".long_files_sql falls back to the glob when no partition list is present", {
+  # test-views.R registers views with a hand-built manifest that has no
+  # `files` element. That must keep working: the glob is valid for the
+  # local paths such a manifest is used with.
+  expect_equal(
+    uscogdata:::.long_files_sql("/tmp/corpus/", list(schema_version = 4L)),
+    "'/tmp/corpus/data/long/**/*.parquet'"
+  )
+  expect_equal(
+    uscogdata:::.long_files_sql("/tmp/corpus/", list(files = list(long_partitions = list()))),
+    "'/tmp/corpus/data/long/**/*.parquet'"
+  )
+})
+
+test_that("the enumerated list matches the bundled fixture's partition count", {
+  skip_if_no_corpus()
+  m <- jsonlite::fromJSON(
+    file.path(fixture_corpus_path(), "manifest.json"), simplifyVector = FALSE
+  )
+  out <- uscogdata:::.long_files_sql(fixture_corpus_path(), m)
+  expect_equal(
+    lengths(regmatches(out, gregexpr("part-0\\.parquet", out))),
+    length(m$files$long_partitions)
+  )
+})
+
+test_that("no view SQL survives rendering with an unsubstituted token", {
+  # Introducing {long_files} broke four test sites that had hand-rolled the
+  # {url} substitution -- each failed with a DuckDB parser error on the
+  # surviving brace. This asserts the whole SQL directory renders clean, so
+  # a future token cannot reintroduce that silently.
+  sql_dir <- system.file("sql", package = "uscogdata")
+  for (f in list.files(sql_dir, pattern = "\\.sql$", full.names = TRUE)) {
+    rendered <- uscogdata:::.render_view_sql(
+      paste(readLines(f, warn = FALSE), collapse = "\n"), "/tmp/corpus/"
+    )
+    expect_false(grepl("\\{[a-z_]+\\}", rendered), label = basename(f))
+  }
+})
+
+test_that("registered `long` view reads through the enumerated list", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    con <- uscogdata:::.ensure_session()
+    n <- DBI::dbGetQuery(con, "SELECT count(*) AS n FROM long")$n
+    expect_gt(n, 0)
+    yrs <- DBI::dbGetQuery(con, "SELECT DISTINCT year FROM long ORDER BY year")$year
+    expect_true(all(c(2011, 2012, 2019, 2020) %in% yrs))
+  })
+})
@@ -4,12 +4,14 @@
 # protect users from silent failures when USCOGDATA_URL is misconfigured
 # or returns non-JSON content.

-test_that("cog_open aborts with actionable error when URL is the placeholder default", {
+test_that("cog_open aborts with actionable error when URL contains the sentinel", {
  uscogdata:::cog_close()
  on.exit(uscogdata:::cog_close(), add = TRUE)

-  placeholder <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
-  withr::with_envvar(c(USCOGDATA_URL = placeholder), {
+  # No longer the package default (that is the public HF corpus). This is a
+  # user who copied a config template and did not finish editing it.
+  sentinel_url <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  withr::with_envvar(c(USCOGDATA_URL = sentinel_url), {
    expect_error(
      uscogdata:::cog_open(),
      class = "uscogdata_url_not_configured"
@@ -35,8 +37,10 @@ test_that("placeholder guard error names both env var and option as remediation"
  uscogdata:::cog_close()
  on.exit(uscogdata:::cog_close(), add = TRUE)

-  placeholder <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
-  withr::with_envvar(c(USCOGDATA_URL = placeholder), {
+  # No longer the package default (that is the public HF corpus). This is a
+  # user who copied a config template and did not finish editing it.
+  sentinel_url <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  withr::with_envvar(c(USCOGDATA_URL = sentinel_url), {
    msg <- tryCatch(uscogdata:::cog_open(), error = conditionMessage)
    expect_match(msg, "USCOGDATA_URL", fixed = TRUE)
    expect_match(msg, "uscogdata.url", fixed = TRUE)
@@ -0,0 +1,29 @@
+# Mirror of test-spending-pagination.R for cog_revenue(), which shares the
+# same .verb_spendrev()/.build_verb_sql() pushdown -- see that file for the
+# incident this fixes.
+
+test_that("cog_revenue limit/offset page correctly and report total_rows", {
+  skip_if_no_corpus()
+  full <- cog_revenue("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_revenue("121011212191", years = 2019:2020, category = NULL,
+                      limit = 5L, offset = 3L)
+  expect_equal(nrow(page), 5L)
+  expect_equal(page[c("year", "canonical_govid", "revenue_subtype", "category")],
+              full[4:8, c("year", "canonical_govid", "revenue_subtype", "category")],
+              ignore_attr = TRUE)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("cog_revenue limit unset by default leaves total_rows absent", {
+  skip_if_no_corpus()
+  r <- cog_revenue("121011212191", 2020L, "Property Tax")
+  expect_null(attr(r, "total_rows"))
+})
+
+test_that("cog_revenue complete + limit conflict aborts the same way as cog_spending", {
+  skip_if_no_corpus()
+  expect_error(
+    cog_revenue("121011212191", 2020L, "Property Tax", complete = TRUE, limit = 5L),
+    class = "uscogdata_complete_pagination_conflict"
+  )
+})
@@ -0,0 +1,95 @@
+# cog-api's paginate() used to slice an already-fully-materialized result:
+# every page of a deep sweep re-ran the whole query and re-listified every
+# row, just to keep 1000 and discard the rest. For a 193,105-row fleet-wide
+# query walked 194 pages deep, that repeated the full cost 194 times and
+# wedged the production server for hours (2026-08-06 incident). limit/offset
+# here push the slice into the SQL itself, so a page costs O(limit), not
+# O(full result).
+
+test_that("limit without offset returns the first page, matching the unpaginated head", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                       limit = 10L)
+  expect_equal(nrow(page), 10L)
+  expect_equal(page[c("year", "canonical_govid", "spend_subtype", "category")],
+              full[1:10, c("year", "canonical_govid", "spend_subtype", "category")],
+              ignore_attr = TRUE)
+})
+
+test_that("offset skips ahead without gaps or overlap", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page2 <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                        limit = 10L, offset = 10L)
+  expect_equal(nrow(page2), 10L)
+  expect_equal(page2[c("year", "canonical_govid", "spend_subtype", "category")],
+              full[11:20, c("year", "canonical_govid", "spend_subtype", "category")],
+              ignore_attr = TRUE)
+})
+
+test_that("walking every page reconstructs the unpaginated result exactly", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  n <- nrow(full)
+  limit <- 7L
+  pages <- list()
+  offset <- 0L
+  repeat {
+    p <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                      limit = limit, offset = offset)
+    if (nrow(p) == 0L) break
+    pages[[length(pages) + 1L]] <- p
+    offset <- offset + limit
+    if (offset > n + limit) stop("test runaway: paging did not terminate")
+  }
+  walked <- dplyr::bind_rows(pages)
+  expect_equal(nrow(walked), n)
+  key_cols <- c("year", "canonical_govid", "spend_subtype", "category", "amt_nominal")
+  expect_equal(walked[key_cols], full[key_cols], ignore_attr = TRUE)
+})
+
+test_that("total_rows attribute reports the full unpaginated count", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                       limit = 5L, offset = 0L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("offset past the end returns zero rows, not an error", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                       limit = 10L, offset = nrow(full) + 100L)
+  expect_equal(nrow(page), 0L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("limit is unset by default -- unpaginated calls are unaffected", {
+  skip_if_no_corpus()
+  r <- cog_spending("121011212191", 2020L, "Corrections")
+  expect_null(attr(r, "total_rows"))
+})
+
+test_that("per_capita and adjust_to_year still apply correctly within a page", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2020L, category = NULL,
+                       per_capita = TRUE, adjust_to_year = 2022L)
+  page <- cog_spending("121011212191", years = 2020L, category = NULL,
+                       per_capita = TRUE, adjust_to_year = 2022L,
+                       limit = 3L, offset = 2L)
+  expect_equal(page[c("amt_nominal", "amt_real", "amt_per_capita_nominal",
+                      "amt_per_capita_real")],
+              full[3:5, c("amt_nominal", "amt_real", "amt_per_capita_nominal",
+                          "amt_per_capita_real")],
+              ignore_attr = TRUE)
+})
+
+test_that("complete = TRUE with limit aborts -- pagination over a partial grid is undefined", {
+  skip_if_no_corpus()
+  expect_error(
+    cog_spending("121011212191", 2020L, "Corrections", complete = TRUE, limit = 5L),
+    class = "uscogdata_complete_pagination_conflict"
+  )
+})
@@ -103,7 +103,7 @@ test_that("inst/sql/22- and 23- harmonized views enforce every WHERE predicate (
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
  }

  con <- DBI::dbConnect(duckdb::duckdb())
@@ -183,7 +183,7 @@ test_that("inst/sql/24- and 25- IG views retain aggregates, COALESCE NULL harmon
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
  }

  con <- DBI::dbConnect(duckdb::duckdb())
@@ -335,7 +335,7 @@ test_that(".harmonization_view_files guard is necessary: registration against a
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", url, txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, url)
  }
  con2 <- DBI::dbConnect(duckdb::duckdb())
  on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
Author	SHA1	Message	Date
jared	44e4953f94	fix: keep doc/ and Meta/ out of the build R-CMD-check / check (push) Successful in 3m58s Details R-CMD-check / check (pull_request) Successful in 3m52s Details Removing ^vignettes$ was right; removing ^doc$ and ^Meta$ with it was not. Those are devtools::build_vignettes() artefacts, not sources -- R CMD build regenerates inst/doc/ from vignettes/ by itself, and shipping the local copies earned a 'non-standard file/directory found at top level' NOTE. R CMD check --as-cran is now 0 errors, 0 warnings, 0 notes.	2026-08-08 18:25:00 -04:00
jared	a5500f0b6a	docs: add CONTRIBUTING with the canonical-on-Gitea PR flow Moves developer, testing and release instructions out of the README, minus the fixture-stripping advice, which was wrong. Explains that a GitHub PR closes itself as merged once the mirror syncs, because the merge preserves the contributor's SHAs -- so a PR closing without a visible Merge click reads as success rather than rejection. Documents the cog-api dependency: its CI clones this package at USCOGDATA_REF, defaulting to main with no pin, so anything merged here reaches the API's next build. Includes the commands to run its suite against a branch first.	2026-08-08 18:20:42 -04:00
jared	da2839f885	docs: recast NEWS around the first public release NEWS described changes relative to states no user had ever seen -- 'Breaking: corpus schema_version 4', 'the package now requires...' -- across the whole pre-release development. To someone deciding whether to depend on this, that reads as instability. 0.3.0 is written as an announcement: what it covers, the verbs, that reading the corpus now works out of the box, four things to know before a first query, and the known limits. The 0.2.0 changelog is kept verbatim. The 0.1.0 development log is dropped; that history is in git. cog_explain() now documents what provenance actually holds, since the README points readers at it -- in particular why series_break_refs and corpus_break_refs are separate fields rather than one list.	2026-08-08 18:20:33 -04:00
jared	331399ab86	docs: rewrite README for a stranger Reordered around a new user: what the data is, where it comes from, install, a quickstart that runs with no configuration, then the full-dollars warning and the concepts that decide whether a published number is right. Adds a 'Where the data comes from' section linking the API documentation site, the live API, the Hugging Face corpus and the Census source, so attribution and provenance are reachable from the top rather than implied. Drops the sibling-repo path, the commented-out install line, the Status block, and the release advice telling you to strip the fixture -- which would break the vignette and leave public CI unable to check without credentials. The quickstart passes years=; cog_spending() has no full-history default, so the obvious one-liner errors on a reader's first call.	2026-08-08 18:20:24 -04:00
jared	5582c6cb57	docs: index all 14 exports in pkgdown, set the site url The reference index covered 6 of 14 exports, so pkgdown errored on the eight missing topics and the docs site did not build at all. Adds a Comparison & aggregation section and a Corpus metadata section, lists both vignettes as articles, and sets url so canonical links and search resolve. build_site() now completes clean: reference metadata ok, no problems.	2026-08-08 17:49:10 -04:00
jared	013af5b0d2	fix: ship the vignettes .Rbuildignore excluded ^vignettes$, ^doc$ and ^Meta$, so an installed uscogdata had no vignettes at all -- while the README instructed users to run vignette("total-spending"), which failed for every one of them. Both build offline: total-spending points USCOGDATA_URL at the bundled fixture, population-denominators is eval = FALSE. Confirmed present in the built tarball as both source and rendered inst/doc/. The test also pins the fixture as never-excluded -- it is what lets R CMD check run with no credentials on r-universe and GitHub Actions.	2026-08-08 17:48:07 -04:00
jared	4a92f36d44	chore: add full MIT text, name the copyright holder properly LICENSE held only the two-line stub and no LICENSE.md existed, so the repo carried no license text for a human browsing it or for GitHub's license detector. usethis::use_mit_license() writes LICENSE.md but leaves an existing LICENSE alone, so the stub kept saying 'Civilytics' while the full text said 'Civilytics Consulting LLC'. Corrected by hand, with a test pinning the two together.	2026-08-08 17:47:18 -04:00
jared	a67735f121	chore: release metadata -- author of record, URLs, schema ceiling, 0.3.0 Authors@R was an org with no human, so citation() and the r-universe maintainer page had nothing to render and ORCID could not collate this with merTools. The given-name vector c("Jared", "E.") matches merTools exactly; person("Jared", "E. Knowles") would render the same but put the middle initial in the family-name slot. MaxCorpusSchema claimed 5 while .validate_schema() accepts 4-7 and the published corpus is 7 -- metadata contradicting code by two versions. 0.3.0 rather than 0.2.0: remote reads go from broken to working and the default URL from placeholder to live, which is user-visible behaviour.	2026-08-08 17:46:39 -04:00
jared	b9f7f7d8d3	test: exercise the public corpus end to end, unconfigured The remote-read defect survived because every test path used a local corpus, and so did the API in production. This is the only test that runs the package the way a new user does: no USCOGDATA_URL, no option, no fixture -- just install and call a verb. Gated on USCOGDATA_LIVE_TEST so offline CI skips rather than fails. Measured against the live corpus while writing this: cog_spending for one government is 3.9s for a single year and 5.9s across 2000-2022. Well above the 1.5-2.8s raw parquet scan, because the verbs also join crosswalks, resolve categories and assemble provenance.	2026-08-08 17:21:53 -04:00
jared	4300b636b1	feat: default to the public corpus so the package works unconfigured The default was a REPLACE_WITH_SHARE_TOKEN sentinel and no document in the package supplied a working URL, so a new user installing uscogdata had no path to a session at all -- just an actionable-looking error with nothing actionable behind it. The default is now the public HuggingFace mirror: CC-BY-4.0, no credential, CDN-backed, and it keeps the origin's uplink out of the read path. USCOGDATA_URL and options(uscogdata.url=) still override, so Nextcloud and cog_mirror() copies are unaffected. The sentinel guard stays for half-edited configs; the two tests covering it set the URL explicitly, so they only needed renaming to stop calling it 'the default'.	2026-08-08 17:18:56 -04:00
jared	99e1e86e37	fix: enumerate long partitions from the manifest instead of globbing DuckDB cannot expand a glob over generic HTTP -- there is no directory listing, and allow_asterisks_in_http_paths only forwards the literal '*/' as a filename, which 404s. So every remote corpus read failed. Only local paths worked, which is how the API (a host mount) and the test fixture run, so nothing ever caught it. Measured against the published corpus: the explicit list returns the same 46,148,034 rows the hf:// glob does, with hive_partitioning still recovering year from the paths. Building it from the manifest keeps the reader host-agnostic rather than binding it to one vendor's protocol. Also extracts .render_view_sql(). Four test sites had hand-rolled the {url} substitution -- one commented as doing it 'exactly as .register_views() does' -- and all four broke on the second token. They now share the one function that knows the vocabulary, and a new test renders every SQL file to prove no token survives.	2026-08-08 17:11:44 -04:00
jared	c042ee0b90	docs: retarget the release at 0.3.0 and preserve the 0.2.0 changelog Spec and plan were written against a branch 25 commits behind main, where the package still read 0.1.0. It is 0.2.0, with a real 0.2.0 changelog in NEWS that the plan would have deleted. 0.3.0 rather than 0.2.0 because remote corpus reads go from broken to working and the default URL from placeholder to live -- user-visible behaviour, so a minor bump. Not 1.0.0: types 4 and 5 remain out of scope. Task 9 now prepends a 0.3.0 section, keeps 0.2.0 verbatim with a diff check to prove it, and drops only the 0.1.0 development churn. Task 4 gains the Version bump.	2026-08-08 17:07:19 -04:00
jared	29dc8199e0	docs: implementation plan for the uscogdata 0.1.0 public release Ten tasks, 58 steps, TDD throughout. Tasks 1-3 fix the P0 (manifest enumeration, working default URL, and the live-corpus test whose absence let the defect survive); 4-7 are metadata and packaging; 8-10 rewrite README, NEWS and CONTRIBUTING. Distribution mechanics stay out of scope -- r-universe publishes check results on registration, so it comes after final verification is green.	2026-08-08 17:03:40 -04:00
jared	619b167ab1	docs: design spec for the uscogdata 0.1.0 public release Covers the P0 finding that the package cannot read the corpus remotely at all -- no working default URL, and Hive globs are unsupported over generic HTTP by DuckDB 1.5.5. Fix is manifest-driven file enumeration (measured: 46,148,034 rows over plain https, identical to the hf:// glob) plus a working public default. Also: seven release-readiness fixes, a README restructured for a stranger, NEWS rewritten as an initial release rather than a pre-release churn log, and the Gitea-canonical/GitHub-mirror/r-universe distribution mechanics.	2026-08-08 17:03:40 -04:00
jared	dfda39051e	docs: point agents at the Civilytics values file before they start Mirrors the same block in cog_explorer/CLAUDE.md. A project-level @ import does not preload, so the instruction to read it is the mechanism.	2026-08-08 17:03:39 -04:00
jared	c587c8ba87	Merge pull request 'fix: push limit/offset into the query instead of materializing then slicing' (#39 ) from fix/pushdown-pagination into main R-CMD-check / check (push) Successful in 3m35s Details Reviewed-on: #39	2026-08-06 14:19:42 -04:00
jared	6a06302036	fix: push limit/offset into the query instead of materializing then slicing R-CMD-check / check (push) Successful in 4m6s Details R-CMD-check / check (pull_request) Successful in 3m33s Details cog-api's paginate() sliced an already-fully-materialized result: every page of a deep sweep re-ran the whole cog_spending()/cog_revenue() query and re-listified every row, just to keep up to 1000 and discard the rest. A 193,105-row/194-page fleet-wide sweep (cog_explorer's Southern guide, corpus summary build) repeated that full cost 194 times and wedged the production server for hours on 2026-08-06 -- single request, CPU-bound, single-threaded plumber process, no other request could get through, not even /health. cog_spending()/cog_revenue() gain optional limit/offset, pushed into .build_verb_sql() as SQL LIMIT/OFFSET behind the existing (already deterministic) ORDER BY. The full unpaginated row count rides along via COUNT(*) OVER() in the same scan -- exposed as a total_rows attribute -- so a caller walking pages never needs a second round trip to ask how many there are. A page now costs O(limit), not O(full result). Mutually exclusive with complete = TRUE (which fills a grid over the FULL requested (year, category) space -- pagination over a partial slice of already-grouped rows has no defined meaning for the cells it would fill) and with recipe (whose result comes from a separate, not-yet-wired query path). Both abort with a clear classed condition rather than silently ignoring the parameter. limit/offset default to NULL; every existing call site is unaffected.	2026-08-06 13:56:27 -04:00
jared	e3ab26c3e6	Merge pull request 'feat: 'All Categories' pseudo-category + n_units_reporting semantics' (#37 ) from feat/all-categories-37 into main R-CMD-check / check (push) Successful in 3m50s Details Reviewed-on: #37	2026-08-05 12:50:46 -04:00
jared	498950afa6	fix: scope all-categories suggestion candidates by subtype, not category (finding 6) R-CMD-check / check (push) Successful in 3m41s Details R-CMD-check / check (pull_request) Successful in 4m52s Details .build_suggestions()'s recipe-candidate sub-select was keyed on `WHERE category IN (<category>)`. The reserved pseudo-category "All Categories" is never itself a row in summary_categories.category, so in all-categories mode `candidates` always came back empty and coverage signposting (uscogdata#9) was structurally impossible for the one mode whose entire premise is "you cannot sum the wrong scope" -- measured on Los Angeles County FY2011: category = "Public Welfare" reports 2 suggestions (incl. $271,589,000 excluded E68), category = "All Categories" reported 0, silently losing that same signal. Apply the branch's own design principle: the concept boundary is subtype, not category. .build_suggestions() now accepts all_categories/subtype_col/ subtype_scope (all optional, default off, so no other caller's behaviour changes) and, when all-categories mode is active, scopes the candidate sub-select by `<subtype_col> IN (<subtype_scope>)` instead -- symmetric with .build_verb_sql()'s own WHERE predicate. The M/L recipe exclusion and the is.null(category) early return are unchanged. After the fix, LA County FY2011 "All Categories" reports 5 suggestions, including welfare_cash_e68_wide for the exact $271,589,000 gap. Adds two covering tests to test-all-categories.R using the bundled fixture (AL state gov, FY2011, "Corrections"): one end-to-end (per-category and all-categories both signpost the same recipe) and one direct on .build_suggestions() proving the subtype-vs-category branch is what changes the query. Updates the 0.2.0 NEWS entry.	2026-08-05 12:38:12 -04:00
jared	a5f86d87b3	fix: close five final-review gaps in all-categories mode - .detect_direct_suppressed() keys on (year, canonical_govid, category); all-categories mode collapses category to one literal value, so the key collides and the detector silently reports FALSE instead of "unknown". Report NA there instead, and stop isTRUE() in .build_provenance() from collapsing that NA back to FALSE. Schema widened to allow null. - Refuse complete = TRUE + category = "All Categories": the completion grid has no per-category cells left to fill once categories are collapsed, so the prior silent 0-rows-filled result was never actually checked. - cog_balances(category = "All Categories") returned zero rows with no error. .validate_verb_inputs() gains allow_all_categories (default FALSE); .verb_spendrev() passes TRUE, cog_balances() does not, so the three verbs share one place to reject it instead of drifting again. - Fix the false `subtype = "operations"` argument claim (no such argument exists) in NEWS.md and an internal spending.R comment. Adds three covering tests to test-all-categories.R for the three behaviour changes above.	2026-08-05 12:30:23 -04:00
jared	61b9c95731	chore: release 0.2.0 R-CMD-check / check (push) Successful in 3m34s Details R-CMD-check / check (pull_request) Successful in 6m30s Details Bumps the minor version because 'All Categories' adds public surface without breaking any existing call. The bump is load-bearing, not cosmetic: cog-api installs this package with install_local(), which no-ops when the version already matches. Without it, Phase 1 would silently test against the 0.1.0 reader and pass while proving nothing.	2026-08-05 12:04:24 -04:00
jared	44e9b40b86	docs: n_units_reporting is category-conditional, not a response rate Closes uscogdata#36. It counts governments with rows for the requested category, so a surveyed government that genuinely spends nothing there is indistinguishable from one never surveyed. In FY2022, a complete census year, Georgia reports 393 of 567 cities for Police -- the gap is cities that contract to the sheriff. Documents the comparison that IS valid: same category, census year vs sample year.	2026-08-05 11:58:25 -04:00
jared	f1e9aa383a	test: prove 'All Categories' passes through the geographic rollup Geographic totals are the expensive case cog-api#37 was filed about -- without this a caller issues one rollup per category and sums them. The pass-through was expected to work by construction; this asserts it rather than assuming it, including under per_capita and inflation adjustment.	2026-08-05 11:53:27 -04:00
jaredandClaude Opus 5	12a9be110f	feat: advertise 'All Categories' from cog_categories() A reserved value nobody can discover is a trap, and this is the view the API's /categories endpoint is built from. Emitted for the two flow vocabularies only -- cog_balances() returns a stock and has no concept to sum within. Also fix test-categories.R to exclude pseudo-category rows from crosswalk-specific assertions (one row per (category, subtype) pair, non-empty item_codes, valid subtypes). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 11:43:02 -04:00
jared	503fa6562f	docs: fix false subtype= claim in All Categories roxygen (F1) The @param category text on cog_spending()/cog_revenue() told users to "Combine with subtype = ..." but neither verb has a subtype argument. Replace with accurate guidance: filter the returned frame's spend_subtype/revenue_subtype column.	2026-08-05 11:35:47 -04:00
jared	11ae99c382	feat: accept category = 'All Categories' on cog_spending/cog_revenue Returns one summed row per (year, govid, subtype) across every category in the requested concept's subtype scope, so a caller never sums categories client-side and cannot sum the wrong scope. Combining it with other category names is an error rather than a silent partial sum.	2026-08-05 11:32:47 -04:00
jared	5e22e940e7	feat: all-categories mode in .build_verb_sql() The concept boundary in this package is subtype, not category, so a total is the existing query with the category dimension collapsed and no category predicate applied. subtype is deliberately kept in the grouping: subtype=operations plus all-categories is 'operating expenditure', which is the measure a fiscal comparison wants. Named 'All Categories' rather than 'Total' because category='Total' would sit one argument from expenditure_concept='total' and mean something different.	2026-08-05 11:21:35 -04:00
jared	2fc9e7585b	Merge pull request 'fix: signpost partially-suppressed categories (#9 )' (#32 ) from fix/partial-coverage-signposting-9 into main R-CMD-check / check (push) Successful in 3m24s Details	2026-08-05 08:15:46 -04:00