ci: multi-platform R CMD check for the GitHub mirror

The canonical Gitea runner is Linux-only, and this package hard-depends on duckdb and httr2 -- both compiled, both with real platform variance -- while having never been checked on Windows or macOS. A large share of the audience is on Windows. Inert here: Gitea reads .gitea/workflows, GitHub reads .github/workflows, so both configs coexist and the Gitea CI stays authoritative for deploys. This goes live the moment the mirror repo exists (#44). The matrix needs no credentials -- setup.R points at the bundled fixture -- which is why that fixture must never be .Rbuildignore'd.
Merge pull request 'feat: uscogdata 0.3.0 — first public release' (#41 ) from feat/public-release-0.3.0 into main
2026-08-08 19:49:52 -04:00 · 2026-08-08 18:56:14 -04:00 · 2026-08-08 18:55:59 -04:00 · 2026-08-08 18:25:00 -04:00 · 2026-08-08 18:20:42 -04:00 · 2026-08-08 18:20:33 -04:00
32 changed files with 2498 additions and 387 deletions
@@ -3,17 +3,17 @@
 ^\.Rproj\.user$
 ^_pkgdown\.yml$
 ^docs$
+^Meta$
+^doc$
 ^pkgdown$
 ^\.github$
 ^LICENSE\.md$
 ^\.git$
 ^\.gitignore$
 \.gitkeep$
-^vignettes$
 ^specs$
 ^plans$
-^doc$
-^Meta$
 ^\.gitea$
 ^CLAUDE\.md$
 ^\.superpowers$
+^CONTRIBUTING\.md$
@@ -0,0 +1,61 @@
+# Multi-platform R CMD check, running on the GitHub mirror.
+#
+# This exists because the canonical Gitea runner is Linux-only, and this
+# package hard-depends on duckdb and httr2 -- both compiled, both with real
+# platform variance -- while having never been checked on Windows or macOS.
+# A large share of the audience is on Windows.
+#
+# Gitea reads .gitea/workflows and GitHub reads .github/workflows, so this
+# file is inert on the canonical repo and coexists with the Gitea CI that
+# remains authoritative for deploys.
+#
+# The suite needs NO credentials: tests/testthat/setup.R points USCOGDATA_URL
+# at the bundled fixture corpus. That is exactly why inst/extdata/fixture_corpus
+# must never be added to .Rbuildignore.
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+name: R-CMD-check
+
+permissions: read-all
+
+jobs:
+  R-CMD-check:
+    runs-on: ${{ matrix.config.os }}
+    name: ${{ matrix.config.os }} (${{ matrix.config.r }})
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - {os: macos-latest,   r: 'release'}
+          - {os: windows-latest, r: 'release'}
+          - {os: ubuntu-latest,  r: 'devel', http-user-agent: 'release'}
+          - {os: ubuntu-latest,  r: 'release'}
+
+    env:
+      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
+      R_KEEP_PKG_SOURCE: yes
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: r-lib/actions/setup-pandoc@v2
+
+      - uses: r-lib/actions/setup-r@v2
+        with:
+          r-version: ${{ matrix.config.r }}
+          http-user-agent: ${{ matrix.config.http-user-agent }}
+          use-public-rspm: true
+
+      - uses: r-lib/actions/setup-r-dependencies@v2
+        with:
+          extra-packages: any::rcmdcheck
+          needs: check
+
+      - uses: r-lib/actions/check-r-package@v2
+        with:
+          upload-snapshots: true
+          build_args: 'c("--no-manual")'
@@ -124,3 +124,10 @@ devtools::test()
  a numbered `.sql` file; build a query in R.
 - No arrow dependency — DuckDB reads parquet natively
 - `withr` is a Suggests-only dep; only used in tests
+
+## Domain context — read this first
+
+**Before doing any work in this repo, read `~/.claude/memory/values/civilytics.md`.**
+It carries the purpose, direction, and constraints for this domain. It is not optional
+context — read it before planning or writing code, not after. (An `@` import will not
+work here; project-level imports don't preload. The read is the mechanism.)
@@ -0,0 +1,105 @@
+# Contributing to uscogdata
+
+Thanks for reading this — a package like this gets better mostly through people
+noticing that a number looks wrong.
+
+## Where the code lives
+
+Development happens on **Gitea**, at
+`gitea.civilytics.org/Civilytics/uscogdata`. The repository at
+`github.com/civilytics/uscogdata` is a **mirror** that accepts issues and pull
+requests.
+
+## What happens to a GitHub pull request
+
+Open it normally. Behind the scenes it is fetched and landed on the canonical
+Gitea repository, then syncs back:
+
+```sh
+git fetch github refs/pull/42/head:pr-42
+git switch main && git merge --no-ff pr-42
+git push origin main            # Gitea -> mirror -> GitHub
+```
+
+Because the merge preserves your commits at their original SHAs, **GitHub marks
+your PR merged on its own** as soon as the mirror syncs. So:
+
+> If your pull request closes as "Merged" without anyone visibly clicking
+> Merge, that is the normal, successful outcome — not a rejection.
+
+Substantial contributions get a `ctb` entry in `DESCRIPTION`, which surfaces in
+`citation("uscogdata")`.
+
+There is no CLA and no DCO sign-off requirement.
+
+## Running the tests
+
+```r
+devtools::test()   # bundled fixture; no network, no credentials
+```
+
+`tests/testthat/setup.R` points `USCOGDATA_URL` at
+`inst/extdata/fixture_corpus/` automatically — a four-year slice (2011, 2012,
+2019, 2020) covering all 50 states. That is the whole data setup.
+
+## Testing against the live corpus
+
+```sh
+USCOGDATA_LIVE_TEST=true Rscript -e 'devtools::test(filter = "live-corpus")'
+```
+
+This is worth understanding rather than skipping. Until 0.3.0 the package
+**could not read a remote corpus at all** — the partitioned view used a glob,
+and DuckDB cannot expand a glob over generic HTTP. It went unnoticed for months
+because every test path used a local corpus (the bundled fixture), and so did
+the production API (a host mount). Nothing exercised the package the way a new
+user does.
+
+`test-live-corpus.R` is the only test that runs with no `USCOGDATA_URL`, no
+option, and no fixture. If you change anything touching view registration,
+manifest handling, or configuration, run it.
+
+## Do not exclude the fixture from the build
+
+There is a temptation to add `^inst/extdata/fixture_corpus$` to
+`.Rbuildignore` because 15 MB feels large for a package. Don't:
+
+- `vignette("total-spending")` reads from it and would fail to build.
+- `R CMD check` on r-universe and GitHub Actions would have no corpus, so the
+  suite could not run without credentials.
+
+This package is not going to CRAN, so its 5 MB guidance does not apply. A
+package-size NOTE in `R CMD check` is expected and acceptable.
+
+## Downstream consumers
+
+`cog-api` depends on this package and its CI clones uscogdata at
+`USCOGDATA_REF`, **defaulting to `main`**. There is no pin. Anything merged
+here reaches the API's next build, so before merging a change to the reader,
+run the API suite against your branch:
+
+```sh
+Rscript -e "remotes::install_local('/path/to/uscogdata', upgrade = 'never')"
+cd /path/to/cog-api/api/tests/testthat
+Rscript -e 'testthat::test_dir(".", stop_on_failure = TRUE)'
+```
+
+The API calls only exported verbs, so internal refactors are usually safe —
+but "usually" is not a release gate.
+
+## Release checklist
+
+1. `devtools::test()` — green against the bundled fixture, offline.
+2. `USCOGDATA_LIVE_TEST=true devtools::test()` — green against the live corpus.
+3. cog-api suite green against this branch (above).
+4. `devtools::check(args = "--as-cran")` — 0 errors, 0 warnings.
+5. `pkgdown::build_site()` completes.
+6. Vignettes resolve from an installed copy:
+   `vignette("total-spending", package = "uscogdata")`.
+7. **Cold-start check**: on a machine that has never had this package,
+   install it and run the README quickstart verbatim with no environment
+   variables set. This is the only check that catches a
+   corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
+8. Bump `Version` and add a `NEWS.md` section.
+9. Tag, then update the r-universe registry pin at
+   `github.com/civilytics/civilytics.r-universe.dev`.
@@ -1,14 +1,21 @@
 Package: uscogdata
 Type: Package
 Title: Curated Reader for the Civilytics US Census of Governments Finance Corpus
-Version: 0.2.0
-Authors@R:
-    person("Civilytics", , , "jknowles@gmail.com", role = c("aut", "cre"))
+Version: 0.3.0
+Authors@R: c(
+    person(c("Jared", "E."), "Knowles",
+           email = "jared@civilytics.com",
+           role = c("aut", "cre"),
+           comment = c(ORCID = "0000-0003-0005-9478")),
+    person("Civilytics Consulting LLC", role = c("cph", "fnd")))
 Description: Curated R verbs over the Civilytics US Census of Governments
    finance corpus. Provides unit-level financial profiles, geographic
    rollups, and peer comparisons with auditable provenance and built-in
    cross-vintage correctness.
 License: MIT + file LICENSE
+URL: https://github.com/civilytics/uscogdata,
+    https://civilytics.r-universe.dev/uscogdata
+BugReports: https://github.com/civilytics/uscogdata/issues
 Encoding: UTF-8
 LazyData: false
 Depends: R (>= 4.1)
@@ -32,4 +39,4 @@ Config/testthat/edition: 3
 VignetteBuilder: knitr
 RoxygenNote: 7.3.3
 MinCorpusSchema: 4
-MaxCorpusSchema: 5
+MaxCorpusSchema: 7
@@ -1,2 +1,2 @@
 YEAR: 2026
-COPYRIGHT HOLDER: Civilytics
+COPYRIGHT HOLDER: Civilytics Consulting LLC
@@ -0,0 +1,21 @@
+# MIT License
+
+Copyright (c) 2026 Civilytics Consulting LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -1,3 +1,63 @@
+# uscogdata 0.3.0
+
+First public release.
+
+`uscogdata` provides curated R verbs over the Civilytics US Census of
+Governments finance corpus: unit-level financial profiles, geographic rollups
+and peer comparisons, with auditable provenance on every result.
+
+## What it covers
+
+Government types 0-3 (state, county, municipality, township), FY1967-FY2024 --
+56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
+or FY1969. Special districts (type 4) and school districts (type 5) are out of
+scope pending validation.
+
+## The verbs
+
+`cog_spending()`, `cog_revenue()` and `cog_balances()` for flows and holdings;
+`cog_gov_search()` to resolve place names (including basket mode for many at
+once); `cog_find_peers()` and `cog_peer_compare()` for cohorts;
+`cog_geographic_rollup()` for aggregates; `cog_categories()`, `cog_recipes()`,
+`cog_manifest()` and `cog_explain()` for metadata and provenance; and
+`cog_mirror()` for a local copy of the corpus.
+
+## Reading the corpus now works out of the box
+
+* The package reads the published corpus over HTTPS **with no configuration**.
+  Previously the default was a placeholder sentinel and no document in the
+  package supplied a working URL, so a new user had no path to a session.
+* Remote reads work at all. The partitioned view used a glob, and DuckDB
+  cannot expand a glob over generic HTTP -- there is no directory listing to
+  expand against. Partition paths are now enumerated from the corpus manifest,
+  which is host-agnostic: an HTTPS mirror, a Nextcloud share and a local
+  `cog_mirror()` copy all take the same path.
+* Nothing is written to disk in remote mode; DuckDB fetches only the row
+  groups a query needs.
+
+## Four things to know before your first query
+
+* **Amounts are in full US dollars.** The raw Census files report thousands;
+  the verbs multiply by 1000 on the way out. Do not multiply again.
+* **Multi-government aggregates disclose their coverage.** The Census is a
+  complete enumeration only in years ending in 2 and 7; every other year is a
+  sample. Every such result carries `provenance$coverage` with per-year
+  `n_units_reporting`.
+* **Absence means two different things.** Before FY2012 an absent cell means
+  Census published $0; from FY2012 it means not reported. `complete = TRUE`
+  labels which.
+* **Series breaks reach you unasked.** Catalogued breaks intersecting your
+  query appear in provenance and in `cog_explain()`.
+
+## Known limits
+
+* Special districts (type 4) and school districts (type 5) are out of scope.
+* Per-capita rollups exclude governments with no F-33 population, which is by
+  design but does silently narrow a rollup.
+* `n_units_reporting` is category-conditional and is not a response rate.
+* Employee-retirement (`X`) codes stop at FY2016, when those systems moved to
+  the Annual Survey of Public Pensions.
+
 # uscogdata 0.2.0

 ## New features
@@ -39,262 +99,3 @@
  is not a response rate: a government that was surveyed and genuinely spends
  nothing in the requested category is indistinguishable from one never
  surveyed (uscogdata#36).
-
-# uscogdata 0.1.0 (development)
-
-## Signposting now catches partially-suppressed categories
-
-* A coverage suggestion used to fire only when a category returned **no rows
-  at all** in a requested year. That missed the more dangerous case: a
-  category that still returns rows while silently dropping component codes
-  the wide era publishes only as aggregates (#9). `cog_spending(category =
-  "Public Welfare")` for FY2011 returned a plausible figure that omitted
-  `E67`/`E68` entirely -- for Los Angeles County, $2,075,461,000 of a true
-  $5,261,404,000, a 39% understatement, with `provenance$suggestions` empty.
-* Suggestions now also fire on **partial** coverage, and every suggestion
-  carries `trigger` (`"empty_year"` or `"suppressed_component"`),
-  `suppressed_amount`, `suppressed_years` and `suppressed_codes`, so a caller
-  can see how much is missing and decide whether to re-run with the recipe.
-* `cog_revenue()` gets the same fix through the shared verb path. Alaska's
-  FY2011 `Miscellaneous Revenue` reported $943,842,000 while dropping
-  $1,899,995,000 of aggregate-published `U4-` rents and royalties.
-* The trigger stays recipe-driven, so it only fires where a harmonization
-  recipe actually exists to name the fix. `higher_ed_e18_wide` and
-  `general_gov_e89_wide` stay silent in every year measured on the bundled
-  fixture, because their components are ordinary classified leaves even
-  pre-2012.
-* The `suppressed_component` trigger (and any `suppressed_amount`/
-  `suppressed_codes` an `empty_year` fire also carries) is scoped to the
-  calling verb's own flow family: `cog_spending()` only ever measures E/F/G
-  component dollars, `cog_revenue()` only T/A/U/B/C/D. A component from the
-  OTHER flow family reports `suppressed_amount = 0` rather than a fabricated
-  claim. The `empty_year` trigger itself is not flow-scoped -- a category
-  belonging to the other flow (e.g. `cog_spending(category = "IG Local")`)
-  still returns zero rows and can still fire, in any year including modern
-  ones, naming the recipe whose own generic join finds real data for this
-  government. That is a mis-scoped query, not a corpus-format gap, so its
-  `suppressed_amount` is correctly 0.
-
-## New: `cog_balances()` for cash-and-security holdings
-
-* New `cog_balances()` exposes the 14 cash-and-security holding codes
-  (`category_type = "balance"`): fund balances, retirement system holdings and
-  insurance trust balances (#25). Holdings are a stock, not a flow, so the verb
-  has no `expenditure_concept` / `revenue_concept` / `complete` arguments, and
-  no `subtype` argument either -- for holdings, `category` is a strict
-  coarsening of `balance_subtype`, so `category = "Fund Balances"` is exactly
-  the `general` family (`W01`/`W31`/`W61`).
-* `cog_balances()` results carry `provenance$balance_caveats`, recording that
-  Census holdings are gross rather than GAAP fund balance, and the measured
-  coverage window of each subtype family.
-
-## Multi-government aggregates now disclose their reporting coverage
-
-* The Census of Governments is a **complete census only in years ending in 2
-  and 7**; every other year is a sample, and the sample varies enormously. On
-  the bundled fixture, Wisconsin's 608-city universe rolls up **597**
-  governments in FY2012 and **112** in FY2019 — an 18%-to-98% swing the
-  return value said nothing about, so a statewide total resting on a fifth of
-  the universe looked exactly like one resting on all of it.
-* `cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` gain
-  `coverage`:
-
-  | value | effect |
-  |---|---|
-  | `"all"` (default) | every unit that reported that year — unchanged behaviour |
-  | `"census"` | census years only; aborts if the range holds none rather than returning nothing |
-  | `"consistent"` | only units reporting in *every* requested year — a balanced panel |
-
-* **Regardless of mode**, every result now carries `provenance$coverage` with
-  per-year `n_units_reporting`, `n_units_expected` and `is_census_year`, plus
-  `provenance$coverage_mode`. `cog_explain()` prints a "Reporting coverage"
-  section. So the default mode can no longer mislead silently.
-* `is_census_year` is a statement about the **survey calendar**, never a claim
-  of completeness: FY1967 is a census year in which only 97 of Wisconsin's 608
-  cities report. `n_units_reporting` is the number that tells the truth.
-* On `cog_peer_compare()` the target is exempt from `"consistent"` balancing —
-  it is the subject of the comparison, not a member of the cohort — and the
-  `summary_*` quantiles are computed after the filter, so they describe the
-  cohort actually returned. `n_units_reporting` counts peers only, against the
-  cohort size.
-* On `cog_find_peers()`, `coverage` governs the cohort **vintage** when `year`
-  is `NULL`: `"census"` snaps to the most recent census year with an observed
-  population, so a cohort is not built from a sample year in which most of the
-  candidate universe is absent.
-
-## `complete = TRUE`: absent cells, labelled with why they are absent
-
-* `cog_spending()` and `cog_revenue()` gain `complete`, defaulting to `FALSE`
-  (today's behaviour). With `complete = TRUE` the requested grid is filled
-  from the corpus's `code_set` table and every row carries a new
-  `value_source` column:
-
-  | `value_source` | meaning | `amt_nominal` |
-  |---|---|---|
-  | `reported` | the corpus carries this cell | as published |
-  | `census_zero` | dense-source year (≤ FY2011), cell absent — Census published `$0` | `0` |
-  | `not_reported` | sparse-source year (≥ FY2012), cell absent — unknown | `NA` |
-
-  The `NA` is deliberate and is the whole point: filling a modern absence
-  with `0` would invent data, which is precisely the error the corpus's
-  representation contract exists to prevent.
-* This restores information the reader lost when the corpus was sparsified
-  (`SB194`, cog_pipeline#64) — a wide-era query whose cells were all `$0`
-  had begun returning nothing at all — and improves on what came before it,
-  since the pre-sparsification corpus could not distinguish a published zero
-  from an unreported cell either.
-* The grid is scoped to each government's **own type**, so a county is never
-  filled with cells only a state can report.
-* Needs a corpus published from 2026-07-29 onward (when `representation` and
-  `code_set` began shipping); aborts with class
-  `uscogdata_representation_unavailable` otherwise. Gated on the manifest
-  listing those tables rather than on `schema_version`, which was never
-  bumped for the change. Not available with `recipe` or
-  `expenditure_concept = "total"` — neither draws its cells from `code_set`.
-* `provenance$completion` reports `applied`, `rows_filled`, and the per-year
-  `absence_means` rule; `cog_explain()` prints a "Completion" section.
-
-## Corpus-wide series breaks now reach users (`corpus_break_refs`)
-
-* Four catalogued series breaks carry `fin_code = "ALL"` — caveats about the
-  corpus as a whole rather than about one item code. `series_break_refs` is
-  built by matching `fin_code` against the item codes in the result, and no
-  row's `item_code` is ever the literal `"ALL"`, so **none of them could ever
-  be surfaced**: `SB085` (dollar precision across the 1976/1977 boundary),
-  `SB087` (imputation exclusion from FY2002), `SB194` (the dense → sparse
-  representation change at FY2012) and `SB086` (the government id scheme
-  change at FY2017).
-* Provenance gains `corpus_break_refs`, selected on the break-year window
-  alone and disjoint from `series_break_refs` by construction, so a consumer
-  can tell a whole-result caveat from a break in one series. `cog_explain()`
-  prints them under their own "Corpus-wide caveats" heading. cog-api passes
-  provenance through verbatim, so the field appears there without an API
-  change.
-* `SB194` is the one that made this urgent: a query spanning FY2011 → FY2012
-  crosses the boundary where an absent cell stops meaning "Census published
-  `$0`" and starts meaning "not reported", and until now nothing said so.
-
-## Bundled fixture regenerated against the sparsified corpus
-
-* `inst/extdata/fixture_corpus/` now tracks the corpus published on
-  2026-07-29 (`pipeline_commit 83f9715`, schema v6). The wide era no longer
-  stores explicit zeros: FY2011 fell from 2,864,212 rows to 496,004, of
-  which none are `$0`. **Absence now means two different things** — in a
-  `dense_source` year (≤ FY2011) an absent cell means Census published `$0`;
-  in a `sparse_source` year (≥ FY2012) it means not reported. The corpus
-  carries that rule in two new tables the fixture now ships,
-  `representation.parquet` and `code_set.parquet`, alongside
-  `census_collection_coverage.parquet` and `lineage_events.parquet`
-  (all ten publish-tree metadata tables, up from six). Catalogued upstream
-  as series break `SB194`.
-* `cog_categories()` gains an `assistance` spending subtype: the J-prefix
-  aid/benefit codes (`J19`, `J67`, `J68`, `J85`) are categorised now that
-  the upstream crosswalk covers every flow code carrying dollars.
-* Two consequences worth knowing about, both visible in provenance rather
-  than in returned dollars. The harmonization block's `na_rows_excluded`
-  counts only rows that exist, so wide-era codes that were zero-padded no
-  longer appear there. Coverage-gap `suggestions` are presence-based for the
-  same reason, so a recipe whose component codes were all `$0` for a given
-  government-year is no longer suggested for it.
-* `tests/testthat/test-fixture-vintage.R` pins these structural facts, so a
-  fixture left behind by a future publish fails loudly instead of letting the
-  suite pass against a corpus that no longer exists.
-
-## Breaking: corpus schema_version 4 (Phase P canonical ids)
-
-* The package now requires corpus `schema_version = 4` (`MinCorpusSchema` /
-  `MaxCorpusSchema` in `DESCRIPTION` are both `4`); older corpora built
-  against schema 3 are rejected by `cog_open()` with a clear version-mismatch
-  error. `canonical_govid` is now uniformly 12 characters across every
-  vintage the corpus covers (previously a mix of 9-char legacy ids and
-  12-char FIPS ids depending on source year) — **every hardcoded
-  `canonical_govid` literal from a pre-Phase-P corpus is now invalid** and
-  must be re-resolved via `cog_gov_search()` or the new `canonical_alias`
-  lookup table. `canonical_fips_xwalk` gains four columns
-  (`legacy_govs_id`, `census_geoid`, `id_source`; `confidence` is renamed to
-  `pop_confidence`) and a companion `canonical_alias` table ships in the
-  corpus for mapping legacy/alternate ids onto the current canonical
-  namespace. The bundled fixture corpus (`inst/extdata/fixture_corpus/`) has
-  been regenerated against the Phase P publish tree, now ships the full
-  `canonical_fips_xwalk` and `canonical_alias` master tables alongside the
-  2019-2020 long partitions, and is reproducible via
-  `data-raw/regenerate_fixture_corpus.R`.
-
-## Clearer errors when `USCOGDATA_URL` is unconfigured or returns non-JSON
-
-* `cog_open()` now aborts with the `uscogdata_url_not_configured` error
-  class when the resolved corpus URL still contains the placeholder
-  `REPLACE_WITH_SHARE_TOKEN` sentinel (or is empty). The message lists both
-  remediation paths (`Sys.setenv(USCOGDATA_URL = ...)` and
-  `options(uscogdata.url = ...)`) and points at the bundled fixture for
-  offline testing. Previously the package proceeded to fetch the placeholder
-  URL, cached the resulting HTML welcome page, and failed downstream with a
-  cryptic `jsonlite` lexical-error.
-* `.fetch_or_cache_manifest()` now parses the HTTP response body before
-  persisting it. Non-JSON responses (login pages, 404 HTML) raise
-  `uscogdata_invalid_manifest` with the URL, Content-Type, and underlying
-  parse error — and never write to the on-disk cache.
-* Manifest cache writes are now atomic (write to `manifest.json.tmp.<pid>`
-  in `cache_dir`, then `file.rename` over the target), so an interrupted
-  fetch cannot replace a previously-good cache.
-* Existing caches with non-JSON content (poisoned by the prior code path)
-  are silently refetched instead of returning a parse error to the caller.
-* Local `USCOGDATA_URL` paths whose `manifest.json` is not valid JSON now
-  surface the same `uscogdata_invalid_manifest` class with file context.
-
-## Per-capita denominators now use per-year Census F-33 population
-
-* `cog_spending()` and `cog_revenue()` previously divided all years' amounts
-  by a single ACS 2018-2022 estimate (`canonical_fips_xwalk.population_acs`),
-  producing biased per-capita values for time-series analysis. They now
-  divide by the F-33 `population` recorded on each gov-year via the new
-  `gov_population_yearly` view. Result tibbles gain a `pop_source` column
-  with values `"census_f33"` or `"unavailable"`. `notes` is updated to
-  concatenate multiple notes with `"; "`.
-
-## Peer cohorts can be set to a chosen year
-
-* `cog_find_peers()` adds a `year` argument (default: most recent year for
-  which the target has an observed population in `gov_population_yearly`).
-  The returned column previously named `population_acs` is now `population`
-  and reflects the cohort year's vintage. The cohort year is attached to the
-  returned tibble as `attr(x, "cohort_year")`.
-* `cog_peer_compare()` now stamps a `cohort_year` column on its result (read
-  from the peers tibble's attribute) and records `cohort_year` plus
-  `cohort_govids` in provenance. When the caller supplies a bare character
-  vector instead of a `cog_find_peers()` result, `cohort_year` is `NA`.
-
-## Rollups exclude govs missing population
-
-* `cog_geographic_rollup(per_capita = TRUE)` drops rows whose government has
-  `pop_source == "unavailable"` and records the dropped govids in
-  `provenance$rollup$excluded_govids`. This excludes special districts
-  (type 4) and school districts (type 5) from per-capita rollups by design.
-
-## New: vignette and provenance metadata
-
-* New vignette `population-denominators` covers the four population sources,
-  the type-4/5 coverage gap, the popyear quirk, and how to build moving-window
-  peer cohorts manually.
-* Provenance gains `transformations$per_capita$popyear_range` and
-  `pop_source_counts`. `cog_explain()` renders both.
-
-## New features
-
-* `cog_gov_search()` gains a **basket mode**: passing vector `name`
-  / `state` / `type` arguments resolves multiple place names in one
-  call and returns a tibble of canonical rows in input order, ready
-  to pipe into `cog_spending()` / `cog_revenue()`. Per-row resolution
-  follows an exact-then-substring matching algorithm with deterministic
-  disambiguation; ambiguous and missing entries are surfaced via a
-  sidecar audit tibble plus a single console summary message.
-* New exports `cog_basket_resolution()` and `cog_basket_unresolved()`
-  expose the basket sidecar for iterative query refinement.
-
-## Breaking changes
-
-* The first formal of `cog_gov_search()` was renamed from `pattern`
-  to `name`. All existing call sites in `cog_explorer/` and the
-  package itself use positional first-arg, so this rename is
-  non-breaking in practice. Callers that pass `pattern = ...` by name
-  must update to `name = ...`.
@@ -5,7 +5,18 @@
 .uscogdata_env <- new.env(parent = emptyenv())

 .uscogdata_defaults <- list(
-  url               = "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/",
+  # Public HuggingFace mirror of the published corpus: CC-BY-4.0, no
+  # credential, CDN-backed. This is the default so `library(uscogdata)`
+  # followed by a verb works with zero configuration -- previously the
+  # default was a REPLACE_WITH_SHARE_TOKEN sentinel and no document in the
+  # package supplied a working URL, so a new user had no path to a session.
+  #
+  # The trailing slash is required: every consumer concatenates onto this
+  # (see .resolve_url(), which enforces it anyway).
+  #
+  # Override with USCOGDATA_URL or options(uscogdata.url=) to read a
+  # Nextcloud share or a local copy made by cog_mirror().
+  url               = "https://huggingface.co/datasets/civilytics/us-cog-finance/resolve/main/",
  cache_dir         = NULL,
  manifest_ttl_secs = 3600L
 )
@@ -11,6 +11,30 @@
 #'   returns `result` invisibly for chaining. `"list"` returns the raw
 #'   provenance list (identical to `attr(result, "provenance")`).
 #' @return Either `result` (invisibly) or the provenance list.
+#' @section Two kinds of series break:
+#' Catalogued breaks reach you without being asked for, in two disjoint
+#' fields, because a caveat about one series and a caveat about the whole
+#' corpus are different claims:
+#'
+#' * **`series_break_refs`** — breaks matched against the item codes actually
+#'   present in this result. A break in one code you queried.
+#' * **`corpus_break_refs`** — breaks catalogued with `fin_code = "ALL"`,
+#'   which are statements about the corpus rather than about any one code:
+#'   dollar precision across the 1976/1977 boundary (`SB085`), imputation
+#'   exclusion from FY2002 (`SB087`), the FY2012 dense-to-sparse
+#'   representation change (`SB194`), and the FY2017 government-identifier
+#'   change (`SB086`). These are selected on the break-year window alone.
+#'
+#' `SB194` is the one most likely to matter: a query spanning FY2011 to FY2012
+#' crosses the boundary where an absent cell stops meaning "Census published
+#' $0" and starts meaning "not reported".
+#' @section Other provenance blocks:
+#' `transformations$units_conversion` records the `$1,000s`-to-dollars
+#' multiply that every amount column has already had applied.
+#' `transformations$per_capita` records the population denominator and its
+#' year range. `coverage` and `coverage_mode` appear on multi-government
+#' results (see [cog_geographic_rollup()]). `completion` appears when
+#' `complete = TRUE`. `balance_caveats` appears on [cog_balances()] results.
 #' @export
 cog_explain <- function(result, format = c("print", "list")) {
  format <- match.arg(format)
@@ -28,7 +28,7 @@
      "*" = "{.code Sys.setenv(USCOGDATA_URL = \"<url-or-local-path>/\")}",
      "*" = "{.code options(uscogdata.url = \"<url-or-local-path>/\")}",
      i = "For an offline smoke test, use the bundled fixture: {.code system.file(\"extdata/fixture_corpus\", package = \"uscogdata\")}.",
-      i = "For the live Civilytics corpus, request the Nextcloud share URL from the package maintainer."
+      i = "The public corpus is the default: unset USCOGDATA_URL to use it, or point it at a local copy made by {.code cog_mirror()}."
    ), class = "uscogdata_url_not_configured")
  }
  invisible(url)
@@ -54,7 +54,7 @@ cog_revenue <- function(govid, years, category = NULL,
                        per_capita = FALSE, adjust_to_year = NULL,
                        basis = c("harmonized", "raw"), recipe = NULL,
                        revenue_concept = c("general", "total"),
-                        complete = FALSE) {
+                        complete = FALSE, limit = NULL, offset = NULL) {
  # flow_prefixes no longer classifies rows (crosswalk revenue_subtype
  # membership does -- General Revenue, i.e. everything except
  # insurance_trust) -- it only scopes the recipe-suggestion machinery to
@@ -73,6 +73,8 @@ cog_revenue <- function(govid, years, category = NULL,
    basis          = basis,
    recipe         = recipe,
    revenue_concept = revenue_concept,
-    complete       = complete
+    complete       = complete,
+    limit          = limit,
+    offset         = offset
  )
 }
@@ -178,6 +178,13 @@
 #'   `recipe` or with `expenditure_concept = "total"` (class
 #'   `uscogdata_complete_unsupported`) — neither draws its cells from
 #'   `code_set`.
+#' @param limit Maximum number of result rows to return, pushed into the SQL
+#'   query itself (`LIMIT`/`OFFSET`) rather than applied after the full
+#'   result is materialized. `NULL` (the default) returns every matching row,
+#'   exactly as before this parameter existed. Mutually exclusive with
+#'   `recipe` and with `complete = TRUE` -- see `offset` and `total_rows`.
+#' @param offset Rows to skip before `limit` starts counting (0-based).
+#'   Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.
 #' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
 #'   `spend_subtype`, `category`, `amt_nominal`, optional `amt_real`,
 #'   optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
@@ -185,13 +192,17 @@
 #'   and `value_source` when `complete = TRUE`.
 #'   Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`,
 #'   whose `completion` block reports `applied`, `rows_filled`, and the
-#'   per-year `absence_means` rule that was applied.
+#'   per-year `absence_means` rule that was applied. When `limit` is set,
+#'   also carries a `total_rows` attribute: the full unpaginated row count,
+#'   computed by the same query (`COUNT(*) OVER()`) rather than a second
+#'   round trip -- so a caller walking pages never has to ask "how many are
+#'   there" separately.
 #' @export
 cog_spending <- function(govid, years, category = NULL,
                         per_capita = FALSE, adjust_to_year = NULL,
                         basis = c("harmonized", "raw"), recipe = NULL,
                         expenditure_concept = c("primary", "direct", "total"),
-                         complete = FALSE) {
+                         complete = FALSE, limit = NULL, offset = NULL) {
  # flow_prefixes no longer classifies rows (crosswalk subtype membership
  # does, per expenditure_concept) -- it only scopes the recipe-suggestion
  # machinery to this verb's recipe families (see R/suggestions.R; the
@@ -210,7 +221,9 @@ cog_spending <- function(govid, years, category = NULL,
    basis         = basis,
    recipe        = recipe,
    expenditure_concept = expenditure_concept,
-    complete      = complete
+    complete      = complete,
+    limit         = limit,
+    offset        = offset
  )
 }

@@ -238,7 +251,7 @@ cog_spending <- function(govid, years, category = NULL,
                           basis = c("harmonized", "raw"), recipe = NULL,
                           expenditure_concept = c("primary", "direct", "total"),
                           revenue_concept = c("general", "total"),
-                           complete = FALSE) {
+                           complete = FALSE, limit = NULL, offset = NULL) {
  basis_explicit <- length(basis) == 1L
  basis <- match.arg(basis, c("harmonized", "raw"))
  # match.arg() itself throws a base `simpleError`, not an rlang-classed
@@ -344,6 +357,38 @@ cog_spending <- function(govid, years, category = NULL,
    )
  }

+  # limit/offset push the page into the SQL itself (see .build_verb_sql()),
+  # so the two things that would make "a page of what" ambiguous are refused
+  # up front rather than silently ignored: complete = TRUE fills a grid over
+  # the FULL requested (year, category) space, and a recipe's result comes
+  # from .run_recipe()'s own query, which this function does not touch.
+  if (!is.null(limit)) {
+    limit <- as.integer(limit)
+    if (length(limit) != 1L || is.na(limit) || limit < 0L) {
+      cli::cli_abort("`limit` must be a single non-negative integer.",
+                     class = "uscogdata_invalid_pagination")
+    }
+    offset <- if (is.null(offset)) 0L else as.integer(offset)
+    if (length(offset) != 1L || is.na(offset) || offset < 0L) {
+      cli::cli_abort("`offset` must be a single non-negative integer.",
+                     class = "uscogdata_invalid_pagination")
+    }
+    if (complete) {
+      cli::cli_abort(c(
+        "`limit`/`offset` cannot be combined with `complete = TRUE`.",
+        "i" = "`complete` fills a grid over the FULL requested (year, category) space; paginating a slice of already-grouped rows has no defined meaning for the cells it would fill.",
+        "*" = "Drop `limit`/`offset`, or drop `complete`."
+      ), class = "uscogdata_complete_pagination_conflict")
+    }
+    if (!is.null(recipe)) {
+      cli::cli_abort(c(
+        "`limit`/`offset` cannot be combined with `recipe`.",
+        "i" = "A recipe's result comes from a separate query (`.run_recipe()`) that pagination is not wired into yet.",
+        "*" = "Drop `limit`/`offset`, or drop `recipe`."
+      ), class = "uscogdata_recipe_pagination_conflict")
+    }
+  }
+
  years   <- as.integer(years)
  if (!is.null(adjust_to_year)) adjust_to_year <- as.integer(adjust_to_year)

@@ -356,6 +401,7 @@ cog_spending <- function(govid, years, category = NULL,

  recipe_block <- NULL
  category_for_prov <- category
+  total_rows <- NULL  # set below only when limit is non-NULL (non-recipe path)
  if (!is.null(recipe)) {
    .require_schema_v5(con, manifest, "recipe =")
    .validate_recipe_id(con, recipe)
@@ -380,8 +426,29 @@ cog_spending <- function(govid, years, category = NULL,
    sql <- .build_verb_sql(view, subtype_col, govid, years,
                           if (all_categories) NULL else category,
                           ig_view, subtype_scope,
-                           all_categories = all_categories)
+                           all_categories = all_categories,
+                           limit = limit, offset = offset)
    result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
+    if (!is.null(limit)) {
+      # COUNT(*) OVER() rides along as an ordinary column so the total comes
+      # from the same scan when this page has any rows -- see
+      # .build_verb_sql(). An empty page (offset past the end) carries no
+      # such row to read it from, so that one case falls back to a second,
+      # unpaginated COUNT(*) query rather than reporting a wrong zero.
+      if (nrow(result) > 0L) {
+        total_rows <- result$pagination_total_rows[[1]]
+        result$pagination_total_rows <- NULL
+      } else {
+        count_sql <- sprintf(
+          "SELECT COUNT(*) AS n FROM (%s) AS _uncounted",
+          .build_verb_sql(view, subtype_col, govid, years,
+                          if (all_categories) NULL else category,
+                          ig_view, subtype_scope,
+                          all_categories = all_categories)
+        )
+        total_rows <- as.integer(DBI::dbGetQuery(con, count_sql)$n[[1]])
+      }
+    }
  }

  # Fill BEFORE per_capita / inflation so the added cells get the same
@@ -546,6 +613,10 @@ cog_spending <- function(govid, years, category = NULL,
  prov$scope$govids_missing <- scope$missing
  attr(result, "provenance") <- prov
  attr(result, ".popyear_range") <- NULL
+  # Attached here, after every downstream transform (per_capita/real-dollar
+  # joins, notes, subtype filtering), the same way provenance is -- an
+  # attribute set before those runs is not guaranteed to survive them.
+  if (!is.null(limit)) attr(result, "total_rows") <- total_rows

  if (length(suggestions) > 0L) .inform_suggestions(suggestions)

@@ -671,7 +742,7 @@ cog_spending <- function(govid, years, category = NULL,
 #' @noRd
 .build_verb_sql <- function(view, subtype_col, govid, years, category,
                            ig_view = NULL, subtype_scope = NULL,
-                            all_categories = FALSE) {
+                            all_categories = FALSE, limit = NULL, offset = NULL) {
  govid_lit <- .sql_lit_chr(govid)
  years_lit <- paste(as.integer(years), collapse = ",")
  # In all-categories mode there is no category filter: the sum is defined by
@@ -731,7 +802,7 @@ cog_spending <- function(govid, years, category = NULL,
  }
  category_group <- if (all_categories) "" else ", category"

-  sprintf(
+  base_sql <- sprintf(
    "SELECT
       year,
       canonical_govid,
@@ -751,6 +822,27 @@ cog_spending <- function(govid, years, category = NULL,
    subtype_col, source_expr, govid_lit, years_lit, category_pred, subtype_pred,
    category_select, category_group
  )
+
+  # limit/offset push the page into the query itself instead of pulling every
+  # matching row across the network only to slice and discard most of it
+  # afterward (the pattern behind the 2026-08-06 production incident: a
+  # 193,105-row/194-page sweep re-ran the full query and re-listified every
+  # row on EVERY page). COUNT(*) OVER() rides along as an ordinary column so
+  # the caller gets the true total from this same scan -- see the call site
+  # in .verb_spendrev(), which reads it off row 1 and strips it back out.
+  # The outer SELECT * wrapping (rather than appending LIMIT/OFFSET directly
+  # to base_sql) is what makes COUNT(*) OVER() see the post-GROUP-BY row
+  # count, not the pre-aggregation one.
+  if (is.null(limit)) {
+    base_sql
+  } else {
+    sprintf(
+      "SELECT *, COUNT(*) OVER() AS pagination_total_rows
+       FROM (%s) AS _paged
+       LIMIT %d OFFSET %d",
+      base_sql, limit, offset
+    )
+  }
 }

 #' @noRd
@@ -78,6 +78,55 @@
  file %in% basename(paths)
 }

+#' Build the SQL path expression for the partitioned `long` table.
+#'
+#' DuckDB cannot expand a glob over generic HTTP: there is no directory
+#' listing to expand against, and `allow_asterisks_in_http_paths` only
+#' forwards the literal `**/*` as a filename, which 404s. Measured against
+#' the published corpus on 2026-08-08, an explicit file list returns the
+#' same 46,148,034 rows the (working) `hf://` glob does, and
+#' `hive_partitioning = true` still recovers `year` from the paths.
+#'
+#' The manifest already enumerates every partition, so we build the list
+#' from it. This is host-agnostic -- Nextcloud, HuggingFace and a local
+#' fixture take the same path -- where an `hf://` URL would tie the reader
+#' to one vendor's protocol and still need special-casing, since manifest
+#' fetching goes through httr2, which cannot speak `hf://`.
+#'
+#' Falls back to the glob when the manifest carries no partition list: a
+#' hand-built manifest in a test (see test-views.R) or a corpus predating
+#' the field. Both are local, where globbing works.
+#' @noRd
+.long_files_sql <- function(url, manifest) {
+  parts <- manifest$files$long_partitions %||% list()
+  if (length(parts) == 0L) {
+    return(.sql_lit_chr(paste0(url, "data/long/**/*.parquet")))
+  }
+  paths <- vapply(parts, function(p) as.character(p$path), character(1))
+  paste0("[", .sql_lit_chr(paste0(url, paths)), "]")
+}
+
+#' Substitute the corpus-location tokens in a view's SQL text.
+#'
+#' One place knows the token vocabulary. `.register_views()` and the tests
+#' that execute a view file directly both route through here. This exists
+#' because four test sites had hand-rolled the `{url}` substitution -- one
+#' of them commented as doing it "exactly as .register_views() does" -- and
+#' every one of them broke the moment a second token was introduced.
+#'
+#' `{long_files}` must be substituted BEFORE `{url}`: it expands to a string
+#' that itself contains the url, so the reverse order leaves the token in
+#' place and DuckDB's parser fails on the brace.
+#'
+#' `manifest` defaults to empty, which routes `.long_files_sql()` to its glob
+#' fallback -- correct for the local temp corpora the direct-execution tests
+#' build.
+#' @noRd
+.render_view_sql <- function(sql, url, manifest = list()) {
+  sql <- gsub("\\{long_files\\}", .long_files_sql(url, manifest), sql, fixed = FALSE)
+  gsub("\\{url\\}", url, sql, fixed = FALSE)
+}
+
 #' Register DuckDB views from inst/sql/ SQL files
 #' @noRd
 .register_views <- function(con, url, manifest) {
@@ -91,7 +140,7 @@
        !.corpus_has_table(manifest, .representation_view_files[[base]])) next
    if (base %in% .balance_view_files && !.corpus_has_balance_subtype(con)) next
    sql <- paste(readLines(f, warn = FALSE), collapse = "\n")
-    sql <- gsub("\\{url\\}", url, sql, fixed = FALSE)
+    sql <- .render_view_sql(sql, url, manifest)
    DBI::dbExecute(con, sql)
  }
 }
@@ -1,24 +1,116 @@
 # uscogdata

-Curated R reader for the Civilytics US Census of Governments finance corpus.
+<!-- badges: start -->
+[![r-universe](https://civilytics.r-universe.dev/badges/uscogdata)](https://civilytics.r-universe.dev/uscogdata)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md)
+<!-- badges: end -->

-Provides unit-level financial profiles, geographic rollups, and peer comparisons
-with auditable provenance and built-in cross-vintage correctness. Reads the
-published corpus (Hive-partitioned parquet + manifest.json) directly from
-Nextcloud via DuckDB httpfs — no local bulk downloads required.
+A curated R reader for the Civilytics US Census of Governments finance corpus —
+every dollar that US state, county, municipal and township governments reported
+raising and spending, from **FY1967 to FY2024**, in one queryable place.

-## Status
+The Census of Governments is the only nationwide source for local government
+finance, and it is hard to use: item codes change meaning across vintages,
+government identifiers were renumbered in 2017, and an absent value means
+"published zero" in one era and "not reported" in the next. This package
+handles each of those problems, and it tells you when it has — every result
+carries provenance describing what was converted, what was aggregated, and
+which known series breaks intersect your query.

-Under active development (Phase 2 of the cog_pipeline project). See
-`../cog_pipeline/docs/reader-specification.md` for the reader contract this
-package implements.
+**Scope:** government types 0–3 (state, county, municipality, township).
+56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
+or FY1969. Special districts (type 4) and school districts (type 5) are
+excluded pending validation.

-## Installation
+## Where the data comes from
+
+The corpus is published and documented at the **[US Census of Governments
+Finance API](https://pages.civilytics.org/cog-api/)**. Start there for how the
+data was built, how the identifier and item-code reconciliation works, and what
+the corpus does and does not cover.
+
+- **[API documentation and walkthroughs](https://pages.civilytics.org/cog-api/)**
+  — reference, data dictionary, and worked examples such as the
+  [Southern states guide](https://pages.civilytics.org/cog-api/cog-api-south-guide.html)
+- **[Live API](https://cog-api.civilytics.org/api/v1/)** — the same corpus over
+  HTTP, for Tableau, Python, or anything that isn't R
+- **[Bulk corpus on Hugging Face](https://huggingface.co/datasets/civilytics/us-cog-finance)**
+  — CC-BY-4.0; the same parquet files this package reads
+- **[Census Bureau source data](https://www.census.gov/programs-surveys/gov-finances.html)**
+  — the underlying public files
+
+## Install

 ```r
-# pak::pkg_install("gitea.civilytics.org/Civilytics/uscogdata")
+install.packages("uscogdata",
+  repos = c("https://civilytics.r-universe.dev",
+            "https://cloud.r-project.org"))
 ```

+Or from source:
+
+```r
+pak::pkg_install("git::https://gitea.civilytics.org/Civilytics/uscogdata.git")
+```
+
+## Quickstart
+
+No configuration, no credentials, no download. The package reads the published
+corpus over HTTPS by default.
+
+```r
+library(uscogdata)
+
+# Resolve a place name to a canonical government id
+madison <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+madison$canonical_govid
+#> [1] "552025209777"
+
+# Police spending, inflation-adjusted and per capita
+spend <- cog_spending(
+  madison$canonical_govid,
+  years          = 2012:2022,
+  category       = "Police",
+  per_capita     = TRUE,
+  adjust_to_year = 2023
+)
+
+# What did that result do to the numbers, and what should you know about them?
+cog_explain(spend)
+```
+
+`years` is required — there is no implicit full-history default.
+
+## Two ways to read the corpus
+
+| | Remote (default) | Mirrored |
+|---|---|---|
+| Setup | none | `cog_mirror(dest)`, 190.6 MB once |
+| Disk used | **0 MB** — HTTP range requests only | 190.6 MB |
+| Per query | ~4 s (one government, one year)<br>~6 s (one government, 23 years) | local speed |
+| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
+
+Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
+works out which row groups it needs, and reads only those. Nothing is cached
+between sessions either, so every query goes back to the network.
+
+The default points at a public HuggingFace mirror of the corpus. If you would
+rather not depend on a third party — for reproducibility, for an air-gapped
+environment, or on principle — **the escape hatch is one function call**:
+
+```r
+cog_mirror("~/cog-corpus")
+Sys.setenv(USCOGDATA_URL = "~/cog-corpus/")
+```
+
+After that, nothing in your analysis touches an external service.
+
+### Configuration
+
+- `USCOGDATA_URL` — corpus root: an HTTPS URL or a local path, **trailing slash required**
+- `USCOGDATA_CACHE_DIR` — where the manifest is cached (default: user cache dir)
+- `USCOGDATA_MANIFEST_TTL_SECS` — manifest re-fetch interval (default 3600)
+
 ## Amounts are in full US dollars

 Every amount column this package returns — `amt_nominal`, `amt_real`,
@@ -29,111 +121,127 @@ own `amt` column preserves that. The verbs multiply by 1000 on the way out, so
 you never have to. The conversion is recorded in every result:

 ```r
-r <- cog_spending("552025209777", 2020L)
-attr(r, "provenance")$transformations$units_conversion
-#> $applied TRUE  $source_unit "$1,000s (raw Census)"  $target_unit "$USD"  $multiplier 1000
+attr(spend, "provenance")$transformations$units_conversion
+#> $applied     TRUE
+#> $source_unit "$1,000s (raw Census)"
+#> $target_unit "$USD"
+#> $multiplier  1000
 ```

 **Do not multiply again.** If you have read elsewhere that COG amounts are in
-`$1,000s` — true of the raw corpus, and of `cog_explorer`'s conventions doc —
-that rule does not apply to anything a `cog_*()` verb hands you. Applying it
-twice overstates every figure by 1000x, and the result looks plausible rather
-than obviously wrong.
+`$1,000s` — which is true of the raw Census files and of the corpus's own `amt`
+column — that rule does not apply to anything a `cog_*()` verb hands you.
+Applying it twice overstates every figure by 1000x, and the result looks
+plausible rather than obviously wrong.

-## Configuration
+## Concepts worth understanding before you publish a number

- `USCOGDATA_URL` — corpus root URL (public Nextcloud share, trailing slash)
- `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
- `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
-
-## Primary vs Direct vs Total spending
+### Primary vs Direct vs Total spending

 `cog_spending(..., expenditure_concept = c("primary", "direct", "total"))`
-controls whose spending a result counts. Concepts are defined as sets of the
-crosswalk's `spend_subtype` values — never item-code first letters, which
-cannot classify correctly (the letter `Y` alone spans revenue, expenditure,
-and balance codes):
+controls *whose* spending a result counts. Concepts are defined as sets of the
+crosswalk's `spend_subtype` values, never item-code first letters — the letter
+`Y` alone spans revenue, expenditure and balance codes.

- `"primary"` (the default) is the government's own service provision:
-  current operations, capital outlay, and assistance payments.
- `"direct"` is Census's published Direct Expenditure: `primary` plus
-  interest on debt and insurance trust benefit payments (e.g. pensions).
- `"total"` additionally adds the intergovernmental leg — money handed to
-  other governments to spend (`M`/`L` codes plus `Q11`/`Q12`/`Q18` state
-  payments to school systems) — which is meaningful for describing one
-  government's own budget over time, but double-counts when summed across
-  governments (a state's payment to a county is the same dollar the county
-  reports as its own direct spending).
+- **`"primary"`** (default) — the government's own service provision: current
+  operations, capital outlay, assistance payments.
+- **`"direct"`** — Census's published Direct Expenditure: `primary` plus
+  interest on debt and insurance trust benefits (e.g. pensions).
+- **`"total"`** — adds the intergovernmental leg, money handed to other
+  governments to spend. Meaningful for one government's own budget over time,
+  but it double-counts when summed across governments: a state's payment to a
+  county is the same dollar the county reports as its own direct spending.

-**Rule of thumb: any figure that spans more than one government uses
-`primary` or `direct`.** `cog_geographic_rollup()` and `cog_peer_compare()`
-enforce this by refusing `expenditure_concept = "total"`. See
-`vignette("total-spending", package = "uscogdata")` for the full
-explanation with worked examples.
+**Rule of thumb: any figure spanning more than one government uses `primary`
+or `direct`.** `cog_geographic_rollup()` and `cog_peer_compare()` enforce that
+by refusing `"total"` outright. Worked examples in
+`vignette("total-spending", package = "uscogdata")`.

-## General vs Total revenue
+### General vs Total revenue

-`cog_revenue(..., revenue_concept = c("general", "total"))` selects between
-Census's two published revenue concepts, again defined as crosswalk
-`revenue_subtype` sets rather than item-code prefixes:
+`cog_revenue(..., revenue_concept = c("general", "total"))`:

- `"general"` (the default) is Census **General Revenue**: own-source
-  (taxes, charges, miscellaneous) plus federal, state and local
-  intergovernmental aid.
- `"total"` is Census **Total Revenue**: `general` plus utility revenue
-  (`A91`–`A94`), liquor store revenue (`A90`), and insurance trust revenue
-  (unemployment and workers' compensation `Y` codes plus the
-  employee-retirement `X` codes).
+- **`"general"`** (default) — Census General Revenue: own-source taxes,
+  charges and miscellaneous, plus federal, state and local aid.
+- **`"total"`** — General plus utility revenue (`A91`–`A94`), liquor store
+  revenue (`A90`), and insurance trust revenue.

-The manual defines the first by subtracting the other three from the second,
-so the two are related by Census's own identity:
+Census defines these by its own identity:

 ```
 Total Revenue = General + Utility + Liquor Store + Insurance Trust
 ```

-Two things worth knowing before switching to `"total"`:
+Two things to know before switching to `"total"`. **Utility revenue is large
+for cities** — measured on the bundled fixture, utility plus liquor store is
+15.9% of city revenue, against 1.2% for states and 1.7% for counties. And the
+**employee-retirement (`X`) codes stop at FY2016**, when those systems moved to
+the separate Annual Survey of Public Pensions, so a `"total"` series steps down
+at the FY2016/FY2017 boundary for reasons of collection scope, not revenue
+(series breaks `SB197`–`SB209`).

- **Utility revenue is large for cities.** Measured on the bundled fixture,
-  utility plus liquor store revenue is 15.9% of city (type 2) revenue, versus
-  1.2% for states and 1.7% for counties. `general` excludes it by definition.
- **The employee-retirement (`X`) codes stop at FY2016**, when those systems
-  moved out of the annual finance file into the separate Annual Survey of
-  Public Pensions. A `"total"` series therefore steps down at the
-  FY2016/FY2017 seam for reasons of collection scope, not revenue (series
-  breaks `SB197`–`SB202`, in the corpus's `series_breaks` table).
+### Reporting coverage: the Census is only sometimes a census

-## Developer notes
+**The Census of Governments is a complete enumeration only in years ending in
+2 and 7.** Every other year is a sample, and the sample varies enormously —
+measured on the bundled fixture, Wisconsin's 608-city universe rolls up 597
+governments in FY2012 and 112 in FY2019.

-### Testing
-
-The package ships a bundled fixture corpus at `inst/extdata/fixture_corpus/` —
-a 15 MB four-year slice (2011, 2012, 2019, 2020) of the full corpus covering
-all 50 states. `tests/testthat/setup.R` automatically points `USCOGDATA_URL`
-at this fixture, so the full test suite runs offline with no network
-dependency:
+A statewide total resting on a fifth of the universe looks exactly like one
+resting on all of it, so every multi-government result now says which it is:

 ```r
-devtools::test()   # uses bundled fixture, no credentials required
+attr(rollup, "provenance")$coverage   # per-year n_units_reporting, is_census_year
 ```

-### Releasing against the live corpus
+`cog_geographic_rollup()`, `cog_peer_compare()` and `cog_find_peers()` take a
+`coverage` argument — `"all"` (default), `"census"` (census years only), or
+`"consistent"` (only units reporting in every requested year, a balanced
+panel).

-Before cutting a release, run the test suite against the published corpus to
-catch any drift between the fixture and the real data:
+`n_units_reporting` is **category-conditional**, and it is not a response rate. A government that was surveyed and genuinely spends
+nothing in the requested category is indistinguishable from one never surveyed.
+
+### Absent cells mean two different things
+
+Before FY2012, an absent cell means Census published `$0`. From FY2012 on, it
+means not reported. `cog_spending(..., complete = TRUE)` fills the requested
+grid and labels every row with which it is, via `value_source`:
+
+| `value_source` | meaning | `amt_nominal` |
+|---|---|---|
+| `reported` | the corpus carries this cell | as published |
+| `census_zero` | dense-source year (≤ FY2011), absent — Census published `$0` | `0` |
+| `not_reported` | sparse-source year (≥ FY2012), absent — unknown | `NA` |
+
+That `NA` is deliberate. Filling a modern absence with `0` would invent data.
+
+### Series breaks surface on their own
+
+Catalogued breaks that intersect your query appear in provenance whether or not
+you went looking for them — `series_break_refs` for breaks in a specific item code, and
+`corpus_break_refs` for caveats about the corpus as a whole (dollar precision
+across the 1976/1977 boundary, the FY2017 identifier change, the FY2012
+dense→sparse representation change). `cog_explain()` prints both.
+
+## How to cite

 ```r
-Sys.setenv(USCOGDATA_URL = "<published-corpus-url-with-trailing-slash>")
-devtools::test()
+citation("uscogdata")
 ```

-When the live-corpus run is clean, strip the fixture from the built package by
-adding this line to `.Rbuildignore`:
+The corpus itself is published under CC-BY-4.0. Cite it as:

-```
-^inst/extdata/fixture_corpus$
-```
+> Civilytics Consulting. US Census of Governments finance corpus.
+> https://huggingface.co/datasets/civilytics/us-cog-finance

-The test suite is URL-agnostic — `setup.R` falls back to `USCOGDATA_URL` when
-the bundled fixture is absent, so no test code changes are needed for the
-release run or after stripping the fixture.
+## Contributing
+
+Development happens on [Gitea](https://gitea.civilytics.org/Civilytics/uscogdata);
+[GitHub](https://github.com/civilytics/uscogdata) is a mirror that accepts
+issues and pull requests. See [CONTRIBUTING.md](CONTRIBUTING.md) for how a
+patch gets from there to here.
+
+## License
+
+MIT © Civilytics Consulting LLC. See [LICENSE.md](LICENSE.md).
@@ -1,4 +1,5 @@
-url: ~
+url: https://civilytics.r-universe.dev/uscogdata
+
 template:
  bootstrap: 5

@@ -15,11 +16,26 @@ reference:
      - cog_gov_search
      - cog_basket_resolution
      - cog_basket_unresolved
-  - title: Session
+  - title: Comparison & aggregation
+    desc: Peer cohorts and geographic aggregates.
    contents:
-      - has_keyword("internal")
+      - cog_find_peers
+      - cog_peer_compare
+      - cog_geographic_rollup
+  - title: Corpus metadata
+    desc: >
+      What the corpus contains, where a given result came from, and how to
+      hold a local copy of it.
+    contents:
+      - cog_categories
+      - cog_recipes
+      - cog_manifest
+      - cog_explain
+      - cog_mirror

 articles:
-  - title: Getting started
+  - title: Concepts
    navbar: ~
-    contents: []
+    contents:
+      - total-spending
+      - population-denominators
@@ -1,7 +1,7 @@
 {
  "schema_version": 6,
-  "built_at": "2026-07-31T00:47:27Z",
-  "pipeline_commit": "aadb46b",
+  "built_at": "2026-08-03T16:51:32Z",
+  "pipeline_commit": "e7394a4",
  "fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated from the sparsified schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit zeros, so FY2011 absence means Census published $0 while FY2012+ absence means not reported. representation.parquet and code_set.parquet carry that rule and ship in full, as do every other metadata table in the publish tree. 2011/2012 straddle both the wide-aggregate -> modern-leaf format boundary (exercised by basis=\"harmonized\" and recipe= queries) and the dense -> sparse representation boundary (SB194); 2019/2020 retain the prior per-capita/CPI regression anchors. Regenerated via data-raw/regenerate_fixture_corpus.R.",
  "data_vintage": {
    "source_vintages": {
@@ -105,7 +105,7 @@
      },
      {
        "path": "data/series_breaks.parquet",
-        "sha256": "06dcc995ff533e57cc65fa25086cc9bf83ba592c58bf7cc99269dc2576f69944",
+        "sha256": "731998516cd802f63fcf7fb66053c7a62b7be955ab0794cad4a4979cb7628b87",
        "description": "series_breaks.parquet"
      },
      {
@@ -1,3 +1,7 @@
 CREATE OR REPLACE VIEW long AS
 SELECT *
-FROM read_parquet('{url}data/long/**/*.parquet', hive_partitioning = true);
+-- {long_files} carries its own quoting: a bracketed list of every partition
+-- the manifest enumerates, or a single quoted glob on fallback. Do NOT wrap
+-- it in quotes. See .long_files_sql() in R/views.R for why a glob alone
+-- cannot work over HTTP.
+FROM read_parquet({long_files}, hive_partitioning = true);
@@ -21,3 +21,33 @@ Prints the structured provenance attached to a tibble returned by any
 `cog_*` verb, or returns it as a list for downstream use (MCP tools,
 dashboards, JSON export).
 }
+\section{Two kinds of series break}{
+
+Catalogued breaks reach you without being asked for, in two disjoint
+fields, because a caveat about one series and a caveat about the whole
+corpus are different claims:
+
+* **`series_break_refs`** — breaks matched against the item codes actually
+  present in this result. A break in one code you queried.
+* **`corpus_break_refs`** — breaks catalogued with `fin_code = "ALL"`,
+  which are statements about the corpus rather than about any one code:
+  dollar precision across the 1976/1977 boundary (`SB085`), imputation
+  exclusion from FY2002 (`SB087`), the FY2012 dense-to-sparse
+  representation change (`SB194`), and the FY2017 government-identifier
+  change (`SB086`). These are selected on the break-year window alone.
+
+`SB194` is the one most likely to matter: a query spanning FY2011 to FY2012
+crosses the boundary where an absent cell stops meaning "Census published
+$0" and starts meaning "not reported".
+}
+
+\section{Other provenance blocks}{
+
+`transformations$units_conversion` records the `$1,000s`-to-dollars
+multiply that every amount column has already had applied.
+`transformations$per_capita` records the population denominator and its
+year range. `coverage` and `coverage_mode` appear on multi-government
+results (see [cog_geographic_rollup()]). `completion` appears when
+`complete = TRUE`. `balance_caveats` appears on [cog_balances()] results.
+}
+
@@ -13,7 +13,9 @@ cog_revenue(
  basis = c("harmonized", "raw"),
  recipe = NULL,
  revenue_concept = c("general", "total"),
-  complete = FALSE
+  complete = FALSE,
+  limit = NULL,
+  offset = NULL
 )
 }
 \arguments{
@@ -115,6 +117,15 @@ possibly-misleading `"harmonized"`/`"raw"` value.}
  `recipe` or with `expenditure_concept = "total"` (class
  `uscogdata_complete_unsupported`) — neither draws its cells from
  `code_set`.}
+
+\item{limit}{Maximum number of result rows to return, pushed into the SQL
+query itself (`LIMIT`/`OFFSET`) rather than applied after the full
+result is materialized. `NULL` (the default) returns every matching row,
+exactly as before this parameter existed. Mutually exclusive with
+`recipe` and with `complete = TRUE` -- see `offset` and `total_rows`.}
+
+\item{offset}{Rows to skip before `limit` starts counting (0-based).
+Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,
@@ -13,7 +13,9 @@ cog_spending(
  basis = c("harmonized", "raw"),
  recipe = NULL,
  expenditure_concept = c("primary", "direct", "total"),
-  complete = FALSE
+  complete = FALSE,
+  limit = NULL,
+  offset = NULL
 )
 }
 \arguments{
@@ -135,6 +137,15 @@ possibly-misleading `"harmonized"`/`"raw"` value.}
  `recipe` or with `expenditure_concept = "total"` (class
  `uscogdata_complete_unsupported`) — neither draws its cells from
  `code_set`.}
+
+\item{limit}{Maximum number of result rows to return, pushed into the SQL
+query itself (`LIMIT`/`OFFSET`) rather than applied after the full
+result is materialized. `NULL` (the default) returns every matching row,
+exactly as before this parameter existed. Mutually exclusive with
+`recipe` and with `complete = TRUE` -- see `offset` and `total_rows`.}
+
+\item{offset}{Rows to skip before `limit` starts counting (0-based).
+Ignored if `limit` is `NULL`; defaults to `0L` when `limit` is set.}
 }
 \value{
 Tibble with columns `year`, `canonical_govid`, `gov_name`,
@@ -144,7 +155,11 @@ Tibble with columns `year`, `canonical_govid`, `gov_name`,
  and `value_source` when `complete = TRUE`.
  Carries a `provenance` attribute matching `inst/schemas/provenance-v1.json`,
  whose `completion` block reports `applied`, `rows_filled`, and the
-  per-year `absence_means` rule that was applied.
+  per-year `absence_means` rule that was applied. When `limit` is set,
+  also carries a `total_rows` attribute: the full unpaginated row count,
+  computed by the same query (`COUNT(*) OVER()`) rather than a second
+  round trip -- so a caller walking pages never has to ask "how many are
+  there" separately.
 }
 \description{
 One row per `(year, canonical_govid, spend_subtype, category)`. Amounts are
@@ -0,0 +1,296 @@
+# `uscogdata` 0.3.0 — public release
+
+**Date:** 2026-08-08 · **Status:** design, awaiting approval
+**Scope:** release-readiness, README, NEWS. Distribution mechanics recorded here as
+decided, sequenced after the package is clean.
+
+`uscogdata` is feature-complete and the corpus it reads has been public on
+HuggingFace since 2026-08-07 (294 downloads as of this writing). The API built on
+it is live. What does not exist is a public *package*: the repo is private, there
+is no install path, and — measured, not assumed — **a stranger who installed it
+today could not read the corpus at all.**
+
+This spec covers making that untrue.
+
+## Decisions locked
+
+| Decision | Choice |
+|---|---|
+| Canonical source | `gitea.civilytics.org/Civilytics/uscogdata`, flipped public |
+| Public mirror | `github.com/civilytics/uscogdata` — issues, PRs, multi-OS check, CDN |
+| Mirror mechanism | Gitea Actions non-force `git push` (not a push mirror) |
+| Binaries | `civilytics.r-universe.dev`, registry pinned to a release tag |
+| Author of record | Jared E. Knowles `<jared@civilytics.com>`, ORCID `0000-0003-0005-9478` |
+| Copyright | Civilytics Consulting LLC (`cph`, `fnd`) |
+| License | MIT (package) · CC-BY-4.0 (corpus) |
+| Corrections intake | Deferred — see *Out of scope* |
+| Other packages | Parked until this one walks the path end to end |
+
+## P0 — the corpus is unreachable
+
+Two independent faults, either of which alone is fatal.
+
+**No corpus URL exists.** `R/config.R` defaults to the literal
+`REPLACE_WITH_SHARE_TOKEN` sentinel, and no file in the repo supplies a working
+one. A new user calling any verb gets `uscogdata_url_not_configured` with no path
+to resolution.
+
+**Remote reads are broken regardless.** Every partitioned view globs:
+
+```sql
+FROM read_parquet('{url}data/long/**/*.parquet', hive_partitioning = true)
+```
+
+DuckDB 1.5.5 refuses globs over generic HTTP. Its suggested
+`allow_asterisks_in_http_paths` escape hatch does not help — it forwards the
+literal `**/*` as a filename and 404s, because plain HTTP exposes no directory
+listing to expand against.
+
+The package therefore works only against a **local path**. That is how the API
+runs it (`CORPUS_HOST_PATH` is a host mount on maxwell) and how the tests run
+(bundled fixture), which is why the fault went unnoticed. The README's headline
+claim — *"Reads the published corpus directly from Nextcloud via DuckDB httpfs —
+no local bulk downloads required"* — is currently false.
+
+### Fix: enumerate from the manifest, do not glob
+
+`manifest.json` already lists every partition under `files.long_partitions[]`
+with `path`, `year`, `sha256`, `row_count` and `size_bytes` — 56 of them.
+Substituting an explicit file list for the glob was measured against the
+published corpus on 2026-08-08:
+
+| Path | Result |
+|---|---|
+| `https://…/data/long/**/*.parquet` (default) | error — globs unsupported over HTTP |
+| same, `allow_asterisks_in_http_paths = true` | error — literal `**/*` 404s |
+| `hf://datasets/civilytics/us-cog-finance/…` glob | 46,148,034 rows |
+| **explicit list over plain https** | **46,148,034 rows** |
+
+`hive_partitioning = true` still recovers `year` from the paths under
+enumeration, so no downstream view or verb changes.
+
+Enumeration is preferred over `hf://` deliberately. It is **host-agnostic** —
+Nextcloud, HuggingFace, or any static server take the same code path — where
+`hf://` would tie the default to one vendor's protocol and still need
+special-casing, since manifest fetching goes through `httr2`, which cannot speak
+`hf://`. Enumeration also *removes* a dependency (globbing) rather than adding
+one, and the manifest's per-file `sha256` becomes available for integrity
+checking later.
+
+Views are registered from `inst/sql/` with `{url}` substitution in
+`R/views.R:.register_views()`. The list must be built once per session from the
+already-fetched manifest and substituted the same way, so the change is confined
+to view registration and does not touch verb code.
+
+### Fix: ship a working default
+
+`R/config.R`'s default becomes the public HuggingFace `resolve/main/` URL:
+CC-BY-4.0, no token to publish, CDN-backed, and it keeps maxwell's uplink out of
+the path — the same reasoning behind the GitHub mirror and r-universe.
+
+This means `library(uscogdata)` followed by a verb works with **zero
+configuration**, which is what makes the package demonstrable in a README and
+later in a post. `USCOGDATA_URL` and `options(uscogdata.url=)` continue to
+override, so the Nextcloud copy and local mirrors are unaffected.
+
+The `uscogdata_url_not_configured` error class stays — it still fires for an
+explicitly-set empty or placeholder URL — but ceases to be the default
+experience.
+
+### Consequence: `cog_mirror()` is promoted
+
+Measured cost of the remote default, from efron on a good connection:
+
+| | |
+|---|---|
+| Whole corpus | **190.6 MB**, 56 partitions, 46,148,034 rows, FY1967–FY2024 |
+| One government, one year | 1.5 s |
+| One government, all 56 years | 2.8 s |
+| Disk written | **0.00 MB** — range requests only; `external_file_cache` is in-memory |
+
+Nothing persists locally beyond the shared `httpfs` extension in `~/.duckdb` (a
+few MB, once per machine, across all DuckDB use). Costs are RAM and per-query
+bandwidth, since nothing caches between sessions.
+
+Those timings are raw scans. Real verbs additionally join crosswalks, resolve
+categories and assemble provenance, so end-to-end verb latency will be higher and
+**must be re-measured once the fix lands** — it cannot be measured today.
+
+The corpus being only 190.6 MB makes `cog_mirror()` a first-class option rather
+than a developer footnote. The README presents **both paths**:
+
+- **Remote (default, zero setup)** — trying it out, teaching, one-off questions.
+- **Mirrored (`cog_mirror()`, 190 MB once)** — repeated or heavy analysis,
+  offline work, reproducibility, or preferring not to depend on HuggingFace.
+
+The second is also the honest answer to the vendor-dependency question raised by
+defaulting to HuggingFace: **the escape hatch is one function call and 190 MB**,
+after which no analysis touches an external service. The README says so
+explicitly. That is the difference between a convenience default and lock-in.
+
+## Release-readiness fixes
+
+| # | Issue | Fix |
+|---|---|---|
+| 1 | `MaxCorpusSchema: 5` in DESCRIPTION; `.validate_schema()` accepts `4,5,6,7`; published corpus is **7** | `MaxCorpusSchema: 7` |
+| 2 | `^vignettes$` in `.Rbuildignore` — both vignettes absent from the installed package, while README tells users to run `vignette("total-spending")` | Remove `^vignettes$`, `^doc$`, `^Meta$`. Both vignettes build offline (`total-spending` reads the bundled fixture; `population-denominators` is `eval = FALSE`) |
+| 3 | `_pkgdown.yml` reference index covers 6 of 14 exports — pkgdown errors on missing topics | Add `cog_categories`, `cog_explain`, `cog_find_peers`, `cog_geographic_rollup`, `cog_manifest`, `cog_mirror`, `cog_peer_compare`, `cog_recipes`; set `url:` |
+| 4 | No `URL:` / `BugReports:` in DESCRIPTION | Add both, pointing at the GitHub mirror |
+| 5 | No `LICENSE.md`; `LICENSE` holder reads `Civilytics` | `usethis::use_mit_license("Civilytics Consulting LLC")` |
+| 6 | README instructs stripping the fixture at release | Delete that section — see below |
+| 7 | `Authors@R` is an org with no human | Jared E. Knowles `aut`/`cre` + ORCID; Civilytics Consulting LLC `cph`/`fnd` |
+
+**On #6.** The advice to add `^inst/extdata/fixture_corpus$` to `.Rbuildignore`
+is CRAN-sized thinking (5 MB limit) and this package is not going to CRAN.
+Stripping the 15 MB fixture would break `total-spending.Rmd`, which reads from
+it, and would leave r-universe and GitHub Actions unable to run the 28 test files
+without a corpus credential. **The fixture is what lets `R CMD check` pass
+anywhere with zero secrets** — precisely what public CI needs. It ships.
+
+## README
+
+The current README addresses someone standing inside the repo tree: status reads
+"Under active development (Phase 2 of the cog_pipeline project)", it points at
+`../cog_pipeline/docs/reader-specification.md`, the install line is commented
+out, and developer, testing and release sections sit above anything a user needs.
+
+Restructured around a stranger, in this order:
+
+1. **What this is** — one paragraph, and what the corpus covers (types 0–3,
+   FY1967–FY2024, 46M rows, 190.6 MB).
+2. **Install** — r-universe first (binaries), git second.
+3. **Quickstart that actually runs** — resolve a government, get its history,
+   print provenance. No configuration step.
+4. **Two ways to read the corpus** — remote default vs `cog_mirror()`, with the
+   measured numbers and the independence note.
+5. **Amounts are in full US dollars** — kept near the top. This is the errata
+   most likely to produce a wrong answer that looks plausible.
+6. **Concepts** — primary/direct/total spending, general/total revenue,
+   coverage. Condensed, linking to the vignettes for the full treatment.
+7. **How to cite** — `citation("uscogdata")`, corpus CC-BY-4.0 attribution.
+8. **Contributing** — canonical-on-Gitea PR flow.
+
+Developer notes, testing instructions and release procedure move to
+`CONTRIBUTING.md`. Every path reference to a sibling repo is removed or replaced
+with a URL that resolves for someone who has only this repo.
+
+## NEWS.md
+
+`NEWS.md` currently holds two sections. `0.2.0` is a legitimate changelog — the
+`"All Categories"` reserved value, the coverage-signposting fix, the
+`n_units_reporting` documentation — and it stays. Beneath it,
+`0.1.0 (development)` is a pre-release churn log: changes described relative to
+states no user has ever seen ("Breaking: corpus schema_version 4", "the package
+now requires…"), spanning the package's entire pre-release development. To a
+newcomer deciding whether to depend on this, that section reads as instability.
+
+**A new `0.3.0` section is added at the top, framed as the first public
+release**: what the package does, what the corpus covers, and the caveats that
+are genuinely load-bearing. **`0.2.0` is kept verbatim.** **`0.1.0 (development)`
+is dropped** — that history stays in git, where it belongs.
+
+The version is `0.3.0` rather than `0.2.0` because this release changes
+user-visible behaviour: remote corpus reads go from broken to working, and the
+default URL from a dead placeholder to a live corpus. It is also not `1.0.0` —
+the corpus still excludes government types 4 and 5 pending validation, so a
+stability promise would overclaim. No git tag exists for any prior version;
+`chore: release 0.2.0` bumped `DESCRIPTION` and `NEWS` only.
+
+The substantive content is migrated, not deleted. These are hard-won and belong
+in documentation rather than buried in a changelog:
+
+| Content | Destination |
+|---|---|
+| Coverage disclosure on multi-government aggregates (census vs sample years) | README concepts + `cog_geographic_rollup()` docs |
+| `complete = TRUE` three-way absence semantics (`reported` / `census_zero` / `not_reported`) | `cog_spending()` / `cog_revenue()` docs |
+| Series-break and corpus-break surfacing | README + `cog_explain()` docs |
+| $1,000s → full dollars conversion | README, already prominent |
+| Per-year F-33 population denominators | `population-denominators` vignette, already there |
+
+This also makes NEWS reusable as raw material for the release announcement,
+which is the stated downstream purpose.
+
+## Distribution mechanics
+
+Recorded as decided; executed after the package is clean and checks are green.
+
+**Sequence matters.** r-universe publishes check results the moment a package is
+registered. Registering before the fixes above land means a red badge on day one,
+which is a worse first impression than a week's delay.
+
+1. `gitleaks` over full history. A coarse grep found nothing across 140 commits
+   and the default corpus URL is still the placeholder sentinel, but a proper
+   scan is the gate on an irreversible action.
+2. Flip the Gitea repo public. Disable Gitea issues on it, so there is exactly
+   one inbox.
+3. Create `github.com/civilytics/uscogdata`. Add `.github/workflows/` for the
+   Windows/macOS/Linux `R CMD check` matrix — the platforms the Gitea runner
+   cannot provide, and which this package has never been tested on despite
+   depending on duckdb and httr2. Gitea reads `.gitea/workflows`, GitHub reads
+   `.github/workflows`; both live in one tree without colliding.
+4. Gitea Actions workflow pushing to GitHub **without `--force`**, so divergence
+   fails loudly in CI rather than silently overwriting.
+5. Add `jared@civilytics.com` as a verified secondary email on the GitHub
+   account — r-universe links maintainer identity by matching DESCRIPTION's email
+   against registered GitHub emails, and the association only takes effect on the
+   next build.
+6. Tag `v0.3.0`. Create `github.com/civilytics/civilytics.r-universe.dev` with a
+   `packages.json` pinned to the tag, pointing at the GitHub mirror rather than
+   Gitea so clone traffic stays off maxwell. Install the r-universe app.
+
+### PR flow
+
+Never press Merge on GitHub. A merge there is overwritten by the next sync, the
+PR still displays "Merged", and nothing says otherwise.
+
+```sh
+git remote add github https://github.com/civilytics/uscogdata.git
+git config --add remote.github.fetch '+refs/pull/*/head:refs/remotes/github/pr/*'
+git fetch github
+git switch -c pr-42 github/pr/42      # test
+git switch main && git merge --no-ff pr-42
+git push origin main                   # Gitea -> mirror -> GitHub
+```
+
+GitHub auto-closes a PR as merged once its head commit becomes an ancestor of the
+base branch, so `--no-ff` — which preserves the contributor's SHAs — makes the PR
+close itself when the mirror pushes. **For external PRs, merge; do not squash or
+rebase.** Squashing rewrites the SHAs, the auto-close never fires, and closing by
+hand reads to a first-time contributor as rejection.
+
+`CONTRIBUTING.md` states this, and a GitHub Action comments it on incoming PRs.
+No CLA; no DCO.
+
+## Verification
+
+The release is not done until all of these pass:
+
+1. `R CMD check --as-cran` clean on Linux, and on Windows and macOS via the
+   GitHub matrix. This package has never been checked on the latter two.
+2. Full test suite (28 files) green against the **bundled fixture**, offline,
+   with no credentials — the property public CI depends on.
+3. Full test suite green against the **live corpus**, which additionally
+   exercises the enumeration fix that the fixture's local path cannot.
+4. `pkgdown::build_site()` completes.
+5. Both vignettes present in the built tarball and
+   `vignette("total-spending", package = "uscogdata")` resolves from an
+   installed copy.
+6. **Cold-start check on a machine that has never seen this package:** install
+   from r-universe, `library(uscogdata)`, run the README quickstart verbatim with
+   no environment variables set. This is the only test that catches the P0 class
+   of fault, and its absence is why the fault survived.
+7. End-to-end verb latency re-measured against the live corpus and the README's
+   numbers updated if they moved.
+
+## Out of scope
+
+- **Corrections intake.** Deferred by decision. Consequence: the release cannot
+  invite data-error reports or make the "traceable and correctable" claim that
+  most distinguishes this corpus from Census's own files. `BugReports:` points at
+  package issues only. A verified correction should eventually terminate as a
+  `lineage_event` or `series_break` row so it propagates through provenance to
+  every consumer — that design is unstarted.
+- **Announcement posts.** Deferred. The API announcement is gated on corrections
+  landing and merits a Civic Pulse edition.
+- **The rest of the R package backlog.** Parked until this one completes the path.
+- **`cog_pipeline` publication.** Stays private.
@@ -66,7 +66,7 @@ test_that("inst/sql/26-balance_long.sql enforces NOT is_aggregate (real SQL text
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
  }

  con <- DBI::dbConnect(duckdb::duckdb())
@@ -64,3 +64,109 @@ test_that(".resolve_url does not invent a slash for an empty setting", {
  withr::local_options(uscogdata.url = "")
  expect_equal(.resolve_url(), "")
 })
+
+test_that("the default corpus URL is real, not a placeholder", {
+  # setup.R points USCOGDATA_URL at the bundled fixture for the whole suite,
+  # so both the env var and the option have to be cleared to see the default.
+  withr::local_envvar(USCOGDATA_URL = NA)
+  withr::local_options(uscogdata.url = NULL)
+  url <- .resolve_url()
+  expect_false(grepl("REPLACE_WITH", url, fixed = TRUE))
+  expect_match(url, "^https://")
+  expect_match(url, "/$")
+})
+
+test_that("an explicitly-set sentinel URL still aborts", {
+  # The guard must survive the default change: a user who half-edited a
+  # copied config still gets the actionable error.
+  withr::local_envvar(
+    USCOGDATA_URL = "https://other.example/s/REPLACE_WITH_SHARE_TOKEN/x/"
+  )
+  expect_error(
+    .check_url_configured(.resolve_url()),
+    class = "uscogdata_url_not_configured"
+  )
+})
+
+test_that("DESCRIPTION carries release metadata", {
+  skip_if_no_source_tree("DESCRIPTION")
+  d <- read.dcf(source_tree_path("DESCRIPTION"))
+  fields <- colnames(d)
+
+  expect_true(all(c("URL", "BugReports") %in% fields))
+  expect_match(d[1, "Authors@R"], "Knowles", fixed = TRUE)
+  expect_match(d[1, "Authors@R"], "0000-0003-0005-9478", fixed = TRUE)
+  expect_match(d[1, "Authors@R"], "Civilytics Consulting LLC", fixed = TRUE)
+
+  # The gate in .validate_schema() accepts up to 7 and the published corpus
+  # IS 7; DESCRIPTION must not claim otherwise.
+  expect_equal(as.integer(d[1, "MaxCorpusSchema"]), 7L)
+
+  # Authors@R must actually parse -- a malformed person() call is only
+  # caught at citation()/build time otherwise.
+  people <- eval(parse(text = d[1, "Authors@R"]))
+  expect_s3_class(people, "person")
+  expect_true("cre" %in% unlist(lapply(people, function(p) p$role)))
+})
+
+test_that("LICENSE and LICENSE.md name the same copyright holder", {
+  skip_if_no_source_tree("LICENSE", "LICENSE.md")
+  holder <- sub("^COPYRIGHT HOLDER:\\s*", "",
+                grep("^COPYRIGHT HOLDER:", readLines(source_tree_path("LICENSE"),
+                                                     warn = FALSE), value = TRUE))
+  full <- paste(readLines(source_tree_path("LICENSE.md"), warn = FALSE), collapse = "\n")
+
+  expect_equal(holder, "Civilytics Consulting LLC")
+  expect_match(full, holder, fixed = TRUE)
+  # usethis::use_mit_license() writes LICENSE.md but leaves an existing
+  # LICENSE alone, which is how the two came to disagree in the first place.
+  expect_match(full, "MIT License", fixed = TRUE)
+})
+
+test_that("vignettes are not excluded from the build", {
+  skip_if_no_source_tree(".Rbuildignore")
+  ignore <- readLines(source_tree_path(".Rbuildignore"), warn = FALSE)
+  expect_false(any(grepl("^\\^vignettes\\$$", ignore)))
+  # The fixture is what lets R CMD check run offline with no credentials on
+  # r-universe and GitHub Actions. It must never be excluded.
+  expect_false(any(grepl("fixture_corpus", ignore, fixed = TRUE)))
+  # doc/ and Meta/ ARE build artefacts of devtools::build_vignettes() and must
+  # stay excluded -- R CMD build regenerates inst/doc/ from vignettes/ on its
+  # own, and leaving them in earns a "non-standard file at top level" NOTE.
+  expect_true(any(grepl("^\\^doc\\$$", ignore)))
+  expect_true(any(grepl("^\\^Meta\\$$", ignore)))
+})
+
+test_that("_pkgdown.yml indexes every exported topic", {
+  skip_if_no_source_tree("_pkgdown.yml", "NAMESPACE")
+  exports <- grep("^export\\(", readLines(source_tree_path("NAMESPACE"), warn = FALSE),
+                  value = TRUE)
+  exports <- sub("^export\\((.*)\\)$", "\\1", exports)
+  yml <- paste(readLines(source_tree_path("_pkgdown.yml"), warn = FALSE), collapse = "\n")
+  missing <- exports[!vapply(exports,
+                             function(e) grepl(paste0("\\b", e, "\\b"), yml),
+                             logical(1))]
+  # pkgdown errors on topics missing from the index, so an unlisted export
+  # means the docs site does not build at all.
+  expect_equal(missing, character(0))
+})
+
+test_that("README is written for a stranger, not a repo insider", {
+  skip_if_no_source_tree("README.md")
+  r <- paste(readLines(source_tree_path("README.md"), warn = FALSE), collapse = "\n")
+
+  # No paths that only resolve inside a maintainer's checkout.
+  expect_false(grepl("../cog_pipeline", r, fixed = TRUE))
+  # A real, uncommented install line.
+  expect_match(r, "install.packages", fixed = TRUE)
+  expect_false(grepl("# pak::pkg_install", r, fixed = TRUE))
+  # The errata most likely to produce a plausible-looking wrong answer.
+  expect_match(r, "full US dollars", fixed = TRUE)
+  # The release advice that conflicts with public CI is gone.
+  expect_false(grepl("Rbuildignore", r, fixed = TRUE))
+  # Both read paths documented.
+  expect_match(r, "cog_mirror", fixed = TRUE)
+  # cog_spending() has no default for `years`; a quickstart that omits it
+  # errors on the reader's first call.
+  expect_match(r, "years\\s*=", perl = TRUE)
+})
@@ -0,0 +1,62 @@
+# Network-gated. Set USCOGDATA_LIVE_TEST=true to run.
+#
+# This file exists because the defect fixed for 0.3.0 -- no remote corpus was
+# readable at all, because DuckDB cannot expand a glob over generic HTTP --
+# survived precisely because every other test path used a LOCAL corpus (the
+# bundled fixture), and so did the API in production (a host mount). Nothing
+# ever exercised the package the way a new user does.
+skip_live <- function() {
+  testthat::skip_if_not(
+    identical(tolower(Sys.getenv("USCOGDATA_LIVE_TEST", "")), "true"),
+    "live-corpus test: set USCOGDATA_LIVE_TEST=true to run"
+  )
+}
+
+# The suite's setup.R pins USCOGDATA_URL to the bundled fixture, so reaching
+# the default requires clearing both the env var and the option.
+with_default_corpus <- function(code) {
+  withr::local_envvar(
+    USCOGDATA_URL = NA, USCOGDATA_FIXTURE_URL = NA,
+    .local_envir = parent.frame()
+  )
+  withr::local_options(uscogdata.url = NULL, .local_envir = parent.frame())
+  cog_close()
+  withr::defer(cog_close(), envir = parent.frame())
+  force(code)
+}
+
+test_that("the package reads the public corpus with no configuration at all", {
+  skip_live()
+  with_default_corpus({
+    g <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+    expect_gt(nrow(g), 0)
+
+    s <- cog_spending(g$canonical_govid[1], years = 2022)
+    expect_gt(nrow(s), 0)
+    expect_true(all(c("amt_nominal", "year", "category") %in% names(s)))
+
+    # Amounts are full dollars, already x1000. A city's annual spending is
+    # millions, not thousands -- this catches a regression that dropped or
+    # doubled the conversion.
+    expect_gt(sum(s$amt_nominal, na.rm = TRUE), 1e6)
+
+    p <- attr(s, "provenance")
+    expect_true(isTRUE(p$transformations$units_conversion$applied))
+    expect_equal(p$transformations$units_conversion$multiplier, 1000)
+  })
+})
+
+test_that("a multi-decade query reads across many partitions", {
+  skip_live()
+  with_default_corpus({
+    g <- cog_gov_search(name = "Madison", state = "WI", type = 2)
+    # `years` is required on cog_spending() -- there is no full-history
+    # default at the reader level (the API's /profile route supplies one).
+    s <- cog_spending(g$canonical_govid[1], years = 2000:2022)
+    # Enumeration builds one read_parquet() path per requested partition. If
+    # the list were truncated, or silently collapsed to a single file, the
+    # returned span is what catches it.
+    expect_gt(diff(range(s$year)), 10)
+    expect_gt(length(unique(s$year)), 5)
+  })
+})
@@ -0,0 +1,64 @@
+test_that(".long_files_sql enumerates every partition the manifest lists", {
+  manifest <- list(files = list(long_partitions = list(
+    list(year = 2011L, path = "data/long/year=2011/part-0.parquet"),
+    list(year = 2012L, path = "data/long/year=2012/part-0.parquet")
+  )))
+  expect_equal(
+    uscogdata:::.long_files_sql("https://example.org/corpus/", manifest),
+    paste0(
+      "['https://example.org/corpus/data/long/year=2011/part-0.parquet',",
+      "'https://example.org/corpus/data/long/year=2012/part-0.parquet']"
+    )
+  )
+})
+
+test_that(".long_files_sql falls back to the glob when no partition list is present", {
+  # test-views.R registers views with a hand-built manifest that has no
+  # `files` element. That must keep working: the glob is valid for the
+  # local paths such a manifest is used with.
+  expect_equal(
+    uscogdata:::.long_files_sql("/tmp/corpus/", list(schema_version = 4L)),
+    "'/tmp/corpus/data/long/**/*.parquet'"
+  )
+  expect_equal(
+    uscogdata:::.long_files_sql("/tmp/corpus/", list(files = list(long_partitions = list()))),
+    "'/tmp/corpus/data/long/**/*.parquet'"
+  )
+})
+
+test_that("the enumerated list matches the bundled fixture's partition count", {
+  skip_if_no_corpus()
+  m <- jsonlite::fromJSON(
+    file.path(fixture_corpus_path(), "manifest.json"), simplifyVector = FALSE
+  )
+  out <- uscogdata:::.long_files_sql(fixture_corpus_path(), m)
+  expect_equal(
+    lengths(regmatches(out, gregexpr("part-0\\.parquet", out))),
+    length(m$files$long_partitions)
+  )
+})
+
+test_that("no view SQL survives rendering with an unsubstituted token", {
+  # Introducing {long_files} broke four test sites that had hand-rolled the
+  # {url} substitution -- each failed with a DuckDB parser error on the
+  # surviving brace. This asserts the whole SQL directory renders clean, so
+  # a future token cannot reintroduce that silently.
+  sql_dir <- system.file("sql", package = "uscogdata")
+  for (f in list.files(sql_dir, pattern = "\\.sql$", full.names = TRUE)) {
+    rendered <- uscogdata:::.render_view_sql(
+      paste(readLines(f, warn = FALSE), collapse = "\n"), "/tmp/corpus/"
+    )
+    expect_false(grepl("\\{[a-z_]+\\}", rendered), label = basename(f))
+  }
+})
+
+test_that("registered `long` view reads through the enumerated list", {
+  skip_if_no_corpus()
+  with_fixture_corpus({
+    con <- uscogdata:::.ensure_session()
+    n <- DBI::dbGetQuery(con, "SELECT count(*) AS n FROM long")$n
+    expect_gt(n, 0)
+    yrs <- DBI::dbGetQuery(con, "SELECT DISTINCT year FROM long ORDER BY year")$year
+    expect_true(all(c(2011, 2012, 2019, 2020) %in% yrs))
+  })
+})
@@ -4,12 +4,14 @@
 # protect users from silent failures when USCOGDATA_URL is misconfigured
 # or returns non-JSON content.

-test_that("cog_open aborts with actionable error when URL is the placeholder default", {
+test_that("cog_open aborts with actionable error when URL contains the sentinel", {
  uscogdata:::cog_close()
  on.exit(uscogdata:::cog_close(), add = TRUE)

-  placeholder <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
-  withr::with_envvar(c(USCOGDATA_URL = placeholder), {
+  # No longer the package default (that is the public HF corpus). This is a
+  # user who copied a config template and did not finish editing it.
+  sentinel_url <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  withr::with_envvar(c(USCOGDATA_URL = sentinel_url), {
    expect_error(
      uscogdata:::cog_open(),
      class = "uscogdata_url_not_configured"
@@ -35,8 +37,10 @@ test_that("placeholder guard error names both env var and option as remediation"
  uscogdata:::cog_close()
  on.exit(uscogdata:::cog_close(), add = TRUE)

-  placeholder <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
-  withr::with_envvar(c(USCOGDATA_URL = placeholder), {
+  # No longer the package default (that is the public HF corpus). This is a
+  # user who copied a config template and did not finish editing it.
+  sentinel_url <- "https://cloud.civilytics.org/s/REPLACE_WITH_SHARE_TOKEN/download/"
+  withr::with_envvar(c(USCOGDATA_URL = sentinel_url), {
    msg <- tryCatch(uscogdata:::cog_open(), error = conditionMessage)
    expect_match(msg, "USCOGDATA_URL", fixed = TRUE)
    expect_match(msg, "uscogdata.url", fixed = TRUE)
@@ -0,0 +1,29 @@
+# Mirror of test-spending-pagination.R for cog_revenue(), which shares the
+# same .verb_spendrev()/.build_verb_sql() pushdown -- see that file for the
+# incident this fixes.
+
+test_that("cog_revenue limit/offset page correctly and report total_rows", {
+  skip_if_no_corpus()
+  full <- cog_revenue("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_revenue("121011212191", years = 2019:2020, category = NULL,
+                      limit = 5L, offset = 3L)
+  expect_equal(nrow(page), 5L)
+  expect_equal(page[c("year", "canonical_govid", "revenue_subtype", "category")],
+              full[4:8, c("year", "canonical_govid", "revenue_subtype", "category")],
+              ignore_attr = TRUE)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("cog_revenue limit unset by default leaves total_rows absent", {
+  skip_if_no_corpus()
+  r <- cog_revenue("121011212191", 2020L, "Property Tax")
+  expect_null(attr(r, "total_rows"))
+})
+
+test_that("cog_revenue complete + limit conflict aborts the same way as cog_spending", {
+  skip_if_no_corpus()
+  expect_error(
+    cog_revenue("121011212191", 2020L, "Property Tax", complete = TRUE, limit = 5L),
+    class = "uscogdata_complete_pagination_conflict"
+  )
+})
@@ -0,0 +1,95 @@
+# cog-api's paginate() used to slice an already-fully-materialized result:
+# every page of a deep sweep re-ran the whole query and re-listified every
+# row, just to keep 1000 and discard the rest. For a 193,105-row fleet-wide
+# query walked 194 pages deep, that repeated the full cost 194 times and
+# wedged the production server for hours (2026-08-06 incident). limit/offset
+# here push the slice into the SQL itself, so a page costs O(limit), not
+# O(full result).
+
+test_that("limit without offset returns the first page, matching the unpaginated head", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                       limit = 10L)
+  expect_equal(nrow(page), 10L)
+  expect_equal(page[c("year", "canonical_govid", "spend_subtype", "category")],
+              full[1:10, c("year", "canonical_govid", "spend_subtype", "category")],
+              ignore_attr = TRUE)
+})
+
+test_that("offset skips ahead without gaps or overlap", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page2 <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                        limit = 10L, offset = 10L)
+  expect_equal(nrow(page2), 10L)
+  expect_equal(page2[c("year", "canonical_govid", "spend_subtype", "category")],
+              full[11:20, c("year", "canonical_govid", "spend_subtype", "category")],
+              ignore_attr = TRUE)
+})
+
+test_that("walking every page reconstructs the unpaginated result exactly", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  n <- nrow(full)
+  limit <- 7L
+  pages <- list()
+  offset <- 0L
+  repeat {
+    p <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                      limit = limit, offset = offset)
+    if (nrow(p) == 0L) break
+    pages[[length(pages) + 1L]] <- p
+    offset <- offset + limit
+    if (offset > n + limit) stop("test runaway: paging did not terminate")
+  }
+  walked <- dplyr::bind_rows(pages)
+  expect_equal(nrow(walked), n)
+  key_cols <- c("year", "canonical_govid", "spend_subtype", "category", "amt_nominal")
+  expect_equal(walked[key_cols], full[key_cols], ignore_attr = TRUE)
+})
+
+test_that("total_rows attribute reports the full unpaginated count", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                       limit = 5L, offset = 0L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("offset past the end returns zero rows, not an error", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2019:2020, category = NULL)
+  page <- cog_spending("121011212191", years = 2019:2020, category = NULL,
+                       limit = 10L, offset = nrow(full) + 100L)
+  expect_equal(nrow(page), 0L)
+  expect_equal(attr(page, "total_rows"), nrow(full))
+})
+
+test_that("limit is unset by default -- unpaginated calls are unaffected", {
+  skip_if_no_corpus()
+  r <- cog_spending("121011212191", 2020L, "Corrections")
+  expect_null(attr(r, "total_rows"))
+})
+
+test_that("per_capita and adjust_to_year still apply correctly within a page", {
+  skip_if_no_corpus()
+  full <- cog_spending("121011212191", years = 2020L, category = NULL,
+                       per_capita = TRUE, adjust_to_year = 2022L)
+  page <- cog_spending("121011212191", years = 2020L, category = NULL,
+                       per_capita = TRUE, adjust_to_year = 2022L,
+                       limit = 3L, offset = 2L)
+  expect_equal(page[c("amt_nominal", "amt_real", "amt_per_capita_nominal",
+                      "amt_per_capita_real")],
+              full[3:5, c("amt_nominal", "amt_real", "amt_per_capita_nominal",
+                          "amt_per_capita_real")],
+              ignore_attr = TRUE)
+})
+
+test_that("complete = TRUE with limit aborts -- pagination over a partial grid is undefined", {
+  skip_if_no_corpus()
+  expect_error(
+    cog_spending("121011212191", 2020L, "Corrections", complete = TRUE, limit = 5L),
+    class = "uscogdata_complete_pagination_conflict"
+  )
+})
@@ -103,7 +103,7 @@ test_that("inst/sql/22- and 23- harmonized views enforce every WHERE predicate (
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
  }

  con <- DBI::dbConnect(duckdb::duckdb())
@@ -183,7 +183,7 @@ test_that("inst/sql/24- and 25- IG views retain aggregates, COALESCE NULL harmon
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, paste0(tmp, "/"))
  }

  con <- DBI::dbConnect(duckdb::duckdb())
@@ -335,7 +335,7 @@ test_that(".harmonization_view_files guard is necessary: registration against a
  sql_dir <- system.file("sql", package = "uscogdata")
  .read_view_sql <- function(filename) {
    txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
-    gsub("\\{url\\}", url, txt, fixed = FALSE)
+    uscogdata:::.render_view_sql(txt, url)
  }
  con2 <- DBI::dbConnect(duckdb::duckdb())
  on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
Author	SHA1	Message	Date
jared	6392a74013	ci: multi-platform R CMD check for the GitHub mirror R-CMD-check / check (push) Successful in 3m27s Details R-CMD-check / check (pull_request) Successful in 3m19s Details The canonical Gitea runner is Linux-only, and this package hard-depends on duckdb and httr2 -- both compiled, both with real platform variance -- while having never been checked on Windows or macOS. A large share of the audience is on Windows. Inert here: Gitea reads .gitea/workflows, GitHub reads .github/workflows, so both configs coexist and the Gitea CI stays authoritative for deploys. This goes live the moment the mirror repo exists (#44). The matrix needs no credentials -- setup.R points at the bundled fixture -- which is why that fixture must never be .Rbuildignore'd.	2026-08-08 19:49:52 -04:00
jared	de2ba0cfb9	Merge pull request 'feat: uscogdata 0.3.0 — first public release' (#41 ) from feat/public-release-0.3.0 into main R-CMD-check / check (push) Successful in 3m27s Details	2026-08-08 18:56:14 -04:00
jared	e912a926c2	Merge pull request 'chore: regenerate fixture against pipeline e7394a4 (SB203-SB209)' (#40 ) from chore/fixture-sb203 into main R-CMD-check / check (push) Successful in 3m29s Details	2026-08-08 18:55:59 -04:00
jared	44e4953f94	fix: keep doc/ and Meta/ out of the build R-CMD-check / check (push) Successful in 3m58s Details R-CMD-check / check (pull_request) Successful in 3m52s Details Removing ^vignettes$ was right; removing ^doc$ and ^Meta$ with it was not. Those are devtools::build_vignettes() artefacts, not sources -- R CMD build regenerates inst/doc/ from vignettes/ by itself, and shipping the local copies earned a 'non-standard file/directory found at top level' NOTE. R CMD check --as-cran is now 0 errors, 0 warnings, 0 notes.	2026-08-08 18:25:00 -04:00
jared	a5500f0b6a	docs: add CONTRIBUTING with the canonical-on-Gitea PR flow Moves developer, testing and release instructions out of the README, minus the fixture-stripping advice, which was wrong. Explains that a GitHub PR closes itself as merged once the mirror syncs, because the merge preserves the contributor's SHAs -- so a PR closing without a visible Merge click reads as success rather than rejection. Documents the cog-api dependency: its CI clones this package at USCOGDATA_REF, defaulting to main with no pin, so anything merged here reaches the API's next build. Includes the commands to run its suite against a branch first.	2026-08-08 18:20:42 -04:00
jared	da2839f885	docs: recast NEWS around the first public release NEWS described changes relative to states no user had ever seen -- 'Breaking: corpus schema_version 4', 'the package now requires...' -- across the whole pre-release development. To someone deciding whether to depend on this, that reads as instability. 0.3.0 is written as an announcement: what it covers, the verbs, that reading the corpus now works out of the box, four things to know before a first query, and the known limits. The 0.2.0 changelog is kept verbatim. The 0.1.0 development log is dropped; that history is in git. cog_explain() now documents what provenance actually holds, since the README points readers at it -- in particular why series_break_refs and corpus_break_refs are separate fields rather than one list.	2026-08-08 18:20:33 -04:00
jared	331399ab86	docs: rewrite README for a stranger Reordered around a new user: what the data is, where it comes from, install, a quickstart that runs with no configuration, then the full-dollars warning and the concepts that decide whether a published number is right. Adds a 'Where the data comes from' section linking the API documentation site, the live API, the Hugging Face corpus and the Census source, so attribution and provenance are reachable from the top rather than implied. Drops the sibling-repo path, the commented-out install line, the Status block, and the release advice telling you to strip the fixture -- which would break the vignette and leave public CI unable to check without credentials. The quickstart passes years=; cog_spending() has no full-history default, so the obvious one-liner errors on a reader's first call.	2026-08-08 18:20:24 -04:00
jared	5582c6cb57	docs: index all 14 exports in pkgdown, set the site url The reference index covered 6 of 14 exports, so pkgdown errored on the eight missing topics and the docs site did not build at all. Adds a Comparison & aggregation section and a Corpus metadata section, lists both vignettes as articles, and sets url so canonical links and search resolve. build_site() now completes clean: reference metadata ok, no problems.	2026-08-08 17:49:10 -04:00
jared	013af5b0d2	fix: ship the vignettes .Rbuildignore excluded ^vignettes$, ^doc$ and ^Meta$, so an installed uscogdata had no vignettes at all -- while the README instructed users to run vignette("total-spending"), which failed for every one of them. Both build offline: total-spending points USCOGDATA_URL at the bundled fixture, population-denominators is eval = FALSE. Confirmed present in the built tarball as both source and rendered inst/doc/. The test also pins the fixture as never-excluded -- it is what lets R CMD check run with no credentials on r-universe and GitHub Actions.	2026-08-08 17:48:07 -04:00
jared	4a92f36d44	chore: add full MIT text, name the copyright holder properly LICENSE held only the two-line stub and no LICENSE.md existed, so the repo carried no license text for a human browsing it or for GitHub's license detector. usethis::use_mit_license() writes LICENSE.md but leaves an existing LICENSE alone, so the stub kept saying 'Civilytics' while the full text said 'Civilytics Consulting LLC'. Corrected by hand, with a test pinning the two together.	2026-08-08 17:47:18 -04:00
jared	a67735f121	chore: release metadata -- author of record, URLs, schema ceiling, 0.3.0 Authors@R was an org with no human, so citation() and the r-universe maintainer page had nothing to render and ORCID could not collate this with merTools. The given-name vector c("Jared", "E.") matches merTools exactly; person("Jared", "E. Knowles") would render the same but put the middle initial in the family-name slot. MaxCorpusSchema claimed 5 while .validate_schema() accepts 4-7 and the published corpus is 7 -- metadata contradicting code by two versions. 0.3.0 rather than 0.2.0: remote reads go from broken to working and the default URL from placeholder to live, which is user-visible behaviour.	2026-08-08 17:46:39 -04:00
jared	b9f7f7d8d3	test: exercise the public corpus end to end, unconfigured The remote-read defect survived because every test path used a local corpus, and so did the API in production. This is the only test that runs the package the way a new user does: no USCOGDATA_URL, no option, no fixture -- just install and call a verb. Gated on USCOGDATA_LIVE_TEST so offline CI skips rather than fails. Measured against the live corpus while writing this: cog_spending for one government is 3.9s for a single year and 5.9s across 2000-2022. Well above the 1.5-2.8s raw parquet scan, because the verbs also join crosswalks, resolve categories and assemble provenance.	2026-08-08 17:21:53 -04:00
jared	4300b636b1	feat: default to the public corpus so the package works unconfigured The default was a REPLACE_WITH_SHARE_TOKEN sentinel and no document in the package supplied a working URL, so a new user installing uscogdata had no path to a session at all -- just an actionable-looking error with nothing actionable behind it. The default is now the public HuggingFace mirror: CC-BY-4.0, no credential, CDN-backed, and it keeps the origin's uplink out of the read path. USCOGDATA_URL and options(uscogdata.url=) still override, so Nextcloud and cog_mirror() copies are unaffected. The sentinel guard stays for half-edited configs; the two tests covering it set the URL explicitly, so they only needed renaming to stop calling it 'the default'.	2026-08-08 17:18:56 -04:00
jared	99e1e86e37	fix: enumerate long partitions from the manifest instead of globbing DuckDB cannot expand a glob over generic HTTP -- there is no directory listing, and allow_asterisks_in_http_paths only forwards the literal '*/' as a filename, which 404s. So every remote corpus read failed. Only local paths worked, which is how the API (a host mount) and the test fixture run, so nothing ever caught it. Measured against the published corpus: the explicit list returns the same 46,148,034 rows the hf:// glob does, with hive_partitioning still recovering year from the paths. Building it from the manifest keeps the reader host-agnostic rather than binding it to one vendor's protocol. Also extracts .render_view_sql(). Four test sites had hand-rolled the {url} substitution -- one commented as doing it 'exactly as .register_views() does' -- and all four broke on the second token. They now share the one function that knows the vocabulary, and a new test renders every SQL file to prove no token survives.	2026-08-08 17:11:44 -04:00
jared	c042ee0b90	docs: retarget the release at 0.3.0 and preserve the 0.2.0 changelog Spec and plan were written against a branch 25 commits behind main, where the package still read 0.1.0. It is 0.2.0, with a real 0.2.0 changelog in NEWS that the plan would have deleted. 0.3.0 rather than 0.2.0 because remote corpus reads go from broken to working and the default URL from placeholder to live -- user-visible behaviour, so a minor bump. Not 1.0.0: types 4 and 5 remain out of scope. Task 9 now prepends a 0.3.0 section, keeps 0.2.0 verbatim with a diff check to prove it, and drops only the 0.1.0 development churn. Task 4 gains the Version bump.	2026-08-08 17:07:19 -04:00
jared	29dc8199e0	docs: implementation plan for the uscogdata 0.1.0 public release Ten tasks, 58 steps, TDD throughout. Tasks 1-3 fix the P0 (manifest enumeration, working default URL, and the live-corpus test whose absence let the defect survive); 4-7 are metadata and packaging; 8-10 rewrite README, NEWS and CONTRIBUTING. Distribution mechanics stay out of scope -- r-universe publishes check results on registration, so it comes after final verification is green.	2026-08-08 17:03:40 -04:00
jared	619b167ab1	docs: design spec for the uscogdata 0.1.0 public release Covers the P0 finding that the package cannot read the corpus remotely at all -- no working default URL, and Hive globs are unsupported over generic HTTP by DuckDB 1.5.5. Fix is manifest-driven file enumeration (measured: 46,148,034 rows over plain https, identical to the hf:// glob) plus a working public default. Also: seven release-readiness fixes, a README restructured for a stranger, NEWS rewritten as an initial release rather than a pre-release churn log, and the Gitea-canonical/GitHub-mirror/r-universe distribution mechanics.	2026-08-08 17:03:40 -04:00
jared	dfda39051e	docs: point agents at the Civilytics values file before they start Mirrors the same block in cog_explorer/CLAUDE.md. A project-level @ import does not preload, so the instruction to read it is the mechanism.	2026-08-08 17:03:39 -04:00
jared	785f3af16d	chore: regenerate fixture against pipeline e7394a4 (SB203-SB209) R-CMD-check / check (pull_request) Successful in 3m50s Details R-CMD-check / check (push) Successful in 4m9s Details Picks up seven new catalogued series breaks, all break_year 2017, recording that the employee-retirement X-codes (X21, X30, X42, X44, X47, Z77, Z78) were last collected in the annual finance file at FY2016 before those systems moved to the Annual Survey of Public Pensions. series_breaks.parquet 202 -> 209 rows; nothing removed. manifest built_at and pipeline_commit updated, and the series_breaks sha256 with them.	2026-08-08 17:02:14 -04:00
jared	c587c8ba87	Merge pull request 'fix: push limit/offset into the query instead of materializing then slicing' (#39 ) from fix/pushdown-pagination into main R-CMD-check / check (push) Successful in 3m35s Details Reviewed-on: #39	2026-08-06 14:19:42 -04:00
jared	6a06302036	fix: push limit/offset into the query instead of materializing then slicing R-CMD-check / check (push) Successful in 4m6s Details R-CMD-check / check (pull_request) Successful in 3m33s Details cog-api's paginate() sliced an already-fully-materialized result: every page of a deep sweep re-ran the whole cog_spending()/cog_revenue() query and re-listified every row, just to keep up to 1000 and discard the rest. A 193,105-row/194-page fleet-wide sweep (cog_explorer's Southern guide, corpus summary build) repeated that full cost 194 times and wedged the production server for hours on 2026-08-06 -- single request, CPU-bound, single-threaded plumber process, no other request could get through, not even /health. cog_spending()/cog_revenue() gain optional limit/offset, pushed into .build_verb_sql() as SQL LIMIT/OFFSET behind the existing (already deterministic) ORDER BY. The full unpaginated row count rides along via COUNT(*) OVER() in the same scan -- exposed as a total_rows attribute -- so a caller walking pages never needs a second round trip to ask how many there are. A page now costs O(limit), not O(full result). Mutually exclusive with complete = TRUE (which fills a grid over the FULL requested (year, category) space -- pagination over a partial slice of already-grouped rows has no defined meaning for the cells it would fill) and with recipe (whose result comes from a separate, not-yet-wired query path). Both abort with a clear classed condition rather than silently ignoring the parameter. limit/offset default to NULL; every existing call site is unaffected.	2026-08-06 13:56:27 -04:00
jared	e3ab26c3e6	Merge pull request 'feat: 'All Categories' pseudo-category + n_units_reporting semantics' (#37 ) from feat/all-categories-37 into main R-CMD-check / check (push) Successful in 3m50s Details Reviewed-on: #37	2026-08-05 12:50:46 -04:00