Compare commits

...
Author SHA1 Message Date
jared 5f81ae386b chore: CI badge, mirror PR explainer, and the tag-pin release step (#47)
R-CMD-check / check (pull_request) Successful in 3m49s
R-CMD-check / check (push) Successful in 3m52s
Three items #47 absorbed from #46, held back so a dead badge would not sit
beside an unresolved r-universe one. Both resolve now that v0.4.0 is tagged.

- R-CMD-check badge pointing at the GitHub mirror's workflow, where the
  4-platform matrix actually runs.

- A pull_request_target workflow explaining the mirror flow on every incoming
  PR. A PR here is landed on Gitea and syncs back, and because the merge
  preserves the contributor's commits at their original SHAs, GitHub marks the
  PR 'Merged' with nobody visibly clicking Merge. To a first-time contributor
  that reads as rejection. Say so before it happens.

  pull_request_target rather than pull_request because a fork PR's token is
  read-only under the latter -- it could not comment, which is the entire job.
  That is only safe because this never checks out or runs contributor code; the
  file says so and says not to add a checkout.

- CONTRIBUTING's release checklist now spells out that the tag goes on Gitea and
  the mirror carries it, and that r-universe does NOT pick up a release until
  packages.json's branch pin is edited. '*release' would automate it but needs a
  GitHub Release object, and the mirror pushes tags only -- so it would silently
  never update. Learned while doing this release.
2026-08-10 19:47:26 -04:00
jared d2caa6de97 Merge pull request 'docs: re-measure the corpus-access table against the published corpus (#56)' (#65) from docs/readme-perf-remeasure-56 into main
Mirror to GitHub / mirror (push) Successful in 9s
R-CMD-check / check (push) Successful in 3m25s
2026-08-10 19:36:42 -04:00
jared 303aa59b07 docs: order the 0.4.0 NEWS sections by user impact, not merge order
R-CMD-check / check (push) Successful in 3m58s
R-CMD-check / check (pull_request) Successful in 3m38s
The three 0.4.0 features landed in the order their PRs merged, which buried the
headline change (cohort predicates, 4.8x) below an operator config knob and a
documentation note. Reordered to: cohorts, pagination, DuckDB budget, docs,
Fixes -- with Fixes last, where it was already.

The stacked PRs each appended their own '## Fixes' heading, so resolving the
conflicts also folded two of them into the single section that belongs there.

Content is byte-identical to what merged; only section order changed. Verified
by diffing the sorted non-blank lines against the previous commit.
2026-08-10 19:29:43 -04:00
jared 81f72321ee docs: re-measure the corpus-access table against the published corpus (#56)
figures predate the row-group rechunk (cog_pipeline#93, published 2026-08-09)
and reported the mirrored column as 'local speed' with no number -- hiding the
largest difference available to a user.

Measured 2026-08-10, fresh R session per arm, against the live corpus at
pipeline_commit 3d28ddd. Madison WI, 16-core Linux workstation.

Three findings the old table could not express:

- A local mirror is 60-80x faster. A one-off question is ~12 s end to end
  remotely against ~0.15 s mirrored. Stated outright now, because it is a
  bigger and cheaper win for users than anything in the R code.

- Opening the session is the LARGEST remote cost (~7.5 s), bigger than any
  individual query, and it lands on the user's first query rather than on
  library(). The old table accounted for it nowhere, so every per-query figure
  was quietly missing it.

- The remote cost is round-trips, not scanning: a repeat query over
  already-touched partitions is ~1.5 s against ~4 s cold, and a full-history
  query costs ~7 s whether it runs first or last (verified by running the arms
  in both orders). This is why #93's 1.4-1.7x, measured through cog-api against
  a local mount, does not show up on the remote path -- there, network latency
  swamps scan time.

Corpus size corrected to ~201 MB: row-group chunking added ~3.4%, and 190.6 was
ambiguous between MB and MiB besides. Measured from the manifest and on disk.
The 0.3.0 NEWS section keeps 190.6 -- it was correct for that release.

Also documented HTTP 429: a burst of remote queries gets rate-limited by the
host. Hit while taking these measurements.
2026-08-10 19:29:05 -04:00
jared 5cb83d8f1d Merge pull request 'feat: limit/offset on cog_gov_search() and cog_balances() (#57)' (#63) from feat/pagination-search-balances-57 into main
Mirror to GitHub / mirror (push) Successful in 12s
R-CMD-check / check (push) Successful in 4m26s
2026-08-10 19:28:09 -04:00
4 changed files with 169 additions and 48 deletions
+64
View File
@@ -0,0 +1,64 @@
# Explain the mirror contribution flow on every incoming pull request.
#
# This repository is a MIRROR. A PR opened here is landed on the canonical Gitea
# repository and syncs back; because the merge preserves the contributor's
# commits at their original SHAs, GitHub marks the PR "Merged" on its own as
# soon as the mirror syncs -- with nobody visibly clicking Merge.
#
# Without this comment, that reads as a rejection: the contributor sees their PR
# close with no review, no merge button pressed, and no explanation. It is
# actually the successful outcome. Say so up front, before it happens.
#
# WHY pull_request_target AND NOT pull_request:
# a `pull_request` run from a fork gets a read-only token, so it cannot post a
# comment -- which is exactly the case this workflow exists to serve.
# `pull_request_target` runs in the context of the BASE repo and gets a writable
# token. That is only safe because this job never checks out or executes the
# contributor's code; it posts a fixed string. Do not add a checkout of
# `github.event.pull_request.head.sha` here -- that combination is the standard
# pull_request_target privilege-escalation hole.
name: Explain the mirror flow
on:
pull_request_target:
types: [opened]
permissions:
pull-requests: write
jobs:
comment:
runs-on: ubuntu-latest
steps:
- name: Post the contribution-flow explainer
uses: actions/github-script@v7
with:
script: |
const body = [
"Thanks for this — and one thing worth knowing before it happens.",
"",
"**This repository is a mirror.** Development happens on Gitea at",
"`gitea.civilytics.org/Civilytics/uscogdata`. Your pull request will be fetched",
"from here, landed there, and synced back.",
"",
"Because that merge preserves your commits at their original SHAs, **GitHub will",
"mark this pull request \"Merged\" on its own** — without anyone visibly clicking",
"the Merge button, and possibly without a review comment on this page first.",
"",
"> If your pull request closes as \"Merged\" and nobody appears to have merged it,",
"> that is the normal, successful outcome — not a rejection.",
"",
"If it is *not* going to be merged, you will get an actual reply saying so.",
"",
"Substantial contributions get a `ctb` entry in `DESCRIPTION`, which surfaces in",
"`citation(\"uscogdata\")`. There is no CLA and no DCO sign-off.",
"",
"Full details: [CONTRIBUTING.md](https://github.com/civilytics/uscogdata/blob/main/CONTRIBUTING.md).",
].join("\n");
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number,
body,
});
+10 -2
View File
@@ -101,5 +101,13 @@ but "usually" is not a release gate.
variables set. This is the only check that catches a variables set. This is the only check that catches a
corpus-unreachable defect, and its absence is why 0.3.0 needed fixing. corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
8. Bump `Version` and add a `NEWS.md` section. 8. Bump `Version` and add a `NEWS.md` section.
9. Tag, then update the r-universe registry pin at 9. Tag on **Gitea** (`git tag -a vX.Y.Z && git push origin vX.Y.Z`). The mirror
`github.com/civilytics/civilytics.r-universe.dev`. workflow carries tags to GitHub on its own — confirm the tag appears at
`github.com/civilytics/uscogdata/tags` before continuing.
10. Update the r-universe registry pin at
`github.com/civilytics/civilytics.r-universe.dev` — edit `packages.json`'s
`branch` to the new tag. **r-universe will not pick up a release until this
is edited**: the pin is a tag, deliberately, so a mid-refactor `main` is
never published as a release. `"branch": "*release"` would track releases
automatically, but it needs a GitHub *Release* object and the mirror pushes
tags only — so it would silently never update.
+65 -41
View File
@@ -1,46 +1,5 @@
# uscogdata 0.4.0 # uscogdata 0.4.0
## DuckDB's resource budget is configurable
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
spellings) cap the DuckDB connection the package opens. Both follow the same
env-var > option > default precedence as `USCOGDATA_URL`.
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
before -- every visible core. That is right for one interactive session on a
dedicated machine and wrong for a server: where several readers share a host, each
otherwise claims the whole machine and they contend. Capping measured ~5% on a
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
cores), which is cheap enough that a server should always cap.
This replaces a workaround in which a consumer reached into the package namespace
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
`SET threads` -- depending both on a private name and on the session already being
open.
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
verbs were left materializing everything and slicing in R. Both now take
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
than a second query.
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
returns the entire 40,336-row crosswalk when called with no filter.
Two refusals rather than silent surprises:
* `cog_balances(recipe = , limit = )` aborts with class
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
separate query that pagination is not wired into.
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
per requested name with a sidecar covering all of them; a page of that is not
a page of anything the caller asked for.
## Cohorts can be named by predicate, not just by id ## Cohorts can be named by predicate, not just by id
`cog_spending()`, `cog_revenue()` and `cog_balances()` gain optional `state` `cog_spending()`, `cog_revenue()` and `cog_balances()` gain optional `state`
@@ -87,6 +46,71 @@ When the cohort is named by predicate there is no id list to report, so
`provenance$scope$cohort` carries `state`, `type` and `n_governments` instead. `provenance$scope$cohort` carries `state`, `type` and `n_governments` instead.
A `govid`-named cohort's provenance is unchanged. A `govid`-named cohort's provenance is unchanged.
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
verbs were left materializing everything and slicing in R. Both now take
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
than a second query.
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
returns the entire 40,336-row crosswalk when called with no filter.
Two refusals rather than silent surprises:
* `cog_balances(recipe = , limit = )` aborts with class
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
separate query that pagination is not wired into.
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
per requested name with a sidecar covering all of them; a page of that is not
a page of anything the caller asked for.
## DuckDB's resource budget is configurable
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
spellings) cap the DuckDB connection the package opens. Both follow the same
env-var > option > default precedence as `USCOGDATA_URL`.
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
before -- every visible core. That is right for one interactive session on a
dedicated machine and wrong for a server: where several readers share a host, each
otherwise claims the whole machine and they contend. Capping measured ~5% on a
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
cores), which is cheap enough that a server should always cap.
This replaces a workaround in which a consumer reached into the package namespace
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
`SET threads` -- depending both on a private name and on the session already being
open.
## Documentation: the corpus-access table is re-measured and honest
The README's "two ways to read the corpus" table carried figures taken before
the corpus was re-chunked into row groups (cog_pipeline#93, published
2026-08-09) and reported the mirrored column as "local speed" with no number at
all. Re-measured 2026-08-10 against the published corpus (`pipeline_commit
3d28ddd`), fresh R session per arm:
* **A local mirror is roughly 60-80x faster.** A one-off question costs ~12 s
end to end remotely against ~0.15 s mirrored. That is the largest single
difference available to a user and it is now stated outright rather than left
as "local speed".
* **Opening the session is the largest remote cost** (~7.5 s -- manifest fetch
plus 23 view registrations over HTTPS), larger than any individual query, and
it lands on the first query rather than on `library(uscogdata)`. The old table
did not account for it anywhere.
* **The remote cost is round-trips, not scanning.** A repeat query over
already-touched partitions is ~1.5 s against ~4 s cold, and a full-history
query costs ~7 s whether it runs first or last.
* The corpus size is **~201 MB**, not 190.6 MB -- row-group chunking added ~3.4%
and the old figure was ambiguous between MB and MiB besides.
* Documented that a burst of remote queries can be rate-limited by the host
(`HTTP 429`), which is another reason to mirror for real work.
## Fixes ## Fixes
* `cog_gov_search()` now orders by `population_acs DESC NULLS LAST, * `cog_gov_search()` now orders by `population_acs DESC NULLS LAST,
+30 -5
View File
@@ -1,6 +1,7 @@
# uscogdata # uscogdata
<!-- badges: start --> <!-- badges: start -->
[![R-CMD-check](https://github.com/civilytics/uscogdata/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/civilytics/uscogdata/actions/workflows/R-CMD-check.yaml)
[![r-universe](https://civilytics.r-universe.dev/badges/uscogdata)](https://civilytics.r-universe.dev/uscogdata) [![r-universe](https://civilytics.r-universe.dev/badges/uscogdata)](https://civilytics.r-universe.dev/uscogdata)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md)
<!-- badges: end --> <!-- badges: end -->
@@ -18,7 +19,7 @@ carries provenance describing what was converted, what was aggregated, and
which known series breaks intersect your query. which known series breaks intersect your query.
**Scope:** government types 0–3 (state, county, municipality, township). **Scope:** government types 0–3 (state, county, municipality, township).
56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968 56 fiscal years, 46,148,034 rows, ~201 MB. There is no source data for FY1968
or FY1969. Special districts (type 4) and school districts (type 5) are or FY1969. Special districts (type 4) and school districts (type 5) are
excluded pending validation. excluded pending validation.
@@ -85,14 +86,38 @@ cog_explain(spend)
| | Remote (default) | Mirrored | | | Remote (default) | Mirrored |
|---|---|---| |---|---|---|
| Setup | none | `cog_mirror(dest)`, 190.6 MB once | | Setup | none | `cog_mirror(dest)`, ~201 MB once |
| Disk used | **0 MB** — HTTP range requests only | 190.6 MB | | Disk used | **0 MB** — HTTP range requests only | ~201 MB |
| Per query | ~4 s (one government, one year)<br>~6 s (one government, 23 years) | local speed | | Opening a session | ~7.5 s | ~0.1 s |
| One government, one year | ~4 s | ~0.05 s |
| One government, full history | ~7 s | ~0.1 s |
| Later queries, same session | ~1.5 s | ~0.05 s |
| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility | | Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
**A local mirror is roughly 60–80x faster, and it is one function call.** That is
by far the largest difference any of these settings makes. If you are going to
ask more than a handful of questions, mirror first.
Measured 2026-08-10 on a 16-core Linux workstation against the published corpus
(schema v7, `pipeline_commit 3d28ddd`), fresh R session per arm. A one-off
question costs about **12 seconds end to end remotely and 0.15 seconds
mirrored**, session setup included.
Two things the per-query rows hide:
- **Opening the session is the single largest remote cost** — larger than any
one query. It fetches the manifest and registers 23 SQL views over HTTPS, and
it lands on your first query, not on `library(uscogdata)`.
- **The cost is network round-trips, not scanning.** A repeat query against
partitions this session has already touched is ~1.5 s rather than ~4 s, and a
full-history query costs ~7 s whether it runs first or last. What you are
paying for is reaching each of the 56 yearly files over HTTPS the first time.
Nothing is written to disk in remote mode: DuckDB fetches the parquet footer, Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
works out which row groups it needs, and reads only those. Nothing is cached works out which row groups it needs, and reads only those. Nothing is cached
between sessions either, so every query goes back to the network. between sessions either, so every query goes back to the network — and a session
that issues many remote queries in quick succession can be rate-limited by the
host (`HTTP Error: ... 429`). Both are further reasons to mirror for real work.
The default points at a public HuggingFace mirror of the corpus. If you would The default points at a public HuggingFace mirror of the corpus. If you would
rather not depend on a third party — for reproducibility, for an air-gapped rather not depend on a third party — for reproducibility, for an air-gapped