Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5f81ae386b
|
||
|
|
d2caa6de97 | ||
|
|
303aa59b07
|
||
|
|
81f72321ee
|
||
|
|
5cb83d8f1d |
@@ -0,0 +1,64 @@
|
||||
# Explain the mirror contribution flow on every incoming pull request.
|
||||
#
|
||||
# This repository is a MIRROR. A PR opened here is landed on the canonical Gitea
|
||||
# repository and syncs back; because the merge preserves the contributor's
|
||||
# commits at their original SHAs, GitHub marks the PR "Merged" on its own as
|
||||
# soon as the mirror syncs -- with nobody visibly clicking Merge.
|
||||
#
|
||||
# Without this comment, that reads as a rejection: the contributor sees their PR
|
||||
# close with no review, no merge button pressed, and no explanation. It is
|
||||
# actually the successful outcome. Say so up front, before it happens.
|
||||
#
|
||||
# WHY pull_request_target AND NOT pull_request:
|
||||
# a `pull_request` run from a fork gets a read-only token, so it cannot post a
|
||||
# comment -- which is exactly the case this workflow exists to serve.
|
||||
# `pull_request_target` runs in the context of the BASE repo and gets a writable
|
||||
# token. That is only safe because this job never checks out or executes the
|
||||
# contributor's code; it posts a fixed string. Do not add a checkout of
|
||||
# `github.event.pull_request.head.sha` here -- that combination is the standard
|
||||
# pull_request_target privilege-escalation hole.
|
||||
name: Explain the mirror flow
|
||||
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [opened]
|
||||
|
||||
permissions:
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
comment:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Post the contribution-flow explainer
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const body = [
|
||||
"Thanks for this — and one thing worth knowing before it happens.",
|
||||
"",
|
||||
"**This repository is a mirror.** Development happens on Gitea at",
|
||||
"`gitea.civilytics.org/Civilytics/uscogdata`. Your pull request will be fetched",
|
||||
"from here, landed there, and synced back.",
|
||||
"",
|
||||
"Because that merge preserves your commits at their original SHAs, **GitHub will",
|
||||
"mark this pull request \"Merged\" on its own** — without anyone visibly clicking",
|
||||
"the Merge button, and possibly without a review comment on this page first.",
|
||||
"",
|
||||
"> If your pull request closes as \"Merged\" and nobody appears to have merged it,",
|
||||
"> that is the normal, successful outcome — not a rejection.",
|
||||
"",
|
||||
"If it is *not* going to be merged, you will get an actual reply saying so.",
|
||||
"",
|
||||
"Substantial contributions get a `ctb` entry in `DESCRIPTION`, which surfaces in",
|
||||
"`citation(\"uscogdata\")`. There is no CLA and no DCO sign-off.",
|
||||
"",
|
||||
"Full details: [CONTRIBUTING.md](https://github.com/civilytics/uscogdata/blob/main/CONTRIBUTING.md).",
|
||||
].join("\n");
|
||||
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.payload.pull_request.number,
|
||||
body,
|
||||
});
|
||||
+10
-2
@@ -101,5 +101,13 @@ but "usually" is not a release gate.
|
||||
variables set. This is the only check that catches a
|
||||
corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
|
||||
8. Bump `Version` and add a `NEWS.md` section.
|
||||
9. Tag, then update the r-universe registry pin at
|
||||
`github.com/civilytics/civilytics.r-universe.dev`.
|
||||
9. Tag on **Gitea** (`git tag -a vX.Y.Z && git push origin vX.Y.Z`). The mirror
|
||||
workflow carries tags to GitHub on its own — confirm the tag appears at
|
||||
`github.com/civilytics/uscogdata/tags` before continuing.
|
||||
10. Update the r-universe registry pin at
|
||||
`github.com/civilytics/civilytics.r-universe.dev` — edit `packages.json`'s
|
||||
`branch` to the new tag. **r-universe will not pick up a release until this
|
||||
is edited**: the pin is a tag, deliberately, so a mid-refactor `main` is
|
||||
never published as a release. `"branch": "*release"` would track releases
|
||||
automatically, but it needs a GitHub *Release* object and the mirror pushes
|
||||
tags only — so it would silently never update.
|
||||
|
||||
@@ -1,46 +1,5 @@
|
||||
# uscogdata 0.4.0
|
||||
|
||||
## DuckDB's resource budget is configurable
|
||||
|
||||
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
|
||||
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
|
||||
spellings) cap the DuckDB connection the package opens. Both follow the same
|
||||
env-var > option > default precedence as `USCOGDATA_URL`.
|
||||
|
||||
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
|
||||
before -- every visible core. That is right for one interactive session on a
|
||||
dedicated machine and wrong for a server: where several readers share a host, each
|
||||
otherwise claims the whole machine and they contend. Capping measured ~5% on a
|
||||
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
|
||||
cores), which is cheap enough that a server should always cap.
|
||||
|
||||
This replaces a workaround in which a consumer reached into the package namespace
|
||||
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
|
||||
`SET threads` -- depending both on a private name and on the session already being
|
||||
open.
|
||||
|
||||
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
|
||||
|
||||
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
|
||||
verbs were left materializing everything and slicing in R. Both now take
|
||||
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
|
||||
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
|
||||
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
|
||||
than a second query.
|
||||
|
||||
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
|
||||
returns the entire 40,336-row crosswalk when called with no filter.
|
||||
|
||||
Two refusals rather than silent surprises:
|
||||
|
||||
* `cog_balances(recipe = , limit = )` aborts with class
|
||||
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
|
||||
separate query that pagination is not wired into.
|
||||
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
|
||||
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
|
||||
per requested name with a sidecar covering all of them; a page of that is not
|
||||
a page of anything the caller asked for.
|
||||
|
||||
## Cohorts can be named by predicate, not just by id
|
||||
|
||||
`cog_spending()`, `cog_revenue()` and `cog_balances()` gain optional `state`
|
||||
@@ -87,6 +46,71 @@ When the cohort is named by predicate there is no id list to report, so
|
||||
`provenance$scope$cohort` carries `state`, `type` and `n_governments` instead.
|
||||
A `govid`-named cohort's provenance is unchanged.
|
||||
|
||||
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
|
||||
|
||||
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
|
||||
verbs were left materializing everything and slicing in R. Both now take
|
||||
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
|
||||
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
|
||||
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
|
||||
than a second query.
|
||||
|
||||
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
|
||||
returns the entire 40,336-row crosswalk when called with no filter.
|
||||
|
||||
Two refusals rather than silent surprises:
|
||||
|
||||
* `cog_balances(recipe = , limit = )` aborts with class
|
||||
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
|
||||
separate query that pagination is not wired into.
|
||||
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
|
||||
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
|
||||
per requested name with a sidecar covering all of them; a page of that is not
|
||||
a page of anything the caller asked for.
|
||||
|
||||
## DuckDB's resource budget is configurable
|
||||
|
||||
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
|
||||
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
|
||||
spellings) cap the DuckDB connection the package opens. Both follow the same
|
||||
env-var > option > default precedence as `USCOGDATA_URL`.
|
||||
|
||||
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
|
||||
before -- every visible core. That is right for one interactive session on a
|
||||
dedicated machine and wrong for a server: where several readers share a host, each
|
||||
otherwise claims the whole machine and they contend. Capping measured ~5% on a
|
||||
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
|
||||
cores), which is cheap enough that a server should always cap.
|
||||
|
||||
This replaces a workaround in which a consumer reached into the package namespace
|
||||
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
|
||||
`SET threads` -- depending both on a private name and on the session already being
|
||||
open.
|
||||
|
||||
## Documentation: the corpus-access table is re-measured and honest
|
||||
|
||||
The README's "two ways to read the corpus" table carried figures taken before
|
||||
the corpus was re-chunked into row groups (cog_pipeline#93, published
|
||||
2026-08-09) and reported the mirrored column as "local speed" with no number at
|
||||
all. Re-measured 2026-08-10 against the published corpus (`pipeline_commit
|
||||
3d28ddd`), fresh R session per arm:
|
||||
|
||||
* **A local mirror is roughly 60-80x faster.** A one-off question costs ~12 s
|
||||
end to end remotely against ~0.15 s mirrored. That is the largest single
|
||||
difference available to a user and it is now stated outright rather than left
|
||||
as "local speed".
|
||||
* **Opening the session is the largest remote cost** (~7.5 s -- manifest fetch
|
||||
plus 23 view registrations over HTTPS), larger than any individual query, and
|
||||
it lands on the first query rather than on `library(uscogdata)`. The old table
|
||||
did not account for it anywhere.
|
||||
* **The remote cost is round-trips, not scanning.** A repeat query over
|
||||
already-touched partitions is ~1.5 s against ~4 s cold, and a full-history
|
||||
query costs ~7 s whether it runs first or last.
|
||||
* The corpus size is **~201 MB**, not 190.6 MB -- row-group chunking added ~3.4%
|
||||
and the old figure was ambiguous between MB and MiB besides.
|
||||
* Documented that a burst of remote queries can be rate-limited by the host
|
||||
(`HTTP 429`), which is another reason to mirror for real work.
|
||||
|
||||
## Fixes
|
||||
|
||||
* `cog_gov_search()` now orders by `population_acs DESC NULLS LAST,
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# uscogdata
|
||||
|
||||
<!-- badges: start -->
|
||||
[](https://github.com/civilytics/uscogdata/actions/workflows/R-CMD-check.yaml)
|
||||
[](https://civilytics.r-universe.dev/uscogdata)
|
||||
[](LICENSE.md)
|
||||
<!-- badges: end -->
|
||||
@@ -18,7 +19,7 @@ carries provenance describing what was converted, what was aggregated, and
|
||||
which known series breaks intersect your query.
|
||||
|
||||
**Scope:** government types 0–3 (state, county, municipality, township).
|
||||
56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
|
||||
56 fiscal years, 46,148,034 rows, ~201 MB. There is no source data for FY1968
|
||||
or FY1969. Special districts (type 4) and school districts (type 5) are
|
||||
excluded pending validation.
|
||||
|
||||
@@ -85,14 +86,38 @@ cog_explain(spend)
|
||||
|
||||
| | Remote (default) | Mirrored |
|
||||
|---|---|---|
|
||||
| Setup | none | `cog_mirror(dest)`, 190.6 MB once |
|
||||
| Disk used | **0 MB** — HTTP range requests only | 190.6 MB |
|
||||
| Per query | ~4 s (one government, one year)<br>~6 s (one government, 23 years) | local speed |
|
||||
| Setup | none | `cog_mirror(dest)`, ~201 MB once |
|
||||
| Disk used | **0 MB** — HTTP range requests only | ~201 MB |
|
||||
| Opening a session | ~7.5 s | ~0.1 s |
|
||||
| One government, one year | ~4 s | ~0.05 s |
|
||||
| One government, full history | ~7 s | ~0.1 s |
|
||||
| Later queries, same session | ~1.5 s | ~0.05 s |
|
||||
| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
|
||||
|
||||
**A local mirror is roughly 60–80x faster, and it is one function call.** That is
|
||||
by far the largest difference any of these settings makes. If you are going to
|
||||
ask more than a handful of questions, mirror first.
|
||||
|
||||
Measured 2026-08-10 on a 16-core Linux workstation against the published corpus
|
||||
(schema v7, `pipeline_commit 3d28ddd`), fresh R session per arm. A one-off
|
||||
question costs about **12 seconds end to end remotely and 0.15 seconds
|
||||
mirrored**, session setup included.
|
||||
|
||||
Two things the per-query rows hide:
|
||||
|
||||
- **Opening the session is the single largest remote cost** — larger than any
|
||||
one query. It fetches the manifest and registers 23 SQL views over HTTPS, and
|
||||
it lands on your first query, not on `library(uscogdata)`.
|
||||
- **The cost is network round-trips, not scanning.** A repeat query against
|
||||
partitions this session has already touched is ~1.5 s rather than ~4 s, and a
|
||||
full-history query costs ~7 s whether it runs first or last. What you are
|
||||
paying for is reaching each of the 56 yearly files over HTTPS the first time.
|
||||
|
||||
Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
|
||||
works out which row groups it needs, and reads only those. Nothing is cached
|
||||
between sessions either, so every query goes back to the network.
|
||||
between sessions either, so every query goes back to the network — and a session
|
||||
that issues many remote queries in quick succession can be rate-limited by the
|
||||
host (`HTTP Error: ... 429`). Both are further reasons to mirror for real work.
|
||||
|
||||
The default points at a public HuggingFace mirror of the corpus. If you would
|
||||
rather not depend on a third party — for reproducibility, for an air-gapped
|
||||
|
||||
Reference in New Issue
Block a user