Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5f81ae386b
|
||
|
|
d2caa6de97 | ||
|
|
303aa59b07
|
||
|
|
81f72321ee
|
||
|
|
5cb83d8f1d |
@@ -0,0 +1,64 @@
|
|||||||
|
# Explain the mirror contribution flow on every incoming pull request.
|
||||||
|
#
|
||||||
|
# This repository is a MIRROR. A PR opened here is landed on the canonical Gitea
|
||||||
|
# repository and syncs back; because the merge preserves the contributor's
|
||||||
|
# commits at their original SHAs, GitHub marks the PR "Merged" on its own as
|
||||||
|
# soon as the mirror syncs -- with nobody visibly clicking Merge.
|
||||||
|
#
|
||||||
|
# Without this comment, that reads as a rejection: the contributor sees their PR
|
||||||
|
# close with no review, no merge button pressed, and no explanation. It is
|
||||||
|
# actually the successful outcome. Say so up front, before it happens.
|
||||||
|
#
|
||||||
|
# WHY pull_request_target AND NOT pull_request:
|
||||||
|
# a `pull_request` run from a fork gets a read-only token, so it cannot post a
|
||||||
|
# comment -- which is exactly the case this workflow exists to serve.
|
||||||
|
# `pull_request_target` runs in the context of the BASE repo and gets a writable
|
||||||
|
# token. That is only safe because this job never checks out or executes the
|
||||||
|
# contributor's code; it posts a fixed string. Do not add a checkout of
|
||||||
|
# `github.event.pull_request.head.sha` here -- that combination is the standard
|
||||||
|
# pull_request_target privilege-escalation hole.
|
||||||
|
name: Explain the mirror flow
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request_target:
|
||||||
|
types: [opened]
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
pull-requests: write
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
comment:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Post the contribution-flow explainer
|
||||||
|
uses: actions/github-script@v7
|
||||||
|
with:
|
||||||
|
script: |
|
||||||
|
const body = [
|
||||||
|
"Thanks for this — and one thing worth knowing before it happens.",
|
||||||
|
"",
|
||||||
|
"**This repository is a mirror.** Development happens on Gitea at",
|
||||||
|
"`gitea.civilytics.org/Civilytics/uscogdata`. Your pull request will be fetched",
|
||||||
|
"from here, landed there, and synced back.",
|
||||||
|
"",
|
||||||
|
"Because that merge preserves your commits at their original SHAs, **GitHub will",
|
||||||
|
"mark this pull request \"Merged\" on its own** — without anyone visibly clicking",
|
||||||
|
"the Merge button, and possibly without a review comment on this page first.",
|
||||||
|
"",
|
||||||
|
"> If your pull request closes as \"Merged\" and nobody appears to have merged it,",
|
||||||
|
"> that is the normal, successful outcome — not a rejection.",
|
||||||
|
"",
|
||||||
|
"If it is *not* going to be merged, you will get an actual reply saying so.",
|
||||||
|
"",
|
||||||
|
"Substantial contributions get a `ctb` entry in `DESCRIPTION`, which surfaces in",
|
||||||
|
"`citation(\"uscogdata\")`. There is no CLA and no DCO sign-off.",
|
||||||
|
"",
|
||||||
|
"Full details: [CONTRIBUTING.md](https://github.com/civilytics/uscogdata/blob/main/CONTRIBUTING.md).",
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
await github.rest.issues.createComment({
|
||||||
|
owner: context.repo.owner,
|
||||||
|
repo: context.repo.repo,
|
||||||
|
issue_number: context.payload.pull_request.number,
|
||||||
|
body,
|
||||||
|
});
|
||||||
+10
-2
@@ -101,5 +101,13 @@ but "usually" is not a release gate.
|
|||||||
variables set. This is the only check that catches a
|
variables set. This is the only check that catches a
|
||||||
corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
|
corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
|
||||||
8. Bump `Version` and add a `NEWS.md` section.
|
8. Bump `Version` and add a `NEWS.md` section.
|
||||||
9. Tag, then update the r-universe registry pin at
|
9. Tag on **Gitea** (`git tag -a vX.Y.Z && git push origin vX.Y.Z`). The mirror
|
||||||
`github.com/civilytics/civilytics.r-universe.dev`.
|
workflow carries tags to GitHub on its own — confirm the tag appears at
|
||||||
|
`github.com/civilytics/uscogdata/tags` before continuing.
|
||||||
|
10. Update the r-universe registry pin at
|
||||||
|
`github.com/civilytics/civilytics.r-universe.dev` — edit `packages.json`'s
|
||||||
|
`branch` to the new tag. **r-universe will not pick up a release until this
|
||||||
|
is edited**: the pin is a tag, deliberately, so a mid-refactor `main` is
|
||||||
|
never published as a release. `"branch": "*release"` would track releases
|
||||||
|
automatically, but it needs a GitHub *Release* object and the mirror pushes
|
||||||
|
tags only — so it would silently never update.
|
||||||
|
|||||||
@@ -1,46 +1,5 @@
|
|||||||
# uscogdata 0.4.0
|
# uscogdata 0.4.0
|
||||||
|
|
||||||
## DuckDB's resource budget is configurable
|
|
||||||
|
|
||||||
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
|
|
||||||
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
|
|
||||||
spellings) cap the DuckDB connection the package opens. Both follow the same
|
|
||||||
env-var > option > default precedence as `USCOGDATA_URL`.
|
|
||||||
|
|
||||||
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
|
|
||||||
before -- every visible core. That is right for one interactive session on a
|
|
||||||
dedicated machine and wrong for a server: where several readers share a host, each
|
|
||||||
otherwise claims the whole machine and they contend. Capping measured ~5% on a
|
|
||||||
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
|
|
||||||
cores), which is cheap enough that a server should always cap.
|
|
||||||
|
|
||||||
This replaces a workaround in which a consumer reached into the package namespace
|
|
||||||
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
|
|
||||||
`SET threads` -- depending both on a private name and on the session already being
|
|
||||||
open.
|
|
||||||
|
|
||||||
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
|
|
||||||
|
|
||||||
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
|
|
||||||
verbs were left materializing everything and slicing in R. Both now take
|
|
||||||
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
|
|
||||||
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
|
|
||||||
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
|
|
||||||
than a second query.
|
|
||||||
|
|
||||||
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
|
|
||||||
returns the entire 40,336-row crosswalk when called with no filter.
|
|
||||||
|
|
||||||
Two refusals rather than silent surprises:
|
|
||||||
|
|
||||||
* `cog_balances(recipe = , limit = )` aborts with class
|
|
||||||
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
|
|
||||||
separate query that pagination is not wired into.
|
|
||||||
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
|
|
||||||
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
|
|
||||||
per requested name with a sidecar covering all of them; a page of that is not
|
|
||||||
a page of anything the caller asked for.
|
|
||||||
|
|
||||||
## Cohorts can be named by predicate, not just by id
|
## Cohorts can be named by predicate, not just by id
|
||||||
|
|
||||||
`cog_spending()`, `cog_revenue()` and `cog_balances()` gain optional `state`
|
`cog_spending()`, `cog_revenue()` and `cog_balances()` gain optional `state`
|
||||||
@@ -87,6 +46,71 @@ When the cohort is named by predicate there is no id list to report, so
|
|||||||
`provenance$scope$cohort` carries `state`, `type` and `n_governments` instead.
|
`provenance$scope$cohort` carries `state`, `type` and `n_governments` instead.
|
||||||
A `govid`-named cohort's provenance is unchanged.
|
A `govid`-named cohort's provenance is unchanged.
|
||||||
|
|
||||||
|
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
|
||||||
|
|
||||||
|
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
|
||||||
|
verbs were left materializing everything and slicing in R. Both now take
|
||||||
|
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
|
||||||
|
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
|
||||||
|
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
|
||||||
|
than a second query.
|
||||||
|
|
||||||
|
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
|
||||||
|
returns the entire 40,336-row crosswalk when called with no filter.
|
||||||
|
|
||||||
|
Two refusals rather than silent surprises:
|
||||||
|
|
||||||
|
* `cog_balances(recipe = , limit = )` aborts with class
|
||||||
|
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
|
||||||
|
separate query that pagination is not wired into.
|
||||||
|
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
|
||||||
|
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
|
||||||
|
per requested name with a sidecar covering all of them; a page of that is not
|
||||||
|
a page of anything the caller asked for.
|
||||||
|
|
||||||
|
## DuckDB's resource budget is configurable
|
||||||
|
|
||||||
|
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
|
||||||
|
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
|
||||||
|
spellings) cap the DuckDB connection the package opens. Both follow the same
|
||||||
|
env-var > option > default precedence as `USCOGDATA_URL`.
|
||||||
|
|
||||||
|
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
|
||||||
|
before -- every visible core. That is right for one interactive session on a
|
||||||
|
dedicated machine and wrong for a server: where several readers share a host, each
|
||||||
|
otherwise claims the whole machine and they contend. Capping measured ~5% on a
|
||||||
|
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
|
||||||
|
cores), which is cheap enough that a server should always cap.
|
||||||
|
|
||||||
|
This replaces a workaround in which a consumer reached into the package namespace
|
||||||
|
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
|
||||||
|
`SET threads` -- depending both on a private name and on the session already being
|
||||||
|
open.
|
||||||
|
|
||||||
|
## Documentation: the corpus-access table is re-measured and honest
|
||||||
|
|
||||||
|
The README's "two ways to read the corpus" table carried figures taken before
|
||||||
|
the corpus was re-chunked into row groups (cog_pipeline#93, published
|
||||||
|
2026-08-09) and reported the mirrored column as "local speed" with no number at
|
||||||
|
all. Re-measured 2026-08-10 against the published corpus (`pipeline_commit
|
||||||
|
3d28ddd`), fresh R session per arm:
|
||||||
|
|
||||||
|
* **A local mirror is roughly 60-80x faster.** A one-off question costs ~12 s
|
||||||
|
end to end remotely against ~0.15 s mirrored. That is the largest single
|
||||||
|
difference available to a user and it is now stated outright rather than left
|
||||||
|
as "local speed".
|
||||||
|
* **Opening the session is the largest remote cost** (~7.5 s -- manifest fetch
|
||||||
|
plus 23 view registrations over HTTPS), larger than any individual query, and
|
||||||
|
it lands on the first query rather than on `library(uscogdata)`. The old table
|
||||||
|
did not account for it anywhere.
|
||||||
|
* **The remote cost is round-trips, not scanning.** A repeat query over
|
||||||
|
already-touched partitions is ~1.5 s against ~4 s cold, and a full-history
|
||||||
|
query costs ~7 s whether it runs first or last.
|
||||||
|
* The corpus size is **~201 MB**, not 190.6 MB -- row-group chunking added ~3.4%
|
||||||
|
and the old figure was ambiguous between MB and MiB besides.
|
||||||
|
* Documented that a burst of remote queries can be rate-limited by the host
|
||||||
|
(`HTTP 429`), which is another reason to mirror for real work.
|
||||||
|
|
||||||
## Fixes
|
## Fixes
|
||||||
|
|
||||||
* `cog_gov_search()` now orders by `population_acs DESC NULLS LAST,
|
* `cog_gov_search()` now orders by `population_acs DESC NULLS LAST,
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
# uscogdata
|
# uscogdata
|
||||||
|
|
||||||
<!-- badges: start -->
|
<!-- badges: start -->
|
||||||
|
[](https://github.com/civilytics/uscogdata/actions/workflows/R-CMD-check.yaml)
|
||||||
[](https://civilytics.r-universe.dev/uscogdata)
|
[](https://civilytics.r-universe.dev/uscogdata)
|
||||||
[](LICENSE.md)
|
[](LICENSE.md)
|
||||||
<!-- badges: end -->
|
<!-- badges: end -->
|
||||||
@@ -18,7 +19,7 @@ carries provenance describing what was converted, what was aggregated, and
|
|||||||
which known series breaks intersect your query.
|
which known series breaks intersect your query.
|
||||||
|
|
||||||
**Scope:** government types 0–3 (state, county, municipality, township).
|
**Scope:** government types 0–3 (state, county, municipality, township).
|
||||||
56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
|
56 fiscal years, 46,148,034 rows, ~201 MB. There is no source data for FY1968
|
||||||
or FY1969. Special districts (type 4) and school districts (type 5) are
|
or FY1969. Special districts (type 4) and school districts (type 5) are
|
||||||
excluded pending validation.
|
excluded pending validation.
|
||||||
|
|
||||||
@@ -85,14 +86,38 @@ cog_explain(spend)
|
|||||||
|
|
||||||
| | Remote (default) | Mirrored |
|
| | Remote (default) | Mirrored |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Setup | none | `cog_mirror(dest)`, 190.6 MB once |
|
| Setup | none | `cog_mirror(dest)`, ~201 MB once |
|
||||||
| Disk used | **0 MB** — HTTP range requests only | 190.6 MB |
|
| Disk used | **0 MB** — HTTP range requests only | ~201 MB |
|
||||||
| Per query | ~4 s (one government, one year)<br>~6 s (one government, 23 years) | local speed |
|
| Opening a session | ~7.5 s | ~0.1 s |
|
||||||
|
| One government, one year | ~4 s | ~0.05 s |
|
||||||
|
| One government, full history | ~7 s | ~0.1 s |
|
||||||
|
| Later queries, same session | ~1.5 s | ~0.05 s |
|
||||||
| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
|
| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
|
||||||
|
|
||||||
|
**A local mirror is roughly 60–80x faster, and it is one function call.** That is
|
||||||
|
by far the largest difference any of these settings makes. If you are going to
|
||||||
|
ask more than a handful of questions, mirror first.
|
||||||
|
|
||||||
|
Measured 2026-08-10 on a 16-core Linux workstation against the published corpus
|
||||||
|
(schema v7, `pipeline_commit 3d28ddd`), fresh R session per arm. A one-off
|
||||||
|
question costs about **12 seconds end to end remotely and 0.15 seconds
|
||||||
|
mirrored**, session setup included.
|
||||||
|
|
||||||
|
Two things the per-query rows hide:
|
||||||
|
|
||||||
|
- **Opening the session is the single largest remote cost** — larger than any
|
||||||
|
one query. It fetches the manifest and registers 23 SQL views over HTTPS, and
|
||||||
|
it lands on your first query, not on `library(uscogdata)`.
|
||||||
|
- **The cost is network round-trips, not scanning.** A repeat query against
|
||||||
|
partitions this session has already touched is ~1.5 s rather than ~4 s, and a
|
||||||
|
full-history query costs ~7 s whether it runs first or last. What you are
|
||||||
|
paying for is reaching each of the 56 yearly files over HTTPS the first time.
|
||||||
|
|
||||||
Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
|
Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
|
||||||
works out which row groups it needs, and reads only those. Nothing is cached
|
works out which row groups it needs, and reads only those. Nothing is cached
|
||||||
between sessions either, so every query goes back to the network.
|
between sessions either, so every query goes back to the network — and a session
|
||||||
|
that issues many remote queries in quick succession can be rate-limited by the
|
||||||
|
host (`HTTP Error: ... 429`). Both are further reasons to mirror for real work.
|
||||||
|
|
||||||
The default points at a public HuggingFace mirror of the corpus. If you would
|
The default points at a public HuggingFace mirror of the corpus. If you would
|
||||||
rather not depend on a third party — for reproducibility, for an air-gapped
|
rather not depend on a third party — for reproducibility, for an air-gapped
|
||||||
|
|||||||
Reference in New Issue
Block a user