Compare commits

...
Author SHA1 Message Date
jared 7274ce3bfe chore: correct the roborev exclusions and refresh the guidelines
Mirror to GitHub / mirror (push) Successful in 9s
R-CMD-check / check (push) Successful in 3m46s
roborev matches excluded_commit_patterns as substrings, and compass's cadence rule
requires `type(ws): subject (#N)`, so `chore:` never matched `chore(engine): ...`.
Adds `chore(`.

Does not add `docs(`. Those commits carry the journal entry and the board
narrative, and the plain-language rule exists to check exactly that prose -- it has
no other commit to fire on.

The guidelines were also stale: they were composed before base.md gained the
plain-language rule, and nothing re-composes them on its own. Refreshed, which is
what put that rule in this repository for the first time.
2026-08-23 23:57:05 -04:00
jared 24e86ed598 chore: retire the three plans whose work has shipped
Mirror to GitHub / mirror (push) Successful in 6s
R-CMD-check / check (push) Successful in 3m21s
186 unticked checkboxes across three plans, none of them outstanding work.
Superpowers-style plans are execution transcripts: nobody ticks the boxes, and
the plan is abandoned at the point the work is done. Left in place they are
indistinguishable from a live backlog -- the old compass retrofit rule, "open
checkboxes become issues", would have filed 186 issues for finished work.

Evidence, from scripts/plans.py plus a check by hand:

  2026-08-04-partial-coverage-signposting  NEWS: "Coverage signposting ..."
  2026-08-08-public-release                NEWS: "First public release."
  2026-04-29-per-year-population-denominator
      no NEWS line matched and 19 of the 21 files it names exist, so the
      classifier called it ambiguous. Confirmed shipped by hand:
      33c0274 docs(news): per-year population denominators, plus the feat
      commits behind it.

specs/ is untouched. A spec explains why the design is what it is and stays
useful; a plan is scaffolding, and once the building stands it is in the way.
All three remain in git history.
2026-08-23 23:32:11 -04:00
jared 1ec20174b7 chore: move compass out of docs/, which pkgdown deletes
Mirror to GitHub / mirror (push) Successful in 7s
R-CMD-check / check (push) Successful in 3m28s
compass.toml, JOURNAL.md, STATUS.md and decisions/ were sitting inside pkgdown's
output directory. Asked directly, pkgdown listed docs/pm and docs/decisions among
the 28 top-level entries clean_site() would delete, and the guard that would have
refused -- check_dest_is_pkgdown() -- was satisfied by docs/pkgdown.yml. After the
move it lists 26 and none of them are compass's.

Nothing was lost. The journal had no entries and there were no decision records
yet, so this was the cheapest moment to move.

The .gitignore workaround goes with it. Re-including two children of an excluded
docs/ forced the rule to be written as /docs/* plus two negations, which changed
the anchoring and made the fixture corpus's own docs/ need a separate rule. A bare
docs/ matches at any depth again, so both are unnecessary.

.Rbuildignore gains ^pm$ -- R CMD check flags a non-standard top-level directory.

Compass reads both layouts, so this repository worked either way; the point is
that docs/ is a directory another tool empties.
2026-08-23 23:26:33 -04:00
Jared Knowles 2e317d3a0f chore: adopt compass for project tracking
Mirror to GitHub / mirror (push) Successful in 10s
R-CMD-check / check (push) Successful in 3m55s
Three workstreams -- the query verbs, the corpus and how it is mounted, and the
docs that explain both. Each can go stale independently, which is what the
workstream boundary is for: a change to a verb obligates the vignettes, a change
to the corpus obligates NEWS.

All six open issues now carry ws/ and type/ labels, applied additively so #36
kept its existing kodor/, severity/, south-guide and verdict/ labels. The
generated board is pinned as issue #68.

.roborev.toml carries review guidelines composed from a shared baseline, the R
package overlay, and this project's own conventions read out of CLAUDE.md: the
ensure_session-then-dbGetQuery order, the tbl_df-with-provenance return
contract, coerce_govid_input at the boundary, the two SQL layers, no arrow, and
withr as tests-only.

The .gitignore change is load-bearing. pkgdown output made docs/ ignored, which
would have left every compass file untracked and unable to travel to another
machine. Git cannot re-include anything beneath an excluded directory, so the
rule had to list children instead. That forced an anchoring change: a bare
"docs/" matches at any depth, "/docs/*" only at the root, so the fixture corpus
docs directory needed an explicit rule to stay excluded as before.
2026-08-23 16:17:20 -04:00
jared f7c606984f Merge pull request 'ci: mirror canonical tag objects, not the lightweight refs checkout builds' (#67) from ci/mirror-canonical-tags into main
Mirror to GitHub / mirror (push) Successful in 9s
R-CMD-check / check (push) Successful in 3m14s
Reviewed-on: #67
2026-08-11 10:03:43 -04:00
jared 9617b86a26 ci: mirror canonical tag objects, not the lightweight refs checkout builds
R-CMD-check / check (push) Successful in 3m45s
R-CMD-check / check (pull_request) Successful in 3m29s
v0.4.0 failed to mirror, and it would have failed at every future release.

On a tag-triggered run, checkout materializes refs/tags/<tag> as a LIGHTWEIGHT
tag at the commit SHA; the annotated tag object Gitea holds is never fetched.
Run 2070 therefore pushed a lightweight v0.4.0 to GitHub and reported success.
Run 2076, on main with fetch-depth 0, did fetch the real annotated object and
was rejected with "already exists" trying to correct it -- git will not clobber
an existing tag. Gitea had e138eeb (annotated), GitHub had d2caa6d (the commit).

Re-fetch canonical tag objects from Gitea before pushing. --force rewrites LOCAL
tag refs only; it is not a force push and does not weaken the non-force
guarantee on main. It is required: without it the fetch is rejected with "would
clobber existing tag" and the lightweight ref survives to be mirrored again.

Verified against a scratch clone -- lightweight d2caa6d becomes annotated
e138eeb, peeling back to the same commit; without --force the tag is unchanged.

The GitHub tag was repaired by hand out of band, so the two remotes already
agree; this stops it recurring.
2026-08-11 09:57:14 -04:00
jared 224e5d0530 Merge pull request 'chore: CI badge, mirror PR explainer, and the tag-pin release step (#47)' (#66) from chore/release-47-badges-mirror-pr into main
Mirror to GitHub / mirror (push) Failing after 11s
R-CMD-check / check (push) Successful in 3m15s
Reviewed-on: #66
2026-08-11 09:44:33 -04:00
jared 5f81ae386b chore: CI badge, mirror PR explainer, and the tag-pin release step (#47)
R-CMD-check / check (pull_request) Successful in 3m49s
R-CMD-check / check (push) Successful in 3m52s
Three items #47 absorbed from #46, held back so a dead badge would not sit
beside an unresolved r-universe one. Both resolve now that v0.4.0 is tagged.

- R-CMD-check badge pointing at the GitHub mirror's workflow, where the
  4-platform matrix actually runs.

- A pull_request_target workflow explaining the mirror flow on every incoming
  PR. A PR here is landed on Gitea and syncs back, and because the merge
  preserves the contributor's commits at their original SHAs, GitHub marks the
  PR 'Merged' with nobody visibly clicking Merge. To a first-time contributor
  that reads as rejection. Say so before it happens.

  pull_request_target rather than pull_request because a fork PR's token is
  read-only under the latter -- it could not comment, which is the entire job.
  That is only safe because this never checks out or runs contributor code; the
  file says so and says not to add a checkout.

- CONTRIBUTING's release checklist now spells out that the tag goes on Gitea and
  the mirror carries it, and that r-universe does NOT pick up a release until
  packages.json's branch pin is edited. '*release' would automate it but needs a
  GitHub Release object, and the mirror pushes tags only -- so it would silently
  never update. Learned while doing this release.
2026-08-10 19:47:26 -04:00
jared d2caa6de97 Merge pull request 'docs: re-measure the corpus-access table against the published corpus (#56)' (#65) from docs/readme-perf-remeasure-56 into main
Mirror to GitHub / mirror (push) Successful in 9s
R-CMD-check / check (push) Successful in 3m25s
2026-08-10 19:36:42 -04:00
jared 303aa59b07 docs: order the 0.4.0 NEWS sections by user impact, not merge order
R-CMD-check / check (push) Successful in 3m58s
R-CMD-check / check (pull_request) Successful in 3m38s
The three 0.4.0 features landed in the order their PRs merged, which buried the
headline change (cohort predicates, 4.8x) below an operator config knob and a
documentation note. Reordered to: cohorts, pagination, DuckDB budget, docs,
Fixes -- with Fixes last, where it was already.

The stacked PRs each appended their own '## Fixes' heading, so resolving the
conflicts also folded two of them into the single section that belongs there.

Content is byte-identical to what merged; only section order changed. Verified
by diffing the sorted non-blank lines against the previous commit.
2026-08-10 19:29:43 -04:00
jared 81f72321ee docs: re-measure the corpus-access table against the published corpus (#56)
figures predate the row-group rechunk (cog_pipeline#93, published 2026-08-09)
and reported the mirrored column as 'local speed' with no number -- hiding the
largest difference available to a user.

Measured 2026-08-10, fresh R session per arm, against the live corpus at
pipeline_commit 3d28ddd. Madison WI, 16-core Linux workstation.

Three findings the old table could not express:

- A local mirror is 60-80x faster. A one-off question is ~12 s end to end
  remotely against ~0.15 s mirrored. Stated outright now, because it is a
  bigger and cheaper win for users than anything in the R code.

- Opening the session is the LARGEST remote cost (~7.5 s), bigger than any
  individual query, and it lands on the user's first query rather than on
  library(). The old table accounted for it nowhere, so every per-query figure
  was quietly missing it.

- The remote cost is round-trips, not scanning: a repeat query over
  already-touched partitions is ~1.5 s against ~4 s cold, and a full-history
  query costs ~7 s whether it runs first or last (verified by running the arms
  in both orders). This is why #93's 1.4-1.7x, measured through cog-api against
  a local mount, does not show up on the remote path -- there, network latency
  swamps scan time.

Corpus size corrected to ~201 MB: row-group chunking added ~3.4%, and 190.6 was
ambiguous between MB and MiB besides. Measured from the manifest and on disk.
The 0.3.0 NEWS section keeps 190.6 -- it was correct for that release.

Also documented HTTP 429: a burst of remote queries gets rate-limited by the
host. Hit while taking these measurements.
2026-08-10 19:29:05 -04:00
jared 5cb83d8f1d Merge pull request 'feat: limit/offset on cog_gov_search() and cog_balances() (#57)' (#63) from feat/pagination-search-balances-57 into main
Mirror to GitHub / mirror (push) Successful in 12s
R-CMD-check / check (push) Successful in 4m26s
2026-08-10 19:28:09 -04:00
15 changed files with 386 additions and 3598 deletions
+1
View File
@@ -3,6 +3,7 @@
^\.Rproj\.user$
^_pkgdown\.yml$
^docs$
^pm$
^Meta$
^doc$
^pkgdown$
+12
View File
@@ -39,5 +39,17 @@ jobs:
echo "PAT_GH is unset -- add it under Settings > Actions > Secrets." >&2
exit 1
fi
# On a tag-triggered run, checkout materializes refs/tags/<tag> as a
# LIGHTWEIGHT tag at the commit SHA -- the annotated tag object Gitea
# holds is never fetched. Mirroring that strips the annotation, and the
# NEXT run on main (which does fetch the real object) is then rejected
# with "already exists" trying to correct it, because git will not
# clobber an existing tag. That is why v0.4.0 failed to mirror.
#
# Re-fetch canonical tag objects from Gitea first. --force here rewrites
# LOCAL tag refs only; it is not a force push and does not weaken the
# non-force guarantee on main documented above.
git fetch --tags --force origin
git push "https://x-access-token:${PAT_GH}@github.com/civilytics/uscogdata.git" \
HEAD:refs/heads/main --tags
+64
View File
@@ -0,0 +1,64 @@
# Explain the mirror contribution flow on every incoming pull request.
#
# This repository is a MIRROR. A PR opened here is landed on the canonical Gitea
# repository and syncs back; because the merge preserves the contributor's
# commits at their original SHAs, GitHub marks the PR "Merged" on its own as
# soon as the mirror syncs -- with nobody visibly clicking Merge.
#
# Without this comment, that reads as a rejection: the contributor sees their PR
# close with no review, no merge button pressed, and no explanation. It is
# actually the successful outcome. Say so up front, before it happens.
#
# WHY pull_request_target AND NOT pull_request:
# a `pull_request` run from a fork gets a read-only token, so it cannot post a
# comment -- which is exactly the case this workflow exists to serve.
# `pull_request_target` runs in the context of the BASE repo and gets a writable
# token. That is only safe because this job never checks out or executes the
# contributor's code; it posts a fixed string. Do not add a checkout of
# `github.event.pull_request.head.sha` here -- that combination is the standard
# pull_request_target privilege-escalation hole.
name: Explain the mirror flow
on:
pull_request_target:
types: [opened]
permissions:
pull-requests: write
jobs:
comment:
runs-on: ubuntu-latest
steps:
- name: Post the contribution-flow explainer
uses: actions/github-script@v7
with:
script: |
const body = [
"Thanks for this — and one thing worth knowing before it happens.",
"",
"**This repository is a mirror.** Development happens on Gitea at",
"`gitea.civilytics.org/Civilytics/uscogdata`. Your pull request will be fetched",
"from here, landed there, and synced back.",
"",
"Because that merge preserves your commits at their original SHAs, **GitHub will",
"mark this pull request \"Merged\" on its own** — without anyone visibly clicking",
"the Merge button, and possibly without a review comment on this page first.",
"",
"> If your pull request closes as \"Merged\" and nobody appears to have merged it,",
"> that is the normal, successful outcome — not a rejection.",
"",
"If it is *not* going to be merged, you will get an actual reply saying so.",
"",
"Substantial contributions get a `ctb` entry in `DESCRIPTION`, which surfaces in",
"`citation(\"uscogdata\")`. There is no CLA and no DCO sign-off.",
"",
"Full details: [CONTRIBUTING.md](https://github.com/civilytics/uscogdata/blob/main/CONTRIBUTING.md).",
].join("\n");
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number,
body,
});
+7
View File
@@ -4,6 +4,10 @@
.Ruserdata
*.Rproj
inst/doc
# pkgdown output. Compass used to keep its files in docs/pm/ and
# docs/decisions/, which forced this to be written as children with two
# re-includes -- git cannot re-include anything beneath an excluded directory.
# Compass lives in pm/ now, so the whole directory can be excluded again.
docs/
/doc/
/Meta/
@@ -12,3 +16,6 @@ docs/
# SDD working artifacts (ledger, briefs, review packages) — plans/ stays tracked
.superpowers/sdd/
.compass-cache/
# roborev snapshots
/.roborev/
+60
View File
@@ -0,0 +1,60 @@
# roborev configuration, initialised by compass.
# Reviews are queued to a background daemon -- they never block a commit.
post_commit_review = 'commit'
excluded_commit_patterns = ['WIP', 'chore:', 'chore(', 'docs:', 'Merge ']
review_guidelines = '''
# --- compass:begin (generated -- edit the sources, not this) ---
- Prefer returning new values to mutating arguments in place. A function that edits
its caller's object is a bug waiting for a second caller.
- Validate at system boundaries -- user input, API responses, file contents, config.
Fail fast with a message naming the field and the file.
- Never swallow an error. Handle it or let it propagate; a bare catch that continues
is worse than a crash.
- No hardcoded secrets, tokens, or credentials, and no secrets in log output or error
messages.
- Parameterise every query. String-built SQL is a defect even when the input looks safe.
- Keep functions under roughly 50 lines and files under roughly 400. Flag nesting
deeper than four levels.
- No magic numbers or hardcoded paths -- name them as constants or read them from config.
- New behaviour needs a test. A bug fix needs a test that fails without the fix.
- Prose a person reads -- an issue title or body, a journal entry, a decision record,
the narrative on the status board -- names the action or the thing, not the shape of
the machinery. Flag "gate", "seam", "surface area", "load-bearing", "first-class",
"primitive", "blast radius". A project's own defined vocabulary is not the target.
- Use the native pipe `|>`, not magrittr `%>%`.
- snake_case for objects and functions; UPPER_SNAKE for constants. Never use `.` as a
word separator in a function name -- it collides with S3 dispatch.
- Validate arguments at the top of exported functions with `stopifnot()` or an explicit
check, and say which argument was wrong.
- Never `setDT()`, `set()`, or otherwise modify by reference a data.table the caller
still owns. `as.data.table()` copies; use it.
- Prefer `vapply()` to `sapply()` -- `sapply()` silently returns a list when the type
varies, which turns a type error into a downstream mystery.
- Use `seq_len(n)` / `seq_along(x)`, never `1:n`, which iterates backwards when n is 0.
- Compare strings with `==` only after checking for NA; use `identical()` for scalars
where NA would be wrong.
- Do not call `library()` inside package or module files; attach packages in scripts and
test helpers only.
- Namespace-qualify calls into other packages (`stats::sd`) in code that is sourced.
- Every exported function needs roxygen with `@param` for each argument (type, meaning,
and why the default is what it is) and `@return`. Add `@examples` for exported API.
- Declare dependencies in DESCRIPTION. Prefer base R or an existing dependency over
adding a new one; a package with zero hard deps is worth keeping that way.
- Signal errors with `stop()` carrying a condition class, so callers can catch the kind
rather than matching on message text.
- Keep internals internal. Export only what a user needs; an accidentally exported
helper becomes an API you have to keep.
- Tests use testthat edition 3. Each test is self-sufficient -- no reliance on state
left by an earlier test or on a fixture built elsewhere in the file.
- Prefer duplication in tests over a helper that hides what is being asserted.
- Every verb calls .ensure_session() first, then queries via DBI::dbGetQuery().
- A verb's return value is always a tbl_df carrying a provenance attribute.
- govid inputs always go through .coerce_govid_input(); it accepts a character vector or a data frame.
- SQL has two layers: view definitions are numbered .sql files in inst/sql/ registered by .register_views(); query construction is inline sprintf() in R. Add a view as a file; build a query in R.
- No arrow dependency -- DuckDB reads parquet natively.
- withr is Suggests-only and must appear in tests alone.
- Tests must pass offline against the bundled fixture; tests/testthat/setup.R sets USCOGDATA_URL for that.
# --- compass:end ---
'''
+10 -2
View File
@@ -101,5 +101,13 @@ but "usually" is not a release gate.
variables set. This is the only check that catches a
corpus-unreachable defect, and its absence is why 0.3.0 needed fixing.
8. Bump `Version` and add a `NEWS.md` section.
9. Tag, then update the r-universe registry pin at
`github.com/civilytics/civilytics.r-universe.dev`.
9. Tag on **Gitea** (`git tag -a vX.Y.Z && git push origin vX.Y.Z`). The mirror
workflow carries tags to GitHub on its own — confirm the tag appears at
`github.com/civilytics/uscogdata/tags` before continuing.
10. Update the r-universe registry pin at
`github.com/civilytics/civilytics.r-universe.dev` — edit `packages.json`'s
`branch` to the new tag. **r-universe will not pick up a release until this
is edited**: the pin is a tag, deliberately, so a mid-refactor `main` is
never published as a release. `"branch": "*release"` would track releases
automatically, but it needs a GitHub *Release* object and the mirror pushes
tags only — so it would silently never update.
+65 -41
View File
@@ -1,46 +1,5 @@
# uscogdata 0.4.0
## DuckDB's resource budget is configurable
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
spellings) cap the DuckDB connection the package opens. Both follow the same
env-var > option > default precedence as `USCOGDATA_URL`.
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
before -- every visible core. That is right for one interactive session on a
dedicated machine and wrong for a server: where several readers share a host, each
otherwise claims the whole machine and they contend. Capping measured ~5% on a
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
cores), which is cheap enough that a server should always cap.
This replaces a workaround in which a consumer reached into the package namespace
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
`SET threads` -- depending both on a private name and on the session already being
open.
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
verbs were left materializing everything and slicing in R. Both now take
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
than a second query.
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
returns the entire 40,336-row crosswalk when called with no filter.
Two refusals rather than silent surprises:
* `cog_balances(recipe = , limit = )` aborts with class
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
separate query that pagination is not wired into.
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
per requested name with a sidecar covering all of them; a page of that is not
a page of anything the caller asked for.
## Cohorts can be named by predicate, not just by id
`cog_spending()`, `cog_revenue()` and `cog_balances()` gain optional `state`
@@ -87,6 +46,71 @@ When the cohort is named by predicate there is no id list to report, so
`provenance$scope$cohort` carries `state`, `type` and `n_governments` instead.
A `govid`-named cohort's provenance is unchanged.
## `cog_gov_search()` and `cog_balances()` gain `limit`/`offset`
Pagination arrived on `cog_spending()`/`cog_revenue()` in 0.3.0; the other two
verbs were left materializing everything and slicing in R. Both now take
`limit`/`offset` with the same semantics: `NULL` default, the page applied in
SQL behind a deterministic `ORDER BY`, and the unpaginated count returned as a
`total_rows` attribute computed by `COUNT(*) OVER()` in the same scan rather
than a second query.
`cog_gov_search()` had no `LIMIT` at all, which made it the one verb that
returns the entire 40,336-row crosswalk when called with no filter.
Two refusals rather than silent surprises:
* `cog_balances(recipe = , limit = )` aborts with class
`uscogdata_recipe_pagination_conflict` -- a recipe's result comes from a
separate query that pagination is not wired into.
* `cog_gov_search()` in basket mode (`length(name) > 1`) aborts with class
`uscogdata_basket_pagination_conflict`. Basket mode returns one resolved row
per requested name with a sidecar covering all of them; a page of that is not
a page of anything the caller asked for.
## DuckDB's resource budget is configurable
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` (with matching
`options(uscogdata.duckdb_threads = )` / `options(uscogdata.duckdb_memory_limit = )`
spellings) cap the DuckDB connection the package opens. Both follow the same
env-var > option > default precedence as `USCOGDATA_URL`.
Unset, **no pragma is issued at all** and DuckDB's own defaults apply exactly as
before -- every visible core. That is right for one interactive session on a
dedicated machine and wrong for a server: where several readers share a host, each
otherwise claims the whole machine and they contend. Capping measured ~5% on a
single-government all-years query (502 ms at 2 threads vs 475 ms uncapped on 16
cores), which is cheap enough that a server should always cap.
This replaces a workaround in which a consumer reached into the package namespace
at boot -- `getFromNamespace(".ensure_session", "uscogdata")()` followed by a manual
`SET threads` -- depending both on a private name and on the session already being
open.
## Documentation: the corpus-access table is re-measured and honest
The README's "two ways to read the corpus" table carried figures taken before
the corpus was re-chunked into row groups (cog_pipeline#93, published
2026-08-09) and reported the mirrored column as "local speed" with no number at
all. Re-measured 2026-08-10 against the published corpus (`pipeline_commit
3d28ddd`), fresh R session per arm:
* **A local mirror is roughly 60-80x faster.** A one-off question costs ~12 s
end to end remotely against ~0.15 s mirrored. That is the largest single
difference available to a user and it is now stated outright rather than left
as "local speed".
* **Opening the session is the largest remote cost** (~7.5 s -- manifest fetch
plus 23 view registrations over HTTPS), larger than any individual query, and
it lands on the first query rather than on `library(uscogdata)`. The old table
did not account for it anywhere.
* **The remote cost is round-trips, not scanning.** A repeat query over
already-touched partitions is ~1.5 s against ~4 s cold, and a full-history
query costs ~7 s whether it runs first or last.
* The corpus size is **~201 MB**, not 190.6 MB -- row-group chunking added ~3.4%
and the old figure was ambiguous between MB and MiB besides.
* Documented that a burst of remote queries can be rate-limited by the host
(`HTTP 429`), which is another reason to mirror for real work.
## Fixes
* `cog_gov_search()` now orders by `population_acs DESC NULLS LAST,
+30 -5
View File
@@ -1,6 +1,7 @@
# uscogdata
<!-- badges: start -->
[![R-CMD-check](https://github.com/civilytics/uscogdata/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/civilytics/uscogdata/actions/workflows/R-CMD-check.yaml)
[![r-universe](https://civilytics.r-universe.dev/badges/uscogdata)](https://civilytics.r-universe.dev/uscogdata)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md)
<!-- badges: end -->
@@ -18,7 +19,7 @@ carries provenance describing what was converted, what was aggregated, and
which known series breaks intersect your query.
**Scope:** government types 0–3 (state, county, municipality, township).
56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
56 fiscal years, 46,148,034 rows, ~201 MB. There is no source data for FY1968
or FY1969. Special districts (type 4) and school districts (type 5) are
excluded pending validation.
@@ -85,14 +86,38 @@ cog_explain(spend)
| | Remote (default) | Mirrored |
|---|---|---|
| Setup | none | `cog_mirror(dest)`, 190.6 MB once |
| Disk used | **0 MB** — HTTP range requests only | 190.6 MB |
| Per query | ~4 s (one government, one year)<br>~6 s (one government, 23 years) | local speed |
| Setup | none | `cog_mirror(dest)`, ~201 MB once |
| Disk used | **0 MB** — HTTP range requests only | ~201 MB |
| Opening a session | ~7.5 s | ~0.1 s |
| One government, one year | ~4 s | ~0.05 s |
| One government, full history | ~7 s | ~0.1 s |
| Later queries, same session | ~1.5 s | ~0.05 s |
| Good for | trying it out, teaching, one-off questions | repeated analysis, offline work, reproducibility |
**A local mirror is roughly 60–80x faster, and it is one function call.** That is
by far the largest difference any of these settings makes. If you are going to
ask more than a handful of questions, mirror first.
Measured 2026-08-10 on a 16-core Linux workstation against the published corpus
(schema v7, `pipeline_commit 3d28ddd`), fresh R session per arm. A one-off
question costs about **12 seconds end to end remotely and 0.15 seconds
mirrored**, session setup included.
Two things the per-query rows hide:
- **Opening the session is the single largest remote cost** — larger than any
one query. It fetches the manifest and registers 23 SQL views over HTTPS, and
it lands on your first query, not on `library(uscogdata)`.
- **The cost is network round-trips, not scanning.** A repeat query against
partitions this session has already touched is ~1.5 s rather than ~4 s, and a
full-history query costs ~7 s whether it runs first or last. What you are
paying for is reaching each of the 56 yearly files over HTTPS the first time.
Nothing is written to disk in remote mode: DuckDB fetches the parquet footer,
works out which row groups it needs, and reads only those. Nothing is cached
between sessions either, so every query goes back to the network.
between sessions either, so every query goes back to the network — and a session
that issues many remote queries in quick succession can be rate-limited by the
host (`HTTP Error: ... 429`). Both are further reasons to mirror for real work.
The default points at a public HuggingFace mirror of the corpus. If you would
rather not depend on a third party — for reproducibility, for an air-gapped
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+14
View File
@@ -0,0 +1,14 @@
# Project journal
Append-only, newest first. **Entries are never edited** — the value of this file is
that it records what was believed at the time, including the parts that turned out
wrong. Where things stand *today* is in `STATUS.md`, which is generated.
Four lines per entry. The analysis belongs in the issue or the decision record; this
file carries the reasoning and the pointers.
- **Why** — the driver. The one line git cannot reconstruct later.
- **Obligates** — issues this change created elsewhere. Numbers, not prose.
- **Refs** — commits, issues, decision records.
---
+67
View File
@@ -0,0 +1,67 @@
# Project status
> Between the compass markers is generated. Edit the sources, not this.
<!-- compass:begin -->
<!-- compass:board -->
## Where this stands
uscogdata is at 0.4.0 and its public surface is settled: the query verbs, the cohort
predicates added in this release, and the provenance contract every verb returns.
The six open issues split cleanly. Two are API work carried out of the #9 review pass
and deliberately deferred there rather than fixed in that branch. Three concern the
corpus layer, and the largest of them, partition-level caching, was named the single
highest-leverage change on the remote path before being deferred. One, the
data-correction intake (#52), is a decision rather than a task: it was parked during
the 0.3.0 design, and the API announcement waits on it, because without it the corpus
cannot make the "traceable and correctable" claim that most distinguishes it from
Census's own files.
Nothing here is blocked on anything else, so the ordering is a judgement about value
rather than a dependency graph.
Compass's own files moved out of `docs/` this session. They were sitting inside
pkgdown's output directory, and `pkgdown::clean_site()` deletes every top-level entry
there except `CNAME` and `dev` — asked directly, it listed `docs/pm` and
`docs/decisions` among the 28 it would remove, with the guard that would have stopped
it satisfied by `docs/pkgdown.yml`. They are in `pm/` now. Nothing was lost: the
journal had no entries and there were no decision records yet, which made this the
cheapest moment to move. The `.gitignore` workaround that re-included two children of
an excluded `docs/` is gone with it.
## Ready to work on next
- **#34** cog_revenue() offers expenditure recipes as suggestions: scope the candidate query by category_type · `ws/api` — nothing is blocking it; something is currently wrong
- **#36** n_units_reporting is category-conditional and cannot be read as a response rate · `ws/corpus` — nothing is blocking it; owed work from an earlier change
- **#2** Extend population data to be households as an alternate spending denominator · `ws/corpus` — nothing is blocking it
- **#33** Decompose .build_suggestions() (106 lines) into named helpers · `ws/api` — nothing is blocking it
- **#52** Release 11/11: design the data-correction intake (deferred; gates the API announcement) · `ws/corpus` — nothing is blocking it
- **#64** Partition-level caching: R/cache.R is still a stub, and the remote path pays for it every session · `ws/corpus` — nothing is blocking it
## Workstreams
| Stream | Commits since | Open | Debt | Owes docs |
|---|---|---|---|---|
| Query verbs and results | 77 | 2 | 0 | no |
| Corpus, mirror, provenance | 39 | 4 | 1 | no |
| Vignettes and guides | 34 | 0 | 0 | **yes** |
## CI
![R-CMD-check](https://gitea.civilytics.org/Civilytics/uscogdata/actions/workflows/ci.yml/badge.svg?branch=main)
![Mirror to GitHub](https://gitea.civilytics.org/Civilytics/uscogdata/actions/workflows/mirror-github.yml/badge.svg?branch=main)
<details>
<summary>Dependency graph and detail</summary>
_Nothing blocks anything else, so there is no graph to draw._
- Marker: `none` (no journal entry yet)
- Commits since: 165
- Open issues: 6
</details>
<!-- compass:end -->
+44
View File
@@ -0,0 +1,44 @@
[project]
name = "uscogdata"
forge = "Civilytics/uscogdata"
# Three strands that go stale independently: what the verbs return, what the
# corpus is and how it is mounted, and how both are explained to a reader.
[[workstream]]
id = "api"
title = "Query verbs and results"
paths = [
"R/revenue.R", "R/spending.R", "R/balances.R", "R/peers.R", "R/search.R",
"R/categories.R", "R/recipes.R", "R/rollup.R", "R/explain.R", "R/basket.R",
"R/suggestions.R", "R/suppression.R", "R/complete.R", "R/cohort.R",
"R/basis.R", "R/adjust.R", "R/pagination.R",
]
docs = ["vignettes/*.Rmd", "README.md"]
[[workstream]]
id = "corpus"
title = "Corpus, mirror, provenance"
paths = [
"R/manifest.R", "R/mirror.R", "R/cache.R", "R/session.R", "R/provenance.R",
"R/coverage.R", "R/config.R", "R/views.R", "R/series_breaks.R",
"R/balance_caveats.R", "R/zzz.R", "data-raw/**", "inst/sql/**",
]
docs = ["vignettes/*.Rmd", "NEWS.md"]
[[workstream]]
id = "docs"
title = "Vignettes and guides"
paths = ["vignettes/**", "README.md", "_pkgdown.yml", "NEWS.md"]
docs = []
[roborev]
project_guidelines = [
"Every verb calls .ensure_session() first, then queries via DBI::dbGetQuery().",
"A verb's return value is always a tbl_df carrying a provenance attribute.",
"govid inputs always go through .coerce_govid_input(); it accepts a character vector or a data frame.",
"SQL has two layers: view definitions are numbered .sql files in inst/sql/ registered by .register_views(); query construction is inline sprintf() in R. Add a view as a file; build a query in R.",
"No arrow dependency -- DuckDB reads parquet natively.",
"withr is Suggests-only and must appear in tests alone.",
"Tests must pass offline against the bundled fixture; tests/testthat/setup.R sets USCOGDATA_URL for that.",
]
+12
View File
@@ -0,0 +1,12 @@
# Decisions
One file per decision, numbered and immutable. A decision that changes is superseded
by a new record, never edited in place — the old reasoning is the point.
The table below is **generated** by `compass:decide`. Do not hand-edit it.
<!-- compass:begin decisions -->
| # | Date | Decision | Status |
|---|---|---|---|
| — | — | *No decisions recorded yet.* | — |
<!-- compass:end decisions -->