Compare commits

...
Author SHA1 Message Date
jared 7274ce3bfe chore: correct the roborev exclusions and refresh the guidelines
Mirror to GitHub / mirror (push) Successful in 9s
R-CMD-check / check (push) Successful in 3m46s
roborev matches excluded_commit_patterns as substrings, and compass's cadence rule
requires `type(ws): subject (#N)`, so `chore:` never matched `chore(engine): ...`.
Adds `chore(`.

Does not add `docs(`. Those commits carry the journal entry and the board
narrative, and the plain-language rule exists to check exactly that prose -- it has
no other commit to fire on.

The guidelines were also stale: they were composed before base.md gained the
plain-language rule, and nothing re-composes them on its own. Refreshed, which is
what put that rule in this repository for the first time.
2026-08-23 23:57:05 -04:00
jared 24e86ed598 chore: retire the three plans whose work has shipped
Mirror to GitHub / mirror (push) Successful in 6s
R-CMD-check / check (push) Successful in 3m21s
186 unticked checkboxes across three plans, none of them outstanding work.
Superpowers-style plans are execution transcripts: nobody ticks the boxes, and
the plan is abandoned at the point the work is done. Left in place they are
indistinguishable from a live backlog -- the old compass retrofit rule, "open
checkboxes become issues", would have filed 186 issues for finished work.

Evidence, from scripts/plans.py plus a check by hand:

  2026-08-04-partial-coverage-signposting  NEWS: "Coverage signposting ..."
  2026-08-08-public-release                NEWS: "First public release."
  2026-04-29-per-year-population-denominator
      no NEWS line matched and 19 of the 21 files it names exist, so the
      classifier called it ambiguous. Confirmed shipped by hand:
      33c0274 docs(news): per-year population denominators, plus the feat
      commits behind it.

specs/ is untouched. A spec explains why the design is what it is and stays
useful; a plan is scaffolding, and once the building stands it is in the way.
All three remain in git history.
2026-08-23 23:32:11 -04:00
jared 1ec20174b7 chore: move compass out of docs/, which pkgdown deletes
Mirror to GitHub / mirror (push) Successful in 7s
R-CMD-check / check (push) Successful in 3m28s
compass.toml, JOURNAL.md, STATUS.md and decisions/ were sitting inside pkgdown's
output directory. Asked directly, pkgdown listed docs/pm and docs/decisions among
the 28 top-level entries clean_site() would delete, and the guard that would have
refused -- check_dest_is_pkgdown() -- was satisfied by docs/pkgdown.yml. After the
move it lists 26 and none of them are compass's.

Nothing was lost. The journal had no entries and there were no decision records
yet, so this was the cheapest moment to move.

The .gitignore workaround goes with it. Re-including two children of an excluded
docs/ forced the rule to be written as /docs/* plus two negations, which changed
the anchoring and made the fixture corpus's own docs/ need a separate rule. A bare
docs/ matches at any depth again, so both are unnecessary.

.Rbuildignore gains ^pm$ -- R CMD check flags a non-standard top-level directory.

Compass reads both layouts, so this repository worked either way; the point is
that docs/ is a directory another tool empties.
2026-08-23 23:26:33 -04:00
Jared Knowles 2e317d3a0f chore: adopt compass for project tracking
Mirror to GitHub / mirror (push) Successful in 10s
R-CMD-check / check (push) Successful in 3m55s
Three workstreams -- the query verbs, the corpus and how it is mounted, and the
docs that explain both. Each can go stale independently, which is what the
workstream boundary is for: a change to a verb obligates the vignettes, a change
to the corpus obligates NEWS.

All six open issues now carry ws/ and type/ labels, applied additively so #36
kept its existing kodor/, severity/, south-guide and verdict/ labels. The
generated board is pinned as issue #68.

.roborev.toml carries review guidelines composed from a shared baseline, the R
package overlay, and this project's own conventions read out of CLAUDE.md: the
ensure_session-then-dbGetQuery order, the tbl_df-with-provenance return
contract, coerce_govid_input at the boundary, the two SQL layers, no arrow, and
withr as tests-only.

The .gitignore change is load-bearing. pkgdown output made docs/ ignored, which
would have left every compass file untracked and unable to travel to another
machine. Git cannot re-include anything beneath an excluded directory, so the
rule had to list children instead. That forced an anchoring change: a bare
"docs/" matches at any depth, "/docs/*" only at the root, so the fixture corpus
docs directory needed an explicit rule to stay excluded as before.
2026-08-23 16:17:20 -04:00
jared f7c606984f Merge pull request 'ci: mirror canonical tag objects, not the lightweight refs checkout builds' (#67) from ci/mirror-canonical-tags into main
Mirror to GitHub / mirror (push) Successful in 9s
R-CMD-check / check (push) Successful in 3m14s
Reviewed-on: #67
2026-08-11 10:03:43 -04:00
jared 9617b86a26 ci: mirror canonical tag objects, not the lightweight refs checkout builds
R-CMD-check / check (push) Successful in 3m45s
R-CMD-check / check (pull_request) Successful in 3m29s
v0.4.0 failed to mirror, and it would have failed at every future release.

On a tag-triggered run, checkout materializes refs/tags/<tag> as a LIGHTWEIGHT
tag at the commit SHA; the annotated tag object Gitea holds is never fetched.
Run 2070 therefore pushed a lightweight v0.4.0 to GitHub and reported success.
Run 2076, on main with fetch-depth 0, did fetch the real annotated object and
was rejected with "already exists" trying to correct it -- git will not clobber
an existing tag. Gitea had e138eeb (annotated), GitHub had d2caa6d (the commit).

Re-fetch canonical tag objects from Gitea before pushing. --force rewrites LOCAL
tag refs only; it is not a force push and does not weaken the non-force
guarantee on main. It is required: without it the fetch is rejected with "would
clobber existing tag" and the lightweight ref survives to be mirrored again.

Verified against a scratch clone -- lightweight d2caa6d becomes annotated
e138eeb, peeling back to the same commit; without --force the tag is unchanged.

The GitHub tag was repaired by hand out of band, so the two remotes already
agree; this stops it recurring.
2026-08-11 09:57:14 -04:00
jared 224e5d0530 Merge pull request 'chore: CI badge, mirror PR explainer, and the tag-pin release step (#47)' (#66) from chore/release-47-badges-mirror-pr into main
Mirror to GitHub / mirror (push) Failing after 11s
R-CMD-check / check (push) Successful in 3m15s
Reviewed-on: #66
2026-08-11 09:44:33 -04:00
11 changed files with 217 additions and 3550 deletions
+1
View File
@@ -3,6 +3,7 @@
^\.Rproj\.user$
^_pkgdown\.yml$
^docs$
^pm$
^Meta$
^doc$
^pkgdown$
+12
View File
@@ -39,5 +39,17 @@ jobs:
echo "PAT_GH is unset -- add it under Settings > Actions > Secrets." >&2
exit 1
fi
# On a tag-triggered run, checkout materializes refs/tags/<tag> as a
# LIGHTWEIGHT tag at the commit SHA -- the annotated tag object Gitea
# holds is never fetched. Mirroring that strips the annotation, and the
# NEXT run on main (which does fetch the real object) is then rejected
# with "already exists" trying to correct it, because git will not
# clobber an existing tag. That is why v0.4.0 failed to mirror.
#
# Re-fetch canonical tag objects from Gitea first. --force here rewrites
# LOCAL tag refs only; it is not a force push and does not weaken the
# non-force guarantee on main documented above.
git fetch --tags --force origin
git push "https://x-access-token:${PAT_GH}@github.com/civilytics/uscogdata.git" \
HEAD:refs/heads/main --tags
+7
View File
@@ -4,6 +4,10 @@
.Ruserdata
*.Rproj
inst/doc
# pkgdown output. Compass used to keep its files in docs/pm/ and
# docs/decisions/, which forced this to be written as children with two
# re-includes -- git cannot re-include anything beneath an excluded directory.
# Compass lives in pm/ now, so the whole directory can be excluded again.
docs/
/doc/
/Meta/
@@ -12,3 +16,6 @@ docs/
# SDD working artifacts (ledger, briefs, review packages) — plans/ stays tracked
.superpowers/sdd/
.compass-cache/
# roborev snapshots
/.roborev/
+60
View File
@@ -0,0 +1,60 @@
# roborev configuration, initialised by compass.
# Reviews are queued to a background daemon -- they never block a commit.
post_commit_review = 'commit'
excluded_commit_patterns = ['WIP', 'chore:', 'chore(', 'docs:', 'Merge ']
review_guidelines = '''
# --- compass:begin (generated -- edit the sources, not this) ---
- Prefer returning new values to mutating arguments in place. A function that edits
its caller's object is a bug waiting for a second caller.
- Validate at system boundaries -- user input, API responses, file contents, config.
Fail fast with a message naming the field and the file.
- Never swallow an error. Handle it or let it propagate; a bare catch that continues
is worse than a crash.
- No hardcoded secrets, tokens, or credentials, and no secrets in log output or error
messages.
- Parameterise every query. String-built SQL is a defect even when the input looks safe.
- Keep functions under roughly 50 lines and files under roughly 400. Flag nesting
deeper than four levels.
- No magic numbers or hardcoded paths -- name them as constants or read them from config.
- New behaviour needs a test. A bug fix needs a test that fails without the fix.
- Prose a person reads -- an issue title or body, a journal entry, a decision record,
the narrative on the status board -- names the action or the thing, not the shape of
the machinery. Flag "gate", "seam", "surface area", "load-bearing", "first-class",
"primitive", "blast radius". A project's own defined vocabulary is not the target.
- Use the native pipe `|>`, not magrittr `%>%`.
- snake_case for objects and functions; UPPER_SNAKE for constants. Never use `.` as a
word separator in a function name -- it collides with S3 dispatch.
- Validate arguments at the top of exported functions with `stopifnot()` or an explicit
check, and say which argument was wrong.
- Never `setDT()`, `set()`, or otherwise modify by reference a data.table the caller
still owns. `as.data.table()` copies; use it.
- Prefer `vapply()` to `sapply()` -- `sapply()` silently returns a list when the type
varies, which turns a type error into a downstream mystery.
- Use `seq_len(n)` / `seq_along(x)`, never `1:n`, which iterates backwards when n is 0.
- Compare strings with `==` only after checking for NA; use `identical()` for scalars
where NA would be wrong.
- Do not call `library()` inside package or module files; attach packages in scripts and
test helpers only.
- Namespace-qualify calls into other packages (`stats::sd`) in code that is sourced.
- Every exported function needs roxygen with `@param` for each argument (type, meaning,
and why the default is what it is) and `@return`. Add `@examples` for exported API.
- Declare dependencies in DESCRIPTION. Prefer base R or an existing dependency over
adding a new one; a package with zero hard deps is worth keeping that way.
- Signal errors with `stop()` carrying a condition class, so callers can catch the kind
rather than matching on message text.
- Keep internals internal. Export only what a user needs; an accidentally exported
helper becomes an API you have to keep.
- Tests use testthat edition 3. Each test is self-sufficient -- no reliance on state
left by an earlier test or on a fixture built elsewhere in the file.
- Prefer duplication in tests over a helper that hides what is being asserted.
- Every verb calls .ensure_session() first, then queries via DBI::dbGetQuery().
- A verb's return value is always a tbl_df carrying a provenance attribute.
- govid inputs always go through .coerce_govid_input(); it accepts a character vector or a data frame.
- SQL has two layers: view definitions are numbered .sql files in inst/sql/ registered by .register_views(); query construction is inline sprintf() in R. Add a view as a file; build a query in R.
- No arrow dependency -- DuckDB reads parquet natively.
- withr is Suggests-only and must appear in tests alone.
- Tests must pass offline against the bundled fixture; tests/testthat/setup.R sets USCOGDATA_URL for that.
# --- compass:end ---
'''
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+14
View File
@@ -0,0 +1,14 @@
# Project journal
Append-only, newest first. **Entries are never edited** — the value of this file is
that it records what was believed at the time, including the parts that turned out
wrong. Where things stand *today* is in `STATUS.md`, which is generated.
Four lines per entry. The analysis belongs in the issue or the decision record; this
file carries the reasoning and the pointers.
- **Why** — the driver. The one line git cannot reconstruct later.
- **Obligates** — issues this change created elsewhere. Numbers, not prose.
- **Refs** — commits, issues, decision records.
---
+67
View File
@@ -0,0 +1,67 @@
# Project status
> Between the compass markers is generated. Edit the sources, not this.
<!-- compass:begin -->
<!-- compass:board -->
## Where this stands
uscogdata is at 0.4.0 and its public surface is settled: the query verbs, the cohort
predicates added in this release, and the provenance contract every verb returns.
The six open issues split cleanly. Two are API work carried out of the #9 review pass
and deliberately deferred there rather than fixed in that branch. Three concern the
corpus layer, and the largest of them, partition-level caching, was named the single
highest-leverage change on the remote path before being deferred. One, the
data-correction intake (#52), is a decision rather than a task: it was parked during
the 0.3.0 design, and the API announcement waits on it, because without it the corpus
cannot make the "traceable and correctable" claim that most distinguishes it from
Census's own files.
Nothing here is blocked on anything else, so the ordering is a judgement about value
rather than a dependency graph.
Compass's own files moved out of `docs/` this session. They were sitting inside
pkgdown's output directory, and `pkgdown::clean_site()` deletes every top-level entry
there except `CNAME` and `dev` — asked directly, it listed `docs/pm` and
`docs/decisions` among the 28 it would remove, with the guard that would have stopped
it satisfied by `docs/pkgdown.yml`. They are in `pm/` now. Nothing was lost: the
journal had no entries and there were no decision records yet, which made this the
cheapest moment to move. The `.gitignore` workaround that re-included two children of
an excluded `docs/` is gone with it.
## Ready to work on next
- **#34** cog_revenue() offers expenditure recipes as suggestions: scope the candidate query by category_type · `ws/api` — nothing is blocking it; something is currently wrong
- **#36** n_units_reporting is category-conditional and cannot be read as a response rate · `ws/corpus` — nothing is blocking it; owed work from an earlier change
- **#2** Extend population data to be households as an alternate spending denominator · `ws/corpus` — nothing is blocking it
- **#33** Decompose .build_suggestions() (106 lines) into named helpers · `ws/api` — nothing is blocking it
- **#52** Release 11/11: design the data-correction intake (deferred; gates the API announcement) · `ws/corpus` — nothing is blocking it
- **#64** Partition-level caching: R/cache.R is still a stub, and the remote path pays for it every session · `ws/corpus` — nothing is blocking it
## Workstreams
| Stream | Commits since | Open | Debt | Owes docs |
|---|---|---|---|---|
| Query verbs and results | 77 | 2 | 0 | no |
| Corpus, mirror, provenance | 39 | 4 | 1 | no |
| Vignettes and guides | 34 | 0 | 0 | **yes** |
## CI
![R-CMD-check](https://gitea.civilytics.org/Civilytics/uscogdata/actions/workflows/ci.yml/badge.svg?branch=main)
![Mirror to GitHub](https://gitea.civilytics.org/Civilytics/uscogdata/actions/workflows/mirror-github.yml/badge.svg?branch=main)
<details>
<summary>Dependency graph and detail</summary>
_Nothing blocks anything else, so there is no graph to draw._
- Marker: `none` (no journal entry yet)
- Commits since: 165
- Open issues: 6
</details>
<!-- compass:end -->
+44
View File
@@ -0,0 +1,44 @@
[project]
name = "uscogdata"
forge = "Civilytics/uscogdata"
# Three strands that go stale independently: what the verbs return, what the
# corpus is and how it is mounted, and how both are explained to a reader.
[[workstream]]
id = "api"
title = "Query verbs and results"
paths = [
"R/revenue.R", "R/spending.R", "R/balances.R", "R/peers.R", "R/search.R",
"R/categories.R", "R/recipes.R", "R/rollup.R", "R/explain.R", "R/basket.R",
"R/suggestions.R", "R/suppression.R", "R/complete.R", "R/cohort.R",
"R/basis.R", "R/adjust.R", "R/pagination.R",
]
docs = ["vignettes/*.Rmd", "README.md"]
[[workstream]]
id = "corpus"
title = "Corpus, mirror, provenance"
paths = [
"R/manifest.R", "R/mirror.R", "R/cache.R", "R/session.R", "R/provenance.R",
"R/coverage.R", "R/config.R", "R/views.R", "R/series_breaks.R",
"R/balance_caveats.R", "R/zzz.R", "data-raw/**", "inst/sql/**",
]
docs = ["vignettes/*.Rmd", "NEWS.md"]
[[workstream]]
id = "docs"
title = "Vignettes and guides"
paths = ["vignettes/**", "README.md", "_pkgdown.yml", "NEWS.md"]
docs = []
[roborev]
project_guidelines = [
"Every verb calls .ensure_session() first, then queries via DBI::dbGetQuery().",
"A verb's return value is always a tbl_df carrying a provenance attribute.",
"govid inputs always go through .coerce_govid_input(); it accepts a character vector or a data frame.",
"SQL has two layers: view definitions are numbered .sql files in inst/sql/ registered by .register_views(); query construction is inline sprintf() in R. Add a view as a file; build a query in R.",
"No arrow dependency -- DuckDB reads parquet natively.",
"withr is Suggests-only and must appear in tests alone.",
"Tests must pass offline against the bundled fixture; tests/testthat/setup.R sets USCOGDATA_URL for that.",
]
+12
View File
@@ -0,0 +1,12 @@
# Decisions
One file per decision, numbered and immutable. A decision that changes is superseded
by a new record, never edited in place — the old reasoning is the point.
The table below is **generated** by `compass:decide`. Do not hand-edit it.
<!-- compass:begin decisions -->
| # | Date | Decision | Status |
|---|---|---|---|
| — | — | *No decisions recorded yet.* | — |
<!-- compass:end decisions -->