105 lines
4.4 KiB
Markdown
105 lines
4.4 KiB
Markdown
# CLAUDE.md — Agent Instructions for uscogdata
|
||
|
||
## Project Overview
|
||
|
||
R package providing a curated reader API for the Civilytics US Census of
|
||
Governments finance corpus. Reads Hive-partitioned parquet + `manifest.json`
|
||
published by `cog_pipeline` via DuckDB (local path or remote URL). This is a
|
||
standalone Gitea repo, sibling to `cog_explorer/cog_pipeline/`.
|
||
|
||
**Gitea remote:** `gitea.civilytics.org/Civilytics/uscogdata`
|
||
**Full implementation plan:** `../cog_pipeline/docs/plan_phase_n_tasks.md` (Tasks 2.1–2.8 + Phase 3)
|
||
**Reader contract spec:** `../cog_pipeline/docs/reader-specification.md`
|
||
|
||
## Architecture
|
||
|
||
### Data flow
|
||
|
||
```
|
||
USCOGDATA_URL (local path or https://)
|
||
→ cog_open() fetches manifest.json, registers DuckDB views from inst/sql/
|
||
→ verbs query views via DBI
|
||
→ results carry provenance attr
|
||
```
|
||
|
||
### Key files
|
||
|
||
- `R/config.R` — `.cfg()`, `.resolve_url()`, `.uscogdata_env` (mutable state)
|
||
- `R/session.R` — `cog_open()`, `cog_close()`, `.ensure_session()`, `.coerce_govid_input()`
|
||
- `R/manifest.R` — `.fetch_or_cache_manifest()`, `.is_local_path()` (local paths bypass HTTP/cache)
|
||
- `R/views.R` — `.register_views()` (substitutes `{url}` into SQL files at `inst/sql/`)
|
||
- `inst/sql/` — 7 SQL view definitions: `long`, `spending_long`, `revenue_long`, `canonical_fips_xwalk`, `summary_categories`, `spending_annotated`, `revenue_annotated`
|
||
- `R/spending.R` / `R/revenue.R` — `cog_spending()` / `cog_revenue()` via shared `.verb_spendrev()`
|
||
- `R/rollup.R` — `cog_geographic_rollup()` (accepts named list of govids by layer)
|
||
- `R/peers.R` — `cog_find_peers()` + `cog_peer_compare()`
|
||
- `R/search.R` — `cog_gov_search()` (name pattern, state, type filters)
|
||
- `R/mirror.R` — `cog_mirror()` (local corpus copy via httr2 WebDAV)
|
||
- `R/explain.R` — `cog_explain()` (prints/returns provenance attr)
|
||
- `R/categories.R` — `cog_categories()` (queries `summary_categories` view)
|
||
- `R/adjust.R` — `.inflate()` helper using bundled CPIAUCSL in `sysdata.rda`
|
||
- `R/provenance.R` — `.build_provenance()` attached as attr to all verb results
|
||
|
||
### Corpus URL resolution
|
||
|
||
`USCOGDATA_URL` env var → `uscogdata.url` option → default placeholder in `config.R`.
|
||
Any value without `://` is treated as a local path by `.is_local_path()` and reads
|
||
`manifest.json` directly from disk (no HTTP, no TTL cache).
|
||
|
||
## Current State (2026-04-27)
|
||
|
||
**Version:** 0.1.0 (pre-release)
|
||
**Branch:** `main`, commit `d65e9fe`
|
||
**Tests:** 181 PASS / 0 FAIL / 0 SKIP
|
||
**CI:** Gitea Actions green (`.gitea/workflows/ci.yml`)
|
||
|
||
### Completed (Tasks 2.1–2.7)
|
||
|
||
All 8 exported verbs implemented and tested:
|
||
`cog_spending`, `cog_revenue`, `cog_explain`, `cog_geographic_rollup`,
|
||
`cog_find_peers`, `cog_peer_compare`, `cog_gov_search`, `cog_mirror`,
|
||
plus `cog_categories`.
|
||
|
||
Bundled fixture corpus at `inst/extdata/fixture_corpus/` (3.6 MB, years
|
||
2019+2020, all 50 states). Tests run fully offline — no credentials needed.
|
||
|
||
### Remaining to v0.1 release
|
||
|
||
1. **Task 2.8 — Docs:** roxygen `@param`/`@return`/`@examples` on all exports;
|
||
full `README.md`; `_pkgdown.yml`; `devtools::document()` + `pkgdown::build_site()`.
|
||
Vignettes can be stubbed for v0.1.
|
||
|
||
2. **Phase 3 — cog_explorer bridge:** create
|
||
`cog_explorer/examples/hello_world_uscogdata.Rmd` (installs from Gitea, runs
|
||
`cog_spending()` against live Nextcloud corpus, renders a chart). Gate N14.
|
||
|
||
3. **Release gates:** run test suite against live corpus
|
||
(`USCOGDATA_URL=<live-url> devtools::test()`), add
|
||
`^inst/extdata/fixture_corpus$` to `.Rbuildignore`, tag `v0.1.0`, cut
|
||
Gitea release.
|
||
|
||
## Testing
|
||
|
||
```r
|
||
devtools::test() # uses bundled fixture, fully offline
|
||
testthat::test_local() # same, used by CI
|
||
```
|
||
|
||
`tests/testthat/setup.R` sets `USCOGDATA_URL` to the bundled fixture automatically.
|
||
`helper-fixture.R` provides `skip_if_no_corpus()`, `fixture_corpus_path()`,
|
||
`with_fixture_corpus()`.
|
||
|
||
To run against the live corpus:
|
||
```r
|
||
Sys.setenv(USCOGDATA_URL = "<live-url-with-trailing-slash>")
|
||
devtools::test()
|
||
```
|
||
|
||
## Coding Conventions
|
||
|
||
- All verbs call `.ensure_session()` first, then query via `DBI::dbGetQuery()`
|
||
- Return value is always a `tbl_df` with a `provenance` attribute
|
||
- govid inputs always go through `.coerce_govid_input()` (accepts character or data frame)
|
||
- SQL lives in `inst/sql/` — never inline SQL strings in R files
|
||
- No arrow dependency — DuckDB reads parquet natively
|
||
- `withr` is a Suggests-only dep; only used in tests
|