jared
c375c55da7
The fixture predated three shipped corpus changes at once: no J rows in summary_categories (it was built before the crosswalk completion), no representation.parquet or code_set.parquet, and a still-dense wide era. Every test in this package and in cog-api runs against it, so both suites were green against a corpus that no longer exists. This is #18's stated prerequisite; it proves nothing about production until it lands. Regenerated from the publish tree at pipeline_commit 83f9715 (schema v6, built 2026-07-29). FY2011 goes from 2,864,212 rows to 496,004 -- 82.7% of the old partition was explicit zeros -- and the fixture now ships all ten publish-tree metadata tables rather than six. The generator's file list is a single constant now, so the copy step and the manifest step cannot drift. Three test repairs, each a real consequence of sparsification rather than a number to bump: test-categories.R "assistance" joined the spending subtype vocabulary with the J-prefix codes. test-spending.R The harmonization block counts rows that exist. Broward's E21/F21/G21 were zero-pads and are gone, so the anchor moves to FL state, whose three NA-mapped rows carry $2.83B -- the amount accounting was previously asserted only against 0 and could not have caught a bug. Broward keeps a test of its own, now asserting the zero-pads are absent. test-expenditure-concept.R Coverage-gap suggestions are presence-based. AL state's only FY2011 B47 cell was an explicit zero, so ig_federal_b47_wide stopped being a candidate there; FL state carries a real amount, so the counterpart guard is exercised against a suggestion that fires. test-fixture-vintage.R pins the structural facts that separate this vintage from its predecessor -- the ten metadata tables, the dense/sparse representation contract, zero explicit zeros in FY2011, code_set coverage, and J19's category. Checked against the old fixture: FY2011 carried 2,368,208 explicit zeros, so the assertion discriminates rather than merely passing. Suites: uscogdata 594 pass / 0 fail / 6 skip (was 576/0/6). cog-api 357 pass / 0 fail / 8 skip against the regenerated fixture, unchanged from its baseline.
uscogdata
Curated R reader for the Civilytics US Census of Governments finance corpus.
Provides unit-level financial profiles, geographic rollups, and peer comparisons with auditable provenance and built-in cross-vintage correctness. Reads the published corpus (Hive-partitioned parquet + manifest.json) directly from Nextcloud via DuckDB httpfs — no local bulk downloads required.
Status
Under active development (Phase 2 of the cog_pipeline project). See
../cog_pipeline/docs/reader-specification.md for the reader contract this
package implements.
Installation
# pak::pkg_install("gitea.civilytics.org/Civilytics/uscogdata")
Configuration
USCOGDATA_URL— corpus root URL (public Nextcloud share, trailing slash)USCOGDATA_CACHE_DIR— optional override for the manifest cache directoryUSCOGDATA_MANIFEST_TTL_SECS— optional manifest re-fetch TTL (default 3600)
Direct vs Total spending
cog_spending(..., expenditure_concept = c("direct", "total")) controls
whose spending a result counts. "direct" (the default) is a government's
own current operations, capital outlay, and other direct spending. "total"
additionally adds in the intergovernmental legs — money it hands to other
governments to spend on its behalf — which is meaningful for describing one
government's own budget over time, but double-counts when summed across
governments (a state's payment to a county is the same dollar the county
reports as its own direct spending).
Rule of thumb: any figure that spans more than one government uses
direct. cog_geographic_rollup() and cog_peer_compare() enforce this
by refusing expenditure_concept = "total". See
vignette("total-spending", package = "uscogdata") for the full
explanation with worked examples.
Developer notes
Testing
The package ships a bundled fixture corpus at inst/extdata/fixture_corpus/ —
a 15 MB four-year slice (2011, 2012, 2019, 2020) of the full corpus covering
all 50 states. tests/testthat/setup.R automatically points USCOGDATA_URL
at this fixture, so the full test suite runs offline with no network
dependency:
devtools::test() # uses bundled fixture, no credentials required
Releasing against the live corpus
Before cutting a release, run the test suite against the published corpus to catch any drift between the fixture and the real data:
Sys.setenv(USCOGDATA_URL = "<published-corpus-url-with-trailing-slash>")
devtools::test()
When the live-corpus run is clean, strip the fixture from the built package by
adding this line to .Rbuildignore:
^inst/extdata/fixture_corpus$
The test suite is URL-agnostic — setup.R falls back to USCOGDATA_URL when
the bundled fixture is absent, so no test code changes are needed for the
release run or after stripping the fixture.