Compare commits

...
Author SHA1 Message Date
jared 4bedf857e9 docs: implementation plan for the uscogdata 0.1.0 public release
R-CMD-check / check (push) Successful in 4m23s
Ten tasks, 58 steps, TDD throughout. Tasks 1-3 fix the P0 (manifest
enumeration, working default URL, and the live-corpus test whose absence
let the defect survive); 4-7 are metadata and packaging; 8-10 rewrite
README, NEWS and CONTRIBUTING.

Distribution mechanics stay out of scope -- r-universe publishes check
results on registration, so it comes after final verification is green.
2026-08-08 16:54:18 -04:00
jared a9de5ba5ab docs: design spec for the uscogdata 0.1.0 public release
Covers the P0 finding that the package cannot read the corpus remotely at
all -- no working default URL, and Hive globs are unsupported over generic
HTTP by DuckDB 1.5.5. Fix is manifest-driven file enumeration (measured:
46,148,034 rows over plain https, identical to the hf:// glob) plus a
working public default.

Also: seven release-readiness fixes, a README restructured for a stranger,
NEWS rewritten as an initial release rather than a pre-release churn log,
and the Gitea-canonical/GitHub-mirror/r-universe distribution mechanics.
2026-08-08 16:46:39 -04:00
jared d0b4bae3cc docs: plan for partial-coverage signposting (#9) 2026-08-04 21:31:15 -04:00
jared e067a5930f test: accept schema_version 7, and keep the upper bound enforced
R-CMD-check / check (pull_request) Successful in 6m3s
R-CMD-check / check (push) Has been cancelled
The merged schema-v7 fix (b59b79b) widened .validate_schema()'s allow-list but
left this test asserting that 7 is REJECTED, so main went red. CI had been
hanging on the apt step before ever reaching the suite, which is why the
failure only surfaced once the HTTPS fix let the job get that far.

Flips 7L from expect_error to expect_silent, and ADDS an 8L rejection case.
That second part is the point: simply deleting the 7L expectation would have
left the test unable to prove any upper bound is enforced at all, so a future
v8 corpus with a genuinely breaking change would pass validation silently. The
test should assert the boundary moved, not that it disappeared.

Suite: 796 PASS, 0 FAIL, 0 WARN, 0 SKIP.
2026-08-04 12:07:59 -04:00
jared 342debaefa ci: fetch apt indexes over HTTPS so the install step stops hanging
R-CMD-check / check (push) Has been cancelled
R-CMD-check / check (pull_request) Has been cancelled
The "Install system libraries" step was stalling indefinitely. It was not
deadlocked on a config prompt and not slow-but-progressing: measured inside the
live runner container, /var/cache/apt/archives stayed at 0 .deb files after 3+
minutes, with apt's http workers parked in S state waiting on the network.

Root cause is the http:// mirror path being pathologically slow from this
runner, not broken. Measured 2026-08-04 from inside the CI container, same
index file, back to back:

  http://archive.ubuntu.com/ubuntu/dists/noble/Release   20.1s
  https://archive.ubuntu.com/ubuntu/dists/noble/Release    3.1s

apt fetches many indexes serially, so ~20s apiece compounds into what looks
like a hang. Rewriting the deb822 sources to https makes the step complete.

Verified before committing, in the running CI container (rocker/r-ver:4.4):
- ca-certificates present and apt 2.8.3 ships the https method built in, so
  nothing has to be installed over http first to bootstrap TLS
- the sed rewrites both URIs (archive + security); the only remaining http://
  is an inert comment line
- '#' is used as the sed delimiter deliberately: '|' collides with the
  alternation and fails with "unknown option to `s'"
- the regex survives YAML block-scalar parsing with backslashes intact

`|| true` guards each sed because the step runs under `sh -e`, so a
missing-sources-file on some other base image must not kill the job.
2026-08-04 11:57:49 -04:00
jared b59b79b2d5 Merge pull request 'fix: accept corpus schema_version 7 (#80)' (#30) from fix/schema-v7 into main
R-CMD-check / check (push) Has been cancelled
Reviewed-on: #30
2026-08-04 11:38:45 -04:00
jared 5668d6b102 fix: accept corpus schema_version 7 (#80)
R-CMD-check / check (pull_request) Has been cancelled
R-CMD-check / check (push) Has been cancelled
2026-08-04 11:22:28 -04:00
jared 0a6d878a36 Merge pull request 'fix: cog_categories() surfaces balance subtypes and accepts type = "balance"' (#29) from fix/cog-categories-balance-subtype into main
R-CMD-check / check (push) Successful in 3m28s
Reviewed-on: #29
2026-08-03 12:20:10 -04:00
jared da726a61f6 fix: cog_categories() surfaces balance subtypes and accepts type = "balance"
R-CMD-check / check (pull_request) Successful in 3m39s
R-CMD-check / check (push) Successful in 3m39s
The balance work added category_type = "balance" rows to the corpus and
cog_balances() to read them, but left cog_categories() -- the discovery
surface -- unable to describe them:

- subtype COALESCEd only spend_subtype and revenue_subtype, so every balance
  row came back with subtype = NA
- type rejected "balance", so there was no way to ask for the holdings
  taxonomy at all

Both matter downstream: cog-api derives its subtype vocabulary from
cog_categories(), so an NA subtype becomes an unusable API parameter. Found
while implementing cog-api#26.

Note cog_balances() itself still takes no subtype argument -- for holdings
category is a strict coarsening of balance_subtype -- but the value belongs
in the discovery surface regardless.

Tests read the expected subtype set independently from the crosswalk parquet
rather than from the function under test.
2026-08-03 12:02:26 -04:00
jared 03c313b46d Merge pull request 'feat: cog_balances(), a reader surface for cash and security holdings (#25)' (#28) from feat/cog-balances-25 into main
R-CMD-check / check (push) Successful in 3m18s
Reviewed-on: #28
2026-08-03 11:52:13 -04:00
jared 2c532bde19 docs: record the two balance_caveats contract facts cog-api#26 must carry
R-CMD-check / check (push) Successful in 3m38s
R-CMD-check / check (pull_request) Successful in 3m35s
Both were settled during implementation and are easy to get wrong from
outside the package:

- coverage_window is corpus-scoped, not result-scoped. It reports the observed
  year extent of every balance subtype, not only those a query returned. The
  sibling field `truncated` is the result-scoped one.
- balance_caveats is present only on cog_balances() results; an API layer that
  assumes it is universal will read NULL from the money verbs.
2026-08-03 11:45:43 -04:00
jared a9e80858d4 docs: correct coverage_window scope and the stale CLAUDE.md Current State block (#25)
F-7: inst/schemas/provenance-v1.json described coverage_window as mapping
each *observed* balance_subtype, but the query at R/balance_caveats.R has
no predicate tied to the query's codes and always returns every subtype in
the mounted corpus. Took option (b) of the two the review offered -- change
the doc, not the code. Reporting all windows is the better product
behaviour (it answers 'is there a family I missed?'), it is what cog-api#26
already forwards verbatim, and option (a) would make the block empty for a
0-row result. Reworded to say the windows are corpus-wide and that
'truncated' is the query-scoped field. Pinned by a new test either way.

F-10: the 'Current State' block was self-contradictory after a partial
update -- headed 2026-04-27, claiming branch main @ d65e9fe, with a
2026-08-03 test count measured on feat/cog-balances-25 underneath it, and
listing README.md / _pkgdown.yml as outstanding when both exist and
_pkgdown.yml was edited by this branch. All numbers below re-measured on
the final tree after every other fix in this wave, not before:
788 tests (testthat::test_local()), 14 exports (NAMESPACE), 14 man/*.Rd,
2 vignettes, no docs/ (pkgdown::build_site() genuinely still outstanding,
as is the .Rbuildignore fixture entry -- both kept in the list).

The related deferred README.md item is closed with no change, per the
review's ruling: README.md enumerates no verbs at all, so naming
cog_balances would make it the only non-cog_spending verb mentioned.
2026-08-03 11:37:15 -04:00
jared fde62eb6cc test(balances): pin the behaviours the final review found untested or weakly asserted (#25)
Findings F-1..F-8. Every assertion below was verified to FAIL before its
fix (or under mutation, where the behaviour already worked) and pass after.

F-3: 'an unknown recipe id is rejected' used a bare expect_error(). Deleting
.validate_recipe_id() leaves .recipe_components() returning 0 rows and
comps$label[[1]] throwing 'subscript out of bounds' -- still an error, so
the test passed on the regression while the user lost the curated message.
Now asserts class = 'uscogdata_unknown_recipe'. Mutation-checked.

F-4: no test ever set per_capita and adjust_to_year together, so the
load-bearing ordering comment at R/balances.R was unverified. Reversing
those two calls silently drops amt_per_capita_real (.attach_real_dollars()
no-ops when amt_per_capita_nominal does not exist yet). New test asserts
presence AND that the per-capita column is deflated by the same factor as
the level column; mutation-checked by reversing the order (2 failures).

F-5: the spec's 'Gating' requirement had no test -- nothing ever called
cog_balances() on a corpus without balance_subtype. Extended the existing
with_corpus_missing_balance_subtype() block to assert class =
'uscogdata_no_balance_support'; mutation-checked by dropping the guard.

F-1/F-2: added mutual-exclusivity and four-argument validation tests, each
pinned to the message or class (all four inputs already produced *some*
error or *some* quiet wrong answer, so bare expect_error() was useless
here). Plus an ordering guard: a data-frame govid must still work, which
is what fails if validation is put before .coerce_govid_input().

F-6: asserts on the RENDERED cog_explain() text (both streams -- cli
routes through conditions that land on stderr), with a negative case
proving money-verb output is unaffected and that the capture is not vacuous.

F-7: pins that coverage_window is corpus-scoped while truncated is
query-scoped; mutation-checked by scoping the windows to observed subtypes.

F-8: pins the memo slot is populated on first call and cleared by
cog_close().
2026-08-03 11:36:32 -04:00
jared 22c2478634 fix(balances): validate the full signature, surface caveats in cog_explain, memoise coverage windows (#25)
Final-review findings F-1, F-2, F-6, F-8 (plus the F-9 @return reword,
which shares R/balances.R).

F-2: .validate_balance_inputs() checked 2 of cog_balances()' 7 arguments.
years = integer(0) leaked a raw DuckDB 'Parser Error ... AND year IN ()'
with the generated SQL echoed back; govid = character(0) and a non-character
category returned 0 rows with no error at all; recipe = c("a","b") threw
'the condition has length > 1' from inside .validate_recipe_id(). Replaced
with a call to the money verbs' own .validate_verb_inputs() (R/spending.R),
which validates the exact superset needed. Deleted the local copy rather
than extending it -- two validators is how they drift. Placed AFTER
.coerce_govid_input(), because .validate_verb_inputs() asserts
is.character(govid) and a data-frame govid is not unwrapped before that.
This is helper reuse of the same kind as .build_verb_sql()/.attach_per_capita();
the verb still does NOT route through .verb_spendrev().

F-1: falls out of F-2 for free -- the recipe/category mutual-exclusivity
guard lives inside .validate_verb_inputs(). Previously recipe silently
discarded category AND overwrote provenance$category with the recipe label,
so a caller asking for Fund Balances got X40/Z77 insurance-trust holdings
with no trace of the dropped filter.

F-6: cog_explain() rendered every provenance caveat block except
balance_caveats. Since .emit_balance_caveats() fires at most once per
session -- and is routinely consumed by a suppressMessages() call or an
unread knitr chunk -- cog_explain() is the only surface left for a caller
who deliberately audits the result. Added a 'Holdings caveats' section
guarded on !is.null(prov$balance_caveats). Also relabels the cosmetic
'Concept: NA' line on balance results as 'not applicable (holdings are a
stock, not a flow)'.

F-8: the coverage-window query has no govid and no year predicate -- its
answer depends only on the mounted corpus -- yet it scanned all of
balance_long on every call (35% of verb runtime on the fixture, and a
per-request throughput ceiling for cog-api#26). Memoised in
.uscogdata_env$balance_coverage_windows, invalidated by cog_close(), the
same pattern as .uscogdata_env$manifest.
2026-08-03 11:36:18 -04:00
jared 225cd60968 docs: fix stale test count and phantom notes column in cog_balances docs (#25)
Re-measured CLAUDE.md's test count on the final tree (764, not 763 --
the earlier number predated the balance_caveats schema test). Removed
notes from cog_balances()'s @return block: it was copied from
cog_spending()'s @return style without checking cog_balances() never
calls .verb_spendrev(), the only place that sets notes. Verified the
remaining documented columns against colnames() observed across every
argument combination (bare, per_capita, adjust_to_year, both, recipe,
category filter).
2026-08-03 11:10:46 -04:00
jared b03f095e49 docs: document cog_balances() and correct stale CLAUDE.md claims (#25)
Adds the NEWS entry, a Financial data pkgdown reference section (none
existed for cog_spending/cog_revenue), and corrects CLAUDE.md's SQL-layer
claim, view count, test count and fixture-year description against
measured values. Also documents balance_caveats in
inst/schemas/provenance-v1.json (test-first: added a schema-documentation
test to test-balances.R, confirmed it failed, then fixed the schema) and
fleshes out cog_balances()'s @return roxygen to enumerate its conditional
columns, regenerating man/cog_balances.Rd.
2026-08-03 11:02:16 -04:00
jared 724b6bd58b docs: disambiguate live-corpus vs fixture year claim in balances test comment (#25) 2026-08-03 10:50:46 -04:00
jared 82e4face4e feat: balance_caveats provenance + once-per-session disclosure (#25) 2026-08-03 10:48:11 -04:00
jared 6c5bdb3048 test: clarify why 2002 must stay in the SB195 recipe test's year vector 2026-08-03 10:40:23 -04:00
jared b8189aeb7f docs: caveat 4 needs a year span crossing FY2002, not just a recipe query
Task 4's implementer found that SB195 does not surface for a
recipe query spanning only 2011-2012. .build_series_break_refs() matches
break_year BETWEEN min(years) AND max(years), and SB195's break_year is 2002.

That is correct behaviour rather than a gap: a series lying entirely after the
book -> market change sits on one consistent basis, so disclosing a break it
never crosses would be noise. .build_corpus_break_refs() applies the same rule
deliberately.

The spec's caveat table overclaimed by omitting the span condition. Corrected.
2026-08-03 10:39:09 -04:00
jared 90d2e6019e feat: recipe= bridges the wide-era holdings series (#25) 2026-08-03 10:37:54 -04:00
jared 769164c824 feat: per_capita and adjust_to_year for cog_balances() (#25) 2026-08-03 10:24:59 -04:00
jared de3a58d105 fix: attach govids_found/govids_missing to cog_balances() provenance
Mirrors R/spending.R:465-466 -- .check_govids_in_scope()'s return was
previously captured only for its message side effect. Also drops a
redundant duplicate assertion in the flow-code guard test.
2026-08-03 10:19:32 -04:00
jared cdb574d3d0 test: drop arrow dependency from cog_balances tests, use direct DuckDB reads
Also add explicit non-empty assertion to the flow-code guard test so it
cannot pass vacuously on a zero-row result.
2026-08-03 10:12:57 -04:00
jared a281a9621f feat: cog_balances() core verb (#25) 2026-08-03 10:01:40 -04:00
jared 825ac394f2 test: replace vacuous is_aggregate assertion with synthetic-parquet coverage
The bundled fixture has no balance item_code with is_aggregate = TRUE, so
asserting COUNT(*) FROM balance_long WHERE is_aggregate = 0 passed whether
or not the view's AND NOT is_aggregate predicate existed. Follows the
synthetic hive-partitioned parquet pattern already used for the 22-/23-
and 24-/25- view predicates in test-views.R: reads the real
inst/sql/26-balance_long.sql text off disk and executes it against a
synthetic corpus containing both an aggregate and non-aggregate row under
a real balance item_code (W01).
2026-08-03 09:54:53 -04:00
jared d09bfd6aef feat: register balance_long / balance_annotated behind a column gate (#25) 2026-08-03 09:46:36 -04:00
jared a11e29a0e0 docs: use Wisconsin state govt (550000227544) as the cog_balances test government
Standardises on the identifier other agents use for state governments, which
is stable across corpus vintages and is the same id used against the live API.

Verified in the bundled fixture, and it is strictly better coverage than the
previous pick: Wisconsin reaches four of the five balance subtypes (adds
workers_comp_trust via Y21) and carries BOTH wide->modern recipe bridges
(X40->Z77 and X41->Z78), so a second recipe test is added. Y61
(other_insurance_trust) is absent for Wisconsin; no test depends on it.

Also notes not to assert on gov_name -- the fixture carries both "WISCONSIN"
and "WISCONSIN STATE GOVT" and the verb COALESCEs them.
2026-08-03 09:42:47 -04:00
jared 7ac4dc6882 docs: implementation plan for cog_balances() (#25)
Six TDD tasks: the two views + registration gate, the core verb, per_capita
and adjust_to_year, recipe=, balance_caveats provenance, docs.

Every internal the plan calls was verified to exist with the signature used
(.build_verb_sql, .shape_recipe_result, .attach_per_capita, .run_recipe,
.require_schema_v5, ...), so the tasks reuse the shared machinery rather than
reimplementing it. The verb deliberately does not route through
.verb_spendrev(), whose concept scoping, IG leg and complete= grid are all
flow-specific.

Test government is ALABAMA STATE GOVT (010000226085), which covers every case
in the bundled fixture: W01/W31/W61 in 2012/2019/2020, X21+Z77 in 2012,
Y07/Y08 throughout, and X40 in 2011 -- so the wide-era recipe bridge is
testable offline.
2026-08-03 09:36:42 -04:00
jared 57212e3399 docs: restore recipe= to cog_balances(); the pipeline was right
Corrects this spec. The earlier draft deferred recipe= and proposed adding
summary_categories rows for X40/X41. Both were wrong, and the pipeline state
they were meant to fix is correct and documented.

cog_pipeline/docs/phase_r_harmonization_review.md records the decisions:

- Sec 0.2: the wide era exposes these split families ONLY as aggregates, so
  the recipe join deliberately does NOT filter is_aggregate. Safe by
  construction -- wide rows are aggregate-only, modern rows leaf-only, every
  component year-scoped.
- Sec 1: the planned X40->Z77 harmonization MAP rows were dropped on purpose;
  continuity ships as recipes instead. That is why harmonization_map carries
  no balance-code rows.

The reader already implements this (R/recipes.R, R/spending.R). Verified
against the live corpus rather than trusting the comment: corrections_combined
FY2007, whose wide leg E05 is likewise aggregate-only, returns $906,743,000.

Also withdraws the claim that SB155/156 and SB195/196 contradict each other.
X40 rows after FY2002 are the wide-era SAS column persisting through the era
boundary; they say nothing about a classification-level rename. The two sets
describe different layers.

What survives is one narrow, non-blocking gap: no series_breaks row exists at
2016/2017 for Z77/Z78/X30, though review doc Sec 2 recommended exactly that.
Recorded as out-of-scope item 1 with the SB197-SB202 precedent.
2026-08-03 09:28:52 -04:00
jared 9f9d40e1c3 docs: drop the subtype argument from cog_balances()
balance is the only category_type whose subtype column is not orthogonal to
category. Measured against the crosswalk: 5 of 6 expenditure subtypes and 1 of
7 revenue subtypes span more than one category, but 0 of 5 balance subtypes do.
Balance is a strict tree -- Fund Balances = {general}, Retirement System
Holdings = {employee_retirement}, Insurance Trust Balances = the three trust
subtypes.

Exposing both arguments would admit no useful combination: of the 15 pairs, 3
are redundant and 12 are guaranteed empty for every government in every year,
failing as an empty tibble that reads as "holds none" rather than as a
contradiction.

Dropping it also keeps the verb aligned -- no uscogdata verb exposes a subtype
argument; the API layers its own subtype row filter on top, which cog-api#26
can do for /balances. #25's one-filter requirement is still met, since
category = "Fund Balances" is exactly W01/W31/W61.

Adds two tests: that one-filter equivalence, and an assertion that the
subtype -> category tree holds, so an upstream change making category lossy
fails here rather than in a user's analysis.
2026-08-03 09:15:50 -04:00
jared d7e14156ff docs: design spec for cog_balances(), the uscogdata#25 holdings surface
Requirement 1 of #25 shipped with #11/#12. This specs requirement 2 only.

Records three upstream gaps found while measuring the corpus, which change
the shipping scope:

- X40/X41 carry ~42.7K rows (1967-2011) but have no summary_categories row,
  so they cannot appear in a category_type='balance' view. Both holdings
  recipes span X40/X41 + Z77/Z78, so recipe= would silently return only the
  2012-2016 leg. recipe= is therefore deferred to v2.
- SB195/SB196 attach to fin_code X40/X41, outside the balance view.
- SB197-SB202 attach to flow codes, not the holdings codes, so the FY2016
  termination of X21/X30/X42/X44/X47/Z77/Z78 has no catalogued break.

Caveats 2-4 are therefore surfaced reader-side via a computed coverage_window
rather than through the existing series-break builders.
2026-08-03 08:44:18 -04:00
jared de7ccbebc7 Merge pull request 'feat: revenue_concept = c("general", "total") off the crosswalk (#12)' (#27) from feat/revenue-concepts-12 into main
R-CMD-check / check (push) Successful in 3m9s
Reviewed-on: #27
2026-07-30 22:09:44 -04:00
jared 4b23dbd9f4 feat: revenue_concept = c("general", "total") off the crosswalk (#12)
R-CMD-check / check (push) Successful in 3m47s
R-CMD-check / check (pull_request) Successful in 3m27s
Closes the last blocked test in the suite. Owner ruled both halves of the
open question yes on 2026-07-30.

`cog_revenue()` gains `revenue_concept`, mirroring `expenditure_concept`,
with Census's two published concepts defined as crosswalk
`revenue_subtype` sets rather than item-code prefixes:

  general = own_source + federal + state + local_aid   (the default)
  total   = general + utility + liquor_store + insurance_trust

The manual defines the first by subtracting the other three from the
second (4.3), so both are computable only once all four families are
named -- which cog_pipeline#79 does. Insurance trust now includes the
employee-retirement X codes (X01/X02/X05/X08) alongside the Y codes.

- inst/sql: revenue_long / revenue_long_harmonized carry EVERY revenue
  subtype; the concept narrows in R via the existing subtype_scope
  machinery, exactly as expenditure_concept narrows spending_long.
- cog_explain() now prints each verb's OWN concept. It previously
  printed `expenditure_concept` unconditionally, so a cog_revenue()
  caller was told "Concept: primary" -- a spending concept their result
  has nothing to do with.
- Fixture regenerated at pipeline_commit aadb46b (330 crosswalk rows).

Corrected two stale expectations in the blocked test while un-skipping
it. It asserted X01+X04+X05+X08 and omitted X02, which applies to state
governments and is nonzero for Wisconsin; X04 is an exhibit code for an
INTRAgovernmental transfer that Census's own "Total Emp Ret Rev"
excludes. Verified against that Census field: the right set is
X01+X02+X05+X08 = $2,283,883k, exactly. And its expected `total` of
$33,377,093k predated the Y codes being classified -- complete Total
Revenue for WI FY2012 is $34,881,961k (general 31,338,293 + Y 1,259,785
+ X 2,283,883).

Behaviour change worth knowing: `general` is now STRICT Census General
Revenue, so utility and liquor store revenue leave the default. Measured
on the fixture that is 15.9% of what cog_revenue() returned for cities,
vs 1.2% for states and 1.7% for counties.

Suite: 716 pass / 0 fail / 0 skip -- the first time this package has had
no skipped tests.

Closes #12
2026-07-30 20:49:49 -04:00
jared 5d77d39711 Merge pull request 'chore: regenerate fixture corpus at pipeline_commit e64a046 (#11 groundwork)' (#26) from feat/expenditure-concepts-11 into main
R-CMD-check / check (push) Successful in 3m3s
Reviewed-on: #26
2026-07-30 16:22:19 -04:00
38 changed files with 5192 additions and 77 deletions
+15
View File
@@ -11,6 +11,21 @@ jobs:
steps:
- name: Install system libraries and Node.js (required by actions/checkout)
run: |
# Switch apt to HTTPS mirrors. Measured from this runner on
# 2026-08-04: the SAME index file takes 20.1s over http:// and 3.1s
# over https://. apt fetches many indexes serially, so http:// does
# not read as "slow" -- it reads as a hang (zero bytes in
# /var/cache/apt/archives after 3+ minutes, apt's http workers parked
# in S state). rocker/r-ver:4.4 already ships ca-certificates and
# apt 2.8.3 has the https method built in, so nothing needs to be
# installed over http first to bootstrap this.
# `|| true` because the step runs under `sh -e`: on an image whose
# sources live in the other location, the missing-file sed must not
# kill the job.
sed -i -E 's#http://(archive|security)\.ubuntu\.com#https://\1.ubuntu.com#g' \
/etc/apt/sources.list.d/ubuntu.sources 2>/dev/null || true
sed -i -E 's#http://(archive|security)\.ubuntu\.com#https://\1.ubuntu.com#g' \
/etc/apt/sources.list 2>/dev/null || true
apt-get update -qq
apt-get install -y --no-install-recommends \
nodejs git \
File diff suppressed because it is too large Load Diff
+36 -14
View File
@@ -28,8 +28,23 @@ USCOGDATA_URL (local path or https://)
- `R/session.R` — `cog_open()`, `cog_close()`, `.ensure_session()`, `.coerce_govid_input()`
- `R/manifest.R` — `.fetch_or_cache_manifest()`, `.is_local_path()` (local paths bypass HTTP/cache)
- `R/views.R` — `.register_views()` (substitutes `{url}` into SQL files at `inst/sql/`)
- `inst/sql/` — 7 SQL view definitions: `long`, `spending_long`, `revenue_long`, `canonical_fips_xwalk`, `summary_categories`, `spending_annotated`, `revenue_annotated`
- `inst/sql/` — **23** SQL view definitions (measured), numbered by load order
(`10-` through `46-`): the `*_long` layer (`long`, `spending_long`,
`revenue_long`, `ig_long`, `balance_long`, plus `_harmonized` variants of
`spending_long`/`revenue_long`/`ig_long`), the `*_annotated` layer
(`spending_annotated`, `revenue_annotated`, `ig_annotated`,
`balance_annotated`, plus `_harmonized` variants of `spending_annotated`/
`revenue_annotated`/`ig_annotated`), and metadata views
(`canonical_fips_xwalk`, `summary_categories`, `gov_population_yearly`,
`harmonization_map`, `harmonization_recipes`, `series_breaks_pq`,
`representation`, `code_set`)
- `R/spending.R` / `R/revenue.R` — `cog_spending()` / `cog_revenue()` via shared `.verb_spendrev()`
- `R/balances.R` — `cog_balances()`. A third money-adjacent verb, but returns a
**stock** (a balance at a point in time) rather than a **flow** (activity
over a fiscal year), so it does NOT route through `.verb_spendrev()` and has
no `expenditure_concept`/`revenue_concept`/`complete`/`subtype` arguments.
`R/balance_caveats.R` attaches `provenance$balance_caveats` (GAAP-vs-gross
disclosure + measured per-subtype coverage windows).
- `R/rollup.R` — `cog_geographic_rollup()` (accepts named list of govids by layer)
- `R/peers.R` — `cog_find_peers()` + `cog_peer_compare()`
- `R/search.R` — `cog_gov_search()` (name pattern, state, type filters)
@@ -45,28 +60,31 @@ USCOGDATA_URL (local path or https://)
Any value without `://` is treated as a local path by `.is_local_path()` and reads
`manifest.json` directly from disk (no HTTP, no TTL cache).
## Current State (2026-04-27)
## Current State (2026-08-03)
**Version:** 0.1.0 (pre-release)
**Branch:** `main`, commit `d65e9fe`
**Tests:** 181 PASS / 0 FAIL / 0 SKIP
**Branch:** `feat/cog-balances-25`, commit `fde62eb`
**Tests:** 788 PASS / 0 FAIL / 0 SKIP / 0 WARN (measured `testthat::test_local()`, 2026-08-03, after the final-review fix wave)
**CI:** Gitea Actions green (`.gitea/workflows/ci.yml`)
### Completed (Tasks 2.1–2.7)
All 8 exported verbs implemented and tested:
`cog_spending`, `cog_revenue`, `cog_explain`, `cog_geographic_rollup`,
`cog_find_peers`, `cog_peer_compare`, `cog_gov_search`, `cog_mirror`,
plus `cog_categories`.
All **14** exports implemented and tested (measured from `NAMESPACE`):
`cog_spending`, `cog_revenue`, `cog_balances`, `cog_explain`,
`cog_geographic_rollup`, `cog_find_peers`, `cog_peer_compare`,
`cog_gov_search`, `cog_mirror`, `cog_categories`, `cog_recipes`,
`cog_manifest`, `cog_basket_resolution`, `cog_basket_unresolved`.
Bundled fixture corpus at `inst/extdata/fixture_corpus/` (3.6 MB, years
2019+2020, all 50 states). Tests run fully offline — no credentials needed.
Bundled fixture corpus at `inst/extdata/fixture_corpus/` (years
2011, 2012, 2019, 2020 — measured via DuckDB `read_parquet(hive_partitioning=1)`,
2026-08-03; all 50 states). Tests run fully offline — no credentials needed.
### Remaining to v0.1 release
1. **Task 2.8 — Docs:** roxygen `@param`/`@return`/`@examples` on all exports;
full `README.md`; `_pkgdown.yml`; `devtools::document()` + `pkgdown::build_site()`.
Vignettes can be stubbed for v0.1.
1. **Task 2.8 — Docs:** mostly done — all 14 exports have a `man/*.Rd`,
`README.md` and `_pkgdown.yml` exist, and `vignettes/` carries
`total-spending.Rmd` + `population-denominators.Rmd`. Outstanding:
`pkgdown::build_site()` has never been run (no `docs/`).
2. **Phase 3 — cog_explorer bridge:** create
`cog_explorer/examples/hello_world_uscogdata.Rmd` (installs from Gitea, runs
@@ -99,6 +117,10 @@ devtools::test()
- All verbs call `.ensure_session()` first, then query via `DBI::dbGetQuery()`
- Return value is always a `tbl_df` with a `provenance` attribute
- govid inputs always go through `.coerce_govid_input()` (accepts character or data frame)
- SQL lives in `inst/sql/` — never inline SQL strings in R files
- SQL has two layers. **View definitions** live in `inst/sql/` and are
registered by `.register_views()`, which globs the directory in sorted order
and substitutes `{url}`. **Query construction** is inline `sprintf()` in R
(`.build_verb_sql()`, `.run_recipe()`, `.attach_per_capita()`). Add a view as
a numbered `.sql` file; build a query in R.
- No arrow dependency — DuckDB reads parquet natively
- `withr` is a Suggests-only dep; only used in tests
+1
View File
@@ -1,5 +1,6 @@
# Generated by roxygen2: do not edit by hand
export(cog_balances)
export(cog_basket_resolution)
export(cog_basket_unresolved)
export(cog_categories)
+13
View File
@@ -1,5 +1,18 @@
# uscogdata 0.1.0 (development)
## New: `cog_balances()` for cash-and-security holdings
* New `cog_balances()` exposes the 14 cash-and-security holding codes
(`category_type = "balance"`): fund balances, retirement system holdings and
insurance trust balances (#25). Holdings are a stock, not a flow, so the verb
has no `expenditure_concept` / `revenue_concept` / `complete` arguments, and
no `subtype` argument either -- for holdings, `category` is a strict
coarsening of `balance_subtype`, so `category = "Fund Balances"` is exactly
the `general` family (`W01`/`W31`/`W61`).
* `cog_balances()` results carry `provenance$balance_caveats`, recording that
Census holdings are gross rather than GAAP fund balance, and the measured
coverage window of each subtype family.
## Multi-government aggregates now disclose their reporting coverage
* The Census of Governments is a **complete census only in years ending in 2
+122
View File
@@ -0,0 +1,122 @@
# R/balance_caveats.R
#
# The four caveats from cog_pipeline/docs/data_dictionary.md § Cash and
# security holdings. Each one silently invalidates an obvious analysis, so
# they travel in provenance (machine-readable, for cog-api#26) rather than
# living only in prose.
#
# Two of the four are already carried by the code-driven series-break
# builders and are deliberately NOT duplicated here:
# * SB195/SB196 -- X40/X41 book -> market at FY2002 -- fire via
# series_break_refs on the recipe path, the only path that observes those
# codes.
# What remains is the GAAP distinction (a constant) and the coverage windows
# (measured, never hardcoded, so they stay correct as the corpus grows).
#' Per-subtype observed year extents, plus which requested families are
#' truncated relative to the requested span.
#' @noRd
.balance_caveats <- function(con, codes_observed, years) {
cw <- .balance_coverage_windows(con)
observed_subtypes <- if (length(codes_observed) == 0L) {
character(0)
} else {
DBI::dbGetQuery(con, sprintf(
"SELECT DISTINCT balance_subtype FROM summary_categories
WHERE item_code IN (%s) AND balance_subtype IS NOT NULL",
.sql_lit_chr(codes_observed)
))$balance_subtype
}
# A family is "truncated" when the caller asked for years outside the span
# that family actually covers -- the FY2016 employee-retirement termination
# and the FY2021 end of the W family are both this shape.
truncated <- character(0)
if (length(years) > 0L) {
for (s in observed_subtypes) {
w <- cw[[s]]
if (is.null(w)) next
if (max(years) > w[2] || min(years) < w[1]) truncated <- c(truncated, s)
}
}
list(
not_gaap = TRUE,
not_gaap_note = paste0(
"Census holdings are gross -- no liabilities are netted -- and are NOT ",
"GAAP fund balance. A reserve ratio built from them overstates what is ",
"actually available."
),
coverage_window = cw,
truncated = sort(unique(truncated))
)
}
#' Per-subtype [min year, max year] extents for EVERY balance subtype in the
#' mounted corpus, memoised for the session.
#'
#' The query carries no govid and no year predicate -- its answer is a property
#' of the mounted corpus alone and cannot change between calls -- but it scans
#' the whole of `balance_long`, which measured 35% of `cog_balances()` runtime
#' on the bundled fixture and would be a per-request throughput ceiling once
#' cog-api#26 serves this verb over HTTP. Memoised in `.uscogdata_env` and
#' invalidated by `cog_close()`, the same pattern as `.uscogdata_env$manifest`.
#'
#' Scope is deliberately corpus-wide rather than query-scoped: a caller asking
#' "is there a family I missed?" needs every window. The observed-scoped field
#' is `truncated`. Documented as such in inst/schemas/provenance-v1.json.
#' @noRd
.balance_coverage_windows <- function(con) {
cached <- .uscogdata_env$balance_coverage_windows
if (!is.null(cached)) return(cached)
windows <- DBI::dbGetQuery(con,
"SELECT c.balance_subtype AS subtype,
MIN(l.year) AS year_min,
MAX(l.year) AS year_max
FROM balance_long l
JOIN summary_categories c USING (item_code)
WHERE c.balance_subtype IS NOT NULL
GROUP BY 1
ORDER BY 1"
)
cw <- stats::setNames(
lapply(seq_len(nrow(windows)),
function(i) as.integer(c(windows$year_min[i], windows$year_max[i]))),
windows$subtype
)
.uscogdata_env$balance_coverage_windows <- cw
cw
}
#' TRUE the first time `key` is seen this session, FALSE thereafter.
#' Reset by cog_close().
#' @noRd
.balance_caveat_once <- function(key) {
seen <- .uscogdata_env$balance_caveats_shown
if (is.null(seen)) seen <- character(0)
if (key %in% seen) return(FALSE)
.uscogdata_env$balance_caveats_shown <- c(seen, key)
TRUE
}
#' Emit at most one message per caveat class per session.
#' @noRd
.emit_balance_caveats <- function(caveats) {
if (.balance_caveat_once("not_gaap")) {
cli::cli_inform(c(
"!" = "Census holdings are gross and are {.strong not} GAAP fund balance.",
"i" = "No liabilities are netted; a reserve ratio built from them overstates available funds."
))
}
if (length(caveats$truncated) > 0L &&
.balance_caveat_once("coverage_window")) {
cli::cli_inform(c(
"!" = "Requested years extend beyond what {.val {caveats$truncated}} actually covers.",
"i" = "See {.code provenance$balance_caveats$coverage_window}."
))
}
invisible(NULL)
}
+159
View File
@@ -0,0 +1,159 @@
# R/balances.R
#
# Cash and security holdings. A third verb rather than an argument on a money
# verb because holdings are a STOCK -- a balance at a point in time -- while
# cog_spending()/cog_revenue() return FLOWS over a fiscal year. The money
# verbs' whole argument vocabulary (expenditure_concept, revenue_concept,
# complete=) describes flows and is meaningless here, so this deliberately
# does NOT route through .verb_spendrev().
#' Cash and security holdings for one or more governments
#'
#' Returns Census cash-and-security holdings (`category_type = "balance"`):
#' fund balances, retirement system holdings and insurance trust balances.
#'
#' @section Holdings are not GAAP fund balance:
#' Census holdings are **gross** -- no liabilities are netted -- so a reserve
#' ratio built from them overstates what is actually available. They are not
#' comparable to a GAAP fund balance from an ACFR.
#'
#' @param govid Canonical govid(s): a character vector, or a data frame with a
#' `canonical_govid` column (e.g. from [cog_gov_search()]).
#' @param years Integer vector of fiscal years.
#' @param category Optional character vector of categories to keep. One of
#' `"Fund Balances"`, `"Insurance Trust Balances"`,
#' `"Retirement System Holdings"`. There is deliberately no `subtype`
#' argument: for holdings, `category` is a strict coarsening of
#' `balance_subtype` (unlike the money verbs, where the two axes cross), so
#' every combination would be either redundant or empty.
#' `category = "Fund Balances"` is exactly the `general` family
#' (`W01`/`W31`/`W61`). `balance_subtype` is returned, so a finer split is
#' one `dplyr::filter()` away.
#' @param per_capita Divide holdings by population. Note this is a **stock per
#' resident** (reserves per person), which is *not* comparable to
#' [cog_spending()]'s per-capita figures -- those are a flow per person.
#' @param adjust_to_year Deflate to this year's dollars (CPI-U).
#' @param basis Accepted for uniformity with the money verbs, but currently a
#' **no-op**: `harmonization_map` carries no balance-code rows, so harmonized
#' and raw space are identical for holdings. Reported in
#' `provenance$basis_note`.
#' @param recipe Optional harmonization recipe id (see [cog_recipes()]).
#' `"cash_securities_z77_wide"` and `"cash_securities_z78_wide"` bridge the
#' wide era to the modern one.
#'
#' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
#' `balance_subtype`, `category`, `amt_nominal`, `codes_included`,
#' `aggregate_fallback`, plus optional `amt_per_capita_nominal` and
#' `pop_source` (when `per_capita = TRUE`), optional `amt_real` (when
#' `adjust_to_year` is set), and optional `amt_per_capita_real` (only when
#' **both** `per_capita = TRUE` and `adjust_to_year` are set -- there is no
#' nominal per-capita column to deflate otherwise). Amounts are full US
#' dollars.
#'
#' Carries a `provenance` attribute matching
#' `inst/schemas/provenance-v1.json`, whose `balance_caveats` block reports
#' `not_gaap`, `not_gaap_note`, `coverage_window` (measured year extents for
#' every balance subtype in the mounted corpus, not only the observed ones)
#' and `truncated` (the observed subtypes whose coverage falls short of the
#' requested years). `expenditure_concept`/`revenue_concept` are `NA` --
#' holdings are a stock, not a flow, so neither concept vocabulary applies.
#' @export
cog_balances <- function(govid, years, category = NULL,
per_capita = FALSE, adjust_to_year = NULL,
basis = c("harmonized", "raw"), recipe = NULL) {
call <- match.call()
basis <- match.arg(basis, c("harmonized", "raw"))
# Coerce FIRST, validate second: .validate_verb_inputs() asserts
# is.character(govid), and a data-frame govid (cog_gov_search() output) has
# not been unwrapped yet at this point.
govid <- .coerce_govid_input(govid)
# The money verbs' validator, reused rather than re-implemented (R/spending.R).
# It covers the exact superset cog_balances() needs -- including the
# recipe/category mutual-exclusivity guard -- so a second local copy would
# only be a place for the two to drift apart. This is the same kind of
# helper reuse as .build_verb_sql()/.attach_per_capita() below; it does NOT
# route the verb through .verb_spendrev(), which stays deliberately unused
# here because its flow vocabulary is meaningless for a stock.
.validate_verb_inputs(govid, years, category, per_capita, adjust_to_year,
recipe)
years <- as.integer(years)
if (!is.null(adjust_to_year)) adjust_to_year <- as.integer(adjust_to_year)
con <- .ensure_session()
.require_balance_support(con)
scope <- .check_govids_in_scope(govid)
basis_note <- paste0(
"`basis` has no effect on holdings: harmonization_map carries no ",
"balance-code rows, so harmonized and raw space are identical here."
)
manifest <- .uscogdata_env$manifest
recipe_block <- NULL
category_for_prov <- category
if (!is.null(recipe)) {
.require_schema_v5(con, manifest, "recipe =")
.validate_recipe_id(con, recipe)
comps <- .recipe_components(con, recipe)
recipe_label <- comps$label[[1]]
result <- .run_recipe(con, recipe, govid, years)
sql <- attr(result, "sql_query")
result <- .shape_recipe_result(result, "balance_subtype", recipe_label)
recipe_block <- list(
recipe_id = recipe, label = recipe_label,
components = .df_to_row_list(comps)
)
category_for_prov <- recipe_label
} else {
sql <- .build_verb_sql("balance_annotated", "balance_subtype",
govid, years, category,
ig_view = NULL, subtype_scope = NULL)
result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
}
# Order matters (matches .verb_spendrev()): per-capita first, so
# .attach_real_dollars() deflates the nominal per-capita column into
# amt_per_capita_real rather than needing amt_per_capita_nominal recomputed.
if (isTRUE(per_capita)) result <- .attach_per_capita(result, con, govid)
if (!is.null(adjust_to_year)) {
result <- .attach_real_dollars(result, adjust_to_year, per_capita)
}
prov <- .build_provenance(
verb = "cog_balances", call = call, govid = govid, years = years,
category = category_for_prov, per_capita = per_capita,
adjust_to_year = adjust_to_year, result = result, sql = sql,
subtype_col = "balance_subtype",
basis = basis, basis_note = basis_note,
# Neither concept vocabulary applies to a stock.
expenditure_concept = NA_character_,
revenue_concept = NA_character_,
recipe = recipe_block
)
prov$scope$govids_found <- scope$found
prov$scope$govids_missing <- scope$missing
prov$balance_caveats <- .balance_caveats(
con, prov$codes_summed$observed, years
)
.emit_balance_caveats(prov$balance_caveats)
attr(result, "provenance") <- prov
result
}
#' Abort unless the mounted corpus classifies balance codes.
#'
#' `balance_subtype` arrived with cog_pipeline #76/#77 without a
#' schema_version bump, so the check is on the column, not the version.
#' @noRd
.require_balance_support <- function(con) {
if (.corpus_has_balance_subtype(con)) return(invisible(TRUE))
cli::cli_abort(
c("This corpus does not classify cash and security holdings.",
i = "`summary_categories` has no {.field balance_subtype} column.",
i = "Republish from cog_pipeline at #76/#77 or later."),
class = "uscogdata_no_balance_support"
)
}
+13 -6
View File
@@ -5,12 +5,19 @@
#' Returns the category taxonomy exposed by the corpus's
#' `summary_categories` view, grouped to one row per
#' `(category, subtype)` pair. Use this to discover valid `category`
#' values for [cog_spending()] / [cog_revenue()] /
#' values for [cog_spending()] / [cog_revenue()] / [cog_balances()] /
#' [cog_geographic_rollup()] and to audit which Census item codes feed
#' each category.
#'
#' @param type Either `NULL` (default, return both spending and revenue
#' rows), `"spending"`, or `"revenue"`.
#' `subtype` COALESCEs the crosswalk's three subtype columns, so it carries
#' `spend_subtype` on expenditure rows, `revenue_subtype` on revenue rows and
#' `balance_subtype` on balance rows. Note that [cog_balances()] itself takes
#' no `subtype` argument — for holdings, `category` is a strict coarsening of
#' `balance_subtype` — but the value is surfaced here because it is the
#' discovery surface downstream consumers build their vocabulary from.
#'
#' @param type Either `NULL` (default, every row: expenditure, revenue and
#' balance), `"spending"`, `"revenue"`, or `"balance"`.
#' @param pattern Optional regex matched case-insensitively against the
#' `category` column (e.g. `"Police"` or `"Tax"`).
#' @return Tibble with columns `category`, `category_type`, `subtype`,
@@ -20,8 +27,8 @@
cog_categories <- function(type = NULL, pattern = NULL) {
if (!is.null(type)) {
if (!is.character(type) || length(type) != 1L ||
!type %in% c("spending", "revenue")) {
cli::cli_abort('`type` must be NULL, "spending", or "revenue".')
!type %in% c("spending", "revenue", "balance")) {
cli::cli_abort('`type` must be NULL, "spending", "revenue", or "balance".')
}
}
if (!is.null(pattern) &&
@@ -48,7 +55,7 @@ cog_categories <- function(type = NULL, pattern = NULL) {
sql <- paste(
"SELECT category, category_type,
COALESCE(spend_subtype, revenue_subtype) AS subtype,
COALESCE(spend_subtype, revenue_subtype, balance_subtype) AS subtype,
COUNT(DISTINCT item_code) AS n_codes,
string_agg(DISTINCT item_code, ',' ORDER BY item_code) AS item_codes
FROM summary_categories",
+39 -1
View File
@@ -60,7 +60,20 @@ cog_explain <- function(result, format = c("print", "list")) {
cli::cli_text("Basis: {prov$basis}{note}")
}
if (!is.null(prov$expenditure_concept)) {
# Each verb reports its OWN concept. Both fields are always present (each
# defaults to its concept's default), so printing `expenditure_concept`
# unconditionally would tell a cog_revenue() caller "Concept: primary",
# which names a spending concept their result has nothing to do with.
if (identical(prov$verb, "cog_revenue")) {
if (!is.null(prov$revenue_concept)) {
cli::cli_text("Concept: {prov$revenue_concept} revenue")
}
} else if (identical(prov$verb, "cog_balances")) {
# Both concept fields are deliberately NA here (a stock has no flow
# concept). Printing the raw NA reads as a missing value rather than an
# intentional one, so say what it means instead.
cli::cli_text("Concept: not applicable (holdings are a stock, not a flow)")
} else if (!is.null(prov$expenditure_concept)) {
concept_note <- if (!is.null(prov$expenditure_concept_note) &&
!is.na(prov$expenditure_concept_note)) {
sprintf(" (%s)", prov$expenditure_concept_note)
@@ -166,6 +179,31 @@ cog_explain <- function(result, format = c("print", "list")) {
cli::cli_ul(.series_break_story_lines(prov$corpus_break_refs))
}
# Balance results only (NULL on money-verb provenance, so they are
# unaffected). This is the ONLY on-demand surface for the GAAP disclosure:
# .emit_balance_caveats() fires at most once per session, and is routinely
# consumed by a suppressMessages() call or by a knitted chunk nobody reads,
# so a caller who deliberately audits a result with cog_explain() must still
# be told.
bc <- prov$balance_caveats
if (!is.null(bc)) {
cli::cli_h2("Holdings caveats")
if (!is.null(bc$not_gaap_note)) cli::cli_alert_warning(bc$not_gaap_note)
if (length(bc$truncated) > 0L) {
cli::cli_text(
"Requested years extend beyond what these families actually cover:"
)
cli::cli_ul(vapply(bc$truncated, function(s) {
w <- bc$coverage_window[[s]]
if (length(w) == 2L) {
sprintf("%s: covered %s-%s in this corpus", s, w[1], w[2])
} else {
s
}
}, character(1)))
}
}
cli::cli_h2("Transformations")
uc <- prov$transformations$units_conversion
if (isTRUE(uc$applied)) {
+1 -1
View File
@@ -138,7 +138,7 @@
#' year, matching canonical_fips_xwalk) rather than as-of-year; as-of-year
#' moved to the *_asof columns. This package's own geography always came from
#' the xwalk (already present-based), so behaviour is unchanged.
.validate_schema <- function(manifest, supported = c(4L, 5L, 6L)) {
.validate_schema <- function(manifest, supported = c(4L, 5L, 6L, 7L)) {
if (!manifest$schema_version %in% supported) {
cli::cli_abort(c(
"Corpus schema version mismatch.",
+3 -1
View File
@@ -6,9 +6,10 @@
per_capita, adjust_to_year, result, sql,
subtype_col, basis = NA_character_,
basis_note = NA_character_,
expenditure_concept = "direct",
expenditure_concept = "primary",
expenditure_concept_note = NA_character_,
expenditure_concept_direct_suppressed = FALSE,
revenue_concept = "general",
harmonization = NULL, recipe = NULL,
suggestions = list(),
completion = NULL) {
@@ -67,6 +68,7 @@
expenditure_concept = expenditure_concept,
expenditure_concept_note = expenditure_concept_note,
expenditure_concept_direct_suppressed = isTRUE(expenditure_concept_direct_suppressed),
revenue_concept = revenue_concept,
harmonization = harmonization %||% list(
applied = FALSE, na_rows_excluded = 0L, na_amount_excluded = 0,
note = NA_character_
+27
View File
@@ -8,6 +8,31 @@
#' multiplies by 1000 and records the conversion in `provenance`).
#'
#' @inheritParams cog_spending
#' @param revenue_concept Which of Census's two published revenue concepts to
#' return. Concepts are defined as sets of the crosswalk's `revenue_subtype`
#' values -- never as item-code first letters, which cannot classify
#' correctly (prefix `Y` spans revenue, expenditure and balance codes, and
#' prefix `X` does the same):
#'
#' * `"general"` (default) -- Census General Revenue: `own_source` +
#' `federal` + `state` + `local_aid`. The manual defines this concept by
#' subtraction (section 4.3: *"General revenue comprises all revenue
#' except that classified as liquor store, utility, or insurance trust
#' revenue"*), so utility (`A91`-`A94`), liquor store (`A90`) and
#' insurance trust revenue are all excluded.
#' * `"total"` -- Census Total Revenue: every revenue subtype, i.e.
#' `general` plus utility, liquor store, and insurance trust revenue
#' (`Y01`/`Y02`/`Y04`/`Y11`/`Y12`/`Y51`/`Y52` and the employee-retirement
#' `X01`/`X02`/`X05`/`X08`).
#'
#' The two are related by Census's own identity, `Total Revenue = General +
#' Utility + Liquor Store + Insurance Trust`.
#'
#' Note that the employee-retirement (`X`) codes stop at FY2016, when those
#' systems moved out of the annual finance file into the separate Annual
#' Survey of Public Pensions, so a `"total"` series steps down at the
#' FY2016/FY2017 seam for reasons that are about collection scope rather
#' than revenue (series breaks `SB197`-`SB202`).
#' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
#' `revenue_subtype`, `category`, `amt_nominal`, optional `amt_real`,
#' optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
@@ -17,6 +42,7 @@
cog_revenue <- function(govid, years, category = NULL,
per_capita = FALSE, adjust_to_year = NULL,
basis = c("harmonized", "raw"), recipe = NULL,
revenue_concept = c("general", "total"),
complete = FALSE) {
# flow_prefixes no longer classifies rows (crosswalk revenue_subtype
# membership does -- General Revenue, i.e. everything except
@@ -35,6 +61,7 @@ cog_revenue <- function(govid, years, category = NULL,
adjust_to_year = adjust_to_year,
basis = basis,
recipe = recipe,
revenue_concept = revenue_concept,
complete = complete
)
}
+4 -1
View File
@@ -12,7 +12,7 @@ cog_open <- function(url = .resolve_url(),
DBI::dbExecute(con, "INSTALL httpfs; LOAD httpfs;")
manifest <- .fetch_or_cache_manifest(url, cache_dir)
.validate_schema(manifest, supported = c(4L, 5L, 6L))
.validate_schema(manifest, supported = c(4L, 5L, 6L, 7L))
.validate_scope(manifest)
.register_views(con, url, manifest)
@@ -95,4 +95,7 @@ cog_close <- function() {
}
.uscogdata_env$con <- NULL
.uscogdata_env$manifest <- NULL
.uscogdata_env$balance_caveats_shown <- NULL
# Memoised corpus-constant; a different corpus may be mounted next.
.uscogdata_env$balance_coverage_windows <- NULL
}
+37 -7
View File
@@ -31,9 +31,26 @@
)
}
# cog_revenue()'s single concept (until uscogdata#12 adds more): Census
# General Revenue -- every crosswalk revenue subtype except insurance_trust.
# The two revenue concepts (uscogdata#12), again as crosswalk subtype sets.
# Census's manual section 4.3 defines the first by SUBTRACTING from the second
# -- "General revenue comprises all revenue except that classified as liquor
# store, utility, or insurance trust revenue" -- giving the identity
#
# Total Revenue = General + Utility + Liquor Store + Insurance Trust
#
# Verified against Census's own computed concept fields (IndFin FY2012,
# Wisconsin state): 31,410,686 + 0 + 0 + 4,469,906 = 35,880,592, exact.
.revenue_subtypes_general <- c("own_source", "federal", "state", "local_aid")
.revenue_subtypes_total <- c(.revenue_subtypes_general, "utility",
"liquor_store", "insurance_trust")
#' @noRd
.revenue_concept_subtypes <- function(concept) {
switch(concept,
general = .revenue_subtypes_general,
total = .revenue_subtypes_total
)
}
#' Summarized spending by category
#'
@@ -198,6 +215,7 @@ cog_spending <- function(govid, years, category = NULL,
per_capita, adjust_to_year,
basis = c("harmonized", "raw"), recipe = NULL,
expenditure_concept = c("primary", "direct", "total"),
revenue_concept = c("general", "total"),
complete = FALSE) {
basis_explicit <- length(basis) == 1L
basis <- match.arg(basis, c("harmonized", "raw"))
@@ -215,16 +233,27 @@ cog_spending <- function(govid, years, category = NULL,
}
)
revenue_concept <- tryCatch(
match.arg(revenue_concept, c("general", "total")),
error = function(e) {
cli::cli_abort(
"`revenue_concept` must be one of {.val general} or {.val total}.",
class = "uscogdata_invalid_revenue_concept",
parent = e
)
}
)
# The concept's subtype scope. Every code path below -- the verb SQL, the
# harmonization exclusion count, and the complete = TRUE grid -- is scoped
# by crosswalk subtype membership, never by item-code prefix. For revenue
# there is a single concept today (General Revenue; uscogdata#12 will add
# more). "total"'s extra intergovernmental leg travels through the ig_*
# views, not through this scope.
# by crosswalk subtype membership, never by item-code prefix. The
# expenditure "total" concept's extra intergovernmental leg is the one
# exception: it travels through the ig_* views rather than this scope,
# because its legacy rows are aggregate-flagged.
subtype_scope <- if (identical(subtype_col, "spend_subtype")) {
.expenditure_concept_subtypes(expenditure_concept)
} else {
.revenue_subtypes_general
.revenue_concept_subtypes(revenue_concept)
}
govid <- .coerce_govid_input(govid, arg = "govid")
@@ -427,6 +456,7 @@ cog_spending <- function(govid, years, category = NULL,
expenditure_concept = expenditure_concept,
expenditure_concept_note = expenditure_concept_note_for_prov,
expenditure_concept_direct_suppressed = direct_suppressed_flag,
revenue_concept = revenue_concept,
harmonization = harmonization,
recipe = recipe_block,
suggestions = suggestions,
+24
View File
@@ -45,6 +45,29 @@
"37-code_set.sql" = "code_set.parquet"
)
# Cash and security holdings (uscogdata#25). 46- selects
# `c.balance_subtype`, a column that arrived with cog_pipeline #76/#77 and
# WITHOUT a schema_version bump -- so neither existing gate applies:
# .harmonization_view_files keys on schema_version, .representation_view_files
# on the presence of a FILE. Here the discriminator is a COLUMN on a table
# that exists either way. CREATE VIEW resolves its source schema eagerly, so
# on an older corpus 46- would fail at registration with "Binder Error:
# Referenced column balance_subtype not found" rather than at query time.
.balance_view_files <- c("26-balance_long.sql", "46-balance_annotated.sql")
#' Does the mounted corpus's `summary_categories` carry `balance_subtype`?
#' Probed against the live connection rather than the manifest, because the
#' manifest describes files, not columns.
#' @noRd
.corpus_has_balance_subtype <- function(con) {
n <- DBI::dbGetQuery(con,
"SELECT COUNT(*) AS n FROM information_schema.columns
WHERE table_name = 'summary_categories'
AND column_name = 'balance_subtype'"
)$n
isTRUE(as.integer(n) > 0L)
}
#' Does the mounted corpus publish `file` (e.g. "code_set.parquet")?
#' Reads the manifest's metadata list rather than stat-ing the URL, so it
#' works identically for a local fixture and a remote share.
@@ -66,6 +89,7 @@
if (base %in% .harmonization_view_files && schema_version < 5L) next
if (base %in% names(.representation_view_files) &&
!.corpus_has_table(manifest, .representation_view_files[[base]])) next
if (base %in% .balance_view_files && !.corpus_has_balance_subtype(con)) next
sql <- paste(readLines(f, warn = FALSE), collapse = "\n")
sql <- gsub("\\{url\\}", url, sql, fixed = FALSE)
DBI::dbExecute(con, sql)
+32
View File
@@ -71,6 +71,38 @@ enforce this by refusing `expenditure_concept = "total"`. See
`vignette("total-spending", package = "uscogdata")` for the full
explanation with worked examples.
## General vs Total revenue
`cog_revenue(..., revenue_concept = c("general", "total"))` selects between
Census's two published revenue concepts, again defined as crosswalk
`revenue_subtype` sets rather than item-code prefixes:
- `"general"` (the default) is Census **General Revenue**: own-source
(taxes, charges, miscellaneous) plus federal, state and local
intergovernmental aid.
- `"total"` is Census **Total Revenue**: `general` plus utility revenue
(`A91`–`A94`), liquor store revenue (`A90`), and insurance trust revenue
(unemployment and workers' compensation `Y` codes plus the
employee-retirement `X` codes).
The manual defines the first by subtracting the other three from the second,
so the two are related by Census's own identity:
```
Total Revenue = General + Utility + Liquor Store + Insurance Trust
```
Two things worth knowing before switching to `"total"`:
- **Utility revenue is large for cities.** Measured on the bundled fixture,
utility plus liquor store revenue is 15.9% of city (type 2) revenue, versus
1.2% for states and 1.7% for counties. `general` excludes it by definition.
- **The employee-retirement (`X`) codes stop at FY2016**, when those systems
moved out of the annual finance file into the separate Annual Survey of
Public Pensions. A `"total"` series therefore steps down at the
FY2016/FY2017 seam for reasons of collection scope, not revenue (series
breaks `SB197`–`SB202`, in the corpus's `series_breaks` table).
## Developer notes
### Testing
+6
View File
@@ -3,6 +3,12 @@ template:
bootstrap: 5
reference:
- title: Financial data
desc: Spending, revenue and balance-sheet holdings for one or more governments.
contents:
- cog_spending
- cog_revenue
- cog_balances
- title: Search & basket
desc: Resolve place names into canonical govids.
contents:
Binary file not shown.
Binary file not shown.
+4 -4
View File
@@ -1,7 +1,7 @@
{
"schema_version": 6,
"built_at": "2026-07-30T20:01:56Z",
"pipeline_commit": "e64a046",
"built_at": "2026-07-31T00:47:27Z",
"pipeline_commit": "aadb46b",
"fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated from the sparsified schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit zeros, so FY2011 absence means Census published $0 while FY2012+ absence means not reported. representation.parquet and code_set.parquet carry that rule and ship in full, as do every other metadata table in the publish tree. 2011/2012 straddle both the wide-aggregate -> modern-leaf format boundary (exercised by basis=\"harmonized\" and recipe= queries) and the dense -> sparse representation boundary (SB194); 2019/2020 retain the prior per-capita/CPI regression anchors. Regenerated via data-raw/regenerate_fixture_corpus.R.",
"data_vintage": {
"source_vintages": {
@@ -105,12 +105,12 @@
},
{
"path": "data/series_breaks.parquet",
"sha256": "381090660c8b8a1bee852e7f63d29b9ecaf10f71870f41c63de92017c83b6f1f",
"sha256": "06dcc995ff533e57cc65fa25086cc9bf83ba592c58bf7cc99269dc2576f69944",
"description": "series_breaks.parquet"
},
{
"path": "data/summary_categories.parquet",
"sha256": "e4918abf8e9e6d1372d7ddc255dc199f9c303e250f4106449241b48c68abee67",
"sha256": "e3b0efa00ce713b8f45829b89cfde24b55333f26101f0495df82d85997d18d8e",
"description": "summary_categories.parquet"
}
]
+16
View File
@@ -25,6 +25,12 @@
"type": "boolean",
"description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'primary' or 'direct'. See the affected rows' `notes` for the recovering recipe, if any."
},
"revenue_concept": {
"type": "string",
"enum": ["general", "total"],
"description": "Which revenue concept produced this result, defined as crosswalk revenue_subtype sets (never item-code prefixes). 'general' (the default) is Census General Revenue: own_source + federal + state + local_aid. 'total' is Census Total Revenue: general plus utility, liquor store and insurance trust revenue. Census defines the first by subtracting the other three from the second (manual section 4.3). Meaningful for cog_revenue() results; spending results carry the default.",
"$comment": "The employee-retirement (X) codes inside insurance_trust stop at FY2016, so a 'total' series steps at the FY2016/FY2017 seam for collection-scope reasons (series breaks SB197-SB202)."
},
"harmonization": { "type": "object" },
"recipe": { "type": ["object", "null"] },
"suggestions": { "type": "array" },
@@ -47,6 +53,16 @@
"items": { "type": "string" },
"description": "Ids of catalogued series breaks whose fin_code is the literal 'ALL' -- caveats about the corpus as a whole (dollar precision across 1976/1977, imputation exclusion from 2002, the dense -> sparse representation change at 2012, the government id scheme change at 2017) rather than about one item code. Selected on the break_year window alone, so they do not depend on which codes a result contains. Disjoint from series_break_refs by construction: an entry qualifies the whole result, not one series."
},
"balance_caveats": {
"type": ["object", "null"],
"description": "Present only on cog_balances() results (null/absent for cog_spending()/cog_revenue()). `not_gaap` is always TRUE and `not_gaap_note` explains that Census holdings are gross -- no liabilities are netted -- so they are NOT comparable to a GAAP fund balance. `coverage_window` maps EVERY balance_subtype present in the mounted corpus -- not only the ones this query observed -- to its measured [min year, max year] there (never hardcoded), so a caller can see which families exist and over what span before deciding they missed one. `truncated` is the query-scoped field: it lists only the subtypes this result actually observed whose coverage_window does not fully span the requested years.",
"properties": {
"not_gaap": { "type": "boolean" },
"not_gaap_note": { "type": "string" },
"coverage_window": { "type": "object" },
"truncated": { "type": "array", "items": { "type": "string" } }
}
},
"manifest": { "type": "object" },
"sql_query": { "type": "string" }
}
+8 -5
View File
@@ -1,15 +1,18 @@
-- Revenue rows, classified by crosswalk MEMBERSHIP rather than item-code
-- first letter (see 20-spending_long.sql for why prefixes cannot work).
-- Scope is Census General Revenue: every crosswalk revenue subtype EXCEPT
-- insurance_trust (Y01/Y02/Y04/Y11/Y12/Y51/Y52). Owner ruling 2026-07-30:
-- the default revenue concept stays general; surfacing insurance-trust
-- revenue through an explicit concept argument is uscogdata#12.
--
-- Carries EVERY revenue subtype. Which of Census's two published concepts a
-- query actually returns is decided per revenue_concept in R
-- (.verb_spendrev), exactly as expenditure_concept narrows spending_long:
-- general = own_source + federal + state + local_aid (the default)
-- total = general + utility + liquor_store + insurance_trust
-- Census defines the first by subtracting the other three from the second
-- (manual section 4.3), so both concepts need all four families present here.
CREATE OR REPLACE VIEW revenue_long AS
SELECT *
FROM long
WHERE item_code IN (
SELECT item_code FROM summary_categories
WHERE category_type = 'revenue'
AND revenue_subtype <> 'insurance_trust'
)
AND NOT is_aggregate;
+1 -2
View File
@@ -1,5 +1,5 @@
-- Harmonized-basis twin of 21-revenue_long.sql: same crosswalk-membership
-- classification (General Revenue = revenue minus insurance_trust), applied
-- classification (every revenue subtype; the concept narrows in R), applied
-- to harmonized_code rather than the published item_code.
CREATE OR REPLACE VIEW revenue_long_harmonized AS
SELECT * REPLACE (harmonized_code AS item_code)
@@ -9,5 +9,4 @@ WHERE NOT is_aggregate
AND harmonized_code IN (
SELECT item_code FROM summary_categories
WHERE category_type = 'revenue'
AND revenue_subtype <> 'insurance_trust'
);
+22
View File
@@ -0,0 +1,22 @@
-- Cash and security holdings, classified by crosswalk MEMBERSHIP on
-- category_type (see 21-revenue_long.sql for why first-letter prefixes cannot
-- do this job -- the X and Y families each span revenue, expenditure AND
-- balance).
--
-- These rows are STOCKS: a balance at a point in time, not a flow over a
-- fiscal year. Summing a stock with a flow is meaningless, which is why they
-- live behind a third view rather than as a subtype of either money view, and
-- why neither spending_long nor revenue_long can reach them.
--
-- `NOT is_aggregate` mirrors spending_long / revenue_long. The wide-era
-- aggregate-only holdings codes (X40/X41) are deliberately outside this view;
-- they are reachable only through the recipe path, which bypasses this filter
-- by design (cog_pipeline/docs/phase_r_harmonization_review.md § 0.2).
CREATE OR REPLACE VIEW balance_long AS
SELECT *
FROM long
WHERE item_code IN (
SELECT item_code FROM summary_categories
WHERE category_type = 'balance'
)
AND NOT is_aggregate;
+16
View File
@@ -0,0 +1,16 @@
CREATE OR REPLACE VIEW balance_annotated AS
SELECT
s.*,
x.gov_name AS xwalk_gov_name,
x.govs_type,
x.type_label,
x.fips_state AS xwalk_fips_state,
x.fips_county AS xwalk_fips_county,
x.fips_place,
x.population_acs,
c.category,
c.category_type,
c.balance_subtype
FROM balance_long s
LEFT JOIN canonical_fips_xwalk x USING (canonical_govid)
LEFT JOIN summary_categories c USING (item_code);
+76
View File
@@ -0,0 +1,76 @@
% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/balances.R
\name{cog_balances}
\alias{cog_balances}
\title{Cash and security holdings for one or more governments}
\usage{
cog_balances(
govid,
years,
category = NULL,
per_capita = FALSE,
adjust_to_year = NULL,
basis = c("harmonized", "raw"),
recipe = NULL
)
}
\arguments{
\item{govid}{Canonical govid(s): a character vector, or a data frame with a
`canonical_govid` column (e.g. from [cog_gov_search()]).}
\item{years}{Integer vector of fiscal years.}
\item{category}{Optional character vector of categories to keep. One of
`"Fund Balances"`, `"Insurance Trust Balances"`,
`"Retirement System Holdings"`. There is deliberately no `subtype`
argument: for holdings, `category` is a strict coarsening of
`balance_subtype` (unlike the money verbs, where the two axes cross), so
every combination would be either redundant or empty.
`category = "Fund Balances"` is exactly the `general` family
(`W01`/`W31`/`W61`). `balance_subtype` is returned, so a finer split is
one `dplyr::filter()` away.}
\item{per_capita}{Divide holdings by population. Note this is a **stock per
resident** (reserves per person), which is *not* comparable to
[cog_spending()]'s per-capita figures -- those are a flow per person.}
\item{adjust_to_year}{Deflate to this year's dollars (CPI-U).}
\item{basis}{Accepted for uniformity with the money verbs, but currently a
**no-op**: `harmonization_map` carries no balance-code rows, so harmonized
and raw space are identical for holdings. Reported in
`provenance$basis_note`.}
\item{recipe}{Optional harmonization recipe id (see [cog_recipes()]).
`"cash_securities_z77_wide"` and `"cash_securities_z78_wide"` bridge the
wide era to the modern one.}
}
\value{
Tibble with columns `year`, `canonical_govid`, `gov_name`,
`balance_subtype`, `category`, `amt_nominal`, `codes_included`,
`aggregate_fallback`, plus optional `amt_per_capita_nominal` and
`pop_source` (when `per_capita = TRUE`), optional `amt_real` (when
`adjust_to_year` is set), and optional `amt_per_capita_real` (only when
**both** `per_capita = TRUE` and `adjust_to_year` are set -- there is no
nominal per-capita column to deflate otherwise). Amounts are full US
dollars.
Carries a `provenance` attribute matching
`inst/schemas/provenance-v1.json`, whose `balance_caveats` block reports
`not_gaap`, `not_gaap_note`, `coverage_window` (measured year extents for
every balance subtype in the mounted corpus, not only the observed ones)
and `truncated` (the observed subtypes whose coverage falls short of the
requested years). `expenditure_concept`/`revenue_concept` are `NA` --
holdings are a stock, not a flow, so neither concept vocabulary applies.
}
\description{
Returns Census cash-and-security holdings (`category_type = "balance"`):
fund balances, retirement system holdings and insurance trust balances.
}
\section{Holdings are not GAAP fund balance}{
Census holdings are **gross** -- no liabilities are netted -- so a reserve
ratio built from them overstates what is actually available. They are not
comparable to a GAAP fund balance from an ACFR.
}
+11 -3
View File
@@ -7,8 +7,8 @@
cog_categories(type = NULL, pattern = NULL)
}
\arguments{
\item{type}{Either `NULL` (default, return both spending and revenue
rows), `"spending"`, or `"revenue"`.}
\item{type}{Either `NULL` (default, every row: expenditure, revenue and
balance), `"spending"`, `"revenue"`, or `"balance"`.}
\item{pattern}{Optional regex matched case-insensitively against the
`category` column (e.g. `"Police"` or `"Tax"`).}
@@ -22,7 +22,15 @@ Tibble with columns `category`, `category_type`, `subtype`,
Returns the category taxonomy exposed by the corpus's
`summary_categories` view, grouped to one row per
`(category, subtype)` pair. Use this to discover valid `category`
values for [cog_spending()] / [cog_revenue()] /
values for [cog_spending()] / [cog_revenue()] / [cog_balances()] /
[cog_geographic_rollup()] and to audit which Census item codes feed
each category.
}
\details{
`subtype` COALESCEs the crosswalk's three subtype columns, so it carries
`spend_subtype` on expenditure rows, `revenue_subtype` on revenue rows and
`balance_subtype` on balance rows. Note that [cog_balances()] itself takes
no `subtype` argument — for holdings, `category` is a strict coarsening of
`balance_subtype` — but the value is surfaced here because it is the
discovery surface downstream consumers build their vocabulary from.
}
+27
View File
@@ -12,6 +12,7 @@ cog_revenue(
adjust_to_year = NULL,
basis = c("harmonized", "raw"),
recipe = NULL,
revenue_concept = c("general", "total"),
complete = FALSE
)
}
@@ -57,6 +58,32 @@ argument is ignored and the result's provenance reports
FALSE`, pointing at the `recipe` block instead) rather than a
possibly-misleading `"harmonized"`/`"raw"` value.}
\item{revenue_concept}{Which of Census's two published revenue concepts to
return. Concepts are defined as sets of the crosswalk's `revenue_subtype`
values -- never as item-code first letters, which cannot classify
correctly (prefix `Y` spans revenue, expenditure and balance codes, and
prefix `X` does the same):
* `"general"` (default) -- Census General Revenue: `own_source` +
`federal` + `state` + `local_aid`. The manual defines this concept by
subtraction (section 4.3: *"General revenue comprises all revenue
except that classified as liquor store, utility, or insurance trust
revenue"*), so utility (`A91`-`A94`), liquor store (`A90`) and
insurance trust revenue are all excluded.
* `"total"` -- Census Total Revenue: every revenue subtype, i.e.
`general` plus utility, liquor store, and insurance trust revenue
(`Y01`/`Y02`/`Y04`/`Y11`/`Y12`/`Y51`/`Y52` and the employee-retirement
`X01`/`X02`/`X05`/`X08`).
The two are related by Census's own identity, `Total Revenue = General +
Utility + Liquor Store + Insurance Trust`.
Note that the employee-retirement (`X`) codes stop at FY2016, when those
systems moved out of the annual finance file into the separate Annual
Survey of Public Pensions, so a `"total"` series steps down at the
FY2016/FY2017 seam for reasons that are about collection scope rather
than revenue (series breaks `SB197`-`SB202`).}
\item{complete}{If `TRUE`, fill the requested grid so that a cell the
corpus does not carry still appears, labelled with **why** it is
missing, and add a `value_source` column to every row:
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+281
View File
@@ -0,0 +1,281 @@
# `cog_balances()` — a reader surface for cash and security holdings
**Issue:** `uscogdata#25` requirement 2 · **Downstream:** `cog-api#26`
**Date:** 2026-08-03 · **Status:** design, awaiting approval
Requirement 1 of `uscogdata#25` (no `balance` row may reach a money verb) shipped
with `#11`/`#12` and is asserted at both view and verb level. This spec covers
requirement 2 only: a way to query holdings.
## Decision: a verb, not an argument
`cog_balances()`, parallel to `cog_spending()` / `cog_revenue()`.
Holdings are a **stock** — a balance at a point in time — while the money verbs
return **flows** over a fiscal year. The flow verbs' whole argument vocabulary
is meaningless for a stock: `expenditure_concept` / `revenue_concept` describe
which flows Census aggregates into a published total, and `complete=` fills a
grid of fiscal-year cells. Overloading a money verb would put a stock behind
arguments that all assume a flow.
## The 14 codes
Measured against the published corpus 2026-08-03, not transcribed from the
issue. `year_min`/`year_max` are observed row extents.
| `balance_subtype` | `category` | codes | observed years |
|---|---|---|---|
| `general` | Fund Balances | `W01`, `W31`, `W61` | 2012–2021 |
| `employee_retirement` | Retirement System Holdings | `X21`, `X42`, `X44` | 1967–2016 |
| | | `X47` | 1988–2016 |
| | | `X30`, `Z77`, `Z78` | 2012–2016 |
| `unemployment_trust` | Insurance Trust Balances | `Y07`, `Y08` | 1967–2023 |
| `workers_comp_trust` | Insurance Trust Balances | `Y21` | 2012–2023 |
| `other_insurance_trust` | Insurance Trust Balances | `Y61` | 2012–2023 |
## Architecture
### Two new views
Mirroring the `revenue_long` / `revenue_annotated` pair exactly:
- `inst/sql/26-balance_long.sql` — `category_type = 'balance' AND NOT is_aggregate`
- `inst/sql/46-balance_annotated.sql` — joins `canonical_fips_xwalk` and
`summary_categories`, exposing `category`, `category_type`, `balance_subtype`
`.register_views()` globs `inst/sql/*.sql` in sorted order, so both register
with no new registration code.
### A third gate list in `R/views.R`
`CREATE VIEW` resolves its source schema eagerly, so a missing **column** fails
at registration time, not at query time. `46-balance_annotated.sql` selects
`c.balance_subtype`, which exists only on corpora built after pipeline `#76`/`#77`.
That arrived without a `schema_version` bump, so neither existing gate applies:
`.harmonization_view_files` keys on `schema_version`, `.representation_view_files`
on the presence of a *file*. The discriminator here is a **column on an existing
table**.
```r
.balance_view_files <- c("26-balance_long.sql", "46-balance_annotated.sql")
```
gated by probing `summary_categories` for `balance_subtype`, with
`cog_balances()` erroring cleanly via `.require_balance_support()` on an older
corpus — mirroring how `.require_schema_v5()` gates the harmonized views.
### `R/balances.R` — a dedicated path, not `.verb_spendrev()`
`.verb_spendrev()` is 825 lines whose concept scoping, intergovernmental leg and
`complete=` grid are all flow-specific, and four verbs depend on it. Threading a
third mode through it adds branching to shared code for no reuse benefit.
Reused unchanged: `.build_provenance()`, `.build_series_break_refs()`,
`.build_corpus_break_refs()`, the population join, `.inflate()`, and
`.coerce_govid_input()`.
Following the package's real two-layer convention: **view definitions** live in
`inst/sql/`; **query construction** is inline `sprintf()` in R, as in
`.verb_spendrev()`. (`CLAUDE.md` currently states "never inline SQL strings in R
files", which the verb layer has never obeyed. Corrected in a separate commit —
see Out of scope.)
## Signature
```r
cog_balances(govid, years,
category = NULL, # Fund Balances | Insurance Trust Balances |
# Retirement System Holdings
per_capita = FALSE,
adjust_to_year = NULL,
basis = c("harmonized", "raw"),
recipe = NULL)
```
Returns a `tbl_df` with a `provenance` attribute, like every other verb.
**Absent by design:** `expenditure_concept`, `revenue_concept`, `complete`,
and `subtype` — see below.
**`per_capita` is offered.** Holdings per resident is a real measure (pension
assets per capita, fund balance per resident). The roxygen `@param` states
plainly that this is a *stock per resident* and is **not** comparable to
`cog_spending()`'s per-capita figures.
**`basis` is currently a no-op** — `harmonization_map` has zero balance-code
rows, so harmonized and raw are identical for holdings. Kept for uniformity
with the money verbs (the API would otherwise special-case), and
`provenance$basis_note` says so outright rather than letting it look meaningful.
**`recipe` ships in v1 and works.** The two holdings recipes bridge the wide era
to the modern one:
```
cash_securities_z77_wide = X40 (1967-2011) + Z77 (2012-2023)
cash_securities_z78_wide = X41 (1967-2011) + Z78 (2012-2023)
```
`X40`/`X41` carry ~42,700 rows that are **100% `is_aggregate = TRUE`**, so they
are invisible to `balance_long`, which filters `NOT is_aggregate` like every
other basis view. That is by design, not a defect:
`cog_pipeline/docs/phase_r_harmonization_review.md` § 0.2 records that the wide
era exposes these split families *only* as aggregates, and that the recipe join
must therefore **not** filter `is_aggregate` — safe by construction, because
wide rows (≤2011) are aggregate-only, modern rows (2012+) are leaf-only, and
every component is year-scoped, so no double-count is possible. § 1 records the
matching decision that the planned `X40→Z77` harmonization *map* rows were
dropped and the continuity ships as recipes instead, which is why
`harmonization_map` has no balance-code rows.
The reader already implements this (`R/recipes.R`, `R/spending.R`), and it is
verified rather than assumed: `corrections_combined` for FY2007 — a recipe whose
wide leg `E05` is likewise aggregate-only — returns $906,743,000 against the
live corpus. So a recipe query reaches rows the verb's own view cannot, exactly
as intended.
### No `subtype` argument: `category` is a strict coarsening
`balance` is the only `category_type` in which `category` and the subtype column
are **not** orthogonal. Measured against the published crosswalk:
| `category_type` | subtypes spanning more than one category |
|---|---|
| expenditure | 5 of 6 (`operations`, `capital`, `interest`, `assistance`, `intergovernmental`) |
| revenue | 1 of 7 (`own_source`) |
| **balance** | **0 of 5** |
For expenditure the two axes are a genuine cross-tab — *function* (Police, Fire)
× *economic character* (operations, capital) — so both earn their place. For
balance the relation is a strict tree:
```
Fund Balances = {general} W01 W31 W61
Retirement System Holdings = {employee_retirement} X21 X30 X42 X44 X47 Z77 Z78
Insurance Trust Balances = {unemployment_trust,
workers_comp_trust,
other_insurance_trust} Y07 Y08 Y21 Y61
```
Exposing both would therefore admit no useful combination. Of the 15 possible
pairs, 3 are redundant (the subtype already implies its category) and **12 are
guaranteed empty for every government in every year** — and an impossible query
would fail by returning an empty tibble, which reads as "this government holds
none" rather than "you asked a contradiction."
Dropping `subtype` also keeps the verb aligned with the rest of the package: no
uscogdata verb exposes a subtype argument. `subtype_col` is internal plumbing in
`.verb_spendrev()`, and the API layers its own `subtype` row filter on top
(`api/R/handlers_governments.R`). `cog-api#26` can do exactly that for
`/balances`.
`#25`'s hard requirement is still met — `category = "Fund Balances"` *is* the
`general` family, precisely `W01`/`W31`/`W61`, in one filter. The only loss is
isolating one of the three insurance funds in a single argument;
`balance_subtype` remains a returned column, so that is one `dplyr::filter()`
away.
## Caveat surfacing
`provenance$balance_caveats`, always present, plus one `cli_inform()` per
session per caveat class when a query actually touches an affected family or
year. Structured so `cog-api#26` can forward the fields verbatim.
Verified against `series_breaks.csv`, not assumed:
| # | Caveat | Covered by existing machinery? |
|---|---|---|
| 1 | Gross holdings, **not GAAP fund balance**; no liabilities netted | No — a constant, new field `not_gaap = TRUE` |
| 2 | `W` is FY2012–2021 only | No — new `coverage_window`, **computed** from the corpus |
| 3 | `X`/`Z` holdings end FY2016 | **Not yet.** No `series_breaks` row exists at 2016/2017 for `Z77`/`Z78`/`X30`. Reader surfaces it via `coverage_window`; flows through `series_break_refs` once the upstream entry lands (see Out of scope) |
| 4 | `X40`/`X41` book → market at FY2002 | **Yes**, via `SB195`/`SB196` on `fin_code` `X40`/`X41`, under **two** conditions: a `recipe` query (the only path that observes those codes) **and** a year span that crosses FY2002. Asserted in the tests rather than assumed |
On caveat 4's second condition: `.build_series_break_refs()` matches
`break_year BETWEEN min(years) AND max(years)`, so a request spanning only
2011–2012 does **not** surface `SB195`. That is correct, not a gap — such a
series sits entirely after the change, on one consistent basis, and flagging a
break it never crosses would be noise. The same rule is applied deliberately in
`.build_corpus_break_refs()`. An earlier draft of this row omitted the span
condition and overclaimed.
`coverage_window` is derived per observed subtype family from the corpus, never
hardcoded, so it stays correct as the corpus grows.
`series_break_refs` and `corpus_break_refs` are otherwise populated by the
existing code-driven builders and need no change.
## Testing
New `tests/testthat/test-balances.R`. The bundled fixture covers all four
fixture years — `W` in 2012/2019/2020, the `X`/`Z` family in 2011/2012, `Y`
throughout — so every test below runs offline.
- **Inverse guard.** No flow code ever appears in `cog_balances()`, complementing
the already-asserted forward guard. Absence is verified against the raw corpus
via `read_parquet` on `data/long`, never through the verb that creates it.
- **FY2016 seam.** The `X`/`Z` family is present in 2012 and absent in 2019;
`coverage_window` reports the termination and the console message fires once.
- **Caveats.** `not_gaap` is always `TRUE`; `coverage_window` matches the
measured table above; the FY2002 valuation caveat fires only when the year
range crosses 2002 *and* touches `employee_retirement`.
- **`per_capita`.** `amt_per_capita_nominal == amt_nominal / population`.
- **`category = "Fund Balances"` is the `general` family.** Returns exactly
`W01`/`W31`/`W61` and nothing else — `#25`'s one-filter requirement, asserted
rather than assumed.
- **The hierarchy holds.** Every `balance_subtype` in the crosswalk maps to
exactly one `category`. Asserted against the crosswalk so that an upstream
change breaking the tree — which would silently make `category` lossy —
fails here rather than in a user's analysis.
- **`recipe` bridges the wide era.** `cash_securities_z77_wide` returns the
`X40` leg for a pre-2012 year, proving the aggregate-only wide rows are
reached — the property `phase_r_harmonization_review.md` § 0.2 depends on. A
regression here would silently truncate a 45-year series to five.
- **`SB195`/`SB196` reach the user on that path.** A `recipe` query spanning
FY2002 carries both in `provenance$series_break_refs`, so the book → market
basis change is disclosed wherever `X40`/`X41` are actually observed.
- **Gating.** `.require_balance_support()` errors cleanly on a corpus whose
`summary_categories` lacks `balance_subtype`.
## Out of scope, tracked separately
1. **Pipeline issue (new), non-blocking.** Catalogue the FY2016 termination of
the seven holdings codes in `series_breaks.csv`. There is currently **no**
entry at 2016/2017 for `Z77`/`Z78`/`X30`, although
`docs/phase_r_harmonization_review.md` § 2 identified the gap and recommended
exactly this — *"candidate new `series_breaks.csv` entries (recommend
`with_caution` documentation rows, no map action)"*. The follow-through never
happened. `SB197`–`SB202` set the precedent, giving the analogous X-flow
codes `coverage_restricted` + `with_caution` at 2017; `with_caution` is also
what keeps this out of the `joinable = "no"` identity-change rule, which
would otherwise oblige a harmonization-map row.
Verify the break corpus-wide and census-to-census before writing the rows.
`cog_balances()` does not wait on this — caveat 3 is covered reader-side by
`coverage_window` meanwhile, and the entry simply adds a second, catalogued
signpost when it lands.
**Superseded:** an earlier draft of this spec proposed adding
`summary_categories` rows for `X40`/`X41` and treated `recipe=` as blocked.
Both were wrong. `X40`/`X41` are deliberately aggregate-only per
`phase_r_harmonization_review.md` § 0.2, the dropped harmonization-map rows
are the documented § 1 decision, and the recipe path reaches them by design.
2. **`cog-api#26`.** Adds `/balances` in all three required places — handler,
`param_contract`, and the `plumber.R` route signature. Lands after this.
**Two contract facts the API must carry forward**, both settled during
implementation and easy to get wrong from the outside:
- `provenance$balance_caveats$coverage_window` is **corpus-scoped, not
result-scoped**. It reports the observed year extent of *every* balance
subtype in the corpus, not only the subtypes a given query returned — so a
`category = "Fund Balances"` query still returns all five windows. That is
deliberate: the windows describe what the corpus holds, which is what a
consumer needs in order to know what it did *not* ask for. The sibling
field `truncated` is the result-scoped one. Documented in
`inst/schemas/provenance-v1.json` and mutation-guarded against silent
inversion.
- `balance_caveats` appears **only** on `cog_balances()` results. It is
absent from `cog_spending()`/`cog_revenue()` provenance, and the schema
says so — an API layer that assumes it is universal will read `NULL`.
3. **`uscogdata/CLAUDE.md` refresh.** Separate commit. It is stale: it claims 7
SQL views (there are 21), 181 tests (716), a two-year fixture (four years),
and a "never inline SQL" rule the verb layer does not follow.
+286
View File
@@ -0,0 +1,286 @@
# `uscogdata` 0.1.0 — public release
**Date:** 2026-08-08 · **Status:** design, awaiting approval
**Scope:** release-readiness, README, NEWS. Distribution mechanics recorded here as
decided, sequenced after the package is clean.
`uscogdata` is feature-complete and the corpus it reads has been public on
HuggingFace since 2026-08-07 (294 downloads as of this writing). The API built on
it is live. What does not exist is a public *package*: the repo is private, there
is no install path, and — measured, not assumed — **a stranger who installed it
today could not read the corpus at all.**
This spec covers making that untrue.
## Decisions locked
| Decision | Choice |
|---|---|
| Canonical source | `gitea.civilytics.org/Civilytics/uscogdata`, flipped public |
| Public mirror | `github.com/civilytics/uscogdata` — issues, PRs, multi-OS check, CDN |
| Mirror mechanism | Gitea Actions non-force `git push` (not a push mirror) |
| Binaries | `civilytics.r-universe.dev`, registry pinned to a release tag |
| Author of record | Jared E. Knowles `<jared@civilytics.com>`, ORCID `0000-0003-0005-9478` |
| Copyright | Civilytics Consulting LLC (`cph`, `fnd`) |
| License | MIT (package) · CC-BY-4.0 (corpus) |
| Corrections intake | Deferred — see *Out of scope* |
| Other packages | Parked until this one walks the path end to end |
## P0 — the corpus is unreachable
Two independent faults, either of which alone is fatal.
**No corpus URL exists.** `R/config.R` defaults to the literal
`REPLACE_WITH_SHARE_TOKEN` sentinel, and no file in the repo supplies a working
one. A new user calling any verb gets `uscogdata_url_not_configured` with no path
to resolution.
**Remote reads are broken regardless.** Every partitioned view globs:
```sql
FROM read_parquet('{url}data/long/**/*.parquet', hive_partitioning = true)
```
DuckDB 1.5.5 refuses globs over generic HTTP. Its suggested
`allow_asterisks_in_http_paths` escape hatch does not help — it forwards the
literal `**/*` as a filename and 404s, because plain HTTP exposes no directory
listing to expand against.
The package therefore works only against a **local path**. That is how the API
runs it (`CORPUS_HOST_PATH` is a host mount on maxwell) and how the tests run
(bundled fixture), which is why the fault went unnoticed. The README's headline
claim — *"Reads the published corpus directly from Nextcloud via DuckDB httpfs —
no local bulk downloads required"* — is currently false.
### Fix: enumerate from the manifest, do not glob
`manifest.json` already lists every partition under `files.long_partitions[]`
with `path`, `year`, `sha256`, `row_count` and `size_bytes` — 56 of them.
Substituting an explicit file list for the glob was measured against the
published corpus on 2026-08-08:
| Path | Result |
|---|---|
| `https://…/data/long/**/*.parquet` (default) | error — globs unsupported over HTTP |
| same, `allow_asterisks_in_http_paths = true` | error — literal `**/*` 404s |
| `hf://datasets/civilytics/us-cog-finance/…` glob | 46,148,034 rows |
| **explicit list over plain https** | **46,148,034 rows** |
`hive_partitioning = true` still recovers `year` from the paths under
enumeration, so no downstream view or verb changes.
Enumeration is preferred over `hf://` deliberately. It is **host-agnostic** —
Nextcloud, HuggingFace, or any static server take the same code path — where
`hf://` would tie the default to one vendor's protocol and still need
special-casing, since manifest fetching goes through `httr2`, which cannot speak
`hf://`. Enumeration also *removes* a dependency (globbing) rather than adding
one, and the manifest's per-file `sha256` becomes available for integrity
checking later.
Views are registered from `inst/sql/` with `{url}` substitution in
`R/views.R:.register_views()`. The list must be built once per session from the
already-fetched manifest and substituted the same way, so the change is confined
to view registration and does not touch verb code.
### Fix: ship a working default
`R/config.R`'s default becomes the public HuggingFace `resolve/main/` URL:
CC-BY-4.0, no token to publish, CDN-backed, and it keeps maxwell's uplink out of
the path — the same reasoning behind the GitHub mirror and r-universe.
This means `library(uscogdata)` followed by a verb works with **zero
configuration**, which is what makes the package demonstrable in a README and
later in a post. `USCOGDATA_URL` and `options(uscogdata.url=)` continue to
override, so the Nextcloud copy and local mirrors are unaffected.
The `uscogdata_url_not_configured` error class stays — it still fires for an
explicitly-set empty or placeholder URL — but ceases to be the default
experience.
### Consequence: `cog_mirror()` is promoted
Measured cost of the remote default, from efron on a good connection:
| | |
|---|---|
| Whole corpus | **190.6 MB**, 56 partitions, 46,148,034 rows, FY1967–FY2024 |
| One government, one year | 1.5 s |
| One government, all 56 years | 2.8 s |
| Disk written | **0.00 MB** — range requests only; `external_file_cache` is in-memory |
Nothing persists locally beyond the shared `httpfs` extension in `~/.duckdb` (a
few MB, once per machine, across all DuckDB use). Costs are RAM and per-query
bandwidth, since nothing caches between sessions.
Those timings are raw scans. Real verbs additionally join crosswalks, resolve
categories and assemble provenance, so end-to-end verb latency will be higher and
**must be re-measured once the fix lands** — it cannot be measured today.
The corpus being only 190.6 MB makes `cog_mirror()` a first-class option rather
than a developer footnote. The README presents **both paths**:
- **Remote (default, zero setup)** — trying it out, teaching, one-off questions.
- **Mirrored (`cog_mirror()`, 190 MB once)** — repeated or heavy analysis,
offline work, reproducibility, or preferring not to depend on HuggingFace.
The second is also the honest answer to the vendor-dependency question raised by
defaulting to HuggingFace: **the escape hatch is one function call and 190 MB**,
after which no analysis touches an external service. The README says so
explicitly. That is the difference between a convenience default and lock-in.
## Release-readiness fixes
| # | Issue | Fix |
|---|---|---|
| 1 | `MaxCorpusSchema: 5` in DESCRIPTION; `.validate_schema()` accepts `4,5,6,7`; published corpus is **7** | `MaxCorpusSchema: 7` |
| 2 | `^vignettes$` in `.Rbuildignore` — both vignettes absent from the installed package, while README tells users to run `vignette("total-spending")` | Remove `^vignettes$`, `^doc$`, `^Meta$`. Both vignettes build offline (`total-spending` reads the bundled fixture; `population-denominators` is `eval = FALSE`) |
| 3 | `_pkgdown.yml` reference index covers 6 of 14 exports — pkgdown errors on missing topics | Add `cog_categories`, `cog_explain`, `cog_find_peers`, `cog_geographic_rollup`, `cog_manifest`, `cog_mirror`, `cog_peer_compare`, `cog_recipes`; set `url:` |
| 4 | No `URL:` / `BugReports:` in DESCRIPTION | Add both, pointing at the GitHub mirror |
| 5 | No `LICENSE.md`; `LICENSE` holder reads `Civilytics` | `usethis::use_mit_license("Civilytics Consulting LLC")` |
| 6 | README instructs stripping the fixture at release | Delete that section — see below |
| 7 | `Authors@R` is an org with no human | Jared E. Knowles `aut`/`cre` + ORCID; Civilytics Consulting LLC `cph`/`fnd` |
**On #6.** The advice to add `^inst/extdata/fixture_corpus$` to `.Rbuildignore`
is CRAN-sized thinking (5 MB limit) and this package is not going to CRAN.
Stripping the 15 MB fixture would break `total-spending.Rmd`, which reads from
it, and would leave r-universe and GitHub Actions unable to run the 28 test files
without a corpus credential. **The fixture is what lets `R CMD check` pass
anywhere with zero secrets** — precisely what public CI needs. It ships.
## README
The current README addresses someone standing inside the repo tree: status reads
"Under active development (Phase 2 of the cog_pipeline project)", it points at
`../cog_pipeline/docs/reader-specification.md`, the install line is commented
out, and developer, testing and release sections sit above anything a user needs.
Restructured around a stranger, in this order:
1. **What this is** — one paragraph, and what the corpus covers (types 0–3,
FY1967–FY2024, 46M rows, 190.6 MB).
2. **Install** — r-universe first (binaries), git second.
3. **Quickstart that actually runs** — resolve a government, get its history,
print provenance. No configuration step.
4. **Two ways to read the corpus** — remote default vs `cog_mirror()`, with the
measured numbers and the independence note.
5. **Amounts are in full US dollars** — kept near the top. This is the errata
most likely to produce a wrong answer that looks plausible.
6. **Concepts** — primary/direct/total spending, general/total revenue,
coverage. Condensed, linking to the vignettes for the full treatment.
7. **How to cite** — `citation("uscogdata")`, corpus CC-BY-4.0 attribution.
8. **Contributing** — canonical-on-Gitea PR flow.
Developer notes, testing instructions and release procedure move to
`CONTRIBUTING.md`. Every path reference to a sibling repo is removed or replaced
with a URL that resolves for someone who has only this repo.
## NEWS.md
The current NEWS is a pre-release churn log: changes described relative to states
no user has seen ("Breaking: corpus schema_version 4", "the package now
requires…"), newest-first across the package's entire pre-release development
(2026-04-23 to 2026-08-04, 140 commits). To a newcomer evaluating whether to
depend on the package, it reads as instability.
**0.1.0 is rewritten as an initial release**: what the package does, what the
corpus covers, and the caveats that are genuinely load-bearing. The pre-release
history is not preserved in NEWS — it is in git, where it belongs.
The substantive content is migrated, not deleted. These are hard-won and belong
in documentation rather than buried in a changelog:
| Content | Destination |
|---|---|
| Coverage disclosure on multi-government aggregates (census vs sample years) | README concepts + `cog_geographic_rollup()` docs |
| `complete = TRUE` three-way absence semantics (`reported` / `census_zero` / `not_reported`) | `cog_spending()` / `cog_revenue()` docs |
| Series-break and corpus-break surfacing | README + `cog_explain()` docs |
| $1,000s → full dollars conversion | README, already prominent |
| Per-year F-33 population denominators | `population-denominators` vignette, already there |
This also makes NEWS reusable as raw material for the release announcement,
which is the stated downstream purpose.
## Distribution mechanics
Recorded as decided; executed after the package is clean and checks are green.
**Sequence matters.** r-universe publishes check results the moment a package is
registered. Registering before the fixes above land means a red badge on day one,
which is a worse first impression than a week's delay.
1. `gitleaks` over full history. A coarse grep found nothing across 140 commits
and the default corpus URL is still the placeholder sentinel, but a proper
scan is the gate on an irreversible action.
2. Flip the Gitea repo public. Disable Gitea issues on it, so there is exactly
one inbox.
3. Create `github.com/civilytics/uscogdata`. Add `.github/workflows/` for the
Windows/macOS/Linux `R CMD check` matrix — the platforms the Gitea runner
cannot provide, and which this package has never been tested on despite
depending on duckdb and httr2. Gitea reads `.gitea/workflows`, GitHub reads
`.github/workflows`; both live in one tree without colliding.
4. Gitea Actions workflow pushing to GitHub **without `--force`**, so divergence
fails loudly in CI rather than silently overwriting.
5. Add `jared@civilytics.com` as a verified secondary email on the GitHub
account — r-universe links maintainer identity by matching DESCRIPTION's email
against registered GitHub emails, and the association only takes effect on the
next build.
6. Tag `v0.1.0`. Create `github.com/civilytics/civilytics.r-universe.dev` with a
`packages.json` pinned to the tag, pointing at the GitHub mirror rather than
Gitea so clone traffic stays off maxwell. Install the r-universe app.
### PR flow
Never press Merge on GitHub. A merge there is overwritten by the next sync, the
PR still displays "Merged", and nothing says otherwise.
```sh
git remote add github https://github.com/civilytics/uscogdata.git
git config --add remote.github.fetch '+refs/pull/*/head:refs/remotes/github/pr/*'
git fetch github
git switch -c pr-42 github/pr/42 # test
git switch main && git merge --no-ff pr-42
git push origin main # Gitea -> mirror -> GitHub
```
GitHub auto-closes a PR as merged once its head commit becomes an ancestor of the
base branch, so `--no-ff` — which preserves the contributor's SHAs — makes the PR
close itself when the mirror pushes. **For external PRs, merge; do not squash or
rebase.** Squashing rewrites the SHAs, the auto-close never fires, and closing by
hand reads to a first-time contributor as rejection.
`CONTRIBUTING.md` states this, and a GitHub Action comments it on incoming PRs.
No CLA; no DCO.
## Verification
The release is not done until all of these pass:
1. `R CMD check --as-cran` clean on Linux, and on Windows and macOS via the
GitHub matrix. This package has never been checked on the latter two.
2. Full test suite (28 files) green against the **bundled fixture**, offline,
with no credentials — the property public CI depends on.
3. Full test suite green against the **live corpus**, which additionally
exercises the enumeration fix that the fixture's local path cannot.
4. `pkgdown::build_site()` completes.
5. Both vignettes present in the built tarball and
`vignette("total-spending", package = "uscogdata")` resolves from an
installed copy.
6. **Cold-start check on a machine that has never seen this package:** install
from r-universe, `library(uscogdata)`, run the README quickstart verbatim with
no environment variables set. This is the only test that catches the P0 class
of fault, and its absence is why the fault survived.
7. End-to-end verb latency re-measured against the live corpus and the README's
numbers updated if they moved.
## Out of scope
- **Corrections intake.** Deferred by decision. Consequence: the release cannot
invite data-error reports or make the "traceable and correctable" claim that
most distinguishes this corpus from Census's own files. `BugReports:` points at
package issues only. A verified correction should eventually terminate as a
`lineage_event` or `series_break` row so it propagates through provenance to
every consumer — that design is unstarted.
- **Announcement posts.** Deferred. The API announcement is gated on corrections
landing and merits a Civic Pulse edition.
- **The rest of the R package backlog.** Parked until this one completes the path.
- **`cog_pipeline` publication.** Stays private.
+33
View File
@@ -154,3 +154,36 @@ with_corpus_missing_ig_categories <- function(code) {
}, add = TRUE)
force(code)
}
# Copy the bundled fixture to a temp dir with summary_categories.parquet
# rewritten to DROP the balance_subtype column, then run `code` against it.
# Models a corpus published before cog_pipeline #76/#77. schema_version is
# left untouched deliberately: that change shipped without a version bump, so
# column presence is the only honest signal -- this helper is what proves the
# package keys off it. Mirrors with_corpus_missing_ig_categories().
with_corpus_missing_balance_subtype <- function(code) {
src <- fixture_corpus_path()
tmp <- withr::local_tempdir(.local_envir = parent.frame())
file.copy(list.files(src, full.names = TRUE), tmp, recursive = TRUE)
cats_path <- file.path(tmp, "data", "summary_categories.parquet")
filtered_path <- file.path(tmp, "data", "summary_categories_filtered.parquet")
write_con <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
DBI::dbExecute(write_con, sprintf(
"COPY (SELECT * EXCLUDE (balance_subtype) FROM read_parquet(%s))
TO %s (FORMAT PARQUET)",
uscogdata:::.sql_lit_chr(cats_path), uscogdata:::.sql_lit_chr(filtered_path)
))
file.remove(cats_path)
file.rename(filtered_path, cats_path)
old_url <- Sys.getenv("USCOGDATA_URL", unset = NA)
uscogdata:::cog_close()
Sys.setenv(USCOGDATA_URL = paste0(tmp, "/"))
on.exit({
uscogdata:::cog_close()
if (is.na(old_url)) Sys.unsetenv("USCOGDATA_URL") else Sys.setenv(USCOGDATA_URL = old_url)
}, add = TRUE)
force(code)
}
+539
View File
@@ -0,0 +1,539 @@
test_that("balance views register and carry only balance codes", {
skip_if_no_corpus()
con <- cog_open()
on.exit(cog_close())
views <- DBI::dbGetQuery(con,
"SELECT table_name FROM information_schema.tables
WHERE table_schema = 'main' AND table_type = 'VIEW'"
)$table_name
expect_true(all(c("balance_long", "balance_annotated") %in% views))
# Every item_code in balance_long is a category_type = 'balance' member.
leak <- DBI::dbGetQuery(con,
"SELECT COUNT(*) AS n FROM balance_long
WHERE item_code NOT IN (
SELECT item_code FROM summary_categories WHERE category_type = 'balance')"
)$n
expect_identical(as.integer(leak), 0L)
# balance_annotated exposes the subtype column the verb groups on.
cols <- DBI::dbGetQuery(con,
"SELECT column_name FROM information_schema.columns
WHERE table_name = 'balance_annotated'"
)$column_name
expect_true(all(c("category", "category_type", "balance_subtype") %in% cols))
})
test_that("inst/sql/26-balance_long.sql enforces NOT is_aggregate (real SQL text, synthetic parquet)", {
# Every category_type = 'balance' item_code in the bundled fixture has
# is_aggregate = FALSE for every row of every year -- there is no real row
# that would be excluded ONLY by the `AND NOT is_aggregate` predicate. An
# assertion against the live fixture (`WHERE is_aggregate` returns 0) is
# therefore vacuous: it passes identically whether or not the view's
# predicate is present. As with the 22-/23- and 24-/25- tests above, this
# reads the real inst/sql/26-balance_long.sql text off disk and executes it
# -- plus its 10-long.sql / 11-summary_categories.sql dependencies -- against
# a synthetic hive-partitioned parquet tree that DOES contain an aggregate
# row under a real balance item_code (W01), so a regression that drops the
# predicate changes which rows survive.
skip_if_no_corpus()
tmp <- withr::local_tempdir()
part_dir <- file.path(tmp, "data", "long", "year=2004")
dir.create(part_dir, recursive = TRUE)
part_path <- file.path(part_dir, "part-0.parquet")
write_con <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
DBI::dbExecute(write_con, sprintf("
COPY (
SELECT * FROM (VALUES
('bal-A', 'W01', 100, false), -- control: ordinary balance row, survives
('bal-B', 'W01', 999999, true) -- excluded ONLY by `NOT is_aggregate`
) AS t(canonical_govid, item_code, amt, is_aggregate)
) TO %s (FORMAT PARQUET)
", uscogdata:::.sql_lit_chr(part_path)))
DBI::dbExecute(write_con, sprintf("
COPY (
SELECT * FROM (VALUES
('W01', 'Fund Balances', 'balance', NULL, NULL, 'general')
) AS t(item_code, category, category_type, spend_subtype, revenue_subtype, balance_subtype)
) TO %s (FORMAT PARQUET)
", uscogdata:::.sql_lit_chr(file.path(tmp, "data", "summary_categories.parquet"))))
sql_dir <- system.file("sql", package = "uscogdata")
.read_view_sql <- function(filename) {
txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
}
con <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
DBI::dbExecute(con, .read_view_sql("10-long.sql"))
DBI::dbExecute(con, .read_view_sql("11-summary_categories.sql"))
DBI::dbExecute(con, .read_view_sql("26-balance_long.sql"))
rows <- DBI::dbGetQuery(con,
"SELECT canonical_govid, item_code, amt FROM balance_long ORDER BY canonical_govid"
)
expect_equal(nrow(rows), 1L)
expect_equal(rows$canonical_govid, "bal-A")
expect_equal(rows$amt, 100)
})
test_that("balance views are skipped on a corpus without balance_subtype", {
skip_if_no_corpus()
with_corpus_missing_balance_subtype({
con <- cog_open()
on.exit(cog_close())
views <- DBI::dbGetQuery(con,
"SELECT table_name FROM information_schema.tables
WHERE table_schema = 'main' AND table_type = 'VIEW'"
)$table_name
# Registration must SKIP them, not error -- an older corpus stays usable.
expect_false(any(c("balance_long", "balance_annotated") %in% views))
expect_true("revenue_long" %in% views)
# ...and calling the verb on such a corpus must hit
# .require_balance_support()'s curated abort (spec § Testing: "Gating"),
# not a DuckDB binder error naming a view that was never registered.
# Asserted on the CLASS: removing the guard still errors, so a bare
# expect_error() would pass on the regression.
expect_error(
cog_balances("550000227544", 2019),
class = "uscogdata_no_balance_support"
)
})
})
test_that("cog_balances returns holdings for a government that has them", {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", 2019)
expect_s3_class(r, "tbl_df")
expect_true(nrow(r) > 0L)
expect_true(all(c("year", "canonical_govid", "gov_name", "balance_subtype",
"category", "amt_nominal") %in% names(r)))
expect_identical(sort(unique(r$category)),
c("Fund Balances", "Insurance Trust Balances"))
expect_false(is.null(attr(r, "provenance")))
expect_identical(attr(r, "provenance")$verb, "cog_balances")
})
})
test_that('category = "Fund Balances" is exactly the general family', {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", 2019, category = "Fund Balances")
expect_identical(unique(r$balance_subtype), "general")
codes <- sort(unlist(strsplit(paste(r$codes_included, collapse = ","), ",")))
expect_identical(codes, c("W01", "W31", "W61"))
})
})
test_that("no flow code can reach cog_balances", {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", c(2011, 2012, 2019, 2020))
got <- unique(unlist(strsplit(paste(r$codes_included, collapse = ","), ",")))
# The expected set is read from the RAW corpus, never from the verb --
# verifying an absence through the filter that creates it proves nothing.
# A fresh, direct DuckDB connection against the raw parquet files (never
# cog_open()'s session, never balance_long/balance_annotated) reads
# parquet natively -- no arrow dependency needed (see CLAUDE.md).
con2 <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
cats_path <- file.path(fixture_corpus_path(), "data", "summary_categories.parquet")
balance_codes <- DBI::dbGetQuery(con2, sprintf(
"SELECT item_code FROM read_parquet(%s) WHERE category_type = 'balance'",
uscogdata:::.sql_lit_chr(cats_path)
))$item_code
expect_true(length(got) > 0L)
expect_true(all(got %in% balance_codes))
})
})
test_that("every balance_subtype maps to exactly one category", {
skip_if_no_corpus()
# Dropping the `subtype` argument is only safe while this tree holds. If the
# pipeline ever gives a balance subtype a second category, `category` becomes
# a lossy filter -- fail HERE rather than in a user's analysis. Read via a
# fresh direct DuckDB connection against the raw parquet file, not through
# any registered view.
con2 <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
cats_path <- file.path(fixture_corpus_path(), "data", "summary_categories.parquet")
b <- DBI::dbGetQuery(con2, sprintf(
"SELECT category, balance_subtype FROM read_parquet(%s) WHERE category_type = 'balance'",
uscogdata:::.sql_lit_chr(cats_path)
))
per_subtype <- tapply(b$category, b$balance_subtype,
function(x) length(unique(x)))
expect_true(all(per_subtype == 1L))
})
test_that("cog_balances records found + missing govids in provenance", {
skip_if_no_corpus()
with_fixture_corpus({
suppressMessages(
r <- cog_balances(c("550000227544", "XXXINVALID"), 2019)
)
prov <- attr(r, "provenance")
expect_equal(sort(prov$scope$govids_found), "550000227544")
expect_equal(sort(prov$scope$govids_missing), "XXXINVALID")
})
})
test_that("per_capita divides holdings by population", {
skip_if_no_corpus()
with_fixture_corpus({
plain <- cog_balances("550000227544", 2019, category = "Fund Balances")
pc <- cog_balances("550000227544", 2019, category = "Fund Balances",
per_capita = TRUE)
expect_true("amt_per_capita_nominal" %in% names(pc))
expect_true("pop_source" %in% names(pc))
expect_identical(pc$amt_nominal, plain$amt_nominal)
# Assert against the denominator read from the corpus, NOT against a
# quantity derived from amt_per_capita_nominal itself -- dividing the
# column back out would be tautological and would pass on any value.
pop <- DBI::dbGetQuery(cog_open(), sprintf(
"SELECT population FROM gov_population_yearly
WHERE canonical_govid = %s AND year = 2019",
uscogdata:::.sql_lit_chr("550000227544")
))$population
expect_length(pop, 1L)
expect_equal(pc$amt_per_capita_nominal, pc$amt_nominal / pop,
tolerance = 1e-8)
prov <- attr(pc, "provenance")
expect_true(prov$transformations$per_capita$applied)
})
})
test_that("adjust_to_year adds real dollars", {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", 2012, category = "Fund Balances",
adjust_to_year = 2020)
expect_true("amt_real" %in% names(r))
# 2012 dollars inflated to 2020 must exceed nominal.
expect_true(all(r$amt_real > r$amt_nominal))
prov <- attr(r, "provenance")
expect_true(prov$transformations$inflation$applied)
expect_identical(prov$transformations$inflation$base_year, 2020L)
})
})
test_that("per_capita and adjust_to_year compose", {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", 2012, category = "Fund Balances",
per_capita = TRUE, adjust_to_year = 2020)
expect_true("amt_per_capita_real" %in% names(r))
# The per-capita column must be deflated by the SAME factor as the level
# column -- this is what the ordering at R/balances.R:101-103 guarantees.
# .attach_real_dollars() silently no-ops on the per-capita leg when
# amt_per_capita_nominal does not exist yet (R/spending.R:664), so
# reversing those two calls drops this column with no error at all.
expect_equal(r$amt_per_capita_real / r$amt_per_capita_nominal,
r$amt_real / r$amt_nominal, tolerance = 1e-8)
# And the documented condition is a conjunction: adjust_to_year ALONE
# must not produce amt_per_capita_real (pins the @return wording).
r2 <- cog_balances("550000227544", 2012, category = "Fund Balances",
adjust_to_year = 2020)
expect_true("amt_real" %in% names(r2))
expect_false("amt_per_capita_real" %in% names(r2))
})
})
# --- input validation ------------------------------------------------------
test_that("cog_balances validates its inputs like the money verbs", {
skip_if_no_corpus()
with_fixture_corpus({
G <- "550000227544"
# Pinned to the message, not bare expect_error(): every one of these
# already produces *some* error or *some* quiet wrong answer today --
# years = integer(0) leaks `Parser Error ... AND year IN ()` with the
# generated SQL, recipe = c("a","b") throws "the condition has length > 1",
# and the govid/category cases return 0 rows with no error at all.
expect_error(cog_balances(G, integer(0)), "non-empty integer vector")
expect_error(cog_balances(character(0), 2019), "non-empty character vector")
expect_error(cog_balances(G, 2019, category = 5), "must be character or NULL")
expect_error(cog_balances(G, 2019, recipe = c("a", "b")),
"length-1 character string")
})
})
test_that("validation runs after govid coercion, so a data frame still works", {
skip_if_no_corpus()
with_fixture_corpus({
# .validate_verb_inputs() asserts is.character(govid); it must therefore
# run AFTER .coerce_govid_input(), never before, or the documented
# data-frame input (cog_gov_search() output) would abort.
df <- data.frame(canonical_govid = "550000227544", stringsAsFactors = FALSE)
r <- suppressMessages(cog_balances(df, 2019))
expect_true(nrow(r) > 0L)
expect_identical(unique(r$canonical_govid), "550000227544")
})
})
test_that("recipe and category are mutually exclusive", {
skip_if_no_corpus()
with_fixture_corpus({
expect_error(
cog_balances("550000227544", c(2011, 2012),
category = "Fund Balances",
recipe = "cash_securities_z77_wide"),
class = "uscogdata_recipe_category_conflict"
)
})
})
# --- recipe = : the wide-era holdings bridge -------------------------------
test_that("recipe bridges the wide era into the modern one", {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", c(2011, 2012),
recipe = "cash_securities_z77_wide")
# .run_recipe()'s SQL returns `long.year` as a DOUBLE (a corpus-wide trait,
# not specific to this recipe -- see the money-verb recipe tests, which
# only ever assert on it with expect_equal), so compare numerically rather
# than with expect_identical()'s type-strict comparison.
expect_equal(sort(r$year), c(2011, 2012))
# The 2011 leg can ONLY come from X40, which is 100% is_aggregate = TRUE
# and therefore invisible to balance_long. If the recipe path ever starts
# filtering aggregates, a 45-year series silently truncates to five --
# this is the regression guard for phase_r_harmonization_review.md § 0.2.
codes <- attr(r, "provenance")$codes_summed$observed
expect_true("X40" %in% codes)
expect_true("Z77" %in% codes)
expect_true(all(r$amt_nominal > 0))
prov <- attr(r, "provenance")
expect_identical(prov$recipe$recipe_id, "cash_securities_z77_wide")
})
})
test_that("the FY2002 book-to-market basis change is disclosed on the recipe path", {
skip_if_no_corpus()
with_fixture_corpus({
# 2002 is in the year vector deliberately, and must stay -- do not
# "simplify" this back to c(2011, 2012).
#
# .build_series_break_refs() (R/series_breaks.R, shared with every verb)
# matches breaks with `break_year BETWEEN min(years) AND max(years)`, and
# SB195's break_year is 2002. A c(2011, 2012) span never crosses the
# FY2002 book -> market change -- that whole span sits after it, on one
# consistent basis -- so NOT disclosing SB195 there is correct behaviour,
# not a gap (same reasoning as the "a request that never crosses the
# boundary is not affected by it" comment on .build_corpus_break_refs()).
#
# The property actually worth testing is: a recipe query that observes
# X40 AND spans FY2002 discloses SB195. This fixture has no 2002
# partition data for X40/Z77 (confirmed: only 2011/2012/2019/2020
# partitions exist), so including 2002 in `years` widens the
# break-matching window without changing which rows the recipe join
# returns -- verified empirically: r$year below is exactly {2011, 2012}
# whether or not 2002 is in the request (see task-4-report.md).
# Removing 2002 would silently turn this back into the non-crossing case
# above and destroy the test's purpose.
r <- cog_balances("550000227544", c(2002, 2011, 2012),
recipe = "cash_securities_z77_wide")
expect_equal(sort(r$year), c(2011, 2012))
refs <- attr(r, "provenance")$series_break_refs
# SB195 sits on fin_code X40; it can only fire where X40 is observed,
# which is exactly the recipe path.
expect_true("SB195" %in% refs)
})
})
test_that("the second holdings bridge works too", {
skip_if_no_corpus()
with_fixture_corpus({
# X41 -> Z78, the securities counterpart. Wisconsin carries X41 in 2011
# and Z78 in 2012, so both legs are exercised.
r <- cog_balances("550000227544", c(2011, 2012),
recipe = "cash_securities_z78_wide")
codes <- attr(r, "provenance")$codes_summed$observed
expect_true(all(c("X41", "Z78") %in% codes))
expect_equal(sort(r$year), c(2011, 2012))
})
})
test_that("an unknown recipe id is rejected", {
skip_if_no_corpus()
with_fixture_corpus({
# Asserted on the CLASS .validate_recipe_id() sets (R/recipes.R:84).
# Without it the test is non-discriminating: deleting the validation call
# leaves .recipe_components() returning 0 rows and comps$label[[1]]
# throwing "subscript out of bounds", which a bare expect_error() accepts
# while the user loses the curated "valid recipe ids are ..." message.
expect_error(
cog_balances("550000227544", 2019, recipe = "no_such_recipe"),
class = "uscogdata_unknown_recipe"
)
})
})
# --- balance_caveats: GAAP disclosure + measured coverage windows ----------
test_that("balance_caveats is always present and flags the GAAP distinction", {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", 2019)
cav <- attr(r, "provenance")$balance_caveats
expect_false(is.null(cav))
expect_true(cav$not_gaap)
})
})
test_that("coverage_window is computed from the corpus, not hardcoded", {
skip_if_no_corpus()
with_fixture_corpus({
r <- cog_balances("550000227544", c(2011, 2012, 2019, 2020))
cav <- attr(r, "provenance")$balance_caveats
# Read the "general" family's true year extent independently, via a
# fresh DuckDB connection against the raw parquet files (never through
# balance_long/.balance_caveats() itself, and never via arrow -- this
# package reads parquet through DuckDB only, see CLAUDE.md). Replicates
# the same predicates 26-balance_long.sql applies (category_type =
# 'balance', NOT is_aggregate) so this is a faithful, independent
# measurement rather than a re-statement of the view under test.
con2 <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
long_glob <- file.path(fixture_corpus_path(), "data", "long", "**", "*.parquet")
cats_path <- file.path(fixture_corpus_path(), "data", "summary_categories.parquet")
obs <- DBI::dbGetQuery(con2, sprintf(
"SELECT MIN(l.year) AS y0, MAX(l.year) AS y1
FROM read_parquet(%s, hive_partitioning = true) l
JOIN read_parquet(%s) c USING (item_code)
WHERE c.balance_subtype = 'general' AND NOT l.is_aggregate",
uscogdata:::.sql_lit_chr(long_glob), uscogdata:::.sql_lit_chr(cats_path)
))
expect_identical(as.integer(cav$coverage_window$general),
c(as.integer(obs$y0), as.integer(obs$y1)))
})
})
test_that("coverage_window covers every corpus subtype, not just observed ones", {
skip_if_no_corpus()
with_fixture_corpus({
# Deliberate contract (provenance-v1.json): the window block is corpus-
# scoped so a caller can ask "is there a family I missed?", while
# `truncated` is the observed-scoped field. A single-category query must
# therefore still report every balance family in the mounted corpus.
r <- cog_balances("550000227544", 2019, category = "Fund Balances")
expect_identical(unique(r$balance_subtype), "general")
con2 <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
cats_path <- file.path(fixture_corpus_path(), "data", "summary_categories.parquet")
all_subtypes <- DBI::dbGetQuery(con2, sprintf(
"SELECT DISTINCT balance_subtype FROM read_parquet(%s)
WHERE balance_subtype IS NOT NULL",
uscogdata:::.sql_lit_chr(cats_path)
))$balance_subtype
cav <- attr(r, "provenance")$balance_caveats
expect_setequal(names(cav$coverage_window), all_subtypes)
expect_true(length(all_subtypes) > 1L)
# ...while `truncated` stays scoped to what this query actually observed.
expect_true(all(cav$truncated %in% unique(r$balance_subtype)))
})
})
test_that("the corpus-constant coverage windows are memoised per session", {
skip_if_no_corpus()
with_fixture_corpus({
# The windows query has no govid/year predicate: its answer depends only
# on which corpus is mounted, so re-running the full balance_long scan on
# every call is pure waste (35% of verb runtime on the fixture). Same
# memoise-and-invalidate pattern as .uscogdata_env$manifest.
expect_null(uscogdata:::.uscogdata_env$balance_coverage_windows)
suppressMessages(cog_balances("550000227544", 2019))
memo <- uscogdata:::.uscogdata_env$balance_coverage_windows
expect_false(is.null(memo))
expect_true("general" %in% names(memo))
uscogdata:::cog_close()
expect_null(uscogdata:::.uscogdata_env$balance_coverage_windows)
})
})
test_that("a request past a family's coverage window is flagged", {
skip_if_no_corpus()
with_fixture_corpus({
# employee_retirement (X21/X30/X47/Z77/Z78) genuinely ends at FY2016 in
# the LIVE corpus -- Census moved employee retirement reporting to the
# Annual Survey of Public Pensions after that year. This bundled FIXTURE
# doesn't carry 2013-2016 at all (only 2011/2012/2019/2020 are present),
# so the family's *observed* max here is 2012, not 2016. Either way the
# requested span (2012, 2019) reaches past what the family covers in
# THIS corpus, which is what makes .balance_caveats() flag it -- the
# assertion below is about the fixture's measured window, not the FY2016
# live-corpus cutoff.
r <- cog_balances("550000227544", c(2012, 2019))
cav <- attr(r, "provenance")$balance_caveats
expect_true("employee_retirement" %in% cav$truncated)
})
})
test_that("the provenance schema documents balance_caveats", {
sch <- jsonlite::fromJSON(
system.file("schemas", "provenance-v1.json", package = "uscogdata"),
simplifyVector = FALSE
)
expect_true("balance_caveats" %in% names(sch$properties))
})
test_that("cog_explain surfaces the balance caveats", {
skip_if_no_corpus()
with_fixture_corpus({
# Asserted on the RENDERED text, not on prov$balance_caveats: the field
# is already covered above, and the once-per-session cli_inform() means
# cog_explain() is the only surface a caller who missed (or suppressed)
# the first message can still audit.
r <- suppressMessages(cog_balances("550000227544", c(2012, 2019)))
# Both streams: cli routes most of its output through conditions that
# land on stderr, so a stdout-only capture would be empty (the pattern
# used throughout test-explain.R).
out <- paste(c(capture.output(cog_explain(r)),
capture.output(cog_explain(r), type = "message")),
collapse = "\n")
expect_match(out, "GAAP")
expect_match(out, "employee_retirement")
})
})
test_that("cog_explain on a money-verb result has no balance caveat section", {
skip_if_no_corpus()
with_fixture_corpus({
r <- suppressMessages(cog_spending("550000227544", 2019))
out <- paste(c(capture.output(cog_explain(r)),
capture.output(cog_explain(r), type = "message")),
collapse = "\n")
# Guard against the capture itself being vacuous: the section must be
# absent from output that demonstrably contains the rest of the report.
expect_match(out, "Data vintage")
expect_false(grepl("GAAP", out))
})
})
test_that("the caveat message fires once per session", {
skip_if_no_corpus()
with_fixture_corpus({
expect_message(cog_balances("550000227544", 2019), "not.*GAAP")
expect_no_message(cog_balances("550000227544", 2020))
})
})
+43 -4
View File
@@ -49,12 +49,14 @@ test_that("cog_categories(type = 'revenue') returns only revenue rows", {
skip_if_no_corpus()
r <- cog_categories(type = "revenue")
expect_true(all(r$category_type == "revenue"))
# `insurance_trust` (Y01/Y02/Y04/Y11/Y12/Y51/Y52) is deliberately NOT
# own_source: Census's "General Revenue" excludes insurance trust revenue,
# and Y01 alone is $1.31T corpus-wide.
# The four non-general subtypes are deliberately NOT own_source: Census's
# General Revenue excludes insurance trust (Y01 alone is $1.31T corpus-wide,
# plus the employee-retirement X codes), utility (A91-A94) and liquor store
# (A90) revenue by definition, which is what makes both of its published
# revenue concepts computable -- see `revenue_concept` in `?cog_revenue`.
expect_true(all(r$subtype %in%
c("own_source", "federal", "state", "local_aid",
"insurance_trust")))
"insurance_trust", "utility", "liquor_store")))
})
test_that("cog_categories(pattern = ...) filters case-insensitively", {
@@ -91,3 +93,40 @@ test_that("cog_categories sorted by category_type, category, subtype", {
test_that("cog_categories rejects invalid type", {
expect_error(cog_categories(type = "both"), "type")
})
test_that("cog_categories() surfaces balance subtypes", {
skip_if_no_corpus()
with_fixture_corpus({
cc <- cog_categories()
b <- cc[cc$category_type == "balance", ]
expect_true(nrow(b) > 0L)
# Every balance row must carry its subtype. Before the COALESCE included
# balance_subtype these were all NA, which silently made the balance
# taxonomy undiscoverable -- cog-api derives its subtype vocabulary from
# this function, so an NA here becomes an unusable API parameter.
expect_false(any(is.na(b$subtype)))
# The exact set, read independently from the crosswalk rather than from
# the function under test.
con2 <- DBI::dbConnect(duckdb::duckdb())
on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
p <- file.path(fixture_corpus_path(), "data", "summary_categories.parquet")
want <- DBI::dbGetQuery(con2, sprintf(
"SELECT DISTINCT balance_subtype FROM read_parquet(%s)
WHERE category_type = 'balance' AND balance_subtype IS NOT NULL
ORDER BY 1", uscogdata:::.sql_lit_chr(p)))$balance_subtype
expect_true(length(want) > 1L)
expect_identical(sort(unique(b$subtype)), sort(want))
})
})
test_that('cog_categories(type = "balance") filters to holdings', {
skip_if_no_corpus()
with_fixture_corpus({
b <- cog_categories(type = "balance")
expect_true(nrow(b) > 0L)
expect_identical(unique(b$category_type), "balance")
expect_false(any(is.na(b$subtype)))
})
})
+12 -2
View File
@@ -111,7 +111,7 @@ test_that("cog_manifest returns the active session's parsed manifest", {
})
})
test_that(".validate_schema accepts schema_version 4, 5 and 6, rejects others", {
test_that(".validate_schema accepts schema_version 4 through 7, rejects others", {
expect_silent(uscogdata:::.validate_schema(list(schema_version = 4L)))
expect_silent(uscogdata:::.validate_schema(list(schema_version = 5L)))
# v6 = FIPS geography harmonization (2026-07-22): _code -> _asof rename +
@@ -119,12 +119,22 @@ test_that(".validate_schema accepts schema_version 4, 5 and 6, rejects others",
# renamed columns and its geography comes from the xwalk, so v6 is accepted
# without behavioural change -- see .validate_schema()'s note.
expect_silent(uscogdata:::.validate_schema(list(schema_version = 6L)))
# v7 = `data_year` APPENDED as column 29 (cog_pipeline #80, 2026-08-03), the
# most recent fiscal year contributing to a collapsed key. Appended, never
# inserted: canonical_govid stays at position 26, so nothing this package
# reads shifts. Verified against the real v7 corpus before widening the
# allow-list -- cog_spending()/cog_balances() return correctly for FY2024 AND
# for FY2012, so the new column is inert here.
expect_silent(uscogdata:::.validate_schema(list(schema_version = 7L)))
expect_error(
uscogdata:::.validate_schema(list(schema_version = 3L)),
"schema_version"
)
# The upper bound still has to be ENFORCED, not just moved. Without this the
# test would no longer prove that an unknown future schema is refused, and a
# v8 corpus with a genuinely breaking change would sail through.
expect_error(
uscogdata:::.validate_schema(list(schema_version = 7L)),
uscogdata:::.validate_schema(list(schema_version = 8L)),
"schema_version"
)
})
@@ -9,47 +9,131 @@
# a published Census revenue concept exactly the way I89 sits inside Census's
# Direct Expenditure concept (finding F-012).
#
# CAVEAT FOR WHOEVER PICKS THIS UP: the argument name below (`revenue_concept =
# "total"`) is this test's *proposal*, not a settled decision. The owner's
# 2026-07-28 resolution covers expenditure concepts only; no revenue-side
# naming has been ruled on. If the eventual argument is named differently,
# change the two calls here -- the asserted dollar invariants are what matter
# and are independent of the naming.
# RULED 2026-07-30. `revenue_concept = c("general", "total")` mirrors
# `expenditure_concept`, and the two values are Census's two published revenue
# concepts, related by the manual's own identity (section 4.3, which defines
# the first by SUBTRACTING from the second):
#
# Total Revenue = General + Utility + Liquor Store + Insurance Trust
#
# so `general` is the four general subtypes (own_source/federal/state/
# local_aid) and `total` is every revenue subtype. Naming utility (A91-A94)
# and liquor store (A90) separately is what makes BOTH computable -- before
# cog_pipeline#79 they sat in own_source, so the default was really
# "General + Utility + Liquor", a concept Census does not publish.
#
# Fixture reproducibility: Madison's own X-prefix revenue (FY1970-FY1986,
# $15,098,000 nominal, $0 thereafter) is outside the bundled fixture's year
# window (2011/2012/2019/2020), so the same invariant is asserted on Wisconsin
# state government FY2012, where the fixture carries nonzero X01/X05/X08.
# state government FY2012, where the fixture carries nonzero X01/X02/X05/X08.
test_that("cog_revenue() can return Census Total Revenue including Insurance Trust (prefix X)", {
testthat::skip("Blocked on uscogdata#12 (finding F-014)")
wi_state <- "550000227544" # WISCONSIN (state government)
# Revenue-shaped Employee Retirement codes, read from the RAW corpus rather
# than through cog_revenue(), which is the filter under test:
# X01 local employee contribution, X04/X05 contributions and transfers from
# other governments, X08 earnings on investments.
x_revenue <- wt_raw_amt(wi_state, 2012L, codes = c("X01", "X04", "X05", "X08"))
expect_equal(x_revenue, 2038800) # 615,835 + 0 + 560,382 + 862,583 ($1,000s)
# X01/X02 employee contributions, X05 contributions from other governments,
# X08 total earnings on investments.
#
# X04 and X06 are deliberately NOT in this set, though an earlier draft of
# this test included X04. Both are exhibit codes for INTRAgovernmental
# transfers (the administering government paying into its own fund), which
# X05's own definition excludes by name. Census agrees: its computed "Total
# Emp Ret Rev" for this government-year is exactly the four codes below.
x_revenue <- wt_raw_amt(wi_state, 2012L, codes = c("X01", "X02", "X05", "X08"))
expect_equal(x_revenue, 2283883) # 615,835 + 245,083 + 560,382 + 862,583
# The Y-prefix insurance trust revenue (unemployment + workers comp), which
# is the other half of the same Census concept.
y_revenue <- wt_raw_amt(wi_state, 2012L, codes = c("Y01", "Y11"))
expect_equal(y_revenue, 1259785)
general <- cog_revenue(govid = wi_state, years = 2012L)
expect_equal(attr(general, "provenance")$revenue_concept, "general")
expect_equal(sum(general$amt_nominal), 31338293000)
total <- cog_revenue(govid = wi_state, years = 2012L, revenue_concept = "total")
expect_equal(sum(total$amt_nominal) - sum(general$amt_nominal), x_revenue * 1000)
expect_equal(sum(total$amt_nominal), 33377093000)
expect_true(all(c("X01", "X05", "X08") %in% wt_codes_included(total)))
expect_equal(attr(total, "provenance")$revenue_concept, "total")
# total - general is the whole insurance trust leg, X and Y together.
# Asserted as a delta as well as a level so this stays correct however the
# utility/liquor families land (both are $0 for WI state in FY2012).
expect_equal(sum(total$amt_nominal) - sum(general$amt_nominal),
(x_revenue + y_revenue) * 1000)
expect_equal(sum(total$amt_nominal), 34881961000)
expect_true(all(c("X01", "X02", "X05", "X08") %in% wt_codes_included(total)))
# Sibling codes under the SAME first letter must stay out: X11/X12 are
# benefit payments (an expenditure) and X21/X30/X47 are cash and securities
# holdings (a balance-sheet stock). This is the F-018 point restated on the
# revenue side -- the split has to come from the crosswalk's spend_type, not
# from the letter X.
# revenue side -- the split comes from the crosswalk, not from the letter X.
expect_false(any(c("X11", "X12", "X21", "X30", "X47") %in% wt_codes_included(total)))
# Every returned row still resolves to a category. summary_categories has
# zero rows for prefix X today, so relaxing the prefix filter alone would
# produce category = NA rows -- see census_of_governments_finance_pipeline#60.
# Every returned row still resolves to a category (cog_pipeline#79 added the
# X crosswalk rows; relaxing a prefix filter alone would have produced
# category = NA rows).
expect_false(any(is.na(total$category)))
})
test_that("revenue_concept = 'general' is the default and is strict Census General Revenue", {
wi_state <- "550000227544"
default <- cog_revenue(govid = wi_state, years = 2012L)
explicit <- cog_revenue(govid = wi_state, years = 2012L,
revenue_concept = "general")
expect_equal(sum(default$amt_nominal), sum(explicit$amt_nominal))
# General Revenue excludes utility, liquor store AND insurance trust
# revenue. WI state carries $0 of utility/liquor in FY2012, so the level
# assertion above cannot see those two -- assert the subtype scope directly.
#
# A subset, not setequal: `state` means "intergovernmental revenue FROM the
# state government" (the C codes), which a STATE government does not receive
# from itself, so it is legitimately absent here.
expect_true(all(default$revenue_subtype %in%
c("own_source", "federal", "state", "local_aid")))
expect_false(any(c("utility", "liquor_store", "insurance_trust") %in%
default$revenue_subtype))
})
test_that("utility and liquor store revenue are inside `total` and outside `general`", {
# A city, where utility revenue is material: this is the case the WI state
# baseline structurally cannot exercise. Measured on the fixture, utility +
# liquor is 15.9% of what cog_revenue() returned for type-2 governments
# before the general/total split, so this is the largest behaviour change
# the concept split introduces.
con <- uscogdata:::.ensure_session()
gov <- DBI::dbGetQuery(con,
"SELECT canonical_govid, SUM(amt) amt FROM long
WHERE year = 2012 AND type = 2 AND NOT is_aggregate
AND item_code IN ('A91','A92','A93','A94')
GROUP BY 1 ORDER BY amt DESC LIMIT 1")$canonical_govid
util_raw <- wt_raw_amt(gov, 2012L, codes = c("A90", "A91", "A92", "A93", "A94"))
expect_gt(util_raw, 0)
general <- cog_revenue(govid = gov, years = 2012L)
total <- cog_revenue(govid = gov, years = 2012L, revenue_concept = "total")
expect_false(any(c("utility", "liquor_store") %in% general$revenue_subtype))
expect_true("utility" %in% total$revenue_subtype)
expect_equal(sum(total$amt_nominal) - sum(general$amt_nominal),
util_raw * 1000 +
wt_raw_amt(gov, 2012L, codes = c("Y01", "Y11", "X01", "X02",
"X05", "X08")) * 1000)
})
test_that("revenue_concept rejects unknown values and never returns a balance row", {
expect_error(
cog_revenue("550000227544", years = 2012L, revenue_concept = "gross"),
class = "uscogdata_invalid_revenue_concept"
)
# uscogdata#25 restated for the widest revenue concept: stocks are not
# flows, and `total` must not quietly admit the X/Y/W/Z balance families.
con <- uscogdata:::.ensure_session()
balance <- DBI::dbGetQuery(con,
"SELECT item_code, category FROM summary_categories WHERE category_type = 'balance'")
total <- cog_revenue("550000227544", years = 2012L, revenue_concept = "total")
expect_false(any(total$category %in% balance$category))
expect_length(intersect(wt_codes_included(total), balance$item_code), 0L)
})
+5 -5
View File
@@ -393,19 +393,19 @@ test_that("spending_long carries exactly the non-IG expenditure crosswalk codes
expect_equal(agg_count, 0)
})
test_that("revenue_long carries exactly the general-revenue crosswalk codes and excludes aggregates", {
test_that("revenue_long carries exactly the revenue crosswalk codes and excludes aggregates", {
skip_if_no_corpus()
con <- cog_open()
on.exit(cog_close())
# General Revenue scope: revenue crosswalk members minus insurance_trust
# (owner ruling 2026-07-30; an explicit wider concept is uscogdata#12).
# The view carries EVERY revenue subtype; which of Census's two published
# concepts a query returns is decided per `revenue_concept` in R
# (uscogdata#12), exactly as `expenditure_concept` narrows spending_long.
stray <- DBI::dbGetQuery(con,
"SELECT DISTINCT s.item_code
FROM revenue_long s
LEFT JOIN summary_categories c USING (item_code)
WHERE c.category_type IS DISTINCT FROM 'revenue'
OR c.revenue_subtype = 'insurance_trust'"
WHERE c.category_type IS DISTINCT FROM 'revenue'"
)$item_code
expect_length(stray, 0L)