feat: cog_balances(), a reader surface for cash and security holdings (#25) #28

Merged
jared merged 22 commits from feat/cog-balances-25 into main 2026-08-03 11:52:13 -04:00
Owner

Closes #25 (requirement 2).

Adds cog_balances(), a third verb exposing the 14 cash-and-security holding
codes (category_type = 'balance'): fund balances, retirement system holdings
and insurance trust balances.

Requirement 1 of #25 — that no balance row may reach a money verb — already
shipped with #11/#12 and is asserted at both view and verb level. This is the
query surface.

Why a verb, not an argument

Holdings are a stock — a balance at a point in time — while cog_spending()
and cog_revenue() return flows over a fiscal year. The money verbs' whole
argument vocabulary describes flows, so the verb has no expenditure_concept,
revenue_concept or complete argument and deliberately does not route through
.verb_spendrev().

It also has no subtype argument. balance is the only category_type
whose subtype column is not orthogonal to category: 5 of 6 expenditure
subtypes and 1 of 7 revenue subtypes span more than one category, but 0 of 5
balance subtypes do. Of the 15 possible (subtype, category) pairs, 3 are
redundant and 12 are guaranteed empty for every government in every year — an
impossible query would have failed as an empty tibble reading "holds none"
rather than "you asked a contradiction". category = "Fund Balances" is exactly
the general family (W01/W31/W61), so #25's one-filter requirement is met.
A test asserts the subtype→category tree still holds in the crosswalk, because
dropping subtype is only safe while it does.

Signature

cog_balances(govid, years,
             category       = NULL,
             per_capita     = FALSE,
             adjust_to_year = NULL,
             basis          = c("harmonized", "raw"),
             recipe         = NULL)

basis is accepted for uniformity but is currently a no-op — harmonization_map
carries no balance-code rows — and provenance$basis_note says so rather than
letting it look meaningful.

The wide-era bridge

recipe = "cash_securities_z77_wide" / "cash_securities_z78_wide" bridge
X40/X41 (1967–2011) to Z77/Z78 (2012–2023). X40/X41 exist in the
corpus only as is_aggregate = TRUE rows and are invisible to the
balance_annotated view by construction; .run_recipe() deliberately does not
filter aggregates, which is what makes the pre-2012 leg reachable. If that ever
regresses a 45-year series silently truncates to five while still looking valid,
so it carries a mutation-verified regression guard.

Disclosure

provenance$balance_caveats carries the GAAP-vs-gross disclosure and per-subtype
coverage windows measured from the corpus, never hardcoded, plus a
once-per-session cli message. cog_explain() renders them. SB195/SB196
(the FY2002 book→market change) are deliberately not duplicated there — they
reach users through the existing series_break_refs machinery on a recipe query
whose year span crosses 2002.

Two contract facts for cog-api#26: coverage_window is corpus-scoped, not
result-scoped
(the sibling truncated field is the result-scoped one), and
balance_caveats appears only on cog_balances() results.

Also in this PR

  • inst/schemas/provenance-v1.json documents balance_caveats.
  • .validate_verb_inputs() is now shared with cog_balances(), closing a gap
    where only 2 of 7 arguments were validated and years = integer(0) leaked raw
    DuckDB SQL to the user.
  • CLAUDE.md corrected: it claimed "never inline SQL strings in R files", which
    the verb layer has never obeyed — view definitions live in inst/sql/,
    query construction is inline sprintf() in R. Also 23 SQL views (was 7),
    788 tests (was 181), four fixture years (was two).

Test plan

  • Full suite 788 pass / 0 fail / 0 skip (was 716), offline against the
    bundled fixture. Test government is Wisconsin state govt 550000227544,
    which carries W in 2012/2019/2020, X/Z in 2011/2012, Y throughout,
    and X40/X41 in 2011 — so both recipe bridges are testable offline.
  • Absence is verified against the raw corpus via DuckDB read_parquet(),
    never through the view or verb under test.
  • Load-bearing behaviours mutation-verified: removing NOT is_aggregate
    from the view, filtering aggregates in the recipe path, and reversing the
    per-capita/inflation call order each make their covering test fail.
  • CI green (installs dependencies fresh and runs rcmdcheck, neither
    exercised locally).
Closes #25 (requirement 2). Adds `cog_balances()`, a third verb exposing the 14 cash-and-security holding codes (`category_type = 'balance'`): fund balances, retirement system holdings and insurance trust balances. Requirement 1 of #25 — that no `balance` row may reach a money verb — already shipped with #11/#12 and is asserted at both view and verb level. This is the query surface. ## Why a verb, not an argument Holdings are a **stock** — a balance at a point in time — while `cog_spending()` and `cog_revenue()` return **flows** over a fiscal year. The money verbs' whole argument vocabulary describes flows, so the verb has no `expenditure_concept`, `revenue_concept` or `complete` argument and deliberately does not route through `.verb_spendrev()`. It also has **no `subtype` argument**. `balance` is the only `category_type` whose subtype column is not orthogonal to `category`: 5 of 6 expenditure subtypes and 1 of 7 revenue subtypes span more than one category, but 0 of 5 balance subtypes do. Of the 15 possible `(subtype, category)` pairs, 3 are redundant and 12 are guaranteed empty for every government in every year — an impossible query would have failed as an empty tibble reading "holds none" rather than "you asked a contradiction". `category = "Fund Balances"` is exactly the `general` family (`W01`/`W31`/`W61`), so #25's one-filter requirement is met. A test asserts the subtype→category tree still holds in the crosswalk, because dropping `subtype` is only safe while it does. ## Signature ```r cog_balances(govid, years, category = NULL, per_capita = FALSE, adjust_to_year = NULL, basis = c("harmonized", "raw"), recipe = NULL) ``` `basis` is accepted for uniformity but is currently a no-op — `harmonization_map` carries no balance-code rows — and `provenance$basis_note` says so rather than letting it look meaningful. ## The wide-era bridge `recipe = "cash_securities_z77_wide"` / `"cash_securities_z78_wide"` bridge `X40`/`X41` (1967–2011) to `Z77`/`Z78` (2012–2023). `X40`/`X41` exist in the corpus **only** as `is_aggregate = TRUE` rows and are invisible to the `balance_annotated` view by construction; `.run_recipe()` deliberately does not filter aggregates, which is what makes the pre-2012 leg reachable. If that ever regresses a 45-year series silently truncates to five while still looking valid, so it carries a mutation-verified regression guard. ## Disclosure `provenance$balance_caveats` carries the GAAP-vs-gross disclosure and per-subtype coverage windows **measured from the corpus**, never hardcoded, plus a once-per-session `cli` message. `cog_explain()` renders them. `SB195`/`SB196` (the FY2002 book→market change) are deliberately not duplicated there — they reach users through the existing `series_break_refs` machinery on a recipe query whose year span crosses 2002. Two contract facts for `cog-api#26`: `coverage_window` is **corpus-scoped, not result-scoped** (the sibling `truncated` field is the result-scoped one), and `balance_caveats` appears only on `cog_balances()` results. ## Also in this PR - `inst/schemas/provenance-v1.json` documents `balance_caveats`. - `.validate_verb_inputs()` is now shared with `cog_balances()`, closing a gap where only 2 of 7 arguments were validated and `years = integer(0)` leaked raw DuckDB SQL to the user. - `CLAUDE.md` corrected: it claimed "never inline SQL strings in R files", which the verb layer has never obeyed — view *definitions* live in `inst/sql/`, query *construction* is inline `sprintf()` in R. Also 23 SQL views (was 7), 788 tests (was 181), four fixture years (was two). ## Test plan - [x] Full suite **788 pass / 0 fail / 0 skip** (was 716), offline against the bundled fixture. Test government is Wisconsin state govt `550000227544`, which carries `W` in 2012/2019/2020, `X`/`Z` in 2011/2012, `Y` throughout, and `X40`/`X41` in 2011 — so both recipe bridges are testable offline. - [x] Absence is verified against the raw corpus via DuckDB `read_parquet()`, never through the view or verb under test. - [x] Load-bearing behaviours mutation-verified: removing `NOT is_aggregate` from the view, filtering aggregates in the recipe path, and reversing the per-capita/inflation call order each make their covering test fail. - [ ] CI green (installs dependencies fresh and runs `rcmdcheck`, neither exercised locally).
jared added 22 commits 2026-08-03 11:48:22 -04:00
Requirement 1 of #25 shipped with #11/#12. This specs requirement 2 only.

Records three upstream gaps found while measuring the corpus, which change
the shipping scope:

- X40/X41 carry ~42.7K rows (1967-2011) but have no summary_categories row,
  so they cannot appear in a category_type='balance' view. Both holdings
  recipes span X40/X41 + Z77/Z78, so recipe= would silently return only the
  2012-2016 leg. recipe= is therefore deferred to v2.
- SB195/SB196 attach to fin_code X40/X41, outside the balance view.
- SB197-SB202 attach to flow codes, not the holdings codes, so the FY2016
  termination of X21/X30/X42/X44/X47/Z77/Z78 has no catalogued break.

Caveats 2-4 are therefore surfaced reader-side via a computed coverage_window
rather than through the existing series-break builders.
balance is the only category_type whose subtype column is not orthogonal to
category. Measured against the crosswalk: 5 of 6 expenditure subtypes and 1 of
7 revenue subtypes span more than one category, but 0 of 5 balance subtypes do.
Balance is a strict tree -- Fund Balances = {general}, Retirement System
Holdings = {employee_retirement}, Insurance Trust Balances = the three trust
subtypes.

Exposing both arguments would admit no useful combination: of the 15 pairs, 3
are redundant and 12 are guaranteed empty for every government in every year,
failing as an empty tibble that reads as "holds none" rather than as a
contradiction.

Dropping it also keeps the verb aligned -- no uscogdata verb exposes a subtype
argument; the API layers its own subtype row filter on top, which cog-api#26
can do for /balances. #25's one-filter requirement is still met, since
category = "Fund Balances" is exactly W01/W31/W61.

Adds two tests: that one-filter equivalence, and an assertion that the
subtype -> category tree holds, so an upstream change making category lossy
fails here rather than in a user's analysis.
Corrects this spec. The earlier draft deferred recipe= and proposed adding
summary_categories rows for X40/X41. Both were wrong, and the pipeline state
they were meant to fix is correct and documented.

cog_pipeline/docs/phase_r_harmonization_review.md records the decisions:

- Sec 0.2: the wide era exposes these split families ONLY as aggregates, so
  the recipe join deliberately does NOT filter is_aggregate. Safe by
  construction -- wide rows are aggregate-only, modern rows leaf-only, every
  component year-scoped.
- Sec 1: the planned X40->Z77 harmonization MAP rows were dropped on purpose;
  continuity ships as recipes instead. That is why harmonization_map carries
  no balance-code rows.

The reader already implements this (R/recipes.R, R/spending.R). Verified
against the live corpus rather than trusting the comment: corrections_combined
FY2007, whose wide leg E05 is likewise aggregate-only, returns $906,743,000.

Also withdraws the claim that SB155/156 and SB195/196 contradict each other.
X40 rows after FY2002 are the wide-era SAS column persisting through the era
boundary; they say nothing about a classification-level rename. The two sets
describe different layers.

What survives is one narrow, non-blocking gap: no series_breaks row exists at
2016/2017 for Z77/Z78/X30, though review doc Sec 2 recommended exactly that.
Recorded as out-of-scope item 1 with the SB197-SB202 precedent.
Six TDD tasks: the two views + registration gate, the core verb, per_capita
and adjust_to_year, recipe=, balance_caveats provenance, docs.

Every internal the plan calls was verified to exist with the signature used
(.build_verb_sql, .shape_recipe_result, .attach_per_capita, .run_recipe,
.require_schema_v5, ...), so the tasks reuse the shared machinery rather than
reimplementing it. The verb deliberately does not route through
.verb_spendrev(), whose concept scoping, IG leg and complete= grid are all
flow-specific.

Test government is ALABAMA STATE GOVT (010000226085), which covers every case
in the bundled fixture: W01/W31/W61 in 2012/2019/2020, X21+Z77 in 2012,
Y07/Y08 throughout, and X40 in 2011 -- so the wide-era recipe bridge is
testable offline.
Standardises on the identifier other agents use for state governments, which
is stable across corpus vintages and is the same id used against the live API.

Verified in the bundled fixture, and it is strictly better coverage than the
previous pick: Wisconsin reaches four of the five balance subtypes (adds
workers_comp_trust via Y21) and carries BOTH wide->modern recipe bridges
(X40->Z77 and X41->Z78), so a second recipe test is added. Y61
(other_insurance_trust) is absent for Wisconsin; no test depends on it.

Also notes not to assert on gov_name -- the fixture carries both "WISCONSIN"
and "WISCONSIN STATE GOVT" and the verb COALESCEs them.
The bundled fixture has no balance item_code with is_aggregate = TRUE, so
asserting COUNT(*) FROM balance_long WHERE is_aggregate = 0 passed whether
or not the view's AND NOT is_aggregate predicate existed. Follows the
synthetic hive-partitioned parquet pattern already used for the 22-/23-
and 24-/25- view predicates in test-views.R: reads the real
inst/sql/26-balance_long.sql text off disk and executes it against a
synthetic corpus containing both an aggregate and non-aggregate row under
a real balance item_code (W01).
Also add explicit non-empty assertion to the flow-code guard test so it
cannot pass vacuously on a zero-row result.
Mirrors R/spending.R:465-466 -- .check_govids_in_scope()'s return was
previously captured only for its message side effect. Also drops a
redundant duplicate assertion in the flow-code guard test.
Task 4's implementer found that SB195 does not surface for a
recipe query spanning only 2011-2012. .build_series_break_refs() matches
break_year BETWEEN min(years) AND max(years), and SB195's break_year is 2002.

That is correct behaviour rather than a gap: a series lying entirely after the
book -> market change sits on one consistent basis, so disclosing a break it
never crosses would be noise. .build_corpus_break_refs() applies the same rule
deliberately.

The spec's caveat table overclaimed by omitting the span condition. Corrected.
Adds the NEWS entry, a Financial data pkgdown reference section (none
existed for cog_spending/cog_revenue), and corrects CLAUDE.md's SQL-layer
claim, view count, test count and fixture-year description against
measured values. Also documents balance_caveats in
inst/schemas/provenance-v1.json (test-first: added a schema-documentation
test to test-balances.R, confirmed it failed, then fixed the schema) and
fleshes out cog_balances()'s @return roxygen to enumerate its conditional
columns, regenerating man/cog_balances.Rd.
Re-measured CLAUDE.md's test count on the final tree (764, not 763 --
the earlier number predated the balance_caveats schema test). Removed
notes from cog_balances()'s @return block: it was copied from
cog_spending()'s @return style without checking cog_balances() never
calls .verb_spendrev(), the only place that sets notes. Verified the
remaining documented columns against colnames() observed across every
argument combination (bare, per_capita, adjust_to_year, both, recipe,
category filter).
Final-review findings F-1, F-2, F-6, F-8 (plus the F-9 @return reword,
which shares R/balances.R).

F-2: .validate_balance_inputs() checked 2 of cog_balances()' 7 arguments.
years = integer(0) leaked a raw DuckDB 'Parser Error ... AND year IN ()'
with the generated SQL echoed back; govid = character(0) and a non-character
category returned 0 rows with no error at all; recipe = c("a","b") threw
'the condition has length > 1' from inside .validate_recipe_id(). Replaced
with a call to the money verbs' own .validate_verb_inputs() (R/spending.R),
which validates the exact superset needed. Deleted the local copy rather
than extending it -- two validators is how they drift. Placed AFTER
.coerce_govid_input(), because .validate_verb_inputs() asserts
is.character(govid) and a data-frame govid is not unwrapped before that.
This is helper reuse of the same kind as .build_verb_sql()/.attach_per_capita();
the verb still does NOT route through .verb_spendrev().

F-1: falls out of F-2 for free -- the recipe/category mutual-exclusivity
guard lives inside .validate_verb_inputs(). Previously recipe silently
discarded category AND overwrote provenance$category with the recipe label,
so a caller asking for Fund Balances got X40/Z77 insurance-trust holdings
with no trace of the dropped filter.

F-6: cog_explain() rendered every provenance caveat block except
balance_caveats. Since .emit_balance_caveats() fires at most once per
session -- and is routinely consumed by a suppressMessages() call or an
unread knitr chunk -- cog_explain() is the only surface left for a caller
who deliberately audits the result. Added a 'Holdings caveats' section
guarded on !is.null(prov$balance_caveats). Also relabels the cosmetic
'Concept: NA' line on balance results as 'not applicable (holdings are a
stock, not a flow)'.

F-8: the coverage-window query has no govid and no year predicate -- its
answer depends only on the mounted corpus -- yet it scanned all of
balance_long on every call (35% of verb runtime on the fixture, and a
per-request throughput ceiling for cog-api#26). Memoised in
.uscogdata_env$balance_coverage_windows, invalidated by cog_close(), the
same pattern as .uscogdata_env$manifest.
Findings F-1..F-8. Every assertion below was verified to FAIL before its
fix (or under mutation, where the behaviour already worked) and pass after.

F-3: 'an unknown recipe id is rejected' used a bare expect_error(). Deleting
.validate_recipe_id() leaves .recipe_components() returning 0 rows and
comps$label[[1]] throwing 'subscript out of bounds' -- still an error, so
the test passed on the regression while the user lost the curated message.
Now asserts class = 'uscogdata_unknown_recipe'. Mutation-checked.

F-4: no test ever set per_capita and adjust_to_year together, so the
load-bearing ordering comment at R/balances.R was unverified. Reversing
those two calls silently drops amt_per_capita_real (.attach_real_dollars()
no-ops when amt_per_capita_nominal does not exist yet). New test asserts
presence AND that the per-capita column is deflated by the same factor as
the level column; mutation-checked by reversing the order (2 failures).

F-5: the spec's 'Gating' requirement had no test -- nothing ever called
cog_balances() on a corpus without balance_subtype. Extended the existing
with_corpus_missing_balance_subtype() block to assert class =
'uscogdata_no_balance_support'; mutation-checked by dropping the guard.

F-1/F-2: added mutual-exclusivity and four-argument validation tests, each
pinned to the message or class (all four inputs already produced *some*
error or *some* quiet wrong answer, so bare expect_error() was useless
here). Plus an ordering guard: a data-frame govid must still work, which
is what fails if validation is put before .coerce_govid_input().

F-6: asserts on the RENDERED cog_explain() text (both streams -- cli
routes through conditions that land on stderr), with a negative case
proving money-verb output is unaffected and that the capture is not vacuous.

F-7: pins that coverage_window is corpus-scoped while truncated is
query-scoped; mutation-checked by scoping the windows to observed subtypes.

F-8: pins the memo slot is populated on first call and cleared by
cog_close().
F-7: inst/schemas/provenance-v1.json described coverage_window as mapping
each *observed* balance_subtype, but the query at R/balance_caveats.R has
no predicate tied to the query's codes and always returns every subtype in
the mounted corpus. Took option (b) of the two the review offered -- change
the doc, not the code. Reporting all windows is the better product
behaviour (it answers 'is there a family I missed?'), it is what cog-api#26
already forwards verbatim, and option (a) would make the block empty for a
0-row result. Reworded to say the windows are corpus-wide and that
'truncated' is the query-scoped field. Pinned by a new test either way.

F-10: the 'Current State' block was self-contradictory after a partial
update -- headed 2026-04-27, claiming branch main @ d65e9fe, with a
2026-08-03 test count measured on feat/cog-balances-25 underneath it, and
listing README.md / _pkgdown.yml as outstanding when both exist and
_pkgdown.yml was edited by this branch. All numbers below re-measured on
the final tree after every other fix in this wave, not before:
788 tests (testthat::test_local()), 14 exports (NAMESPACE), 14 man/*.Rd,
2 vignettes, no docs/ (pkgdown::build_site() genuinely still outstanding,
as is the .Rbuildignore fixture entry -- both kept in the list).

The related deferred README.md item is closed with no change, per the
review's ruling: README.md enumerates no verbs at all, so naming
cog_balances would make it the only non-cog_spending verb mentioned.
docs: record the two balance_caveats contract facts cog-api#26 must carry
R-CMD-check / check (push) Successful in 3m38s
R-CMD-check / check (pull_request) Successful in 3m35s
2c532bde19
Both were settled during implementation and are easy to get wrong from
outside the package:

- coverage_window is corpus-scoped, not result-scoped. It reports the observed
  year extent of every balance subtype, not only those a query returned. The
  sibling field `truncated` is the result-scoped one.
- balance_caveats is present only on cog_balances() results; an API layer that
  assumes it is universal will read NULL from the money verbs.
jared merged commit 03c313b46d into main 2026-08-03 11:52:13 -04:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Civilytics/uscogdata#28