docs: design spec for cog_balances(), the uscogdata#25 holdings surface
Requirement 1 of #25 shipped with #11/#12. This specs requirement 2 only. Records three upstream gaps found while measuring the corpus, which change the shipping scope: - X40/X41 carry ~42.7K rows (1967-2011) but have no summary_categories row, so they cannot appear in a category_type='balance' view. Both holdings recipes span X40/X41 + Z77/Z78, so recipe= would silently return only the 2012-2016 leg. recipe= is therefore deferred to v2. - SB195/SB196 attach to fin_code X40/X41, outside the balance view. - SB197-SB202 attach to flow codes, not the holdings codes, so the FY2016 termination of X21/X30/X42/X44/X47/Z77/Z78 has no catalogued break. Caveats 2-4 are therefore surfaced reader-side via a computed coverage_window rather than through the existing series-break builders.
This commit is contained in:
@@ -0,0 +1,188 @@
|
||||
# `cog_balances()` — a reader surface for cash and security holdings
|
||||
|
||||
**Issue:** `uscogdata#25` requirement 2 · **Downstream:** `cog-api#26`
|
||||
**Date:** 2026-08-03 · **Status:** design, awaiting approval
|
||||
|
||||
Requirement 1 of `uscogdata#25` (no `balance` row may reach a money verb) shipped
|
||||
with `#11`/`#12` and is asserted at both view and verb level. This spec covers
|
||||
requirement 2 only: a way to query holdings.
|
||||
|
||||
## Decision: a verb, not an argument
|
||||
|
||||
`cog_balances()`, parallel to `cog_spending()` / `cog_revenue()`.
|
||||
|
||||
Holdings are a **stock** — a balance at a point in time — while the money verbs
|
||||
return **flows** over a fiscal year. The flow verbs' whole argument vocabulary
|
||||
is meaningless for a stock: `expenditure_concept` / `revenue_concept` describe
|
||||
which flows Census aggregates into a published total, and `complete=` fills a
|
||||
grid of fiscal-year cells. Overloading a money verb would put a stock behind
|
||||
arguments that all assume a flow.
|
||||
|
||||
## The 14 codes
|
||||
|
||||
Measured against the published corpus 2026-08-03, not transcribed from the
|
||||
issue. `year_min`/`year_max` are observed row extents.
|
||||
|
||||
| `balance_subtype` | `category` | codes | observed years |
|
||||
|---|---|---|---|
|
||||
| `general` | Fund Balances | `W01`, `W31`, `W61` | 2012–2021 |
|
||||
| `employee_retirement` | Retirement System Holdings | `X21`, `X42`, `X44` | 1967–2016 |
|
||||
| | | `X47` | 1988–2016 |
|
||||
| | | `X30`, `Z77`, `Z78` | 2012–2016 |
|
||||
| `unemployment_trust` | Insurance Trust Balances | `Y07`, `Y08` | 1967–2023 |
|
||||
| `workers_comp_trust` | Insurance Trust Balances | `Y21` | 2012–2023 |
|
||||
| `other_insurance_trust` | Insurance Trust Balances | `Y61` | 2012–2023 |
|
||||
|
||||
## Architecture
|
||||
|
||||
### Two new views
|
||||
|
||||
Mirroring the `revenue_long` / `revenue_annotated` pair exactly:
|
||||
|
||||
- `inst/sql/26-balance_long.sql` — `category_type = 'balance' AND NOT is_aggregate`
|
||||
- `inst/sql/46-balance_annotated.sql` — joins `canonical_fips_xwalk` and
|
||||
`summary_categories`, exposing `category`, `category_type`, `balance_subtype`
|
||||
|
||||
`.register_views()` globs `inst/sql/*.sql` in sorted order, so both register
|
||||
with no new registration code.
|
||||
|
||||
### A third gate list in `R/views.R`
|
||||
|
||||
`CREATE VIEW` resolves its source schema eagerly, so a missing **column** fails
|
||||
at registration time, not at query time. `46-balance_annotated.sql` selects
|
||||
`c.balance_subtype`, which exists only on corpora built after pipeline `#76`/`#77`.
|
||||
That arrived without a `schema_version` bump, so neither existing gate applies:
|
||||
`.harmonization_view_files` keys on `schema_version`, `.representation_view_files`
|
||||
on the presence of a *file*. The discriminator here is a **column on an existing
|
||||
table**.
|
||||
|
||||
```r
|
||||
.balance_view_files <- c("26-balance_long.sql", "46-balance_annotated.sql")
|
||||
```
|
||||
|
||||
gated by probing `summary_categories` for `balance_subtype`, with
|
||||
`cog_balances()` erroring cleanly via `.require_balance_support()` on an older
|
||||
corpus — mirroring how `.require_schema_v5()` gates the harmonized views.
|
||||
|
||||
### `R/balances.R` — a dedicated path, not `.verb_spendrev()`
|
||||
|
||||
`.verb_spendrev()` is 825 lines whose concept scoping, intergovernmental leg and
|
||||
`complete=` grid are all flow-specific, and four verbs depend on it. Threading a
|
||||
third mode through it adds branching to shared code for no reuse benefit.
|
||||
|
||||
Reused unchanged: `.build_provenance()`, `.build_series_break_refs()`,
|
||||
`.build_corpus_break_refs()`, the population join, `.inflate()`, and
|
||||
`.coerce_govid_input()`.
|
||||
|
||||
Following the package's real two-layer convention: **view definitions** live in
|
||||
`inst/sql/`; **query construction** is inline `sprintf()` in R, as in
|
||||
`.verb_spendrev()`. (`CLAUDE.md` currently states "never inline SQL strings in R
|
||||
files", which the verb layer has never obeyed. Corrected in a separate commit —
|
||||
see Out of scope.)
|
||||
|
||||
## Signature
|
||||
|
||||
```r
|
||||
cog_balances(govid, years,
|
||||
subtype = NULL, # general | employee_retirement |
|
||||
# unemployment_trust | workers_comp_trust |
|
||||
# other_insurance_trust
|
||||
category = NULL, # Fund Balances | Insurance Trust Balances |
|
||||
# Retirement System Holdings
|
||||
per_capita = FALSE,
|
||||
adjust_to_year = NULL,
|
||||
basis = c("harmonized", "raw"))
|
||||
```
|
||||
|
||||
Returns a `tbl_df` with a `provenance` attribute, like every other verb.
|
||||
|
||||
**Absent by design:** `expenditure_concept`, `revenue_concept`, `complete`.
|
||||
|
||||
**`per_capita` is offered.** Holdings per resident is a real measure (pension
|
||||
assets per capita, fund balance per resident). The roxygen `@param` states
|
||||
plainly that this is a *stock per resident* and is **not** comparable to
|
||||
`cog_spending()`'s per-capita figures.
|
||||
|
||||
**`basis` is currently a no-op** — `harmonization_map` has zero balance-code
|
||||
rows, so harmonized and raw are identical for holdings. Kept for uniformity
|
||||
with the money verbs (the API would otherwise special-case), and
|
||||
`provenance$basis_note` says so outright rather than letting it look meaningful.
|
||||
|
||||
**`recipe` is deliberately omitted from v1.** See below.
|
||||
|
||||
## Decision point: `recipe=` deferred to v2
|
||||
|
||||
The two holdings recipes are unusable from this verb today, and shipping the
|
||||
argument anyway would produce a silently truncated series.
|
||||
|
||||
`cash_securities_z77_wide` = `X40` (1967–2011) + `Z77` (2012–2023);
|
||||
`cash_securities_z78_wide` = `X41` (1967–2011) + `Z78` (2012–2023).
|
||||
|
||||
**`X40` and `X41` have zero rows in `summary_categories`.** They carry ~42,700
|
||||
rows in the corpus across 1967–2011, but with no `category_type` they cannot
|
||||
appear in a `category_type = 'balance'` view. Applying either recipe would
|
||||
return only the modern leg — 2012–2016 — dropping 45 years while looking like a
|
||||
valid continuous series. That is precisely the "plausible rather than obviously
|
||||
wrong" failure mode `#25` exists to prevent.
|
||||
|
||||
The fix is upstream, in the pipeline crosswalk. Until it lands, `cog_balances()`
|
||||
does not accept `recipe`, and an attempt to pass one is an error naming the
|
||||
blocking issue rather than a silent partial result.
|
||||
|
||||
## Caveat surfacing
|
||||
|
||||
`provenance$balance_caveats`, always present, plus one `cli_inform()` per
|
||||
session per caveat class when a query actually touches an affected family or
|
||||
year. Structured so `cog-api#26` can forward the fields verbatim.
|
||||
|
||||
Three of the four caveats need reader-side work. **My earlier assumption that the
|
||||
catalogued series breaks would cover them is wrong**, verified against
|
||||
`series_breaks.parquet`:
|
||||
|
||||
| # | Caveat | Covered by existing machinery? |
|
||||
|---|---|---|
|
||||
| 1 | Gross holdings, **not GAAP fund balance**; no liabilities netted | No — constant, new field `not_gaap = TRUE` |
|
||||
| 2 | `W` is FY2012–2021 only | No — new `coverage_window`, **computed** from the corpus |
|
||||
| 3 | `X`/`Z` family ends FY2016 | **No.** SB197–SB202 attach to `X02`/`X05`/`X08`/`X11`/`X12`, which are *flow* codes. The holdings codes have no catalogued break for their FY2016 termination. Surfaced via `coverage_window` |
|
||||
| 4 | `X40`/`X41` book → market at FY2002 | **No.** SB195/SB196 attach to `fin_code` `X40`/`X41`, which are outside the balance view (see above). Surfaced as an explicit caveat when the year range crosses 2002 and touches `employee_retirement` |
|
||||
|
||||
`coverage_window` is derived per observed subtype family from the corpus, never
|
||||
hardcoded, so it stays correct as the corpus grows.
|
||||
|
||||
`series_break_refs` and `corpus_break_refs` are still populated by the existing
|
||||
code-driven builders — they simply contribute nothing for the four caveats
|
||||
above, and will start contributing once the upstream gaps close.
|
||||
|
||||
## Testing
|
||||
|
||||
New `tests/testthat/test-balances.R`. The bundled fixture covers all four
|
||||
fixture years — `W` in 2012/2019/2020, the `X`/`Z` family in 2011/2012, `Y`
|
||||
throughout — so every test below runs offline.
|
||||
|
||||
- **Inverse guard.** No flow code ever appears in `cog_balances()`, complementing
|
||||
the already-asserted forward guard. Absence is verified against the raw corpus
|
||||
via `read_parquet` on `data/long`, never through the verb that creates it.
|
||||
- **FY2016 seam.** The `X`/`Z` family is present in 2012 and absent in 2019;
|
||||
`coverage_window` reports the termination and the console message fires once.
|
||||
- **Caveats.** `not_gaap` is always `TRUE`; `coverage_window` matches the
|
||||
measured table above; the FY2002 valuation caveat fires only when the year
|
||||
range crosses 2002 *and* touches `employee_retirement`.
|
||||
- **`per_capita`.** `amt_per_capita_nominal == amt_nominal / population`.
|
||||
- **`recipe` rejection.** Passing `recipe` errors with a message naming the
|
||||
blocking issue.
|
||||
- **Gating.** `.require_balance_support()` errors cleanly on a corpus whose
|
||||
`summary_categories` lacks `balance_subtype`.
|
||||
|
||||
## Out of scope, tracked separately
|
||||
|
||||
1. **Pipeline issue (new).** Add `summary_categories` rows for `X40`/`X41`
|
||||
(`category_type = 'balance'`, `balance_subtype = 'employee_retirement'`), and
|
||||
catalogue a series break for the FY2016 termination of the holdings codes.
|
||||
Unblocks `recipe=` and lets caveats 3 and 4 flow through the existing
|
||||
builders. Requires a corpus rebuild + republish, which is a separate,
|
||||
human-approved operation.
|
||||
2. **`cog-api#26`.** Adds `/balances` in all three required places — handler,
|
||||
`param_contract`, and the `plumber.R` route signature. Lands after this.
|
||||
3. **`uscogdata/CLAUDE.md` refresh.** Separate commit. It is stale: it claims 7
|
||||
SQL views (there are 21), 181 tests (716), a two-year fixture (four years),
|
||||
and a "never inline SQL" rule the verb layer does not follow.
|
||||
Reference in New Issue
Block a user