Both were settled during implementation and are easy to get wrong from
outside the package:
- coverage_window is corpus-scoped, not result-scoped. It reports the observed
year extent of every balance subtype, not only those a query returned. The
sibling field `truncated` is the result-scoped one.
- balance_caveats is present only on cog_balances() results; an API layer that
assumes it is universal will read NULL from the money verbs.
F-7: inst/schemas/provenance-v1.json described coverage_window as mapping
each *observed* balance_subtype, but the query at R/balance_caveats.R has
no predicate tied to the query's codes and always returns every subtype in
the mounted corpus. Took option (b) of the two the review offered -- change
the doc, not the code. Reporting all windows is the better product
behaviour (it answers 'is there a family I missed?'), it is what cog-api#26
already forwards verbatim, and option (a) would make the block empty for a
0-row result. Reworded to say the windows are corpus-wide and that
'truncated' is the query-scoped field. Pinned by a new test either way.
F-10: the 'Current State' block was self-contradictory after a partial
update -- headed 2026-04-27, claiming branch main @ d65e9fe, with a
2026-08-03 test count measured on feat/cog-balances-25 underneath it, and
listing README.md / _pkgdown.yml as outstanding when both exist and
_pkgdown.yml was edited by this branch. All numbers below re-measured on
the final tree after every other fix in this wave, not before:
788 tests (testthat::test_local()), 14 exports (NAMESPACE), 14 man/*.Rd,
2 vignettes, no docs/ (pkgdown::build_site() genuinely still outstanding,
as is the .Rbuildignore fixture entry -- both kept in the list).
The related deferred README.md item is closed with no change, per the
review's ruling: README.md enumerates no verbs at all, so naming
cog_balances would make it the only non-cog_spending verb mentioned.
Findings F-1..F-8. Every assertion below was verified to FAIL before its
fix (or under mutation, where the behaviour already worked) and pass after.
F-3: 'an unknown recipe id is rejected' used a bare expect_error(). Deleting
.validate_recipe_id() leaves .recipe_components() returning 0 rows and
comps$label[[1]] throwing 'subscript out of bounds' -- still an error, so
the test passed on the regression while the user lost the curated message.
Now asserts class = 'uscogdata_unknown_recipe'. Mutation-checked.
F-4: no test ever set per_capita and adjust_to_year together, so the
load-bearing ordering comment at R/balances.R was unverified. Reversing
those two calls silently drops amt_per_capita_real (.attach_real_dollars()
no-ops when amt_per_capita_nominal does not exist yet). New test asserts
presence AND that the per-capita column is deflated by the same factor as
the level column; mutation-checked by reversing the order (2 failures).
F-5: the spec's 'Gating' requirement had no test -- nothing ever called
cog_balances() on a corpus without balance_subtype. Extended the existing
with_corpus_missing_balance_subtype() block to assert class =
'uscogdata_no_balance_support'; mutation-checked by dropping the guard.
F-1/F-2: added mutual-exclusivity and four-argument validation tests, each
pinned to the message or class (all four inputs already produced *some*
error or *some* quiet wrong answer, so bare expect_error() was useless
here). Plus an ordering guard: a data-frame govid must still work, which
is what fails if validation is put before .coerce_govid_input().
F-6: asserts on the RENDERED cog_explain() text (both streams -- cli
routes through conditions that land on stderr), with a negative case
proving money-verb output is unaffected and that the capture is not vacuous.
F-7: pins that coverage_window is corpus-scoped while truncated is
query-scoped; mutation-checked by scoping the windows to observed subtypes.
F-8: pins the memo slot is populated on first call and cleared by
cog_close().
Final-review findings F-1, F-2, F-6, F-8 (plus the F-9 @return reword,
which shares R/balances.R).
F-2: .validate_balance_inputs() checked 2 of cog_balances()' 7 arguments.
years = integer(0) leaked a raw DuckDB 'Parser Error ... AND year IN ()'
with the generated SQL echoed back; govid = character(0) and a non-character
category returned 0 rows with no error at all; recipe = c("a","b") threw
'the condition has length > 1' from inside .validate_recipe_id(). Replaced
with a call to the money verbs' own .validate_verb_inputs() (R/spending.R),
which validates the exact superset needed. Deleted the local copy rather
than extending it -- two validators is how they drift. Placed AFTER
.coerce_govid_input(), because .validate_verb_inputs() asserts
is.character(govid) and a data-frame govid is not unwrapped before that.
This is helper reuse of the same kind as .build_verb_sql()/.attach_per_capita();
the verb still does NOT route through .verb_spendrev().
F-1: falls out of F-2 for free -- the recipe/category mutual-exclusivity
guard lives inside .validate_verb_inputs(). Previously recipe silently
discarded category AND overwrote provenance$category with the recipe label,
so a caller asking for Fund Balances got X40/Z77 insurance-trust holdings
with no trace of the dropped filter.
F-6: cog_explain() rendered every provenance caveat block except
balance_caveats. Since .emit_balance_caveats() fires at most once per
session -- and is routinely consumed by a suppressMessages() call or an
unread knitr chunk -- cog_explain() is the only surface left for a caller
who deliberately audits the result. Added a 'Holdings caveats' section
guarded on !is.null(prov$balance_caveats). Also relabels the cosmetic
'Concept: NA' line on balance results as 'not applicable (holdings are a
stock, not a flow)'.
F-8: the coverage-window query has no govid and no year predicate -- its
answer depends only on the mounted corpus -- yet it scanned all of
balance_long on every call (35% of verb runtime on the fixture, and a
per-request throughput ceiling for cog-api#26). Memoised in
.uscogdata_env$balance_coverage_windows, invalidated by cog_close(), the
same pattern as .uscogdata_env$manifest.
Re-measured CLAUDE.md's test count on the final tree (764, not 763 --
the earlier number predated the balance_caveats schema test). Removed
notes from cog_balances()'s @return block: it was copied from
cog_spending()'s @return style without checking cog_balances() never
calls .verb_spendrev(), the only place that sets notes. Verified the
remaining documented columns against colnames() observed across every
argument combination (bare, per_capita, adjust_to_year, both, recipe,
category filter).
Adds the NEWS entry, a Financial data pkgdown reference section (none
existed for cog_spending/cog_revenue), and corrects CLAUDE.md's SQL-layer
claim, view count, test count and fixture-year description against
measured values. Also documents balance_caveats in
inst/schemas/provenance-v1.json (test-first: added a schema-documentation
test to test-balances.R, confirmed it failed, then fixed the schema) and
fleshes out cog_balances()'s @return roxygen to enumerate its conditional
columns, regenerating man/cog_balances.Rd.
Task 4's implementer found that SB195 does not surface for a
recipe query spanning only 2011-2012. .build_series_break_refs() matches
break_year BETWEEN min(years) AND max(years), and SB195's break_year is 2002.
That is correct behaviour rather than a gap: a series lying entirely after the
book -> market change sits on one consistent basis, so disclosing a break it
never crosses would be noise. .build_corpus_break_refs() applies the same rule
deliberately.
The spec's caveat table overclaimed by omitting the span condition. Corrected.
Mirrors R/spending.R:465-466 -- .check_govids_in_scope()'s return was
previously captured only for its message side effect. Also drops a
redundant duplicate assertion in the flow-code guard test.
The bundled fixture has no balance item_code with is_aggregate = TRUE, so
asserting COUNT(*) FROM balance_long WHERE is_aggregate = 0 passed whether
or not the view's AND NOT is_aggregate predicate existed. Follows the
synthetic hive-partitioned parquet pattern already used for the 22-/23-
and 24-/25- view predicates in test-views.R: reads the real
inst/sql/26-balance_long.sql text off disk and executes it against a
synthetic corpus containing both an aggregate and non-aggregate row under
a real balance item_code (W01).
Standardises on the identifier other agents use for state governments, which
is stable across corpus vintages and is the same id used against the live API.
Verified in the bundled fixture, and it is strictly better coverage than the
previous pick: Wisconsin reaches four of the five balance subtypes (adds
workers_comp_trust via Y21) and carries BOTH wide->modern recipe bridges
(X40->Z77 and X41->Z78), so a second recipe test is added. Y61
(other_insurance_trust) is absent for Wisconsin; no test depends on it.
Also notes not to assert on gov_name -- the fixture carries both "WISCONSIN"
and "WISCONSIN STATE GOVT" and the verb COALESCEs them.
Six TDD tasks: the two views + registration gate, the core verb, per_capita
and adjust_to_year, recipe=, balance_caveats provenance, docs.
Every internal the plan calls was verified to exist with the signature used
(.build_verb_sql, .shape_recipe_result, .attach_per_capita, .run_recipe,
.require_schema_v5, ...), so the tasks reuse the shared machinery rather than
reimplementing it. The verb deliberately does not route through
.verb_spendrev(), whose concept scoping, IG leg and complete= grid are all
flow-specific.
Test government is ALABAMA STATE GOVT (010000226085), which covers every case
in the bundled fixture: W01/W31/W61 in 2012/2019/2020, X21+Z77 in 2012,
Y07/Y08 throughout, and X40 in 2011 -- so the wide-era recipe bridge is
testable offline.
Corrects this spec. The earlier draft deferred recipe= and proposed adding
summary_categories rows for X40/X41. Both were wrong, and the pipeline state
they were meant to fix is correct and documented.
cog_pipeline/docs/phase_r_harmonization_review.md records the decisions:
- Sec 0.2: the wide era exposes these split families ONLY as aggregates, so
the recipe join deliberately does NOT filter is_aggregate. Safe by
construction -- wide rows are aggregate-only, modern rows leaf-only, every
component year-scoped.
- Sec 1: the planned X40->Z77 harmonization MAP rows were dropped on purpose;
continuity ships as recipes instead. That is why harmonization_map carries
no balance-code rows.
The reader already implements this (R/recipes.R, R/spending.R). Verified
against the live corpus rather than trusting the comment: corrections_combined
FY2007, whose wide leg E05 is likewise aggregate-only, returns $906,743,000.
Also withdraws the claim that SB155/156 and SB195/196 contradict each other.
X40 rows after FY2002 are the wide-era SAS column persisting through the era
boundary; they say nothing about a classification-level rename. The two sets
describe different layers.
What survives is one narrow, non-blocking gap: no series_breaks row exists at
2016/2017 for Z77/Z78/X30, though review doc Sec 2 recommended exactly that.
Recorded as out-of-scope item 1 with the SB197-SB202 precedent.
balance is the only category_type whose subtype column is not orthogonal to
category. Measured against the crosswalk: 5 of 6 expenditure subtypes and 1 of
7 revenue subtypes span more than one category, but 0 of 5 balance subtypes do.
Balance is a strict tree -- Fund Balances = {general}, Retirement System
Holdings = {employee_retirement}, Insurance Trust Balances = the three trust
subtypes.
Exposing both arguments would admit no useful combination: of the 15 pairs, 3
are redundant and 12 are guaranteed empty for every government in every year,
failing as an empty tibble that reads as "holds none" rather than as a
contradiction.
Dropping it also keeps the verb aligned -- no uscogdata verb exposes a subtype
argument; the API layers its own subtype row filter on top, which cog-api#26
can do for /balances. #25's one-filter requirement is still met, since
category = "Fund Balances" is exactly W01/W31/W61.
Adds two tests: that one-filter equivalence, and an assertion that the
subtype -> category tree holds, so an upstream change making category lossy
fails here rather than in a user's analysis.
Requirement 1 of #25 shipped with #11/#12. This specs requirement 2 only.
Records three upstream gaps found while measuring the corpus, which change
the shipping scope:
- X40/X41 carry ~42.7K rows (1967-2011) but have no summary_categories row,
so they cannot appear in a category_type='balance' view. Both holdings
recipes span X40/X41 + Z77/Z78, so recipe= would silently return only the
2012-2016 leg. recipe= is therefore deferred to v2.
- SB195/SB196 attach to fin_code X40/X41, outside the balance view.
- SB197-SB202 attach to flow codes, not the holdings codes, so the FY2016
termination of X21/X30/X42/X44/X47/Z77/Z78 has no catalogued break.
Caveats 2-4 are therefore surfaced reader-side via a computed coverage_window
rather than through the existing series-break builders.
jared