fix: surface ALL-scoped series breaks in provenance (#19)
R-CMD-check / check (push) Successful in 3m1s
R-CMD-check / check (pull_request) Successful in 3m1s

.build_series_break_refs() matches `fin_code IN (<codes in the result>)`.
No row's item_code is ever the literal "ALL", so the four corpus-wide
entries could never match and reached no user:

  SB085  1977  dollar precision across the 1976/1977 boundary
  SB087  2002  imputation exclusion FY2002-2006
  SB194  2012  dense -> sparse representation change
  SB086  2017  government id scheme change

SB194 is why this matters now. cog_pipeline#64 DoD 4 was "series_breaks.csv
carries an ALL @ 2012 entry describing the representation change, SO
cog_explain() surfaces it". The entry shipped; the reader dropped it. A
query spanning FY2011 -> FY2012 crosses the boundary where an absent cell
stops meaning "Census published $0" and starts meaning "not reported", and
nothing said so.

Provenance gains `corpus_break_refs`, built by .build_corpus_break_refs()
on the break_year window alone -- which codes a result happens to contain
is irrelevant to a caveat about the corpus. A separate field rather than
more entries in series_break_refs, because an ALL caveat qualifies the
whole result and folding the two together invites reading it as a caveat
about one series; .build_series_break_refs() now excludes 'ALL' explicitly
so the two stay disjoint by construction. cog_explain() prints them under
their own "Corpus-wide caveats" heading, and cog-api passes provenance
through verbatim, so the field reaches the API with no change there.

On the year rule: all four entries are BOUNDARY caveats -- their own
join_advice speaks of crossing 1976/1977, of FY2002-2006, of absence not
being comparable across FY2012, of pre- vs post-2017 ids -- so the same
`break_year BETWEEN min(years) AND max(years)` rule the code-specific path
uses is the right one, and matches the issue's DoD 1. The issue's DoD 3
also asks that a FY2011 query surface SB085; that cannot hold under DoD 1
and does not hold under any reading of SB085's text, whose boundary is
1976/1977. Tested with a range that actually spans it, and flagged on the
issue.

Stacked on fix/regen-fixture-corpus-18: SB194 does not exist in main's
bundled fixture, which predates the break being catalogued.

Suite: 606 pass / 0 fail / 6 skip (was 594/0/6).
cog-api 357 / 0 / 8, unchanged.
This commit is contained in:
2026-07-30 10:27:48 -04:00
parent c375c55da7
commit 1d553a788f
8 changed files with 183 additions and 10 deletions
+20
View File
@@ -1,5 +1,25 @@
# uscogdata 0.1.0 (development)
## Corpus-wide series breaks now reach users (`corpus_break_refs`)
* Four catalogued series breaks carry `fin_code = "ALL"` — caveats about the
corpus as a whole rather than about one item code. `series_break_refs` is
built by matching `fin_code` against the item codes in the result, and no
row's `item_code` is ever the literal `"ALL"`, so **none of them could ever
be surfaced**: `SB085` (dollar precision across the 1976/1977 boundary),
`SB087` (imputation exclusion from FY2002), `SB194` (the dense → sparse
representation change at FY2012) and `SB086` (the government id scheme
change at FY2017).
* Provenance gains `corpus_break_refs`, selected on the break-year window
alone and disjoint from `series_break_refs` by construction, so a consumer
can tell a whole-result caveat from a break in one series. `cog_explain()`
prints them under their own "Corpus-wide caveats" heading. cog-api passes
provenance through verbatim, so the field appears there without an API
change.
* `SB194` is the one that made this urgent: a query spanning FY2011 → FY2012
crosses the boundary where an absent cell stops meaning "Census published
`$0`" and starts meaning "not reported", and until now nothing said so.
## Bundled fixture regenerated against the sparsified corpus
* `inst/extdata/fixture_corpus/` now tracks the corpus published on