Closes#19. Stacked on #20 — base is fix/regen-fixture-corpus-18, because SB194 does not exist in main's bundled fixture (it predates the break being catalogued). Merge #20 first and this retargets to main cleanly.
The defect
.build_series_break_refs() matches fin_code IN (<codes in the result>). No row's item_code is ever the literal "ALL", so the four corpus-wide entries could never match and reached no user:
id
year
what it says
SB085
1977
dollar precision across the 1976/1977 boundary
SB087
2002
imputation exclusion FY2002-2006
SB194
2012
dense → sparse representation change
SB086
2017
government id scheme change
SB194 is why this is urgent: cog_pipeline#64 DoD 4 was "series_breaks.csv carries an ALL @ 2012 entry describing the representation change, so cog_explain() surfaces it". The entry shipped; the reader dropped it.
The fix
Provenance gains corpus_break_refs, built by .build_corpus_break_refs() on the break_year window alone — which codes a result happens to contain is irrelevant to a caveat about the corpus.
A separate field rather than more entries in series_break_refs (DoD 2): an ALL caveat qualifies the whole result, and folding the two together invites reading it as a caveat about one series. .build_series_break_refs() now excludes 'ALL' explicitly, so the two are disjoint by construction and a test asserts it.
cog_explain() prints them under their own Corpus-wide caveats heading. cog-api passes provenance through verbatim (provenance_of() → ok_envelope()), so the field reaches the API with no change there.
Verified end-to-end on a FY2011→2012 query:
series_break_refs: []
corpus_break_refs: ["SB194"]
── Corpus-wide caveats ──
• SB194 (2012): Absence is NOT comparable across FY2012. Rule: year<=2011 + cell
absent => Census published $0 (census_zero); year>=2012 + cell absent => not
reported, unknown (not_reported). …
One deviation from the issue, flagged
DoD 1 says ALL entries fire when the query's year range spans their break_year; DoD 3 asks that a FY2011 query surface SB085, whose break_year is 1977. Those cannot both hold. All four entries are boundary caveats — their own join_advice speaks of crossing 1976/1977, of FY2002-2006, of absence not being comparable across FY2012, of pre- vs post-2017 ids — so I implemented DoD 1's rule, which is also the rule the code-specific path already uses. SB085 is tested with a range that actually spans 1977. Commented on #19.
Verification
suite
before
after
uscogdata
594 / 0 / 6
606 / 0 / 6
cog-api
357 / 0 / 8
357 / 0 / 8
Tests were written first and confirmed red for the right reasons (4 failures + object '.build_corpus_break_refs' not found) before the implementation landed.
Closes #19. **Stacked on #20** — base is `fix/regen-fixture-corpus-18`, because `SB194` does not exist in main's bundled fixture (it predates the break being catalogued). Merge #20 first and this retargets to main cleanly.
## The defect
`.build_series_break_refs()` matches `fin_code IN (<codes in the result>)`. No row's `item_code` is ever the literal `"ALL"`, so the four corpus-wide entries could never match and reached no user:
| id | year | what it says |
|---|---:|---|
| `SB085` | 1977 | dollar precision across the 1976/1977 boundary |
| `SB087` | 2002 | imputation exclusion FY2002-2006 |
| `SB194` | 2012 | dense → sparse representation change |
| `SB086` | 2017 | government id scheme change |
`SB194` is why this is urgent: cog_pipeline#64 DoD 4 was "`series_breaks.csv` carries an ALL @ 2012 entry describing the representation change, **so `cog_explain()` surfaces it**". The entry shipped; the reader dropped it.
## The fix
Provenance gains `corpus_break_refs`, built by `.build_corpus_break_refs()` on the **break_year window alone** — which codes a result happens to contain is irrelevant to a caveat about the corpus.
A separate field rather than more entries in `series_break_refs` (DoD 2): an ALL caveat qualifies the whole result, and folding the two together invites reading it as a caveat about one series. `.build_series_break_refs()` now excludes `'ALL'` explicitly, so the two are disjoint by construction and a test asserts it.
`cog_explain()` prints them under their own **Corpus-wide caveats** heading. cog-api passes provenance through verbatim (`provenance_of()` → `ok_envelope()`), so the field reaches the API with no change there.
Verified end-to-end on a FY2011→2012 query:
```
series_break_refs: []
corpus_break_refs: ["SB194"]
── Corpus-wide caveats ──
• SB194 (2012): Absence is NOT comparable across FY2012. Rule: year<=2011 + cell
absent => Census published $0 (census_zero); year>=2012 + cell absent => not
reported, unknown (not_reported). …
```
## One deviation from the issue, flagged
DoD 1 says ALL entries fire when the query's year range **spans their break_year**; DoD 3 asks that a **FY2011** query surface `SB085`, whose break_year is **1977**. Those cannot both hold. All four entries are *boundary* caveats — their own `join_advice` speaks of crossing 1976/1977, of FY2002-2006, of absence not being comparable across FY2012, of pre- vs post-2017 ids — so I implemented DoD 1's rule, which is also the rule the code-specific path already uses. `SB085` is tested with a range that actually spans 1977. Commented on #19.
## Verification
| suite | before | after |
|---|---|---|
| uscogdata | 594 / 0 / 6 | **606 / 0 / 6** |
| cog-api | 357 / 0 / 8 | **357 / 0 / 8** |
Tests were written first and confirmed red for the right reasons (4 failures + `object '.build_corpus_break_refs' not found`) before the implementation landed.
.build_series_break_refs() matches `fin_code IN (<codes in the result>)`.
No row's item_code is ever the literal "ALL", so the four corpus-wide
entries could never match and reached no user:
SB085 1977 dollar precision across the 1976/1977 boundary
SB087 2002 imputation exclusion FY2002-2006
SB194 2012 dense -> sparse representation change
SB086 2017 government id scheme change
SB194 is why this matters now. cog_pipeline#64 DoD 4 was "series_breaks.csv
carries an ALL @ 2012 entry describing the representation change, SO
cog_explain() surfaces it". The entry shipped; the reader dropped it. A
query spanning FY2011 -> FY2012 crosses the boundary where an absent cell
stops meaning "Census published $0" and starts meaning "not reported", and
nothing said so.
Provenance gains `corpus_break_refs`, built by .build_corpus_break_refs()
on the break_year window alone -- which codes a result happens to contain
is irrelevant to a caveat about the corpus. A separate field rather than
more entries in series_break_refs, because an ALL caveat qualifies the
whole result and folding the two together invites reading it as a caveat
about one series; .build_series_break_refs() now excludes 'ALL' explicitly
so the two stay disjoint by construction. cog_explain() prints them under
their own "Corpus-wide caveats" heading, and cog-api passes provenance
through verbatim, so the field reaches the API with no change there.
On the year rule: all four entries are BOUNDARY caveats -- their own
join_advice speaks of crossing 1976/1977, of FY2002-2006, of absence not
being comparable across FY2012, of pre- vs post-2017 ids -- so the same
`break_year BETWEEN min(years) AND max(years)` rule the code-specific path
uses is the right one, and matches the issue's DoD 1. The issue's DoD 3
also asks that a FY2011 query surface SB085; that cannot hold under DoD 1
and does not hold under any reading of SB085's text, whose boundary is
1976/1977. Tested with a range that actually spans it, and flagged on the
issue.
Stacked on fix/regen-fixture-corpus-18: SB194 does not exist in main's
bundled fixture, which predates the break being catalogued.
Suite: 606 pass / 0 fail / 6 skip (was 594/0/6).
cog-api 357 / 0 / 8, unchanged.
jared
merged commit ebac39e6de into main2026-07-30 10:33:24 -04:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #19. Stacked on #20 — base is
fix/regen-fixture-corpus-18, becauseSB194does not exist in main's bundled fixture (it predates the break being catalogued). Merge #20 first and this retargets to main cleanly.The defect
.build_series_break_refs()matchesfin_code IN (<codes in the result>). No row'sitem_codeis ever the literal"ALL", so the four corpus-wide entries could never match and reached no user:SB085SB087SB194SB086SB194is why this is urgent: cog_pipeline#64 DoD 4 was "series_breaks.csvcarries an ALL @ 2012 entry describing the representation change, socog_explain()surfaces it". The entry shipped; the reader dropped it.The fix
Provenance gains
corpus_break_refs, built by.build_corpus_break_refs()on the break_year window alone — which codes a result happens to contain is irrelevant to a caveat about the corpus.A separate field rather than more entries in
series_break_refs(DoD 2): an ALL caveat qualifies the whole result, and folding the two together invites reading it as a caveat about one series..build_series_break_refs()now excludes'ALL'explicitly, so the two are disjoint by construction and a test asserts it.cog_explain()prints them under their own Corpus-wide caveats heading. cog-api passes provenance through verbatim (provenance_of()→ok_envelope()), so the field reaches the API with no change there.Verified end-to-end on a FY2011→2012 query:
One deviation from the issue, flagged
DoD 1 says ALL entries fire when the query's year range spans their break_year; DoD 3 asks that a FY2011 query surface
SB085, whose break_year is 1977. Those cannot both hold. All four entries are boundary caveats — their ownjoin_advicespeaks of crossing 1976/1977, of FY2002-2006, of absence not being comparable across FY2012, of pre- vs post-2017 ids — so I implemented DoD 1's rule, which is also the rule the code-specific path already uses.SB085is tested with a range that actually spans 1977. Commented on #19.Verification
Tests were written first and confirmed red for the right reasons (4 failures +
object '.build_corpus_break_refs' not found) before the implementation landed.