fin_code = "ALL" series breaks can never surface through cog_explain() #19

New Issue

2026-07-29T19:57:52-04:00

jared commented

2026-07-29 19:57:52 -04:00

R/series_breaks.R looks breaks up with

WHERE fin_code IN (%s) AND break_year BETWEEN %d AND %d

matching against the item codes present in the result. An entry whose fin_code is the literal string "ALL" can therefore never match, because no row's item_code is "ALL".

Four catalogued corpus-wide breaks are affected -- every one of them a caveat that applies to every code:

id	year	what it says
`SB085`	1977	FY1967-1976 collected in whole dollars; derived values carry +-$1k noise
`SB087`	2002	imputation exclusion FY2002-2006
`SB086`	2017	government ID scheme change
`SB194`	2012	dense -> sparse representation change (absence means `$0` before FY2012, `not reported` after)

Verified against the live API just now -- a FY2011 Madison Police query returns provenance.series_break_refs: [], and SB085 (which applies to it) is not surfaced.

SB194 is the one that makes this urgent: census_of_governments_finance_pipeline#64 DoD 4 was "series_breaks.csv carries an ALL @ 2012 entry describing the representation change, so cog_explain() surfaces it." The entry exists and was published, but the reader path drops it, so that intent is not actually met. The pre-existing three have been invisible the same way for longer.

Definition of done

ALL-scoped entries are returned for any query whose year range spans their break_year, regardless of which codes the result contains.
They are distinguishable from code-specific breaks in the output -- an ALL caveat applies to the whole result, not to one series.
A test asserting a FY2011 query surfaces SB085, and a query spanning 2011->2012 surfaces SB194.

Note cog_code_info() in the pipeline repo has the same exclusion (sb[sb$fin_code != "ALL", ] in R/reshape.R), deliberately, because it is a per-code lookup. This issue is about the explain path, which is not per-code.

`R/series_breaks.R` looks breaks up with ```sql WHERE fin_code IN (%s) AND break_year BETWEEN %d AND %d ``` matching against the item codes present in the result. An entry whose `fin_code` is the literal string `"ALL"` can therefore **never** match, because no row's `item_code` is `"ALL"`. Four catalogued corpus-wide breaks are affected -- every one of them a caveat that applies to *every* code: | id | year | what it says | |---|---:|---| | `SB085` | 1977 | FY1967-1976 collected in whole dollars; derived values carry +-$1k noise | | `SB087` | 2002 | imputation exclusion FY2002-2006 | | `SB086` | 2017 | government ID scheme change | | `SB194` | 2012 | **dense -> sparse representation change** (absence means `$0` before FY2012, `not reported` after) | Verified against the live API just now -- a FY2011 Madison Police query returns `provenance.series_break_refs: []`, and `SB085` (which applies to it) is not surfaced. `SB194` is the one that makes this urgent: `census_of_governments_finance_pipeline#64` DoD 4 was "`series_breaks.csv` carries an `ALL @ 2012` entry describing the representation change, **so `cog_explain()` surfaces it**." The entry exists and was published, but the reader path drops it, so that intent is not actually met. The pre-existing three have been invisible the same way for longer. ## Definition of done 1. `ALL`-scoped entries are returned for any query whose year range spans their `break_year`, regardless of which codes the result contains. 2. They are distinguishable from code-specific breaks in the output -- an `ALL` caveat applies to the whole result, not to one series. 3. A test asserting a FY2011 query surfaces `SB085`, and a query spanning 2011->2012 surfaces `SB194`. Note `cog_code_info()` in the pipeline repo has the same exclusion (`sb[sb$fin_code != "ALL", ]` in `R/reshape.R`), deliberately, because it is a per-code lookup. This issue is about the *explain* path, which is not per-code.

jared referenced this issue from a commit

2026-07-30 10:27:50 -04:00

fix: surface ALL-scoped series breaks in provenance (#19)

jared referenced a pull request that will close this issue

2026-07-30 10:28:08 -04:00

fix: surface ALL-scoped series breaks in provenance (#19) #21

jared commented

2026-07-30 10:28:26 -04:00

Implemented in PR #21 (stacked on #20).

DoD 1 and DoD 3 conflict, and I followed DoD 1.

DoD 1: ALL-scoped entries fire when the query's year range spans their break_year.
DoD 3: a FY2011 query surfaces SB085 — whose break_year is 1977.

A FY2011 query does not span 1977, so both cannot hold. Reading SB085's own row settles it:

join_advice: Source files for FY1967-1976 were in whole dollars rounded to thousands. All subtraction-based derived formulas may produce ±1 spurious values across the 1976/1977 boundary.

All four ALL entries are boundary caveats, not era caveats — crossing 1976/1977, FY2002-2006, absence not being comparable across FY2012, pre- vs post-2017 ids. So break_year BETWEEN min(years) AND max(years) — the same rule the code-specific path already uses — is the right one, and a FY2011-only query is genuinely unaffected by SB085.

SB085 is tested with a range that actually spans 1977 (years = 1975:1980, plus a negative case at 1978:1980), and the SB194 half of DoD 3 is tested exactly as written (a query spanning 2011→2012).

If the intent was instead era scoping — e.g. "any query touching FY1967-1976 gets SB085, any query touching the wide era gets SB194" — that is a different and defensible rule, but a different one; say so and I will change it.

DoD 2 is met with a separate corpus_break_refs provenance field rather than extra entries in series_break_refs: an ALL caveat qualifies the whole result, and merging them invites reading one as a caveat about a single series. The two are disjoint by construction and a test asserts it. cog-api passes provenance through verbatim, so the field reaches the API with no change there.

Implemented in PR #21 (stacked on #20). **DoD 1 and DoD 3 conflict, and I followed DoD 1.** DoD 1: ALL-scoped entries fire when the query's year range spans their `break_year`. DoD 3: a **FY2011** query surfaces `SB085` — whose `break_year` is **1977**. A FY2011 query does not span 1977, so both cannot hold. Reading `SB085`'s own row settles it: > **join_advice:** Source files for FY1967-1976 were in whole dollars rounded to thousands. All subtraction-based derived formulas may produce ±1 spurious values across the **1976/1977 boundary**. All four ALL entries are *boundary* caveats, not era caveats — crossing 1976/1977, FY2002-2006, absence not being comparable across FY2012, pre- vs post-2017 ids. So `break_year BETWEEN min(years) AND max(years)` — the same rule the code-specific path already uses — is the right one, and a FY2011-only query is genuinely unaffected by `SB085`. `SB085` is tested with a range that actually spans 1977 (`years = 1975:1980`, plus a negative case at `1978:1980`), and the `SB194` half of DoD 3 is tested exactly as written (a query spanning 2011→2012). If the intent was instead *era* scoping — e.g. "any query touching FY1967-1976 gets `SB085`, any query touching the wide era gets `SB194`" — that is a different and defensible rule, but a different one; say so and I will change it. **DoD 2** is met with a separate `corpus_break_refs` provenance field rather than extra entries in `series_break_refs`: an ALL caveat qualifies the whole result, and merging them invites reading one as a caveat about a single series. The two are disjoint by construction and a test asserts it. cog-api passes provenance through verbatim, so the field reaches the API with no change there.

jared closed this issue

2026-07-30 10:33:25 -04:00

jared referenced this issue from a commit

2026-07-30 10:33:26 -04:00

Merge pull request 'fix: surface ALL-scoped series breaks in provenance (#19)' (#21) from fix/all-scoped-series-breaks-19 into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Civilytics/uscogdata#19

fin_code = "ALL" series breaks can never surface through cog_explain() #19

Definition of done

`fin_code = "ALL"` series breaks can never surface through cog_explain() #19