Add complete= so wide-era census zeros are recoverable after sparsification
#18
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Civilytics/uscogdata#18
Reference in New Issue
Block a user
Follow-on from
census_of_governments_finance_pipeline#64DoD 7. Filed after the corpus was sparsified and published on 2026-07-29 (pipeline_commit 83f9715).What changed under us
The published corpus no longer stores the wide era's explicit zeros. 331.4M rows -> 45.9M (-86%). Absence now means two different things:
dense_source)sparse_source)Two new corpus artifacts carry the rule:
data/representation.parquet(per year:representation,absence_means,code_set_id) anddata/code_set.parquet(code_set_id,year,type,item_code,is_aggregate,n_units-- 84,693 rows across 55 years). The government universe is already incanonical_fips_xwalk. Catalogued asSB194.The user-visible consequence
A wide-era query whose rows were all
$0now returns nothing instead of$0rows, and the reader offers no way to get them back. Verified against the live corpus:cog_spending(govid="552025209777", years=2005L)returns 27 rows, 0 of them zero-amount. Before sparsification the all-zero categories came back as explicit$0.This is strictly less information than the reader used to expose, which is why #64 filed this follow-on.
What to build
complete = TRUEon the money verbs: fill the requested grid fromcode_set(scoped to each government'stype-- filling against the union of all types invents rows like "$0 state IG transfer to school districts" for counties) and stamp each row'svalue_source:reportedlongcensus_zeronot_reportedStrictly more information than the reader had before sparsification, where a wide-era zero and a modern absent cell were different shapes with nothing explaining the difference.
Prerequisite
The bundled fixture corpus is stale and must be regenerated first.
inst/extdata/fixture_corpus/has noJrows insummary_categories(it predates the #65 crosswalk work), norepresentation.parquet, nocode_set.parquet, and a still-dense wide era. Every test in this package and in cog-api runs against it, so it currently does not represent what is published.Definition of done
complete=implemented oncog_spending()/cog_revenue(), defaulting to today's behaviour.value_sourceon every returned row whencomplete = TRUE.complete = TRUEreproduces the pre-sparsification row set exactly, zeros included.cog-apifollow-on filed (threadingcomplete=behind its own pagination fixes).The pipeline asserts this same round-trip on every build -- see
tests/testthat/test-end-to-end.Rin that repo for the reference implementation of the densification.Reopened. PR #20 landed DoD 1 only — the fixture regeneration this issue names as its prerequisite. My PR body said "closes #18's stated prerequisite" and Gitea's keyword parser took the
closes #18out of it; my mistake.Still open:
pipeline_commit 83f9715, wide era sparse,representation.parquet+code_set.parquetnow shipped).complete=oncog_spending()/cog_revenue(), defaulting to today's behaviour.value_source(reported/census_zero/not_reported) on every row whencomplete = TRUE.complete = TRUEreproduces the pre-sparsification row set exactly, zeros included.complete=behind its own pagination fixes — cog-api#6).The fixture now carries everything DoD 2-4 need, so the remaining work is self-contained. Note the reader has no views over
representation/code_setyet: the parquets ship in the fixture and the publish tree, butinst/sql/registers nothing for them, so step 2 starts with two new view definitions.