docs: warn raw-parquet readers that survey_weight is not an aggregation weight

The v5 schema passes the legacy IndFin Weight column through verbatim as
survey_weight. Census documents it as informational-only, and its encoding
is inconsistent across vintages (reciprocal scale most years, direct in
2003, placeholder 1 in 1967-2001 gap years, all-0 in 2007-2012, NA modern),
so weighting amt by it produces silently wrong totals. No uscogdata function
reads the column; this warning is for direct DuckDB/arrow consumers.
Evidence: cog_pipeline/.superpowers/sdd/weight-semantics-findings.md.
This commit is contained in:
2026-07-23 09:45:48 -04:00
parent ac4289d369
commit 49d80e060e
+16
View File
@@ -25,6 +25,22 @@ package implements.
- `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
- `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
## Raw-parquet caveat: `survey_weight` is not an aggregation weight
Users reading the corpus parquet directly (DuckDB, arrow) will see a
`survey_weight` column (schema v5, col 26). It is legacy Census IndFin
sample-design **metadata passed through verbatim** — the Census Bureau's own
source documentation says it "is for informational purposes only and should
not be used to derive any other statistics" (`_ReadMe_First_IndFin.txt`;
likewise `UserGuide.xls` Data User Note 8: "Do not use the weight field to
derive state or national totals"). The raw encoding is also inconsistent
across vintages (reciprocal scale most years, direct scale in 2003, a `1`
placeholder in 1967/70/71/73/2001, all-`0` in 2007–2012, `NA` for all
modern-source rows), so `sum(amt * survey_weight/10000)`-style expressions
produce silently wrong totals — including exact zeros for 2007–2012. Sum
`amt` unweighted; no uscogdata function reads this column. Full evidence:
`cog_pipeline/.superpowers/sdd/weight-semantics-findings.md`.
## Developer notes
### Testing