docs: warn raw-parquet readers that survey_weight is not an aggregation weight
The v5 schema passes the legacy IndFin Weight column through verbatim as survey_weight. Census documents it as informational-only, and its encoding is inconsistent across vintages (reciprocal scale most years, direct in 2003, placeholder 1 in 1967-2001 gap years, all-0 in 2007-2012, NA modern), so weighting amt by it produces silently wrong totals. No uscogdata function reads the column; this warning is for direct DuckDB/arrow consumers. Evidence: cog_pipeline/.superpowers/sdd/weight-semantics-findings.md.
This commit is contained in:
@@ -25,6 +25,22 @@ package implements.
|
|||||||
- `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
|
- `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
|
||||||
- `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
|
- `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
|
||||||
|
|
||||||
|
## Raw-parquet caveat: `survey_weight` is not an aggregation weight
|
||||||
|
|
||||||
|
Users reading the corpus parquet directly (DuckDB, arrow) will see a
|
||||||
|
`survey_weight` column (schema v5, col 26). It is legacy Census IndFin
|
||||||
|
sample-design **metadata passed through verbatim** — the Census Bureau's own
|
||||||
|
source documentation says it "is for informational purposes only and should
|
||||||
|
not be used to derive any other statistics" (`_ReadMe_First_IndFin.txt`;
|
||||||
|
likewise `UserGuide.xls` Data User Note 8: "Do not use the weight field to
|
||||||
|
derive state or national totals"). The raw encoding is also inconsistent
|
||||||
|
across vintages (reciprocal scale most years, direct scale in 2003, a `1`
|
||||||
|
placeholder in 1967/70/71/73/2001, all-`0` in 2007–2012, `NA` for all
|
||||||
|
modern-source rows), so `sum(amt * survey_weight/10000)`-style expressions
|
||||||
|
produce silently wrong totals — including exact zeros for 2007–2012. Sum
|
||||||
|
`amt` unweighted; no uscogdata function reads this column. Full evidence:
|
||||||
|
`cog_pipeline/.superpowers/sdd/weight-semantics-findings.md`.
|
||||||
|
|
||||||
## Developer notes
|
## Developer notes
|
||||||
|
|
||||||
### Testing
|
### Testing
|
||||||
|
|||||||
Reference in New Issue
Block a user