jared
2cb04ad9c1
Scopes replacing the analytic distribution approximation with real posterior draws, fetched client-side from the Hugging Face parquet dataset using duckdb-wasm — no new backend endpoint required.
188 lines
9.0 KiB
Markdown
188 lines
9.0 KiB
Markdown
# Empirical draw distributions via DuckDB-Wasm — Design Spec
|
||
|
||
**Date:** 2026-08-11
|
||
**Status:** Draft for review
|
||
|
||
---
|
||
|
||
## 1. Purpose & context
|
||
|
||
Two of this app's three charts currently show a *modeled* distribution shape that is
|
||
not the real posterior — `src/utils/distributionApprox.js` fits a two-piece-normal
|
||
curve to each group's `(median, lower, upper)` summary stats returned by the
|
||
`/estimates` API, because the API's raw posterior draws are only available in bulk
|
||
as Hive-partitioned Parquet on Hugging Face
|
||
(`civilytics/crdc-school-arrest-rates`), meant for DuckDB/bulk consumption, not
|
||
browser fetches (see `AGENTS.md` §"API Endpoint Availability" and
|
||
`HANDOFF.md` §"Future Enhancements").
|
||
|
||
This spec replaces the approximation with the **real** empirical draws (500 per
|
||
group), fetched client-side using `@duckdb/duckdb-wasm` to query the actual Parquet
|
||
shard for the district's state directly from Hugging Face — no new backend
|
||
endpoint, no change to the existing summary API calls.
|
||
|
||
**Verified feasibility (2026-08-11):**
|
||
- The HF dataset is public, non-gated. Each `(model_id, YEAR, LEA_STATE)` partition
|
||
is a single file (`data_0.parquet`).
|
||
- File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed
|
||
by resolving the `resolve/main/...` redirect to the actual CDN blob.
|
||
- The redirect target sends `access-control-allow-origin: *` and
|
||
`accept-ranges: bytes` — browser `fetch()` works directly, no proxy needed.
|
||
- Schema (from `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`):
|
||
`LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred`, sorted
|
||
within each shard by `(LEAID, RACE, SEX)`. `LEA_STATE`/`YEAR`/`model_id` are
|
||
Hive-partition columns (encoded in the path, not repeated in every row).
|
||
`stu_enroll` is **not** in the draws table — it's already available in this app
|
||
from the existing `/estimates` summary call.
|
||
|
||
---
|
||
|
||
## 2. Architecture
|
||
|
||
```
|
||
district selected (leaid, state)
|
||
│
|
||
▼
|
||
resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state)
|
||
e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/
|
||
resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet
|
||
│
|
||
▼
|
||
fetch() the shard (native fetch, follows the HF→CDN redirect automatically)
|
||
│
|
||
▼
|
||
duckdb-wasm: registerFileBuffer + query
|
||
SELECT RACE, SEX, pred FROM shard WHERE LEAID = '<leaid>'
|
||
│
|
||
▼
|
||
join `pred` (posterior count draws) against stu_enroll already in app state
|
||
(from the existing /estimates summary call) → rate-per-1000 draws per group
|
||
│
|
||
▼
|
||
KDE per race×sex group → smooth density curve, same shape the charts draw today
|
||
```
|
||
|
||
Given verified shard sizes (≤6.4MB), the design fetches the **whole shard** with a
|
||
plain `fetch()` and queries it in-memory via duckdb-wasm, rather than relying on
|
||
fine-grained HTTP range / row-group pruning. This is simpler and more robust than
|
||
depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN
|
||
redirect chain under partial-range requests — an unverified behavior — for a
|
||
saving that wouldn't matter at these file sizes.
|
||
|
||
`@duckdb/duckdb-wasm` is MIT-licensed and runs entirely client-side in a Web
|
||
Worker. It introduces no new server dependency and no new hosted service beyond
|
||
the Hugging Face dataset the `crdc-arrests` project's `/draws` endpoint already
|
||
points to (per `2026-05-30-draws-api-design.md`, decision #5) — this spec doesn't
|
||
introduce that dependency, it makes the demo app actually use data that was
|
||
already published there for exactly this purpose.
|
||
|
||
**Deployment risk:** duckdb-wasm's threaded ("eh") bundle requires
|
||
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` response headers
|
||
(for `SharedArrayBuffer`), which the git-pages static host does not send today.
|
||
This design uses the **single-threaded ("mvp") bundle** instead — at these file
|
||
sizes threading has no meaningful benefit, and it avoids needing new headers on
|
||
both the git-pages and Docker/nginx deploy paths.
|
||
|
||
---
|
||
|
||
## 3. File-level changes
|
||
|
||
### New files
|
||
- **`src/utils/duckdbClient.js`** — lazy-initialized singleton. Dynamic-imports
|
||
`@duckdb/duckdb-wasm`, selects the MVP (non-threaded) bundle, starts the worker
|
||
once. Dynamic `import()` keeps the ~3–5MB wasm payload out of the main bundle;
|
||
it only loads when a chart actually needs draws.
|
||
- **`src/utils/kde.js`** — Gaussian KDE over an array of numbers (Silverman
|
||
bandwidth). Takes over the role `distributionApprox.js`'s `densityCurve` plays
|
||
today, fed real empirical draws instead of a parametric fit.
|
||
- **`src/hooks/useDrawDistribution.js`** — given
|
||
`{ leaid, state, model, year, groups }` (groups = race/sex + `stu_enroll` already
|
||
in app state), resolves the HF URL, fetches, registers the buffer with
|
||
duckdb-wasm, runs the query, joins enrollment, and returns
|
||
`{ status: 'loading' | 'ready' | 'error', drawsByGroup }`. Owns an in-memory
|
||
`Map` cache keyed by `model+state+year` so re-selecting a model in Chart 3's
|
||
dropdown, or viewing another district in the same state, reuses the shard
|
||
already fetched.
|
||
|
||
### Modified files
|
||
- **`RateDensityRidgeline.jsx`** — on model-dropdown change, calls
|
||
`useDrawDistribution` for the selected model; replaces
|
||
`fitSkewedInterval`/`densityCurve` with the hook's real draws → `kde.js`. Shows
|
||
an inline spinner in the ridge area while that model's shard is loading (Charts
|
||
1–2 aren't blocked).
|
||
- **`RateByGroupBar.jsx`** — fetches draws for `unified_m3_mod` (the one model
|
||
this chart uses) alongside its existing data fetch. `q1`/`q3` become exact
|
||
empirical quantiles from the 500 real draws — removes this chart's use of
|
||
`fitSkewedInterval`'s fitted quantile function entirely.
|
||
- **`ChartPanel.jsx`** — passes `district.leaid`, `state`, and each group's
|
||
`stu_enroll` (already fetched) down to the two charts above.
|
||
- **`ApproxNote.jsx`** — becomes conditional: renders the "estimated shape" note
|
||
only when a chart is in fallback mode; charts backed by real draws show no note
|
||
(or a neutral "500 posterior draws" caption).
|
||
- **`package.json` / `vite.config.mjs`** — add `@duckdb/duckdb-wasm`; wasm/worker
|
||
assets are pulled in via Vite's native `?url` imports, which already respect the
|
||
`/crdc-demo/` `base` path — no bundler plugin needed.
|
||
|
||
### Unchanged
|
||
- **`ArrestsOverTime.jsx`** — already uses real summary stats (point-range from
|
||
`/estimates`), no approximation involved; out of scope.
|
||
- **`useApi.js`** — still the source for medians, intervals, and enrollment.
|
||
- **`distributionApprox.js`** — kept as the fallback path (see §4).
|
||
|
||
---
|
||
|
||
## 4. Error handling & caching
|
||
|
||
**Fallback:** `distributionApprox.js` is retained. `useDrawDistribution` catches
|
||
fetch/wasm/query failures and returns `status: 'error'`; both chart components
|
||
branch on that to render the current analytic-approximation path with
|
||
`<ApproxNote />` visible. A Hugging Face outage, a network failure, or an
|
||
unsupported browser degrades to today's behavior rather than breaking the chart.
|
||
|
||
**Caching:** in-memory only (a `Map` inside the hook), scoped to the browser
|
||
session. No IndexedDB/persistent cache in this iteration — a demo session
|
||
typically covers one or two districts, and shards are cheap enough to refetch on
|
||
reload.
|
||
|
||
---
|
||
|
||
## 5. Testing
|
||
|
||
This project has no automated test suite (per `AGENTS.md`); this follows the
|
||
existing manual-verification convention:
|
||
|
||
1. `npm run dev`; walk a small state (DC or WY, ~130–200KB shard) and a large one
|
||
(CA or TX, several MB) through the full district-search flow.
|
||
2. Confirm both charts render from real draws; confirm the Network tab shows the
|
||
expected parquet fetch(es) and sizes.
|
||
3. Simulate failure (block the `huggingface.co` / CDN domain in devtools) and
|
||
confirm both charts fall back to the analytic approximation with the note
|
||
visible, rather than breaking.
|
||
4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is
|
||
instant on re-selection (cache hit).
|
||
|
||
---
|
||
|
||
## 6. Open risk to de-risk first
|
||
|
||
Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query
|
||
correctly when deployed under the `/crdc-demo/` subpath on git-pages, and does
|
||
nginx/git-pages serve `.wasm` with a usable content type? Everything else in this
|
||
design is standard Vite asset handling already exercised elsewhere in the app, but
|
||
this specific combination (wasm worker + subpath base + static host) hasn't been
|
||
verified end-to-end and should be checked with a throwaway spike rather than
|
||
assumed.
|
||
|
||
---
|
||
|
||
## 7. Explicitly out of scope
|
||
|
||
- `ArrestsOverTime.jsx` (Chart 1) — no approximation to replace.
|
||
- A new server-side `/draws`-streaming API endpoint — explicitly rejected in favor
|
||
of client-side wasm access, per the brainstorming decision that led to this spec.
|
||
- Persistent (IndexedDB) caching of fetched shards.
|
||
- Fine-grained HTTP range / row-group-level partial reads — shard sizes are small
|
||
enough that whole-file fetch is simpler and sufficiently fast.
|
||
- Extending empirical draws to national/exceedance-probability views — those
|
||
charts aren't part of the current 3-chart demo.
|