docs: add design spec for empirical draw distributions via duckdb-wasm
Scopes replacing the analytic distribution approximation with real posterior draws, fetched client-side from the Hugging Face parquet dataset using duckdb-wasm — no new backend endpoint required.
This commit is contained in:
@@ -0,0 +1,187 @@
|
||||
# Empirical draw distributions via DuckDB-Wasm — Design Spec
|
||||
|
||||
**Date:** 2026-08-11
|
||||
**Status:** Draft for review
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose & context
|
||||
|
||||
Two of this app's three charts currently show a *modeled* distribution shape that is
|
||||
not the real posterior — `src/utils/distributionApprox.js` fits a two-piece-normal
|
||||
curve to each group's `(median, lower, upper)` summary stats returned by the
|
||||
`/estimates` API, because the API's raw posterior draws are only available in bulk
|
||||
as Hive-partitioned Parquet on Hugging Face
|
||||
(`civilytics/crdc-school-arrest-rates`), meant for DuckDB/bulk consumption, not
|
||||
browser fetches (see `AGENTS.md` §"API Endpoint Availability" and
|
||||
`HANDOFF.md` §"Future Enhancements").
|
||||
|
||||
This spec replaces the approximation with the **real** empirical draws (500 per
|
||||
group), fetched client-side using `@duckdb/duckdb-wasm` to query the actual Parquet
|
||||
shard for the district's state directly from Hugging Face — no new backend
|
||||
endpoint, no change to the existing summary API calls.
|
||||
|
||||
**Verified feasibility (2026-08-11):**
|
||||
- The HF dataset is public, non-gated. Each `(model_id, YEAR, LEA_STATE)` partition
|
||||
is a single file (`data_0.parquet`).
|
||||
- File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed
|
||||
by resolving the `resolve/main/...` redirect to the actual CDN blob.
|
||||
- The redirect target sends `access-control-allow-origin: *` and
|
||||
`accept-ranges: bytes` — browser `fetch()` works directly, no proxy needed.
|
||||
- Schema (from `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`):
|
||||
`LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred`, sorted
|
||||
within each shard by `(LEAID, RACE, SEX)`. `LEA_STATE`/`YEAR`/`model_id` are
|
||||
Hive-partition columns (encoded in the path, not repeated in every row).
|
||||
`stu_enroll` is **not** in the draws table — it's already available in this app
|
||||
from the existing `/estimates` summary call.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
```
|
||||
district selected (leaid, state)
|
||||
│
|
||||
▼
|
||||
resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state)
|
||||
e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/
|
||||
resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet
|
||||
│
|
||||
▼
|
||||
fetch() the shard (native fetch, follows the HF→CDN redirect automatically)
|
||||
│
|
||||
▼
|
||||
duckdb-wasm: registerFileBuffer + query
|
||||
SELECT RACE, SEX, pred FROM shard WHERE LEAID = '<leaid>'
|
||||
│
|
||||
▼
|
||||
join `pred` (posterior count draws) against stu_enroll already in app state
|
||||
(from the existing /estimates summary call) → rate-per-1000 draws per group
|
||||
│
|
||||
▼
|
||||
KDE per race×sex group → smooth density curve, same shape the charts draw today
|
||||
```
|
||||
|
||||
Given verified shard sizes (≤6.4MB), the design fetches the **whole shard** with a
|
||||
plain `fetch()` and queries it in-memory via duckdb-wasm, rather than relying on
|
||||
fine-grained HTTP range / row-group pruning. This is simpler and more robust than
|
||||
depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN
|
||||
redirect chain under partial-range requests — an unverified behavior — for a
|
||||
saving that wouldn't matter at these file sizes.
|
||||
|
||||
`@duckdb/duckdb-wasm` is MIT-licensed and runs entirely client-side in a Web
|
||||
Worker. It introduces no new server dependency and no new hosted service beyond
|
||||
the Hugging Face dataset the `crdc-arrests` project's `/draws` endpoint already
|
||||
points to (per `2026-05-30-draws-api-design.md`, decision #5) — this spec doesn't
|
||||
introduce that dependency, it makes the demo app actually use data that was
|
||||
already published there for exactly this purpose.
|
||||
|
||||
**Deployment risk:** duckdb-wasm's threaded ("eh") bundle requires
|
||||
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` response headers
|
||||
(for `SharedArrayBuffer`), which the git-pages static host does not send today.
|
||||
This design uses the **single-threaded ("mvp") bundle** instead — at these file
|
||||
sizes threading has no meaningful benefit, and it avoids needing new headers on
|
||||
both the git-pages and Docker/nginx deploy paths.
|
||||
|
||||
---
|
||||
|
||||
## 3. File-level changes
|
||||
|
||||
### New files
|
||||
- **`src/utils/duckdbClient.js`** — lazy-initialized singleton. Dynamic-imports
|
||||
`@duckdb/duckdb-wasm`, selects the MVP (non-threaded) bundle, starts the worker
|
||||
once. Dynamic `import()` keeps the ~3–5MB wasm payload out of the main bundle;
|
||||
it only loads when a chart actually needs draws.
|
||||
- **`src/utils/kde.js`** — Gaussian KDE over an array of numbers (Silverman
|
||||
bandwidth). Takes over the role `distributionApprox.js`'s `densityCurve` plays
|
||||
today, fed real empirical draws instead of a parametric fit.
|
||||
- **`src/hooks/useDrawDistribution.js`** — given
|
||||
`{ leaid, state, model, year, groups }` (groups = race/sex + `stu_enroll` already
|
||||
in app state), resolves the HF URL, fetches, registers the buffer with
|
||||
duckdb-wasm, runs the query, joins enrollment, and returns
|
||||
`{ status: 'loading' | 'ready' | 'error', drawsByGroup }`. Owns an in-memory
|
||||
`Map` cache keyed by `model+state+year` so re-selecting a model in Chart 3's
|
||||
dropdown, or viewing another district in the same state, reuses the shard
|
||||
already fetched.
|
||||
|
||||
### Modified files
|
||||
- **`RateDensityRidgeline.jsx`** — on model-dropdown change, calls
|
||||
`useDrawDistribution` for the selected model; replaces
|
||||
`fitSkewedInterval`/`densityCurve` with the hook's real draws → `kde.js`. Shows
|
||||
an inline spinner in the ridge area while that model's shard is loading (Charts
|
||||
1–2 aren't blocked).
|
||||
- **`RateByGroupBar.jsx`** — fetches draws for `unified_m3_mod` (the one model
|
||||
this chart uses) alongside its existing data fetch. `q1`/`q3` become exact
|
||||
empirical quantiles from the 500 real draws — removes this chart's use of
|
||||
`fitSkewedInterval`'s fitted quantile function entirely.
|
||||
- **`ChartPanel.jsx`** — passes `district.leaid`, `state`, and each group's
|
||||
`stu_enroll` (already fetched) down to the two charts above.
|
||||
- **`ApproxNote.jsx`** — becomes conditional: renders the "estimated shape" note
|
||||
only when a chart is in fallback mode; charts backed by real draws show no note
|
||||
(or a neutral "500 posterior draws" caption).
|
||||
- **`package.json` / `vite.config.mjs`** — add `@duckdb/duckdb-wasm`; wasm/worker
|
||||
assets are pulled in via Vite's native `?url` imports, which already respect the
|
||||
`/crdc-demo/` `base` path — no bundler plugin needed.
|
||||
|
||||
### Unchanged
|
||||
- **`ArrestsOverTime.jsx`** — already uses real summary stats (point-range from
|
||||
`/estimates`), no approximation involved; out of scope.
|
||||
- **`useApi.js`** — still the source for medians, intervals, and enrollment.
|
||||
- **`distributionApprox.js`** — kept as the fallback path (see §4).
|
||||
|
||||
---
|
||||
|
||||
## 4. Error handling & caching
|
||||
|
||||
**Fallback:** `distributionApprox.js` is retained. `useDrawDistribution` catches
|
||||
fetch/wasm/query failures and returns `status: 'error'`; both chart components
|
||||
branch on that to render the current analytic-approximation path with
|
||||
`<ApproxNote />` visible. A Hugging Face outage, a network failure, or an
|
||||
unsupported browser degrades to today's behavior rather than breaking the chart.
|
||||
|
||||
**Caching:** in-memory only (a `Map` inside the hook), scoped to the browser
|
||||
session. No IndexedDB/persistent cache in this iteration — a demo session
|
||||
typically covers one or two districts, and shards are cheap enough to refetch on
|
||||
reload.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing
|
||||
|
||||
This project has no automated test suite (per `AGENTS.md`); this follows the
|
||||
existing manual-verification convention:
|
||||
|
||||
1. `npm run dev`; walk a small state (DC or WY, ~130–200KB shard) and a large one
|
||||
(CA or TX, several MB) through the full district-search flow.
|
||||
2. Confirm both charts render from real draws; confirm the Network tab shows the
|
||||
expected parquet fetch(es) and sizes.
|
||||
3. Simulate failure (block the `huggingface.co` / CDN domain in devtools) and
|
||||
confirm both charts fall back to the analytic approximation with the note
|
||||
visible, rather than breaking.
|
||||
4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is
|
||||
instant on re-selection (cache hit).
|
||||
|
||||
---
|
||||
|
||||
## 6. Open risk to de-risk first
|
||||
|
||||
Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query
|
||||
correctly when deployed under the `/crdc-demo/` subpath on git-pages, and does
|
||||
nginx/git-pages serve `.wasm` with a usable content type? Everything else in this
|
||||
design is standard Vite asset handling already exercised elsewhere in the app, but
|
||||
this specific combination (wasm worker + subpath base + static host) hasn't been
|
||||
verified end-to-end and should be checked with a throwaway spike rather than
|
||||
assumed.
|
||||
|
||||
---
|
||||
|
||||
## 7. Explicitly out of scope
|
||||
|
||||
- `ArrestsOverTime.jsx` (Chart 1) — no approximation to replace.
|
||||
- A new server-side `/draws`-streaming API endpoint — explicitly rejected in favor
|
||||
of client-side wasm access, per the brainstorming decision that led to this spec.
|
||||
- Persistent (IndexedDB) caching of fetched shards.
|
||||
- Fine-grained HTTP range / row-group-level partial reads — shard sizes are small
|
||||
enough that whole-file fetch is simpler and sufficiently fast.
|
||||
- Extending empirical draws to national/exceedance-probability views — those
|
||||
charts aren't part of the current 3-chart demo.
|
||||
Reference in New Issue
Block a user