# Empirical draw distributions via DuckDB-Wasm — Design Spec **Date:** 2026-08-11 **Status:** Draft for review --- ## 1. Purpose & context Two of this app's three charts currently show a *modeled* distribution shape that is not the real posterior — `src/utils/distributionApprox.js` fits a two-piece-normal curve to each group's `(median, lower, upper)` summary stats returned by the `/estimates` API, because the API's raw posterior draws are only available in bulk as Hive-partitioned Parquet on Hugging Face (`civilytics/crdc-school-arrest-rates`), meant for DuckDB/bulk consumption, not browser fetches (see `AGENTS.md` §"API Endpoint Availability" and `HANDOFF.md` §"Future Enhancements"). This spec replaces the approximation with the **real** empirical draws (500 per group), fetched client-side using `@duckdb/duckdb-wasm` to query the actual Parquet shard for the district's state directly from Hugging Face — no new backend endpoint, no change to the existing summary API calls. **Verified feasibility (2026-08-11):** - The HF dataset is public, non-gated. Each `(model_id, YEAR, LEA_STATE)` partition is a single file (`data_0.parquet`). - File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed by resolving the `resolve/main/...` redirect to the actual CDN blob. - The redirect target sends `access-control-allow-origin: *` and `accept-ranges: bytes` — browser `fetch()` works directly, no proxy needed. - Schema (from `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`): `LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred`, sorted within each shard by `(LEAID, RACE, SEX)`. `LEA_STATE`/`YEAR`/`model_id` are Hive-partition columns (encoded in the path, not repeated in every row). `stu_enroll` is **not** in the draws table — it's already available in this app from the existing `/estimates` summary call. --- ## 2. Architecture ``` district selected (leaid, state) │ ▼ resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state) e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/ resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet │ ▼ fetch() the shard (native fetch, follows the HF→CDN redirect automatically) │ ▼ duckdb-wasm: registerFileBuffer + query SELECT RACE, SEX, pred FROM shard WHERE LEAID = '' │ ▼ join `pred` (posterior count draws) against stu_enroll already in app state (from the existing /estimates summary call) → rate-per-1000 draws per group │ ▼ KDE per race×sex group → smooth density curve, same shape the charts draw today ``` Given verified shard sizes (≤6.4MB), the design fetches the **whole shard** with a plain `fetch()` and queries it in-memory via duckdb-wasm, rather than relying on fine-grained HTTP range / row-group pruning. This is simpler and more robust than depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN redirect chain under partial-range requests — an unverified behavior — for a saving that wouldn't matter at these file sizes. `@duckdb/duckdb-wasm` is MIT-licensed and runs entirely client-side in a Web Worker. It introduces no new server dependency and no new hosted service beyond the Hugging Face dataset the `crdc-arrests` project's `/draws` endpoint already points to (per `2026-05-30-draws-api-design.md`, decision #5) — this spec doesn't introduce that dependency, it makes the demo app actually use data that was already published there for exactly this purpose. **Deployment risk:** duckdb-wasm's threaded ("eh") bundle requires `Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` response headers (for `SharedArrayBuffer`), which the git-pages static host does not send today. This design uses the **single-threaded ("mvp") bundle** instead — at these file sizes threading has no meaningful benefit, and it avoids needing new headers on both the git-pages and Docker/nginx deploy paths. --- ## 3. File-level changes ### New files - **`src/utils/duckdbClient.js`** — lazy-initialized singleton. Dynamic-imports `@duckdb/duckdb-wasm`, selects the MVP (non-threaded) bundle, starts the worker once. Dynamic `import()` keeps the ~3–5MB wasm payload out of the main bundle; it only loads when a chart actually needs draws. - **`src/utils/kde.js`** — Gaussian KDE over an array of numbers (Silverman bandwidth). Takes over the role `distributionApprox.js`'s `densityCurve` plays today, fed real empirical draws instead of a parametric fit. - **`src/hooks/useDrawDistribution.js`** — given `{ leaid, state, model, year, groups }` (groups = race/sex + `stu_enroll` already in app state), resolves the HF URL, fetches, registers the buffer with duckdb-wasm, runs the query, joins enrollment, and returns `{ status: 'loading' | 'ready' | 'error', drawsByGroup }`. Owns an in-memory `Map` cache keyed by `model+state+year` so re-selecting a model in Chart 3's dropdown, or viewing another district in the same state, reuses the shard already fetched. ### Modified files - **`RateDensityRidgeline.jsx`** — on model-dropdown change, calls `useDrawDistribution` for the selected model; replaces `fitSkewedInterval`/`densityCurve` with the hook's real draws → `kde.js`. Shows an inline spinner in the ridge area while that model's shard is loading (Charts 1–2 aren't blocked). - **`RateByGroupBar.jsx`** — fetches draws for `unified_m3_mod` (the one model this chart uses) alongside its existing data fetch. `q1`/`q3` become exact empirical quantiles from the 500 real draws — removes this chart's use of `fitSkewedInterval`'s fitted quantile function entirely. - **`ChartPanel.jsx`** — passes `district.leaid`, `state`, and each group's `stu_enroll` (already fetched) down to the two charts above. - **`ApproxNote.jsx`** — becomes conditional: renders the "estimated shape" note only when a chart is in fallback mode; charts backed by real draws show no note (or a neutral "500 posterior draws" caption). - **`package.json` / `vite.config.mjs`** — add `@duckdb/duckdb-wasm`; wasm/worker assets are pulled in via Vite's native `?url` imports, which already respect the `/crdc-demo/` `base` path — no bundler plugin needed. ### Unchanged - **`ArrestsOverTime.jsx`** — already uses real summary stats (point-range from `/estimates`), no approximation involved; out of scope. - **`useApi.js`** — still the source for medians, intervals, and enrollment. - **`distributionApprox.js`** — kept as the fallback path (see §4). --- ## 4. Error handling & caching **Fallback:** `distributionApprox.js` is retained. `useDrawDistribution` catches fetch/wasm/query failures and returns `status: 'error'`; both chart components branch on that to render the current analytic-approximation path with `` visible. A Hugging Face outage, a network failure, or an unsupported browser degrades to today's behavior rather than breaking the chart. **Caching:** in-memory only (a `Map` inside the hook), scoped to the browser session. No IndexedDB/persistent cache in this iteration — a demo session typically covers one or two districts, and shards are cheap enough to refetch on reload. --- ## 5. Testing This project has no automated test suite (per `AGENTS.md`); this follows the existing manual-verification convention: 1. `npm run dev`; walk a small state (DC or WY, ~130–200KB shard) and a large one (CA or TX, several MB) through the full district-search flow. 2. Confirm both charts render from real draws; confirm the Network tab shows the expected parquet fetch(es) and sizes. 3. Simulate failure (block the `huggingface.co` / CDN domain in devtools) and confirm both charts fall back to the analytic approximation with the note visible, rather than breaking. 4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is instant on re-selection (cache hit). --- ## 6. Open risk to de-risk first Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query correctly when deployed under the `/crdc-demo/` subpath on git-pages, and does nginx/git-pages serve `.wasm` with a usable content type? Everything else in this design is standard Vite asset handling already exercised elsewhere in the app, but this specific combination (wasm worker + subpath base + static host) hasn't been verified end-to-end and should be checked with a throwaway spike rather than assumed. --- ## 7. Explicitly out of scope - `ArrestsOverTime.jsx` (Chart 1) — no approximation to replace. - A new server-side `/draws`-streaming API endpoint — explicitly rejected in favor of client-side wasm access, per the brainstorming decision that led to this spec. - Persistent (IndexedDB) caching of fetched shards. - Fine-grained HTTP range / row-group-level partial reads — shard sizes are small enough that whole-file fetch is simpler and sufficiently fast. - Extending empirical draws to national/exceedance-probability views — those charts aren't part of the current 3-chart demo.