crdc-demo/docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md

Empirical draw distributions via DuckDB-Wasm — Design Spec

Date: 2026-08-11 Status: Draft for review

---

1. Purpose & context

Two of this app's three charts currently show a modeled distribution shape that is not the real posterior — src/utils/distributionApprox.js fits a two-piece-normal curve to each group's (median, lower, upper) summary stats returned by the /estimates API, because the API's raw posterior draws are only available in bulk as Hive-partitioned Parquet on Hugging Face (civilytics/crdc-school-arrest-rates), meant for DuckDB/bulk consumption, not browser fetches (see AGENTS.md §"API Endpoint Availability" and HANDOFF.md §"Future Enhancements").

This spec replaces the approximation with the real empirical draws (500 per group), fetched client-side using @duckdb/duckdb-wasm to query the actual Parquet shard for the district's state directly from Hugging Face — no new backend endpoint, no change to the existing summary API calls.

Verified feasibility (2026-08-11):

• The HF dataset is public, non-gated. Each (model_id, YEAR, LEA_STATE) partition is a single file (data_0.parquet).
• File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed by resolving the resolve/main/... redirect to the actual CDN blob.
• The redirect target sends access-control-allow-origin: * and accept-ranges: bytes — browser fetch() works directly, no proxy needed.
• Schema (from crdc-arrests/R/postprocess.R + R/export_parquet.R): LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred, sorted within each shard by (LEAID, RACE, SEX). LEA_STATE/YEAR/model_id are Hive-partition columns (encoded in the path, not repeated in every row). stu_enroll is not in the draws table — it's already available in this app from the existing /estimates summary call.

---

2. Architecture

district selected (leaid, state)
        │
        ▼
resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state)
  e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/
       resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet
        │
        ▼
fetch() the shard (native fetch, follows the HF→CDN redirect automatically)
        │
        ▼
duckdb-wasm: registerFileBuffer + query
  SELECT RACE, SEX, pred FROM shard WHERE LEAID = '<leaid>'
        │
        ▼
join `pred` (posterior count draws) against stu_enroll already in app state
  (from the existing /estimates summary call) → rate-per-1000 draws per group
        │
        ▼
KDE per race×sex group → smooth density curve, same shape the charts draw today

Given verified shard sizes (≤6.4MB), the design fetches the whole shard with a plain fetch() and queries it in-memory via duckdb-wasm, rather than relying on fine-grained HTTP range / row-group pruning. This is simpler and more robust than depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN redirect chain under partial-range requests — an unverified behavior — for a saving that wouldn't matter at these file sizes.

@duckdb/duckdb-wasm is MIT-licensed and runs entirely client-side in a Web Worker. It introduces no new server dependency and no new hosted service beyond the Hugging Face dataset the crdc-arrests project's /draws endpoint already points to (per 2026-05-30-draws-api-design.md, decision #5) — this spec doesn't introduce that dependency, it makes the demo app actually use data that was already published there for exactly this purpose.

Deployment risk: duckdb-wasm's threaded ("eh") bundle requires Cross-Origin-Opener-Policy / Cross-Origin-Embedder-Policy response headers (for SharedArrayBuffer), which the git-pages static host does not send today. This design uses the single-threaded ("mvp") bundle instead — at these file sizes threading has no meaningful benefit, and it avoids needing new headers on both the git-pages and Docker/nginx deploy paths.

---

3. File-level changes

New files

• src/utils/duckdbClient.js — lazy-initialized singleton. Dynamic-imports @duckdb/duckdb-wasm, selects the MVP (non-threaded) bundle, starts the worker once. Dynamic import() keeps the ~3–5MB wasm payload out of the main bundle; it only loads when a chart actually needs draws.
• src/utils/kde.js — Gaussian KDE over an array of numbers (Silverman bandwidth). Takes over the role distributionApprox.js's densityCurve plays today, fed real empirical draws instead of a parametric fit.
• src/hooks/useDrawDistribution.js — given { leaid, state, model, year, groups } (groups = race/sex + stu_enroll already in app state), resolves the HF URL, fetches, registers the buffer with duckdb-wasm, runs the query, joins enrollment, and returns { status: 'loading' | 'ready' | 'error', drawsByGroup }. Owns an in-memory Map cache keyed by model+state+year so re-selecting a model in Chart 3's dropdown, or viewing another district in the same state, reuses the shard already fetched.

Modified files

• RateDensityRidgeline.jsx — on model-dropdown change, calls useDrawDistribution for the selected model; replaces fitSkewedInterval/densityCurve with the hook's real draws → kde.js. Shows an inline spinner in the ridge area while that model's shard is loading (Charts 1–2 aren't blocked).
• RateByGroupBar.jsx — fetches draws for unified_m3_mod (the one model this chart uses) alongside its existing data fetch. q1/q3 become exact empirical quantiles from the 500 real draws — removes this chart's use of fitSkewedInterval's fitted quantile function entirely.
• ChartPanel.jsx — passes district.leaid, state, and each group's stu_enroll (already fetched) down to the two charts above.
• ApproxNote.jsx — becomes conditional: renders the "estimated shape" note only when a chart is in fallback mode; charts backed by real draws show no note (or a neutral "500 posterior draws" caption).
• package.json / vite.config.mjs — add @duckdb/duckdb-wasm; wasm/worker assets are pulled in via Vite's native ?url imports, which already respect the /crdc-demo/ base path — no bundler plugin needed.

Unchanged

• ArrestsOverTime.jsx — already uses real summary stats (point-range from /estimates), no approximation involved; out of scope.
• useApi.js — still the source for medians, intervals, and enrollment.
• distributionApprox.js — kept as the fallback path (see §4).

---

4. Error handling & caching

Fallback: distributionApprox.js is retained. useDrawDistribution catches fetch/wasm/query failures and returns status: 'error'; both chart components branch on that to render the current analytic-approximation path with <ApproxNote /> visible. A Hugging Face outage, a network failure, or an unsupported browser degrades to today's behavior rather than breaking the chart.

Caching: in-memory only (a Map inside the hook), scoped to the browser session. No IndexedDB/persistent cache in this iteration — a demo session typically covers one or two districts, and shards are cheap enough to refetch on reload.

---

5. Testing

This project has no automated test suite (per AGENTS.md); this follows the existing manual-verification convention:

1. npm run dev; walk a small state (DC or WY, ~130–200KB shard) and a large one (CA or TX, several MB) through the full district-search flow.
2. Confirm both charts render from real draws; confirm the Network tab shows the expected parquet fetch(es) and sizes.
3. Simulate failure (block the huggingface.co / CDN domain in devtools) and confirm both charts fall back to the analytic approximation with the note visible, rather than breaking.
4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is instant on re-selection (cache hit).

---

6. Open risk to de-risk first

Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query correctly when deployed under the /crdc-demo/ subpath on git-pages, and does nginx/git-pages serve .wasm with a usable content type? Everything else in this design is standard Vite asset handling already exercised elsewhere in the app, but this specific combination (wasm worker + subpath base + static host) hasn't been verified end-to-end and should be checked with a throwaway spike rather than assumed.

---

7. Explicitly out of scope

• ArrestsOverTime.jsx (Chart 1) — no approximation to replace.
• A new server-side /draws-streaming API endpoint — explicitly rejected in favor of client-side wasm access, per the brainstorming decision that led to this spec.
• Persistent (IndexedDB) caching of fetched shards.
• Fine-grained HTTP range / row-group-level partial reads — shard sizes are small enough that whole-file fetch is simpler and sufficiently fast.
• Extending empirical draws to national/exceedance-probability views — those charts aren't part of the current 3-chart demo.