From 2cb04ad9c1d454d7a8509d4af586f3552a81c2b6 Mon Sep 17 00:00:00 2001 From: Jared Knowles Date: Tue, 11 Aug 2026 07:41:21 -0400 Subject: [PATCH] docs: add design spec for empirical draw distributions via duckdb-wasm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scopes replacing the analytic distribution approximation with real posterior draws, fetched client-side from the Hugging Face parquet dataset using duckdb-wasm — no new backend endpoint required. --- .../2026-08-11-empirical-draws-wasm-design.md | 187 ++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md diff --git a/docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md b/docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md new file mode 100644 index 0000000..80e3d08 --- /dev/null +++ b/docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md @@ -0,0 +1,187 @@ +# Empirical draw distributions via DuckDB-Wasm — Design Spec + +**Date:** 2026-08-11 +**Status:** Draft for review + +--- + +## 1. Purpose & context + +Two of this app's three charts currently show a *modeled* distribution shape that is +not the real posterior — `src/utils/distributionApprox.js` fits a two-piece-normal +curve to each group's `(median, lower, upper)` summary stats returned by the +`/estimates` API, because the API's raw posterior draws are only available in bulk +as Hive-partitioned Parquet on Hugging Face +(`civilytics/crdc-school-arrest-rates`), meant for DuckDB/bulk consumption, not +browser fetches (see `AGENTS.md` §"API Endpoint Availability" and +`HANDOFF.md` §"Future Enhancements"). + +This spec replaces the approximation with the **real** empirical draws (500 per +group), fetched client-side using `@duckdb/duckdb-wasm` to query the actual Parquet +shard for the district's state directly from Hugging Face — no new backend +endpoint, no change to the existing summary API calls. + +**Verified feasibility (2026-08-11):** +- The HF dataset is public, non-gated. Each `(model_id, YEAR, LEA_STATE)` partition + is a single file (`data_0.parquet`). +- File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed + by resolving the `resolve/main/...` redirect to the actual CDN blob. +- The redirect target sends `access-control-allow-origin: *` and + `accept-ranges: bytes` — browser `fetch()` works directly, no proxy needed. +- Schema (from `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`): + `LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred`, sorted + within each shard by `(LEAID, RACE, SEX)`. `LEA_STATE`/`YEAR`/`model_id` are + Hive-partition columns (encoded in the path, not repeated in every row). + `stu_enroll` is **not** in the draws table — it's already available in this app + from the existing `/estimates` summary call. + +--- + +## 2. Architecture + +``` +district selected (leaid, state) + │ + ▼ +resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state) + e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/ + resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet + │ + ▼ +fetch() the shard (native fetch, follows the HF→CDN redirect automatically) + │ + ▼ +duckdb-wasm: registerFileBuffer + query + SELECT RACE, SEX, pred FROM shard WHERE LEAID = '' + │ + ▼ +join `pred` (posterior count draws) against stu_enroll already in app state + (from the existing /estimates summary call) → rate-per-1000 draws per group + │ + ▼ +KDE per race×sex group → smooth density curve, same shape the charts draw today +``` + +Given verified shard sizes (≤6.4MB), the design fetches the **whole shard** with a +plain `fetch()` and queries it in-memory via duckdb-wasm, rather than relying on +fine-grained HTTP range / row-group pruning. This is simpler and more robust than +depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN +redirect chain under partial-range requests — an unverified behavior — for a +saving that wouldn't matter at these file sizes. + +`@duckdb/duckdb-wasm` is MIT-licensed and runs entirely client-side in a Web +Worker. It introduces no new server dependency and no new hosted service beyond +the Hugging Face dataset the `crdc-arrests` project's `/draws` endpoint already +points to (per `2026-05-30-draws-api-design.md`, decision #5) — this spec doesn't +introduce that dependency, it makes the demo app actually use data that was +already published there for exactly this purpose. + +**Deployment risk:** duckdb-wasm's threaded ("eh") bundle requires +`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` response headers +(for `SharedArrayBuffer`), which the git-pages static host does not send today. +This design uses the **single-threaded ("mvp") bundle** instead — at these file +sizes threading has no meaningful benefit, and it avoids needing new headers on +both the git-pages and Docker/nginx deploy paths. + +--- + +## 3. File-level changes + +### New files +- **`src/utils/duckdbClient.js`** — lazy-initialized singleton. Dynamic-imports + `@duckdb/duckdb-wasm`, selects the MVP (non-threaded) bundle, starts the worker + once. Dynamic `import()` keeps the ~3–5MB wasm payload out of the main bundle; + it only loads when a chart actually needs draws. +- **`src/utils/kde.js`** — Gaussian KDE over an array of numbers (Silverman + bandwidth). Takes over the role `distributionApprox.js`'s `densityCurve` plays + today, fed real empirical draws instead of a parametric fit. +- **`src/hooks/useDrawDistribution.js`** — given + `{ leaid, state, model, year, groups }` (groups = race/sex + `stu_enroll` already + in app state), resolves the HF URL, fetches, registers the buffer with + duckdb-wasm, runs the query, joins enrollment, and returns + `{ status: 'loading' | 'ready' | 'error', drawsByGroup }`. Owns an in-memory + `Map` cache keyed by `model+state+year` so re-selecting a model in Chart 3's + dropdown, or viewing another district in the same state, reuses the shard + already fetched. + +### Modified files +- **`RateDensityRidgeline.jsx`** — on model-dropdown change, calls + `useDrawDistribution` for the selected model; replaces + `fitSkewedInterval`/`densityCurve` with the hook's real draws → `kde.js`. Shows + an inline spinner in the ridge area while that model's shard is loading (Charts + 1–2 aren't blocked). +- **`RateByGroupBar.jsx`** — fetches draws for `unified_m3_mod` (the one model + this chart uses) alongside its existing data fetch. `q1`/`q3` become exact + empirical quantiles from the 500 real draws — removes this chart's use of + `fitSkewedInterval`'s fitted quantile function entirely. +- **`ChartPanel.jsx`** — passes `district.leaid`, `state`, and each group's + `stu_enroll` (already fetched) down to the two charts above. +- **`ApproxNote.jsx`** — becomes conditional: renders the "estimated shape" note + only when a chart is in fallback mode; charts backed by real draws show no note + (or a neutral "500 posterior draws" caption). +- **`package.json` / `vite.config.mjs`** — add `@duckdb/duckdb-wasm`; wasm/worker + assets are pulled in via Vite's native `?url` imports, which already respect the + `/crdc-demo/` `base` path — no bundler plugin needed. + +### Unchanged +- **`ArrestsOverTime.jsx`** — already uses real summary stats (point-range from + `/estimates`), no approximation involved; out of scope. +- **`useApi.js`** — still the source for medians, intervals, and enrollment. +- **`distributionApprox.js`** — kept as the fallback path (see §4). + +--- + +## 4. Error handling & caching + +**Fallback:** `distributionApprox.js` is retained. `useDrawDistribution` catches +fetch/wasm/query failures and returns `status: 'error'`; both chart components +branch on that to render the current analytic-approximation path with +`` visible. A Hugging Face outage, a network failure, or an +unsupported browser degrades to today's behavior rather than breaking the chart. + +**Caching:** in-memory only (a `Map` inside the hook), scoped to the browser +session. No IndexedDB/persistent cache in this iteration — a demo session +typically covers one or two districts, and shards are cheap enough to refetch on +reload. + +--- + +## 5. Testing + +This project has no automated test suite (per `AGENTS.md`); this follows the +existing manual-verification convention: + +1. `npm run dev`; walk a small state (DC or WY, ~130–200KB shard) and a large one + (CA or TX, several MB) through the full district-search flow. +2. Confirm both charts render from real draws; confirm the Network tab shows the + expected parquet fetch(es) and sizes. +3. Simulate failure (block the `huggingface.co` / CDN domain in devtools) and + confirm both charts fall back to the analytic approximation with the note + visible, rather than breaking. +4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is + instant on re-selection (cache hit). + +--- + +## 6. Open risk to de-risk first + +Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query +correctly when deployed under the `/crdc-demo/` subpath on git-pages, and does +nginx/git-pages serve `.wasm` with a usable content type? Everything else in this +design is standard Vite asset handling already exercised elsewhere in the app, but +this specific combination (wasm worker + subpath base + static host) hasn't been +verified end-to-end and should be checked with a throwaway spike rather than +assumed. + +--- + +## 7. Explicitly out of scope + +- `ArrestsOverTime.jsx` (Chart 1) — no approximation to replace. +- A new server-side `/draws`-streaming API endpoint — explicitly rejected in favor + of client-side wasm access, per the brainstorming decision that led to this spec. +- Persistent (IndexedDB) caching of fetched shards. +- Fine-grained HTTP range / row-group-level partial reads — shard sizes are small + enough that whole-file fetch is simpler and sufficiently fast. +- Extending empirical draws to national/exceedance-probability views — those + charts aren't part of the current 3-chart demo.