diff --git a/docs/superpowers/plans/2026-08-11-empirical-draws-wasm.md b/docs/superpowers/plans/2026-08-11-empirical-draws-wasm.md new file mode 100644 index 0000000..b1fc97b --- /dev/null +++ b/docs/superpowers/plans/2026-08-11-empirical-draws-wasm.md @@ -0,0 +1,1001 @@ +# Empirical Draw Distributions via DuckDB-Wasm Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the analytic distribution approximation in Chart 2 (`RateByGroupBar`) and Chart 3 (`RateDensityRidgeline`) with the real posterior draws (500 per group), fetched client-side from the Hugging Face parquet dataset via `@duckdb/duckdb-wasm`, with graceful fallback to the existing approximation on any failure. + +**Architecture:** A lazy-loaded `@duckdb/duckdb-wasm` singleton (MVP/single-threaded bundle only) fetches the one HF parquet shard matching a district's `(model_id, YEAR, LEA_STATE)`, queries it in-browser for that district's `LEAID`, and joins the resulting count draws against enrollment already in app state to get rate-per-1000 draws per race×sex group. A React hook (`useDrawDistribution`) owns fetch/cache/query/join; chart components consume it and fall back to today's `fitSkewedInterval`/`densityCurve` approximation whenever draws aren't (yet, or ever) available. + +**Tech Stack:** React 19, Vite 8, `@duckdb/duckdb-wasm` ^1.32.0 (new), no new test framework — pure-logic modules get `node --test` (Node's built-in test runner, zero new deps); wasm/network/chart-rendering integration is verified manually per this project's existing convention (no automated UI test suite — see `AGENTS.md`). + +## Global Constraints + +- Use the **single-threaded MVP bundle only** (`duckdb-mvp.wasm` + `duckdb-browser-mvp.worker.js`) — never the `eh`/`coi` bundles, which require `Cross-Origin-Opener-Policy`/`Cross-Origin-Embedder-Policy` headers this static host doesn't send. +- **No new backend endpoint.** All draw access is client-side against the existing public HF dataset. +- **Fallback is mandatory, not optional.** `distributionApprox.js` stays in the codebase; every chart that can show real draws must also handle `status === 'error'` (or draws missing for a specific group) by rendering the existing approximation. +- **Cache in-memory only**, scoped to the browser session (a module-level `Map`) — no IndexedDB/persistent cache. +- **Lazy per-model fetch** for Chart 3's model dropdown — fetch only the selected model's shard, not all 4 quadrant models upfront. +- HF base URL (verified public, non-gated, CORS-open 2026-08-11): + `https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet` +- Parquet schema (verified against `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`): each shard (`model_id=X/YEAR=Y/LEA_STATE=Z/data_0.parquet`) contains columns `LEAID, RACE, SEX, subgroup_id, draw_id, pred` — `stu_enroll` is **not** in this table; it comes from the `/estimates` summary API this app already calls. +- Spec: `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` + +--- + +### Task 1: `duckdbClient.js` singleton + dependency (de-risking spike) + +**Files:** +- Create: `src/utils/duckdbClient.js` +- Modify: `package.json:15-21` (devDependencies block) +- Modify: `vite.config.mjs` + +**Interfaces:** +- Produces: `getDb(): Promise` — a lazy-initialized, memoized singleton. Every later task that needs a DuckDB connection calls `getDb()` and then `db.connect()`. + +This task exists specifically to de-risk the one unverified assumption from the design spec (§6): that duckdb-wasm's MVP bundle loads and queries correctly when the app is served from the `/crdc-demo/` subpath, with Vite handling the wasm/worker asset URLs correctly. Everything here is verified live in the browser, not just written and trusted. + +**Verified facts this task relies on** (checked 2026-08-11 against `@duckdb/duckdb-wasm@1.32.0`'s published `.d.ts` files): +- `new duckdb.AsyncDuckDB(logger, worker)` then `await db.instantiate(mainModuleURL, pthreadWorkerURL)` — for MVP, `pthreadWorkerURL` is `null`. +- The package's `exports` map explicitly whitelists `./dist/duckdb-mvp.wasm` and `./dist/duckdb-browser-mvp.worker.js` as importable subpaths, so Vite's `?url` import pattern resolves them. +- `duckdb.ConsoleLogger` and `duckdb.LogLevel` are exported from the package root. + +- [ ] **Step 1: Add the dependency** + +```bash +npm install --save-dev @duckdb/duckdb-wasm@^1.32.0 +``` + +Note: this pins the stable `1.32.0` release. The npm `latest` dist-tag for this package currently points at a `-dev` prerelease (`1.33.1-dev57.0` as of 2026-08-11) — don't use `npm install @duckdb/duckdb-wasm` without the version pin, it will grab the prerelease. + +- [ ] **Step 2: Exclude the package from Vite's dev-server dependency pre-bundling** + +Edit `vite.config.mjs`: + +```js +import { defineConfig } from 'vite' +import react from '@vitejs/plugin-react-swc' + +// The app is served at /crdc-demo/ on pages.civilytics.org. +// Set base to the subdirectory so asset paths resolve correctly. +export default defineConfig({ + plugins: [react()], + server: { port: 5173 }, + build: { outDir: 'dist' }, + base: '/crdc-demo/', // Required for subdirectory deployment on git-pages + optimizeDeps: { + // duckdb-wasm ships its own worker + wasm binaries resolved via `?url` + // imports; esbuild's dev-server pre-bundling can rewrite those import + // paths and break worker instantiation. Exclude it from pre-bundling. + exclude: ['@duckdb/duckdb-wasm'], + }, +}) +``` + +- [ ] **Step 3: Create the singleton module** + +Create `src/utils/duckdbClient.js`: + +```js +/** + * Lazy-initialized singleton AsyncDuckDB instance running the single-threaded + * MVP wasm bundle only — never eh/coi, which need Cross-Origin-Opener-Policy / + * Cross-Origin-Embedder-Policy response headers this static host doesn't send. + * See docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md. + */ + +let dbPromise = null + +/** @returns {Promise} */ +export function getDb() { + if (!dbPromise) dbPromise = initDb() + return dbPromise +} + +async function initDb() { + const duckdb = await import('@duckdb/duckdb-wasm') + const mvpWorkerUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js?url')).default + const mvpWasmUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm?url')).default + + const worker = new Worker(mvpWorkerUrl) + const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING) + const db = new duckdb.AsyncDuckDB(logger, worker) + await db.instantiate(mvpWasmUrl, null) + return db +} +``` + +- [ ] **Step 4: Verify live in the browser (this is the de-risking check)** + +Run: `npm run dev` + +Open `http://localhost:5173/crdc-demo/` in a browser, open devtools console, and run: + +```js +const mod = await import('/src/utils/duckdbClient.js') +const db = await mod.getDb() +const conn = await db.connect() +const result = await conn.query('SELECT 42 AS answer') +console.log(result.toArray().map((r) => r.toJSON())) +await conn.close() +``` + +Expected: logs `[{ answer: 42 }]` with no errors. Also check the Network tab: requests for `duckdb-mvp.wasm` and `duckdb-browser-mvp.worker.js` (or similarly named hashed files) return `200`, not `404`. + +If this fails with a worker-instantiation or MIME-type error, that confirms the risk flagged in the spec — stop and resolve it here (e.g., check whether `optimizeDeps.exclude` took effect after a dev-server restart) before proceeding to later tasks, since every later task depends on this module working. + +- [ ] **Step 5: Commit** + +```bash +git add package.json package-lock.json vite.config.mjs src/utils/duckdbClient.js +git commit -m "feat: add duckdb-wasm singleton for client-side parquet queries" +``` + +--- + +### Task 2: `kde.js` — empirical density utilities + +**Files:** +- Create: `src/utils/kde.js` +- Create: `src/utils/kde.test.js` +- Modify: `package.json` (add a `test` script) + +**Interfaces:** +- Consumes: nothing (pure functions, no dependency on Task 1). +- Produces: + - `quantile(draws: number[], p: number): number` — linear-interpolated quantile, does not mutate `draws`. + - `silvermanBandwidth(draws: number[]): number` — always > 0. + - `kdeCurve(draws: number[], options?: {min?: number, max?: number, n?: number}): Array<{x: number, y: number}>` — same `{x, y}` point-array shape as `distributionApprox.js`'s `densityCurve`, so chart code can switch between the two without changing how it draws the curve. + +This is independent of Task 1 and can be done in parallel with it. + +- [ ] **Step 1: Add the test script** + +Edit `package.json`, add to `"scripts"`: + +```json +{ + "scripts": { + "dev": "vite", + "build": "vite build", + "preview": "vite preview", + "test": "node --test src/", + "lint": "eslint src/ --ext .js,.jsx,.ts,.tsx", + "format": "prettier --write \"src/**/*.{js,jsx,css}\"" + } +} +``` + +- [ ] **Step 2: Write the failing tests** + +Create `src/utils/kde.test.js`: + +```js +import { test } from 'node:test' +import assert from 'node:assert/strict' +import { quantile, silvermanBandwidth, kdeCurve } from './kde.js' + +test('quantile: median of an odd-length array', () => { + assert.equal(quantile([3, 1, 2], 0.5), 2) +}) + +test('quantile: linear interpolation between two ranks', () => { + // sorted: [10, 20, 30, 40] — p=0.25 -> index 0.75 -> interpolate 10..20 + assert.equal(quantile([40, 10, 30, 20], 0.25), 17.5) +}) + +test('quantile: does not mutate its input array', () => { + const input = [5, 3, 4, 1, 2] + quantile(input, 0.5) + assert.deepEqual(input, [5, 3, 4, 1, 2]) +}) + +test('silvermanBandwidth: positive, finite floor for identical draws', () => { + const h = silvermanBandwidth([7, 7, 7, 7, 7]) + assert.ok(h > 0 && Number.isFinite(h)) +}) + +test('silvermanBandwidth: positive, finite floor for a single draw', () => { + const h = silvermanBandwidth([7]) + assert.ok(h > 0 && Number.isFinite(h)) +}) + +test('kdeCurve: returns n points spanning [min, max]', () => { + const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8] + const curve = kdeCurve(draws, { min: 0, max: 10, n: 60 }) + assert.equal(curve.length, 60) + assert.equal(curve[0].x, 0) + assert.ok(Math.abs(curve[curve.length - 1].x - 10) < 1e-9) +}) + +test('kdeCurve: density integrates to ~1 over a wide domain (trapezoidal check)', () => { + const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8] + const curve = kdeCurve(draws, { min: -20, max: 30, n: 2000 }) + let area = 0 + for (let i = 1; i < curve.length; i++) { + const dx = curve[i].x - curve[i - 1].x + area += (dx * (curve[i].y + curve[i - 1].y)) / 2 + } + assert.ok(Math.abs(area - 1) < 0.01, `expected area ~1, got ${area}`) +}) +``` + +- [ ] **Step 3: Run tests to verify they fail** + +Run: `npm test` +Expected: FAIL — `Cannot find module './kde.js'` (the module doesn't exist yet). + +- [ ] **Step 4: Write the implementation** + +Create `src/utils/kde.js`: + +```js +/** + * Empirical density utilities for posterior draw arrays — Gaussian KDE with + * Silverman's rule-of-thumb bandwidth, plus a linear-interpolated quantile. + * Used in place of distributionApprox.js's analytic fitSkewedInterval/ + * densityCurve approximation whenever real posterior draws are available. + */ + +const SQRT_2PI = Math.sqrt(2 * Math.PI) +const MIN_BANDWIDTH = 1e-3 + +/** + * Linear-interpolated quantile (R type-7). Does not mutate `draws`. + * @param {number[]} draws + * @param {number} p - probability in [0, 1] + * @returns {number} + */ +export function quantile(draws, p) { + const sorted = [...draws].sort((a, b) => a - b) + const idx = p * (sorted.length - 1) + const lo = Math.floor(idx) + const hi = Math.ceil(idx) + if (lo === hi) return sorted[lo] + const frac = idx - lo + return sorted[lo] * (1 - frac) + sorted[hi] * frac +} + +function standardDeviation(draws) { + const n = draws.length + const mean = draws.reduce((sum, d) => sum + d, 0) / n + const variance = draws.reduce((sum, d) => sum + (d - mean) ** 2, 0) / (n - 1) + return Math.sqrt(variance) +} + +/** + * Silverman's rule-of-thumb bandwidth (robust variant using min(sd, IQR/1.34)), + * floored so a near-degenerate draw set (e.g. almost all zeros) never + * collapses the kernel to a spike. + * @param {number[]} draws + * @returns {number} + */ +export function silvermanBandwidth(draws) { + const n = draws.length + if (n < 2) return MIN_BANDWIDTH + const sd = standardDeviation(draws) + const iqr = quantile(draws, 0.75) - quantile(draws, 0.25) + const spread = Math.min(sd, iqr / 1.34) || sd || MIN_BANDWIDTH + return Math.max(0.9 * spread * Math.pow(n, -0.2), MIN_BANDWIDTH) +} + +/** + * n evenly spaced {x, y} points of a Gaussian KDE over `draws` — same shape + * contract as distributionApprox.js's densityCurve, so chart code can switch + * between the two without changing its rendering path. + * @param {number[]} draws + * @param {{min?: number, max?: number, n?: number}} [options] + * @returns {Array<{x: number, y: number}>} + */ +export function kdeCurve(draws, { min = 0, max, n = 60 } = {}) { + const hi = max ?? Math.max(...draws) * 1.1 + const h = silvermanBandwidth(draws) + const step = (hi - min) / (n - 1) + const points = [] + for (let i = 0; i < n; i++) { + const x = min + step * i + let sum = 0 + for (const d of draws) { + const z = (x - d) / h + sum += Math.exp(-0.5 * z * z) / SQRT_2PI + } + points.push({ x, y: sum / (draws.length * h) }) + } + return points +} +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `npm test` +Expected: PASS — 7 tests, 0 failures. + +- [ ] **Step 6: Commit** + +```bash +git add package.json src/utils/kde.js src/utils/kde.test.js +git commit -m "feat: add empirical KDE/quantile utilities for real posterior draws" +``` + +--- + +### Task 3: `useDrawDistribution` hook + wire into Chart 2 (`RateByGroupBar`) + +**Files:** +- Create: `src/hooks/useDrawDistribution.js` +- Modify: `src/charts/RateByGroupBar.jsx` (full file — see below) +- Modify: `src/components/ChartPanel.jsx:98` + +**Interfaces:** +- Consumes: `getDb()` from Task 1 (`src/utils/duckdbClient.js`); `quantile()` from Task 2 (`src/utils/kde.js`). +- Produces: `useDrawDistribution({ leaid, state, model, year, groups }): { status: 'loading'|'ready'|'error', drawsByGroup: Record | null }`, where `groups: Array<{race: string, sex: string, stuEnroll: number}>` and `drawsByGroup` keys are `` `${race}_${sex}` `` mapping to arrays of rate-per-1000 draws. Task 4 reuses this hook unchanged. + +Note on a deliberate deviation from the design spec: the spec (§3) listed `ApproxNote.jsx` as a file that "becomes conditional." This plan achieves that same end behavior — the note shows only when a chart is in fallback mode — by conditionally rendering `` at each chart's call site (`{status !== 'ready' && }`, see Step 2 below and Task 4 Step 1) rather than moving conditional logic inside `ApproxNote.jsx` itself. `ApproxNote.jsx` is not modified by this plan. + +- [ ] **Step 1: Write the hook** + +Create `src/hooks/useDrawDistribution.js`: + +```js +import { useEffect, useState } from 'react' +import { getDb } from '../utils/duckdbClient.js' + +const HF_BASE = 'https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet' + +// Module-level cache: one registered duckdb-wasm file buffer per +// (model, year, state) shard, shared across every component instance and +// district navigated to in this browser session. See Global Constraints — +// in-memory only, no persistence across page loads. +const shardCache = new Map() + +function shardKey(model, year, state) { + return `${model}__${year}__${state}` +} + +function ensureShardRegistered(db, model, year, state) { + const key = shardKey(model, year, state) + if (!shardCache.has(key)) { + shardCache.set( + key, + (async () => { + const url = `${HF_BASE}/model_id=${model}/YEAR=${year}/LEA_STATE=${state}/data_0.parquet` + const res = await fetch(url) + if (!res.ok) throw new Error(`Failed to fetch draw shard: HTTP ${res.status}`) + const buffer = new Uint8Array(await res.arrayBuffer()) + const fileName = `${key}.parquet` + await db.registerFileBuffer(fileName, buffer) + return fileName + })(), + ) + } + return shardCache.get(key) +} + +/** + * Fetches real posterior draws for one district/model/year from the Hugging + * Face parquet dataset via duckdb-wasm, converts predicted counts to + * rate-per-1,000 using each group's stu_enroll (not present in the draws + * table itself — joined here from data this app already has), and returns + * them keyed by "RACE_SEX". + * + * @param {{leaid: string, state: string, model: string, year: string, + * groups: Array<{race: string, sex: string, stuEnroll: number}>}} params + * @returns {{status: 'loading'|'ready'|'error', drawsByGroup: Record | null}} + */ +export function useDrawDistribution({ leaid, state, model, year, groups }) { + const [status, setStatus] = useState('loading') + const [drawsByGroup, setDrawsByGroup] = useState(null) + + // groups is typically a fresh array literal every render; derive a stable + // primitive so the effect only re-runs when its actual content changes. + const groupsSignature = (groups || []).map((g) => `${g.race}:${g.sex}:${g.stuEnroll}`).join(',') + + useEffect(() => { + if (!leaid || !state || !model || !year || !groups?.length) return + let cancelled = false + setStatus('loading') + + async function run() { + let conn + try { + const db = await getDb() + const fileName = await ensureShardRegistered(db, model, year, state) + conn = await db.connect() + const stmt = await conn.prepare(`SELECT RACE, SEX, pred FROM read_parquet('${fileName}') WHERE LEAID = ?`) + const table = await stmt.query(leaid) + await stmt.close() + const rows = table.toArray().map((r) => r.toJSON()) + + const enrollByGroup = {} + for (const g of groups) enrollByGroup[`${g.race}_${g.sex}`] = g.stuEnroll || 0 + + const byGroup = {} + for (const row of rows) { + const key = `${row.RACE}_${row.SEX}` + const enroll = enrollByGroup[key] + if (!enroll) continue + const rate = (Number(row.pred) / enroll) * 1000 + ;(byGroup[key] ??= []).push(rate) + } + + if (!cancelled) { + setDrawsByGroup(byGroup) + setStatus('ready') + } + } catch (err) { + console.error('useDrawDistribution failed:', err) + if (!cancelled) setStatus('error') + } finally { + if (conn) await conn.close() + } + } + + run() + return () => { + cancelled = true + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [leaid, state, model, year, groupsSignature]) + + return { status, drawsByGroup } +} +``` + +Note the `fileName` used in the SQL string comes from `shardKey()`, which is built from `model`/`year`/`state` — enum-constrained values from this app's own `MODEL_QUADRANTS`/`CRDC_WAVES`/state-selector lists (see `src/hooks/useApi.js`), never raw user input, so string interpolation into the `read_parquet(...)` path argument is safe here. `leaid` **is** passed as a bound `?` parameter, not interpolated, since it's a value users can influence indirectly via the district search / deep-link URL. + +- [ ] **Step 2: Rewrite `RateByGroupBar.jsx` to use real draws with fallback** + +Replace the full contents of `src/charts/RateByGroupBar.jsx`: + +```jsx +import ChartLegend from '../components/ChartLegend.jsx' +import ApproxNote from '../components/ApproxNote.jsx' +import { raceColor, OBSERVED_MARK_COLOR, SHORT_RACE_LABEL } from '../utils/colors.js' +import { fitSkewedInterval } from '../utils/distributionApprox.js' +import { quantile } from '../utils/kde.js' +import { niceTicks } from '../utils/niceTicks.js' +import { useDrawDistribution } from '../hooks/useDrawDistribution.js' + +/** + * Arrest rate by student group, most recent year, disaggregated into two + * panels (Female / Male), each a horizontal box-and-whisker across the 4 + * race categories. Whisker = the model's reported 90% interval; box = the + * 25th-75th percentile of the group's 500 real posterior draws (or, if draws + * are unavailable, the fitSkewedInterval approximation); white tick = + * median; dark diamond = observed rate. + */ + +const RACE_ORDER = ['WH', 'BL', 'HI', 'AM'] +const SEX_PANELS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }] +const ROW_HEIGHT = 34 +const MODEL = 'unified_m3_mod' // matches ChartPanel's WAVE_MODEL, which built `data` + +function buildBox(d, draws) { + const median = Math.max(d.modeledMedian || 0, 0) + const lower = Math.max(Math.min(d.rateLower ?? median, median), 0) + const upper = Math.max(d.rateUpper ?? median, median) + + if (draws && draws.length > 0) { + return { lower, upper, median, q1: Math.max(quantile(draws, 0.25), 0), q3: Math.max(quantile(draws, 0.75), median) } + } + const fit = fitSkewedInterval({ median, lower, upper }) + return { lower, upper, median, q1: Math.max(fit.quantile(0.25), 0), q3: Math.max(fit.quantile(0.75), median) } +} + +export default function RateByGroupBar({ data, leaid, state }) { + const groups = data.map((d) => ({ race: d.race, sex: d.sex, stuEnroll: d.enrollment })) + const { status, drawsByGroup } = useDrawDistribution({ leaid, state, model: MODEL, year: '21-22', groups }) + + const maxRate = Math.max( + ...data.map((d) => Math.max(d.observedRate, d.rateUpper ?? d.modeledMedian ?? 0)), + 0.5 + ) + const { ticks, niceMax } = niceTicks(maxRate, 4) + + return ( +
+

+ Arrest rate by student group — 2021–22 (per 1,000) +

+ {status !== 'ready' && } + +
+ {SEX_PANELS.map(({ sex, label }) => ( + d.sex === sex)} + drawsByGroup={drawsByGroup} + ticks={ticks} + niceMax={niceMax} + /> + ))} +
+ + data.some((d) => d.race === race)).map((race) => ({ + shape: 'swatch', color: raceColor(race), label: SHORT_RACE_LABEL[race], + })), + { shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' }, + ]} /> + +

+ {status === 'ready' + ? "Box = 25th–75th percentile of 500 real posterior draws; whisker = the model's reported 90% interval; white tick = median." + : "Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported 90% interval; white tick = median."} + {' '}The dark diamond is the observed rate. +

+
+ ) +} + +function SexPanel({ label, rows, drawsByGroup, ticks, niceMax }) { + const width = 300 + const margin = { top: 30, right: 16, bottom: 34, left: 66 } + const innerWidth = width - margin.left - margin.right + const byRace = RACE_ORDER.map((race) => rows.find((d) => d.race === race)).filter(Boolean) + const bodyHeight = byRace.length * ROW_HEIGHT + const height = margin.top + bodyHeight + margin.bottom + const xScale = (val) => margin.left + (val / niceMax) * innerWidth + + return ( +
+ + + {label} + + + + + {ticks.map((val) => { + const x = xScale(val) + return ( + + + + {val} + + + ) + })} + + {byRace.map((d, i) => { + const rowY = margin.top + i * ROW_HEIGHT + const midY = rowY + ROW_HEIGHT / 2 + const boxTop = midY - ROW_HEIGHT * 0.26 + const boxBottom = midY + ROW_HEIGHT * 0.26 + const color = raceColor(d.race) + const draws = drawsByGroup?.[`${d.race}_${d.sex}`] + const box = buildBox(d, draws) + const observedX = xScale(d.observedRate) + + return ( + + + + + + + + + {SHORT_RACE_LABEL[d.race] || d.race} + + + ) + })} + + + Rate per 1,000 students + + +
+ ) +} +``` + +- [ ] **Step 3: Pass `leaid`/`state` from `ChartPanel.jsx`** + +In `src/components/ChartPanel.jsx`, change line 98 from: + +```jsx + +``` + +to: + +```jsx + +``` + +- [ ] **Step 4: Verify manually in the browser** + +Run: `npm run dev`, open `http://localhost:5173/crdc-demo/`. + +1. Pick a **small state** (e.g. Wyoming or DC) and any district. Confirm Chart 2 renders. Open the Network tab and confirm a request to + `.../parquet/model_id=unified_m3_mod/YEAR=21-22/LEA_STATE=WY/data_0.parquet` (or `DC`) returns `200` with a small payload (order of 100–300KB per the sizes verified in the design spec). +2. Confirm the caption under Chart 2 reads "Box = 25th–75th percentile of 500 real posterior draws..." (the ready-state text) once the shard loads, and that no `` is showing. +3. Pick a **large state** (California or Texas) and confirm the same, with a larger (up to ~6.4MB) shard request. +4. In devtools, block the domain `huggingface.co` (Network tab → right-click the request → "Block request domain", or use devtools' request-blocking panel), reload, and pick a district. Confirm Chart 2 still renders (using the fallback box calculation) and now shows `` with the original "fitted approximation" caption text. Unblock the domain afterward. + +- [ ] **Step 5: Commit** + +```bash +git add src/hooks/useDrawDistribution.js src/charts/RateByGroupBar.jsx src/components/ChartPanel.jsx +git commit -m "feat: drive Chart 2's box plot from real posterior draws, with fallback" +``` + +--- + +### Task 4: Wire real draws into Chart 3 (`RateDensityRidgeline`) + +**Files:** +- Modify: `src/charts/RateDensityRidgeline.jsx` (full file — see below) +- Modify: `src/components/ChartPanel.jsx:100` + +**Interfaces:** +- Consumes: `useDrawDistribution` from Task 3 (`src/hooks/useDrawDistribution.js`); `kdeCurve` from Task 2 (`src/utils/kde.js`). +- Produces: nothing new consumed by later tasks. + +This chart is more involved than Task 3 because of the model dropdown: switching models must lazily fetch that model's shard (cached on repeat selection, per Global Constraints), and each of the 8 race×sex groups needs independent real-draws-vs-fallback handling (not an all-or-nothing chart-level switch), since `drawsByGroup` could in principle be missing an individual group even when the shard loaded fine. + +- [ ] **Step 1: Rewrite `RateDensityRidgeline.jsx`** + +Replace the full contents of `src/charts/RateDensityRidgeline.jsx`: + +```jsx +import { useState } from 'react' +import { MODEL_QUADRANTS } from '../hooks/useApi.js' +import { raceColor, OBSERVED_MARK_COLOR, RACE_LABELS, SHORT_RACE_LABEL } from '../utils/colors.js' +import { fitSkewedInterval, densityCurve } from '../utils/distributionApprox.js' +import { kdeCurve } from '../utils/kde.js' +import { niceTicks } from '../utils/niceTicks.js' +import { useDrawDistribution } from '../hooks/useDrawDistribution.js' +import ChartLegend from '../components/ChartLegend.jsx' +import ApproxNote from '../components/ApproxNote.jsx' + +/** + * Modeled posterior density per race×sex group, for one selected model + * (dropdown, default three-year + referral rate), split into Female/Male + * columns. Each ridge is drawn from that group's 500 real posterior draws + * (Gaussian KDE) when available, falling back per-group to the analytic + * fitSkewedInterval/densityCurve approximation otherwise. + */ + +const RACE_ORDER = ['WH', 'BL', 'HI', 'AM'] +const SEX_COLUMNS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }] +const DEFAULT_MODEL = 'unified_m4_mod' // Three-year + referral rate + +function buildGroupRow(row, draws) { + const enroll = row.stu_enroll || 0 + const observedRate = enroll > 0 ? ((row.observed_arrests || 0) / enroll) * 1000 : 0 + const rateMedian = (row.rate_median || 0) * 1000 + const rateLower = (row.rate_lower || 0) * 1000 + const rateUpper = (row.rate_upper || 0) * 1000 + return { + race: row.race, + sex: row.sex, + observedRate, + draws, + fit: fitSkewedInterval({ median: rateMedian, lower: rateLower, upper: rateUpper }), + } +} + +export default function RateDensityRidgeline({ quadData, leaid, state }) { + const [selectedModel, setSelectedModel] = useState(DEFAULT_MODEL) + const rows = (quadData && quadData[selectedModel]) || [] + const groups = rows.map((r) => ({ race: r.race, sex: r.sex, stuEnroll: r.stu_enroll || 0 })) + const { status, drawsByGroup } = useDrawDistribution({ leaid, state, model: selectedModel, year: '21-22', groups }) + + const modelSelect = ( + + ) + + return ( +
+
+
+

+ Predicted arrest rates by student group +

+ {status !== 'ready' && } +
+ {modelSelect} +
+ + {rows.length === 0 ? ( +

No model data available.

+ ) : ( + <> + + ({ shape: 'swatch', color: raceColor(race), label: RACE_LABELS[race] })), + { shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' }, + ]} /> + + )} +
+ ) +} + +function RidgeColumns({ rows, drawsByGroup }) { + // Shared x-domain across both columns, so Female/Male are directly comparable. + const allUpper = rows.map((r) => (r.rate_upper || 0) * 1000) + const allObserved = rows + .filter((r) => (r.stu_enroll || 0) > 0) + .map((r) => ((r.observed_arrests || 0) / r.stu_enroll) * 1000) + const rawMax = Math.min(Math.max(...allUpper, ...allObserved, 1) * 1.15, 30) + const { ticks, niceMax } = niceTicks(rawMax, 5) + + return ( +
+ {SEX_COLUMNS.map(({ sex, label }) => ( + r.sex === sex)} + drawsByGroup={drawsByGroup} + ticks={ticks} + maxRate={niceMax} + /> + ))} +
+ ) +} + +function SexRidgeColumn({ label, rows, drawsByGroup, ticks, maxRate }) { + const groups = RACE_ORDER + .map((race) => rows.find((r) => r.race === race)) + .filter(Boolean) + .map((row) => buildGroupRow(row, drawsByGroup?.[`${row.race}_${row.sex}`])) + + const width = 300 + const rowHeight = 58 + const margin = { top: 26, right: 16, bottom: 26, left: 56 } + const innerWidth = width - margin.left - margin.right + const height = margin.top + Math.max(groups.length, 1) * rowHeight + margin.bottom + + const xScale = (val) => margin.left + (val / maxRate) * innerWidth + + const curves = groups.map((g) => + g.draws && g.draws.length > 0 + ? kdeCurve(g.draws, { min: 0, max: maxRate, n: 60 }) + : densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }), + ) + const maxPdf = Math.max(...curves.flatMap((c) => c.map((p) => p.y)), 1e-9) + const peakHeight = rowHeight * 0.82 + + return ( +
+ + + {label} + + + {groups.length === 0 ? ( + + No data for this group + + ) : ( + <> + {ticks.map((val) => { + const x = xScale(val) + return ( + + + + {val} + + + ) + })} + + {groups.map((g, i) => { + const rowTop = margin.top + i * rowHeight + const baselineY = rowTop + rowHeight * 0.9 + const curve = curves[i] + const color = raceColor(g.race) + + const topPath = curve + .map((p, j) => `${j === 0 ? 'M' : 'L'}${xScale(p.x)},${baselineY - (p.y / maxPdf) * peakHeight}`) + .join(' ') + const areaPath = `${topPath} L${xScale(maxRate)},${baselineY} L${xScale(0)},${baselineY} Z` + + const obsX = xScale(Math.min(g.observedRate, maxRate)) + const obsY = baselineY - peakHeight * 0.15 + + return ( + + + {SHORT_RACE_LABEL[g.race] || g.race} + + + + + ) + })} + + + Arrests per 1,000 students + + + )} + +
+ ) +} +``` + +- [ ] **Step 2: Pass `leaid`/`state` from `ChartPanel.jsx`** + +In `src/components/ChartPanel.jsx`, change line 100 from: + +```jsx + +``` + +to: + +```jsx + +``` + +- [ ] **Step 3: Verify manually in the browser** + +Run: `npm run dev`, open `http://localhost:5173/crdc-demo/`, pick any district. + +1. Confirm Chart 3 renders with the default model (`unified_m4_mod`) and the Network tab shows exactly one new parquet request (for that model) — not four. +2. Switch the model dropdown to a different quadrant model. Confirm a new parquet request fires for that model, and the ridges visibly update once it resolves. +3. Switch back to the first model. Confirm **no new network request** fires (cache hit — the `shardCache` Map in `useDrawDistribution.js` should serve it instantly) and the ridges render immediately. +4. Block the `huggingface.co` domain (as in Task 3 Step 4), reload, pick a district, and confirm Chart 3 still renders using the analytic-approximation ridges with `` showing, rather than breaking. Unblock the domain afterward. + +- [ ] **Step 4: Commit** + +```bash +git add src/charts/RateDensityRidgeline.jsx src/components/ChartPanel.jsx +git commit -m "feat: drive Chart 3's ridgelines from real posterior draws, with fallback" +``` + +--- + +### Task 5: Production build verification + +**Files:** none (verification only). + +**Interfaces:** none — this task confirms Tasks 1–4 survive Vite's production build/hashing, which behaves differently from the dev server (this is the one remaining gap between Task 1's dev-mode check and the real git-pages deployment). + +- [ ] **Step 1: Build and preview** + +```bash +npm run build +npm run preview +``` + +- [ ] **Step 2: Verify in the built app** + +Open the URL `npm run preview` prints (it will include the `/crdc-demo/` base path). Repeat the checks from Task 3 Step 4 and Task 4 Step 3 (small-state district, large-state district, model-dropdown switching) against this production build rather than the dev server. Confirm: +- No console errors. +- The wasm/worker asset requests in the Network tab resolve to hashed filenames under `assets/` (or wherever Vite places them) and return `200`. +- Both charts render identically to what you saw under `npm run dev`. + +- [ ] **Step 3: Commit** + +No code changes in this task — nothing to commit. If any issue surfaced, fix it in the relevant task's files and re-run Steps 1–2 before moving on. + +--- + +### Task 6: Update project documentation + +**Files:** +- Modify: `AGENTS.md` +- Modify: `README.md` +- Modify: `HANDOFF.md` + +`AGENTS.md` currently documents the *old* architecture as fact — it will actively mislead a future agent if left unchanged. Specifically: +- §"Common Pitfalls & Gotchas" → "4. API Endpoint Availability" says *"If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation."* This is now false for Charts 2 and 3. +- `HANDOFF.md` → "Future Enhancements" lists real posterior draws as unimplemented future work. This plan implements it. + +- [ ] **Step 1: Update `AGENTS.md`** + +In the "Common Pitfalls & Gotchas" → "4. API Endpoint Availability" section, replace: + +```markdown +### 4. API Endpoint Availability + +Not all endpoints are available to browser-based clients: +- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds) +- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser + +If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds. +``` + +with: + +```markdown +### 4. API Endpoint Availability + +Not all endpoints are available to browser-based clients: +- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds) +- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below. + +### 5. Real posterior draws via duckdb-wasm + +Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual +500-draw-per-group posterior from the public Hugging Face parquet dataset +(`civilytics/crdc-school-arrest-rates`), queried client-side with +`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` + +`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is +involved. If that fetch fails (network, unsupported browser, HF outage), +both charts fall back to the `distributionApprox.js` analytic approximation +and show the "estimated shape" note — **do not delete `distributionApprox.js` +or `ApproxNote.jsx`**, they're the fallback path, not dead code. + +See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for +the full design. +``` + +Then renumber the old "5. CORS Configuration" section to "6." (and any other subsequent numbered items in that list) so the numbering stays sequential. + +- [ ] **Step 2: Update `README.md`** + +In the "What It Does" list, change item 5 from: + +```markdown +5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates +``` + +to: + +```markdown +5. **Predicted rates by student group** (Chart 5) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched. +``` + +In the "API Endpoints Used" table, change the `/api/v1/draws?...` row's "Purpose" from: + +```markdown +| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app | +``` + +to: + +```markdown +| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` | +``` + +- [ ] **Step 3: Update `HANDOFF.md`** + +In "What Needs to Be Done Next" → "Future Enhancements", change: + +```markdown +- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required. +``` + +to: + +```markdown +- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`. +``` + +- [ ] **Step 4: Commit** + +```bash +git add AGENTS.md README.md HANDOFF.md +git commit -m "docs: reflect real posterior-draw architecture in AGENTS/README/HANDOFF" +```