1002 lines
43 KiB
Markdown
1002 lines
43 KiB
Markdown
# Empirical Draw Distributions via DuckDB-Wasm Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Replace the analytic distribution approximation in Chart 2 (`RateByGroupBar`) and Chart 3 (`RateDensityRidgeline`) with the real posterior draws (500 per group), fetched client-side from the Hugging Face parquet dataset via `@duckdb/duckdb-wasm`, with graceful fallback to the existing approximation on any failure.
|
||
|
||
**Architecture:** A lazy-loaded `@duckdb/duckdb-wasm` singleton (MVP/single-threaded bundle only) fetches the one HF parquet shard matching a district's `(model_id, YEAR, LEA_STATE)`, queries it in-browser for that district's `LEAID`, and joins the resulting count draws against enrollment already in app state to get rate-per-1000 draws per race×sex group. A React hook (`useDrawDistribution`) owns fetch/cache/query/join; chart components consume it and fall back to today's `fitSkewedInterval`/`densityCurve` approximation whenever draws aren't (yet, or ever) available.
|
||
|
||
**Tech Stack:** React 19, Vite 8, `@duckdb/duckdb-wasm` ^1.32.0 (new), no new test framework — pure-logic modules get `node --test` (Node's built-in test runner, zero new deps); wasm/network/chart-rendering integration is verified manually per this project's existing convention (no automated UI test suite — see `AGENTS.md`).
|
||
|
||
## Global Constraints
|
||
|
||
- Use the **single-threaded MVP bundle only** (`duckdb-mvp.wasm` + `duckdb-browser-mvp.worker.js`) — never the `eh`/`coi` bundles, which require `Cross-Origin-Opener-Policy`/`Cross-Origin-Embedder-Policy` headers this static host doesn't send.
|
||
- **No new backend endpoint.** All draw access is client-side against the existing public HF dataset.
|
||
- **Fallback is mandatory, not optional.** `distributionApprox.js` stays in the codebase; every chart that can show real draws must also handle `status === 'error'` (or draws missing for a specific group) by rendering the existing approximation.
|
||
- **Cache in-memory only**, scoped to the browser session (a module-level `Map`) — no IndexedDB/persistent cache.
|
||
- **Lazy per-model fetch** for Chart 3's model dropdown — fetch only the selected model's shard, not all 4 quadrant models upfront.
|
||
- HF base URL (verified public, non-gated, CORS-open 2026-08-11):
|
||
`https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet`
|
||
- Parquet schema (verified against `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`): each shard (`model_id=X/YEAR=Y/LEA_STATE=Z/data_0.parquet`) contains columns `LEAID, RACE, SEX, subgroup_id, draw_id, pred` — `stu_enroll` is **not** in this table; it comes from the `/estimates` summary API this app already calls.
|
||
- Spec: `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`
|
||
|
||
---
|
||
|
||
### Task 1: `duckdbClient.js` singleton + dependency (de-risking spike)
|
||
|
||
**Files:**
|
||
- Create: `src/utils/duckdbClient.js`
|
||
- Modify: `package.json:15-21` (devDependencies block)
|
||
- Modify: `vite.config.mjs`
|
||
|
||
**Interfaces:**
|
||
- Produces: `getDb(): Promise<AsyncDuckDB>` — a lazy-initialized, memoized singleton. Every later task that needs a DuckDB connection calls `getDb()` and then `db.connect()`.
|
||
|
||
This task exists specifically to de-risk the one unverified assumption from the design spec (§6): that duckdb-wasm's MVP bundle loads and queries correctly when the app is served from the `/crdc-demo/` subpath, with Vite handling the wasm/worker asset URLs correctly. Everything here is verified live in the browser, not just written and trusted.
|
||
|
||
**Verified facts this task relies on** (checked 2026-08-11 against `@duckdb/duckdb-wasm@1.32.0`'s published `.d.ts` files):
|
||
- `new duckdb.AsyncDuckDB(logger, worker)` then `await db.instantiate(mainModuleURL, pthreadWorkerURL)` — for MVP, `pthreadWorkerURL` is `null`.
|
||
- The package's `exports` map explicitly whitelists `./dist/duckdb-mvp.wasm` and `./dist/duckdb-browser-mvp.worker.js` as importable subpaths, so Vite's `?url` import pattern resolves them.
|
||
- `duckdb.ConsoleLogger` and `duckdb.LogLevel` are exported from the package root.
|
||
|
||
- [ ] **Step 1: Add the dependency**
|
||
|
||
```bash
|
||
npm install --save-dev @duckdb/duckdb-wasm@^1.32.0
|
||
```
|
||
|
||
Note: this pins the stable `1.32.0` release. The npm `latest` dist-tag for this package currently points at a `-dev` prerelease (`1.33.1-dev57.0` as of 2026-08-11) — don't use `npm install @duckdb/duckdb-wasm` without the version pin, it will grab the prerelease.
|
||
|
||
- [ ] **Step 2: Exclude the package from Vite's dev-server dependency pre-bundling**
|
||
|
||
Edit `vite.config.mjs`:
|
||
|
||
```js
|
||
import { defineConfig } from 'vite'
|
||
import react from '@vitejs/plugin-react-swc'
|
||
|
||
// The app is served at /crdc-demo/ on pages.civilytics.org.
|
||
// Set base to the subdirectory so asset paths resolve correctly.
|
||
export default defineConfig({
|
||
plugins: [react()],
|
||
server: { port: 5173 },
|
||
build: { outDir: 'dist' },
|
||
base: '/crdc-demo/', // Required for subdirectory deployment on git-pages
|
||
optimizeDeps: {
|
||
// duckdb-wasm ships its own worker + wasm binaries resolved via `?url`
|
||
// imports; esbuild's dev-server pre-bundling can rewrite those import
|
||
// paths and break worker instantiation. Exclude it from pre-bundling.
|
||
exclude: ['@duckdb/duckdb-wasm'],
|
||
},
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 3: Create the singleton module**
|
||
|
||
Create `src/utils/duckdbClient.js`:
|
||
|
||
```js
|
||
/**
|
||
* Lazy-initialized singleton AsyncDuckDB instance running the single-threaded
|
||
* MVP wasm bundle only — never eh/coi, which need Cross-Origin-Opener-Policy /
|
||
* Cross-Origin-Embedder-Policy response headers this static host doesn't send.
|
||
* See docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md.
|
||
*/
|
||
|
||
let dbPromise = null
|
||
|
||
/** @returns {Promise<import('@duckdb/duckdb-wasm').AsyncDuckDB>} */
|
||
export function getDb() {
|
||
if (!dbPromise) dbPromise = initDb()
|
||
return dbPromise
|
||
}
|
||
|
||
async function initDb() {
|
||
const duckdb = await import('@duckdb/duckdb-wasm')
|
||
const mvpWorkerUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js?url')).default
|
||
const mvpWasmUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm?url')).default
|
||
|
||
const worker = new Worker(mvpWorkerUrl)
|
||
const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING)
|
||
const db = new duckdb.AsyncDuckDB(logger, worker)
|
||
await db.instantiate(mvpWasmUrl, null)
|
||
return db
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Verify live in the browser (this is the de-risking check)**
|
||
|
||
Run: `npm run dev`
|
||
|
||
Open `http://localhost:5173/crdc-demo/` in a browser, open devtools console, and run:
|
||
|
||
```js
|
||
const mod = await import('/src/utils/duckdbClient.js')
|
||
const db = await mod.getDb()
|
||
const conn = await db.connect()
|
||
const result = await conn.query('SELECT 42 AS answer')
|
||
console.log(result.toArray().map((r) => r.toJSON()))
|
||
await conn.close()
|
||
```
|
||
|
||
Expected: logs `[{ answer: 42 }]` with no errors. Also check the Network tab: requests for `duckdb-mvp.wasm` and `duckdb-browser-mvp.worker.js` (or similarly named hashed files) return `200`, not `404`.
|
||
|
||
If this fails with a worker-instantiation or MIME-type error, that confirms the risk flagged in the spec — stop and resolve it here (e.g., check whether `optimizeDeps.exclude` took effect after a dev-server restart) before proceeding to later tasks, since every later task depends on this module working.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add package.json package-lock.json vite.config.mjs src/utils/duckdbClient.js
|
||
git commit -m "feat: add duckdb-wasm singleton for client-side parquet queries"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: `kde.js` — empirical density utilities
|
||
|
||
**Files:**
|
||
- Create: `src/utils/kde.js`
|
||
- Create: `src/utils/kde.test.js`
|
||
- Modify: `package.json` (add a `test` script)
|
||
|
||
**Interfaces:**
|
||
- Consumes: nothing (pure functions, no dependency on Task 1).
|
||
- Produces:
|
||
- `quantile(draws: number[], p: number): number` — linear-interpolated quantile, does not mutate `draws`.
|
||
- `silvermanBandwidth(draws: number[]): number` — always > 0.
|
||
- `kdeCurve(draws: number[], options?: {min?: number, max?: number, n?: number}): Array<{x: number, y: number}>` — same `{x, y}` point-array shape as `distributionApprox.js`'s `densityCurve`, so chart code can switch between the two without changing how it draws the curve.
|
||
|
||
This is independent of Task 1 and can be done in parallel with it.
|
||
|
||
- [ ] **Step 1: Add the test script**
|
||
|
||
Edit `package.json`, add to `"scripts"`:
|
||
|
||
```json
|
||
{
|
||
"scripts": {
|
||
"dev": "vite",
|
||
"build": "vite build",
|
||
"preview": "vite preview",
|
||
"test": "node --test src/",
|
||
"lint": "eslint src/ --ext .js,.jsx,.ts,.tsx",
|
||
"format": "prettier --write \"src/**/*.{js,jsx,css}\""
|
||
}
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: Write the failing tests**
|
||
|
||
Create `src/utils/kde.test.js`:
|
||
|
||
```js
|
||
import { test } from 'node:test'
|
||
import assert from 'node:assert/strict'
|
||
import { quantile, silvermanBandwidth, kdeCurve } from './kde.js'
|
||
|
||
test('quantile: median of an odd-length array', () => {
|
||
assert.equal(quantile([3, 1, 2], 0.5), 2)
|
||
})
|
||
|
||
test('quantile: linear interpolation between two ranks', () => {
|
||
// sorted: [10, 20, 30, 40] — p=0.25 -> index 0.75 -> interpolate 10..20
|
||
assert.equal(quantile([40, 10, 30, 20], 0.25), 17.5)
|
||
})
|
||
|
||
test('quantile: does not mutate its input array', () => {
|
||
const input = [5, 3, 4, 1, 2]
|
||
quantile(input, 0.5)
|
||
assert.deepEqual(input, [5, 3, 4, 1, 2])
|
||
})
|
||
|
||
test('silvermanBandwidth: positive, finite floor for identical draws', () => {
|
||
const h = silvermanBandwidth([7, 7, 7, 7, 7])
|
||
assert.ok(h > 0 && Number.isFinite(h))
|
||
})
|
||
|
||
test('silvermanBandwidth: positive, finite floor for a single draw', () => {
|
||
const h = silvermanBandwidth([7])
|
||
assert.ok(h > 0 && Number.isFinite(h))
|
||
})
|
||
|
||
test('kdeCurve: returns n points spanning [min, max]', () => {
|
||
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
|
||
const curve = kdeCurve(draws, { min: 0, max: 10, n: 60 })
|
||
assert.equal(curve.length, 60)
|
||
assert.equal(curve[0].x, 0)
|
||
assert.ok(Math.abs(curve[curve.length - 1].x - 10) < 1e-9)
|
||
})
|
||
|
||
test('kdeCurve: density integrates to ~1 over a wide domain (trapezoidal check)', () => {
|
||
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
|
||
const curve = kdeCurve(draws, { min: -20, max: 30, n: 2000 })
|
||
let area = 0
|
||
for (let i = 1; i < curve.length; i++) {
|
||
const dx = curve[i].x - curve[i - 1].x
|
||
area += (dx * (curve[i].y + curve[i - 1].y)) / 2
|
||
}
|
||
assert.ok(Math.abs(area - 1) < 0.01, `expected area ~1, got ${area}`)
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 3: Run tests to verify they fail**
|
||
|
||
Run: `npm test`
|
||
Expected: FAIL — `Cannot find module './kde.js'` (the module doesn't exist yet).
|
||
|
||
- [ ] **Step 4: Write the implementation**
|
||
|
||
Create `src/utils/kde.js`:
|
||
|
||
```js
|
||
/**
|
||
* Empirical density utilities for posterior draw arrays — Gaussian KDE with
|
||
* Silverman's rule-of-thumb bandwidth, plus a linear-interpolated quantile.
|
||
* Used in place of distributionApprox.js's analytic fitSkewedInterval/
|
||
* densityCurve approximation whenever real posterior draws are available.
|
||
*/
|
||
|
||
const SQRT_2PI = Math.sqrt(2 * Math.PI)
|
||
const MIN_BANDWIDTH = 1e-3
|
||
|
||
/**
|
||
* Linear-interpolated quantile (R type-7). Does not mutate `draws`.
|
||
* @param {number[]} draws
|
||
* @param {number} p - probability in [0, 1]
|
||
* @returns {number}
|
||
*/
|
||
export function quantile(draws, p) {
|
||
const sorted = [...draws].sort((a, b) => a - b)
|
||
const idx = p * (sorted.length - 1)
|
||
const lo = Math.floor(idx)
|
||
const hi = Math.ceil(idx)
|
||
if (lo === hi) return sorted[lo]
|
||
const frac = idx - lo
|
||
return sorted[lo] * (1 - frac) + sorted[hi] * frac
|
||
}
|
||
|
||
function standardDeviation(draws) {
|
||
const n = draws.length
|
||
const mean = draws.reduce((sum, d) => sum + d, 0) / n
|
||
const variance = draws.reduce((sum, d) => sum + (d - mean) ** 2, 0) / (n - 1)
|
||
return Math.sqrt(variance)
|
||
}
|
||
|
||
/**
|
||
* Silverman's rule-of-thumb bandwidth (robust variant using min(sd, IQR/1.34)),
|
||
* floored so a near-degenerate draw set (e.g. almost all zeros) never
|
||
* collapses the kernel to a spike.
|
||
* @param {number[]} draws
|
||
* @returns {number}
|
||
*/
|
||
export function silvermanBandwidth(draws) {
|
||
const n = draws.length
|
||
if (n < 2) return MIN_BANDWIDTH
|
||
const sd = standardDeviation(draws)
|
||
const iqr = quantile(draws, 0.75) - quantile(draws, 0.25)
|
||
const spread = Math.min(sd, iqr / 1.34) || sd || MIN_BANDWIDTH
|
||
return Math.max(0.9 * spread * Math.pow(n, -0.2), MIN_BANDWIDTH)
|
||
}
|
||
|
||
/**
|
||
* n evenly spaced {x, y} points of a Gaussian KDE over `draws` — same shape
|
||
* contract as distributionApprox.js's densityCurve, so chart code can switch
|
||
* between the two without changing its rendering path.
|
||
* @param {number[]} draws
|
||
* @param {{min?: number, max?: number, n?: number}} [options]
|
||
* @returns {Array<{x: number, y: number}>}
|
||
*/
|
||
export function kdeCurve(draws, { min = 0, max, n = 60 } = {}) {
|
||
const hi = max ?? Math.max(...draws) * 1.1
|
||
const h = silvermanBandwidth(draws)
|
||
const step = (hi - min) / (n - 1)
|
||
const points = []
|
||
for (let i = 0; i < n; i++) {
|
||
const x = min + step * i
|
||
let sum = 0
|
||
for (const d of draws) {
|
||
const z = (x - d) / h
|
||
sum += Math.exp(-0.5 * z * z) / SQRT_2PI
|
||
}
|
||
points.push({ x, y: sum / (draws.length * h) })
|
||
}
|
||
return points
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 5: Run tests to verify they pass**
|
||
|
||
Run: `npm test`
|
||
Expected: PASS — 7 tests, 0 failures.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add package.json src/utils/kde.js src/utils/kde.test.js
|
||
git commit -m "feat: add empirical KDE/quantile utilities for real posterior draws"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: `useDrawDistribution` hook + wire into Chart 2 (`RateByGroupBar`)
|
||
|
||
**Files:**
|
||
- Create: `src/hooks/useDrawDistribution.js`
|
||
- Modify: `src/charts/RateByGroupBar.jsx` (full file — see below)
|
||
- Modify: `src/components/ChartPanel.jsx:98`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `getDb()` from Task 1 (`src/utils/duckdbClient.js`); `quantile()` from Task 2 (`src/utils/kde.js`).
|
||
- Produces: `useDrawDistribution({ leaid, state, model, year, groups }): { status: 'loading'|'ready'|'error', drawsByGroup: Record<string, number[]> | null }`, where `groups: Array<{race: string, sex: string, stuEnroll: number}>` and `drawsByGroup` keys are `` `${race}_${sex}` `` mapping to arrays of rate-per-1000 draws. Task 4 reuses this hook unchanged.
|
||
|
||
Note on a deliberate deviation from the design spec: the spec (§3) listed `ApproxNote.jsx` as a file that "becomes conditional." This plan achieves that same end behavior — the note shows only when a chart is in fallback mode — by conditionally rendering `<ApproxNote />` at each chart's call site (`{status !== 'ready' && <ApproxNote />}`, see Step 2 below and Task 4 Step 1) rather than moving conditional logic inside `ApproxNote.jsx` itself. `ApproxNote.jsx` is not modified by this plan.
|
||
|
||
- [ ] **Step 1: Write the hook**
|
||
|
||
Create `src/hooks/useDrawDistribution.js`:
|
||
|
||
```js
|
||
import { useEffect, useState } from 'react'
|
||
import { getDb } from '../utils/duckdbClient.js'
|
||
|
||
const HF_BASE = 'https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet'
|
||
|
||
// Module-level cache: one registered duckdb-wasm file buffer per
|
||
// (model, year, state) shard, shared across every component instance and
|
||
// district navigated to in this browser session. See Global Constraints —
|
||
// in-memory only, no persistence across page loads.
|
||
const shardCache = new Map()
|
||
|
||
function shardKey(model, year, state) {
|
||
return `${model}__${year}__${state}`
|
||
}
|
||
|
||
function ensureShardRegistered(db, model, year, state) {
|
||
const key = shardKey(model, year, state)
|
||
if (!shardCache.has(key)) {
|
||
shardCache.set(
|
||
key,
|
||
(async () => {
|
||
const url = `${HF_BASE}/model_id=${model}/YEAR=${year}/LEA_STATE=${state}/data_0.parquet`
|
||
const res = await fetch(url)
|
||
if (!res.ok) throw new Error(`Failed to fetch draw shard: HTTP ${res.status}`)
|
||
const buffer = new Uint8Array(await res.arrayBuffer())
|
||
const fileName = `${key}.parquet`
|
||
await db.registerFileBuffer(fileName, buffer)
|
||
return fileName
|
||
})(),
|
||
)
|
||
}
|
||
return shardCache.get(key)
|
||
}
|
||
|
||
/**
|
||
* Fetches real posterior draws for one district/model/year from the Hugging
|
||
* Face parquet dataset via duckdb-wasm, converts predicted counts to
|
||
* rate-per-1,000 using each group's stu_enroll (not present in the draws
|
||
* table itself — joined here from data this app already has), and returns
|
||
* them keyed by "RACE_SEX".
|
||
*
|
||
* @param {{leaid: string, state: string, model: string, year: string,
|
||
* groups: Array<{race: string, sex: string, stuEnroll: number}>}} params
|
||
* @returns {{status: 'loading'|'ready'|'error', drawsByGroup: Record<string, number[]> | null}}
|
||
*/
|
||
export function useDrawDistribution({ leaid, state, model, year, groups }) {
|
||
const [status, setStatus] = useState('loading')
|
||
const [drawsByGroup, setDrawsByGroup] = useState(null)
|
||
|
||
// groups is typically a fresh array literal every render; derive a stable
|
||
// primitive so the effect only re-runs when its actual content changes.
|
||
const groupsSignature = (groups || []).map((g) => `${g.race}:${g.sex}:${g.stuEnroll}`).join(',')
|
||
|
||
useEffect(() => {
|
||
if (!leaid || !state || !model || !year || !groups?.length) return
|
||
let cancelled = false
|
||
setStatus('loading')
|
||
|
||
async function run() {
|
||
let conn
|
||
try {
|
||
const db = await getDb()
|
||
const fileName = await ensureShardRegistered(db, model, year, state)
|
||
conn = await db.connect()
|
||
const stmt = await conn.prepare(`SELECT RACE, SEX, pred FROM read_parquet('${fileName}') WHERE LEAID = ?`)
|
||
const table = await stmt.query(leaid)
|
||
await stmt.close()
|
||
const rows = table.toArray().map((r) => r.toJSON())
|
||
|
||
const enrollByGroup = {}
|
||
for (const g of groups) enrollByGroup[`${g.race}_${g.sex}`] = g.stuEnroll || 0
|
||
|
||
const byGroup = {}
|
||
for (const row of rows) {
|
||
const key = `${row.RACE}_${row.SEX}`
|
||
const enroll = enrollByGroup[key]
|
||
if (!enroll) continue
|
||
const rate = (Number(row.pred) / enroll) * 1000
|
||
;(byGroup[key] ??= []).push(rate)
|
||
}
|
||
|
||
if (!cancelled) {
|
||
setDrawsByGroup(byGroup)
|
||
setStatus('ready')
|
||
}
|
||
} catch (err) {
|
||
console.error('useDrawDistribution failed:', err)
|
||
if (!cancelled) setStatus('error')
|
||
} finally {
|
||
if (conn) await conn.close()
|
||
}
|
||
}
|
||
|
||
run()
|
||
return () => {
|
||
cancelled = true
|
||
}
|
||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||
}, [leaid, state, model, year, groupsSignature])
|
||
|
||
return { status, drawsByGroup }
|
||
}
|
||
```
|
||
|
||
Note the `fileName` used in the SQL string comes from `shardKey()`, which is built from `model`/`year`/`state` — enum-constrained values from this app's own `MODEL_QUADRANTS`/`CRDC_WAVES`/state-selector lists (see `src/hooks/useApi.js`), never raw user input, so string interpolation into the `read_parquet(...)` path argument is safe here. `leaid` **is** passed as a bound `?` parameter, not interpolated, since it's a value users can influence indirectly via the district search / deep-link URL.
|
||
|
||
- [ ] **Step 2: Rewrite `RateByGroupBar.jsx` to use real draws with fallback**
|
||
|
||
Replace the full contents of `src/charts/RateByGroupBar.jsx`:
|
||
|
||
```jsx
|
||
import ChartLegend from '../components/ChartLegend.jsx'
|
||
import ApproxNote from '../components/ApproxNote.jsx'
|
||
import { raceColor, OBSERVED_MARK_COLOR, SHORT_RACE_LABEL } from '../utils/colors.js'
|
||
import { fitSkewedInterval } from '../utils/distributionApprox.js'
|
||
import { quantile } from '../utils/kde.js'
|
||
import { niceTicks } from '../utils/niceTicks.js'
|
||
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
|
||
|
||
/**
|
||
* Arrest rate by student group, most recent year, disaggregated into two
|
||
* panels (Female / Male), each a horizontal box-and-whisker across the 4
|
||
* race categories. Whisker = the model's reported 90% interval; box = the
|
||
* 25th-75th percentile of the group's 500 real posterior draws (or, if draws
|
||
* are unavailable, the fitSkewedInterval approximation); white tick =
|
||
* median; dark diamond = observed rate.
|
||
*/
|
||
|
||
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
|
||
const SEX_PANELS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
|
||
const ROW_HEIGHT = 34
|
||
const MODEL = 'unified_m3_mod' // matches ChartPanel's WAVE_MODEL, which built `data`
|
||
|
||
function buildBox(d, draws) {
|
||
const median = Math.max(d.modeledMedian || 0, 0)
|
||
const lower = Math.max(Math.min(d.rateLower ?? median, median), 0)
|
||
const upper = Math.max(d.rateUpper ?? median, median)
|
||
|
||
if (draws && draws.length > 0) {
|
||
return { lower, upper, median, q1: Math.max(quantile(draws, 0.25), 0), q3: Math.max(quantile(draws, 0.75), median) }
|
||
}
|
||
const fit = fitSkewedInterval({ median, lower, upper })
|
||
return { lower, upper, median, q1: Math.max(fit.quantile(0.25), 0), q3: Math.max(fit.quantile(0.75), median) }
|
||
}
|
||
|
||
export default function RateByGroupBar({ data, leaid, state }) {
|
||
const groups = data.map((d) => ({ race: d.race, sex: d.sex, stuEnroll: d.enrollment }))
|
||
const { status, drawsByGroup } = useDrawDistribution({ leaid, state, model: MODEL, year: '21-22', groups })
|
||
|
||
const maxRate = Math.max(
|
||
...data.map((d) => Math.max(d.observedRate, d.rateUpper ?? d.modeledMedian ?? 0)),
|
||
0.5
|
||
)
|
||
const { ticks, niceMax } = niceTicks(maxRate, 4)
|
||
|
||
return (
|
||
<div className="cv-card" style={{ padding: 'var(--space-2)' }}>
|
||
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
|
||
Arrest rate by student group — 2021–22 (per 1,000)
|
||
</h3>
|
||
{status !== 'ready' && <ApproxNote />}
|
||
|
||
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
|
||
{SEX_PANELS.map(({ sex, label }) => (
|
||
<SexPanel
|
||
key={sex}
|
||
label={label}
|
||
rows={data.filter((d) => d.sex === sex)}
|
||
drawsByGroup={drawsByGroup}
|
||
ticks={ticks}
|
||
niceMax={niceMax}
|
||
/>
|
||
))}
|
||
</div>
|
||
|
||
<ChartLegend items={[
|
||
...RACE_ORDER.filter((race) => data.some((d) => d.race === race)).map((race) => ({
|
||
shape: 'swatch', color: raceColor(race), label: SHORT_RACE_LABEL[race],
|
||
})),
|
||
{ shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
|
||
]} />
|
||
|
||
<p style={{ fontSize: '0.72rem', color: 'var(--cv-ink-3)', marginTop: 'var(--space-1)' }}>
|
||
{status === 'ready'
|
||
? "Box = 25th–75th percentile of 500 real posterior draws; whisker = the model's reported 90% interval; white tick = median."
|
||
: "Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported 90% interval; white tick = median."}
|
||
{' '}The dark diamond is the observed rate.
|
||
</p>
|
||
</div>
|
||
)
|
||
}
|
||
|
||
function SexPanel({ label, rows, drawsByGroup, ticks, niceMax }) {
|
||
const width = 300
|
||
const margin = { top: 30, right: 16, bottom: 34, left: 66 }
|
||
const innerWidth = width - margin.left - margin.right
|
||
const byRace = RACE_ORDER.map((race) => rows.find((d) => d.race === race)).filter(Boolean)
|
||
const bodyHeight = byRace.length * ROW_HEIGHT
|
||
const height = margin.top + bodyHeight + margin.bottom
|
||
const xScale = (val) => margin.left + (val / niceMax) * innerWidth
|
||
|
||
return (
|
||
<div style={{ border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)', padding: 'var(--space-1)' }}>
|
||
<svg width="100%" viewBox={`0 0 ${width} ${height}`} style={{ maxWidth: '100%' }}>
|
||
<text x={width / 2} y={16} textAnchor="middle" fontSize="0.78rem" fontWeight={700} fill="var(--cv-ink-2)">
|
||
{label}
|
||
</text>
|
||
|
||
<rect x={margin.left} y={margin.top} width={innerWidth} height={bodyHeight} fill="var(--cv-paper-2)" rx={4} />
|
||
|
||
{ticks.map((val) => {
|
||
const x = xScale(val)
|
||
return (
|
||
<g key={val}>
|
||
<line x1={x} y1={margin.top} x2={x} y2={margin.top + bodyHeight} stroke="var(--cv-rule)" strokeWidth={1} />
|
||
<text x={x} y={margin.top + bodyHeight + 14} textAnchor="middle" fontSize="0.6rem" fill="var(--cv-ink-3)">
|
||
{val}
|
||
</text>
|
||
</g>
|
||
)
|
||
})}
|
||
|
||
{byRace.map((d, i) => {
|
||
const rowY = margin.top + i * ROW_HEIGHT
|
||
const midY = rowY + ROW_HEIGHT / 2
|
||
const boxTop = midY - ROW_HEIGHT * 0.26
|
||
const boxBottom = midY + ROW_HEIGHT * 0.26
|
||
const color = raceColor(d.race)
|
||
const draws = drawsByGroup?.[`${d.race}_${d.sex}`]
|
||
const box = buildBox(d, draws)
|
||
const observedX = xScale(d.observedRate)
|
||
|
||
return (
|
||
<g key={d.race}>
|
||
<line x1={xScale(box.lower)} y1={midY} x2={xScale(box.upper)} y2={midY} stroke={color} strokeWidth={1.5} />
|
||
<line x1={xScale(box.lower)} y1={boxTop} x2={xScale(box.lower)} y2={boxBottom} stroke={color} strokeWidth={1.5} />
|
||
<line x1={xScale(box.upper)} y1={boxTop} x2={xScale(box.upper)} y2={boxBottom} stroke={color} strokeWidth={1.5} />
|
||
<rect
|
||
x={xScale(box.q1)} y={boxTop} width={Math.max(xScale(box.q3) - xScale(box.q1), 1)} height={boxBottom - boxTop}
|
||
fill={color} fillOpacity={0.55} stroke={color} strokeWidth={1} rx={1.5}
|
||
/>
|
||
<line x1={xScale(box.median)} y1={boxTop} x2={xScale(box.median)} y2={boxBottom} stroke="#fff" strokeWidth={1.5} />
|
||
<rect
|
||
x={observedX - 4} y={midY - 4} width={8} height={8}
|
||
fill={OBSERVED_MARK_COLOR} stroke="#fff" strokeWidth={1}
|
||
transform={`rotate(45 ${observedX} ${midY})`}
|
||
/>
|
||
<text x={margin.left - 8} y={midY + 4} textAnchor="end" fontSize="0.72rem" fill="var(--cv-ink)">
|
||
{SHORT_RACE_LABEL[d.race] || d.race}
|
||
</text>
|
||
</g>
|
||
)
|
||
})}
|
||
|
||
<text x={margin.left + innerWidth / 2} y={height - 6} textAnchor="middle" fontSize="0.62rem" fill="var(--cv-ink-3)">
|
||
Rate per 1,000 students
|
||
</text>
|
||
</svg>
|
||
</div>
|
||
)
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 3: Pass `leaid`/`state` from `ChartPanel.jsx`**
|
||
|
||
In `src/components/ChartPanel.jsx`, change line 98 from:
|
||
|
||
```jsx
|
||
<RateByGroupBar data={rateByGroup} />
|
||
```
|
||
|
||
to:
|
||
|
||
```jsx
|
||
<RateByGroupBar data={rateByGroup} leaid={district.leaid} state={state} />
|
||
```
|
||
|
||
- [ ] **Step 4: Verify manually in the browser**
|
||
|
||
Run: `npm run dev`, open `http://localhost:5173/crdc-demo/`.
|
||
|
||
1. Pick a **small state** (e.g. Wyoming or DC) and any district. Confirm Chart 2 renders. Open the Network tab and confirm a request to
|
||
`.../parquet/model_id=unified_m3_mod/YEAR=21-22/LEA_STATE=WY/data_0.parquet` (or `DC`) returns `200` with a small payload (order of 100–300KB per the sizes verified in the design spec).
|
||
2. Confirm the caption under Chart 2 reads "Box = 25th–75th percentile of 500 real posterior draws..." (the ready-state text) once the shard loads, and that no `<ApproxNote />` is showing.
|
||
3. Pick a **large state** (California or Texas) and confirm the same, with a larger (up to ~6.4MB) shard request.
|
||
4. In devtools, block the domain `huggingface.co` (Network tab → right-click the request → "Block request domain", or use devtools' request-blocking panel), reload, and pick a district. Confirm Chart 2 still renders (using the fallback box calculation) and now shows `<ApproxNote />` with the original "fitted approximation" caption text. Unblock the domain afterward.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/hooks/useDrawDistribution.js src/charts/RateByGroupBar.jsx src/components/ChartPanel.jsx
|
||
git commit -m "feat: drive Chart 2's box plot from real posterior draws, with fallback"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: Wire real draws into Chart 3 (`RateDensityRidgeline`)
|
||
|
||
**Files:**
|
||
- Modify: `src/charts/RateDensityRidgeline.jsx` (full file — see below)
|
||
- Modify: `src/components/ChartPanel.jsx:100`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `useDrawDistribution` from Task 3 (`src/hooks/useDrawDistribution.js`); `kdeCurve` from Task 2 (`src/utils/kde.js`).
|
||
- Produces: nothing new consumed by later tasks.
|
||
|
||
This chart is more involved than Task 3 because of the model dropdown: switching models must lazily fetch that model's shard (cached on repeat selection, per Global Constraints), and each of the 8 race×sex groups needs independent real-draws-vs-fallback handling (not an all-or-nothing chart-level switch), since `drawsByGroup` could in principle be missing an individual group even when the shard loaded fine.
|
||
|
||
- [ ] **Step 1: Rewrite `RateDensityRidgeline.jsx`**
|
||
|
||
Replace the full contents of `src/charts/RateDensityRidgeline.jsx`:
|
||
|
||
```jsx
|
||
import { useState } from 'react'
|
||
import { MODEL_QUADRANTS } from '../hooks/useApi.js'
|
||
import { raceColor, OBSERVED_MARK_COLOR, RACE_LABELS, SHORT_RACE_LABEL } from '../utils/colors.js'
|
||
import { fitSkewedInterval, densityCurve } from '../utils/distributionApprox.js'
|
||
import { kdeCurve } from '../utils/kde.js'
|
||
import { niceTicks } from '../utils/niceTicks.js'
|
||
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
|
||
import ChartLegend from '../components/ChartLegend.jsx'
|
||
import ApproxNote from '../components/ApproxNote.jsx'
|
||
|
||
/**
|
||
* Modeled posterior density per race×sex group, for one selected model
|
||
* (dropdown, default three-year + referral rate), split into Female/Male
|
||
* columns. Each ridge is drawn from that group's 500 real posterior draws
|
||
* (Gaussian KDE) when available, falling back per-group to the analytic
|
||
* fitSkewedInterval/densityCurve approximation otherwise.
|
||
*/
|
||
|
||
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
|
||
const SEX_COLUMNS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
|
||
const DEFAULT_MODEL = 'unified_m4_mod' // Three-year + referral rate
|
||
|
||
function buildGroupRow(row, draws) {
|
||
const enroll = row.stu_enroll || 0
|
||
const observedRate = enroll > 0 ? ((row.observed_arrests || 0) / enroll) * 1000 : 0
|
||
const rateMedian = (row.rate_median || 0) * 1000
|
||
const rateLower = (row.rate_lower || 0) * 1000
|
||
const rateUpper = (row.rate_upper || 0) * 1000
|
||
return {
|
||
race: row.race,
|
||
sex: row.sex,
|
||
observedRate,
|
||
draws,
|
||
fit: fitSkewedInterval({ median: rateMedian, lower: rateLower, upper: rateUpper }),
|
||
}
|
||
}
|
||
|
||
export default function RateDensityRidgeline({ quadData, leaid, state }) {
|
||
const [selectedModel, setSelectedModel] = useState(DEFAULT_MODEL)
|
||
const rows = (quadData && quadData[selectedModel]) || []
|
||
const groups = rows.map((r) => ({ race: r.race, sex: r.sex, stuEnroll: r.stu_enroll || 0 }))
|
||
const { status, drawsByGroup } = useDrawDistribution({ leaid, state, model: selectedModel, year: '21-22', groups })
|
||
|
||
const modelSelect = (
|
||
<select
|
||
value={selectedModel}
|
||
onChange={(e) => setSelectedModel(e.target.value)}
|
||
style={{ padding: '0.25rem 0.5rem', fontFamily: 'var(--font-sans)', fontSize: '0.8rem' }}
|
||
>
|
||
{MODEL_QUADRANTS.map((q) => (
|
||
<option key={q.model} value={q.model}>{q.label}</option>
|
||
))}
|
||
</select>
|
||
)
|
||
|
||
return (
|
||
<div className="cv-card" style={{ padding: 'var(--space-2)' }}>
|
||
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'flex-start', flexWrap: 'wrap', gap: 'var(--space-2)' }}>
|
||
<div>
|
||
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
|
||
Predicted arrest rates by student group
|
||
</h3>
|
||
{status !== 'ready' && <ApproxNote />}
|
||
</div>
|
||
{modelSelect}
|
||
</div>
|
||
|
||
{rows.length === 0 ? (
|
||
<p style={{ color: 'var(--cv-ink-3)', marginTop: 'var(--space-2)' }}>No model data available.</p>
|
||
) : (
|
||
<>
|
||
<RidgeColumns rows={rows} drawsByGroup={drawsByGroup} />
|
||
<ChartLegend items={[
|
||
...RACE_ORDER.map((race) => ({ shape: 'swatch', color: raceColor(race), label: RACE_LABELS[race] })),
|
||
{ shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
|
||
]} />
|
||
</>
|
||
)}
|
||
</div>
|
||
)
|
||
}
|
||
|
||
function RidgeColumns({ rows, drawsByGroup }) {
|
||
// Shared x-domain across both columns, so Female/Male are directly comparable.
|
||
const allUpper = rows.map((r) => (r.rate_upper || 0) * 1000)
|
||
const allObserved = rows
|
||
.filter((r) => (r.stu_enroll || 0) > 0)
|
||
.map((r) => ((r.observed_arrests || 0) / r.stu_enroll) * 1000)
|
||
const rawMax = Math.min(Math.max(...allUpper, ...allObserved, 1) * 1.15, 30)
|
||
const { ticks, niceMax } = niceTicks(rawMax, 5)
|
||
|
||
return (
|
||
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
|
||
{SEX_COLUMNS.map(({ sex, label }) => (
|
||
<SexRidgeColumn
|
||
key={sex}
|
||
label={label}
|
||
rows={rows.filter((r) => r.sex === sex)}
|
||
drawsByGroup={drawsByGroup}
|
||
ticks={ticks}
|
||
maxRate={niceMax}
|
||
/>
|
||
))}
|
||
</div>
|
||
)
|
||
}
|
||
|
||
function SexRidgeColumn({ label, rows, drawsByGroup, ticks, maxRate }) {
|
||
const groups = RACE_ORDER
|
||
.map((race) => rows.find((r) => r.race === race))
|
||
.filter(Boolean)
|
||
.map((row) => buildGroupRow(row, drawsByGroup?.[`${row.race}_${row.sex}`]))
|
||
|
||
const width = 300
|
||
const rowHeight = 58
|
||
const margin = { top: 26, right: 16, bottom: 26, left: 56 }
|
||
const innerWidth = width - margin.left - margin.right
|
||
const height = margin.top + Math.max(groups.length, 1) * rowHeight + margin.bottom
|
||
|
||
const xScale = (val) => margin.left + (val / maxRate) * innerWidth
|
||
|
||
const curves = groups.map((g) =>
|
||
g.draws && g.draws.length > 0
|
||
? kdeCurve(g.draws, { min: 0, max: maxRate, n: 60 })
|
||
: densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }),
|
||
)
|
||
const maxPdf = Math.max(...curves.flatMap((c) => c.map((p) => p.y)), 1e-9)
|
||
const peakHeight = rowHeight * 0.82
|
||
|
||
return (
|
||
<div style={{ border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)', padding: 'var(--space-1)' }}>
|
||
<svg width="100%" viewBox={`0 0 ${width} ${height}`} style={{ maxWidth: '100%' }}>
|
||
<text x={width / 2} y={16} textAnchor="middle" fontSize="0.78rem" fontWeight={700} fill="var(--cv-ink-2)">
|
||
{label}
|
||
</text>
|
||
|
||
{groups.length === 0 ? (
|
||
<text x={width / 2} y={height / 2} textAnchor="middle" fontSize="0.65rem" fill="var(--cv-ink-3)">
|
||
No data for this group
|
||
</text>
|
||
) : (
|
||
<>
|
||
{ticks.map((val) => {
|
||
const x = xScale(val)
|
||
return (
|
||
<g key={val}>
|
||
<line x1={x} y1={margin.top} x2={x} y2={margin.top + groups.length * rowHeight} stroke="var(--cv-rule)" strokeWidth={1} />
|
||
<text x={x} y={margin.top + groups.length * rowHeight + 14} textAnchor="middle" fontSize="0.58rem" fill="var(--cv-ink-3)">
|
||
{val}
|
||
</text>
|
||
</g>
|
||
)
|
||
})}
|
||
|
||
{groups.map((g, i) => {
|
||
const rowTop = margin.top + i * rowHeight
|
||
const baselineY = rowTop + rowHeight * 0.9
|
||
const curve = curves[i]
|
||
const color = raceColor(g.race)
|
||
|
||
const topPath = curve
|
||
.map((p, j) => `${j === 0 ? 'M' : 'L'}${xScale(p.x)},${baselineY - (p.y / maxPdf) * peakHeight}`)
|
||
.join(' ')
|
||
const areaPath = `${topPath} L${xScale(maxRate)},${baselineY} L${xScale(0)},${baselineY} Z`
|
||
|
||
const obsX = xScale(Math.min(g.observedRate, maxRate))
|
||
const obsY = baselineY - peakHeight * 0.15
|
||
|
||
return (
|
||
<g key={g.race}>
|
||
<text x={margin.left - 8} y={rowTop + rowHeight / 2 + 4} textAnchor="end" fontSize="0.68rem" fill="var(--cv-ink)">
|
||
{SHORT_RACE_LABEL[g.race] || g.race}
|
||
</text>
|
||
<path d={areaPath} fill={color} opacity={0.6} stroke="#fff" strokeWidth={0.5} />
|
||
<rect
|
||
x={obsX - 4} y={obsY - 4} width={8} height={8}
|
||
fill={OBSERVED_MARK_COLOR} stroke="#fff" strokeWidth={1}
|
||
transform={`rotate(45 ${obsX} ${obsY})`}
|
||
/>
|
||
</g>
|
||
)
|
||
})}
|
||
|
||
<text x={width / 2} y={height - 6} textAnchor="middle" fontSize="0.58rem" fill="var(--cv-ink-3)">
|
||
Arrests per 1,000 students
|
||
</text>
|
||
</>
|
||
)}
|
||
</svg>
|
||
</div>
|
||
)
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: Pass `leaid`/`state` from `ChartPanel.jsx`**
|
||
|
||
In `src/components/ChartPanel.jsx`, change line 100 from:
|
||
|
||
```jsx
|
||
<RateDensityRidgeline quadData={data.quadData} />
|
||
```
|
||
|
||
to:
|
||
|
||
```jsx
|
||
<RateDensityRidgeline quadData={data.quadData} leaid={district.leaid} state={state} />
|
||
```
|
||
|
||
- [ ] **Step 3: Verify manually in the browser**
|
||
|
||
Run: `npm run dev`, open `http://localhost:5173/crdc-demo/`, pick any district.
|
||
|
||
1. Confirm Chart 3 renders with the default model (`unified_m4_mod`) and the Network tab shows exactly one new parquet request (for that model) — not four.
|
||
2. Switch the model dropdown to a different quadrant model. Confirm a new parquet request fires for that model, and the ridges visibly update once it resolves.
|
||
3. Switch back to the first model. Confirm **no new network request** fires (cache hit — the `shardCache` Map in `useDrawDistribution.js` should serve it instantly) and the ridges render immediately.
|
||
4. Block the `huggingface.co` domain (as in Task 3 Step 4), reload, pick a district, and confirm Chart 3 still renders using the analytic-approximation ridges with `<ApproxNote />` showing, rather than breaking. Unblock the domain afterward.
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add src/charts/RateDensityRidgeline.jsx src/components/ChartPanel.jsx
|
||
git commit -m "feat: drive Chart 3's ridgelines from real posterior draws, with fallback"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Production build verification
|
||
|
||
**Files:** none (verification only).
|
||
|
||
**Interfaces:** none — this task confirms Tasks 1–4 survive Vite's production build/hashing, which behaves differently from the dev server (this is the one remaining gap between Task 1's dev-mode check and the real git-pages deployment).
|
||
|
||
- [ ] **Step 1: Build and preview**
|
||
|
||
```bash
|
||
npm run build
|
||
npm run preview
|
||
```
|
||
|
||
- [ ] **Step 2: Verify in the built app**
|
||
|
||
Open the URL `npm run preview` prints (it will include the `/crdc-demo/` base path). Repeat the checks from Task 3 Step 4 and Task 4 Step 3 (small-state district, large-state district, model-dropdown switching) against this production build rather than the dev server. Confirm:
|
||
- No console errors.
|
||
- The wasm/worker asset requests in the Network tab resolve to hashed filenames under `assets/` (or wherever Vite places them) and return `200`.
|
||
- Both charts render identically to what you saw under `npm run dev`.
|
||
|
||
- [ ] **Step 3: Commit**
|
||
|
||
No code changes in this task — nothing to commit. If any issue surfaced, fix it in the relevant task's files and re-run Steps 1–2 before moving on.
|
||
|
||
---
|
||
|
||
### Task 6: Update project documentation
|
||
|
||
**Files:**
|
||
- Modify: `AGENTS.md`
|
||
- Modify: `README.md`
|
||
- Modify: `HANDOFF.md`
|
||
|
||
`AGENTS.md` currently documents the *old* architecture as fact — it will actively mislead a future agent if left unchanged. Specifically:
|
||
- §"Common Pitfalls & Gotchas" → "4. API Endpoint Availability" says *"If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation."* This is now false for Charts 2 and 3.
|
||
- `HANDOFF.md` → "Future Enhancements" lists real posterior draws as unimplemented future work. This plan implements it.
|
||
|
||
- [ ] **Step 1: Update `AGENTS.md`**
|
||
|
||
In the "Common Pitfalls & Gotchas" → "4. API Endpoint Availability" section, replace:
|
||
|
||
```markdown
|
||
### 4. API Endpoint Availability
|
||
|
||
Not all endpoints are available to browser-based clients:
|
||
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
|
||
- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser
|
||
|
||
If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds.
|
||
```
|
||
|
||
with:
|
||
|
||
```markdown
|
||
### 4. API Endpoint Availability
|
||
|
||
Not all endpoints are available to browser-based clients:
|
||
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
|
||
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
|
||
|
||
### 5. Real posterior draws via duckdb-wasm
|
||
|
||
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
|
||
500-draw-per-group posterior from the public Hugging Face parquet dataset
|
||
(`civilytics/crdc-school-arrest-rates`), queried client-side with
|
||
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
|
||
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
|
||
involved. If that fetch fails (network, unsupported browser, HF outage),
|
||
both charts fall back to the `distributionApprox.js` analytic approximation
|
||
and show the "estimated shape" note — **do not delete `distributionApprox.js`
|
||
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
|
||
|
||
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
|
||
the full design.
|
||
```
|
||
|
||
Then renumber the old "5. CORS Configuration" section to "6." (and any other subsequent numbered items in that list) so the numbering stays sequential.
|
||
|
||
- [ ] **Step 2: Update `README.md`**
|
||
|
||
In the "What It Does" list, change item 5 from:
|
||
|
||
```markdown
|
||
5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates
|
||
```
|
||
|
||
to:
|
||
|
||
```markdown
|
||
5. **Predicted rates by student group** (Chart 5) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
|
||
```
|
||
|
||
In the "API Endpoints Used" table, change the `/api/v1/draws?...` row's "Purpose" from:
|
||
|
||
```markdown
|
||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app |
|
||
```
|
||
|
||
to:
|
||
|
||
```markdown
|
||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
|
||
```
|
||
|
||
- [ ] **Step 3: Update `HANDOFF.md`**
|
||
|
||
In "What Needs to Be Done Next" → "Future Enhancements", change:
|
||
|
||
```markdown
|
||
- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required.
|
||
```
|
||
|
||
to:
|
||
|
||
```markdown
|
||
- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`.
|
||
```
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add AGENTS.md README.md HANDOFF.md
|
||
git commit -m "docs: reflect real posterior-draw architecture in AGENTS/README/HANDOFF"
|
||
```
|