Civilytics/crdc-demo
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
crdc-demo/docs/superpowers/plans/2026-08-11-empirical-draws-wasm.md
T

43 KiB
Raw Permalink Blame History

Empirical Draw Distributions via DuckDB-Wasm Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the analytic distribution approximation in Chart 2 (RateByGroupBar) and Chart 3 (RateDensityRidgeline) with the real posterior draws (500 per group), fetched client-side from the Hugging Face parquet dataset via @duckdb/duckdb-wasm, with graceful fallback to the existing approximation on any failure.

Architecture: A lazy-loaded @duckdb/duckdb-wasm singleton (MVP/single-threaded bundle only) fetches the one HF parquet shard matching a district's (model_id, YEAR, LEA_STATE), queries it in-browser for that district's LEAID, and joins the resulting count draws against enrollment already in app state to get rate-per-1000 draws per race×sex group. A React hook (useDrawDistribution) owns fetch/cache/query/join; chart components consume it and fall back to today's fitSkewedInterval/densityCurve approximation whenever draws aren't (yet, or ever) available.

Tech Stack: React 19, Vite 8, @duckdb/duckdb-wasm ^1.32.0 (new), no new test framework — pure-logic modules get node --test (Node's built-in test runner, zero new deps); wasm/network/chart-rendering integration is verified manually per this project's existing convention (no automated UI test suite — see AGENTS.md).

Global Constraints

  • Use the single-threaded MVP bundle only (duckdb-mvp.wasm + duckdb-browser-mvp.worker.js) — never the eh/coi bundles, which require Cross-Origin-Opener-Policy/Cross-Origin-Embedder-Policy headers this static host doesn't send.
  • No new backend endpoint. All draw access is client-side against the existing public HF dataset.
  • Fallback is mandatory, not optional. distributionApprox.js stays in the codebase; every chart that can show real draws must also handle status === 'error' (or draws missing for a specific group) by rendering the existing approximation.
  • Cache in-memory only, scoped to the browser session (a module-level Map) — no IndexedDB/persistent cache.
  • Lazy per-model fetch for Chart 3's model dropdown — fetch only the selected model's shard, not all 4 quadrant models upfront.
  • HF base URL (verified public, non-gated, CORS-open 2026-08-11): https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet
  • Parquet schema (verified against crdc-arrests/R/postprocess.R + R/export_parquet.R): each shard (model_id=X/YEAR=Y/LEA_STATE=Z/data_0.parquet) contains columns LEAID, RACE, SEX, subgroup_id, draw_id, pred — stu_enroll is not in this table; it comes from the /estimates summary API this app already calls.
  • Spec: docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md

Task 1: duckdbClient.js singleton + dependency (de-risking spike)

Files:

  • Create: src/utils/duckdbClient.js
  • Modify: package.json:15-21 (devDependencies block)
  • Modify: vite.config.mjs

Interfaces:

  • Produces: getDb(): Promise<AsyncDuckDB> — a lazy-initialized, memoized singleton. Every later task that needs a DuckDB connection calls getDb() and then db.connect().

This task exists specifically to de-risk the one unverified assumption from the design spec (§6): that duckdb-wasm's MVP bundle loads and queries correctly when the app is served from the /crdc-demo/ subpath, with Vite handling the wasm/worker asset URLs correctly. Everything here is verified live in the browser, not just written and trusted.

Verified facts this task relies on (checked 2026-08-11 against @duckdb/duckdb-wasm@1.32.0's published .d.ts files):

  • new duckdb.AsyncDuckDB(logger, worker) then await db.instantiate(mainModuleURL, pthreadWorkerURL) — for MVP, pthreadWorkerURL is null.

  • The package's exports map explicitly whitelists ./dist/duckdb-mvp.wasm and ./dist/duckdb-browser-mvp.worker.js as importable subpaths, so Vite's ?url import pattern resolves them.

  • duckdb.ConsoleLogger and duckdb.LogLevel are exported from the package root.

  • Step 1: Add the dependency

npm install --save-dev @duckdb/duckdb-wasm@^1.32.0

Note: this pins the stable 1.32.0 release. The npm latest dist-tag for this package currently points at a -dev prerelease (1.33.1-dev57.0 as of 2026-08-11) — don't use npm install @duckdb/duckdb-wasm without the version pin, it will grab the prerelease.

  • Step 2: Exclude the package from Vite's dev-server dependency pre-bundling

Edit vite.config.mjs:

import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react-swc'

// The app is served at /crdc-demo/ on pages.civilytics.org.
// Set base to the subdirectory so asset paths resolve correctly.
export default defineConfig({
  plugins: [react()],
  server: { port: 5173 },
  build: { outDir: 'dist' },
  base: '/crdc-demo/',  // Required for subdirectory deployment on git-pages
  optimizeDeps: {
    // duckdb-wasm ships its own worker + wasm binaries resolved via `?url`
    // imports; esbuild's dev-server pre-bundling can rewrite those import
    // paths and break worker instantiation. Exclude it from pre-bundling.
    exclude: ['@duckdb/duckdb-wasm'],
  },
})
  • Step 3: Create the singleton module

Create src/utils/duckdbClient.js:

/**
 * Lazy-initialized singleton AsyncDuckDB instance running the single-threaded
 * MVP wasm bundle only — never eh/coi, which need Cross-Origin-Opener-Policy /
 * Cross-Origin-Embedder-Policy response headers this static host doesn't send.
 * See docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md.
 */

let dbPromise = null

/** @returns {Promise<import('@duckdb/duckdb-wasm').AsyncDuckDB>} */
export function getDb() {
  if (!dbPromise) dbPromise = initDb()
  return dbPromise
}

async function initDb() {
  const duckdb = await import('@duckdb/duckdb-wasm')
  const mvpWorkerUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js?url')).default
  const mvpWasmUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm?url')).default

  const worker = new Worker(mvpWorkerUrl)
  const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING)
  const db = new duckdb.AsyncDuckDB(logger, worker)
  await db.instantiate(mvpWasmUrl, null)
  return db
}
  • Step 4: Verify live in the browser (this is the de-risking check)

Run: npm run dev

Open http://localhost:5173/crdc-demo/ in a browser, open devtools console, and run:

const mod = await import('/src/utils/duckdbClient.js')
const db = await mod.getDb()
const conn = await db.connect()
const result = await conn.query('SELECT 42 AS answer')
console.log(result.toArray().map((r) => r.toJSON()))
await conn.close()

Expected: logs [{ answer: 42 }] with no errors. Also check the Network tab: requests for duckdb-mvp.wasm and duckdb-browser-mvp.worker.js (or similarly named hashed files) return 200, not 404.

If this fails with a worker-instantiation or MIME-type error, that confirms the risk flagged in the spec — stop and resolve it here (e.g., check whether optimizeDeps.exclude took effect after a dev-server restart) before proceeding to later tasks, since every later task depends on this module working.

  • Step 5: Commit
git add package.json package-lock.json vite.config.mjs src/utils/duckdbClient.js
git commit -m "feat: add duckdb-wasm singleton for client-side parquet queries"

Task 2: kde.js — empirical density utilities

Files:

  • Create: src/utils/kde.js
  • Create: src/utils/kde.test.js
  • Modify: package.json (add a test script)

Interfaces:

  • Consumes: nothing (pure functions, no dependency on Task 1).
  • Produces:
    • quantile(draws: number[], p: number): number — linear-interpolated quantile, does not mutate draws.
    • silvermanBandwidth(draws: number[]): number — always > 0.
    • kdeCurve(draws: number[], options?: {min?: number, max?: number, n?: number}): Array<{x: number, y: number}> — same {x, y} point-array shape as distributionApprox.js's densityCurve, so chart code can switch between the two without changing how it draws the curve.

This is independent of Task 1 and can be done in parallel with it.

  • Step 1: Add the test script

Edit package.json, add to "scripts":

{
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "preview": "vite preview",
    "test": "node --test src/",
    "lint": "eslint src/ --ext .js,.jsx,.ts,.tsx",
    "format": "prettier --write \"src/**/*.{js,jsx,css}\""
  }
}
  • Step 2: Write the failing tests

Create src/utils/kde.test.js:

import { test } from 'node:test'
import assert from 'node:assert/strict'
import { quantile, silvermanBandwidth, kdeCurve } from './kde.js'

test('quantile: median of an odd-length array', () => {
  assert.equal(quantile([3, 1, 2], 0.5), 2)
})

test('quantile: linear interpolation between two ranks', () => {
  // sorted: [10, 20, 30, 40] — p=0.25 -> index 0.75 -> interpolate 10..20
  assert.equal(quantile([40, 10, 30, 20], 0.25), 17.5)
})

test('quantile: does not mutate its input array', () => {
  const input = [5, 3, 4, 1, 2]
  quantile(input, 0.5)
  assert.deepEqual(input, [5, 3, 4, 1, 2])
})

test('silvermanBandwidth: positive, finite floor for identical draws', () => {
  const h = silvermanBandwidth([7, 7, 7, 7, 7])
  assert.ok(h > 0 && Number.isFinite(h))
})

test('silvermanBandwidth: positive, finite floor for a single draw', () => {
  const h = silvermanBandwidth([7])
  assert.ok(h > 0 && Number.isFinite(h))
})

test('kdeCurve: returns n points spanning [min, max]', () => {
  const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
  const curve = kdeCurve(draws, { min: 0, max: 10, n: 60 })
  assert.equal(curve.length, 60)
  assert.equal(curve[0].x, 0)
  assert.ok(Math.abs(curve[curve.length - 1].x - 10) < 1e-9)
})

test('kdeCurve: density integrates to ~1 over a wide domain (trapezoidal check)', () => {
  const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
  const curve = kdeCurve(draws, { min: -20, max: 30, n: 2000 })
  let area = 0
  for (let i = 1; i < curve.length; i++) {
    const dx = curve[i].x - curve[i - 1].x
    area += (dx * (curve[i].y + curve[i - 1].y)) / 2
  }
  assert.ok(Math.abs(area - 1) < 0.01, `expected area ~1, got ${area}`)
})
  • Step 3: Run tests to verify they fail

Run: npm test Expected: FAIL — Cannot find module './kde.js' (the module doesn't exist yet).

  • Step 4: Write the implementation

Create src/utils/kde.js:

/**
 * Empirical density utilities for posterior draw arrays — Gaussian KDE with
 * Silverman's rule-of-thumb bandwidth, plus a linear-interpolated quantile.
 * Used in place of distributionApprox.js's analytic fitSkewedInterval/
 * densityCurve approximation whenever real posterior draws are available.
 */

const SQRT_2PI = Math.sqrt(2 * Math.PI)
const MIN_BANDWIDTH = 1e-3

/**
 * Linear-interpolated quantile (R type-7). Does not mutate `draws`.
 * @param {number[]} draws
 * @param {number} p - probability in [0, 1]
 * @returns {number}
 */
export function quantile(draws, p) {
  const sorted = [...draws].sort((a, b) => a - b)
  const idx = p * (sorted.length - 1)
  const lo = Math.floor(idx)
  const hi = Math.ceil(idx)
  if (lo === hi) return sorted[lo]
  const frac = idx - lo
  return sorted[lo] * (1 - frac) + sorted[hi] * frac
}

function standardDeviation(draws) {
  const n = draws.length
  const mean = draws.reduce((sum, d) => sum + d, 0) / n
  const variance = draws.reduce((sum, d) => sum + (d - mean) ** 2, 0) / (n - 1)
  return Math.sqrt(variance)
}

/**
 * Silverman's rule-of-thumb bandwidth (robust variant using min(sd, IQR/1.34)),
 * floored so a near-degenerate draw set (e.g. almost all zeros) never
 * collapses the kernel to a spike.
 * @param {number[]} draws
 * @returns {number}
 */
export function silvermanBandwidth(draws) {
  const n = draws.length
  if (n < 2) return MIN_BANDWIDTH
  const sd = standardDeviation(draws)
  const iqr = quantile(draws, 0.75) - quantile(draws, 0.25)
  const spread = Math.min(sd, iqr / 1.34) || sd || MIN_BANDWIDTH
  return Math.max(0.9 * spread * Math.pow(n, -0.2), MIN_BANDWIDTH)
}

/**
 * n evenly spaced {x, y} points of a Gaussian KDE over `draws` — same shape
 * contract as distributionApprox.js's densityCurve, so chart code can switch
 * between the two without changing its rendering path.
 * @param {number[]} draws
 * @param {{min?: number, max?: number, n?: number}} [options]
 * @returns {Array<{x: number, y: number}>}
 */
export function kdeCurve(draws, { min = 0, max, n = 60 } = {}) {
  const hi = max ?? Math.max(...draws) * 1.1
  const h = silvermanBandwidth(draws)
  const step = (hi - min) / (n - 1)
  const points = []
  for (let i = 0; i < n; i++) {
    const x = min + step * i
    let sum = 0
    for (const d of draws) {
      const z = (x - d) / h
      sum += Math.exp(-0.5 * z * z) / SQRT_2PI
    }
    points.push({ x, y: sum / (draws.length * h) })
  }
  return points
}
  • Step 5: Run tests to verify they pass

Run: npm test Expected: PASS — 7 tests, 0 failures.

  • Step 6: Commit
git add package.json src/utils/kde.js src/utils/kde.test.js
git commit -m "feat: add empirical KDE/quantile utilities for real posterior draws"

Task 3: useDrawDistribution hook + wire into Chart 2 (RateByGroupBar)

Files:

  • Create: src/hooks/useDrawDistribution.js
  • Modify: src/charts/RateByGroupBar.jsx (full file — see below)
  • Modify: src/components/ChartPanel.jsx:98

Interfaces:

  • Consumes: getDb() from Task 1 (src/utils/duckdbClient.js); quantile() from Task 2 (src/utils/kde.js).
  • Produces: useDrawDistribution({ leaid, state, model, year, groups }): { status: 'loading'|'ready'|'error', drawsByGroup: Record<string, number[]> | null }, where groups: Array<{race: string, sex: string, stuEnroll: number}> and drawsByGroup keys are `${race}_${sex}` mapping to arrays of rate-per-1000 draws. Task 4 reuses this hook unchanged.

Note on a deliberate deviation from the design spec: the spec (§3) listed ApproxNote.jsx as a file that "becomes conditional." This plan achieves that same end behavior — the note shows only when a chart is in fallback mode — by conditionally rendering <ApproxNote /> at each chart's call site ({status !== 'ready' && <ApproxNote />}, see Step 2 below and Task 4 Step 1) rather than moving conditional logic inside ApproxNote.jsx itself. ApproxNote.jsx is not modified by this plan.

  • Step 1: Write the hook

Create src/hooks/useDrawDistribution.js:

import { useEffect, useState } from 'react'
import { getDb } from '../utils/duckdbClient.js'

const HF_BASE = 'https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet'

// Module-level cache: one registered duckdb-wasm file buffer per
// (model, year, state) shard, shared across every component instance and
// district navigated to in this browser session. See Global Constraints —
// in-memory only, no persistence across page loads.
const shardCache = new Map()

function shardKey(model, year, state) {
  return `${model}__${year}__${state}`
}

function ensureShardRegistered(db, model, year, state) {
  const key = shardKey(model, year, state)
  if (!shardCache.has(key)) {
    shardCache.set(
      key,
      (async () => {
        const url = `${HF_BASE}/model_id=${model}/YEAR=${year}/LEA_STATE=${state}/data_0.parquet`
        const res = await fetch(url)
        if (!res.ok) throw new Error(`Failed to fetch draw shard: HTTP ${res.status}`)
        const buffer = new Uint8Array(await res.arrayBuffer())
        const fileName = `${key}.parquet`
        await db.registerFileBuffer(fileName, buffer)
        return fileName
      })(),
    )
  }
  return shardCache.get(key)
}

/**
 * Fetches real posterior draws for one district/model/year from the Hugging
 * Face parquet dataset via duckdb-wasm, converts predicted counts to
 * rate-per-1,000 using each group's stu_enroll (not present in the draws
 * table itself — joined here from data this app already has), and returns
 * them keyed by "RACE_SEX".
 *
 * @param {{leaid: string, state: string, model: string, year: string,
 *   groups: Array<{race: string, sex: string, stuEnroll: number}>}} params
 * @returns {{status: 'loading'|'ready'|'error', drawsByGroup: Record<string, number[]> | null}}
 */
export function useDrawDistribution({ leaid, state, model, year, groups }) {
  const [status, setStatus] = useState('loading')
  const [drawsByGroup, setDrawsByGroup] = useState(null)

  // groups is typically a fresh array literal every render; derive a stable
  // primitive so the effect only re-runs when its actual content changes.
  const groupsSignature = (groups || []).map((g) => `${g.race}:${g.sex}:${g.stuEnroll}`).join(',')

  useEffect(() => {
    if (!leaid || !state || !model || !year || !groups?.length) return
    let cancelled = false
    setStatus('loading')

    async function run() {
      let conn
      try {
        const db = await getDb()
        const fileName = await ensureShardRegistered(db, model, year, state)
        conn = await db.connect()
        const stmt = await conn.prepare(`SELECT RACE, SEX, pred FROM read_parquet('${fileName}') WHERE LEAID = ?`)
        const table = await stmt.query(leaid)
        await stmt.close()
        const rows = table.toArray().map((r) => r.toJSON())

        const enrollByGroup = {}
        for (const g of groups) enrollByGroup[`${g.race}_${g.sex}`] = g.stuEnroll || 0

        const byGroup = {}
        for (const row of rows) {
          const key = `${row.RACE}_${row.SEX}`
          const enroll = enrollByGroup[key]
          if (!enroll) continue
          const rate = (Number(row.pred) / enroll) * 1000
          ;(byGroup[key] ??= []).push(rate)
        }

        if (!cancelled) {
          setDrawsByGroup(byGroup)
          setStatus('ready')
        }
      } catch (err) {
        console.error('useDrawDistribution failed:', err)
        if (!cancelled) setStatus('error')
      } finally {
        if (conn) await conn.close()
      }
    }

    run()
    return () => {
      cancelled = true
    }
    // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [leaid, state, model, year, groupsSignature])

  return { status, drawsByGroup }
}

Note the fileName used in the SQL string comes from shardKey(), which is built from model/year/state — enum-constrained values from this app's own MODEL_QUADRANTS/CRDC_WAVES/state-selector lists (see src/hooks/useApi.js), never raw user input, so string interpolation into the read_parquet(...) path argument is safe here. leaid is passed as a bound ? parameter, not interpolated, since it's a value users can influence indirectly via the district search / deep-link URL.

  • Step 2: Rewrite RateByGroupBar.jsx to use real draws with fallback

Replace the full contents of src/charts/RateByGroupBar.jsx:

import ChartLegend from '../components/ChartLegend.jsx'
import ApproxNote from '../components/ApproxNote.jsx'
import { raceColor, OBSERVED_MARK_COLOR, SHORT_RACE_LABEL } from '../utils/colors.js'
import { fitSkewedInterval } from '../utils/distributionApprox.js'
import { quantile } from '../utils/kde.js'
import { niceTicks } from '../utils/niceTicks.js'
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'

/**
 * Arrest rate by student group, most recent year, disaggregated into two
 * panels (Female / Male), each a horizontal box-and-whisker across the 4
 * race categories. Whisker = the model's reported 90% interval; box = the
 * 25th-75th percentile of the group's 500 real posterior draws (or, if draws
 * are unavailable, the fitSkewedInterval approximation); white tick =
 * median; dark diamond = observed rate.
 */

const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
const SEX_PANELS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
const ROW_HEIGHT = 34
const MODEL = 'unified_m3_mod' // matches ChartPanel's WAVE_MODEL, which built `data`

function buildBox(d, draws) {
  const median = Math.max(d.modeledMedian || 0, 0)
  const lower = Math.max(Math.min(d.rateLower ?? median, median), 0)
  const upper = Math.max(d.rateUpper ?? median, median)

  if (draws && draws.length > 0) {
    return { lower, upper, median, q1: Math.max(quantile(draws, 0.25), 0), q3: Math.max(quantile(draws, 0.75), median) }
  }
  const fit = fitSkewedInterval({ median, lower, upper })
  return { lower, upper, median, q1: Math.max(fit.quantile(0.25), 0), q3: Math.max(fit.quantile(0.75), median) }
}

export default function RateByGroupBar({ data, leaid, state }) {
  const groups = data.map((d) => ({ race: d.race, sex: d.sex, stuEnroll: d.enrollment }))
  const { status, drawsByGroup } = useDrawDistribution({ leaid, state, model: MODEL, year: '21-22', groups })

  const maxRate = Math.max(
    ...data.map((d) => Math.max(d.observedRate, d.rateUpper ?? d.modeledMedian ?? 0)),
    0.5
  )
  const { ticks, niceMax } = niceTicks(maxRate, 4)

  return (
    <div className="cv-card" style={{ padding: 'var(--space-2)' }}>
      <h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
        Arrest rate by student group — 2021–22 (per 1,000)
      </h3>
      {status !== 'ready' && <ApproxNote />}

      <div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
        {SEX_PANELS.map(({ sex, label }) => (
          <SexPanel
            key={sex}
            label={label}
            rows={data.filter((d) => d.sex === sex)}
            drawsByGroup={drawsByGroup}
            ticks={ticks}
            niceMax={niceMax}
          />
        ))}
      </div>

      <ChartLegend items={[
        ...RACE_ORDER.filter((race) => data.some((d) => d.race === race)).map((race) => ({
          shape: 'swatch', color: raceColor(race), label: SHORT_RACE_LABEL[race],
        })),
        { shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
      ]} />

      <p style={{ fontSize: '0.72rem', color: 'var(--cv-ink-3)', marginTop: 'var(--space-1)' }}>
        {status === 'ready'
          ? "Box = 25th–75th percentile of 500 real posterior draws; whisker = the model's reported 90% interval; white tick = median."
          : "Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported 90% interval; white tick = median."}
        {' '}The dark diamond is the observed rate.
      </p>
    </div>
  )
}

function SexPanel({ label, rows, drawsByGroup, ticks, niceMax }) {
  const width = 300
  const margin = { top: 30, right: 16, bottom: 34, left: 66 }
  const innerWidth = width - margin.left - margin.right
  const byRace = RACE_ORDER.map((race) => rows.find((d) => d.race === race)).filter(Boolean)
  const bodyHeight = byRace.length * ROW_HEIGHT
  const height = margin.top + bodyHeight + margin.bottom
  const xScale = (val) => margin.left + (val / niceMax) * innerWidth

  return (
    <div style={{ border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)', padding: 'var(--space-1)' }}>
      <svg width="100%" viewBox={`0 0 ${width} ${height}`} style={{ maxWidth: '100%' }}>
        <text x={width / 2} y={16} textAnchor="middle" fontSize="0.78rem" fontWeight={700} fill="var(--cv-ink-2)">
          {label}
        </text>

        <rect x={margin.left} y={margin.top} width={innerWidth} height={bodyHeight} fill="var(--cv-paper-2)" rx={4} />

        {ticks.map((val) => {
          const x = xScale(val)
          return (
            <g key={val}>
              <line x1={x} y1={margin.top} x2={x} y2={margin.top + bodyHeight} stroke="var(--cv-rule)" strokeWidth={1} />
              <text x={x} y={margin.top + bodyHeight + 14} textAnchor="middle" fontSize="0.6rem" fill="var(--cv-ink-3)">
                {val}
              </text>
            </g>
          )
        })}

        {byRace.map((d, i) => {
          const rowY = margin.top + i * ROW_HEIGHT
          const midY = rowY + ROW_HEIGHT / 2
          const boxTop = midY - ROW_HEIGHT * 0.26
          const boxBottom = midY + ROW_HEIGHT * 0.26
          const color = raceColor(d.race)
          const draws = drawsByGroup?.[`${d.race}_${d.sex}`]
          const box = buildBox(d, draws)
          const observedX = xScale(d.observedRate)

          return (
            <g key={d.race}>
              <line x1={xScale(box.lower)} y1={midY} x2={xScale(box.upper)} y2={midY} stroke={color} strokeWidth={1.5} />
              <line x1={xScale(box.lower)} y1={boxTop} x2={xScale(box.lower)} y2={boxBottom} stroke={color} strokeWidth={1.5} />
              <line x1={xScale(box.upper)} y1={boxTop} x2={xScale(box.upper)} y2={boxBottom} stroke={color} strokeWidth={1.5} />
              <rect
                x={xScale(box.q1)} y={boxTop} width={Math.max(xScale(box.q3) - xScale(box.q1), 1)} height={boxBottom - boxTop}
                fill={color} fillOpacity={0.55} stroke={color} strokeWidth={1} rx={1.5}
              />
              <line x1={xScale(box.median)} y1={boxTop} x2={xScale(box.median)} y2={boxBottom} stroke="#fff" strokeWidth={1.5} />
              <rect
                x={observedX - 4} y={midY - 4} width={8} height={8}
                fill={OBSERVED_MARK_COLOR} stroke="#fff" strokeWidth={1}
                transform={`rotate(45 ${observedX} ${midY})`}
              />
              <text x={margin.left - 8} y={midY + 4} textAnchor="end" fontSize="0.72rem" fill="var(--cv-ink)">
                {SHORT_RACE_LABEL[d.race] || d.race}
              </text>
            </g>
          )
        })}

        <text x={margin.left + innerWidth / 2} y={height - 6} textAnchor="middle" fontSize="0.62rem" fill="var(--cv-ink-3)">
          Rate per 1,000 students
        </text>
      </svg>
    </div>
  )
}
  • Step 3: Pass leaid/state from ChartPanel.jsx

In src/components/ChartPanel.jsx, change line 98 from:

        <RateByGroupBar data={rateByGroup} />

to:

        <RateByGroupBar data={rateByGroup} leaid={district.leaid} state={state} />
  • Step 4: Verify manually in the browser

Run: npm run dev, open http://localhost:5173/crdc-demo/.

  1. Pick a small state (e.g. Wyoming or DC) and any district. Confirm Chart 2 renders. Open the Network tab and confirm a request to .../parquet/model_id=unified_m3_mod/YEAR=21-22/LEA_STATE=WY/data_0.parquet (or DC) returns 200 with a small payload (order of 100–300KB per the sizes verified in the design spec).
  2. Confirm the caption under Chart 2 reads "Box = 25th–75th percentile of 500 real posterior draws..." (the ready-state text) once the shard loads, and that no <ApproxNote /> is showing.
  3. Pick a large state (California or Texas) and confirm the same, with a larger (up to ~6.4MB) shard request.
  4. In devtools, block the domain huggingface.co (Network tab → right-click the request → "Block request domain", or use devtools' request-blocking panel), reload, and pick a district. Confirm Chart 2 still renders (using the fallback box calculation) and now shows <ApproxNote /> with the original "fitted approximation" caption text. Unblock the domain afterward.
  • Step 5: Commit
git add src/hooks/useDrawDistribution.js src/charts/RateByGroupBar.jsx src/components/ChartPanel.jsx
git commit -m "feat: drive Chart 2's box plot from real posterior draws, with fallback"

Task 4: Wire real draws into Chart 3 (RateDensityRidgeline)

Files:

  • Modify: src/charts/RateDensityRidgeline.jsx (full file — see below)
  • Modify: src/components/ChartPanel.jsx:100

Interfaces:

  • Consumes: useDrawDistribution from Task 3 (src/hooks/useDrawDistribution.js); kdeCurve from Task 2 (src/utils/kde.js).
  • Produces: nothing new consumed by later tasks.

This chart is more involved than Task 3 because of the model dropdown: switching models must lazily fetch that model's shard (cached on repeat selection, per Global Constraints), and each of the 8 race×sex groups needs independent real-draws-vs-fallback handling (not an all-or-nothing chart-level switch), since drawsByGroup could in principle be missing an individual group even when the shard loaded fine.

  • Step 1: Rewrite RateDensityRidgeline.jsx

Replace the full contents of src/charts/RateDensityRidgeline.jsx:

import { useState } from 'react'
import { MODEL_QUADRANTS } from '../hooks/useApi.js'
import { raceColor, OBSERVED_MARK_COLOR, RACE_LABELS, SHORT_RACE_LABEL } from '../utils/colors.js'
import { fitSkewedInterval, densityCurve } from '../utils/distributionApprox.js'
import { kdeCurve } from '../utils/kde.js'
import { niceTicks } from '../utils/niceTicks.js'
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
import ChartLegend from '../components/ChartLegend.jsx'
import ApproxNote from '../components/ApproxNote.jsx'

/**
 * Modeled posterior density per race×sex group, for one selected model
 * (dropdown, default three-year + referral rate), split into Female/Male
 * columns. Each ridge is drawn from that group's 500 real posterior draws
 * (Gaussian KDE) when available, falling back per-group to the analytic
 * fitSkewedInterval/densityCurve approximation otherwise.
 */

const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
const SEX_COLUMNS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
const DEFAULT_MODEL = 'unified_m4_mod' // Three-year + referral rate

function buildGroupRow(row, draws) {
  const enroll = row.stu_enroll || 0
  const observedRate = enroll > 0 ? ((row.observed_arrests || 0) / enroll) * 1000 : 0
  const rateMedian = (row.rate_median || 0) * 1000
  const rateLower = (row.rate_lower || 0) * 1000
  const rateUpper = (row.rate_upper || 0) * 1000
  return {
    race: row.race,
    sex: row.sex,
    observedRate,
    draws,
    fit: fitSkewedInterval({ median: rateMedian, lower: rateLower, upper: rateUpper }),
  }
}

export default function RateDensityRidgeline({ quadData, leaid, state }) {
  const [selectedModel, setSelectedModel] = useState(DEFAULT_MODEL)
  const rows = (quadData && quadData[selectedModel]) || []
  const groups = rows.map((r) => ({ race: r.race, sex: r.sex, stuEnroll: r.stu_enroll || 0 }))
  const { status, drawsByGroup } = useDrawDistribution({ leaid, state, model: selectedModel, year: '21-22', groups })

  const modelSelect = (
    <select
      value={selectedModel}
      onChange={(e) => setSelectedModel(e.target.value)}
      style={{ padding: '0.25rem 0.5rem', fontFamily: 'var(--font-sans)', fontSize: '0.8rem' }}
    >
      {MODEL_QUADRANTS.map((q) => (
        <option key={q.model} value={q.model}>{q.label}</option>
      ))}
    </select>
  )

  return (
    <div className="cv-card" style={{ padding: 'var(--space-2)' }}>
      <div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'flex-start', flexWrap: 'wrap', gap: 'var(--space-2)' }}>
        <div>
          <h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
            Predicted arrest rates by student group
          </h3>
          {status !== 'ready' && <ApproxNote />}
        </div>
        {modelSelect}
      </div>

      {rows.length === 0 ? (
        <p style={{ color: 'var(--cv-ink-3)', marginTop: 'var(--space-2)' }}>No model data available.</p>
      ) : (
        <>
          <RidgeColumns rows={rows} drawsByGroup={drawsByGroup} />
          <ChartLegend items={[
            ...RACE_ORDER.map((race) => ({ shape: 'swatch', color: raceColor(race), label: RACE_LABELS[race] })),
            { shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
          ]} />
        </>
      )}
    </div>
  )
}

function RidgeColumns({ rows, drawsByGroup }) {
  // Shared x-domain across both columns, so Female/Male are directly comparable.
  const allUpper = rows.map((r) => (r.rate_upper || 0) * 1000)
  const allObserved = rows
    .filter((r) => (r.stu_enroll || 0) > 0)
    .map((r) => ((r.observed_arrests || 0) / r.stu_enroll) * 1000)
  const rawMax = Math.min(Math.max(...allUpper, ...allObserved, 1) * 1.15, 30)
  const { ticks, niceMax } = niceTicks(rawMax, 5)

  return (
    <div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
      {SEX_COLUMNS.map(({ sex, label }) => (
        <SexRidgeColumn
          key={sex}
          label={label}
          rows={rows.filter((r) => r.sex === sex)}
          drawsByGroup={drawsByGroup}
          ticks={ticks}
          maxRate={niceMax}
        />
      ))}
    </div>
  )
}

function SexRidgeColumn({ label, rows, drawsByGroup, ticks, maxRate }) {
  const groups = RACE_ORDER
    .map((race) => rows.find((r) => r.race === race))
    .filter(Boolean)
    .map((row) => buildGroupRow(row, drawsByGroup?.[`${row.race}_${row.sex}`]))

  const width = 300
  const rowHeight = 58
  const margin = { top: 26, right: 16, bottom: 26, left: 56 }
  const innerWidth = width - margin.left - margin.right
  const height = margin.top + Math.max(groups.length, 1) * rowHeight + margin.bottom

  const xScale = (val) => margin.left + (val / maxRate) * innerWidth

  const curves = groups.map((g) =>
    g.draws && g.draws.length > 0
      ? kdeCurve(g.draws, { min: 0, max: maxRate, n: 60 })
      : densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }),
  )
  const maxPdf = Math.max(...curves.flatMap((c) => c.map((p) => p.y)), 1e-9)
  const peakHeight = rowHeight * 0.82

  return (
    <div style={{ border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)', padding: 'var(--space-1)' }}>
      <svg width="100%" viewBox={`0 0 ${width} ${height}`} style={{ maxWidth: '100%' }}>
        <text x={width / 2} y={16} textAnchor="middle" fontSize="0.78rem" fontWeight={700} fill="var(--cv-ink-2)">
          {label}
        </text>

        {groups.length === 0 ? (
          <text x={width / 2} y={height / 2} textAnchor="middle" fontSize="0.65rem" fill="var(--cv-ink-3)">
            No data for this group
          </text>
        ) : (
          <>
            {ticks.map((val) => {
              const x = xScale(val)
              return (
                <g key={val}>
                  <line x1={x} y1={margin.top} x2={x} y2={margin.top + groups.length * rowHeight} stroke="var(--cv-rule)" strokeWidth={1} />
                  <text x={x} y={margin.top + groups.length * rowHeight + 14} textAnchor="middle" fontSize="0.58rem" fill="var(--cv-ink-3)">
                    {val}
                  </text>
                </g>
              )
            })}

            {groups.map((g, i) => {
              const rowTop = margin.top + i * rowHeight
              const baselineY = rowTop + rowHeight * 0.9
              const curve = curves[i]
              const color = raceColor(g.race)

              const topPath = curve
                .map((p, j) => `${j === 0 ? 'M' : 'L'}${xScale(p.x)},${baselineY - (p.y / maxPdf) * peakHeight}`)
                .join(' ')
              const areaPath = `${topPath} L${xScale(maxRate)},${baselineY} L${xScale(0)},${baselineY} Z`

              const obsX = xScale(Math.min(g.observedRate, maxRate))
              const obsY = baselineY - peakHeight * 0.15

              return (
                <g key={g.race}>
                  <text x={margin.left - 8} y={rowTop + rowHeight / 2 + 4} textAnchor="end" fontSize="0.68rem" fill="var(--cv-ink)">
                    {SHORT_RACE_LABEL[g.race] || g.race}
                  </text>
                  <path d={areaPath} fill={color} opacity={0.6} stroke="#fff" strokeWidth={0.5} />
                  <rect
                    x={obsX - 4} y={obsY - 4} width={8} height={8}
                    fill={OBSERVED_MARK_COLOR} stroke="#fff" strokeWidth={1}
                    transform={`rotate(45 ${obsX} ${obsY})`}
                  />
                </g>
              )
            })}

            <text x={width / 2} y={height - 6} textAnchor="middle" fontSize="0.58rem" fill="var(--cv-ink-3)">
              Arrests per 1,000 students
            </text>
          </>
        )}
      </svg>
    </div>
  )
}
  • Step 2: Pass leaid/state from ChartPanel.jsx

In src/components/ChartPanel.jsx, change line 100 from:

        <RateDensityRidgeline quadData={data.quadData} />

to:

        <RateDensityRidgeline quadData={data.quadData} leaid={district.leaid} state={state} />
  • Step 3: Verify manually in the browser

Run: npm run dev, open http://localhost:5173/crdc-demo/, pick any district.

  1. Confirm Chart 3 renders with the default model (unified_m4_mod) and the Network tab shows exactly one new parquet request (for that model) — not four.
  2. Switch the model dropdown to a different quadrant model. Confirm a new parquet request fires for that model, and the ridges visibly update once it resolves.
  3. Switch back to the first model. Confirm no new network request fires (cache hit — the shardCache Map in useDrawDistribution.js should serve it instantly) and the ridges render immediately.
  4. Block the huggingface.co domain (as in Task 3 Step 4), reload, pick a district, and confirm Chart 3 still renders using the analytic-approximation ridges with <ApproxNote /> showing, rather than breaking. Unblock the domain afterward.
  • Step 4: Commit
git add src/charts/RateDensityRidgeline.jsx src/components/ChartPanel.jsx
git commit -m "feat: drive Chart 3's ridgelines from real posterior draws, with fallback"

Task 5: Production build verification

Files: none (verification only).

Interfaces: none — this task confirms Tasks 1–4 survive Vite's production build/hashing, which behaves differently from the dev server (this is the one remaining gap between Task 1's dev-mode check and the real git-pages deployment).

  • Step 1: Build and preview
npm run build
npm run preview
  • Step 2: Verify in the built app

Open the URL npm run preview prints (it will include the /crdc-demo/ base path). Repeat the checks from Task 3 Step 4 and Task 4 Step 3 (small-state district, large-state district, model-dropdown switching) against this production build rather than the dev server. Confirm:

  • No console errors.

  • The wasm/worker asset requests in the Network tab resolve to hashed filenames under assets/ (or wherever Vite places them) and return 200.

  • Both charts render identically to what you saw under npm run dev.

  • Step 3: Commit

No code changes in this task — nothing to commit. If any issue surfaced, fix it in the relevant task's files and re-run Steps 1–2 before moving on.

Task 6: Update project documentation

Files:

  • Modify: AGENTS.md
  • Modify: README.md
  • Modify: HANDOFF.md

AGENTS.md currently documents the old architecture as fact — it will actively mislead a future agent if left unchanged. Specifically:

  • §"Common Pitfalls & Gotchas" → "4. API Endpoint Availability" says "If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation." This is now false for Charts 2 and 3.

  • HANDOFF.md → "Future Enhancements" lists real posterior draws as unimplemented future work. This plan implements it.

  • Step 1: Update AGENTS.md

In the "Common Pitfalls & Gotchas" → "4. API Endpoint Availability" section, replace:

### 4. API Endpoint Availability

Not all endpoints are available to browser-based clients:
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser

If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds.

with:

### 4. API Endpoint Availability

Not all endpoints are available to browser-based clients:
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.

### 5. Real posterior draws via duckdb-wasm

Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
500-draw-per-group posterior from the public Hugging Face parquet dataset
(`civilytics/crdc-school-arrest-rates`), queried client-side with
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
involved. If that fetch fails (network, unsupported browser, HF outage),
both charts fall back to the `distributionApprox.js` analytic approximation
and show the "estimated shape" note — **do not delete `distributionApprox.js`
or `ApproxNote.jsx`**, they're the fallback path, not dead code.

See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
the full design.

Then renumber the old "5. CORS Configuration" section to "6." (and any other subsequent numbered items in that list) so the numbering stays sequential.

  • Step 2: Update README.md

In the "What It Does" list, change item 5 from:

5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates

to:

5. **Predicted rates by student group** (Chart 5) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.

In the "API Endpoints Used" table, change the /api/v1/draws?... row's "Purpose" from:

| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app |

to:

| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
  • Step 3: Update HANDOFF.md

In "What Needs to Be Done Next" → "Future Enhancements", change:

- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required.

to:

- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`.
  • Step 4: Commit
git add AGENTS.md README.md HANDOFF.md
git commit -m "docs: reflect real posterior-draw architecture in AGENTS/README/HANDOFF"
Reference in New Issue View Git Blame Copy Permalink