# CRDC demo — rebuild the visuals around the white-paper figures ## Context The demo app at `pages.civilytics.org/crdc-demo/` works, but the charts don't show off the modelling. The navigation (state → district) is good and stays. The three existing charts get cut to one, replaced by a summary table and two charts ported from the white paper: `wp_fig_group_density` (Fig 6) and `wp_fig_group_difference` (Fig 7) in `crdc-arrests/R/paper_figures.R:472-568`. Two real defects surfaced while scoping: 1. **The "suggested districts" list is ranked over a truncated set.** `DistrictSearch.jsx:22` calls `/estimates?state=XX&year=21-22&limit=500`, but that endpoint returns rows `ORDER BY LEAID` (`crdc-arrests/api/R/handlers_estimates.R:46`) at 8 rows per district. So the app ranks the ~62 lowest-LEAID districts in the state. California has 11,488 rows. 2. **The analytic fallback assumes 90% bounds but the API returns 95%.** `distributionApprox.js` defaults `intervalMass = 0.90`; the API default is `interval=95` (`crdc-arrests/api/R/handlers_estimates.R`). The new charts compute intervals from real draws, so this only affects the fallback path — fix the default while we're in there. ### What the data supports (verified, not assumed) - The HF parquet carries `LEAID, RACE, SEX, pred, draw_id, subgroup_id, batch_num`. `draw_id` is dense 1–500 for every group. The current query at `useDrawDistribution.js:93` just doesn't select it — **that one column is what unlocks both pooling and differences.** - **Sex pooling is the project's own house method.** `build_state_summary()` in `crdc-arrests/R/summarize_draws.R:152-182` pools across LEAs by summing `pred` and `stu_enroll` within each draw, then summarizing across draws. Pooling M+F within a district is the identical operation on a different axis. Honest effort estimate: **~3–4 hours**, most of it UI and labelling, not statistics. - **Caveat to word carefully:** `draw_id` is renumbered 1–500 per write batch (`crdc-arrests/R/postprocess.R:106-133`), and a district's groups land in different batches, so cross-group draw pairing is effectively independent. Measured correlation between Black-male and Hispanic-male `pred` in Clark County was 0.019 even *within* a batch — observation noise from `posterior_predict` dominates. Published Fig 7 has the same property, so the app matches the paper. Captions should say "posterior predictive draws", never "paired parameter draws". - Enrollment covers only AM/BL/HI/WH (verified against the API). Clark County sums to 148,928 against a district enrollment near 304,000. The table must label this. ## Decisions taken | Question | Decision | |---|---| | Model specs | One selected spec by default; opt-in "compare all four" expands to 4 ridge rows | | Existing charts | Keep `ArrestsOverTime`; delete `RateByGroupBar` and `RateDensityRidgeline` | | Pooling trigger | Whole-district: pool when total observed arrests across the 8 cells < 20 | | Rendering | React owns the DOM; add d3 submodules for scales/paths/interpolation | --- ## Work ### 1. Draws pipeline — expose `draw_id`, return counts not rates **`src/hooks/useDrawDistribution.js`** — the one structural change everything else rests on. - Query becomes `SELECT RACE, SEX, draw_id, pred FROM read_parquet(...) WHERE LEAID = ?`. - Return **raw counts indexed by draw**, not rates: `countsByGroup[key][draw_id - 1] = pred`. Indexing by `draw_id` rather than push-order means row ordering from DuckDB is irrelevant. - Accept `models: string[]` instead of a single `model`, returning `{status, byModel, nDraws}`. A fixed-length array avoids conditional hooks when "compare all four" is on. The existing module-level `shardCache` already keys on `(model, year, state)`, so four models is four cache entries with no other change. - Reject a group whose count array has holes (fewer entries than `nDraws`) — a partial group must fall back, not silently render a short draw set. - Keep the reset-before-guard ordering at `useDrawDistribution.js:79-85`; it exists to stop one model's draws being shown under another model's label. **`src/utils/pooling.js`** (new, + test) — pure functions, no React: - `poolBySex(countsByGroup, enrollByGroup)` → sums counts within each draw index across `SEX ∈ {F,M}` and sums enrollment, keyed by race alone. - `toRates(counts, enroll)` → per-1,000 array. - `shouldPoolBySex(rows)` → total `observed_arrests` across rows < `POOL_BY_SEX_ARREST_THRESHOLD` (20, a named constant with the rationale in a comment). **`src/utils/drawGroups.js`** — extend `groupKey` to handle a pooled key (race only) and update `hasDrawsForAll` for the new count-array shape. ### 2. Frequentist interval **`src/utils/agrestiCoull.js`** (new, + test) — direct port of `crdc-arrests/R/paper_figures.R:219-237`, including the zero-numerator branch (`ci_upper = -log(1 - level)`, the rule of three; `ci_lower = 0`). Note the R function returns `c(upper, lower, sd, se, phat)` — **upper first**. Return a named object here instead. Needs a `qnorm`/probit; `distributionApprox.js` already has one — reuse it rather than adding a second. Tests should pin at least one case against R output (e.g. `agresti_coull(15, 499, 0.95)`). ### 3. Density profile — handle discrete posteriors honestly **`src/utils/densityProfile.js`** (new, + test). In sparse districts the posterior predictive is a discrete count distribution. Carson City NV (`3200390`) has a 53-student AI/AN female cell where one arrest is 18.9 per 1,000 — the draws take four distinct values and a Gaussian KDE renders them as a lumpy smear that reads as a rendering bug. - `densityProfile(counts, enroll, domain)` returns `{kind: 'kde'|'mass', points}`. - `kind: 'mass'` when the draws take ≤ 12 distinct values: probability mass at each achievable rate, drawn as a filled staircase so it visually rhymes with the smooth areas beside it. - Otherwise delegate to the existing `kdeCurve` in `src/utils/kde.js` — its bandwidth clamp (`BANDWIDTH_FLOOR_DIVISOR` / `BANDWIDTH_CEILING_DIVISOR`) was tuned for exactly these zero-inflated posteriors and should not be touched. ### 4. Summary table (top of results) **`src/components/DistrictSummaryTable.jsx`** (new). One row per student group plus a total: | Student group | Students | Observed arrests | Rate per 1,000 | - Sorted by observed arrests descending. Zero-arrest rows de-emphasized, not hidden. - Each row carries the checkbox that drives chart A — the table *is* the legend and the control. Default checked = `observed_arrests > 0`; if no group has any, check the two largest by enrollment and say so. - Footnote: students counted are those in the four modeled race groups (AI/AN, Black, Hispanic, White), not total district enrollment. - When pooling is active, rows collapse to four races and a banner states the rule in one sentence, with a switch to force it off. ### 5. Chart A — "Arrest rate probability density" **`src/charts/RateDensityPanel.jsx`** (new). Replaces `RateDensityRidgeline.jsx`. - Two stacked sub-panels, Female over Male, sharing one x-axis (per 1,000). Collapses to a single panel when pooled. This preserves the palette contract documented at `src/utils/colors.js:9-13`: race is hue, sex is position — never a second hue. - Within a sub-panel, selected groups overlap as filled areas (fill ~0.4 opacity, 2px stroke in the race color), direct-labelled at each peak so there's no legend hunting. - Below each sub-panel's baseline, a thin rail stacks one Agresti–Coull point-range per selected group in the matching color — the R figure's `position_nudge` idea, but un-overplotted. - Segmented control for the four unified quadrant specs. A "Compare all four specifications" switch expands to four ridge rows (matching Fig 1's structure) and triggers four shard fetches — cheap for NV (~100KB each), ~6.3MB each for CA, so it stays opt-in with a spinner. - x-domain: max of the density supports and the frequentist upper bounds, with the existing cap logic from `RateDensityRidgeline.jsx:101` and `niceTicks`. - Caption states 500 posterior predictive draws and a 95% Agresti–Coull observed interval. ### 6. Chart B — "Model Estimated Differences" **`src/charts/GroupDifference.jsx`** (new). - Two group pickers; defaults are the two groups with the most observed arrests (pooled groups when pooling is on). Δ = rate(A) − rate(B) per 1,000, computed per draw index. - Single density, filled with an SVG `linearGradient` mapped across x. Use a **diverging ramp centered at zero** — navy for Δ<0, paper at 0, ember for Δ>0 — rather than the paper's YlOrRd: the quantity is signed, and diverging-at-zero is the honest encoding. On-brand via `tokens.css`. - Dashed vertical rule at 0 in `--cv-danger`, matching the paper's red line. - Large readout `Pr(Δ > 0)` with a plain-language sentence beneath ("In 94.4% of posterior draws, the Black male arrest rate exceeds the White male rate"), plus median Δ and an 80%/95% interval as a point-range. - Degrade explicitly when fewer than two groups have usable draws — say why, don't render empty. ### 7. Fix the district suggestions **`scripts/build-top-districts.mjs`** (new) — pages `/estimates?state=XX&year=21-22&limit=1000` using `meta.total` (confirmed present in the envelope) across all 51 states, aggregates observed arrests per LEAID, and writes `public/data/top_districts.json` with the top 15 per state (leaid, name, arrests, enrollment, rate). Roughly 140 requests as a one-off; the output is ~50KB and gets committed, following the `public/data/national_rates.json` precedent. **`src/components/DistrictSearch.jsx`** — read the fixture instead of calling `fetchStateDistricts` at runtime. The search screen loses a multi-second fetch and the ranking becomes correct. Keep live name search on `/districts` unchanged. Drop the hardcoded "Try Derby (KS), Paterson (NJ)…" hint at `DistrictSearch.jsx:164` — the real list supersedes it. ### 8. Wiring, deletions, docs - **`src/components/ChartPanel.jsx`** — owns pooling state, selected groups, selected spec, and the difference pair; passes them down. Keep `ArrestsOverTime`. Drop `QUADRANT_MODELS` prefetch of all four models' *summaries* if only the selected one is needed. - **Delete**: `src/charts/RateByGroupBar.jsx`, `src/charts/RateDensityRidgeline.jsx`. - **Keep**: `distributionApprox.js` and `ApproxNote.jsx` — still the fallback when draws can't be fetched (`AGENTS.md:91-101`). Fix its `intervalMass` default to 0.95 to match the API. - **`package.json`** — add `d3-scale`, `d3-shape`, `d3-array`, `d3-interpolate` as real `dependencies` (the existing deps are all miscategorised under `devDependencies`; leave that alone unless it's breaking the build). - **Docs**: `AGENTS.md` still describes 6 charts, D3 selections, and `DistrictVsNational` / `ModelDrawsComparison` / `ExceedanceProbability` — none of which exist. Rewrite the architecture, data-flow, and interval sections. Update `README.md`'s chart list. --- ## Verification 1. `npm run build` clean; `npm test` (`node --test 'src/**/*.test.js'`) passes, including new tests for `agrestiCoull`, `pooling`, `densityProfile`, and the extended `drawGroups`. 2. `npm run dev`, then walk these districts: - **Clark County NV (`3200060`, 100 arrests / 148,928 students)** — pooling stays off, all four races render, differences chart defaults to the top two groups. - **Carson City NV (`3200390`, 6 arrests / 4,073 students)** — pooling auto-engages (6 < 20), banner appears, table collapses to four races. The AI/AN cell should render as a discrete mass profile, not a smear. Verified against the draws: pooling narrows AI/AN's 90% interval from 37.7 to 27.0 per 1,000, and Hispanic male's from 4.9 to 2.5. - **Washoe County NV (`3200480`)** — cross-check the summary table's observed counts and rates against `/api/v1/estimates/3200480?model=unified_m4_mod&year=21-22`. - **A California district** — confirm the suggestion fixture ranks correctly (this is the case the current code gets wrong), and that "compare all four" warns/spins before pulling ~25MB of shards. 3. Toggle every group off, then on; switch specs; flip pooling manually — no stale draws from a previous model should ever appear under a new label. 4. Compare chart A against `wp_fig_group_density` output for Clark County: same curve shapes, same point-range positions. 5. Browser console clean on hard refresh; check the Network tab shows one shard fetch per (model, state) and no repeats when navigating between districts. ## Effort Roughly **2–3 focused days** end to end: ~1 day for the draws/pooling/util layer with tests, ~1 day for the two charts, ~half a day for the table, the suggestion fixture, and docs. At ~10 hours a week that's about two calendar weeks. The R Shiny alternative would be slower, not faster — it trades a zero-server static site for a container, an R runtime, and server-side access to either the 91GB draws DuckDB or the 51-state parquet tree, and turns every toggle into a round-trip re-render. The React app already fetches real draws client-side and already carries the design tokens.