Compare commits

...
15 Commits
Author SHA1 Message Date
jared db5111dab9 docs: correct the error-bar convention and README to match the actual app
Deploy to git-pages / deploy (push) Successful in 22s
AGENTS.md's "Error Bar Convention" paragraph was wrong in four ways after the
last docs pass: it pointed "above" at a section that is below it, described the
fallback as "synthetic draw generation" when it generates no draws at all, cited
an `intervalWidth / 3.29` expression in RateDensityRidgeline.jsx that does not
exist, and listed ModelDrawsComparison.jsx, a file that was deleted when the
demo was reduced to 3 charts. Rewritten against the current code: the fallback
is fitSkewedInterval's analytic two-piece normal, whose divisor is
probit((1 + intervalMass) / 2) with intervalMass defaulting to 0.90, and the
"if you change this" list now names the three files that actually encode 90%.

README: 6 charts -> 3, the ridgeline is Chart 3 not Chart 5, Chart 2's use of
real draws is now mentioned, the D3.js v7 claim is dropped (d3 is not a
dependency), @duckdb/duckdb-wasm is listed in the tech stack, the file tree
matches src/, and the container-size note accounts for the wasm engine.
2026-08-11 10:37:37 -04:00
jared 096455944e fix(nginx): cache and compress the duckdb-wasm asset
The wasm engine is the single largest asset in the build (39,362,651 bytes) but
the immutable-cache location regex did not include `wasm`, so on the
Docker/nginx path it got no Cache-Control at all, and no gzip directive existed
anywhere -- nginx's default gzip_types is text/html only, so it shipped
uncompressed on every cold load.

Verified against nginx:alpine (1.31.3) with the real dist/ mounted:
`nginx -t` passes, and the wasm now returns Content-Type: application/wasm,
Content-Encoding: gzip, Cache-Control: public, max-age=31536000, immutable --
8,766,496 bytes on the wire instead of 39,362,651.

Dynamic gzip rather than gzip_static because the Vite build emits no
pre-compressed .gz files.
2026-08-11 10:37:27 -04:00
jared 420fc41571 fix: validate the deep-link state param before propagating it
?state= was used verbatim and ends up interpolated into a fetch URL and into
the Hugging Face parquet shard path duckdb-wasm reads. Impact is low -- it is a
client-only fetch to a public HTTPS URL, and the wasm sandbox loads no httpfs
-- but validating at the boundary is cheap and correct. Deep links now require
/^[A-Z]{2}$/ (after trim + uppercase, so ?state=az still works) and fall back
to the state selector with a warning when they do not match.
2026-08-11 10:37:26 -04:00
jared 70a12cad22 fix: gate the "real posterior draws" claim on per-group coverage
useDrawDistribution's status is an any-group signal -- 'ready' means at least
one group came back with draws -- but both charts used it chart-wide to hide
<ApproxNote /> and print a caption claiming every box/ridge is real draws.
Individual boxes and ridges already fall back per-group, so the note and
caption were the only things over-claiming.

This is reachable with real data, not just in theory: for LEAID 0400311 (AZ,
unified_m4_mod) the estimates API returns all 8 race x sex groups but the
parquet shard contains draws for WH_M only. The chart reported 'ready' and hid
the note while 7 of its 8 ridges were the analytic approximation.

New src/utils/drawGroups.js owns the "RACE_SEX" key format (previously
duplicated across the hook and both charts) and a hasDrawsForAll() coverage
check. Each chart now checks the groups it actually renders -- RACE_ORDER x its
sex panels/columns -- rather than everything the API returned. A null map
(loading, or a fetch that failed outright) fails the check, so the error path
still shows the note.

Also fixes a stale-state hazard in the same hook: the input guard ran before
setStatus('loading')/setDrawsByGroup(null), so switching to a model whose
groups list is empty (that model's upstream fetch failed) left status at
'ready' with the previous model's draws still in state. Verified by
instrumenting the hook: with the old ordering, switching from a ready model to
an empty one kept status 'ready' and all 8 previous draw keys; with the reset
moved above the guard it correctly resets to 'loading' with no draws.
2026-08-11 10:37:15 -04:00
jared 138a083c6f fix: clamp KDE bandwidth to the plotting domain, not absolute units
Small districts produce rare-event count posteriors that are heavily
zero-inflated, and the absolute 1e-3 floor / raw-sd fallback broke on both
extremes of that data:

- All-identical draws (e.g. 500 zeros, the norm for a group with a handful of
  students) collapsed to h = 1e-3, a near-delta spike of density ~399. Because
  SexRidgeColumn shares one maxPdf per column, that single spike flattened every
  other ridge in the column to sub-pixel height. Measured on a real district
  (0400315 AZ, unified_m4_mod): the three informative ridges rendered at
  0.07-0.13px of a 47.56px row.
- When IQR is 0 -- the normal case when most draws are 0 -- min(sd, iqr/1.34)
  was falsy and the rule fell all the way back to raw sd, which a few extreme
  draws inflate until the ridge is a flat line claiming maximal uncertainty.

Bandwidth is now clamped to [domainWidth/50, domainWidth/6] and the robust rule
degrades by picking the smallest *positive* spread estimate instead of
discarding robustness entirely. The floor is slightly wider than one render
step at n = 60, so a degenerate draw set resolves as a narrow bump; it does not
bind on an ordinary posterior (spread wider than ~8% of the domain keeps its
own Silverman bandwidth). On the district above the informative ridges now
render at 5.66-5.70px, a ~60x improvement.

Five new tests cover both failure modes; all five fail against the old formula.
2026-08-11 10:37:01 -04:00
jaredandClaude Sonnet 5 b5ebf6bec2 docs: clarify that synthetic draw generation is the fallback mechanism
AGENTS.md Error Bar Convention section now explicitly states that synthetic
draw generation from normal approximation is only used when real draws can't
be fetched (network error, unsupported browser, HF outage). Makes clear the
fallback is secondary, not primary, to the real-draw mechanism described in
the new §5.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-11 10:02:19 -04:00
jaredandClaude Sonnet 5 523c21d78c docs: reflect real posterior-draw architecture in AGENTS/README/HANDOFF
- AGENTS.md: Replace §4 (API Endpoint Availability) with updated text; insert
  new §5 (Real posterior draws via duckdb-wasm) documenting the shift from
  synthetic normal-approximation draws to client-side fetches via duckdb-wasm
  against the public Hugging Face parquet dataset. Include actual payload size
  (~39MB uncompressed / ~8.86MB gzipped). Renumber subsequent items.
- README.md: Update Chart 5 description in "What It Does" to reflect real draws
  + fallback behavior. Update API table to clarify that /api/v1/draws is not
  called from app but informs the Hugging Face URL the app fetches directly.
- HANDOFF.md: Mark "Raw posterior draws" as done (2026-08-11) with reference
  to the design spec.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-11 09:57:12 -04:00
jared 20c08ae198 fix: clear stale drawsByGroup when a model switch starts or fails
useDrawDistribution never cleared drawsByGroup on a new fetch or on
error, only status. A chart that varies `model` across renders (Chart
3's dropdown) could switch from a model that had succeeded to one
that then failed, and keep rendering the *previous* model's real
posterior draws — keyed by the same RACE_SEX strings — under the
newly-selected model's summary stats, with ApproxNote visible
suggesting (wrongly) that the fallback approximation was in use.

Clear drawsByGroup to null both when a new fetch starts and in the
catch branch, so a failed model switch never mixes draws from two
different models.
2026-08-11 09:43:09 -04:00
jared b5941a78d1 feat: drive Chart 3's ridgelines from real posterior draws, with fallback
Wire useDrawDistribution into RateDensityRidgeline so each race×sex ridge
uses a Gaussian KDE over that group's 500 real posterior draws when the
selected model's shard has loaded, falling back per-group to the analytic
fitSkewedInterval approximation otherwise. ApproxNote now only shows when
status !== 'ready'. Pass leaid/state through from ChartPanel.
2026-08-11 09:30:46 -04:00
jared ea6d554e3d fix: don't poison shard cache on failure; require real draws for ready status
Two Important review findings on useDrawDistribution:
- ensureShardRegistered cached the rejected promise on fetch/registration
  failure, permanently stuck for that (model,year,state) key until a full
  page reload. Now deletes the cache entry on failure so the next caller
  retries fresh.
- status could become 'ready' even when the LEAID-bound query returned zero
  usable rows (district not present in the draws shard), silently mislabeling
  a fitSkewedInterval-only render as real posterior draws. Now requires at
  least one group with actual draws before reporting 'ready'.
2026-08-11 09:22:01 -04:00
jared d5bf18482a feat: drive Chart 2's box plot from real posterior draws, with fallback 2026-08-11 09:07:53 -04:00
jared 91e21feb74 feat: add empirical KDE/quantile utilities for real posterior draws 2026-08-11 08:55:06 -04:00
jared 88b09a5d2e feat: add duckdb-wasm singleton for client-side parquet queries
Adds src/utils/duckdbClient.js, a lazy-initialized getDb() singleton
wrapping the AsyncDuckDB MVP (single-threaded) wasm bundle. Pins
@duckdb/duckdb-wasm to ^1.32.0 (npm's latest dist-tag currently points
at a -dev prerelease). Excludes the package from Vite's dev-server
dependency pre-bundling so its worker/wasm ?url imports resolve
correctly.

Verified live in the browser under the /crdc-demo/ base path: wasm and
worker assets load with 200, and a SELECT 42 query round-trips
correctly through the singleton, confirming the risk flagged in the
design spec (duckdb-wasm loading correctly from a subpath-served Vite
dev server) does not materialize.
2026-08-11 08:49:05 -04:00
jared 991a6912f8 docs: add implementation plan for empirical draw distributions via duckdb-wasm 2026-08-11 08:42:01 -04:00
jared 2cb04ad9c1 docs: add design spec for empirical draw distributions via duckdb-wasm
Scopes replacing the analytic distribution approximation with real
posterior draws, fetched client-side from the Hugging Face parquet
dataset using duckdb-wasm — no new backend endpoint required.
2026-08-11 07:41:21 -04:00
19 changed files with 2079 additions and 63 deletions
+24 -6
View File
@@ -46,11 +46,14 @@ App.jsx (router)
### Error Bar Convention: 90% Intervals
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**. When converting HPD bounds to standard deviations for synthetic draw generation (used in Chart 5's ridgelines), the divisor used is **3.29** (corresponding to z = 1.645 for a two-tailed 90% interval).
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**.
The 90% convention is baked into the analytic fallback in `src/utils/distributionApprox.js` (the path used when real posterior draws can't be fetched — see the duckdb-wasm section below). That fallback generates **no draws**: `fitSkewedInterval` fits a two-piece normal directly from `{median, lower, upper}`, converting each half-interval to its own sigma with `z = probit((1 + intervalMass) / 2)` and `intervalMass` defaulting to `0.90` (so z ≈ 1.645). For a symmetric interval that is equivalent to the old fixed "full width / 3.29" divisor, but it is computed from `intervalMass` rather than hardcoded, and each side gets its own sigma so the fitted shape stays skewed.
If you change this convention, update:
- `RateDensityRidgeline.jsx` — SD calculation (`intervalWidth / 3.29`) and label text
- `ModelDrawsComparison.jsx` — Legend labels mentioning "90%"
- `distributionApprox.js` — the `intervalMass = 0.90` default in `fitSkewedInterval`, and its JSDoc claim that the API's bounds are a 90% interval
- `RateByGroupBar.jsx` — the caption and component JSDoc, both of which say "the model's reported 90% interval"
- `ArrestsOverTime.jsx` — the legend label "Modeled (median + 90% interval)"
- Any documentation referencing confidence/credible intervals
## Common Pitfalls & Gotchas
@@ -83,11 +86,26 @@ Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for
Not all endpoints are available to browser-based clients:
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds.
### 5. Real posterior draws via duckdb-wasm
### 5. CORS Configuration
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
500-draw-per-group posterior from the public Hugging Face parquet dataset
(`civilytics/crdc-school-arrest-rates`), queried client-side with
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
involved. If that fetch fails (network, unsupported browser, HF outage),
both charts fall back to the `distributionApprox.js` analytic approximation
and show the "estimated shape" note — **do not delete `distributionApprox.js`
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
the full design.
The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.
### 6. CORS Configuration
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
+1 -1
View File
@@ -33,7 +33,7 @@ The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PRO
## What Needs to Be Done Next
### Future Enhancements
- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required.
- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`.
- **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling.
## Key Files
+25 -22
View File
@@ -18,22 +18,19 @@ npm run preview # serve built files locally
## What It Does
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 6 charts comparing observed data against Bayesian model estimates:
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 3 charts comparing observed data against Bayesian model estimates:
1. **Arrests over time** (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG
2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG
3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG
4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate)
5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates
6. **Exceedance probability** (Chart 6) — P(district > national) per student group
2. **Rate by student group** (Chart 2) — box-and-whisker per race×sex group for the most recent year, split into Female/Male panels. The box is the 25th–75th percentile of that group's real 500-draw posterior (same duckdb-wasm fetch as Chart 3), the whisker is the model's reported 90% interval, and a diamond marks the observed rate; individual boxes fall back to an analytic approximation when a group's draws can't be fetched.
3. **Predicted rates by student group** (Chart 3) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with a model-selector dropdown for the four Bayesian specifications and diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
## Architecture
### Tech Stack
- **React 19** + **Vite** (static site generation, no backend required)
- Plain CSS custom properties for styling (matches Civilytics design tokens exactly)
- D3.js v7 for data-driven visualizations (density ridges, scales, axes)
- Inline SVG rendering — charts are built with vanilla DOM/D3, not charting libraries
- Inline SVG rendering with hand-rolled scales — no charting library, no D3 dependency
- Embeds **DuckDB-Wasm** (`@duckdb/duckdb-wasm`, ~39MB uncompressed / ~8.8MB gzipped, loaded on demand only after a district is selected) to query real posterior draws client-side from a public Hugging Face Parquet dataset
- Calls the public read-only API directly from the browser
### File Structure
@@ -41,26 +38,32 @@ Visitors select a U.S. state, search for a school district (with suggestions of
crdc-demo/
├── index.html # Entry point
├── vite.config.mjs # Vite build config
├── public/ # Static assets (wordmark, favicon)
│ └── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
├── public/ # Static assets (wordmark, favicon, fixtures)
│ ├── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
│ └── data/national_rates.json # Static national rates fixture for comparisons
├── src/
│ ├── main.jsx # React entry
│ ├── App.jsx # Main router (state → search → loading → charts)
│ ├── hooks/useApi.js # API client with retry/backoff + endpoint wrappers
│ ├── hooks/
│ │ ├── useApi.js # API client with retry/backoff + endpoint wrappers
│ │ └── useDrawDistribution.js # Fetches real posterior draws (duckdb-wasm + HF parquet)
│ ├── components/
│ │ ├── StateSelector.jsx # Landing screen — state dropdown/grid
│ │ ├── DistrictSearch.jsx # Search + "interesting" district suggestions
│ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch
│ │ └── ChartPanel.jsx # Orchestrates all 6 charts + data fetching
│ │ ├── ChartLegend.jsx # Shared legend row
│ │ ├── ApproxNote.jsx # "shape estimated from interval bounds" caption
│ │ └── ChartPanel.jsx # Orchestrates all 3 charts + data fetching
│ ├── charts/
│ │ ├── ArrestsOverTime.jsx # Chart 1 — line chart by wave (SVG)
│ │ ├── RateByGroupBar.jsx # Chart 2 — bar chart by group (SVG)
│ │ ├── DistrictVsNational.jsx # Chart 3 — comparison vs. national avg
│ │ ├── ModelDrawsComparison.jsx # Chart 4 — quadrant model comparison (D3)
│ │ ├── RateDensityRidgeline.jsx # Chart 5 — density ridges per student group (D3)
│ │ └── ExceedanceProbability.jsx # Chart 6 — P(district > nat) per group
│ ├── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
│ └── data/national_rates.json # Static national rates fixture for comparisons
│ │ ├── ArrestsOverTime.jsx # Chart 1 — counts by wave (SVG)
│ │ ├── RateByGroupBar.jsx # Chart 2 — box-and-whisker by group (SVG)
│ │ └── RateDensityRidgeline.jsx # Chart 3 — density ridges per student group (SVG)
│ ├── utils/
│ │ ├── duckdbClient.js # Lazy duckdb-wasm bundle loader (dynamic import)
│ │ ├── kde.js # Empirical density from real draws (+ kde.test.js)
│ │ ├── drawGroups.js # Draw-map key format + coverage check (+ .test.js)
│ │ └── distributionApprox.js # Analytic fallback when draws are unavailable
│ └── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
├── Dockerfile # Multi-stage build → nginx static server
├── docker-compose.yml # Local dev / self-hosted deployment
└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch
@@ -72,7 +75,7 @@ crdc-demo/
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district |
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app |
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
## Deployment
@@ -114,7 +117,7 @@ docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo:
docker push registry.civilytics.org/crdc-demo:latest
```
The container is ~5MB (nginx Alpine) and serves static files with immutable cache headers. A `/healthz` endpoint supports container orchestration health checks.
The container is nginx Alpine (~5MB) plus the built site, which is dominated by the ~39MB DuckDB-Wasm engine. `nginx.conf` serves that engine gzipped (~8.8MB on the wire) with immutable cache headers, so it is fetched once per browser. A `/healthz` endpoint supports container orchestration health checks.
### Environment Variables
| Variable | Default | Description |
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,187 @@
# Empirical draw distributions via DuckDB-Wasm — Design Spec
**Date:** 2026-08-11
**Status:** Draft for review
---
## 1. Purpose & context
Two of this app's three charts currently show a *modeled* distribution shape that is
not the real posterior — `src/utils/distributionApprox.js` fits a two-piece-normal
curve to each group's `(median, lower, upper)` summary stats returned by the
`/estimates` API, because the API's raw posterior draws are only available in bulk
as Hive-partitioned Parquet on Hugging Face
(`civilytics/crdc-school-arrest-rates`), meant for DuckDB/bulk consumption, not
browser fetches (see `AGENTS.md` §"API Endpoint Availability" and
`HANDOFF.md` §"Future Enhancements").
This spec replaces the approximation with the **real** empirical draws (500 per
group), fetched client-side using `@duckdb/duckdb-wasm` to query the actual Parquet
shard for the district's state directly from Hugging Face — no new backend
endpoint, no change to the existing summary API calls.
**Verified feasibility (2026-08-11):**
- The HF dataset is public, non-gated. Each `(model_id, YEAR, LEA_STATE)` partition
is a single file (`data_0.parquet`).
- File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed
by resolving the `resolve/main/...` redirect to the actual CDN blob.
- The redirect target sends `access-control-allow-origin: *` and
`accept-ranges: bytes` — browser `fetch()` works directly, no proxy needed.
- Schema (from `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`):
`LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred`, sorted
within each shard by `(LEAID, RACE, SEX)`. `LEA_STATE`/`YEAR`/`model_id` are
Hive-partition columns (encoded in the path, not repeated in every row).
`stu_enroll` is **not** in the draws table — it's already available in this app
from the existing `/estimates` summary call.
---
## 2. Architecture
```
district selected (leaid, state)
│
▼
resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state)
e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/
resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet
│
▼
fetch() the shard (native fetch, follows the HF→CDN redirect automatically)
│
▼
duckdb-wasm: registerFileBuffer + query
SELECT RACE, SEX, pred FROM shard WHERE LEAID = '<leaid>'
│
▼
join `pred` (posterior count draws) against stu_enroll already in app state
(from the existing /estimates summary call) → rate-per-1000 draws per group
│
▼
KDE per race×sex group → smooth density curve, same shape the charts draw today
```
Given verified shard sizes (≤6.4MB), the design fetches the **whole shard** with a
plain `fetch()` and queries it in-memory via duckdb-wasm, rather than relying on
fine-grained HTTP range / row-group pruning. This is simpler and more robust than
depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN
redirect chain under partial-range requests — an unverified behavior — for a
saving that wouldn't matter at these file sizes.
`@duckdb/duckdb-wasm` is MIT-licensed and runs entirely client-side in a Web
Worker. It introduces no new server dependency and no new hosted service beyond
the Hugging Face dataset the `crdc-arrests` project's `/draws` endpoint already
points to (per `2026-05-30-draws-api-design.md`, decision #5) — this spec doesn't
introduce that dependency, it makes the demo app actually use data that was
already published there for exactly this purpose.
**Deployment risk:** duckdb-wasm's threaded ("eh") bundle requires
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` response headers
(for `SharedArrayBuffer`), which the git-pages static host does not send today.
This design uses the **single-threaded ("mvp") bundle** instead — at these file
sizes threading has no meaningful benefit, and it avoids needing new headers on
both the git-pages and Docker/nginx deploy paths.
---
## 3. File-level changes
### New files
- **`src/utils/duckdbClient.js`** — lazy-initialized singleton. Dynamic-imports
`@duckdb/duckdb-wasm`, selects the MVP (non-threaded) bundle, starts the worker
once. Dynamic `import()` keeps the ~3–5MB wasm payload out of the main bundle;
it only loads when a chart actually needs draws.
- **`src/utils/kde.js`** — Gaussian KDE over an array of numbers (Silverman
bandwidth). Takes over the role `distributionApprox.js`'s `densityCurve` plays
today, fed real empirical draws instead of a parametric fit.
- **`src/hooks/useDrawDistribution.js`** — given
`{ leaid, state, model, year, groups }` (groups = race/sex + `stu_enroll` already
in app state), resolves the HF URL, fetches, registers the buffer with
duckdb-wasm, runs the query, joins enrollment, and returns
`{ status: 'loading' | 'ready' | 'error', drawsByGroup }`. Owns an in-memory
`Map` cache keyed by `model+state+year` so re-selecting a model in Chart 3's
dropdown, or viewing another district in the same state, reuses the shard
already fetched.
### Modified files
- **`RateDensityRidgeline.jsx`** — on model-dropdown change, calls
`useDrawDistribution` for the selected model; replaces
`fitSkewedInterval`/`densityCurve` with the hook's real draws → `kde.js`. Shows
an inline spinner in the ridge area while that model's shard is loading (Charts
1–2 aren't blocked).
- **`RateByGroupBar.jsx`** — fetches draws for `unified_m3_mod` (the one model
this chart uses) alongside its existing data fetch. `q1`/`q3` become exact
empirical quantiles from the 500 real draws — removes this chart's use of
`fitSkewedInterval`'s fitted quantile function entirely.
- **`ChartPanel.jsx`** — passes `district.leaid`, `state`, and each group's
`stu_enroll` (already fetched) down to the two charts above.
- **`ApproxNote.jsx`** — becomes conditional: renders the "estimated shape" note
only when a chart is in fallback mode; charts backed by real draws show no note
(or a neutral "500 posterior draws" caption).
- **`package.json` / `vite.config.mjs`** — add `@duckdb/duckdb-wasm`; wasm/worker
assets are pulled in via Vite's native `?url` imports, which already respect the
`/crdc-demo/` `base` path — no bundler plugin needed.
### Unchanged
- **`ArrestsOverTime.jsx`** — already uses real summary stats (point-range from
`/estimates`), no approximation involved; out of scope.
- **`useApi.js`** — still the source for medians, intervals, and enrollment.
- **`distributionApprox.js`** — kept as the fallback path (see §4).
---
## 4. Error handling & caching
**Fallback:** `distributionApprox.js` is retained. `useDrawDistribution` catches
fetch/wasm/query failures and returns `status: 'error'`; both chart components
branch on that to render the current analytic-approximation path with
`<ApproxNote />` visible. A Hugging Face outage, a network failure, or an
unsupported browser degrades to today's behavior rather than breaking the chart.
**Caching:** in-memory only (a `Map` inside the hook), scoped to the browser
session. No IndexedDB/persistent cache in this iteration — a demo session
typically covers one or two districts, and shards are cheap enough to refetch on
reload.
---
## 5. Testing
This project has no automated test suite (per `AGENTS.md`); this follows the
existing manual-verification convention:
1. `npm run dev`; walk a small state (DC or WY, ~130–200KB shard) and a large one
(CA or TX, several MB) through the full district-search flow.
2. Confirm both charts render from real draws; confirm the Network tab shows the
expected parquet fetch(es) and sizes.
3. Simulate failure (block the `huggingface.co` / CDN domain in devtools) and
confirm both charts fall back to the analytic approximation with the note
visible, rather than breaking.
4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is
instant on re-selection (cache hit).
---
## 6. Open risk to de-risk first
Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query
correctly when deployed under the `/crdc-demo/` subpath on git-pages, and does
nginx/git-pages serve `.wasm` with a usable content type? Everything else in this
design is standard Vite asset handling already exercised elsewhere in the app, but
this specific combination (wasm worker + subpath base + static host) hasn't been
verified end-to-end and should be checked with a throwaway spike rather than
assumed.
---
## 7. Explicitly out of scope
- `ArrestsOverTime.jsx` (Chart 1) — no approximation to replace.
- A new server-side `/draws`-streaming API endpoint — explicitly rejected in favor
of client-side wasm access, per the brainstorming decision that led to this spec.
- Persistent (IndexedDB) caching of fetched shards.
- Fine-grained HTTP range / row-group-level partial reads — shard sizes are small
enough that whole-file fetch is simpler and sufficiently fast.
- Extending empirical draws to national/exceedance-probability views — those
charts aren't part of the current 3-chart demo.
+22 -2
View File
@@ -37,12 +37,32 @@ server {
root /usr/share/nginx/html/crdc-demo/; # Adjust if base is different
index index.html;
# Compress text assets and — most importantly — the duckdb-wasm engine,
# which is ~39MB uncompressed and ~8.8MB gzipped. Without this it ships
# uncompressed on every cold load. Dynamic gzip rather than gzip_static
# because the Vite build emits no pre-compressed .gz files.
gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_proxied any;
gzip_comp_level 6;
gzip_types
text/plain
text/css
application/javascript
text/javascript
application/json
image/svg+xml
application/wasm;
location / {
try_files $uri $uri/ /index.html;
}
# Civilytics design tokens: immutable cache headers for static assets
location ~* \.(js|css|png|jpg|jpeg|gif|svg|woff2|ttf)$ {
# Civilytics design tokens: immutable cache headers for static assets.
# `wasm` belongs here too — the duckdb engine is content-hashed by Vite and
# is by far the largest asset in the build, so it must not be re-fetched.
location ~* \.(js|css|wasm|png|jpg|jpeg|gif|svg|woff2|ttf)$ {
expires 1y;
add_header Cache-Control "public, max-age=31536000, immutable";
try_files $uri =404;
+238
View File
@@ -9,6 +9,7 @@
"version": "0.1.0",
"license": "MIT",
"devDependencies": {
"@duckdb/duckdb-wasm": "^1.32.0",
"@vitejs/plugin-react-swc": "^4.3.3",
"eslint": "^8.57.1",
"prettier": "^3.9.6",
@@ -17,6 +18,16 @@
"vite": "^8.2.1"
}
},
"node_modules/@duckdb/duckdb-wasm": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/@duckdb/duckdb-wasm/-/duckdb-wasm-1.32.0.tgz",
"integrity": "sha512-IewXTNYEjsZCPE9weUWgtjGxUlMRo7qhX0GF6tq/KjK8bnY+RAl4cyUdYUfcdzbyb4b9ZxPC+FOsCcxgaKFWMg==",
"dev": true,
"license": "MIT",
"dependencies": {
"apache-arrow": "^17.0.0"
}
},
"node_modules/@eslint-community/eslint-utils": {
"version": "4.10.1",
"resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.10.1.tgz",
@@ -663,6 +674,16 @@
"dev": true,
"license": "Apache-2.0"
},
"node_modules/@swc/helpers": {
"version": "0.5.23",
"resolved": "https://registry.npmjs.org/@swc/helpers/-/helpers-0.5.23.tgz",
"integrity": "sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"tslib": "^2.8.0"
}
},
"node_modules/@swc/types": {
"version": "0.1.28",
"resolved": "https://registry.npmjs.org/@swc/types/-/types-0.1.28.tgz",
@@ -673,6 +694,30 @@
"@swc/counter": "^0.1.3"
}
},
"node_modules/@types/command-line-args": {
"version": "5.2.3",
"resolved": "https://registry.npmjs.org/@types/command-line-args/-/command-line-args-5.2.3.tgz",
"integrity": "sha512-uv0aG6R0Y8WHZLTamZwtfsDLVRnOa+n+n5rEvFWL5Na5gZ8V2Teab/duDPFzIIIhs9qizDpcavCusCLJZu62Kw==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/command-line-usage": {
"version": "5.0.4",
"resolved": "https://registry.npmjs.org/@types/command-line-usage/-/command-line-usage-5.0.4.tgz",
"integrity": "sha512-BwR5KP3Es/CSht0xqBcUXS3qCAUVXwpRKsV2+arxeb65atasuXG9LykC9Ab10Cw3s2raH92ZqOeILaQbsB2ACg==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/node": {
"version": "20.19.43",
"resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.43.tgz",
"integrity": "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA==",
"dev": true,
"license": "MIT",
"dependencies": {
"undici-types": "~6.21.0"
}
},
"node_modules/@ungap/structured-clone": {
"version": "1.3.3",
"resolved": "https://registry.npmjs.org/@ungap/structured-clone/-/structured-clone-1.3.3.tgz",
@@ -763,6 +808,27 @@
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
}
},
"node_modules/apache-arrow": {
"version": "17.0.0",
"resolved": "https://registry.npmjs.org/apache-arrow/-/apache-arrow-17.0.0.tgz",
"integrity": "sha512-X0p7auzdnGuhYMVKYINdQssS4EcKec9TCXyez/qtJt32DrIMGbzqiaMiQ0X6fQlQpw8Fl0Qygcv4dfRAr5Gu9Q==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"@swc/helpers": "^0.5.11",
"@types/command-line-args": "^5.2.3",
"@types/command-line-usage": "^5.0.4",
"@types/node": "^20.13.0",
"command-line-args": "^5.2.1",
"command-line-usage": "^7.0.1",
"flatbuffers": "^24.3.25",
"json-bignum": "^0.0.3",
"tslib": "^2.6.2"
},
"bin": {
"arrow2csv": "bin/arrow2csv.cjs"
}
},
"node_modules/argparse": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
@@ -770,6 +836,16 @@
"dev": true,
"license": "Python-2.0"
},
"node_modules/array-back": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/array-back/-/array-back-3.1.0.tgz",
"integrity": "sha512-TkuxA4UCOvxuDK6NZYXCalszEzj+TLszyASooky+i742l9TqsOdYCMJJupxRic61hwquNtppB3hgcuq9SVSH1Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
@@ -815,6 +891,22 @@
"url": "https://github.com/chalk/chalk?sponsor=1"
}
},
"node_modules/chalk-template": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/chalk-template/-/chalk-template-0.4.0.tgz",
"integrity": "sha512-/ghrgmhfY8RaSdeo43hNXxpoHAtxdbskUHjPpfqUWGttFgycUhYPGx3YZBCnUCvOa7Doivn1IZec3DEGFoMgLg==",
"dev": true,
"license": "MIT",
"dependencies": {
"chalk": "^4.1.2"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/chalk-template?sponsor=1"
}
},
"node_modules/color-convert": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
@@ -835,6 +927,58 @@
"dev": true,
"license": "MIT"
},
"node_modules/command-line-args": {
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/command-line-args/-/command-line-args-5.2.1.tgz",
"integrity": "sha512-H4UfQhZyakIjC74I9d34fGYDwk3XpSr17QhEd0Q3I9Xq1CETHo4Hcuo87WyWHpAF1aSLjLRf5lD9ZGX2qStUvg==",
"dev": true,
"license": "MIT",
"dependencies": {
"array-back": "^3.1.0",
"find-replace": "^3.0.0",
"lodash.camelcase": "^4.3.0",
"typical": "^4.0.0"
},
"engines": {
"node": ">=4.0.0"
}
},
"node_modules/command-line-usage": {
"version": "7.0.4",
"resolved": "https://registry.npmjs.org/command-line-usage/-/command-line-usage-7.0.4.tgz",
"integrity": "sha512-85UdvzTNx/+s5CkSgBm/0hzP80RFHAa7PsfeADE5ezZF3uHz3/Tqj9gIKGT9PTtpycc3Ua64T0oVulGfKxzfqg==",
"dev": true,
"license": "MIT",
"dependencies": {
"array-back": "^6.2.2",
"chalk-template": "^0.4.0",
"table-layout": "^4.1.1",
"typical": "^7.3.0"
},
"engines": {
"node": ">=12.20.0"
}
},
"node_modules/command-line-usage/node_modules/array-back": {
"version": "6.2.3",
"resolved": "https://registry.npmjs.org/array-back/-/array-back-6.2.3.tgz",
"integrity": "sha512-SGDvmg6QTYiTxCBkYVmThcoa67uLl35pyzRHdpCGBOcqFy6BtwnphoFPk7LhJshD+Yk1Kt35WGWeZPTgwR4Fhw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12.17"
}
},
"node_modules/command-line-usage/node_modules/typical": {
"version": "7.3.0",
"resolved": "https://registry.npmjs.org/typical/-/typical-7.3.0.tgz",
"integrity": "sha512-ya4mg/30vm+DOWfBg4YK3j2WD6TWtRkCbasOJr40CseYENzCUby/7rIvXA99JGsQHeNxLbnXdyLLxKSv3tauFw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12.17"
}
},
"node_modules/concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
@@ -1131,6 +1275,19 @@
"node": "^10.12.0 || >=12.0.0"
}
},
"node_modules/find-replace": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/find-replace/-/find-replace-3.0.0.tgz",
"integrity": "sha512-6Tb2myMioCAgv5kfvP5/PkZZ/ntTpVK39fHY7WkWBgvbeE+VHd/tZuZ4mrC+bxh4cfOZeYKVPaJIZtZXV7GNCQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"array-back": "^3.0.1"
},
"engines": {
"node": ">=4.0.0"
}
},
"node_modules/find-up": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/find-up/-/find-up-5.0.0.tgz",
@@ -1163,6 +1320,13 @@
"node": "^10.12.0 || >=12.0.0"
}
},
"node_modules/flatbuffers": {
"version": "24.12.23",
"resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-24.12.23.tgz",
"integrity": "sha512-dLVCAISd5mhls514keQzmEG6QHmUUsNuWsb4tFafIUwvvgDjXhtfAYSKOzt5SWOy+qByV5pbsDZ+Vb7HUOBEdA==",
"dev": true,
"license": "Apache-2.0"
},
"node_modules/flatted": {
"version": "3.4.4",
"resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.4.tgz",
@@ -1379,6 +1543,15 @@
"js-yaml": "bin/js-yaml.js"
}
},
"node_modules/json-bignum": {
"version": "0.0.3",
"resolved": "https://registry.npmjs.org/json-bignum/-/json-bignum-0.0.3.tgz",
"integrity": "sha512-2WHyXj3OfHSgNyuzDbSxI1w2jgw5gkWSWhS7Qg4bWXx1nLk3jnbwfUeS0PSba3IzpTUWdHxBieELUzXRjQB2zg==",
"dev": true,
"engines": {
"node": ">=0.8"
}
},
"node_modules/json-buffer": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/json-buffer/-/json-buffer-3.0.1.tgz",
@@ -1701,6 +1874,13 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/lodash.camelcase": {
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/lodash.camelcase/-/lodash.camelcase-4.3.0.tgz",
"integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA==",
"dev": true,
"license": "MIT"
},
"node_modules/lodash.merge": {
"version": "4.6.2",
"resolved": "https://registry.npmjs.org/lodash.merge/-/lodash.merge-4.6.2.tgz",
@@ -2160,6 +2340,30 @@
"node": ">=8"
}
},
"node_modules/table-layout": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/table-layout/-/table-layout-4.1.1.tgz",
"integrity": "sha512-iK5/YhZxq5GO5z8wb0bY1317uDF3Zjpha0QFFLA8/trAoiLbQD0HUbMesEaxyzUgDxi2QlcbM8IvqOlEjgoXBA==",
"dev": true,
"license": "MIT",
"dependencies": {
"array-back": "^6.2.2",
"wordwrapjs": "^5.1.0"
},
"engines": {
"node": ">=12.17"
}
},
"node_modules/table-layout/node_modules/array-back": {
"version": "6.2.3",
"resolved": "https://registry.npmjs.org/array-back/-/array-back-6.2.3.tgz",
"integrity": "sha512-SGDvmg6QTYiTxCBkYVmThcoa67uLl35pyzRHdpCGBOcqFy6BtwnphoFPk7LhJshD+Yk1Kt35WGWeZPTgwR4Fhw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12.17"
}
},
"node_modules/text-table": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/text-table/-/text-table-0.2.0.tgz",
@@ -2184,6 +2388,13 @@
"url": "https://github.com/sponsors/SuperchupuDev"
}
},
"node_modules/tslib": {
"version": "2.8.1",
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"dev": true,
"license": "0BSD"
},
"node_modules/type-check": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz",
@@ -2210,6 +2421,23 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/typical": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/typical/-/typical-4.0.0.tgz",
"integrity": "sha512-VAH4IvQ7BDFYglMd7BPRDfLgxZZX4O4TFcRDA6EN5X7erNJJq+McIEp8np9aVtxrCJ6qx4GTYVfOWNjcqwZgRw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/undici-types": {
"version": "6.21.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
"integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
"dev": true,
"license": "MIT"
},
"node_modules/uri-js": {
"version": "4.4.1",
"resolved": "https://registry.npmjs.org/uri-js/-/uri-js-4.4.1.tgz",
@@ -2324,6 +2552,16 @@
"node": ">=0.10.0"
}
},
"node_modules/wordwrapjs": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/wordwrapjs/-/wordwrapjs-5.1.1.tgz",
"integrity": "sha512-0yweIbkINJodk27gX9LBGMzyQdBDan3s/dEAiwBOj+Mf0PPyWL6/rikalkv8EeD0E8jm4o5RXEOrFTP3NXbhJg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12.17"
}
},
"node_modules/wrappy": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
+2
View File
@@ -6,6 +6,7 @@
"dev": "vite",
"build": "vite build",
"preview": "vite preview",
"test": "node --test 'src/**/*.test.js'",
"lint": "eslint src/ --ext .js,.jsx,.ts,.tsx",
"format": "prettier --write \"src/**/*.{js,jsx,css}\""
},
@@ -18,6 +19,7 @@
"author": "Civilytics Consulting LLC",
"license": "MIT",
"devDependencies": {
"@duckdb/duckdb-wasm": "^1.32.0",
"@vitejs/plugin-react-swc": "^4.3.3",
"eslint": "^8.57.1",
"prettier": "^3.9.6",
+16 -2
View File
@@ -6,9 +6,14 @@ import ChartPanel from './components/ChartPanel.jsx'
import Footer from './components/Footer.jsx'
import { fetchDistrictEstimates } from './hooks/useApi.js'
// Two-letter USPS state code. The deep-link value flows into API request URLs
// and into the Hugging Face parquet shard path duckdb-wasm reads, so it gets
// validated here at the boundary rather than propagated verbatim.
const STATE_CODE_PATTERN = /^[A-Z]{2}$/
/**
* CRDC Arrests API Demo App — main router.
* Flow: state → district search (with interesting suggestions) → loading animation → 6 charts
* Flow: state → district search (with interesting suggestions) → loading animation → charts
*/
export default function App() {
const [step, setStep] = useState('state') // 'state' | 'search' | 'loading' | 'results'
@@ -19,7 +24,16 @@ export default function App() {
useEffect(() => {
const params = new URLSearchParams(window.location.search)
const leaid = params.get('leaid')
const stateParam = params.get('state')
const rawState = params.get('state')
const stateParam = rawState ? rawState.trim().toUpperCase() : null
if (rawState && !STATE_CODE_PATTERN.test(stateParam)) {
// Malformed deep link — drop it and start at the state selector rather
// than passing an arbitrary string into fetch URLs and shard paths.
console.warn('Ignoring deep link: `state` is not a two-letter state code.')
return
}
if (leaid && stateParam) {
// Deep link (including browser back/forward landing on this URL): resolve
// the district name via a real estimates row, keyed by LEAID. The previous
+45 -16
View File
@@ -2,33 +2,52 @@ import ChartLegend from '../components/ChartLegend.jsx'
import ApproxNote from '../components/ApproxNote.jsx'
import { raceColor, OBSERVED_MARK_COLOR, SHORT_RACE_LABEL } from '../utils/colors.js'
import { fitSkewedInterval } from '../utils/distributionApprox.js'
import { groupKey, hasDrawsForAll } from '../utils/drawGroups.js'
import { quantile } from '../utils/kde.js'
import { niceTicks } from '../utils/niceTicks.js'
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
/**
* Arrest rate by student group, most recent year, disaggregated into two
* panels (Female / Male), each a horizontal box-and-whisker across the 4
* race categories. Whisker = the model's reported 90% interval, box = the
* fitted approximation's 25th-75th percentile, white tick = median, dark
* diamond = observed rate.
* race categories. Whisker = the model's reported 90% interval; box = the
* 25th-75th percentile of the group's 500 real posterior draws (or, if draws
* are unavailable, the fitSkewedInterval approximation); white tick =
* median; dark diamond = observed rate.
*/
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
const SEX_PANELS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
const ROW_HEIGHT = 34
const MODEL = 'unified_m3_mod' // matches ChartPanel's WAVE_MODEL, which built `data`
function buildBox(d) {
function buildBox(d, draws) {
const median = Math.max(d.modeledMedian || 0, 0)
const lower = Math.max(Math.min(d.rateLower ?? median, median), 0)
const upper = Math.max(d.rateUpper ?? median, median)
const fit = fitSkewedInterval({ median, lower, upper })
return {
lower, upper, median,
q1: Math.max(fit.quantile(0.25), 0),
q3: Math.max(fit.quantile(0.75), median),
if (draws && draws.length > 0) {
return { lower, upper, median, q1: Math.max(quantile(draws, 0.25), 0), q3: Math.max(quantile(draws, 0.75), median) }
}
const fit = fitSkewedInterval({ median, lower, upper })
return { lower, upper, median, q1: Math.max(fit.quantile(0.25), 0), q3: Math.max(fit.quantile(0.75), median) }
}
export default function RateByGroupBar({ data }) {
export default function RateByGroupBar({ data, leaid, state }) {
const groups = data.map((d) => ({ race: d.race, sex: d.sex, stuEnroll: d.enrollment }))
const { drawsByGroup } = useDrawDistribution({ leaid, state, model: MODEL, year: '21-22', groups })
// Only the RACE_ORDER × SEX_PANELS cells are actually drawn, so coverage is
// judged against those — not against everything the API returned. The hook's
// 'ready' status is an any-group signal and would over-claim here: individual
// boxes still fall back to fitSkewedInterval whenever their group is missing
// from the shard. A null map (loading, or the whole fetch failed) fails this
// check too, so the error path still shows the note.
const renderedGroups = data.filter(
(d) => RACE_ORDER.includes(d.race) && SEX_PANELS.some((p) => p.sex === d.sex),
)
const allGroupsHaveDraws = hasDrawsForAll(drawsByGroup, renderedGroups)
const maxRate = Math.max(
...data.map((d) => Math.max(d.observedRate, d.rateUpper ?? d.modeledMedian ?? 0)),
0.5
@@ -40,11 +59,18 @@ export default function RateByGroupBar({ data }) {
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
Arrest rate by student group — 2021–22 (per 1,000)
</h3>
<ApproxNote />
{!allGroupsHaveDraws && <ApproxNote />}
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
{SEX_PANELS.map(({ sex, label }) => (
<SexPanel key={sex} label={label} rows={data.filter((d) => d.sex === sex)} ticks={ticks} niceMax={niceMax} />
<SexPanel
key={sex}
label={label}
rows={data.filter((d) => d.sex === sex)}
drawsByGroup={drawsByGroup}
ticks={ticks}
niceMax={niceMax}
/>
))}
</div>
@@ -56,14 +82,16 @@ export default function RateByGroupBar({ data }) {
]} />
<p style={{ fontSize: '0.72rem', color: 'var(--cv-ink-3)', marginTop: 'var(--space-1)' }}>
Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported
90% interval; white tick = median. The dark diamond is the observed rate.
{allGroupsHaveDraws
? "Box = 25th–75th percentile of 500 real posterior draws; whisker = the model's reported 90% interval; white tick = median."
: "Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported 90% interval; white tick = median."}
{' '}The dark diamond is the observed rate.
</p>
</div>
)
}
function SexPanel({ label, rows, ticks, niceMax }) {
function SexPanel({ label, rows, drawsByGroup, ticks, niceMax }) {
const width = 300
const margin = { top: 30, right: 16, bottom: 34, left: 66 }
const innerWidth = width - margin.left - margin.right
@@ -99,7 +127,8 @@ function SexPanel({ label, rows, ticks, niceMax }) {
const boxTop = midY - ROW_HEIGHT * 0.26
const boxBottom = midY + ROW_HEIGHT * 0.26
const color = raceColor(d.race)
const box = buildBox(d)
const draws = drawsByGroup?.[groupKey(d.race, d.sex)]
const box = buildBox(d, draws)
const observedX = xScale(d.observedRate)
return (
+43 -12
View File
@@ -2,23 +2,26 @@ import { useState } from 'react'
import { MODEL_QUADRANTS } from '../hooks/useApi.js'
import { raceColor, OBSERVED_MARK_COLOR, RACE_LABELS, SHORT_RACE_LABEL } from '../utils/colors.js'
import { fitSkewedInterval, densityCurve } from '../utils/distributionApprox.js'
import { groupKey, hasDrawsForAll } from '../utils/drawGroups.js'
import { kdeCurve } from '../utils/kde.js'
import { niceTicks } from '../utils/niceTicks.js'
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
import ChartLegend from '../components/ChartLegend.jsx'
import ApproxNote from '../components/ApproxNote.jsx'
/**
* Modeled posterior density per race×sex group, for one selected model
* (dropdown, default three-year + referral rate), split into Female/Male
* columns — matches whitepaper-fig-clark-density-1.png's ridge style. Plain
* SVG — each ridge is 60 analytic points from fitSkewedInterval/densityCurve,
* already smooth without a binning/curveBasis smoothing pass.
* columns. Each ridge is drawn from that group's 500 real posterior draws
* (Gaussian KDE) when available, falling back per-group to the analytic
* fitSkewedInterval/densityCurve approximation otherwise.
*/
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
const SEX_COLUMNS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
const DEFAULT_MODEL = 'unified_m4_mod' // Three-year + referral rate
function buildGroupRow(row) {
function buildGroupRow(row, draws) {
const enroll = row.stu_enroll || 0
const observedRate = enroll > 0 ? ((row.observed_arrests || 0) / enroll) * 1000 : 0
const rateMedian = (row.rate_median || 0) * 1000
@@ -28,13 +31,27 @@ function buildGroupRow(row) {
race: row.race,
sex: row.sex,
observedRate,
draws,
fit: fitSkewedInterval({ median: rateMedian, lower: rateLower, upper: rateUpper }),
}
}
export default function RateDensityRidgeline({ quadData }) {
export default function RateDensityRidgeline({ quadData, leaid, state }) {
const [selectedModel, setSelectedModel] = useState(DEFAULT_MODEL)
const rows = (quadData && quadData[selectedModel]) || []
const groups = rows.map((r) => ({ race: r.race, sex: r.sex, stuEnroll: r.stu_enroll || 0 }))
const { drawsByGroup } = useDrawDistribution({ leaid, state, model: selectedModel, year: '21-22', groups })
// Only the RACE_ORDER × SEX_COLUMNS cells get a ridge, so coverage is judged
// against those. The hook's 'ready' status is an any-group signal: individual
// ridges still fall back to densityCurve when their group is missing from the
// shard, so gating the note on `status` alone would hide it while part of the
// chart is an approximation. A null map (loading or a failed fetch) also fails
// this check, so the error path still shows the note.
const renderedGroups = rows.filter(
(r) => RACE_ORDER.includes(r.race) && SEX_COLUMNS.some((c) => c.sex === r.sex),
)
const allGroupsHaveDraws = hasDrawsForAll(drawsByGroup, renderedGroups)
const modelSelect = (
<select
@@ -55,7 +72,7 @@ export default function RateDensityRidgeline({ quadData }) {
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
Predicted arrest rates by student group
</h3>
<ApproxNote />
{!allGroupsHaveDraws && <ApproxNote />}
</div>
{modelSelect}
</div>
@@ -64,7 +81,7 @@ export default function RateDensityRidgeline({ quadData }) {
<p style={{ color: 'var(--cv-ink-3)', marginTop: 'var(--space-2)' }}>No model data available.</p>
) : (
<>
<RidgeColumns rows={rows} />
<RidgeColumns rows={rows} drawsByGroup={drawsByGroup} />
<ChartLegend items={[
...RACE_ORDER.map((race) => ({ shape: 'swatch', color: raceColor(race), label: RACE_LABELS[race] })),
{ shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
@@ -75,7 +92,7 @@ export default function RateDensityRidgeline({ quadData }) {
)
}
function RidgeColumns({ rows }) {
function RidgeColumns({ rows, drawsByGroup }) {
// Shared x-domain across both columns, so Female/Male are directly comparable.
const allUpper = rows.map((r) => (r.rate_upper || 0) * 1000)
const allObserved = rows
@@ -87,14 +104,24 @@ function RidgeColumns({ rows }) {
return (
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
{SEX_COLUMNS.map(({ sex, label }) => (
<SexRidgeColumn key={sex} label={label} rows={rows.filter((r) => r.sex === sex)} ticks={ticks} maxRate={niceMax} />
<SexRidgeColumn
key={sex}
label={label}
rows={rows.filter((r) => r.sex === sex)}
drawsByGroup={drawsByGroup}
ticks={ticks}
maxRate={niceMax}
/>
))}
</div>
)
}
function SexRidgeColumn({ label, rows, ticks, maxRate }) {
const groups = RACE_ORDER.map((race) => rows.find((r) => r.race === race)).filter(Boolean).map(buildGroupRow)
function SexRidgeColumn({ label, rows, drawsByGroup, ticks, maxRate }) {
const groups = RACE_ORDER
.map((race) => rows.find((r) => r.race === race))
.filter(Boolean)
.map((row) => buildGroupRow(row, drawsByGroup?.[groupKey(row.race, row.sex)]))
const width = 300
const rowHeight = 58
@@ -104,7 +131,11 @@ function SexRidgeColumn({ label, rows, ticks, maxRate }) {
const xScale = (val) => margin.left + (val / maxRate) * innerWidth
const curves = groups.map((g) => densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }))
const curves = groups.map((g) =>
g.draws && g.draws.length > 0
? kdeCurve(g.draws, { min: 0, max: maxRate, n: 60 })
: densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }),
)
const maxPdf = Math.max(...curves.flatMap((c) => c.map((p) => p.y)), 1e-9)
const peakHeight = rowHeight * 0.82
+2 -2
View File
@@ -95,9 +95,9 @@ export default function ChartPanel({ district, state }) {
}}>
<ArrestsOverTime data={timeSeriesData} modelId={WAVE_MODEL} />
<RateByGroupBar data={rateByGroup} />
<RateByGroupBar data={rateByGroup} leaid={district.leaid} state={state} />
<RateDensityRidgeline quadData={data.quadData} />
<RateDensityRidgeline quadData={data.quadData} leaid={district.leaid} state={state} />
</div>
{/* Methodology footer */}
+150
View File
@@ -0,0 +1,150 @@
import { useEffect, useState } from 'react'
import { getDb } from '../utils/duckdbClient.js'
import { groupKey } from '../utils/drawGroups.js'
const HF_BASE = 'https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet'
// Module-level cache: one registered duckdb-wasm file buffer per
// (model, year, state) shard, shared across every component instance and
// district navigated to in this browser session. See Global Constraints —
// in-memory only, no persistence across page loads.
const shardCache = new Map()
function shardKey(model, year, state) {
return `${model}__${year}__${state}`
}
function ensureShardRegistered(db, model, year, state) {
const key = shardKey(model, year, state)
if (!shardCache.has(key)) {
shardCache.set(
key,
(async () => {
try {
const url = `${HF_BASE}/model_id=${model}/YEAR=${year}/LEA_STATE=${state}/data_0.parquet`
const res = await fetch(url)
if (!res.ok) throw new Error(`Failed to fetch draw shard: HTTP ${res.status}`)
const buffer = new Uint8Array(await res.arrayBuffer())
const fileName = `${key}.parquet`
await db.registerFileBuffer(fileName, buffer)
return fileName
} catch (err) {
// Don't let a transient failure (network blip, HF outage) poison the
// cache forever — remove the rejected entry so the next caller for
// this shard gets a fresh attempt instead of the same dead promise.
shardCache.delete(key)
throw err
}
})(),
)
}
return shardCache.get(key)
}
/**
* Fetches real posterior draws for one district/model/year from the Hugging
* Face parquet dataset via duckdb-wasm, converts predicted counts to
* rate-per-1,000 using each group's stu_enroll (not present in the draws
* table itself — joined here from data this app already has), and returns
* them keyed by "RACE_SEX" (see `groupKey` in utils/drawGroups.js).
*
* `status` is an ANY-group signal: 'ready' means at least one group came back
* with real draws, not that every requested group did. Groups can be missing
* individually (falsy enrollment, or absent from the parquet shard), so a caller
* that wants to claim "these are all real draws" must check its own rendered
* groups against `drawsByGroup` — use `hasDrawsForAll` from utils/drawGroups.js.
*
* @param {{leaid: string, state: string, model: string, year: string,
* groups: Array<{race: string, sex: string, stuEnroll: number}>}} params
* @returns {{status: 'loading'|'ready'|'error', drawsByGroup: Record<string, number[]> | null}}
*/
export function useDrawDistribution({ leaid, state, model, year, groups }) {
const [status, setStatus] = useState('loading')
const [drawsByGroup, setDrawsByGroup] = useState(null)
// groups is typically a fresh array literal every render; derive a stable
// primitive so the effect only re-runs when its actual content changes.
const groupsSignature = (groups || []).map((g) => `${g.race}:${g.sex}:${g.stuEnroll}`).join(',')
useEffect(() => {
let cancelled = false
// Reset BEFORE the input guard below, not after. Clearing any previous
// model/district's draws has to happen on every input change, including the
// ones that have nothing to fetch. Without this ordering, a chart that
// varies `model` across renders (e.g. RateDensityRidgeline's dropdown) and
// lands on a model whose `groups` is empty (that model's upstream fetch
// failed) would keep reporting 'ready' and keep handing back the *previous*
// model's real draws — keyed by the same RACE_SEX strings — under the newly
// selected model's summary stats, silently mixing two models' data.
setStatus('loading')
setDrawsByGroup(null)
// Nothing to fetch: stay in 'loading' with no draws, which every consumer
// already treats as "fall back to the approximation". No cleanup needed —
// nothing async was started.
if (!leaid || !state || !model || !year || !groups?.length) return
async function run() {
let conn
try {
const db = await getDb()
const fileName = await ensureShardRegistered(db, model, year, state)
conn = await db.connect()
const stmt = await conn.prepare(`SELECT RACE, SEX, pred FROM read_parquet('${fileName}') WHERE LEAID = ?`)
const table = await stmt.query(leaid)
await stmt.close()
const rows = table.toArray().map((r) => r.toJSON())
const enrollByGroup = {}
for (const g of groups) enrollByGroup[groupKey(g.race, g.sex)] = g.stuEnroll || 0
const byGroup = {}
for (const row of rows) {
const key = groupKey(row.RACE, row.SEX)
const enroll = enrollByGroup[key]
if (!enroll) continue
const rate = (Number(row.pred) / enroll) * 1000
;(byGroup[key] ??= []).push(rate)
}
// A successful query with zero matching rows (this district isn't in
// the draws shard, or none of its rows matched a known group) is not
// "ready" — there's no real data to show, so treat it like a failure
// and let the caller fall back, rather than silently claiming real
// draws while every group actually uses the fitted approximation.
const hasDraws = Object.values(byGroup).some((draws) => draws.length > 0)
if (!cancelled) {
if (hasDraws) {
setDrawsByGroup(byGroup)
setStatus('ready')
} else {
console.warn('useDrawDistribution: query succeeded but returned no usable draws for', { leaid, model, year, state })
setDrawsByGroup(null)
setStatus('error')
}
}
} catch (err) {
console.error('useDrawDistribution failed:', err)
if (!cancelled) {
// Belt-and-suspenders alongside the setDrawsByGroup(null) at the
// top of this effect: a failed fetch/query must never leave a
// *previous* model's real draws in place under the newly-selected
// model's status/summary stats.
setDrawsByGroup(null)
setStatus('error')
}
} finally {
if (conn) await conn.close()
}
}
run()
return () => {
cancelled = true
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [leaid, state, model, year, groupsSignature])
return { status, drawsByGroup }
}
+33
View File
@@ -0,0 +1,33 @@
/**
* Shared key format and coverage check for the per-group posterior draws
* returned by `useDrawDistribution`. Both the hook (which builds the map) and
* the charts (which read it, and decide whether to claim "real draws") go
* through here so the key format lives in exactly one place.
*/
/** Canonical key for one race×sex group in a `drawsByGroup` map. */
export function groupKey(race, sex) {
return `${race}_${sex}`
}
/**
* True only when *every* group in `groups` has a non-empty draw array.
*
* `useDrawDistribution`'s `status` is an any-group signal: it reports 'ready'
* as soon as one group has real draws. Charts fall back per-group, so the
* chart-wide "these are real posterior draws" note/caption must be gated on
* complete coverage instead — a group can be missing because its enrollment is
* falsy or because its (LEAID, RACE, SEX) isn't in the parquet shard.
*
* Returns false for an empty group list (nothing rendered means nothing to
* claim) and for a null map (loading, or the fetch failed outright).
*
* @param {Record<string, number[]> | null | undefined} drawsByGroup
* @param {Array<{race: string, sex: string}>} groups - the groups a chart is
* actually rendering, not everything the API returned.
* @returns {boolean}
*/
export function hasDrawsForAll(drawsByGroup, groups) {
if (!drawsByGroup || !groups?.length) return false
return groups.every((g) => (drawsByGroup[groupKey(g.race, g.sex)]?.length ?? 0) > 0)
}
+41
View File
@@ -0,0 +1,41 @@
import { test } from 'node:test'
import assert from 'node:assert/strict'
import { groupKey, hasDrawsForAll } from './drawGroups.js'
test('groupKey: joins race and sex with an underscore', () => {
assert.equal(groupKey('BL', 'F'), 'BL_F')
})
test('hasDrawsForAll: true when every rendered group has draws', () => {
const map = { WH_F: [1, 2], BL_F: [3] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), true)
})
test('hasDrawsForAll: false when one rendered group is missing', () => {
// The any-group 'ready' status would still be true here — this is exactly the
// case where the chart must keep showing the approximation note.
const map = { WH_F: [1, 2] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), false)
})
test('hasDrawsForAll: false when a rendered group has an empty draw array', () => {
const map = { WH_F: [1, 2], BL_F: [] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), false)
})
test('hasDrawsForAll: false for a null map (loading or failed fetch)', () => {
assert.equal(hasDrawsForAll(null, [{ race: 'WH', sex: 'F' }]), false)
assert.equal(hasDrawsForAll(undefined, [{ race: 'WH', sex: 'F' }]), false)
})
test('hasDrawsForAll: false for an empty or missing group list', () => {
assert.equal(hasDrawsForAll({ WH_F: [1] }, []), false)
assert.equal(hasDrawsForAll({ WH_F: [1] }, undefined), false)
})
test('hasDrawsForAll: ignores groups the chart is not rendering', () => {
// Extra keys in the map (e.g. a race outside RACE_ORDER) must not block the
// claim for the groups actually on screen.
const map = { WH_F: [1], BL_F: [2], AS_F: [3] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), true)
})
+26
View File
@@ -0,0 +1,26 @@
/**
* Lazy-initialized singleton AsyncDuckDB instance running the single-threaded
* MVP wasm bundle only — never eh/coi, which need Cross-Origin-Opener-Policy /
* Cross-Origin-Embedder-Policy response headers this static host doesn't send.
* See docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md.
*/
let dbPromise = null
/** @returns {Promise<import('@duckdb/duckdb-wasm').AsyncDuckDB>} */
export function getDb() {
if (!dbPromise) dbPromise = initDb()
return dbPromise
}
async function initDb() {
const duckdb = await import('@duckdb/duckdb-wasm')
const mvpWorkerUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js?url')).default
const mvpWasmUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm?url')).default
const worker = new Worker(mvpWorkerUrl)
const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING)
const db = new duckdb.AsyncDuckDB(logger, worker)
await db.instantiate(mvpWasmUrl, null)
return db
}
+110
View File
@@ -0,0 +1,110 @@
/**
* Empirical density utilities for posterior draw arrays — Gaussian KDE with
* Silverman's rule-of-thumb bandwidth, plus a linear-interpolated quantile.
* Used in place of distributionApprox.js's analytic fitSkewedInterval/
* densityCurve approximation whenever real posterior draws are available.
*/
const SQRT_2PI = Math.sqrt(2 * Math.PI)
// Absolute last-resort floor, used only when no plotting domain is known.
const MIN_BANDWIDTH = 1e-3
// Bandwidth is clamped relative to the plotting domain, not to absolute units,
// because these draws are rate-per-1,000 values whose scale varies by orders of
// magnitude between districts. domainWidth/50 is slightly wider than one render
// step at the charts' n = 60 (step = domainWidth/59), so a degenerate draw set
// (e.g. 500 identical zeros, common for small districts) resolves as a narrow
// bump instead of a delta spike that flattens every other ridge sharing the
// column's maxPdf. It is also loose enough not to bind on an ordinary posterior:
// a spread wider than ~8% of the domain keeps its own Silverman bandwidth.
// domainWidth/6 stops a handful of extreme draws from inflating sd until the
// curve is a flat line.
const BANDWIDTH_FLOOR_DIVISOR = 50
const BANDWIDTH_CEILING_DIVISOR = 6
/**
* Linear-interpolated quantile (R type-7). Does not mutate `draws`.
* @param {number[]} draws
* @param {number} p - probability in [0, 1]
* @returns {number}
*/
export function quantile(draws, p) {
const sorted = [...draws].sort((a, b) => a - b)
const idx = p * (sorted.length - 1)
const lo = Math.floor(idx)
const hi = Math.ceil(idx)
if (lo === hi) return sorted[lo]
const frac = idx - lo
return sorted[lo] * (1 - frac) + sorted[hi] * frac
}
function standardDeviation(draws) {
const n = draws.length
const mean = draws.reduce((sum, d) => sum + d, 0) / n
const variance = draws.reduce((sum, d) => sum + (d - mean) ** 2, 0) / (n - 1)
return Math.sqrt(variance)
}
/**
* Silverman's rule-of-thumb bandwidth (robust variant using the smallest
* *positive* spread estimate among sd and IQR/1.34), clamped to a fraction of
* the plotting domain.
*
* Both ends of the clamp matter for the zero-inflated count posteriors small
* districts produce. Without the floor, an all-identical draw set (sd = IQR = 0)
* collapses to a delta-function spike. Without the ceiling, a group whose IQR is
* 0 (the normal case when most draws are 0) falls back to raw sd, which a
* handful of extreme draws inflates until the ridge is a featureless flat line.
*
* @param {number[]} draws
* @param {number} [domainWidth] - width of the x-range the curve will be drawn
* over. Omit only when no domain is known; the clamp then degrades to the
* absolute MIN_BANDWIDTH floor.
* @returns {number}
*/
export function silvermanBandwidth(draws, domainWidth = 0) {
const width = domainWidth > 0 ? domainWidth : 0
const floor = width ? width / BANDWIDTH_FLOOR_DIVISOR : MIN_BANDWIDTH
const ceiling = width ? width / BANDWIDTH_CEILING_DIVISOR : Infinity
const n = draws.length
if (n < 2) return floor
const sd = standardDeviation(draws)
const iqr = quantile(draws, 0.75) - quantile(draws, 0.25)
// Degrade gracefully: keep the robust rule when IQR is informative, use sd
// when it isn't, and let the floor handle a fully degenerate draw set —
// rather than treating a zero spread as "no estimate available".
const candidates = [sd, iqr / 1.34].filter((v) => v > 0)
const spread = candidates.length ? Math.min(...candidates) : 0
const raw = 0.9 * spread * Math.pow(n, -0.2)
return Math.min(Math.max(raw, floor), ceiling)
}
/**
* n evenly spaced {x, y} points of a Gaussian KDE over `draws` — same shape
* contract as distributionApprox.js's densityCurve, so chart code can switch
* between the two without changing its rendering path.
* @param {number[]} draws
* @param {{min?: number, max?: number, n?: number}} [options]
* @returns {Array<{x: number, y: number}>}
*/
export function kdeCurve(draws, { min = 0, max, n = 60 } = {}) {
const hi = max ?? Math.max(...draws) * 1.1
// Guarded against a degenerate/inverted domain so the bandwidth clamp can
// never be handed a negative width.
const domainWidth = Math.max(hi - min, 0)
const h = silvermanBandwidth(draws, domainWidth)
const step = (hi - min) / (n - 1)
const points = []
for (let i = 0; i < n; i++) {
const x = min + step * i
let sum = 0
for (const d of draws) {
const z = (x - d) / h
sum += Math.exp(-0.5 * z * z) / SQRT_2PI
}
points.push({ x, y: sum / (draws.length * h) })
}
return points
}
+107
View File
@@ -0,0 +1,107 @@
import { test } from 'node:test'
import assert from 'node:assert/strict'
import { quantile, silvermanBandwidth, kdeCurve } from './kde.js'
test('quantile: median of an odd-length array', () => {
assert.equal(quantile([3, 1, 2], 0.5), 2)
})
test('quantile: linear interpolation between two ranks', () => {
// sorted: [10, 20, 30, 40] — p=0.25 -> index 0.75 -> interpolate 10..20
assert.equal(quantile([40, 10, 30, 20], 0.25), 17.5)
})
test('quantile: does not mutate its input array', () => {
const input = [5, 3, 4, 1, 2]
quantile(input, 0.5)
assert.deepEqual(input, [5, 3, 4, 1, 2])
})
test('silvermanBandwidth: positive, finite floor for identical draws', () => {
const h = silvermanBandwidth([7, 7, 7, 7, 7], 10)
assert.ok(h > 0 && Number.isFinite(h))
assert.equal(h, 10 / 50, 'degenerate spread falls back to the domain-relative floor')
})
test('silvermanBandwidth: positive, finite floor for a single draw', () => {
const h = silvermanBandwidth([7], 10)
assert.ok(h > 0 && Number.isFinite(h))
assert.equal(h, 10 / 50)
})
test('silvermanBandwidth: positive, finite floor when no domain is supplied', () => {
assert.ok(silvermanBandwidth([7, 7, 7, 7, 7]) > 0)
assert.ok(silvermanBandwidth([7]) > 0)
})
test('silvermanBandwidth: clamped to [domainWidth/50, domainWidth/6]', () => {
const domainWidth = 30
// sd is inflated by 50 extreme draws; without the ceiling this is ~15.6.
const outlierDraws = [...Array(450).fill(0), ...Array(50).fill(200)]
assert.equal(silvermanBandwidth(outlierDraws, domainWidth), domainWidth / 6)
// Fully degenerate: sd = IQR = 0.
assert.equal(silvermanBandwidth(Array(500).fill(0), domainWidth), domainWidth / 50)
})
test('silvermanBandwidth: leaves an ordinary spread untouched by the clamp', () => {
const draws = Array.from({ length: 500 }, (_, i) => 3 + Math.sin(i) * 1.2 + (i % 7) * 0.15)
const h = silvermanBandwidth(draws, 10)
assert.ok(h > 10 / 50 && h < 10 / 6, `expected an unclamped bandwidth, got ${h}`)
})
test('kdeCurve: all-identical draws do not produce a delta-function spike', () => {
// 500 identical zeros is the common case for a small district's rare-event
// count posterior. The old absolute 1e-3 floor gave maxY ~399 here (peak
// ~3989x a uniform density over the same domain), which flattened every other
// ridge sharing the column's maxPdf to sub-pixel height.
const domainWidth = 10
const curve = kdeCurve(Array(500).fill(0), { min: 0, max: domainWidth, n: 60 })
const maxY = Math.max(...curve.map((p) => p.y))
assert.ok(Number.isFinite(maxY) && maxY > 0)
assert.ok(
maxY * domainWidth < 25,
`peak density should stay within ~25x a uniform density over the domain, got ${maxY * domainWidth}x`,
)
})
test('kdeCurve: zero-inflated draws keep their shape (not over-smoothed to flat)', () => {
const domainWidth = 10
const draws = [...Array(450).fill(0), ...Array(50).fill(1)]
const curve = kdeCurve(draws, { min: 0, max: domainWidth, n: 60 })
const ys = curve.map((p) => p.y)
const maxY = Math.max(...ys)
const meanY = ys.reduce((sum, y) => sum + y, 0) / ys.length
assert.ok(maxY / meanY > 3, `expected a peaked curve, got peak/mean ${maxY / meanY}`)
})
test('kdeCurve: outlier-inflated sd does not flatten the curve', () => {
// IQR is 0 here (most draws are 0), so the rule falls back to sd — which these
// 50 extreme draws inflate to ~60. Unclamped that gives h ~15.6 on a domain of
// 30, i.e. peak/mean ~1.6: a near-flat line claiming maximal uncertainty.
const domainWidth = 30
const draws = [...Array(450).fill(0), ...Array(50).fill(200)]
const curve = kdeCurve(draws, { min: 0, max: domainWidth, n: 60 })
const ys = curve.map((p) => p.y)
const maxY = Math.max(...ys)
const meanY = ys.reduce((sum, y) => sum + y, 0) / ys.length
assert.ok(maxY / meanY > 3, `expected a peaked curve, got peak/mean ${maxY / meanY}`)
})
test('kdeCurve: returns n points spanning [min, max]', () => {
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
const curve = kdeCurve(draws, { min: 0, max: 10, n: 60 })
assert.equal(curve.length, 60)
assert.equal(curve[0].x, 0)
assert.ok(Math.abs(curve[curve.length - 1].x - 10) < 1e-9)
})
test('kdeCurve: density integrates to ~1 over a wide domain (trapezoidal check)', () => {
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
const curve = kdeCurve(draws, { min: -20, max: 30, n: 2000 })
let area = 0
for (let i = 1; i < curve.length; i++) {
const dx = curve[i].x - curve[i - 1].x
area += (dx * (curve[i].y + curve[i - 1].y)) / 2
}
assert.ok(Math.abs(area - 1) < 0.01, `expected area ~1, got ${area}`)
})
+6
View File
@@ -8,4 +8,10 @@ export default defineConfig({
server: { port: 5173 },
build: { outDir: 'dist' },
base: '/crdc-demo/', // Required for subdirectory deployment on git-pages
optimizeDeps: {
// duckdb-wasm ships its own worker + wasm binaries resolved via `?url`
// imports; esbuild's dev-server pre-bundling can rewrite those import
// paths and break worker instantiation. Exclude it from pre-bundling.
exclude: ['@duckdb/duckdb-wasm'],
},
})