Compare commits
15
Commits
466ebdb9a6
...
db5111dab9
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
db5111dab9
|
||
|
|
096455944e
|
||
|
|
420fc41571
|
||
|
|
70a12cad22
|
||
|
|
138a083c6f
|
||
|
|
b5ebf6bec2
|
||
|
|
523c21d78c
|
||
|
|
20c08ae198
|
||
|
|
b5941a78d1
|
||
|
|
ea6d554e3d
|
||
|
|
d5bf18482a
|
||
|
|
91e21feb74
|
||
|
|
88b09a5d2e
|
||
|
|
991a6912f8
|
||
|
|
2cb04ad9c1
|
@@ -46,11 +46,14 @@ App.jsx (router)
|
||||
|
||||
### Error Bar Convention: 90% Intervals
|
||||
|
||||
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**. When converting HPD bounds to standard deviations for synthetic draw generation (used in Chart 5's ridgelines), the divisor used is **3.29** (corresponding to z = 1.645 for a two-tailed 90% interval).
|
||||
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**.
|
||||
|
||||
The 90% convention is baked into the analytic fallback in `src/utils/distributionApprox.js` (the path used when real posterior draws can't be fetched — see the duckdb-wasm section below). That fallback generates **no draws**: `fitSkewedInterval` fits a two-piece normal directly from `{median, lower, upper}`, converting each half-interval to its own sigma with `z = probit((1 + intervalMass) / 2)` and `intervalMass` defaulting to `0.90` (so z ≈ 1.645). For a symmetric interval that is equivalent to the old fixed "full width / 3.29" divisor, but it is computed from `intervalMass` rather than hardcoded, and each side gets its own sigma so the fitted shape stays skewed.
|
||||
|
||||
If you change this convention, update:
|
||||
- `RateDensityRidgeline.jsx` — SD calculation (`intervalWidth / 3.29`) and label text
|
||||
- `ModelDrawsComparison.jsx` — Legend labels mentioning "90%"
|
||||
- `distributionApprox.js` — the `intervalMass = 0.90` default in `fitSkewedInterval`, and its JSDoc claim that the API's bounds are a 90% interval
|
||||
- `RateByGroupBar.jsx` — the caption and component JSDoc, both of which say "the model's reported 90% interval"
|
||||
- `ArrestsOverTime.jsx` — the legend label "Modeled (median + 90% interval)"
|
||||
- Any documentation referencing confidence/credible intervals
|
||||
|
||||
## Common Pitfalls & Gotchas
|
||||
@@ -83,11 +86,26 @@ Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for
|
||||
|
||||
Not all endpoints are available to browser-based clients:
|
||||
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
|
||||
- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser
|
||||
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
|
||||
|
||||
If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds.
|
||||
### 5. Real posterior draws via duckdb-wasm
|
||||
|
||||
### 5. CORS Configuration
|
||||
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
|
||||
500-draw-per-group posterior from the public Hugging Face parquet dataset
|
||||
(`civilytics/crdc-school-arrest-rates`), queried client-side with
|
||||
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
|
||||
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
|
||||
involved. If that fetch fails (network, unsupported browser, HF outage),
|
||||
both charts fall back to the `distributionApprox.js` analytic approximation
|
||||
and show the "estimated shape" note — **do not delete `distributionApprox.js`
|
||||
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
|
||||
|
||||
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
|
||||
the full design.
|
||||
|
||||
The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.
|
||||
|
||||
### 6. CORS Configuration
|
||||
|
||||
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
|
||||
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PRO
|
||||
## What Needs to Be Done Next
|
||||
|
||||
### Future Enhancements
|
||||
- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required.
|
||||
- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`.
|
||||
- **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling.
|
||||
|
||||
## Key Files
|
||||
|
||||
@@ -18,22 +18,19 @@ npm run preview # serve built files locally
|
||||
|
||||
## What It Does
|
||||
|
||||
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 6 charts comparing observed data against Bayesian model estimates:
|
||||
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 3 charts comparing observed data against Bayesian model estimates:
|
||||
|
||||
1. **Arrests over time** (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG
|
||||
2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG
|
||||
3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG
|
||||
4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate)
|
||||
5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates
|
||||
6. **Exceedance probability** (Chart 6) — P(district > national) per student group
|
||||
2. **Rate by student group** (Chart 2) — box-and-whisker per race×sex group for the most recent year, split into Female/Male panels. The box is the 25th–75th percentile of that group's real 500-draw posterior (same duckdb-wasm fetch as Chart 3), the whisker is the model's reported 90% interval, and a diamond marks the observed rate; individual boxes fall back to an analytic approximation when a group's draws can't be fetched.
|
||||
3. **Predicted rates by student group** (Chart 3) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with a model-selector dropdown for the four Bayesian specifications and diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Tech Stack
|
||||
- **React 19** + **Vite** (static site generation, no backend required)
|
||||
- Plain CSS custom properties for styling (matches Civilytics design tokens exactly)
|
||||
- D3.js v7 for data-driven visualizations (density ridges, scales, axes)
|
||||
- Inline SVG rendering — charts are built with vanilla DOM/D3, not charting libraries
|
||||
- Inline SVG rendering with hand-rolled scales — no charting library, no D3 dependency
|
||||
- Embeds **DuckDB-Wasm** (`@duckdb/duckdb-wasm`, ~39MB uncompressed / ~8.8MB gzipped, loaded on demand only after a district is selected) to query real posterior draws client-side from a public Hugging Face Parquet dataset
|
||||
- Calls the public read-only API directly from the browser
|
||||
|
||||
### File Structure
|
||||
@@ -41,26 +38,32 @@ Visitors select a U.S. state, search for a school district (with suggestions of
|
||||
crdc-demo/
|
||||
├── index.html # Entry point
|
||||
├── vite.config.mjs # Vite build config
|
||||
├── public/ # Static assets (wordmark, favicon)
|
||||
│ └── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
|
||||
├── public/ # Static assets (wordmark, favicon, fixtures)
|
||||
│ ├── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
|
||||
│ └── data/national_rates.json # Static national rates fixture for comparisons
|
||||
├── src/
|
||||
│ ├── main.jsx # React entry
|
||||
│ ├── App.jsx # Main router (state → search → loading → charts)
|
||||
│ ├── hooks/useApi.js # API client with retry/backoff + endpoint wrappers
|
||||
│ ├── hooks/
|
||||
│ │ ├── useApi.js # API client with retry/backoff + endpoint wrappers
|
||||
│ │ └── useDrawDistribution.js # Fetches real posterior draws (duckdb-wasm + HF parquet)
|
||||
│ ├── components/
|
||||
│ │ ├── StateSelector.jsx # Landing screen — state dropdown/grid
|
||||
│ │ ├── DistrictSearch.jsx # Search + "interesting" district suggestions
|
||||
│ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch
|
||||
│ │ └── ChartPanel.jsx # Orchestrates all 6 charts + data fetching
|
||||
│ │ ├── ChartLegend.jsx # Shared legend row
|
||||
│ │ ├── ApproxNote.jsx # "shape estimated from interval bounds" caption
|
||||
│ │ └── ChartPanel.jsx # Orchestrates all 3 charts + data fetching
|
||||
│ ├── charts/
|
||||
│ │ ├── ArrestsOverTime.jsx # Chart 1 — line chart by wave (SVG)
|
||||
│ │ ├── RateByGroupBar.jsx # Chart 2 — bar chart by group (SVG)
|
||||
│ │ ├── DistrictVsNational.jsx # Chart 3 — comparison vs. national avg
|
||||
│ │ ├── ModelDrawsComparison.jsx # Chart 4 — quadrant model comparison (D3)
|
||||
│ │ ├── RateDensityRidgeline.jsx # Chart 5 — density ridges per student group (D3)
|
||||
│ │ └── ExceedanceProbability.jsx # Chart 6 — P(district > nat) per group
|
||||
│ ├── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
|
||||
│ └── data/national_rates.json # Static national rates fixture for comparisons
|
||||
│ │ ├── ArrestsOverTime.jsx # Chart 1 — counts by wave (SVG)
|
||||
│ │ ├── RateByGroupBar.jsx # Chart 2 — box-and-whisker by group (SVG)
|
||||
│ │ └── RateDensityRidgeline.jsx # Chart 3 — density ridges per student group (SVG)
|
||||
│ ├── utils/
|
||||
│ │ ├── duckdbClient.js # Lazy duckdb-wasm bundle loader (dynamic import)
|
||||
│ │ ├── kde.js # Empirical density from real draws (+ kde.test.js)
|
||||
│ │ ├── drawGroups.js # Draw-map key format + coverage check (+ .test.js)
|
||||
│ │ └── distributionApprox.js # Analytic fallback when draws are unavailable
|
||||
│ └── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
|
||||
├── Dockerfile # Multi-stage build → nginx static server
|
||||
├── docker-compose.yml # Local dev / self-hosted deployment
|
||||
└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch
|
||||
@@ -72,7 +75,7 @@ crdc-demo/
|
||||
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
|
||||
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
|
||||
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district |
|
||||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app |
|
||||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
|
||||
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
|
||||
|
||||
## Deployment
|
||||
@@ -114,7 +117,7 @@ docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo:
|
||||
docker push registry.civilytics.org/crdc-demo:latest
|
||||
```
|
||||
|
||||
The container is ~5MB (nginx Alpine) and serves static files with immutable cache headers. A `/healthz` endpoint supports container orchestration health checks.
|
||||
The container is nginx Alpine (~5MB) plus the built site, which is dominated by the ~39MB DuckDB-Wasm engine. `nginx.conf` serves that engine gzipped (~8.8MB on the wire) with immutable cache headers, so it is fetched once per browser. A `/healthz` endpoint supports container orchestration health checks.
|
||||
|
||||
### Environment Variables
|
||||
| Variable | Default | Description |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,187 @@
|
||||
# Empirical draw distributions via DuckDB-Wasm — Design Spec
|
||||
|
||||
**Date:** 2026-08-11
|
||||
**Status:** Draft for review
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose & context
|
||||
|
||||
Two of this app's three charts currently show a *modeled* distribution shape that is
|
||||
not the real posterior — `src/utils/distributionApprox.js` fits a two-piece-normal
|
||||
curve to each group's `(median, lower, upper)` summary stats returned by the
|
||||
`/estimates` API, because the API's raw posterior draws are only available in bulk
|
||||
as Hive-partitioned Parquet on Hugging Face
|
||||
(`civilytics/crdc-school-arrest-rates`), meant for DuckDB/bulk consumption, not
|
||||
browser fetches (see `AGENTS.md` §"API Endpoint Availability" and
|
||||
`HANDOFF.md` §"Future Enhancements").
|
||||
|
||||
This spec replaces the approximation with the **real** empirical draws (500 per
|
||||
group), fetched client-side using `@duckdb/duckdb-wasm` to query the actual Parquet
|
||||
shard for the district's state directly from Hugging Face — no new backend
|
||||
endpoint, no change to the existing summary API calls.
|
||||
|
||||
**Verified feasibility (2026-08-11):**
|
||||
- The HF dataset is public, non-gated. Each `(model_id, YEAR, LEA_STATE)` partition
|
||||
is a single file (`data_0.parquet`).
|
||||
- File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed
|
||||
by resolving the `resolve/main/...` redirect to the actual CDN blob.
|
||||
- The redirect target sends `access-control-allow-origin: *` and
|
||||
`accept-ranges: bytes` — browser `fetch()` works directly, no proxy needed.
|
||||
- Schema (from `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`):
|
||||
`LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred`, sorted
|
||||
within each shard by `(LEAID, RACE, SEX)`. `LEA_STATE`/`YEAR`/`model_id` are
|
||||
Hive-partition columns (encoded in the path, not repeated in every row).
|
||||
`stu_enroll` is **not** in the draws table — it's already available in this app
|
||||
from the existing `/estimates` summary call.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
```
|
||||
district selected (leaid, state)
|
||||
│
|
||||
▼
|
||||
resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state)
|
||||
e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/
|
||||
resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet
|
||||
│
|
||||
▼
|
||||
fetch() the shard (native fetch, follows the HF→CDN redirect automatically)
|
||||
│
|
||||
▼
|
||||
duckdb-wasm: registerFileBuffer + query
|
||||
SELECT RACE, SEX, pred FROM shard WHERE LEAID = '<leaid>'
|
||||
│
|
||||
▼
|
||||
join `pred` (posterior count draws) against stu_enroll already in app state
|
||||
(from the existing /estimates summary call) → rate-per-1000 draws per group
|
||||
│
|
||||
▼
|
||||
KDE per race×sex group → smooth density curve, same shape the charts draw today
|
||||
```
|
||||
|
||||
Given verified shard sizes (≤6.4MB), the design fetches the **whole shard** with a
|
||||
plain `fetch()` and queries it in-memory via duckdb-wasm, rather than relying on
|
||||
fine-grained HTTP range / row-group pruning. This is simpler and more robust than
|
||||
depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN
|
||||
redirect chain under partial-range requests — an unverified behavior — for a
|
||||
saving that wouldn't matter at these file sizes.
|
||||
|
||||
`@duckdb/duckdb-wasm` is MIT-licensed and runs entirely client-side in a Web
|
||||
Worker. It introduces no new server dependency and no new hosted service beyond
|
||||
the Hugging Face dataset the `crdc-arrests` project's `/draws` endpoint already
|
||||
points to (per `2026-05-30-draws-api-design.md`, decision #5) — this spec doesn't
|
||||
introduce that dependency, it makes the demo app actually use data that was
|
||||
already published there for exactly this purpose.
|
||||
|
||||
**Deployment risk:** duckdb-wasm's threaded ("eh") bundle requires
|
||||
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` response headers
|
||||
(for `SharedArrayBuffer`), which the git-pages static host does not send today.
|
||||
This design uses the **single-threaded ("mvp") bundle** instead — at these file
|
||||
sizes threading has no meaningful benefit, and it avoids needing new headers on
|
||||
both the git-pages and Docker/nginx deploy paths.
|
||||
|
||||
---
|
||||
|
||||
## 3. File-level changes
|
||||
|
||||
### New files
|
||||
- **`src/utils/duckdbClient.js`** — lazy-initialized singleton. Dynamic-imports
|
||||
`@duckdb/duckdb-wasm`, selects the MVP (non-threaded) bundle, starts the worker
|
||||
once. Dynamic `import()` keeps the ~3–5MB wasm payload out of the main bundle;
|
||||
it only loads when a chart actually needs draws.
|
||||
- **`src/utils/kde.js`** — Gaussian KDE over an array of numbers (Silverman
|
||||
bandwidth). Takes over the role `distributionApprox.js`'s `densityCurve` plays
|
||||
today, fed real empirical draws instead of a parametric fit.
|
||||
- **`src/hooks/useDrawDistribution.js`** — given
|
||||
`{ leaid, state, model, year, groups }` (groups = race/sex + `stu_enroll` already
|
||||
in app state), resolves the HF URL, fetches, registers the buffer with
|
||||
duckdb-wasm, runs the query, joins enrollment, and returns
|
||||
`{ status: 'loading' | 'ready' | 'error', drawsByGroup }`. Owns an in-memory
|
||||
`Map` cache keyed by `model+state+year` so re-selecting a model in Chart 3's
|
||||
dropdown, or viewing another district in the same state, reuses the shard
|
||||
already fetched.
|
||||
|
||||
### Modified files
|
||||
- **`RateDensityRidgeline.jsx`** — on model-dropdown change, calls
|
||||
`useDrawDistribution` for the selected model; replaces
|
||||
`fitSkewedInterval`/`densityCurve` with the hook's real draws → `kde.js`. Shows
|
||||
an inline spinner in the ridge area while that model's shard is loading (Charts
|
||||
1–2 aren't blocked).
|
||||
- **`RateByGroupBar.jsx`** — fetches draws for `unified_m3_mod` (the one model
|
||||
this chart uses) alongside its existing data fetch. `q1`/`q3` become exact
|
||||
empirical quantiles from the 500 real draws — removes this chart's use of
|
||||
`fitSkewedInterval`'s fitted quantile function entirely.
|
||||
- **`ChartPanel.jsx`** — passes `district.leaid`, `state`, and each group's
|
||||
`stu_enroll` (already fetched) down to the two charts above.
|
||||
- **`ApproxNote.jsx`** — becomes conditional: renders the "estimated shape" note
|
||||
only when a chart is in fallback mode; charts backed by real draws show no note
|
||||
(or a neutral "500 posterior draws" caption).
|
||||
- **`package.json` / `vite.config.mjs`** — add `@duckdb/duckdb-wasm`; wasm/worker
|
||||
assets are pulled in via Vite's native `?url` imports, which already respect the
|
||||
`/crdc-demo/` `base` path — no bundler plugin needed.
|
||||
|
||||
### Unchanged
|
||||
- **`ArrestsOverTime.jsx`** — already uses real summary stats (point-range from
|
||||
`/estimates`), no approximation involved; out of scope.
|
||||
- **`useApi.js`** — still the source for medians, intervals, and enrollment.
|
||||
- **`distributionApprox.js`** — kept as the fallback path (see §4).
|
||||
|
||||
---
|
||||
|
||||
## 4. Error handling & caching
|
||||
|
||||
**Fallback:** `distributionApprox.js` is retained. `useDrawDistribution` catches
|
||||
fetch/wasm/query failures and returns `status: 'error'`; both chart components
|
||||
branch on that to render the current analytic-approximation path with
|
||||
`<ApproxNote />` visible. A Hugging Face outage, a network failure, or an
|
||||
unsupported browser degrades to today's behavior rather than breaking the chart.
|
||||
|
||||
**Caching:** in-memory only (a `Map` inside the hook), scoped to the browser
|
||||
session. No IndexedDB/persistent cache in this iteration — a demo session
|
||||
typically covers one or two districts, and shards are cheap enough to refetch on
|
||||
reload.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing
|
||||
|
||||
This project has no automated test suite (per `AGENTS.md`); this follows the
|
||||
existing manual-verification convention:
|
||||
|
||||
1. `npm run dev`; walk a small state (DC or WY, ~130–200KB shard) and a large one
|
||||
(CA or TX, several MB) through the full district-search flow.
|
||||
2. Confirm both charts render from real draws; confirm the Network tab shows the
|
||||
expected parquet fetch(es) and sizes.
|
||||
3. Simulate failure (block the `huggingface.co` / CDN domain in devtools) and
|
||||
confirm both charts fall back to the analytic approximation with the note
|
||||
visible, rather than breaking.
|
||||
4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is
|
||||
instant on re-selection (cache hit).
|
||||
|
||||
---
|
||||
|
||||
## 6. Open risk to de-risk first
|
||||
|
||||
Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query
|
||||
correctly when deployed under the `/crdc-demo/` subpath on git-pages, and does
|
||||
nginx/git-pages serve `.wasm` with a usable content type? Everything else in this
|
||||
design is standard Vite asset handling already exercised elsewhere in the app, but
|
||||
this specific combination (wasm worker + subpath base + static host) hasn't been
|
||||
verified end-to-end and should be checked with a throwaway spike rather than
|
||||
assumed.
|
||||
|
||||
---
|
||||
|
||||
## 7. Explicitly out of scope
|
||||
|
||||
- `ArrestsOverTime.jsx` (Chart 1) — no approximation to replace.
|
||||
- A new server-side `/draws`-streaming API endpoint — explicitly rejected in favor
|
||||
of client-side wasm access, per the brainstorming decision that led to this spec.
|
||||
- Persistent (IndexedDB) caching of fetched shards.
|
||||
- Fine-grained HTTP range / row-group-level partial reads — shard sizes are small
|
||||
enough that whole-file fetch is simpler and sufficiently fast.
|
||||
- Extending empirical draws to national/exceedance-probability views — those
|
||||
charts aren't part of the current 3-chart demo.
|
||||
+22
-2
@@ -37,12 +37,32 @@ server {
|
||||
root /usr/share/nginx/html/crdc-demo/; # Adjust if base is different
|
||||
index index.html;
|
||||
|
||||
# Compress text assets and — most importantly — the duckdb-wasm engine,
|
||||
# which is ~39MB uncompressed and ~8.8MB gzipped. Without this it ships
|
||||
# uncompressed on every cold load. Dynamic gzip rather than gzip_static
|
||||
# because the Vite build emits no pre-compressed .gz files.
|
||||
gzip on;
|
||||
gzip_vary on;
|
||||
gzip_min_length 1024;
|
||||
gzip_proxied any;
|
||||
gzip_comp_level 6;
|
||||
gzip_types
|
||||
text/plain
|
||||
text/css
|
||||
application/javascript
|
||||
text/javascript
|
||||
application/json
|
||||
image/svg+xml
|
||||
application/wasm;
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
|
||||
# Civilytics design tokens: immutable cache headers for static assets
|
||||
location ~* \.(js|css|png|jpg|jpeg|gif|svg|woff2|ttf)$ {
|
||||
# Civilytics design tokens: immutable cache headers for static assets.
|
||||
# `wasm` belongs here too — the duckdb engine is content-hashed by Vite and
|
||||
# is by far the largest asset in the build, so it must not be re-fetched.
|
||||
location ~* \.(js|css|wasm|png|jpg|jpeg|gif|svg|woff2|ttf)$ {
|
||||
expires 1y;
|
||||
add_header Cache-Control "public, max-age=31536000, immutable";
|
||||
try_files $uri =404;
|
||||
|
||||
Generated
+238
@@ -9,6 +9,7 @@
|
||||
"version": "0.1.0",
|
||||
"license": "MIT",
|
||||
"devDependencies": {
|
||||
"@duckdb/duckdb-wasm": "^1.32.0",
|
||||
"@vitejs/plugin-react-swc": "^4.3.3",
|
||||
"eslint": "^8.57.1",
|
||||
"prettier": "^3.9.6",
|
||||
@@ -17,6 +18,16 @@
|
||||
"vite": "^8.2.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@duckdb/duckdb-wasm": {
|
||||
"version": "1.32.0",
|
||||
"resolved": "https://registry.npmjs.org/@duckdb/duckdb-wasm/-/duckdb-wasm-1.32.0.tgz",
|
||||
"integrity": "sha512-IewXTNYEjsZCPE9weUWgtjGxUlMRo7qhX0GF6tq/KjK8bnY+RAl4cyUdYUfcdzbyb4b9ZxPC+FOsCcxgaKFWMg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"apache-arrow": "^17.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@eslint-community/eslint-utils": {
|
||||
"version": "4.10.1",
|
||||
"resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.10.1.tgz",
|
||||
@@ -663,6 +674,16 @@
|
||||
"dev": true,
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@swc/helpers": {
|
||||
"version": "0.5.23",
|
||||
"resolved": "https://registry.npmjs.org/@swc/helpers/-/helpers-0.5.23.tgz",
|
||||
"integrity": "sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"tslib": "^2.8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@swc/types": {
|
||||
"version": "0.1.28",
|
||||
"resolved": "https://registry.npmjs.org/@swc/types/-/types-0.1.28.tgz",
|
||||
@@ -673,6 +694,30 @@
|
||||
"@swc/counter": "^0.1.3"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/command-line-args": {
|
||||
"version": "5.2.3",
|
||||
"resolved": "https://registry.npmjs.org/@types/command-line-args/-/command-line-args-5.2.3.tgz",
|
||||
"integrity": "sha512-uv0aG6R0Y8WHZLTamZwtfsDLVRnOa+n+n5rEvFWL5Na5gZ8V2Teab/duDPFzIIIhs9qizDpcavCusCLJZu62Kw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/command-line-usage": {
|
||||
"version": "5.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@types/command-line-usage/-/command-line-usage-5.0.4.tgz",
|
||||
"integrity": "sha512-BwR5KP3Es/CSht0xqBcUXS3qCAUVXwpRKsV2+arxeb65atasuXG9LykC9Ab10Cw3s2raH92ZqOeILaQbsB2ACg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "20.19.43",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.43.tgz",
|
||||
"integrity": "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~6.21.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@ungap/structured-clone": {
|
||||
"version": "1.3.3",
|
||||
"resolved": "https://registry.npmjs.org/@ungap/structured-clone/-/structured-clone-1.3.3.tgz",
|
||||
@@ -763,6 +808,27 @@
|
||||
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/apache-arrow": {
|
||||
"version": "17.0.0",
|
||||
"resolved": "https://registry.npmjs.org/apache-arrow/-/apache-arrow-17.0.0.tgz",
|
||||
"integrity": "sha512-X0p7auzdnGuhYMVKYINdQssS4EcKec9TCXyez/qtJt32DrIMGbzqiaMiQ0X6fQlQpw8Fl0Qygcv4dfRAr5Gu9Q==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@swc/helpers": "^0.5.11",
|
||||
"@types/command-line-args": "^5.2.3",
|
||||
"@types/command-line-usage": "^5.0.4",
|
||||
"@types/node": "^20.13.0",
|
||||
"command-line-args": "^5.2.1",
|
||||
"command-line-usage": "^7.0.1",
|
||||
"flatbuffers": "^24.3.25",
|
||||
"json-bignum": "^0.0.3",
|
||||
"tslib": "^2.6.2"
|
||||
},
|
||||
"bin": {
|
||||
"arrow2csv": "bin/arrow2csv.cjs"
|
||||
}
|
||||
},
|
||||
"node_modules/argparse": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
|
||||
@@ -770,6 +836,16 @@
|
||||
"dev": true,
|
||||
"license": "Python-2.0"
|
||||
},
|
||||
"node_modules/array-back": {
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmjs.org/array-back/-/array-back-3.1.0.tgz",
|
||||
"integrity": "sha512-TkuxA4UCOvxuDK6NZYXCalszEzj+TLszyASooky+i742l9TqsOdYCMJJupxRic61hwquNtppB3hgcuq9SVSH1Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/balanced-match": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
|
||||
@@ -815,6 +891,22 @@
|
||||
"url": "https://github.com/chalk/chalk?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/chalk-template": {
|
||||
"version": "0.4.0",
|
||||
"resolved": "https://registry.npmjs.org/chalk-template/-/chalk-template-0.4.0.tgz",
|
||||
"integrity": "sha512-/ghrgmhfY8RaSdeo43hNXxpoHAtxdbskUHjPpfqUWGttFgycUhYPGx3YZBCnUCvOa7Doivn1IZec3DEGFoMgLg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"chalk": "^4.1.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/chalk/chalk-template?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/color-convert": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
|
||||
@@ -835,6 +927,58 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/command-line-args": {
|
||||
"version": "5.2.1",
|
||||
"resolved": "https://registry.npmjs.org/command-line-args/-/command-line-args-5.2.1.tgz",
|
||||
"integrity": "sha512-H4UfQhZyakIjC74I9d34fGYDwk3XpSr17QhEd0Q3I9Xq1CETHo4Hcuo87WyWHpAF1aSLjLRf5lD9ZGX2qStUvg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"array-back": "^3.1.0",
|
||||
"find-replace": "^3.0.0",
|
||||
"lodash.camelcase": "^4.3.0",
|
||||
"typical": "^4.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=4.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/command-line-usage": {
|
||||
"version": "7.0.4",
|
||||
"resolved": "https://registry.npmjs.org/command-line-usage/-/command-line-usage-7.0.4.tgz",
|
||||
"integrity": "sha512-85UdvzTNx/+s5CkSgBm/0hzP80RFHAa7PsfeADE5ezZF3uHz3/Tqj9gIKGT9PTtpycc3Ua64T0oVulGfKxzfqg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"array-back": "^6.2.2",
|
||||
"chalk-template": "^0.4.0",
|
||||
"table-layout": "^4.1.1",
|
||||
"typical": "^7.3.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12.20.0"
|
||||
}
|
||||
},
|
||||
"node_modules/command-line-usage/node_modules/array-back": {
|
||||
"version": "6.2.3",
|
||||
"resolved": "https://registry.npmjs.org/array-back/-/array-back-6.2.3.tgz",
|
||||
"integrity": "sha512-SGDvmg6QTYiTxCBkYVmThcoa67uLl35pyzRHdpCGBOcqFy6BtwnphoFPk7LhJshD+Yk1Kt35WGWeZPTgwR4Fhw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
}
|
||||
},
|
||||
"node_modules/command-line-usage/node_modules/typical": {
|
||||
"version": "7.3.0",
|
||||
"resolved": "https://registry.npmjs.org/typical/-/typical-7.3.0.tgz",
|
||||
"integrity": "sha512-ya4mg/30vm+DOWfBg4YK3j2WD6TWtRkCbasOJr40CseYENzCUby/7rIvXA99JGsQHeNxLbnXdyLLxKSv3tauFw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
}
|
||||
},
|
||||
"node_modules/concat-map": {
|
||||
"version": "0.0.1",
|
||||
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
|
||||
@@ -1131,6 +1275,19 @@
|
||||
"node": "^10.12.0 || >=12.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/find-replace": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/find-replace/-/find-replace-3.0.0.tgz",
|
||||
"integrity": "sha512-6Tb2myMioCAgv5kfvP5/PkZZ/ntTpVK39fHY7WkWBgvbeE+VHd/tZuZ4mrC+bxh4cfOZeYKVPaJIZtZXV7GNCQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"array-back": "^3.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=4.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/find-up": {
|
||||
"version": "5.0.0",
|
||||
"resolved": "https://registry.npmjs.org/find-up/-/find-up-5.0.0.tgz",
|
||||
@@ -1163,6 +1320,13 @@
|
||||
"node": "^10.12.0 || >=12.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/flatbuffers": {
|
||||
"version": "24.12.23",
|
||||
"resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-24.12.23.tgz",
|
||||
"integrity": "sha512-dLVCAISd5mhls514keQzmEG6QHmUUsNuWsb4tFafIUwvvgDjXhtfAYSKOzt5SWOy+qByV5pbsDZ+Vb7HUOBEdA==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/flatted": {
|
||||
"version": "3.4.4",
|
||||
"resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.4.tgz",
|
||||
@@ -1379,6 +1543,15 @@
|
||||
"js-yaml": "bin/js-yaml.js"
|
||||
}
|
||||
},
|
||||
"node_modules/json-bignum": {
|
||||
"version": "0.0.3",
|
||||
"resolved": "https://registry.npmjs.org/json-bignum/-/json-bignum-0.0.3.tgz",
|
||||
"integrity": "sha512-2WHyXj3OfHSgNyuzDbSxI1w2jgw5gkWSWhS7Qg4bWXx1nLk3jnbwfUeS0PSba3IzpTUWdHxBieELUzXRjQB2zg==",
|
||||
"dev": true,
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/json-buffer": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/json-buffer/-/json-buffer-3.0.1.tgz",
|
||||
@@ -1701,6 +1874,13 @@
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/lodash.camelcase": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/lodash.camelcase/-/lodash.camelcase-4.3.0.tgz",
|
||||
"integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/lodash.merge": {
|
||||
"version": "4.6.2",
|
||||
"resolved": "https://registry.npmjs.org/lodash.merge/-/lodash.merge-4.6.2.tgz",
|
||||
@@ -2160,6 +2340,30 @@
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/table-layout": {
|
||||
"version": "4.1.1",
|
||||
"resolved": "https://registry.npmjs.org/table-layout/-/table-layout-4.1.1.tgz",
|
||||
"integrity": "sha512-iK5/YhZxq5GO5z8wb0bY1317uDF3Zjpha0QFFLA8/trAoiLbQD0HUbMesEaxyzUgDxi2QlcbM8IvqOlEjgoXBA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"array-back": "^6.2.2",
|
||||
"wordwrapjs": "^5.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
}
|
||||
},
|
||||
"node_modules/table-layout/node_modules/array-back": {
|
||||
"version": "6.2.3",
|
||||
"resolved": "https://registry.npmjs.org/array-back/-/array-back-6.2.3.tgz",
|
||||
"integrity": "sha512-SGDvmg6QTYiTxCBkYVmThcoa67uLl35pyzRHdpCGBOcqFy6BtwnphoFPk7LhJshD+Yk1Kt35WGWeZPTgwR4Fhw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
}
|
||||
},
|
||||
"node_modules/text-table": {
|
||||
"version": "0.2.0",
|
||||
"resolved": "https://registry.npmjs.org/text-table/-/text-table-0.2.0.tgz",
|
||||
@@ -2184,6 +2388,13 @@
|
||||
"url": "https://github.com/sponsors/SuperchupuDev"
|
||||
}
|
||||
},
|
||||
"node_modules/tslib": {
|
||||
"version": "2.8.1",
|
||||
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
|
||||
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
|
||||
"dev": true,
|
||||
"license": "0BSD"
|
||||
},
|
||||
"node_modules/type-check": {
|
||||
"version": "0.4.0",
|
||||
"resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz",
|
||||
@@ -2210,6 +2421,23 @@
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/typical": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/typical/-/typical-4.0.0.tgz",
|
||||
"integrity": "sha512-VAH4IvQ7BDFYglMd7BPRDfLgxZZX4O4TFcRDA6EN5X7erNJJq+McIEp8np9aVtxrCJ6qx4GTYVfOWNjcqwZgRw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "6.21.0",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
|
||||
"integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/uri-js": {
|
||||
"version": "4.4.1",
|
||||
"resolved": "https://registry.npmjs.org/uri-js/-/uri-js-4.4.1.tgz",
|
||||
@@ -2324,6 +2552,16 @@
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/wordwrapjs": {
|
||||
"version": "5.1.1",
|
||||
"resolved": "https://registry.npmjs.org/wordwrapjs/-/wordwrapjs-5.1.1.tgz",
|
||||
"integrity": "sha512-0yweIbkINJodk27gX9LBGMzyQdBDan3s/dEAiwBOj+Mf0PPyWL6/rikalkv8EeD0E8jm4o5RXEOrFTP3NXbhJg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
}
|
||||
},
|
||||
"node_modules/wrappy": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview",
|
||||
"test": "node --test 'src/**/*.test.js'",
|
||||
"lint": "eslint src/ --ext .js,.jsx,.ts,.tsx",
|
||||
"format": "prettier --write \"src/**/*.{js,jsx,css}\""
|
||||
},
|
||||
@@ -18,6 +19,7 @@
|
||||
"author": "Civilytics Consulting LLC",
|
||||
"license": "MIT",
|
||||
"devDependencies": {
|
||||
"@duckdb/duckdb-wasm": "^1.32.0",
|
||||
"@vitejs/plugin-react-swc": "^4.3.3",
|
||||
"eslint": "^8.57.1",
|
||||
"prettier": "^3.9.6",
|
||||
|
||||
+16
-2
@@ -6,9 +6,14 @@ import ChartPanel from './components/ChartPanel.jsx'
|
||||
import Footer from './components/Footer.jsx'
|
||||
import { fetchDistrictEstimates } from './hooks/useApi.js'
|
||||
|
||||
// Two-letter USPS state code. The deep-link value flows into API request URLs
|
||||
// and into the Hugging Face parquet shard path duckdb-wasm reads, so it gets
|
||||
// validated here at the boundary rather than propagated verbatim.
|
||||
const STATE_CODE_PATTERN = /^[A-Z]{2}$/
|
||||
|
||||
/**
|
||||
* CRDC Arrests API Demo App — main router.
|
||||
* Flow: state → district search (with interesting suggestions) → loading animation → 6 charts
|
||||
* Flow: state → district search (with interesting suggestions) → loading animation → charts
|
||||
*/
|
||||
export default function App() {
|
||||
const [step, setStep] = useState('state') // 'state' | 'search' | 'loading' | 'results'
|
||||
@@ -19,7 +24,16 @@ export default function App() {
|
||||
useEffect(() => {
|
||||
const params = new URLSearchParams(window.location.search)
|
||||
const leaid = params.get('leaid')
|
||||
const stateParam = params.get('state')
|
||||
const rawState = params.get('state')
|
||||
const stateParam = rawState ? rawState.trim().toUpperCase() : null
|
||||
|
||||
if (rawState && !STATE_CODE_PATTERN.test(stateParam)) {
|
||||
// Malformed deep link — drop it and start at the state selector rather
|
||||
// than passing an arbitrary string into fetch URLs and shard paths.
|
||||
console.warn('Ignoring deep link: `state` is not a two-letter state code.')
|
||||
return
|
||||
}
|
||||
|
||||
if (leaid && stateParam) {
|
||||
// Deep link (including browser back/forward landing on this URL): resolve
|
||||
// the district name via a real estimates row, keyed by LEAID. The previous
|
||||
|
||||
@@ -2,33 +2,52 @@ import ChartLegend from '../components/ChartLegend.jsx'
|
||||
import ApproxNote from '../components/ApproxNote.jsx'
|
||||
import { raceColor, OBSERVED_MARK_COLOR, SHORT_RACE_LABEL } from '../utils/colors.js'
|
||||
import { fitSkewedInterval } from '../utils/distributionApprox.js'
|
||||
import { groupKey, hasDrawsForAll } from '../utils/drawGroups.js'
|
||||
import { quantile } from '../utils/kde.js'
|
||||
import { niceTicks } from '../utils/niceTicks.js'
|
||||
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
|
||||
|
||||
/**
|
||||
* Arrest rate by student group, most recent year, disaggregated into two
|
||||
* panels (Female / Male), each a horizontal box-and-whisker across the 4
|
||||
* race categories. Whisker = the model's reported 90% interval, box = the
|
||||
* fitted approximation's 25th-75th percentile, white tick = median, dark
|
||||
* diamond = observed rate.
|
||||
* race categories. Whisker = the model's reported 90% interval; box = the
|
||||
* 25th-75th percentile of the group's 500 real posterior draws (or, if draws
|
||||
* are unavailable, the fitSkewedInterval approximation); white tick =
|
||||
* median; dark diamond = observed rate.
|
||||
*/
|
||||
|
||||
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
|
||||
const SEX_PANELS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
|
||||
const ROW_HEIGHT = 34
|
||||
const MODEL = 'unified_m3_mod' // matches ChartPanel's WAVE_MODEL, which built `data`
|
||||
|
||||
function buildBox(d) {
|
||||
function buildBox(d, draws) {
|
||||
const median = Math.max(d.modeledMedian || 0, 0)
|
||||
const lower = Math.max(Math.min(d.rateLower ?? median, median), 0)
|
||||
const upper = Math.max(d.rateUpper ?? median, median)
|
||||
const fit = fitSkewedInterval({ median, lower, upper })
|
||||
return {
|
||||
lower, upper, median,
|
||||
q1: Math.max(fit.quantile(0.25), 0),
|
||||
q3: Math.max(fit.quantile(0.75), median),
|
||||
|
||||
if (draws && draws.length > 0) {
|
||||
return { lower, upper, median, q1: Math.max(quantile(draws, 0.25), 0), q3: Math.max(quantile(draws, 0.75), median) }
|
||||
}
|
||||
const fit = fitSkewedInterval({ median, lower, upper })
|
||||
return { lower, upper, median, q1: Math.max(fit.quantile(0.25), 0), q3: Math.max(fit.quantile(0.75), median) }
|
||||
}
|
||||
|
||||
export default function RateByGroupBar({ data }) {
|
||||
export default function RateByGroupBar({ data, leaid, state }) {
|
||||
const groups = data.map((d) => ({ race: d.race, sex: d.sex, stuEnroll: d.enrollment }))
|
||||
const { drawsByGroup } = useDrawDistribution({ leaid, state, model: MODEL, year: '21-22', groups })
|
||||
|
||||
// Only the RACE_ORDER × SEX_PANELS cells are actually drawn, so coverage is
|
||||
// judged against those — not against everything the API returned. The hook's
|
||||
// 'ready' status is an any-group signal and would over-claim here: individual
|
||||
// boxes still fall back to fitSkewedInterval whenever their group is missing
|
||||
// from the shard. A null map (loading, or the whole fetch failed) fails this
|
||||
// check too, so the error path still shows the note.
|
||||
const renderedGroups = data.filter(
|
||||
(d) => RACE_ORDER.includes(d.race) && SEX_PANELS.some((p) => p.sex === d.sex),
|
||||
)
|
||||
const allGroupsHaveDraws = hasDrawsForAll(drawsByGroup, renderedGroups)
|
||||
|
||||
const maxRate = Math.max(
|
||||
...data.map((d) => Math.max(d.observedRate, d.rateUpper ?? d.modeledMedian ?? 0)),
|
||||
0.5
|
||||
@@ -40,11 +59,18 @@ export default function RateByGroupBar({ data }) {
|
||||
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
|
||||
Arrest rate by student group — 2021–22 (per 1,000)
|
||||
</h3>
|
||||
<ApproxNote />
|
||||
{!allGroupsHaveDraws && <ApproxNote />}
|
||||
|
||||
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
|
||||
{SEX_PANELS.map(({ sex, label }) => (
|
||||
<SexPanel key={sex} label={label} rows={data.filter((d) => d.sex === sex)} ticks={ticks} niceMax={niceMax} />
|
||||
<SexPanel
|
||||
key={sex}
|
||||
label={label}
|
||||
rows={data.filter((d) => d.sex === sex)}
|
||||
drawsByGroup={drawsByGroup}
|
||||
ticks={ticks}
|
||||
niceMax={niceMax}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
|
||||
@@ -56,14 +82,16 @@ export default function RateByGroupBar({ data }) {
|
||||
]} />
|
||||
|
||||
<p style={{ fontSize: '0.72rem', color: 'var(--cv-ink-3)', marginTop: 'var(--space-1)' }}>
|
||||
Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported
|
||||
90% interval; white tick = median. The dark diamond is the observed rate.
|
||||
{allGroupsHaveDraws
|
||||
? "Box = 25th–75th percentile of 500 real posterior draws; whisker = the model's reported 90% interval; white tick = median."
|
||||
: "Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported 90% interval; white tick = median."}
|
||||
{' '}The dark diamond is the observed rate.
|
||||
</p>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function SexPanel({ label, rows, ticks, niceMax }) {
|
||||
function SexPanel({ label, rows, drawsByGroup, ticks, niceMax }) {
|
||||
const width = 300
|
||||
const margin = { top: 30, right: 16, bottom: 34, left: 66 }
|
||||
const innerWidth = width - margin.left - margin.right
|
||||
@@ -99,7 +127,8 @@ function SexPanel({ label, rows, ticks, niceMax }) {
|
||||
const boxTop = midY - ROW_HEIGHT * 0.26
|
||||
const boxBottom = midY + ROW_HEIGHT * 0.26
|
||||
const color = raceColor(d.race)
|
||||
const box = buildBox(d)
|
||||
const draws = drawsByGroup?.[groupKey(d.race, d.sex)]
|
||||
const box = buildBox(d, draws)
|
||||
const observedX = xScale(d.observedRate)
|
||||
|
||||
return (
|
||||
|
||||
@@ -2,23 +2,26 @@ import { useState } from 'react'
|
||||
import { MODEL_QUADRANTS } from '../hooks/useApi.js'
|
||||
import { raceColor, OBSERVED_MARK_COLOR, RACE_LABELS, SHORT_RACE_LABEL } from '../utils/colors.js'
|
||||
import { fitSkewedInterval, densityCurve } from '../utils/distributionApprox.js'
|
||||
import { groupKey, hasDrawsForAll } from '../utils/drawGroups.js'
|
||||
import { kdeCurve } from '../utils/kde.js'
|
||||
import { niceTicks } from '../utils/niceTicks.js'
|
||||
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
|
||||
import ChartLegend from '../components/ChartLegend.jsx'
|
||||
import ApproxNote from '../components/ApproxNote.jsx'
|
||||
|
||||
/**
|
||||
* Modeled posterior density per race×sex group, for one selected model
|
||||
* (dropdown, default three-year + referral rate), split into Female/Male
|
||||
* columns — matches whitepaper-fig-clark-density-1.png's ridge style. Plain
|
||||
* SVG — each ridge is 60 analytic points from fitSkewedInterval/densityCurve,
|
||||
* already smooth without a binning/curveBasis smoothing pass.
|
||||
* columns. Each ridge is drawn from that group's 500 real posterior draws
|
||||
* (Gaussian KDE) when available, falling back per-group to the analytic
|
||||
* fitSkewedInterval/densityCurve approximation otherwise.
|
||||
*/
|
||||
|
||||
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
|
||||
const SEX_COLUMNS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
|
||||
const DEFAULT_MODEL = 'unified_m4_mod' // Three-year + referral rate
|
||||
|
||||
function buildGroupRow(row) {
|
||||
function buildGroupRow(row, draws) {
|
||||
const enroll = row.stu_enroll || 0
|
||||
const observedRate = enroll > 0 ? ((row.observed_arrests || 0) / enroll) * 1000 : 0
|
||||
const rateMedian = (row.rate_median || 0) * 1000
|
||||
@@ -28,13 +31,27 @@ function buildGroupRow(row) {
|
||||
race: row.race,
|
||||
sex: row.sex,
|
||||
observedRate,
|
||||
draws,
|
||||
fit: fitSkewedInterval({ median: rateMedian, lower: rateLower, upper: rateUpper }),
|
||||
}
|
||||
}
|
||||
|
||||
export default function RateDensityRidgeline({ quadData }) {
|
||||
export default function RateDensityRidgeline({ quadData, leaid, state }) {
|
||||
const [selectedModel, setSelectedModel] = useState(DEFAULT_MODEL)
|
||||
const rows = (quadData && quadData[selectedModel]) || []
|
||||
const groups = rows.map((r) => ({ race: r.race, sex: r.sex, stuEnroll: r.stu_enroll || 0 }))
|
||||
const { drawsByGroup } = useDrawDistribution({ leaid, state, model: selectedModel, year: '21-22', groups })
|
||||
|
||||
// Only the RACE_ORDER × SEX_COLUMNS cells get a ridge, so coverage is judged
|
||||
// against those. The hook's 'ready' status is an any-group signal: individual
|
||||
// ridges still fall back to densityCurve when their group is missing from the
|
||||
// shard, so gating the note on `status` alone would hide it while part of the
|
||||
// chart is an approximation. A null map (loading or a failed fetch) also fails
|
||||
// this check, so the error path still shows the note.
|
||||
const renderedGroups = rows.filter(
|
||||
(r) => RACE_ORDER.includes(r.race) && SEX_COLUMNS.some((c) => c.sex === r.sex),
|
||||
)
|
||||
const allGroupsHaveDraws = hasDrawsForAll(drawsByGroup, renderedGroups)
|
||||
|
||||
const modelSelect = (
|
||||
<select
|
||||
@@ -55,7 +72,7 @@ export default function RateDensityRidgeline({ quadData }) {
|
||||
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
|
||||
Predicted arrest rates by student group
|
||||
</h3>
|
||||
<ApproxNote />
|
||||
{!allGroupsHaveDraws && <ApproxNote />}
|
||||
</div>
|
||||
{modelSelect}
|
||||
</div>
|
||||
@@ -64,7 +81,7 @@ export default function RateDensityRidgeline({ quadData }) {
|
||||
<p style={{ color: 'var(--cv-ink-3)', marginTop: 'var(--space-2)' }}>No model data available.</p>
|
||||
) : (
|
||||
<>
|
||||
<RidgeColumns rows={rows} />
|
||||
<RidgeColumns rows={rows} drawsByGroup={drawsByGroup} />
|
||||
<ChartLegend items={[
|
||||
...RACE_ORDER.map((race) => ({ shape: 'swatch', color: raceColor(race), label: RACE_LABELS[race] })),
|
||||
{ shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
|
||||
@@ -75,7 +92,7 @@ export default function RateDensityRidgeline({ quadData }) {
|
||||
)
|
||||
}
|
||||
|
||||
function RidgeColumns({ rows }) {
|
||||
function RidgeColumns({ rows, drawsByGroup }) {
|
||||
// Shared x-domain across both columns, so Female/Male are directly comparable.
|
||||
const allUpper = rows.map((r) => (r.rate_upper || 0) * 1000)
|
||||
const allObserved = rows
|
||||
@@ -87,14 +104,24 @@ function RidgeColumns({ rows }) {
|
||||
return (
|
||||
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
|
||||
{SEX_COLUMNS.map(({ sex, label }) => (
|
||||
<SexRidgeColumn key={sex} label={label} rows={rows.filter((r) => r.sex === sex)} ticks={ticks} maxRate={niceMax} />
|
||||
<SexRidgeColumn
|
||||
key={sex}
|
||||
label={label}
|
||||
rows={rows.filter((r) => r.sex === sex)}
|
||||
drawsByGroup={drawsByGroup}
|
||||
ticks={ticks}
|
||||
maxRate={niceMax}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function SexRidgeColumn({ label, rows, ticks, maxRate }) {
|
||||
const groups = RACE_ORDER.map((race) => rows.find((r) => r.race === race)).filter(Boolean).map(buildGroupRow)
|
||||
function SexRidgeColumn({ label, rows, drawsByGroup, ticks, maxRate }) {
|
||||
const groups = RACE_ORDER
|
||||
.map((race) => rows.find((r) => r.race === race))
|
||||
.filter(Boolean)
|
||||
.map((row) => buildGroupRow(row, drawsByGroup?.[groupKey(row.race, row.sex)]))
|
||||
|
||||
const width = 300
|
||||
const rowHeight = 58
|
||||
@@ -104,7 +131,11 @@ function SexRidgeColumn({ label, rows, ticks, maxRate }) {
|
||||
|
||||
const xScale = (val) => margin.left + (val / maxRate) * innerWidth
|
||||
|
||||
const curves = groups.map((g) => densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }))
|
||||
const curves = groups.map((g) =>
|
||||
g.draws && g.draws.length > 0
|
||||
? kdeCurve(g.draws, { min: 0, max: maxRate, n: 60 })
|
||||
: densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }),
|
||||
)
|
||||
const maxPdf = Math.max(...curves.flatMap((c) => c.map((p) => p.y)), 1e-9)
|
||||
const peakHeight = rowHeight * 0.82
|
||||
|
||||
|
||||
@@ -95,9 +95,9 @@ export default function ChartPanel({ district, state }) {
|
||||
}}>
|
||||
<ArrestsOverTime data={timeSeriesData} modelId={WAVE_MODEL} />
|
||||
|
||||
<RateByGroupBar data={rateByGroup} />
|
||||
<RateByGroupBar data={rateByGroup} leaid={district.leaid} state={state} />
|
||||
|
||||
<RateDensityRidgeline quadData={data.quadData} />
|
||||
<RateDensityRidgeline quadData={data.quadData} leaid={district.leaid} state={state} />
|
||||
</div>
|
||||
|
||||
{/* Methodology footer */}
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
import { useEffect, useState } from 'react'
|
||||
import { getDb } from '../utils/duckdbClient.js'
|
||||
import { groupKey } from '../utils/drawGroups.js'
|
||||
|
||||
const HF_BASE = 'https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet'
|
||||
|
||||
// Module-level cache: one registered duckdb-wasm file buffer per
|
||||
// (model, year, state) shard, shared across every component instance and
|
||||
// district navigated to in this browser session. See Global Constraints —
|
||||
// in-memory only, no persistence across page loads.
|
||||
const shardCache = new Map()
|
||||
|
||||
function shardKey(model, year, state) {
|
||||
return `${model}__${year}__${state}`
|
||||
}
|
||||
|
||||
function ensureShardRegistered(db, model, year, state) {
|
||||
const key = shardKey(model, year, state)
|
||||
if (!shardCache.has(key)) {
|
||||
shardCache.set(
|
||||
key,
|
||||
(async () => {
|
||||
try {
|
||||
const url = `${HF_BASE}/model_id=${model}/YEAR=${year}/LEA_STATE=${state}/data_0.parquet`
|
||||
const res = await fetch(url)
|
||||
if (!res.ok) throw new Error(`Failed to fetch draw shard: HTTP ${res.status}`)
|
||||
const buffer = new Uint8Array(await res.arrayBuffer())
|
||||
const fileName = `${key}.parquet`
|
||||
await db.registerFileBuffer(fileName, buffer)
|
||||
return fileName
|
||||
} catch (err) {
|
||||
// Don't let a transient failure (network blip, HF outage) poison the
|
||||
// cache forever — remove the rejected entry so the next caller for
|
||||
// this shard gets a fresh attempt instead of the same dead promise.
|
||||
shardCache.delete(key)
|
||||
throw err
|
||||
}
|
||||
})(),
|
||||
)
|
||||
}
|
||||
return shardCache.get(key)
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches real posterior draws for one district/model/year from the Hugging
|
||||
* Face parquet dataset via duckdb-wasm, converts predicted counts to
|
||||
* rate-per-1,000 using each group's stu_enroll (not present in the draws
|
||||
* table itself — joined here from data this app already has), and returns
|
||||
* them keyed by "RACE_SEX" (see `groupKey` in utils/drawGroups.js).
|
||||
*
|
||||
* `status` is an ANY-group signal: 'ready' means at least one group came back
|
||||
* with real draws, not that every requested group did. Groups can be missing
|
||||
* individually (falsy enrollment, or absent from the parquet shard), so a caller
|
||||
* that wants to claim "these are all real draws" must check its own rendered
|
||||
* groups against `drawsByGroup` — use `hasDrawsForAll` from utils/drawGroups.js.
|
||||
*
|
||||
* @param {{leaid: string, state: string, model: string, year: string,
|
||||
* groups: Array<{race: string, sex: string, stuEnroll: number}>}} params
|
||||
* @returns {{status: 'loading'|'ready'|'error', drawsByGroup: Record<string, number[]> | null}}
|
||||
*/
|
||||
export function useDrawDistribution({ leaid, state, model, year, groups }) {
|
||||
const [status, setStatus] = useState('loading')
|
||||
const [drawsByGroup, setDrawsByGroup] = useState(null)
|
||||
|
||||
// groups is typically a fresh array literal every render; derive a stable
|
||||
// primitive so the effect only re-runs when its actual content changes.
|
||||
const groupsSignature = (groups || []).map((g) => `${g.race}:${g.sex}:${g.stuEnroll}`).join(',')
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false
|
||||
// Reset BEFORE the input guard below, not after. Clearing any previous
|
||||
// model/district's draws has to happen on every input change, including the
|
||||
// ones that have nothing to fetch. Without this ordering, a chart that
|
||||
// varies `model` across renders (e.g. RateDensityRidgeline's dropdown) and
|
||||
// lands on a model whose `groups` is empty (that model's upstream fetch
|
||||
// failed) would keep reporting 'ready' and keep handing back the *previous*
|
||||
// model's real draws — keyed by the same RACE_SEX strings — under the newly
|
||||
// selected model's summary stats, silently mixing two models' data.
|
||||
setStatus('loading')
|
||||
setDrawsByGroup(null)
|
||||
|
||||
// Nothing to fetch: stay in 'loading' with no draws, which every consumer
|
||||
// already treats as "fall back to the approximation". No cleanup needed —
|
||||
// nothing async was started.
|
||||
if (!leaid || !state || !model || !year || !groups?.length) return
|
||||
|
||||
async function run() {
|
||||
let conn
|
||||
try {
|
||||
const db = await getDb()
|
||||
const fileName = await ensureShardRegistered(db, model, year, state)
|
||||
conn = await db.connect()
|
||||
const stmt = await conn.prepare(`SELECT RACE, SEX, pred FROM read_parquet('${fileName}') WHERE LEAID = ?`)
|
||||
const table = await stmt.query(leaid)
|
||||
await stmt.close()
|
||||
const rows = table.toArray().map((r) => r.toJSON())
|
||||
|
||||
const enrollByGroup = {}
|
||||
for (const g of groups) enrollByGroup[groupKey(g.race, g.sex)] = g.stuEnroll || 0
|
||||
|
||||
const byGroup = {}
|
||||
for (const row of rows) {
|
||||
const key = groupKey(row.RACE, row.SEX)
|
||||
const enroll = enrollByGroup[key]
|
||||
if (!enroll) continue
|
||||
const rate = (Number(row.pred) / enroll) * 1000
|
||||
;(byGroup[key] ??= []).push(rate)
|
||||
}
|
||||
|
||||
// A successful query with zero matching rows (this district isn't in
|
||||
// the draws shard, or none of its rows matched a known group) is not
|
||||
// "ready" — there's no real data to show, so treat it like a failure
|
||||
// and let the caller fall back, rather than silently claiming real
|
||||
// draws while every group actually uses the fitted approximation.
|
||||
const hasDraws = Object.values(byGroup).some((draws) => draws.length > 0)
|
||||
|
||||
if (!cancelled) {
|
||||
if (hasDraws) {
|
||||
setDrawsByGroup(byGroup)
|
||||
setStatus('ready')
|
||||
} else {
|
||||
console.warn('useDrawDistribution: query succeeded but returned no usable draws for', { leaid, model, year, state })
|
||||
setDrawsByGroup(null)
|
||||
setStatus('error')
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error('useDrawDistribution failed:', err)
|
||||
if (!cancelled) {
|
||||
// Belt-and-suspenders alongside the setDrawsByGroup(null) at the
|
||||
// top of this effect: a failed fetch/query must never leave a
|
||||
// *previous* model's real draws in place under the newly-selected
|
||||
// model's status/summary stats.
|
||||
setDrawsByGroup(null)
|
||||
setStatus('error')
|
||||
}
|
||||
} finally {
|
||||
if (conn) await conn.close()
|
||||
}
|
||||
}
|
||||
|
||||
run()
|
||||
return () => {
|
||||
cancelled = true
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [leaid, state, model, year, groupsSignature])
|
||||
|
||||
return { status, drawsByGroup }
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* Shared key format and coverage check for the per-group posterior draws
|
||||
* returned by `useDrawDistribution`. Both the hook (which builds the map) and
|
||||
* the charts (which read it, and decide whether to claim "real draws") go
|
||||
* through here so the key format lives in exactly one place.
|
||||
*/
|
||||
|
||||
/** Canonical key for one race×sex group in a `drawsByGroup` map. */
|
||||
export function groupKey(race, sex) {
|
||||
return `${race}_${sex}`
|
||||
}
|
||||
|
||||
/**
|
||||
* True only when *every* group in `groups` has a non-empty draw array.
|
||||
*
|
||||
* `useDrawDistribution`'s `status` is an any-group signal: it reports 'ready'
|
||||
* as soon as one group has real draws. Charts fall back per-group, so the
|
||||
* chart-wide "these are real posterior draws" note/caption must be gated on
|
||||
* complete coverage instead — a group can be missing because its enrollment is
|
||||
* falsy or because its (LEAID, RACE, SEX) isn't in the parquet shard.
|
||||
*
|
||||
* Returns false for an empty group list (nothing rendered means nothing to
|
||||
* claim) and for a null map (loading, or the fetch failed outright).
|
||||
*
|
||||
* @param {Record<string, number[]> | null | undefined} drawsByGroup
|
||||
* @param {Array<{race: string, sex: string}>} groups - the groups a chart is
|
||||
* actually rendering, not everything the API returned.
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function hasDrawsForAll(drawsByGroup, groups) {
|
||||
if (!drawsByGroup || !groups?.length) return false
|
||||
return groups.every((g) => (drawsByGroup[groupKey(g.race, g.sex)]?.length ?? 0) > 0)
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
import { test } from 'node:test'
|
||||
import assert from 'node:assert/strict'
|
||||
import { groupKey, hasDrawsForAll } from './drawGroups.js'
|
||||
|
||||
test('groupKey: joins race and sex with an underscore', () => {
|
||||
assert.equal(groupKey('BL', 'F'), 'BL_F')
|
||||
})
|
||||
|
||||
test('hasDrawsForAll: true when every rendered group has draws', () => {
|
||||
const map = { WH_F: [1, 2], BL_F: [3] }
|
||||
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), true)
|
||||
})
|
||||
|
||||
test('hasDrawsForAll: false when one rendered group is missing', () => {
|
||||
// The any-group 'ready' status would still be true here — this is exactly the
|
||||
// case where the chart must keep showing the approximation note.
|
||||
const map = { WH_F: [1, 2] }
|
||||
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), false)
|
||||
})
|
||||
|
||||
test('hasDrawsForAll: false when a rendered group has an empty draw array', () => {
|
||||
const map = { WH_F: [1, 2], BL_F: [] }
|
||||
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), false)
|
||||
})
|
||||
|
||||
test('hasDrawsForAll: false for a null map (loading or failed fetch)', () => {
|
||||
assert.equal(hasDrawsForAll(null, [{ race: 'WH', sex: 'F' }]), false)
|
||||
assert.equal(hasDrawsForAll(undefined, [{ race: 'WH', sex: 'F' }]), false)
|
||||
})
|
||||
|
||||
test('hasDrawsForAll: false for an empty or missing group list', () => {
|
||||
assert.equal(hasDrawsForAll({ WH_F: [1] }, []), false)
|
||||
assert.equal(hasDrawsForAll({ WH_F: [1] }, undefined), false)
|
||||
})
|
||||
|
||||
test('hasDrawsForAll: ignores groups the chart is not rendering', () => {
|
||||
// Extra keys in the map (e.g. a race outside RACE_ORDER) must not block the
|
||||
// claim for the groups actually on screen.
|
||||
const map = { WH_F: [1], BL_F: [2], AS_F: [3] }
|
||||
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), true)
|
||||
})
|
||||
@@ -0,0 +1,26 @@
|
||||
/**
|
||||
* Lazy-initialized singleton AsyncDuckDB instance running the single-threaded
|
||||
* MVP wasm bundle only — never eh/coi, which need Cross-Origin-Opener-Policy /
|
||||
* Cross-Origin-Embedder-Policy response headers this static host doesn't send.
|
||||
* See docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md.
|
||||
*/
|
||||
|
||||
let dbPromise = null
|
||||
|
||||
/** @returns {Promise<import('@duckdb/duckdb-wasm').AsyncDuckDB>} */
|
||||
export function getDb() {
|
||||
if (!dbPromise) dbPromise = initDb()
|
||||
return dbPromise
|
||||
}
|
||||
|
||||
async function initDb() {
|
||||
const duckdb = await import('@duckdb/duckdb-wasm')
|
||||
const mvpWorkerUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js?url')).default
|
||||
const mvpWasmUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm?url')).default
|
||||
|
||||
const worker = new Worker(mvpWorkerUrl)
|
||||
const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING)
|
||||
const db = new duckdb.AsyncDuckDB(logger, worker)
|
||||
await db.instantiate(mvpWasmUrl, null)
|
||||
return db
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* Empirical density utilities for posterior draw arrays — Gaussian KDE with
|
||||
* Silverman's rule-of-thumb bandwidth, plus a linear-interpolated quantile.
|
||||
* Used in place of distributionApprox.js's analytic fitSkewedInterval/
|
||||
* densityCurve approximation whenever real posterior draws are available.
|
||||
*/
|
||||
|
||||
const SQRT_2PI = Math.sqrt(2 * Math.PI)
|
||||
// Absolute last-resort floor, used only when no plotting domain is known.
|
||||
const MIN_BANDWIDTH = 1e-3
|
||||
// Bandwidth is clamped relative to the plotting domain, not to absolute units,
|
||||
// because these draws are rate-per-1,000 values whose scale varies by orders of
|
||||
// magnitude between districts. domainWidth/50 is slightly wider than one render
|
||||
// step at the charts' n = 60 (step = domainWidth/59), so a degenerate draw set
|
||||
// (e.g. 500 identical zeros, common for small districts) resolves as a narrow
|
||||
// bump instead of a delta spike that flattens every other ridge sharing the
|
||||
// column's maxPdf. It is also loose enough not to bind on an ordinary posterior:
|
||||
// a spread wider than ~8% of the domain keeps its own Silverman bandwidth.
|
||||
// domainWidth/6 stops a handful of extreme draws from inflating sd until the
|
||||
// curve is a flat line.
|
||||
const BANDWIDTH_FLOOR_DIVISOR = 50
|
||||
const BANDWIDTH_CEILING_DIVISOR = 6
|
||||
|
||||
/**
|
||||
* Linear-interpolated quantile (R type-7). Does not mutate `draws`.
|
||||
* @param {number[]} draws
|
||||
* @param {number} p - probability in [0, 1]
|
||||
* @returns {number}
|
||||
*/
|
||||
export function quantile(draws, p) {
|
||||
const sorted = [...draws].sort((a, b) => a - b)
|
||||
const idx = p * (sorted.length - 1)
|
||||
const lo = Math.floor(idx)
|
||||
const hi = Math.ceil(idx)
|
||||
if (lo === hi) return sorted[lo]
|
||||
const frac = idx - lo
|
||||
return sorted[lo] * (1 - frac) + sorted[hi] * frac
|
||||
}
|
||||
|
||||
function standardDeviation(draws) {
|
||||
const n = draws.length
|
||||
const mean = draws.reduce((sum, d) => sum + d, 0) / n
|
||||
const variance = draws.reduce((sum, d) => sum + (d - mean) ** 2, 0) / (n - 1)
|
||||
return Math.sqrt(variance)
|
||||
}
|
||||
|
||||
/**
|
||||
* Silverman's rule-of-thumb bandwidth (robust variant using the smallest
|
||||
* *positive* spread estimate among sd and IQR/1.34), clamped to a fraction of
|
||||
* the plotting domain.
|
||||
*
|
||||
* Both ends of the clamp matter for the zero-inflated count posteriors small
|
||||
* districts produce. Without the floor, an all-identical draw set (sd = IQR = 0)
|
||||
* collapses to a delta-function spike. Without the ceiling, a group whose IQR is
|
||||
* 0 (the normal case when most draws are 0) falls back to raw sd, which a
|
||||
* handful of extreme draws inflates until the ridge is a featureless flat line.
|
||||
*
|
||||
* @param {number[]} draws
|
||||
* @param {number} [domainWidth] - width of the x-range the curve will be drawn
|
||||
* over. Omit only when no domain is known; the clamp then degrades to the
|
||||
* absolute MIN_BANDWIDTH floor.
|
||||
* @returns {number}
|
||||
*/
|
||||
export function silvermanBandwidth(draws, domainWidth = 0) {
|
||||
const width = domainWidth > 0 ? domainWidth : 0
|
||||
const floor = width ? width / BANDWIDTH_FLOOR_DIVISOR : MIN_BANDWIDTH
|
||||
const ceiling = width ? width / BANDWIDTH_CEILING_DIVISOR : Infinity
|
||||
|
||||
const n = draws.length
|
||||
if (n < 2) return floor
|
||||
|
||||
const sd = standardDeviation(draws)
|
||||
const iqr = quantile(draws, 0.75) - quantile(draws, 0.25)
|
||||
// Degrade gracefully: keep the robust rule when IQR is informative, use sd
|
||||
// when it isn't, and let the floor handle a fully degenerate draw set —
|
||||
// rather than treating a zero spread as "no estimate available".
|
||||
const candidates = [sd, iqr / 1.34].filter((v) => v > 0)
|
||||
const spread = candidates.length ? Math.min(...candidates) : 0
|
||||
|
||||
const raw = 0.9 * spread * Math.pow(n, -0.2)
|
||||
return Math.min(Math.max(raw, floor), ceiling)
|
||||
}
|
||||
|
||||
/**
|
||||
* n evenly spaced {x, y} points of a Gaussian KDE over `draws` — same shape
|
||||
* contract as distributionApprox.js's densityCurve, so chart code can switch
|
||||
* between the two without changing its rendering path.
|
||||
* @param {number[]} draws
|
||||
* @param {{min?: number, max?: number, n?: number}} [options]
|
||||
* @returns {Array<{x: number, y: number}>}
|
||||
*/
|
||||
export function kdeCurve(draws, { min = 0, max, n = 60 } = {}) {
|
||||
const hi = max ?? Math.max(...draws) * 1.1
|
||||
// Guarded against a degenerate/inverted domain so the bandwidth clamp can
|
||||
// never be handed a negative width.
|
||||
const domainWidth = Math.max(hi - min, 0)
|
||||
const h = silvermanBandwidth(draws, domainWidth)
|
||||
const step = (hi - min) / (n - 1)
|
||||
const points = []
|
||||
for (let i = 0; i < n; i++) {
|
||||
const x = min + step * i
|
||||
let sum = 0
|
||||
for (const d of draws) {
|
||||
const z = (x - d) / h
|
||||
sum += Math.exp(-0.5 * z * z) / SQRT_2PI
|
||||
}
|
||||
points.push({ x, y: sum / (draws.length * h) })
|
||||
}
|
||||
return points
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
import { test } from 'node:test'
|
||||
import assert from 'node:assert/strict'
|
||||
import { quantile, silvermanBandwidth, kdeCurve } from './kde.js'
|
||||
|
||||
test('quantile: median of an odd-length array', () => {
|
||||
assert.equal(quantile([3, 1, 2], 0.5), 2)
|
||||
})
|
||||
|
||||
test('quantile: linear interpolation between two ranks', () => {
|
||||
// sorted: [10, 20, 30, 40] — p=0.25 -> index 0.75 -> interpolate 10..20
|
||||
assert.equal(quantile([40, 10, 30, 20], 0.25), 17.5)
|
||||
})
|
||||
|
||||
test('quantile: does not mutate its input array', () => {
|
||||
const input = [5, 3, 4, 1, 2]
|
||||
quantile(input, 0.5)
|
||||
assert.deepEqual(input, [5, 3, 4, 1, 2])
|
||||
})
|
||||
|
||||
test('silvermanBandwidth: positive, finite floor for identical draws', () => {
|
||||
const h = silvermanBandwidth([7, 7, 7, 7, 7], 10)
|
||||
assert.ok(h > 0 && Number.isFinite(h))
|
||||
assert.equal(h, 10 / 50, 'degenerate spread falls back to the domain-relative floor')
|
||||
})
|
||||
|
||||
test('silvermanBandwidth: positive, finite floor for a single draw', () => {
|
||||
const h = silvermanBandwidth([7], 10)
|
||||
assert.ok(h > 0 && Number.isFinite(h))
|
||||
assert.equal(h, 10 / 50)
|
||||
})
|
||||
|
||||
test('silvermanBandwidth: positive, finite floor when no domain is supplied', () => {
|
||||
assert.ok(silvermanBandwidth([7, 7, 7, 7, 7]) > 0)
|
||||
assert.ok(silvermanBandwidth([7]) > 0)
|
||||
})
|
||||
|
||||
test('silvermanBandwidth: clamped to [domainWidth/50, domainWidth/6]', () => {
|
||||
const domainWidth = 30
|
||||
// sd is inflated by 50 extreme draws; without the ceiling this is ~15.6.
|
||||
const outlierDraws = [...Array(450).fill(0), ...Array(50).fill(200)]
|
||||
assert.equal(silvermanBandwidth(outlierDraws, domainWidth), domainWidth / 6)
|
||||
// Fully degenerate: sd = IQR = 0.
|
||||
assert.equal(silvermanBandwidth(Array(500).fill(0), domainWidth), domainWidth / 50)
|
||||
})
|
||||
|
||||
test('silvermanBandwidth: leaves an ordinary spread untouched by the clamp', () => {
|
||||
const draws = Array.from({ length: 500 }, (_, i) => 3 + Math.sin(i) * 1.2 + (i % 7) * 0.15)
|
||||
const h = silvermanBandwidth(draws, 10)
|
||||
assert.ok(h > 10 / 50 && h < 10 / 6, `expected an unclamped bandwidth, got ${h}`)
|
||||
})
|
||||
|
||||
test('kdeCurve: all-identical draws do not produce a delta-function spike', () => {
|
||||
// 500 identical zeros is the common case for a small district's rare-event
|
||||
// count posterior. The old absolute 1e-3 floor gave maxY ~399 here (peak
|
||||
// ~3989x a uniform density over the same domain), which flattened every other
|
||||
// ridge sharing the column's maxPdf to sub-pixel height.
|
||||
const domainWidth = 10
|
||||
const curve = kdeCurve(Array(500).fill(0), { min: 0, max: domainWidth, n: 60 })
|
||||
const maxY = Math.max(...curve.map((p) => p.y))
|
||||
assert.ok(Number.isFinite(maxY) && maxY > 0)
|
||||
assert.ok(
|
||||
maxY * domainWidth < 25,
|
||||
`peak density should stay within ~25x a uniform density over the domain, got ${maxY * domainWidth}x`,
|
||||
)
|
||||
})
|
||||
|
||||
test('kdeCurve: zero-inflated draws keep their shape (not over-smoothed to flat)', () => {
|
||||
const domainWidth = 10
|
||||
const draws = [...Array(450).fill(0), ...Array(50).fill(1)]
|
||||
const curve = kdeCurve(draws, { min: 0, max: domainWidth, n: 60 })
|
||||
const ys = curve.map((p) => p.y)
|
||||
const maxY = Math.max(...ys)
|
||||
const meanY = ys.reduce((sum, y) => sum + y, 0) / ys.length
|
||||
assert.ok(maxY / meanY > 3, `expected a peaked curve, got peak/mean ${maxY / meanY}`)
|
||||
})
|
||||
|
||||
test('kdeCurve: outlier-inflated sd does not flatten the curve', () => {
|
||||
// IQR is 0 here (most draws are 0), so the rule falls back to sd — which these
|
||||
// 50 extreme draws inflate to ~60. Unclamped that gives h ~15.6 on a domain of
|
||||
// 30, i.e. peak/mean ~1.6: a near-flat line claiming maximal uncertainty.
|
||||
const domainWidth = 30
|
||||
const draws = [...Array(450).fill(0), ...Array(50).fill(200)]
|
||||
const curve = kdeCurve(draws, { min: 0, max: domainWidth, n: 60 })
|
||||
const ys = curve.map((p) => p.y)
|
||||
const maxY = Math.max(...ys)
|
||||
const meanY = ys.reduce((sum, y) => sum + y, 0) / ys.length
|
||||
assert.ok(maxY / meanY > 3, `expected a peaked curve, got peak/mean ${maxY / meanY}`)
|
||||
})
|
||||
|
||||
test('kdeCurve: returns n points spanning [min, max]', () => {
|
||||
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
|
||||
const curve = kdeCurve(draws, { min: 0, max: 10, n: 60 })
|
||||
assert.equal(curve.length, 60)
|
||||
assert.equal(curve[0].x, 0)
|
||||
assert.ok(Math.abs(curve[curve.length - 1].x - 10) < 1e-9)
|
||||
})
|
||||
|
||||
test('kdeCurve: density integrates to ~1 over a wide domain (trapezoidal check)', () => {
|
||||
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
|
||||
const curve = kdeCurve(draws, { min: -20, max: 30, n: 2000 })
|
||||
let area = 0
|
||||
for (let i = 1; i < curve.length; i++) {
|
||||
const dx = curve[i].x - curve[i - 1].x
|
||||
area += (dx * (curve[i].y + curve[i - 1].y)) / 2
|
||||
}
|
||||
assert.ok(Math.abs(area - 1) < 0.01, `expected area ~1, got ${area}`)
|
||||
})
|
||||
@@ -8,4 +8,10 @@ export default defineConfig({
|
||||
server: { port: 5173 },
|
||||
build: { outDir: 'dist' },
|
||||
base: '/crdc-demo/', // Required for subdirectory deployment on git-pages
|
||||
optimizeDeps: {
|
||||
// duckdb-wasm ships its own worker + wasm binaries resolved via `?url`
|
||||
// imports; esbuild's dev-server pre-bundling can rewrite those import
|
||||
// paths and break worker instantiation. Exclude it from pre-bundling.
|
||||
exclude: ['@duckdb/duckdb-wasm'],
|
||||
},
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user