diff --git a/AGENTS.md b/AGENTS.md index a0cf7f1..a800afc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,11 +83,26 @@ Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for Not all endpoints are available to browser-based clients: - `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds) -- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser +- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below. -If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds. +### 5. Real posterior draws via duckdb-wasm -### 5. CORS Configuration +Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual +500-draw-per-group posterior from the public Hugging Face parquet dataset +(`civilytics/crdc-school-arrest-rates`), queried client-side with +`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` + +`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is +involved. If that fetch fails (network, unsupported browser, HF outage), +both charts fall back to the `distributionApprox.js` analytic approximation +and show the "estimated shape" note — **do not delete `distributionApprox.js` +or `ApproxNote.jsx`**, they're the fallback path, not dead code. + +See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for +the full design. + +The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path. + +### 6. CORS Configuration The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured: - The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable diff --git a/HANDOFF.md b/HANDOFF.md index 206b8e0..3ddb9f9 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -33,7 +33,7 @@ The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PRO ## What Needs to Be Done Next ### Future Enhancements -- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required. +- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`. - **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling. ## Key Files diff --git a/README.md b/README.md index 81d77f3..79a22ee 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ Visitors select a U.S. state, search for a school district (with suggestions of 2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG 3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG 4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate) -5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates +5. **Predicted rates by student group** (Chart 5) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched. 6. **Exceedance probability** (Chart 6) — P(district > national) per student group ## Architecture @@ -72,7 +72,7 @@ crdc-demo/ | `/api/v1/models` | List available Bayesian model specs | Once (cached) | | `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke | | `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district | -| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app | +| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` | | `/data/national_rates.json` | Static national rates fixture (committed) | Once per session | ## Deployment