docs: reflect real posterior-draw architecture in AGENTS/README/HANDOFF

- AGENTS.md: Replace §4 (API Endpoint Availability) with updated text; insert
  new §5 (Real posterior draws via duckdb-wasm) documenting the shift from
  synthetic normal-approximation draws to client-side fetches via duckdb-wasm
  against the public Hugging Face parquet dataset. Include actual payload size
  (~39MB uncompressed / ~8.86MB gzipped). Renumber subsequent items.
- README.md: Update Chart 5 description in "What It Does" to reflect real draws
  + fallback behavior. Update API table to clarify that /api/v1/draws is not
  called from app but informs the Hugging Face URL the app fetches directly.
- HANDOFF.md: Mark "Raw posterior draws" as done (2026-08-11) with reference
  to the design spec.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-11 09:57:12 -04:00
co-authored by Claude Sonnet 5
parent 20c08ae198
commit 523c21d78c
3 changed files with 21 additions and 6 deletions
+18 -3
View File
@@ -83,11 +83,26 @@ Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for
Not all endpoints are available to browser-based clients:
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds.
### 5. Real posterior draws via duckdb-wasm
### 5. CORS Configuration
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
500-draw-per-group posterior from the public Hugging Face parquet dataset
(`civilytics/crdc-school-arrest-rates`), queried client-side with
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
involved. If that fetch fails (network, unsupported browser, HF outage),
both charts fall back to the `distributionApprox.js` analytic approximation
and show the "estimated shape" note — **do not delete `distributionApprox.js`
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
the full design.
The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.
### 6. CORS Configuration
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
+1 -1
View File
@@ -33,7 +33,7 @@ The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PRO
## What Needs to Be Done Next
### Future Enhancements
- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required.
- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`.
- **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling.
## Key Files
+2 -2
View File
@@ -24,7 +24,7 @@ Visitors select a U.S. state, search for a school district (with suggestions of
2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG
3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG
4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate)
5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates
5. **Predicted rates by student group** (Chart 5) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
6. **Exceedance probability** (Chart 6) — P(district > national) per student group
## Architecture
@@ -72,7 +72,7 @@ crdc-demo/
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district |
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app |
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
## Deployment