docs: reflect real posterior-draw architecture in AGENTS/README/HANDOFF
- AGENTS.md: Replace §4 (API Endpoint Availability) with updated text; insert new §5 (Real posterior draws via duckdb-wasm) documenting the shift from synthetic normal-approximation draws to client-side fetches via duckdb-wasm against the public Hugging Face parquet dataset. Include actual payload size (~39MB uncompressed / ~8.86MB gzipped). Renumber subsequent items. - README.md: Update Chart 5 description in "What It Does" to reflect real draws + fallback behavior. Update API table to clarify that /api/v1/draws is not called from app but informs the Hugging Face URL the app fetches directly. - HANDOFF.md: Mark "Raw posterior draws" as done (2026-08-11) with reference to the design spec. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -83,11 +83,26 @@ Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for
|
||||
|
||||
Not all endpoints are available to browser-based clients:
|
||||
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
|
||||
- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser
|
||||
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
|
||||
|
||||
If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds.
|
||||
### 5. Real posterior draws via duckdb-wasm
|
||||
|
||||
### 5. CORS Configuration
|
||||
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
|
||||
500-draw-per-group posterior from the public Hugging Face parquet dataset
|
||||
(`civilytics/crdc-school-arrest-rates`), queried client-side with
|
||||
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
|
||||
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
|
||||
involved. If that fetch fails (network, unsupported browser, HF outage),
|
||||
both charts fall back to the `distributionApprox.js` analytic approximation
|
||||
and show the "estimated shape" note — **do not delete `distributionApprox.js`
|
||||
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
|
||||
|
||||
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
|
||||
the full design.
|
||||
|
||||
The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.
|
||||
|
||||
### 6. CORS Configuration
|
||||
|
||||
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
|
||||
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PRO
|
||||
## What Needs to Be Done Next
|
||||
|
||||
### Future Enhancements
|
||||
- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required.
|
||||
- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`.
|
||||
- **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling.
|
||||
|
||||
## Key Files
|
||||
|
||||
@@ -24,7 +24,7 @@ Visitors select a U.S. state, search for a school district (with suggestions of
|
||||
2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG
|
||||
3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG
|
||||
4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate)
|
||||
5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates
|
||||
5. **Predicted rates by student group** (Chart 5) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
|
||||
6. **Exceedance probability** (Chart 6) — P(district > national) per student group
|
||||
|
||||
## Architecture
|
||||
@@ -72,7 +72,7 @@ crdc-demo/
|
||||
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
|
||||
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
|
||||
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district |
|
||||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app |
|
||||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
|
||||
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
|
||||
|
||||
## Deployment
|
||||
|
||||
Reference in New Issue
Block a user