jaredandClaude Sonnet 5
523c21d78c
- AGENTS.md: Replace §4 (API Endpoint Availability) with updated text; insert new §5 (Real posterior draws via duckdb-wasm) documenting the shift from synthetic normal-approximation draws to client-side fetches via duckdb-wasm against the public Hugging Face parquet dataset. Include actual payload size (~39MB uncompressed / ~8.86MB gzipped). Renumber subsequent items. - README.md: Update Chart 5 description in "What It Does" to reflect real draws + fallback behavior. Update API table to clarify that /api/v1/draws is not called from app but informs the Hugging Face URL the app fetches directly. - HANDOFF.md: Mark "Raw posterior draws" as done (2026-08-11) with reference to the design spec. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
151 lines
8.2 KiB
Markdown
151 lines
8.2 KiB
Markdown
# Agent Guide — CRDC Demo App
|
||
|
||
This document captures context, decisions, and guidance for agents working on this codebase. It is the primary source of truth for how to make changes safely.
|
||
|
||
## Project Overview
|
||
|
||
A React + Vite static web app demonstrating the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/). Visitors select a state, search for a school district, and see 6 charts comparing observed arrest data against Bayesian model estimates. Deployed via Gitea Actions to `pages.civilytics.org/crdc-demo/`.
|
||
|
||
## Architecture Summary
|
||
|
||
- **Frontend**: React 19 + Vite (static site generation)
|
||
- **Styling**: Plain CSS custom properties matching Civilytics design tokens (`src/styles/tokens.css`)
|
||
- **Charts**: Mixed approach:
|
||
- Charts 1–3 use inline SVG with manual scales (no D3 dependency for these)
|
||
- Chart 4 (ModelDrawsComparison) uses D3.js v7 for data-driven rendering of quadrant comparisons
|
||
- Chart 5 (RateDensityRidgeline) uses D3.js v7 for density ridge visualizations
|
||
- **API**: Calls public read-only API directly from browser; no backend required
|
||
- **Deployment**: Static site deployed via Gitea Actions (`.gitea/workflows/pages.yml`)
|
||
|
||
## Key Components and Data Flow
|
||
|
||
```
|
||
App.jsx (router)
|
||
→ StateSelector (landing screen: state dropdown/grid)
|
||
→ DistrictSearch (search + "interesting" suggestions from /estimates?state=&year=)
|
||
→ LoadingAnimation (fetches all data in parallel, shows animated histogram grid)
|
||
→ ChartPanel (receives district object, fetches structured estimates for 6 charts)
|
||
├── ArrestsOverTime — SVG line chart by wave (3 years)
|
||
├── RateByGroupBar — SVG bar chart: observed vs modeled per group
|
||
├── DistrictVsNational — SVG comparison to national average
|
||
├── ModelDrawsComparison — D3 quadrant charts (4 model types × 1 year)
|
||
├── RateDensityRidgeline — D3 density ridges per race×sex group
|
||
└── ExceedanceProbability — P(district > national) per student group
|
||
```
|
||
|
||
### Data Fetching Strategy (`ChartPanel.jsx`)
|
||
|
||
`ChartPanel` fetches all chart data on mount (after `LoadingAnimation` pre-fetched via batch calls):
|
||
|
||
1. **Wave data** for Charts 1–3: Fetches `unified_m3_mod` model estimates for years `['21-22', '17-18', '15-16']`.
|
||
- Uses three-year models because they return observed arrest counts across all waves (one-year models only have data for the most recent wave).
|
||
|
||
2. **Quad data** for Charts 4–6: Fetches estimates from all four quadrant models (`unified_m1_mod`, `unified_m2_mod`, `unified_m3_mod`, `unified_m4_mod`) for year `21-22` only (one year, as the most recent wave).
|
||
|
||
3. **National rates**: Loaded once from a static JSON fixture or cached by `LoadingAnimation`.
|
||
|
||
### Error Bar Convention: 90% Intervals
|
||
|
||
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**. When converting HPD bounds to standard deviations for synthetic draw generation (used in Chart 5's ridgelines), the divisor used is **3.29** (corresponding to z = 1.645 for a two-tailed 90% interval).
|
||
|
||
If you change this convention, update:
|
||
- `RateDensityRidgeline.jsx` — SD calculation (`intervalWidth / 3.29`) and label text
|
||
- `ModelDrawsComparison.jsx` — Legend labels mentioning "90%"
|
||
- Any documentation referencing confidence/credible intervals
|
||
|
||
## Common Pitfalls & Gotchas
|
||
|
||
### 1. Null Safety in Chart Components
|
||
|
||
Several API responses may return empty arrays or missing fields for districts with no arrests:
|
||
```javascript
|
||
// Always use optional chaining and defaults:
|
||
const yearRow = (quadModels[q.key] || []).find(r => r.year === year)
|
||
const predMedian = yearRow?.count_median || 0
|
||
const enroll = row.stu_enroll || 1 // Prevent division by zero
|
||
```
|
||
|
||
### 2. D3 useEffect Dependency Arrays
|
||
|
||
When using `useEffect` for D3 rendering, always include all data dependencies to prevent stale renders:
|
||
```javascript
|
||
// Correct — includes all props used inside the effect
|
||
}, [quadData, selectedModel, rateByGroup])
|
||
```
|
||
|
||
### 3. SVG Dimensions and Responsiveness
|
||
|
||
Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for wide content):
|
||
- Chart cards have `max-width: 70rem` in `ChartPanel.jsx` (vs the default text width of ~60rem)
|
||
- SVG elements should set both `width="100%"` and a fixed `viewBox` for proper scaling
|
||
|
||
### 4. API Endpoint Availability
|
||
|
||
Not all endpoints are available to browser-based clients:
|
||
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
|
||
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
|
||
|
||
### 5. Real posterior draws via duckdb-wasm
|
||
|
||
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
|
||
500-draw-per-group posterior from the public Hugging Face parquet dataset
|
||
(`civilytics/crdc-school-arrest-rates`), queried client-side with
|
||
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
|
||
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
|
||
involved. If that fetch fails (network, unsupported browser, HF outage),
|
||
both charts fall back to the `distributionApprox.js` analytic approximation
|
||
and show the "estimated shape" note — **do not delete `distributionApprox.js`
|
||
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
|
||
|
||
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
|
||
the full design.
|
||
|
||
The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.
|
||
|
||
### 6. CORS Configuration
|
||
|
||
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
|
||
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
|
||
- If unset, the app attempts direct fetch — works when served from Docker/nginx or same-origin
|
||
|
||
## Deployment Checklist
|
||
|
||
Before pushing to production:
|
||
|
||
1. **Build succeeds**: `npm run build` (check for new errors)2. **No console errors in browser** after hard refresh3. **All 6 charts render** with sample districts (test "Denver", "Mobile County")
|
||
4. **Loading animation** appears briefly, then transitions to ChartPanel
|
||
|
||
### Commit Message Convention
|
||
|
||
Use descriptive commit messages that explain the *why*, not just the what:
|
||
```bash
|
||
# Good
|
||
git commit -m "Fix: use three-year model for wave data (returns observed counts across all years)"
|
||
git commit -m "Add D3 density ridges to Chart 5 with diamond markers for observed rates"
|
||
|
||
# Avoid vague messages
|
||
git commit -m "Fix charts" # Too generic
|
||
git commit -m "Update code" # No context
|
||
```
|
||
|
||
## Testing Strategy
|
||
|
||
There are no automated tests in this project. Manual verification is required:1. **Visual check**: Load a district and verify all 6 charts render correctly2. **Error console**: Check browser DevTools for JavaScript errors3. **Data accuracy**: Compare observed values against API response (check Network tab)
|
||
4. **Responsiveness**: Resize window to ensure layout adapts
|
||
|
||
## Style Guide References
|
||
|
||
- R code style: Follows tidyverse principles (`r-style-guide` skill in agent knowledge base)- Chart aesthetic decisions should match patterns from `social_media_posts.md` and `white_paper.qmd`
|
||
- Colors, typography, spacing are defined as CSS custom properties in `src/styles/tokens.css`
|
||
|
||
## Related Repositories
|
||
|
||
- **crdc-arrests** — The API server (Plumber/R) at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/`
|
||
- **civilyticsR** — R package with wordmark and visualization functions at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/civilyticsR/`
|
||
|
||
## Git Conventions
|
||
|
||
- Remote: `https://gitea.civilytics.org/Civilytics/crdc-demo.git`
|
||
- Default branch: `main` (not `master`)
|
||
- Gitea Actions workflow auto-deploys on push to `main` via `.gitea/workflows/pages.yml`
|
||
- Always pull before making changes: `git pull origin main` |