jared
db5111dab9
Deploy to git-pages / deploy (push) Successful in 22s
AGENTS.md's "Error Bar Convention" paragraph was wrong in four ways after the last docs pass: it pointed "above" at a section that is below it, described the fallback as "synthetic draw generation" when it generates no draws at all, cited an `intervalWidth / 3.29` expression in RateDensityRidgeline.jsx that does not exist, and listed ModelDrawsComparison.jsx, a file that was deleted when the demo was reduced to 3 charts. Rewritten against the current code: the fallback is fitSkewedInterval's analytic two-piece normal, whose divisor is probit((1 + intervalMass) / 2) with intervalMass defaulting to 0.90, and the "if you change this" list now names the three files that actually encode 90%. README: 6 charts -> 3, the ridgeline is Chart 3 not Chart 5, Chart 2's use of real draws is now mentioned, the D3.js v7 claim is dropped (d3 is not a dependency), @duckdb/duckdb-wasm is listed in the tech stack, the file tree matches src/, and the container-size note accounts for the wasm engine.
154 lines
8.9 KiB
Markdown
154 lines
8.9 KiB
Markdown
# Agent Guide — CRDC Demo App
|
||
|
||
This document captures context, decisions, and guidance for agents working on this codebase. It is the primary source of truth for how to make changes safely.
|
||
|
||
## Project Overview
|
||
|
||
A React + Vite static web app demonstrating the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/). Visitors select a state, search for a school district, and see 6 charts comparing observed arrest data against Bayesian model estimates. Deployed via Gitea Actions to `pages.civilytics.org/crdc-demo/`.
|
||
|
||
## Architecture Summary
|
||
|
||
- **Frontend**: React 19 + Vite (static site generation)
|
||
- **Styling**: Plain CSS custom properties matching Civilytics design tokens (`src/styles/tokens.css`)
|
||
- **Charts**: Mixed approach:
|
||
- Charts 1–3 use inline SVG with manual scales (no D3 dependency for these)
|
||
- Chart 4 (ModelDrawsComparison) uses D3.js v7 for data-driven rendering of quadrant comparisons
|
||
- Chart 5 (RateDensityRidgeline) uses D3.js v7 for density ridge visualizations
|
||
- **API**: Calls public read-only API directly from browser; no backend required
|
||
- **Deployment**: Static site deployed via Gitea Actions (`.gitea/workflows/pages.yml`)
|
||
|
||
## Key Components and Data Flow
|
||
|
||
```
|
||
App.jsx (router)
|
||
→ StateSelector (landing screen: state dropdown/grid)
|
||
→ DistrictSearch (search + "interesting" suggestions from /estimates?state=&year=)
|
||
→ LoadingAnimation (fetches all data in parallel, shows animated histogram grid)
|
||
→ ChartPanel (receives district object, fetches structured estimates for 6 charts)
|
||
├── ArrestsOverTime — SVG line chart by wave (3 years)
|
||
├── RateByGroupBar — SVG bar chart: observed vs modeled per group
|
||
├── DistrictVsNational — SVG comparison to national average
|
||
├── ModelDrawsComparison — D3 quadrant charts (4 model types × 1 year)
|
||
├── RateDensityRidgeline — D3 density ridges per race×sex group
|
||
└── ExceedanceProbability — P(district > national) per student group
|
||
```
|
||
|
||
### Data Fetching Strategy (`ChartPanel.jsx`)
|
||
|
||
`ChartPanel` fetches all chart data on mount (after `LoadingAnimation` pre-fetched via batch calls):
|
||
|
||
1. **Wave data** for Charts 1–3: Fetches `unified_m3_mod` model estimates for years `['21-22', '17-18', '15-16']`.
|
||
- Uses three-year models because they return observed arrest counts across all waves (one-year models only have data for the most recent wave).
|
||
|
||
2. **Quad data** for Charts 4–6: Fetches estimates from all four quadrant models (`unified_m1_mod`, `unified_m2_mod`, `unified_m3_mod`, `unified_m4_mod`) for year `21-22` only (one year, as the most recent wave).
|
||
|
||
3. **National rates**: Loaded once from a static JSON fixture or cached by `LoadingAnimation`.
|
||
|
||
### Error Bar Convention: 90% Intervals
|
||
|
||
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**.
|
||
|
||
The 90% convention is baked into the analytic fallback in `src/utils/distributionApprox.js` (the path used when real posterior draws can't be fetched — see the duckdb-wasm section below). That fallback generates **no draws**: `fitSkewedInterval` fits a two-piece normal directly from `{median, lower, upper}`, converting each half-interval to its own sigma with `z = probit((1 + intervalMass) / 2)` and `intervalMass` defaulting to `0.90` (so z ≈ 1.645). For a symmetric interval that is equivalent to the old fixed "full width / 3.29" divisor, but it is computed from `intervalMass` rather than hardcoded, and each side gets its own sigma so the fitted shape stays skewed.
|
||
|
||
If you change this convention, update:
|
||
- `distributionApprox.js` — the `intervalMass = 0.90` default in `fitSkewedInterval`, and its JSDoc claim that the API's bounds are a 90% interval
|
||
- `RateByGroupBar.jsx` — the caption and component JSDoc, both of which say "the model's reported 90% interval"
|
||
- `ArrestsOverTime.jsx` — the legend label "Modeled (median + 90% interval)"
|
||
- Any documentation referencing confidence/credible intervals
|
||
|
||
## Common Pitfalls & Gotchas
|
||
|
||
### 1. Null Safety in Chart Components
|
||
|
||
Several API responses may return empty arrays or missing fields for districts with no arrests:
|
||
```javascript
|
||
// Always use optional chaining and defaults:
|
||
const yearRow = (quadModels[q.key] || []).find(r => r.year === year)
|
||
const predMedian = yearRow?.count_median || 0
|
||
const enroll = row.stu_enroll || 1 // Prevent division by zero
|
||
```
|
||
|
||
### 2. D3 useEffect Dependency Arrays
|
||
|
||
When using `useEffect` for D3 rendering, always include all data dependencies to prevent stale renders:
|
||
```javascript
|
||
// Correct — includes all props used inside the effect
|
||
}, [quadData, selectedModel, rateByGroup])
|
||
```
|
||
|
||
### 3. SVG Dimensions and Responsiveness
|
||
|
||
Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for wide content):
|
||
- Chart cards have `max-width: 70rem` in `ChartPanel.jsx` (vs the default text width of ~60rem)
|
||
- SVG elements should set both `width="100%"` and a fixed `viewBox` for proper scaling
|
||
|
||
### 4. API Endpoint Availability
|
||
|
||
Not all endpoints are available to browser-based clients:
|
||
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
|
||
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
|
||
|
||
### 5. Real posterior draws via duckdb-wasm
|
||
|
||
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
|
||
500-draw-per-group posterior from the public Hugging Face parquet dataset
|
||
(`civilytics/crdc-school-arrest-rates`), queried client-side with
|
||
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
|
||
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
|
||
involved. If that fetch fails (network, unsupported browser, HF outage),
|
||
both charts fall back to the `distributionApprox.js` analytic approximation
|
||
and show the "estimated shape" note — **do not delete `distributionApprox.js`
|
||
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
|
||
|
||
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
|
||
the full design.
|
||
|
||
The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.
|
||
|
||
### 6. CORS Configuration
|
||
|
||
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
|
||
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
|
||
- If unset, the app attempts direct fetch — works when served from Docker/nginx or same-origin
|
||
|
||
## Deployment Checklist
|
||
|
||
Before pushing to production:
|
||
|
||
1. **Build succeeds**: `npm run build` (check for new errors)2. **No console errors in browser** after hard refresh3. **All 6 charts render** with sample districts (test "Denver", "Mobile County")
|
||
4. **Loading animation** appears briefly, then transitions to ChartPanel
|
||
|
||
### Commit Message Convention
|
||
|
||
Use descriptive commit messages that explain the *why*, not just the what:
|
||
```bash
|
||
# Good
|
||
git commit -m "Fix: use three-year model for wave data (returns observed counts across all years)"
|
||
git commit -m "Add D3 density ridges to Chart 5 with diamond markers for observed rates"
|
||
|
||
# Avoid vague messages
|
||
git commit -m "Fix charts" # Too generic
|
||
git commit -m "Update code" # No context
|
||
```
|
||
|
||
## Testing Strategy
|
||
|
||
There are no automated tests in this project. Manual verification is required:1. **Visual check**: Load a district and verify all 6 charts render correctly2. **Error console**: Check browser DevTools for JavaScript errors3. **Data accuracy**: Compare observed values against API response (check Network tab)
|
||
4. **Responsiveness**: Resize window to ensure layout adapts
|
||
|
||
## Style Guide References
|
||
|
||
- R code style: Follows tidyverse principles (`r-style-guide` skill in agent knowledge base)- Chart aesthetic decisions should match patterns from `social_media_posts.md` and `white_paper.qmd`
|
||
- Colors, typography, spacing are defined as CSS custom properties in `src/styles/tokens.css`
|
||
|
||
## Related Repositories
|
||
|
||
- **crdc-arrests** — The API server (Plumber/R) at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/`
|
||
- **civilyticsR** — R package with wordmark and visualization functions at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/civilyticsR/`
|
||
|
||
## Git Conventions
|
||
|
||
- Remote: `https://gitea.civilytics.org/Civilytics/crdc-demo.git`
|
||
- Default branch: `main` (not `master`)
|
||
- Gitea Actions workflow auto-deploys on push to `main` via `.gitea/workflows/pages.yml`
|
||
- Always pull before making changes: `git pull origin main` |