Civilytics/crdc-demo
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refresh AGENTS.md, HANDOFF.md, and README with recent work

Browse Source 
- Create AGENTS.md: comprehensive agent guide covering architecture,
  data flow, error bar convention (90% intervals), D3 usage patterns,
  null safety pitfalls, deployment checklist, and related repos
- Update HANDOFF.md: mark CORS as resolved, document chart improvements
  (Chart 5 ridgeline rewrite with synthetic draws + smooth rendering)
- Refresh README.md: accurate tech stack (D3 v7), updated file structure
  with new chart files, corrected API endpoint table
This commit is contained in:
jared
2026-08-10 17:10:35 -04:00
parent 3fe004356e
commit 6c8e1e8409
3 changed files with 184 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+136
View File
@@ -0,0 +1,136 @@
# Agent Guide — CRDC Demo App
This document captures context, decisions, and guidance for agents working on this codebase. It is the primary source of truth for how to make changes safely.
## Project Overview
A React + Vite static web app demonstrating the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/). Visitors select a state, search for a school district, and see 6 charts comparing observed arrest data against Bayesian model estimates. Deployed via Gitea Actions to `pages.civilytics.org/crdc-demo/`.
## Architecture Summary
- **Frontend**: React 19 + Vite (static site generation)
- **Styling**: Plain CSS custom properties matching Civilytics design tokens (`src/styles/tokens.css`)
- **Charts**: Mixed approach:
- Charts 1–3 use inline SVG with manual scales (no D3 dependency for these)
- Chart 4 (ModelDrawsComparison) uses D3.js v7 for data-driven rendering of quadrant comparisons
- Chart 5 (RateDensityRidgeline) uses D3.js v7 for density ridge visualizations
- **API**: Calls public read-only API directly from browser; no backend required
- **Deployment**: Static site deployed via Gitea Actions (`.gitea/workflows/pages.yml`)
## Key Components and Data Flow
```
App.jsx (router)
→ StateSelector (landing screen: state dropdown/grid)
→ DistrictSearch (search + "interesting" suggestions from /estimates?state=&year=)
→ LoadingAnimation (fetches all data in parallel, shows animated histogram grid)
→ ChartPanel (receives district object, fetches structured estimates for 6 charts)
├── ArrestsOverTime — SVG line chart by wave (3 years)
├── RateByGroupBar — SVG bar chart: observed vs modeled per group
├── DistrictVsNational — SVG comparison to national average
├── ModelDrawsComparison — D3 quadrant charts (4 model types × 1 year)
├── RateDensityRidgeline — D3 density ridges per race×sex group
└── ExceedanceProbability — P(district > national) per student group
```
### Data Fetching Strategy (`ChartPanel.jsx`)
`ChartPanel` fetches all chart data on mount (after `LoadingAnimation` pre-fetched via batch calls):
1. **Wave data** for Charts 1–3: Fetches `unified_m3_mod` model estimates for years `['21-22', '17-18', '15-16']`.
- Uses three-year models because they return observed arrest counts across all waves (one-year models only have data for the most recent wave).
2. **Quad data** for Charts 4–6: Fetches estimates from all four quadrant models (`unified_m1_mod`, `unified_m2_mod`, `unified_m3_mod`, `unified_m4_mod`) for year `21-22` only (one year, as the most recent wave).
3. **National rates**: Loaded once from a static JSON fixture or cached by `LoadingAnimation`.
### Error Bar Convention: 90% Intervals
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**. When converting HPD bounds to standard deviations for synthetic draw generation (used in Chart 5's ridgelines), the divisor used is **3.29** (corresponding to z = 1.645 for a two-tailed 90% interval).
If you change this convention, update:
- `RateDensityRidgeline.jsx` — SD calculation (`intervalWidth / 3.29`) and label text
- `ModelDrawsComparison.jsx` — Legend labels mentioning "90%"
- Any documentation referencing confidence/credible intervals
## Common Pitfalls & Gotchas
### 1. Null Safety in Chart Components
Several API responses may return empty arrays or missing fields for districts with no arrests:
```javascript
// Always use optional chaining and defaults:
const yearRow = (quadModels[q.key] || []).find(r => r.year === year)
const predMedian = yearRow?.count_median || 0
const enroll = row.stu_enroll || 1 // Prevent division by zero
```
### 2. D3 useEffect Dependency Arrays
When using `useEffect` for D3 rendering, always include all data dependencies to prevent stale renders:
```javascript
// Correct — includes all props used inside the effect
}, [quadData, selectedModel, rateByGroup])
```
### 3. SVG Dimensions and Responsiveness
Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for wide content):
- Chart cards have `max-width: 70rem` in `ChartPanel.jsx` (vs the default text width of ~60rem)
- SVG elements should set both `width="100%"` and a fixed `viewBox` for proper scaling
### 4. API Endpoint Availability
Not all endpoints are available to browser-based clients:
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser
If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds.
### 5. CORS Configuration
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
- If unset, the app attempts direct fetch — works when served from Docker/nginx or same-origin
## Deployment Checklist
Before pushing to production:
1. **Build succeeds**: `npm run build` (check for new errors)2. **No console errors in browser** after hard refresh3. **All 6 charts render** with sample districts (test "Denver", "Mobile County")
4. **Loading animation** appears briefly, then transitions to ChartPanel
### Commit Message Convention
Use descriptive commit messages that explain the *why*, not just the what:
```bash
# Good
git commit -m "Fix: use three-year model for wave data (returns observed counts across all years)"
git commit -m "Add D3 density ridges to Chart 5 with diamond markers for observed rates"
# Avoid vague messages
git commit -m "Fix charts" # Too generic
git commit -m "Update code" # No context
```
## Testing Strategy
There are no automated tests in this project. Manual verification is required:1. **Visual check**: Load a district and verify all 6 charts render correctly2. **Error console**: Check browser DevTools for JavaScript errors3. **Data accuracy**: Compare observed values against API response (check Network tab)
4. **Responsiveness**: Resize window to ensure layout adapts
## Style Guide References
- R code style: Follows tidyverse principles (`r-style-guide` skill in agent knowledge base)- Chart aesthetic decisions should match patterns from `social_media_posts.md` and `white_paper.qmd`
- Colors, typography, spacing are defined as CSS custom properties in `src/styles/tokens.css`
## Related Repositories
- **crdc-arrests** — The API server (Plumber/R) at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/`
- **civilyticsR** — R package with wordmark and visualization functions at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/civilyticsR/`
## Git Conventions
- Remote: `https://gitea.civilytics.org/Civilytics/crdc-demo.git`
- Default branch: `main` (not `master`)
- Gitea Actions workflow auto-deploys on push to `main` via `.gitea/workflows/pages.yml`
- Always pull before making changes: `git pull origin main`
HANDOFF.md
+30 -15
View File
@@ -1,25 +1,40 @@
# Handoff: Fix CORS for CRDC Arrests API Demo App
## Current Status
The demo app at `https://pages.civilytics.org/crdc-demo/` loads HTML correctly but shows a blank white screen because browser-based fetch requests to the CRDC API are blocked by CORS (the API doesn't send `Access-Control-Allow-Origin` headers).
**RESOLVED**: The demo app at `https://pages.civilytics.org/crdc-demo/` is fully functional. All charts render correctly and there are no blocking errors in the console.
## What's Done
1. ✅ **App code**: React + Vite app fully built with 6 charts, Civilytics visual identity, loading animation, district search — all committed and building cleanly
2. ✅ **Git Pages deployment**: Gitea Actions workflow (`.gitea/workflows/pages.yml`) successfully builds `dist/` and deploys to a `pages` branch on `Civilytics/crdc-demo` repo at gitea.civilytics.org
3. ✅ **CORS proxy code added**:
- Updated `src/hooks/useApi.js`, `LoadingAnimation.jsx`, `ChartPanel.jsx` with `VITE_PROXY_URL` support (routes through a query-param-based proxy)
- Added `proxy.php` — simple PHP CORS proxy for git-pages hosts that support PHP
- Updated Dockerfile nginx.conf to include `/api/v1/` reverse proxy with CORS headers
The original CORS issue was resolved by deploying updated `plumber.R` with CORS headers to the running API server (`crdc-api.civilytics.org`).
## Historical Context (resolved)
### Original Problem
The demo app loaded HTML correctly but showed a blank white screen because browser-based fetch requests to the CRDC API were blocked by CORS (the API didn't send `Access-Control-Allow-Origin` headers).
### Resolution
The CORS issue was fixed at the API source:
- Updated `crdc-arrests/api/plumber.R` with CORS headers (`Access-Control-Allow-Origin: *`) in the existing `cacheHeaders` filter and OPTIONS preflight handling.
- Deployed to the running API server so it took effect at https://crdc-api.civilytics.org/.
The app now calls the public read-only API directly from the browser without requiring a proxy.
## Current Work (completed)
Since resolving CORS, additional improvements were made:
### What's Done
1. ✅ **CORS resolved**: API server (`crdc-api.civilytics.org`) now sends `Access-Control-Allow-Origin: *` headers — deployed via updated `plumber.R`
2. ✅ **App code**: React + Vite app fully built with 6 charts, Civilytics visual identity (wordmark from civilyticsR package), loading animation, district search
3. ✅ **Git Pages deployment**: Gitea Actions workflow (`.gitea/workflows/pages.yml`) successfully builds `dist/` and deploys to a `pages` branch on `Civilytics/crdc-demo` repo at gitea.civilytics.org
4. ✅ **Chart 5 rewrite**: Replaced interval-proxy ridgeline with proper D3 density ridges (`RateDensityRidgeline.jsx`) using synthetic draws from normal approximation of posterior intervals, smooth `curveBasis` rendering, and diamond markers for observed rates
5. ✅ **Chart 4 enhancement**: Added D3-based quadrant visualization in `ModelDrawsComparison.jsx` showing error bars with proper null safety
### Remaining Proxy Code (optional)
The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PROXY_URL` support) was added but is **not needed** since the API now sends CORS headers. It remains as a safety net for environments where the API can't be modified.
## What Needs to Be Done Next
### Option A: Fix at the API source (recommended, cleanest)
The CRDC API is a Plumber/R service deployed behind Cloudflare. The code change has already been made in `crdc-arrests/api/plumber.R` — added CORS headers (`Access-Control-Allow-Origin: *`) to the existing `cacheHeaders` filter and OPTIONS preflight handling.
**Action needed**: Deploy this updated plumber.R to the running API server so it takes effect at https://crdc-api.civilytics.org/. The change is in `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/api/plumber.R` (lines 56-78).
### Option B: Fix on the git-pages server
If you can't restart the API, configure the nginx serving `pages.civilytics.org` to add CORS headers when proxying requests to `crdc-api.civilytics.org`. Or deploy `proxy.php` and set `VITE_PROXY_URL=/crdc-demo/proxy.php` at build time.
### Future Enhancements
- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required.
- **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling.
## Key Files
- **API fix**: `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/api/plumber.R` (CORS headers added to cacheHeaders filter)
README.md
+18 -15
View File
@@ -20,19 +20,20 @@ npm run preview # serve built files locally
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 6 charts comparing observed data against Bayesian model estimates:
1. **Arrests over time** — raw counts by CRDC wave with per-1k rate labels
2. **Rate by student group** — bar chart, most recent year (observed vs. modeled)
3. **District vs. national** — highest-rate group compared to the U.S. average
4. **Model predictions vs. observed** — four quadrants (one-year/three-year × baseline/covariate)
5. **Observed rate vs. model distribution** — ridgeline-style comparison across models
6. **Exceedance probability** — P(district > national) per student group
1. **Arrests over time** (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG
2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG
3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG
4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate)
5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates
6. **Exceedance probability** (Chart 6) — P(district > national) per student group
## Architecture
### Tech Stack
- **React 19** + **Vite** (static site generation, no backend required)
- Plain CSS custom properties for styling (matches Civilytics design tokens exactly)
- SVG-based charts rendered inline (no D3 or charting library dependencies)
- D3.js v7 for data-driven visualizations (density ridges, scales, axes)
- Inline SVG rendering — charts are built with vanilla DOM/D3, not charting libraries
- Calls the public read-only API directly from the browser
### File Structure
@@ -40,6 +41,8 @@ Visitors select a U.S. state, search for a school district (with suggestions of
crdc-demo/
├── index.html # Entry point
├── vite.config.mjs # Vite build config
├── public/ # Static assets (wordmark, favicon)
│ └── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
├── src/
│ ├── main.jsx # React entry
│ ├── App.jsx # Main router (state → search → loading → charts)
@@ -50,18 +53,17 @@ crdc-demo/
│ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch
│ │ └── ChartPanel.jsx # Orchestrates all 6 charts + data fetching
│ ├── charts/
│ │ ├── ArrestsOverTime.jsx # Chart 1 — line chart by wave
│ │ ├── RateByGroupBar.jsx # Chart 2 — bar chart by group
│ │ ├── DistrictVsNational.jsx # Chart 3 — comparison vs. national avg
│ │ ├── ModelDrawsComparison.jsx # Chart 4 — quadrant model comparison
│ │ ├── ObservedRateDensity.jsx # Chart 5 — ridgeline proxy from intervals
│ │ └── ExceedanceProbability.jsx # Chart 6 — P(district > nat) per group
│ │ ├── ArrestsOverTime.jsx # Chart 1 — line chart by wave (SVG)
│ │ ├── RateByGroupBar.jsx # Chart 2 — bar chart by group (SVG)
│ │ ├── DistrictVsNational.jsx # Chart 3 — comparison vs. national avg
│ │ ├── ModelDrawsComparison.jsx # Chart 4 — quadrant model comparison (D3)
│ │ ├── RateDensityRidgeline.jsx # Chart 5 — density ridges per student group (D3)
│ │ └── ExceedanceProbability.jsx # Chart 6 — P(district > nat) per group
│ ├── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
│ └── data/national_rates.json # Static national rates fixture for comparisons
├── Dockerfile # Multi-stage build → nginx static server
├── docker-compose.yml # Local dev / self-hosted deployment
├── _config.yml # Git Pages (self-hosted equivalent) config
└── .github/workflows/deploy.yml # CI/CD — builds and uploads artifacts
└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch
```
### API Endpoints Used
@@ -70,6 +72,7 @@ crdc-demo/
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district |
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app |
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
## Deployment