From 6c8e1e8409ba0cea95bc99b29873890fb6b04f44 Mon Sep 17 00:00:00 2001 From: Jared Knowles Date: Mon, 10 Aug 2026 17:10:35 -0400 Subject: [PATCH] docs: refresh AGENTS.md, HANDOFF.md, and README with recent work - Create AGENTS.md: comprehensive agent guide covering architecture, data flow, error bar convention (90% intervals), D3 usage patterns, null safety pitfalls, deployment checklist, and related repos - Update HANDOFF.md: mark CORS as resolved, document chart improvements (Chart 5 ridgeline rewrite with synthetic draws + smooth rendering) - Refresh README.md: accurate tech stack (D3 v7), updated file structure with new chart files, corrected API endpoint table --- AGENTS.md | 136 +++++++++++++++++++++++++++++++++++++++++++++++++++++ HANDOFF.md | 45 ++++++++++++------ README.md | 33 +++++++------ 3 files changed, 184 insertions(+), 30 deletions(-) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..a0cf7f1 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,136 @@ +# Agent Guide — CRDC Demo App + +This document captures context, decisions, and guidance for agents working on this codebase. It is the primary source of truth for how to make changes safely. + +## Project Overview + +A React + Vite static web app demonstrating the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/). Visitors select a state, search for a school district, and see 6 charts comparing observed arrest data against Bayesian model estimates. Deployed via Gitea Actions to `pages.civilytics.org/crdc-demo/`. + +## Architecture Summary + +- **Frontend**: React 19 + Vite (static site generation) +- **Styling**: Plain CSS custom properties matching Civilytics design tokens (`src/styles/tokens.css`) +- **Charts**: Mixed approach: + - Charts 1–3 use inline SVG with manual scales (no D3 dependency for these) + - Chart 4 (ModelDrawsComparison) uses D3.js v7 for data-driven rendering of quadrant comparisons + - Chart 5 (RateDensityRidgeline) uses D3.js v7 for density ridge visualizations +- **API**: Calls public read-only API directly from browser; no backend required +- **Deployment**: Static site deployed via Gitea Actions (`.gitea/workflows/pages.yml`) + +## Key Components and Data Flow + +``` +App.jsx (router) + → StateSelector (landing screen: state dropdown/grid) + → DistrictSearch (search + "interesting" suggestions from /estimates?state=&year=) + → LoadingAnimation (fetches all data in parallel, shows animated histogram grid) + → ChartPanel (receives district object, fetches structured estimates for 6 charts) + ├── ArrestsOverTime — SVG line chart by wave (3 years) + ├── RateByGroupBar — SVG bar chart: observed vs modeled per group + ├── DistrictVsNational — SVG comparison to national average + ├── ModelDrawsComparison — D3 quadrant charts (4 model types × 1 year) + ├── RateDensityRidgeline — D3 density ridges per race×sex group + └── ExceedanceProbability — P(district > national) per student group +``` + +### Data Fetching Strategy (`ChartPanel.jsx`) + +`ChartPanel` fetches all chart data on mount (after `LoadingAnimation` pre-fetched via batch calls): + +1. **Wave data** for Charts 1–3: Fetches `unified_m3_mod` model estimates for years `['21-22', '17-18', '15-16']`. + - Uses three-year models because they return observed arrest counts across all waves (one-year models only have data for the most recent wave). + +2. **Quad data** for Charts 4–6: Fetches estimates from all four quadrant models (`unified_m1_mod`, `unified_m2_mod`, `unified_m3_mod`, `unified_m4_mod`) for year `21-22` only (one year, as the most recent wave). + +3. **National rates**: Loaded once from a static JSON fixture or cached by `LoadingAnimation`. + +### Error Bar Convention: 90% Intervals + +The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**. When converting HPD bounds to standard deviations for synthetic draw generation (used in Chart 5's ridgelines), the divisor used is **3.29** (corresponding to z = 1.645 for a two-tailed 90% interval). + +If you change this convention, update: +- `RateDensityRidgeline.jsx` — SD calculation (`intervalWidth / 3.29`) and label text +- `ModelDrawsComparison.jsx` — Legend labels mentioning "90%" +- Any documentation referencing confidence/credible intervals + +## Common Pitfalls & Gotchas + +### 1. Null Safety in Chart Components + +Several API responses may return empty arrays or missing fields for districts with no arrests: +```javascript +// Always use optional chaining and defaults: +const yearRow = (quadModels[q.key] || []).find(r => r.year === year) +const predMedian = yearRow?.count_median || 0 +const enroll = row.stu_enroll || 1 // Prevent division by zero +``` + +### 2. D3 useEffect Dependency Arrays + +When using `useEffect` for D3 rendering, always include all data dependencies to prevent stale renders: +```javascript +// Correct — includes all props used inside the effect +}, [quadData, selectedModel, rateByGroup]) +``` + +### 3. SVG Dimensions and Responsiveness + +Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for wide content): +- Chart cards have `max-width: 70rem` in `ChartPanel.jsx` (vs the default text width of ~60rem) +- SVG elements should set both `width="100%"` and a fixed `viewBox` for proper scaling + +### 4. API Endpoint Availability + +Not all endpoints are available to browser-based clients: +- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds) +- `/api/v1/draws?...` — ❌ Returns Parquet shard URL only; meant for bulk processing via DuckDB, **not** usable from the browser + +If you need raw posterior draws in the browser app, a new API endpoint would be required. Currently, Chart 5 generates synthetic draws using normal approximation (`d3.randomNormal`) based on the interval bounds. + +### 5. CORS Configuration + +The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured: +- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable +- If unset, the app attempts direct fetch — works when served from Docker/nginx or same-origin + +## Deployment Checklist + +Before pushing to production: + +1. **Build succeeds**: `npm run build` (check for new errors)2. **No console errors in browser** after hard refresh3. **All 6 charts render** with sample districts (test "Denver", "Mobile County") +4. **Loading animation** appears briefly, then transitions to ChartPanel + +### Commit Message Convention + +Use descriptive commit messages that explain the *why*, not just the what: +```bash +# Good +git commit -m "Fix: use three-year model for wave data (returns observed counts across all years)" +git commit -m "Add D3 density ridges to Chart 5 with diamond markers for observed rates" + +# Avoid vague messages +git commit -m "Fix charts" # Too generic +git commit -m "Update code" # No context +``` + +## Testing Strategy + +There are no automated tests in this project. Manual verification is required:1. **Visual check**: Load a district and verify all 6 charts render correctly2. **Error console**: Check browser DevTools for JavaScript errors3. **Data accuracy**: Compare observed values against API response (check Network tab) +4. **Responsiveness**: Resize window to ensure layout adapts + +## Style Guide References + +- R code style: Follows tidyverse principles (`r-style-guide` skill in agent knowledge base)- Chart aesthetic decisions should match patterns from `social_media_posts.md` and `white_paper.qmd` +- Colors, typography, spacing are defined as CSS custom properties in `src/styles/tokens.css` + +## Related Repositories + +- **crdc-arrests** — The API server (Plumber/R) at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/` +- **civilyticsR** — R package with wordmark and visualization functions at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/civilyticsR/` + +## Git Conventions + +- Remote: `https://gitea.civilytics.org/Civilytics/crdc-demo.git` +- Default branch: `main` (not `master`) +- Gitea Actions workflow auto-deploys on push to `main` via `.gitea/workflows/pages.yml` +- Always pull before making changes: `git pull origin main` \ No newline at end of file diff --git a/HANDOFF.md b/HANDOFF.md index b9bc400..206b8e0 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,25 +1,40 @@ # Handoff: Fix CORS for CRDC Arrests API Demo App ## Current Status -The demo app at `https://pages.civilytics.org/crdc-demo/` loads HTML correctly but shows a blank white screen because browser-based fetch requests to the CRDC API are blocked by CORS (the API doesn't send `Access-Control-Allow-Origin` headers). +**RESOLVED**: The demo app at `https://pages.civilytics.org/crdc-demo/` is fully functional. All charts render correctly and there are no blocking errors in the console. -## What's Done -1. ✅ **App code**: React + Vite app fully built with 6 charts, Civilytics visual identity, loading animation, district search — all committed and building cleanly -2. ✅ **Git Pages deployment**: Gitea Actions workflow (`.gitea/workflows/pages.yml`) successfully builds `dist/` and deploys to a `pages` branch on `Civilytics/crdc-demo` repo at gitea.civilytics.org -3. ✅ **CORS proxy code added**: - - Updated `src/hooks/useApi.js`, `LoadingAnimation.jsx`, `ChartPanel.jsx` with `VITE_PROXY_URL` support (routes through a query-param-based proxy) - - Added `proxy.php` — simple PHP CORS proxy for git-pages hosts that support PHP - - Updated Dockerfile nginx.conf to include `/api/v1/` reverse proxy with CORS headers +The original CORS issue was resolved by deploying updated `plumber.R` with CORS headers to the running API server (`crdc-api.civilytics.org`). + +## Historical Context (resolved) + +### Original Problem +The demo app loaded HTML correctly but showed a blank white screen because browser-based fetch requests to the CRDC API were blocked by CORS (the API didn't send `Access-Control-Allow-Origin` headers). + +### Resolution +The CORS issue was fixed at the API source: +- Updated `crdc-arrests/api/plumber.R` with CORS headers (`Access-Control-Allow-Origin: *`) in the existing `cacheHeaders` filter and OPTIONS preflight handling. +- Deployed to the running API server so it took effect at https://crdc-api.civilytics.org/. + +The app now calls the public read-only API directly from the browser without requiring a proxy. + +## Current Work (completed) + +Since resolving CORS, additional improvements were made: +### What's Done +1. ✅ **CORS resolved**: API server (`crdc-api.civilytics.org`) now sends `Access-Control-Allow-Origin: *` headers — deployed via updated `plumber.R` +2. ✅ **App code**: React + Vite app fully built with 6 charts, Civilytics visual identity (wordmark from civilyticsR package), loading animation, district search +3. ✅ **Git Pages deployment**: Gitea Actions workflow (`.gitea/workflows/pages.yml`) successfully builds `dist/` and deploys to a `pages` branch on `Civilytics/crdc-demo` repo at gitea.civilytics.org +4. ✅ **Chart 5 rewrite**: Replaced interval-proxy ridgeline with proper D3 density ridges (`RateDensityRidgeline.jsx`) using synthetic draws from normal approximation of posterior intervals, smooth `curveBasis` rendering, and diamond markers for observed rates +5. ✅ **Chart 4 enhancement**: Added D3-based quadrant visualization in `ModelDrawsComparison.jsx` showing error bars with proper null safety + +### Remaining Proxy Code (optional) +The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PROXY_URL` support) was added but is **not needed** since the API now sends CORS headers. It remains as a safety net for environments where the API can't be modified. ## What Needs to Be Done Next -### Option A: Fix at the API source (recommended, cleanest) -The CRDC API is a Plumber/R service deployed behind Cloudflare. The code change has already been made in `crdc-arrests/api/plumber.R` — added CORS headers (`Access-Control-Allow-Origin: *`) to the existing `cacheHeaders` filter and OPTIONS preflight handling. - -**Action needed**: Deploy this updated plumber.R to the running API server so it takes effect at https://crdc-api.civilytics.org/. The change is in `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/api/plumber.R` (lines 56-78). - -### Option B: Fix on the git-pages server -If you can't restart the API, configure the nginx serving `pages.civilytics.org` to add CORS headers when proxying requests to `crdc-api.civilytics.org`. Or deploy `proxy.php` and set `VITE_PROXY_URL=/crdc-demo/proxy.php` at build time. +### Future Enhancements +- **Raw posterior draws**: The `/api/v1/draws?...` endpoint returns Parquet shard URLs for bulk processing, not browser-friendly draw data. If actual posterior distributions are needed in Chart 5 (instead of synthetic normal approximation), a new API endpoint would be required. +- **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling. ## Key Files - **API fix**: `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/api/plumber.R` (CORS headers added to cacheHeaders filter) diff --git a/README.md b/README.md index 8c5a765..81d77f3 100644 --- a/README.md +++ b/README.md @@ -20,19 +20,20 @@ npm run preview # serve built files locally Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 6 charts comparing observed data against Bayesian model estimates: -1. **Arrests over time** — raw counts by CRDC wave with per-1k rate labels -2. **Rate by student group** — bar chart, most recent year (observed vs. modeled) -3. **District vs. national** — highest-rate group compared to the U.S. average -4. **Model predictions vs. observed** — four quadrants (one-year/three-year × baseline/covariate) -5. **Observed rate vs. model distribution** — ridgeline-style comparison across models -6. **Exceedance probability** — P(district > national) per student group +1. **Arrests over time** (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG +2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG +3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG +4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate) +5. **Predicted rates by student group** (Chart 5) — D3 density ridges showing posterior distributions with diamond markers for observed rates +6. **Exceedance probability** (Chart 6) — P(district > national) per student group ## Architecture ### Tech Stack - **React 19** + **Vite** (static site generation, no backend required) - Plain CSS custom properties for styling (matches Civilytics design tokens exactly) -- SVG-based charts rendered inline (no D3 or charting library dependencies) +- D3.js v7 for data-driven visualizations (density ridges, scales, axes) +- Inline SVG rendering — charts are built with vanilla DOM/D3, not charting libraries - Calls the public read-only API directly from the browser ### File Structure @@ -40,6 +41,8 @@ Visitors select a U.S. state, search for a school district (with suggestions of crdc-demo/ ├── index.html # Entry point ├── vite.config.mjs # Vite build config +├── public/ # Static assets (wordmark, favicon) +│ └── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package ├── src/ │ ├── main.jsx # React entry │ ├── App.jsx # Main router (state → search → loading → charts) @@ -50,18 +53,17 @@ crdc-demo/ │ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch │ │ └── ChartPanel.jsx # Orchestrates all 6 charts + data fetching │ ├── charts/ -│ │ ├── ArrestsOverTime.jsx # Chart 1 — line chart by wave -│ │ ├── RateByGroupBar.jsx # Chart 2 — bar chart by group -│ │ ├── DistrictVsNational.jsx # Chart 3 — comparison vs. national avg -│ │ ├── ModelDrawsComparison.jsx # Chart 4 — quadrant model comparison -│ │ ├── ObservedRateDensity.jsx # Chart 5 — ridgeline proxy from intervals -│ │ └── ExceedanceProbability.jsx # Chart 6 — P(district > nat) per group +│ │ ├── ArrestsOverTime.jsx # Chart 1 — line chart by wave (SVG) +│ │ ├── RateByGroupBar.jsx # Chart 2 — bar chart by group (SVG) +│ │ ├── DistrictVsNational.jsx # Chart 3 — comparison vs. national avg +│ │ ├── ModelDrawsComparison.jsx # Chart 4 — quadrant model comparison (D3) +│ │ ├── RateDensityRidgeline.jsx # Chart 5 — density ridges per student group (D3) +│ │ └── ExceedanceProbability.jsx # Chart 6 — P(district > nat) per group │ ├── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing) │ └── data/national_rates.json # Static national rates fixture for comparisons ├── Dockerfile # Multi-stage build → nginx static server ├── docker-compose.yml # Local dev / self-hosted deployment -├── _config.yml # Git Pages (self-hosted equivalent) config -└── .github/workflows/deploy.yml # CI/CD — builds and uploads artifacts +└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch ``` ### API Endpoints Used @@ -70,6 +72,7 @@ crdc-demo/ | `/api/v1/models` | List available Bayesian model specs | Once (cached) | | `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke | | `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district | +| `/api/v1/draws?...` | Locate raw-posterior Parquet shard (bulk only, not used in browser) | Not called from app | | `/data/national_rates.json` | Static national rates fixture (committed) | Once per session | ## Deployment