crdc-demo/AGENTS.md

Agent Guide — CRDC Demo App

This document captures context, decisions, and guidance for agents working on this codebase. It is the primary source of truth for how to make changes safely.

Project Overview

A React + Vite static web app demonstrating the CRDC School Arrest Rate API. Visitors select a state, search for a school district, and see 6 charts comparing observed arrest data against Bayesian model estimates. Deployed via Gitea Actions to pages.civilytics.org/crdc-demo/.

Architecture Summary

• Frontend: React 19 + Vite (static site generation)
• Styling: Plain CSS custom properties matching Civilytics design tokens (src/styles/tokens.css)
• Charts: Mixed approach:
  • Charts 1–3 use inline SVG with manual scales (no D3 dependency for these)
  • Chart 4 (ModelDrawsComparison) uses D3.js v7 for data-driven rendering of quadrant comparisons
  • Chart 5 (RateDensityRidgeline) uses D3.js v7 for density ridge visualizations
• API: Calls public read-only API directly from browser; no backend required
• Deployment: Static site deployed via Gitea Actions (.gitea/workflows/pages.yml)

Key Components and Data Flow

App.jsx (router)
  → StateSelector (landing screen: state dropdown/grid)
    → DistrictSearch (search + "interesting" suggestions from /estimates?state=&year=)
      → LoadingAnimation (fetches all data in parallel, shows animated histogram grid)
        → ChartPanel (receives district object, fetches structured estimates for 6 charts)
          ├── ArrestsOverTime       — SVG line chart by wave (3 years)
          ├── RateByGroupBar         — SVG bar chart: observed vs modeled per group
          ├── DistrictVsNational     — SVG comparison to national average
          ├── ModelDrawsComparison    — D3 quadrant charts (4 model types × 1 year)
          ├── RateDensityRidgeline   — D3 density ridges per race×sex group
          └── ExceedanceProbability   — P(district > national) per student group

Data Fetching Strategy (ChartPanel.jsx)

ChartPanel fetches all chart data on mount (after LoadingAnimation pre-fetched via batch calls):

1. Wave data for Charts 1–3: Fetches unified_m3_mod model estimates for years ['21-22', '17-18', '15-16'].

   • Uses three-year models because they return observed arrest counts across all waves (one-year models only have data for the most recent wave).

2. Quad data for Charts 4–6: Fetches estimates from all four quadrant models (unified_m1_mod, unified_m2_mod, unified_m3_mod, unified_m4_mod) for year 21-22 only (one year, as the most recent wave).

3. National rates: Loaded once from a static JSON fixture or cached by LoadingAnimation.

Error Bar Convention: 90% Intervals

The API returns 95% HPD intervals (count_lower, count_upper). However, all chart labels and calculations in this app use 90% intervals.

The 90% convention is baked into the analytic fallback in src/utils/distributionApprox.js (the path used when real posterior draws can't be fetched — see the duckdb-wasm section below). That fallback generates no draws: fitSkewedInterval fits a two-piece normal directly from {median, lower, upper}, converting each half-interval to its own sigma with z = probit((1 + intervalMass) / 2) and intervalMass defaulting to 0.90 (so z ≈ 1.645). For a symmetric interval that is equivalent to the old fixed "full width / 3.29" divisor, but it is computed from intervalMass rather than hardcoded, and each side gets its own sigma so the fitted shape stays skewed.

If you change this convention, update:

• distributionApprox.js — the intervalMass = 0.90 default in fitSkewedInterval, and its JSDoc claim that the API's bounds are a 90% interval
• RateByGroupBar.jsx — the caption and component JSDoc, both of which say "the model's reported 90% interval"
• ArrestsOverTime.jsx — the legend label "Modeled (median + 90% interval)"
• Any documentation referencing confidence/credible intervals

Common Pitfalls & Gotchas

1. Null Safety in Chart Components

Several API responses may return empty arrays or missing fields for districts with no arrests:

// Always use optional chaining and defaults:
const yearRow = (quadModels[q.key] || []).find(r => r.year === year)
const predMedian = yearRow?.count_median || 0
const enroll = row.stu_enroll || 1  // Prevent division by zero

2. D3 useEffect Dependency Arrays

When using useEffect for D3 rendering, always include all data dependencies to prevent stale renders:

// Correct — includes all props used inside the effect
}, [quadData, selectedModel, rateByGroup])

3. SVG Dimensions and Responsiveness

Charts use fixed dimensions with responsive containers (overflowX: 'auto' for wide content):

• Chart cards have max-width: 70rem in ChartPanel.jsx (vs the default text width of ~60rem)
• SVG elements should set both width="100%" and a fixed viewBox for proper scaling

4. API Endpoint Availability

Not all endpoints are available to browser-based clients:

• /api/v1/estimates/{leaid} — ✅ Returns estimates summary (median, lower, upper bounds)
• /api/v1/draws?... — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app does use the real draws in the shard it points to — see below.

5. Real posterior draws via duckdb-wasm

Charts 2 (RateByGroupBar) and 3 (RateDensityRidgeline) fetch the actual 500-draw-per-group posterior from the public Hugging Face parquet dataset (civilytics/crdc-school-arrest-rates), queried client-side with @duckdb/duckdb-wasm (src/utils/duckdbClient.js + src/hooks/useDrawDistribution.js). No server-side draws endpoint is involved. If that fetch fails (network, unsupported browser, HF outage), both charts fall back to the distributionApprox.js analytic approximation and show the "estimated shape" note — do not delete distributionApprox.js or ApproxNote.jsx, they're the fallback path, not dead code.

See docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md for the full design.

The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped @duckdb/duckdb-wasm package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic import() only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.

6. CORS Configuration

The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:

• The app auto-detects proxy availability via VITE_PROXY_URL environment variable
• If unset, the app attempts direct fetch — works when served from Docker/nginx or same-origin

Deployment Checklist

Before pushing to production:

1. Build succeeds: npm run build (check for new errors)2. No console errors in browser after hard refresh3. All 6 charts render with sample districts (test "Denver", "Mobile County")
2. Loading animation appears briefly, then transitions to ChartPanel

Commit Message Convention

Use descriptive commit messages that explain the why, not just the what:

# Good
git commit -m "Fix: use three-year model for wave data (returns observed counts across all years)"
git commit -m "Add D3 density ridges to Chart 5 with diamond markers for observed rates"

# Avoid vague messages
git commit -m "Fix charts"          # Too generic
git commit -m "Update code"         # No context

Testing Strategy

There are no automated tests in this project. Manual verification is required:1. Visual check: Load a district and verify all 6 charts render correctly2. Error console: Check browser DevTools for JavaScript errors3. Data accuracy: Compare observed values against API response (check Network tab) 4. Responsiveness: Resize window to ensure layout adapts

Style Guide References

• R code style: Follows tidyverse principles (r-style-guide skill in agent knowledge base)- Chart aesthetic decisions should match patterns from social_media_posts.md and white_paper.qmd
• Colors, typography, spacing are defined as CSS custom properties in src/styles/tokens.css

Related Repositories

• crdc-arrests — The API server (Plumber/R) at /home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/
• civilyticsR — R package with wordmark and visualization functions at /home/jared/Nextcloud/Civilytics/Code/Civilytics/civilyticsR/

Git Conventions

• Remote: https://gitea.civilytics.org/Civilytics/crdc-demo.git
• Default branch: main (not master)
• Gitea Actions workflow auto-deploys on push to main via .gitea/workflows/pages.yml
• Always pull before making changes: git pull origin main