jared
4396a56a4a
Deploy to git-pages / deploy (push) Successful in 23s
AGENTS.md described 6 charts, D3 selections, and DistrictVsNational / ModelDrawsComparison / ExceedanceProbability — none of which exist. An agent reading it as authoritative would have been actively misled, so it now opens by saying src/ wins any disagreement. Rewritten: the real component tree and data flow, ChartPanel as the owner of all cross-chart state, the pooled/unpooled key namespace trap, the four properties of the draws pipeline that are easy to break, the multi-part shard layout and why discovery uses the tree listing, the "posterior predictive draws" wording rule and the reason for it, the 95% convention, and a table of which tuning decisions carry stated rationale and should not be re-derived (the CVD-validated palette, the KDE bandwidth clamp, the pooling threshold, the mass/KDE cutoff, the axis cap). Adds a testing section — there are automated tests now — and notes that the deployment check should use a multi-part state like California, since a single-part state cannot catch a regression in part discovery. Records that App.jsx's "Search another district" is a full page reload that discards the shard cache. README: the chart list becomes the summary table plus the two ported figures, the file tree matches src/, the d3 role is stated precisely (scales and paths, not selections), and the endpoint table warns that /estimates?state= ranks by LEAID on a short read. Also commits the plan this work followed.
163 lines
11 KiB
Markdown
163 lines
11 KiB
Markdown
# CRDC Arrests API Demo
|
||
|
||
A demonstration web application for the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/), showing Bayesian model comparisons of school-based arrest rates across U.S. districts. Built with React + Vite, deployable as a static site or Docker container.
|
||
|
||
## Quick Start (Development)
|
||
|
||
```bash
|
||
# Install dependencies
|
||
npm ci
|
||
|
||
# Start dev server on localhost:5173
|
||
npm run dev
|
||
|
||
# Build for production
|
||
npm run build # outputs to dist/
|
||
npm run preview # serve built files locally
|
||
```
|
||
|
||
## What It Does
|
||
|
||
Visitors select a U.S. state, search for a school district (with suggestions of the districts reporting the most arrests), and see what was actually reported alongside what the Bayesian models estimate. The results page ports the white paper's Figs 6 and 7 (`wp_fig_group_density` / `wp_fig_group_difference`):
|
||
|
||
1. **Reported arrests and enrollment** — a summary table, one row per student group plus a district total: students, observed arrests, and rate per 1,000. Its checkboxes double as the legend and the group control for the density panel. In sparse districts (fewer than 20 arrests district-wide) Female and Male are pooled within each race, with a banner explaining the rule and a switch to override it.
|
||
2. **Arrest rate probability density** — each selected group's posterior predictive distribution as a filled area, Female over Male sharing one axis, direct-labelled at the peak. Beneath each panel, a rail of 95% Agresti–Coull point ranges for the observed rate. A segmented control switches between the four Bayesian specifications; an opt-in toggle compares all four at once. Draws that take only a handful of distinct values are drawn as discrete probability mass rather than smoothed — in a small district the posterior predictive genuinely *is* discrete.
|
||
3. **Model estimated differences** — the posterior of Δ = rate(A) − rate(B) per 1,000, computed at each draw, filled with a diverging ramp centred at zero, with a dashed rule at no-difference and a plain-language `Pr(Δ > 0)` readout.
|
||
4. **Arrests over time** — observed counts by CRDC wave against the three-year model's median and 95% interval, inline SVG.
|
||
|
||
Distributions come from the published 500-draw posteriors, fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset, and fall back per group to an analytic approximation (with a visible note) when those draws can't be fetched.
|
||
|
||
## Architecture
|
||
|
||
### Tech Stack
|
||
- **React 19** + **Vite** (static site generation, no backend required)
|
||
- Plain CSS custom properties for styling (matches Civilytics design tokens exactly)
|
||
- Inline SVG rendering that **React owns** — no charting library. `d3-scale`, `d3-shape`, `d3-array` and `d3-interpolate` supply scales, path generators and colour interpolation only; no d3 selections and no `useEffect` DOM mutation
|
||
- Embeds **DuckDB-Wasm** (`@duckdb/duckdb-wasm`, ~39MB uncompressed / ~8.8MB gzipped, loaded on demand only after a district is selected) to query real posterior draws client-side from a public Hugging Face Parquet dataset
|
||
- Calls the public read-only API directly from the browser
|
||
|
||
### File Structure
|
||
```
|
||
crdc-demo/
|
||
├── index.html # Entry point
|
||
├── vite.config.mjs # Vite build config
|
||
├── public/ # Static assets (wordmark, favicon, fixtures)
|
||
│ ├── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
|
||
│ └── data/
|
||
│ ├── national_rates.json # National rates fixture for comparisons
|
||
│ └── top_districts.json # Top 15 districts per state by observed arrests
|
||
├── scripts/
|
||
│ └── build-top-districts.mjs # One-off generator for top_districts.json
|
||
├── src/
|
||
│ ├── main.jsx # React entry
|
||
│ ├── App.jsx # Main router (state → search → loading → charts)
|
||
│ ├── hooks/
|
||
│ │ ├── useApi.js # API client with retry/backoff + endpoint wrappers
|
||
│ │ └── useDrawDistribution.js # Posterior draw counts by draw_id (duckdb-wasm + HF parquet)
|
||
│ ├── components/
|
||
│ │ ├── StateSelector.jsx # Landing screen — state dropdown/grid
|
||
│ │ ├── DistrictSearch.jsx # Search + suggestions from the committed fixture
|
||
│ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch
|
||
│ │ ├── ChartLegend.jsx # Shared legend row
|
||
│ │ ├── ApproxNote.jsx # "shape estimated from interval bounds" caption
|
||
│ │ ├── DistrictSummaryTable.jsx # Observed arrests table — also the chart's legend/control
|
||
│ │ └── ChartPanel.jsx # Owns cross-chart state + data fetching
|
||
│ ├── charts/
|
||
│ │ ├── ArrestsOverTime.jsx # Observed vs. modelled counts by wave (SVG)
|
||
│ │ ├── RateDensityPanel.jsx # Chart A — posterior density per group + AC rail
|
||
│ │ └── GroupDifference.jsx # Chart B — posterior of Δ between two groups
|
||
│ ├── utils/
|
||
│ │ ├── duckdbClient.js # Lazy duckdb-wasm bundle loader (dynamic import)
|
||
│ │ ├── kde.js # Empirical density from real draws (+ .test.js)
|
||
│ │ ├── densityProfile.js # Discrete-mass vs. KDE profile choice (+ .test.js)
|
||
│ │ ├── agrestiCoull.js # Frequentist interval, ported from R (+ .test.js)
|
||
│ │ ├── pooling.js # Sex pooling for sparse districts (+ .test.js)
|
||
│ │ ├── districtGroups.js # Display-row derivation and defaults (+ .test.js)
|
||
│ │ ├── groupDifference.js # Per-draw Δ and its summary (+ .test.js)
|
||
│ │ ├── rateDomain.js # Shared x-axis domain and clip flag (+ .test.js)
|
||
│ │ ├── drawGroups.js # Draw-map key format + coverage check (+ .test.js)
|
||
│ │ └── distributionApprox.js # Analytic fallback when draws are unavailable
|
||
│ └── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
|
||
├── Dockerfile # Multi-stage build → nginx static server
|
||
├── docker-compose.yml # Local dev / self-hosted deployment
|
||
└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch
|
||
```
|
||
|
||
### API Endpoints Used
|
||
| Endpoint | Purpose | Frequency |
|
||
|---|---|---|
|
||
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
|
||
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
|
||
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | 3 waves + 1 per selected spec |
|
||
| `/api/v1/estimates?state=XX&year=Y` | Not called at runtime — rows come back `ORDER BY LEAID` at 8 per district, so any short read ranks the lowest-LEAID districts. Paged with `meta.total` by `scripts/build-top-districts.mjs` | Build-time only |
|
||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
|
||
| `/data/top_districts.json` | Suggested districts per state (committed fixture) | Once per session |
|
||
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
|
||
|
||
## Deployment
|
||
|
||
### Option A: Git Pages via Gitea Actions (recommended)
|
||
The app is a static site deployed automatically via Gitea Actions. On every push to `main`, the workflow in
|
||
`.gitea/workflows/pages.yml` builds the Vite app and copies `dist/` to a `pages` branch served by the
|
||
git-pages webhook receiver.
|
||
|
||
**URL**: `https://pages.civilytics.org/crdc-demo/`
|
||
|
||
> **CORS note**: The CRDC API (`crdc-api.civilytics.org`) does not send CORS headers. When deployed as a static site,
|
||
> browser-based fetch requests will be blocked by the same-origin policy. To fix this, deploy `proxy.php` to your git-pages host
|
||
> and set the `VITE_PROXY_URL` environment variable (e.g., `/crdc-demo/proxy.php`). The app automatically detects
|
||
> proxy availability — when unset, it attempts direct API calls (works via Docker/nginx or same-origin setups).
|
||
|
||
The workflow follows the same pattern as `Civilytics/sln-school-comparison`:
|
||
1. Triggers on push to `main` (only when source files change)
|
||
2. Builds with Node 22 + Vite (`npm ci && npm run build`)
|
||
3. Copies `dist/` contents to an orphan `pages` branch
|
||
4. Forces push — the git-pages webhook picks it up automatically
|
||
|
||
> **Note**: Ensure `${{ secrets.GITHUB_TOKEN }}` is configured in repo settings on gitea.civilytics.org.
|
||
|
||
### Option A.5: CORS Proxy for Git Pages
|
||
If you can't deploy PHP, create a simple reverse proxy in nginx or use an edge function that:
|
||
1. Accepts `?target=/api/v1/...` as a query parameter
|
||
2. Forwards the request to `https://crdc-api.civilytics.org/api/v1/...
|
||
3. Returns the response with `Access-Control-Allow-Origin: *`
|
||
4. Set `VITE_PROXY_URL` at build time to point at this endpoint.
|
||
|
||
### Option B: Docker (self-hosted fleet) — includes CORS proxy nginx config:
|
||
|
||
```bash
|
||
docker compose up -d # builds + serves on localhost:8080
|
||
|
||
# Or build manually for your registry:
|
||
docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo:latest .
|
||
docker push registry.civilytics.org/crdc-demo:latest
|
||
```
|
||
|
||
The container is nginx Alpine (~5MB) plus the built site, which is dominated by the ~39MB DuckDB-Wasm engine. `nginx.conf` serves that engine gzipped (~8.8MB on the wire) with immutable cache headers, so it is fetched once per browser. A `/healthz` endpoint supports container orchestration health checks.
|
||
|
||
### Environment Variables
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `VITE_API_BASE` | `https://crdc-api.civilytics.org/api/v1` | Override to point at a staging API (set in Dockerfile build or docker-compose) |
|
||
|
||
## Visual Design
|
||
|
||
This app follows the Civilytics visual identity as defined in:
|
||
- `theme/_tokens.scss` — colors, typography, spacing tokens
|
||
- `social_media_posts.md` — chart patterns from published social media figures
|
||
- `white_paper.qmd` / `R/paper_figures.R` — white paper figure builders
|
||
|
||
Key design decisions:
|
||
- **Color palette**: Warm paper background (`#FAF7F2`), civic navy text (`#0E1A2B`), ember accent (`#C25311`)
|
||
- **Data viz colors**: Teal, plum, moss, brass from the supporting palette
|
||
- **Fonts**: Libre Franklin/Inter for display and body, JetBrains Mono for code
|
||
- **Chart patterns**: Pointrange comparisons (model vs. observed), ridgeline-style density bars, faceted small multiples
|
||
|
||
## Citation & Attribution
|
||
|
||
The CRDC School Arrest Rate API: Knowles, J.E., & Miller, H. (2025). *CRDC School Arrest Rate API v1*. Civilytics Consulting. https://crdc-api.civilytics.org/api/v1/
|
||
|
||
Data source: US Department of Education, Office for Civil Rights. Civil Rights Data Collection (2021-22).
|
||
|
||
This research was supported by a grant from the American Educational Research Association which receives funds for its "AERA Grants Program" from the National Science Foundation under NSF award NSF-DRL #1749275. Opinions reflect those of the author and do not necessarily reflect those AERA or NSF.
|