jared
db5111dab9
Deploy to git-pages / deploy (push) Successful in 22s
AGENTS.md's "Error Bar Convention" paragraph was wrong in four ways after the last docs pass: it pointed "above" at a section that is below it, described the fallback as "synthetic draw generation" when it generates no draws at all, cited an `intervalWidth / 3.29` expression in RateDensityRidgeline.jsx that does not exist, and listed ModelDrawsComparison.jsx, a file that was deleted when the demo was reduced to 3 charts. Rewritten against the current code: the fallback is fitSkewedInterval's analytic two-piece normal, whose divisor is probit((1 + intervalMass) / 2) with intervalMass defaulting to 0.90, and the "if you change this" list now names the three files that actually encode 90%. README: 6 charts -> 3, the ridgeline is Chart 3 not Chart 5, Chart 2's use of real draws is now mentioned, the D3.js v7 claim is dropped (d3 is not a dependency), @duckdb/duckdb-wasm is listed in the tech stack, the file tree matches src/, and the container-size note accounts for the wasm engine.
147 lines
8.9 KiB
Markdown
147 lines
8.9 KiB
Markdown
# CRDC Arrests API Demo
|
||
|
||
A demonstration web application for the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/), showing Bayesian model comparisons of school-based arrest rates across U.S. districts. Built with React + Vite, deployable as a static site or Docker container.
|
||
|
||
## Quick Start (Development)
|
||
|
||
```bash
|
||
# Install dependencies
|
||
npm ci
|
||
|
||
# Start dev server on localhost:5173
|
||
npm run dev
|
||
|
||
# Build for production
|
||
npm run build # outputs to dist/
|
||
npm run preview # serve built files locally
|
||
```
|
||
|
||
## What It Does
|
||
|
||
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 3 charts comparing observed data against Bayesian model estimates:
|
||
|
||
1. **Arrests over time** (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG
|
||
2. **Rate by student group** (Chart 2) — box-and-whisker per race×sex group for the most recent year, split into Female/Male panels. The box is the 25th–75th percentile of that group's real 500-draw posterior (same duckdb-wasm fetch as Chart 3), the whisker is the model's reported 90% interval, and a diamond marks the observed rate; individual boxes fall back to an analytic approximation when a group's draws can't be fetched.
|
||
3. **Predicted rates by student group** (Chart 3) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with a model-selector dropdown for the four Bayesian specifications and diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
|
||
|
||
## Architecture
|
||
|
||
### Tech Stack
|
||
- **React 19** + **Vite** (static site generation, no backend required)
|
||
- Plain CSS custom properties for styling (matches Civilytics design tokens exactly)
|
||
- Inline SVG rendering with hand-rolled scales — no charting library, no D3 dependency
|
||
- Embeds **DuckDB-Wasm** (`@duckdb/duckdb-wasm`, ~39MB uncompressed / ~8.8MB gzipped, loaded on demand only after a district is selected) to query real posterior draws client-side from a public Hugging Face Parquet dataset
|
||
- Calls the public read-only API directly from the browser
|
||
|
||
### File Structure
|
||
```
|
||
crdc-demo/
|
||
├── index.html # Entry point
|
||
├── vite.config.mjs # Vite build config
|
||
├── public/ # Static assets (wordmark, favicon, fixtures)
|
||
│ ├── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
|
||
│ └── data/national_rates.json # Static national rates fixture for comparisons
|
||
├── src/
|
||
│ ├── main.jsx # React entry
|
||
│ ├── App.jsx # Main router (state → search → loading → charts)
|
||
│ ├── hooks/
|
||
│ │ ├── useApi.js # API client with retry/backoff + endpoint wrappers
|
||
│ │ └── useDrawDistribution.js # Fetches real posterior draws (duckdb-wasm + HF parquet)
|
||
│ ├── components/
|
||
│ │ ├── StateSelector.jsx # Landing screen — state dropdown/grid
|
||
│ │ ├── DistrictSearch.jsx # Search + "interesting" district suggestions
|
||
│ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch
|
||
│ │ ├── ChartLegend.jsx # Shared legend row
|
||
│ │ ├── ApproxNote.jsx # "shape estimated from interval bounds" caption
|
||
│ │ └── ChartPanel.jsx # Orchestrates all 3 charts + data fetching
|
||
│ ├── charts/
|
||
│ │ ├── ArrestsOverTime.jsx # Chart 1 — counts by wave (SVG)
|
||
│ │ ├── RateByGroupBar.jsx # Chart 2 — box-and-whisker by group (SVG)
|
||
│ │ └── RateDensityRidgeline.jsx # Chart 3 — density ridges per student group (SVG)
|
||
│ ├── utils/
|
||
│ │ ├── duckdbClient.js # Lazy duckdb-wasm bundle loader (dynamic import)
|
||
│ │ ├── kde.js # Empirical density from real draws (+ kde.test.js)
|
||
│ │ ├── drawGroups.js # Draw-map key format + coverage check (+ .test.js)
|
||
│ │ └── distributionApprox.js # Analytic fallback when draws are unavailable
|
||
│ └── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
|
||
├── Dockerfile # Multi-stage build → nginx static server
|
||
├── docker-compose.yml # Local dev / self-hosted deployment
|
||
└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch
|
||
```
|
||
|
||
### API Endpoints Used
|
||
| Endpoint | Purpose | Frequency |
|
||
|---|---|---|
|
||
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
|
||
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
|
||
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district |
|
||
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
|
||
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
|
||
|
||
## Deployment
|
||
|
||
### Option A: Git Pages via Gitea Actions (recommended)
|
||
The app is a static site deployed automatically via Gitea Actions. On every push to `main`, the workflow in
|
||
`.gitea/workflows/pages.yml` builds the Vite app and copies `dist/` to a `pages` branch served by the
|
||
git-pages webhook receiver.
|
||
|
||
**URL**: `https://pages.civilytics.org/crdc-demo/`
|
||
|
||
> **CORS note**: The CRDC API (`crdc-api.civilytics.org`) does not send CORS headers. When deployed as a static site,
|
||
> browser-based fetch requests will be blocked by the same-origin policy. To fix this, deploy `proxy.php` to your git-pages host
|
||
> and set the `VITE_PROXY_URL` environment variable (e.g., `/crdc-demo/proxy.php`). The app automatically detects
|
||
> proxy availability — when unset, it attempts direct API calls (works via Docker/nginx or same-origin setups).
|
||
|
||
The workflow follows the same pattern as `Civilytics/sln-school-comparison`:
|
||
1. Triggers on push to `main` (only when source files change)
|
||
2. Builds with Node 22 + Vite (`npm ci && npm run build`)
|
||
3. Copies `dist/` contents to an orphan `pages` branch
|
||
4. Forces push — the git-pages webhook picks it up automatically
|
||
|
||
> **Note**: Ensure `${{ secrets.GITHUB_TOKEN }}` is configured in repo settings on gitea.civilytics.org.
|
||
|
||
### Option A.5: CORS Proxy for Git Pages
|
||
If you can't deploy PHP, create a simple reverse proxy in nginx or use an edge function that:
|
||
1. Accepts `?target=/api/v1/...` as a query parameter
|
||
2. Forwards the request to `https://crdc-api.civilytics.org/api/v1/...
|
||
3. Returns the response with `Access-Control-Allow-Origin: *`
|
||
4. Set `VITE_PROXY_URL` at build time to point at this endpoint.
|
||
|
||
### Option B: Docker (self-hosted fleet) — includes CORS proxy nginx config:
|
||
|
||
```bash
|
||
docker compose up -d # builds + serves on localhost:8080
|
||
|
||
# Or build manually for your registry:
|
||
docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo:latest .
|
||
docker push registry.civilytics.org/crdc-demo:latest
|
||
```
|
||
|
||
The container is nginx Alpine (~5MB) plus the built site, which is dominated by the ~39MB DuckDB-Wasm engine. `nginx.conf` serves that engine gzipped (~8.8MB on the wire) with immutable cache headers, so it is fetched once per browser. A `/healthz` endpoint supports container orchestration health checks.
|
||
|
||
### Environment Variables
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `VITE_API_BASE` | `https://crdc-api.civilytics.org/api/v1` | Override to point at a staging API (set in Dockerfile build or docker-compose) |
|
||
|
||
## Visual Design
|
||
|
||
This app follows the Civilytics visual identity as defined in:
|
||
- `theme/_tokens.scss` — colors, typography, spacing tokens
|
||
- `social_media_posts.md` — chart patterns from published social media figures
|
||
- `white_paper.qmd` / `R/paper_figures.R` — white paper figure builders
|
||
|
||
Key design decisions:
|
||
- **Color palette**: Warm paper background (`#FAF7F2`), civic navy text (`#0E1A2B`), ember accent (`#C25311`)
|
||
- **Data viz colors**: Teal, plum, moss, brass from the supporting palette
|
||
- **Fonts**: Libre Franklin/Inter for display and body, JetBrains Mono for code
|
||
- **Chart patterns**: Pointrange comparisons (model vs. observed), ridgeline-style density bars, faceted small multiples
|
||
|
||
## Citation & Attribution
|
||
|
||
The CRDC School Arrest Rate API: Knowles, J.E., & Miller, H. (2025). *CRDC School Arrest Rate API v1*. Civilytics Consulting. https://crdc-api.civilytics.org/api/v1/
|
||
|
||
Data source: US Department of Education, Office for Civil Rights. Civil Rights Data Collection (2021-22).
|
||
|
||
This research was supported by a grant from the American Educational Research Association which receives funds for its "AERA Grants Program" from the National Science Foundation under NSF award NSF-DRL #1749275. Opinions reflect those of the author and do not necessarily reflect those AERA or NSF.
|