jared
c1920c11fb
Ports Figs 6 and 7 from crdc-arrests/R/paper_figures.R (wp_fig_group_density and
wp_fig_group_difference) so someone who has read the paper sees the paper. The
three old charts become a summary table and two charts; ArrestsOverTime stays.
Draws pipeline. useDrawDistribution now selects draw_id and returns predicted
counts indexed by draw rather than pre-divided rates. That one column is what
unlocks the rest: pooling has to sum numerators and denominators separately, and
a between-group difference has to be taken at a common draw index. Indexing by
draw_id rather than push order makes DuckDB's row ordering irrelevant and turns
a missing draw into a hole, which isCompleteDrawSet then rejects — a group
present for 300 of 500 draws would otherwise get an interval computed off a
biased subsample that looks identical on screen to a complete one. It also takes
models[] instead of a single model, so "compare all four" needs no conditional
hooks. The reset-before-guard ordering is preserved.
Also fixes a pre-existing bug in that pipeline: a state's draws are split across
data_0.parquet, data_1.parquet, … and the part count varies by state. The app
only ever fetched data_0. Nevada has one part, so this was invisible in every
Nevada test; California has eight, totalling 6.2MB, of which data_0 is 37KB and
holds 11 of California's 1,715 districts. Every other CA district looked absent
from the published data and silently fell back to the approximation. Parts are
now discovered from the Hugging Face tree listing API and fetched in parallel —
listing rather than probing data_N until a 404, because the browser logs a 404
as a console error however cleanly the fetch handles it, and a red error on
every load is indistinguishable from a real one. HEAD probing is the fallback.
New pure utils, all written against tests first:
agrestiCoull faithful port incl. the zero-numerator rule of three and the
negative lower bound at (1, 53); pinned to five R outputs
pooling sex pooling for sparse districts; numerator and denominator
are always drawn from the same set of groups
densityProfile discrete probability mass below 12 distinct values, KDE above;
KDE delegates to kde.js, whose bandwidth clamp is untouched
districtGroups display-row derivation, defaults, pooled vs unpooled keys
groupDifference per-draw delta; refuses to pair mismatched draw sets
rateDomain shared x-axis, with a clip flag so the cap is never silent
Chart A keeps the palette contract: race is hue, sex is position. Female and
Male are stacked panels sharing one axis, collapsing to one panel when pooled —
never a second hue. Chart B uses a diverging ramp centred at zero rather than
the paper's sequential YlOrRd, because delta is signed and a sequential ramp
encodes "more" where the data means "which direction".
Captions say "posterior predictive draws", never "paired parameter draws":
draw_id is renumbered per write batch upstream and a district's groups land in
different batches, so cross-group pairing is effectively independent (measured
cor ~= 0.02). The published figure has the same property; what neither can claim
is a paired-parameter contrast.
Sparse districts (under 20 arrests district-wide) pool Female and Male within
each race, with a banner stating the rule and a switch to override it. A pooled
group carries no modelled interval — summing two groups' interval bounds is not
a pooled interval, and there is no honest way to fake one without the draws.
React still owns the DOM. d3-scale/shape/array/interpolate supply scales, path
generators and colour interpolation; no selections, no useEffect DOM mutation.
Deletes RateByGroupBar and RateDensityRidgeline. Keeps distributionApprox.js and
ApproxNote.jsx — still the per-group fallback when draws can't be fetched.
112 tests pass; npm run build clean.
CRDC Arrests API Demo
A demonstration web application for the CRDC School Arrest Rate API, showing Bayesian model comparisons of school-based arrest rates across U.S. districts. Built with React + Vite, deployable as a static site or Docker container.
Quick Start (Development)
# Install dependencies
npm ci
# Start dev server on localhost:5173
npm run dev
# Build for production
npm run build # outputs to dist/
npm run preview # serve built files locally
What It Does
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 3 charts comparing observed data against Bayesian model estimates:
- Arrests over time (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG
- Rate by student group (Chart 2) — box-and-whisker per race×sex group for the most recent year, split into Female/Male panels. The box is the 25th–75th percentile of that group's real 500-draw posterior (same duckdb-wasm fetch as Chart 3), the whisker is the model's reported 90% interval, and a diamond marks the observed rate; individual boxes fall back to an analytic approximation when a group's draws can't be fetched.
- Predicted rates by student group (Chart 3) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with a model-selector dropdown for the four Bayesian specifications and diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
Architecture
Tech Stack
- React 19 + Vite (static site generation, no backend required)
- Plain CSS custom properties for styling (matches Civilytics design tokens exactly)
- Inline SVG rendering with hand-rolled scales — no charting library, no D3 dependency
- Embeds DuckDB-Wasm (
@duckdb/duckdb-wasm, ~39MB uncompressed / ~8.8MB gzipped, loaded on demand only after a district is selected) to query real posterior draws client-side from a public Hugging Face Parquet dataset - Calls the public read-only API directly from the browser
File Structure
crdc-demo/
├── index.html # Entry point
├── vite.config.mjs # Vite build config
├── public/ # Static assets (wordmark, favicon, fixtures)
│ ├── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
│ └── data/national_rates.json # Static national rates fixture for comparisons
├── src/
│ ├── main.jsx # React entry
│ ├── App.jsx # Main router (state → search → loading → charts)
│ ├── hooks/
│ │ ├── useApi.js # API client with retry/backoff + endpoint wrappers
│ │ └── useDrawDistribution.js # Fetches real posterior draws (duckdb-wasm + HF parquet)
│ ├── components/
│ │ ├── StateSelector.jsx # Landing screen — state dropdown/grid
│ │ ├── DistrictSearch.jsx # Search + "interesting" district suggestions
│ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch
│ │ ├── ChartLegend.jsx # Shared legend row
│ │ ├── ApproxNote.jsx # "shape estimated from interval bounds" caption
│ │ └── ChartPanel.jsx # Orchestrates all 3 charts + data fetching
│ ├── charts/
│ │ ├── ArrestsOverTime.jsx # Chart 1 — counts by wave (SVG)
│ │ ├── RateByGroupBar.jsx # Chart 2 — box-and-whisker by group (SVG)
│ │ └── RateDensityRidgeline.jsx # Chart 3 — density ridges per student group (SVG)
│ ├── utils/
│ │ ├── duckdbClient.js # Lazy duckdb-wasm bundle loader (dynamic import)
│ │ ├── kde.js # Empirical density from real draws (+ kde.test.js)
│ │ ├── drawGroups.js # Draw-map key format + coverage check (+ .test.js)
│ │ └── distributionApprox.js # Analytic fallback when draws are unavailable
│ └── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
├── Dockerfile # Multi-stage build → nginx static server
├── docker-compose.yml # Local dev / self-hosted deployment
└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch
API Endpoints Used
| Endpoint | Purpose | Frequency |
|---|---|---|
/api/v1/models |
List available Bayesian model specs | Once (cached) |
/api/v1/districts?q=&state= |
District name/geo lookup → LEAID | On keystroke |
/api/v1/estimates/{leaid}?model=X&year=Y |
Estimates for one district/model/year/group | ~40 calls per district |
/api/v1/draws?... |
Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see src/hooks/useDrawDistribution.js |
/data/national_rates.json |
Static national rates fixture (committed) | Once per session |
Deployment
Option A: Git Pages via Gitea Actions (recommended)
The app is a static site deployed automatically via Gitea Actions. On every push to main, the workflow in
.gitea/workflows/pages.yml builds the Vite app and copies dist/ to a pages branch served by the
git-pages webhook receiver.
URL: https://pages.civilytics.org/crdc-demo/
CORS note: The CRDC API (
crdc-api.civilytics.org) does not send CORS headers. When deployed as a static site, browser-based fetch requests will be blocked by the same-origin policy. To fix this, deployproxy.phpto your git-pages host and set theVITE_PROXY_URLenvironment variable (e.g.,/crdc-demo/proxy.php). The app automatically detects proxy availability — when unset, it attempts direct API calls (works via Docker/nginx or same-origin setups).
The workflow follows the same pattern as Civilytics/sln-school-comparison:
- Triggers on push to
main(only when source files change) - Builds with Node 22 + Vite (
npm ci && npm run build) - Copies
dist/contents to an orphanpagesbranch - Forces push — the git-pages webhook picks it up automatically
Note
: Ensure
${{ secrets.GITHUB_TOKEN }}is configured in repo settings on gitea.civilytics.org.
Option A.5: CORS Proxy for Git Pages
If you can't deploy PHP, create a simple reverse proxy in nginx or use an edge function that:
- Accepts
?target=/api/v1/...as a query parameter - Forwards the request to `https://crdc-api.civilytics.org/api/v1/...
- Returns the response with
Access-Control-Allow-Origin: * - Set
VITE_PROXY_URLat build time to point at this endpoint.
Option B: Docker (self-hosted fleet) — includes CORS proxy nginx config:
docker compose up -d # builds + serves on localhost:8080
# Or build manually for your registry:
docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo:latest .
docker push registry.civilytics.org/crdc-demo:latest
The container is nginx Alpine (~5MB) plus the built site, which is dominated by the ~39MB DuckDB-Wasm engine. nginx.conf serves that engine gzipped (~8.8MB on the wire) with immutable cache headers, so it is fetched once per browser. A /healthz endpoint supports container orchestration health checks.
Environment Variables
| Variable | Default | Description |
|---|---|---|
VITE_API_BASE |
https://crdc-api.civilytics.org/api/v1 |
Override to point at a staging API (set in Dockerfile build or docker-compose) |
Visual Design
This app follows the Civilytics visual identity as defined in:
theme/_tokens.scss— colors, typography, spacing tokenssocial_media_posts.md— chart patterns from published social media figureswhite_paper.qmd/R/paper_figures.R— white paper figure builders
Key design decisions:
- Color palette: Warm paper background (
#FAF7F2), civic navy text (#0E1A2B), ember accent (#C25311) - Data viz colors: Teal, plum, moss, brass from the supporting palette
- Fonts: Libre Franklin/Inter for display and body, JetBrains Mono for code
- Chart patterns: Pointrange comparisons (model vs. observed), ridgeline-style density bars, faceted small multiples
Citation & Attribution
The CRDC School Arrest Rate API: Knowles, J.E., & Miller, H. (2025). CRDC School Arrest Rate API v1. Civilytics Consulting. https://crdc-api.civilytics.org/api/v1/
Data source: US Department of Education, Office for Civil Rights. Civil Rights Data Collection (2021-22).
This research was supported by a grant from the American Educational Research Association which receives funds for its "AERA Grants Program" from the National Science Foundation under NSF award NSF-DRL #1749275. Opinions reflect those of the author and do not necessarily reflect those AERA or NSF.