diff --git a/AGENTS.md b/AGENTS.md index 5ef935e..8575683 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,11 +46,14 @@ App.jsx (router) ### Error Bar Convention: 90% Intervals -The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**. When converting HPD bounds to standard deviations for the synthetic draw generation fallback (used in the ridgeline chart if real draws can't be fetched — see "Real posterior draws via duckdb-wasm" above), the divisor used is **3.29** (corresponding to z = 1.645 for a two-tailed 90% interval). +The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**. + +The 90% convention is baked into the analytic fallback in `src/utils/distributionApprox.js` (the path used when real posterior draws can't be fetched — see the duckdb-wasm section below). That fallback generates **no draws**: `fitSkewedInterval` fits a two-piece normal directly from `{median, lower, upper}`, converting each half-interval to its own sigma with `z = probit((1 + intervalMass) / 2)` and `intervalMass` defaulting to `0.90` (so z ≈ 1.645). For a symmetric interval that is equivalent to the old fixed "full width / 3.29" divisor, but it is computed from `intervalMass` rather than hardcoded, and each side gets its own sigma so the fitted shape stays skewed. If you change this convention, update: -- `RateDensityRidgeline.jsx` — SD calculation (`intervalWidth / 3.29`) and label text -- `ModelDrawsComparison.jsx` — Legend labels mentioning "90%" +- `distributionApprox.js` — the `intervalMass = 0.90` default in `fitSkewedInterval`, and its JSDoc claim that the API's bounds are a 90% interval +- `RateByGroupBar.jsx` — the caption and component JSDoc, both of which say "the model's reported 90% interval" +- `ArrestsOverTime.jsx` — the legend label "Modeled (median + 90% interval)" - Any documentation referencing confidence/credible intervals ## Common Pitfalls & Gotchas diff --git a/README.md b/README.md index 79a22ee..cd5931f 100644 --- a/README.md +++ b/README.md @@ -18,22 +18,19 @@ npm run preview # serve built files locally ## What It Does -Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 6 charts comparing observed data against Bayesian model estimates: +Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 3 charts comparing observed data against Bayesian model estimates: 1. **Arrests over time** (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG -2. **Rate by student group** (Chart 2) — bar chart, most recent year (observed vs. modeled), SVG -3. **District vs. national** (Chart 3) — highest-rate group compared to the U.S. average, SVG -4. **Model predictions vs. observed** (Chart 4) — four quadrants (one-year/three-year × baseline/covariate) -5. **Predicted rates by student group** (Chart 5) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched. -6. **Exceedance probability** (Chart 6) — P(district > national) per student group +2. **Rate by student group** (Chart 2) — box-and-whisker per race×sex group for the most recent year, split into Female/Male panels. The box is the 25th–75th percentile of that group's real 500-draw posterior (same duckdb-wasm fetch as Chart 3), the whisker is the model's reported 90% interval, and a diamond marks the observed rate; individual boxes fall back to an analytic approximation when a group's draws can't be fetched. +3. **Predicted rates by student group** (Chart 3) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with a model-selector dropdown for the four Bayesian specifications and diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched. ## Architecture ### Tech Stack - **React 19** + **Vite** (static site generation, no backend required) - Plain CSS custom properties for styling (matches Civilytics design tokens exactly) -- D3.js v7 for data-driven visualizations (density ridges, scales, axes) -- Inline SVG rendering — charts are built with vanilla DOM/D3, not charting libraries +- Inline SVG rendering with hand-rolled scales — no charting library, no D3 dependency +- Embeds **DuckDB-Wasm** (`@duckdb/duckdb-wasm`, ~39MB uncompressed / ~8.8MB gzipped, loaded on demand only after a district is selected) to query real posterior draws client-side from a public Hugging Face Parquet dataset - Calls the public read-only API directly from the browser ### File Structure @@ -41,26 +38,32 @@ Visitors select a U.S. state, search for a school district (with suggestions of crdc-demo/ ├── index.html # Entry point ├── vite.config.mjs # Vite build config -├── public/ # Static assets (wordmark, favicon) -│ └── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package +├── public/ # Static assets (wordmark, favicon, fixtures) +│ ├── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package +│ └── data/national_rates.json # Static national rates fixture for comparisons ├── src/ │ ├── main.jsx # React entry │ ├── App.jsx # Main router (state → search → loading → charts) -│ ├── hooks/useApi.js # API client with retry/backoff + endpoint wrappers +│ ├── hooks/ +│ │ ├── useApi.js # API client with retry/backoff + endpoint wrappers +│ │ └── useDrawDistribution.js # Fetches real posterior draws (duckdb-wasm + HF parquet) │ ├── components/ │ │ ├── StateSelector.jsx # Landing screen — state dropdown/grid │ │ ├── DistrictSearch.jsx # Search + "interesting" district suggestions │ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch -│ │ └── ChartPanel.jsx # Orchestrates all 6 charts + data fetching +│ │ ├── ChartLegend.jsx # Shared legend row +│ │ ├── ApproxNote.jsx # "shape estimated from interval bounds" caption +│ │ └── ChartPanel.jsx # Orchestrates all 3 charts + data fetching │ ├── charts/ -│ │ ├── ArrestsOverTime.jsx # Chart 1 — line chart by wave (SVG) -│ │ ├── RateByGroupBar.jsx # Chart 2 — bar chart by group (SVG) -│ │ ├── DistrictVsNational.jsx # Chart 3 — comparison vs. national avg -│ │ ├── ModelDrawsComparison.jsx # Chart 4 — quadrant model comparison (D3) -│ │ ├── RateDensityRidgeline.jsx # Chart 5 — density ridges per student group (D3) -│ │ └── ExceedanceProbability.jsx # Chart 6 — P(district > nat) per group -│ ├── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing) -│ └── data/national_rates.json # Static national rates fixture for comparisons +│ │ ├── ArrestsOverTime.jsx # Chart 1 — counts by wave (SVG) +│ │ ├── RateByGroupBar.jsx # Chart 2 — box-and-whisker by group (SVG) +│ │ └── RateDensityRidgeline.jsx # Chart 3 — density ridges per student group (SVG) +│ ├── utils/ +│ │ ├── duckdbClient.js # Lazy duckdb-wasm bundle loader (dynamic import) +│ │ ├── kde.js # Empirical density from real draws (+ kde.test.js) +│ │ ├── drawGroups.js # Draw-map key format + coverage check (+ .test.js) +│ │ └── distributionApprox.js # Analytic fallback when draws are unavailable +│ └── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing) ├── Dockerfile # Multi-stage build → nginx static server ├── docker-compose.yml # Local dev / self-hosted deployment └── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch @@ -114,7 +117,7 @@ docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo: docker push registry.civilytics.org/crdc-demo:latest ``` -The container is ~5MB (nginx Alpine) and serves static files with immutable cache headers. A `/healthz` endpoint supports container orchestration health checks. +The container is nginx Alpine (~5MB) plus the built site, which is dominated by the ~39MB DuckDB-Wasm engine. `nginx.conf` serves that engine gzipped (~8.8MB on the wire) with immutable cache headers, so it is fetched once per browser. A `/healthz` endpoint supports container orchestration health checks. ### Environment Variables | Variable | Default | Description |