Compare commits

..
45 Commits
Author SHA1 Message Date
jared db5111dab9 docs: correct the error-bar convention and README to match the actual app
Deploy to git-pages / deploy (push) Successful in 22s
AGENTS.md's "Error Bar Convention" paragraph was wrong in four ways after the
last docs pass: it pointed "above" at a section that is below it, described the
fallback as "synthetic draw generation" when it generates no draws at all, cited
an `intervalWidth / 3.29` expression in RateDensityRidgeline.jsx that does not
exist, and listed ModelDrawsComparison.jsx, a file that was deleted when the
demo was reduced to 3 charts. Rewritten against the current code: the fallback
is fitSkewedInterval's analytic two-piece normal, whose divisor is
probit((1 + intervalMass) / 2) with intervalMass defaulting to 0.90, and the
"if you change this" list now names the three files that actually encode 90%.

README: 6 charts -> 3, the ridgeline is Chart 3 not Chart 5, Chart 2's use of
real draws is now mentioned, the D3.js v7 claim is dropped (d3 is not a
dependency), @duckdb/duckdb-wasm is listed in the tech stack, the file tree
matches src/, and the container-size note accounts for the wasm engine.
2026-08-11 10:37:37 -04:00
jared 096455944e fix(nginx): cache and compress the duckdb-wasm asset
The wasm engine is the single largest asset in the build (39,362,651 bytes) but
the immutable-cache location regex did not include `wasm`, so on the
Docker/nginx path it got no Cache-Control at all, and no gzip directive existed
anywhere -- nginx's default gzip_types is text/html only, so it shipped
uncompressed on every cold load.

Verified against nginx:alpine (1.31.3) with the real dist/ mounted:
`nginx -t` passes, and the wasm now returns Content-Type: application/wasm,
Content-Encoding: gzip, Cache-Control: public, max-age=31536000, immutable --
8,766,496 bytes on the wire instead of 39,362,651.

Dynamic gzip rather than gzip_static because the Vite build emits no
pre-compressed .gz files.
2026-08-11 10:37:27 -04:00
jared 420fc41571 fix: validate the deep-link state param before propagating it
?state= was used verbatim and ends up interpolated into a fetch URL and into
the Hugging Face parquet shard path duckdb-wasm reads. Impact is low -- it is a
client-only fetch to a public HTTPS URL, and the wasm sandbox loads no httpfs
-- but validating at the boundary is cheap and correct. Deep links now require
/^[A-Z]{2}$/ (after trim + uppercase, so ?state=az still works) and fall back
to the state selector with a warning when they do not match.
2026-08-11 10:37:26 -04:00
jared 70a12cad22 fix: gate the "real posterior draws" claim on per-group coverage
useDrawDistribution's status is an any-group signal -- 'ready' means at least
one group came back with draws -- but both charts used it chart-wide to hide
<ApproxNote /> and print a caption claiming every box/ridge is real draws.
Individual boxes and ridges already fall back per-group, so the note and
caption were the only things over-claiming.

This is reachable with real data, not just in theory: for LEAID 0400311 (AZ,
unified_m4_mod) the estimates API returns all 8 race x sex groups but the
parquet shard contains draws for WH_M only. The chart reported 'ready' and hid
the note while 7 of its 8 ridges were the analytic approximation.

New src/utils/drawGroups.js owns the "RACE_SEX" key format (previously
duplicated across the hook and both charts) and a hasDrawsForAll() coverage
check. Each chart now checks the groups it actually renders -- RACE_ORDER x its
sex panels/columns -- rather than everything the API returned. A null map
(loading, or a fetch that failed outright) fails the check, so the error path
still shows the note.

Also fixes a stale-state hazard in the same hook: the input guard ran before
setStatus('loading')/setDrawsByGroup(null), so switching to a model whose
groups list is empty (that model's upstream fetch failed) left status at
'ready' with the previous model's draws still in state. Verified by
instrumenting the hook: with the old ordering, switching from a ready model to
an empty one kept status 'ready' and all 8 previous draw keys; with the reset
moved above the guard it correctly resets to 'loading' with no draws.
2026-08-11 10:37:15 -04:00
jared 138a083c6f fix: clamp KDE bandwidth to the plotting domain, not absolute units
Small districts produce rare-event count posteriors that are heavily
zero-inflated, and the absolute 1e-3 floor / raw-sd fallback broke on both
extremes of that data:

- All-identical draws (e.g. 500 zeros, the norm for a group with a handful of
  students) collapsed to h = 1e-3, a near-delta spike of density ~399. Because
  SexRidgeColumn shares one maxPdf per column, that single spike flattened every
  other ridge in the column to sub-pixel height. Measured on a real district
  (0400315 AZ, unified_m4_mod): the three informative ridges rendered at
  0.07-0.13px of a 47.56px row.
- When IQR is 0 -- the normal case when most draws are 0 -- min(sd, iqr/1.34)
  was falsy and the rule fell all the way back to raw sd, which a few extreme
  draws inflate until the ridge is a flat line claiming maximal uncertainty.

Bandwidth is now clamped to [domainWidth/50, domainWidth/6] and the robust rule
degrades by picking the smallest *positive* spread estimate instead of
discarding robustness entirely. The floor is slightly wider than one render
step at n = 60, so a degenerate draw set resolves as a narrow bump; it does not
bind on an ordinary posterior (spread wider than ~8% of the domain keeps its
own Silverman bandwidth). On the district above the informative ridges now
render at 5.66-5.70px, a ~60x improvement.

Five new tests cover both failure modes; all five fail against the old formula.
2026-08-11 10:37:01 -04:00
jaredandClaude Sonnet 5 b5ebf6bec2 docs: clarify that synthetic draw generation is the fallback mechanism
AGENTS.md Error Bar Convention section now explicitly states that synthetic
draw generation from normal approximation is only used when real draws can't
be fetched (network error, unsupported browser, HF outage). Makes clear the
fallback is secondary, not primary, to the real-draw mechanism described in
the new §5.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-11 10:02:19 -04:00
jaredandClaude Sonnet 5 523c21d78c docs: reflect real posterior-draw architecture in AGENTS/README/HANDOFF
- AGENTS.md: Replace §4 (API Endpoint Availability) with updated text; insert
  new §5 (Real posterior draws via duckdb-wasm) documenting the shift from
  synthetic normal-approximation draws to client-side fetches via duckdb-wasm
  against the public Hugging Face parquet dataset. Include actual payload size
  (~39MB uncompressed / ~8.86MB gzipped). Renumber subsequent items.
- README.md: Update Chart 5 description in "What It Does" to reflect real draws
  + fallback behavior. Update API table to clarify that /api/v1/draws is not
  called from app but informs the Hugging Face URL the app fetches directly.
- HANDOFF.md: Mark "Raw posterior draws" as done (2026-08-11) with reference
  to the design spec.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-11 09:57:12 -04:00
jared 20c08ae198 fix: clear stale drawsByGroup when a model switch starts or fails
useDrawDistribution never cleared drawsByGroup on a new fetch or on
error, only status. A chart that varies `model` across renders (Chart
3's dropdown) could switch from a model that had succeeded to one
that then failed, and keep rendering the *previous* model's real
posterior draws — keyed by the same RACE_SEX strings — under the
newly-selected model's summary stats, with ApproxNote visible
suggesting (wrongly) that the fallback approximation was in use.

Clear drawsByGroup to null both when a new fetch starts and in the
catch branch, so a failed model switch never mixes draws from two
different models.
2026-08-11 09:43:09 -04:00
jared b5941a78d1 feat: drive Chart 3's ridgelines from real posterior draws, with fallback
Wire useDrawDistribution into RateDensityRidgeline so each race×sex ridge
uses a Gaussian KDE over that group's 500 real posterior draws when the
selected model's shard has loaded, falling back per-group to the analytic
fitSkewedInterval approximation otherwise. ApproxNote now only shows when
status !== 'ready'. Pass leaid/state through from ChartPanel.
2026-08-11 09:30:46 -04:00
jared ea6d554e3d fix: don't poison shard cache on failure; require real draws for ready status
Two Important review findings on useDrawDistribution:
- ensureShardRegistered cached the rejected promise on fetch/registration
  failure, permanently stuck for that (model,year,state) key until a full
  page reload. Now deletes the cache entry on failure so the next caller
  retries fresh.
- status could become 'ready' even when the LEAID-bound query returned zero
  usable rows (district not present in the draws shard), silently mislabeling
  a fitSkewedInterval-only render as real posterior draws. Now requires at
  least one group with actual draws before reporting 'ready'.
2026-08-11 09:22:01 -04:00
jared d5bf18482a feat: drive Chart 2's box plot from real posterior draws, with fallback 2026-08-11 09:07:53 -04:00
jared 91e21feb74 feat: add empirical KDE/quantile utilities for real posterior draws 2026-08-11 08:55:06 -04:00
jared 88b09a5d2e feat: add duckdb-wasm singleton for client-side parquet queries
Adds src/utils/duckdbClient.js, a lazy-initialized getDb() singleton
wrapping the AsyncDuckDB MVP (single-threaded) wasm bundle. Pins
@duckdb/duckdb-wasm to ^1.32.0 (npm's latest dist-tag currently points
at a -dev prerelease). Excludes the package from Vite's dev-server
dependency pre-bundling so its worker/wasm ?url imports resolve
correctly.

Verified live in the browser under the /crdc-demo/ base path: wasm and
worker assets load with 200, and a SELECT 42 query round-trips
correctly through the singleton, confirming the risk flagged in the
design spec (duckdb-wasm loading correctly from a subpath-served Vite
dev server) does not materialize.
2026-08-11 08:49:05 -04:00
jared 991a6912f8 docs: add implementation plan for empirical draw distributions via duckdb-wasm 2026-08-11 08:42:01 -04:00
jared 2cb04ad9c1 docs: add design spec for empirical draw distributions via duckdb-wasm
Scopes replacing the analytic distribution approximation with real
posterior draws, fetched client-side from the Hugging Face parquet
dataset using duckdb-wasm — no new backend endpoint required.
2026-08-11 07:41:21 -04:00
jaredandClaude Sonnet 5 466ebdb9a6 Align modeled intervals with their observed points, use diamonds, name the model
Deploy to git-pages / deploy (push) Successful in 11s
The modeled point-range was dodged 18px to the side of its observed point,
which read as misaligned rather than paired. Removes the dodge so both sit
on the same x column - directly comparable at a glance, matching the
reference whitepaper figures' own convention.

Observed points were plain circles, inconsistent with every other chart in
the app where a diamond means "observed." Switches them to diamonds, drawn
last so they stay on top of the modeled marks sharing their column.

Adds a caption naming the model the modeled interval is drawn from (three-
year, no covariate / unified_m3_mod), sourced from ChartPanel's existing
WAVE_MODEL constant via the shared MODEL_QUADRANT_LABEL lookup rather than
hardcoded, so it can't drift if the model choice changes.

Also gives the rate-per-1k labels a paper-colored text halo (paintOrder:
stroke) so they stay legible now that they can sit directly over the
modeled whisker line instead of needing to dodge around it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-10 19:44:28 -04:00
jaredandClaude Sonnet 5 fa35d97269 Clean up the hero chart: remove watermark, fix label collisions, round axis ticks
Deploy to git-pages / deploy (push) Successful in 12s
The district-name watermark inside the arrests-over-time chart was
redundant with the page's own heading and was the direct cause of a label
collision (a high point's rate label rendered on top of it). Removes it.

Two more collisions, both edge cases the screenshot happened to hit: the
first wave's label sat close enough to the plot's top that it could
overlap the topmost gridline text, and a point sitting flush on the plot's
left edge had its center-anchored label bleed into the y-axis tick labels.
Fixes both with more vertical clearance and edge-aware text anchoring
(first point anchors right, last point anchors left, matching the same
fix already applied to chart 4's wave ticks before that chart was cut).

Adds a shared niceTicks() utility (Heckbert's nice-numbers algorithm) so
axis ticks read as round numbers (0/100/200) instead of arbitrary
fractions of the data max (0/113/226/339) — applied to all three charts
for consistency. Also fixes a legend/mark mismatch in the arrests-over-time
chart (the modeled series legend showed a diamond; the actual mark is a
dot) by adding a proper 'dot' shape to ChartLegend.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-10 19:37:02 -04:00
jaredandClaude Sonnet 5 a1ffa600b3 Reduce demo to 3 charts, fix back-button district name bug
Deploy to git-pages / deploy (push) Successful in 26s
Fixes the deep-link/back-button bug where the district name showed
"Unknown District" on return: App.jsx was passing the LEAID as a text
search query to searchDistricts() (a name search), which never matches a
numeric ID. Resolves it instead via fetchDistrictEstimates(leaid, ...) and
reads lea_name/state directly from the returned row - correct regardless
of how the page was reached (fresh load, refresh, or browser back/forward).

Cuts the demo from 6 charts to 3, per request: arrests over time (kept),
arrest rate by student group restructured into Female/Male box-and-whisker
panels (kept), and the posterior density ridge chart restructured from a
2x2 model-quadrant grid into a single selected model (dropdown, default
three-year + referral rate) with Female/Male ridge columns. Removes
DistrictVsNational, ModelDrawsComparison, and ExceedanceProbability
entirely, along with the student-group filter (no longer needed - the
remaining charts always show the full breakdown) and the national-rates
fetch/plumbing that only those removed charts used.

Also updates LoadingAnimation's copy and dedupes its STUDENT_GROUPS/
MODEL_QUADRANTS constants against the shared ones in useApi.js.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-10 19:27:24 -04:00
jaredandClaude Sonnet 5 cf8e65bb8e Fix chart bugs found on deployed site, redesign Chart 2 as box-and-whisker
Deploy to git-pages / deploy (push) Successful in 12s
Chart 4 (model predictions vs. observed) was picking the first row for a
given wave/model instead of summing across selected groups, so any district
whose first-returned group had zero counts (e.g. a suppressed race group)
showed a collapsed, flat modeled marker instead of the real aggregate
interval. Also fixes the rightmost wave label ("2021-22") clipping off the
edge of the small-multiple panels by anchoring edge ticks away from the
viewBox boundary instead of centering on it.

Chart 6's y-axis group labels (e.g. "American Indian / Alaska Native
Female") were wider than their margin and clipped past the left edge of the
SVG. Adds a shared shortGroupLabel() to colors.js and uses it here, matching
the convention already used in charts 2 and 5.

Chart 2 previously drew a plain bar to the modeled median next to an
observed diamond that often sat well past the bar's end, reading as if the
bar itself were an uncertainty range when it wasn't. Replaces it with an
actual horizontal box-and-whisker: whisker = the API's reported 90%
interval, box = the fitted approximation's 25th/75th percentiles (via a new
fit.quantile() inverse-CDF, exact round-trip of fitSkewedInterval's own
construction), median tick, observed diamond overlaid.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-10 18:58:35 -04:00
jaredandClaude Sonnet 5 67caa02fe2 Redesign chart visualizations to match Civilytics white-paper style
Deploy to git-pages / deploy (push) Successful in 18s
Rebuilds all 6 charts around a validated categorical race palette, row-based
sex encoding, and a consistent observed-vs-modeled mark convention (diamond
vs. filled bar/density) instead of ad hoc per-chart color schemes. Adds a
shared student-group filter (defaults to all 8 groups) that scopes every
chart's data from one place in ChartPanel.

Drops the D3 dependency entirely in favor of plain SVG, removing the
imperative-DOM bug class behind this app's repeated "fix the fix" commits.
Replaces the fake symmetric-normal posterior approximation with a skewed,
median-preserving fit to the API's interval bounds, clearly labeled as an
approximation. Fixes two broken SVG fill attributes, a decorative model
dropdown that never affected its chart, dead code, an orphaned component,
and a broken CSS token reference.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-10 18:45:37 -04:00
jared 6c8e1e8409 docs: refresh AGENTS.md, HANDOFF.md, and README with recent work
- Create AGENTS.md: comprehensive agent guide covering architecture,
  data flow, error bar convention (90% intervals), D3 usage patterns,
  null safety pitfalls, deployment checklist, and related repos
- Update HANDOFF.md: mark CORS as resolved, document chart improvements
  (Chart 5 ridgeline rewrite with synthetic draws + smooth rendering)
- Refresh README.md: accurate tech stack (D3 v7), updated file structure
  with new chart files, corrected API endpoint table
2026-08-10 17:10:35 -04:00
jared 3fe004356e Fix ridgeline chart to render smooth density curves instead of individual columns
Deploy to git-pages / deploy (push) Successful in 13s
- Rewrite RateDensityRidgeline.jsx with proper histogram-to-density conversion
- Build top-edge points from binned draws, then smooth with d3.curveBasis
- Construct complete area path: bottom edge + smoothed top + close
- Position observed rate diamonds above the ridge peak
2026-08-10 17:05:38 -04:00
jared fc0b7e9b6d Fix: remove broken .replace() call causing TypeError in RateDensityRidgeline
Deploy to git-pages / deploy (push) Successful in 12s
2026-08-10 16:56:29 -04:00
jared c32aa46f7c Fix: null check before accessing modelDraws[q.key]
Deploy to git-pages / deploy (push) Successful in 12s
2026-08-10 16:51:53 -04:00
jared c64682228d Fix: null checks for quadData in ChartPanel and child components
Deploy to git-pages / deploy (push) Successful in 14s
2026-08-10 16:43:30 -04:00
jared 83417368f2 Fix: handle null modelDraws gracefully
Deploy to git-pages / deploy (push) Successful in 13s
2026-08-10 16:37:40 -04:00
jared be647d0882 Fix: remove draws fetch (API endpoint not available for browser)
Deploy to git-pages / deploy (push) Successful in 12s
The /estimates/{leaid}/draws endpoint doesn't exist - only bulk Parquet
shard access is available. Falls back to interval-based rendering.
2026-08-10 16:32:40 -04:00
jared 54814564a9 Hook up raw posterior draws from API to ModelDrawsComparison
Deploy to git-pages / deploy (push) Successful in 11s
- Add fetchDistrictDraws call in ChartPanel for all 4 quadrant models
- Pass modelDraws to ModelDrawsComparison component
- Rewrite Chart 4 with D3 showing:
  - Density histogram from raw draws when available (per-year distribution)
  - Fall back to interval-based rendering otherwise
  - 90% credible intervals from quantile calculation
  - Legend explaining colors (red=observed, teal=one-year, navy=three-year)
2026-08-10 16:27:47 -04:00
jared f65a70fc3e Add D3 and new ridgeline chart matching R aesthetic
Deploy to git-pages / deploy (push) Successful in 19s
- Install d3@7
- Add fetchDistrictDraws API function (for future use with raw draws)
- Create RateDensityRidgeline component using D3.js
  - Density ridges showing posterior distributions per race group
  - Diamond markers for observed rates (matching R design)
  - Dropdown to switch between model specifications
  - Matches Civilytics color palette (navy fill, danger diamonds)
2026-08-10 16:15:33 -04:00
jared c4c0aa63bd Use three-year model (unified_m3_mod) to fetch observed data for all waves
Deploy to git-pages / deploy (push) Successful in 11s
One-year models don't return historical arrest data; three-year models
include observed counts across all CRDC waves (15-16, 17-18, 21-22).
2026-08-10 13:19:39 -04:00
jared 0fa5844a16 Replace logo with Civilytics wordmark from civilyticsR package
Deploy to git-pages / deploy (push) Successful in 11s
2026-08-10 12:54:55 -04:00
jared b135f480c0 Fix chart sizing, model predictions, 90% intervals, and deep link district name
Deploy to git-pages / deploy (push) Successful in 17s
- Increase SVG dimensions in all 6 charts to prevent label truncation
- Fix ModelDrawsComparison: pass quadData so predictions render for three-year models
- Change error bars from 95% to 90% (legend text + SD calculation)
- Widen chart viewport: max-width 70rem vs default 60rem text width
- Fix deep link: fetch district name from API instead of showing 'Loading...'
2026-08-10 12:50:19 -04:00
jared eb396abce7 Restore first three charts (ArrestsOverTime, RateByGroupBar, DistrictVsNational) that were accidentally removed in previous edit
Deploy to git-pages / deploy (push) Successful in 11s
2026-08-10 12:27:04 -04:00
jared 77757348f8 Switch ChartPanel from 3-column grid to single vertical stack
Deploy to git-pages / deploy (push) Successful in 10s
Charts and their titles/captions need more horizontal space for readability. Single column layout ensures each chart renders at full container width without cramped side-by-side placement.
2026-08-10 12:25:31 -04:00
jared 26f72da6db Disable source map generation for cleaner production builds
Deploy to git-pages / deploy (push) Successful in 17s
Source maps cause harmless but noisy 404 errors in browser dev tools when deployed to git-pages, since the hashed filenames change between local and CI builds. Disabling them eliminates these warnings without affecting app functionality.
2026-08-10 12:23:27 -04:00
jared dd69bb6f8f Fix JS syntax error in ObservedRateDensity.jsx — const declarations were embedded inside JSX
Deploy to git-pages / deploy (push) Successful in 11s
The obsWidth, lowerX, upperWidth, and medX variables were declared inline within the SVG <g> element's children, which is invalid JavaScript/JSX. This caused 'ReferenceError: obsWidth is not defined' at runtime when rendering Chart 5.

Fix: Moved all const declarations to the top of the map callback, before the return statement.
2026-08-10 12:21:00 -04:00
jared 116730b685 Fix national_rates.json deployment — copy to public/ directory
Deploy to git-pages / deploy (push) Successful in 19s
- Add missing data/national_rates.json fixture (was only in src/data/, not served by Vite)
- LoadingAnimation.jsx: Use import.meta.env.BASE_URL for national rates path
- Ensures file is available at /crdc-demo/data/national_rates.json on git-pages
2026-08-10 12:17:42 -04:00
jared 2c900b003f Fix static asset paths for /crdc-demo subdirectory deployment
Deploy to git-pages / deploy (push) Successful in 10s
- App.jsx: Use import.meta.env.BASE_URL prefix for civilytics-logo.svg (was hardcoded as '/civilytics-logo.svg' → 404 on pages.civilytics.org/crdc-demo/)
- ChartPanel.jsx: Same fix for /data/national_rates.json path — was causing JSON.parse errors because the 404 HTML response couldn't be parsed
2026-08-10 11:44:26 -04:00
jared 8ba7440a34 Fix routing, data fetching, and rendering issues in CRDC demo app:
Deploy to git-pages / deploy (push) Successful in 10s
- App.jsx: Fix 'search another district' button href to use /crdc-demo/ instead of / (root)
- LoadingAnimation.jsx: Replace broken inline fetch helper with centralized api client from useApi.js — was bypassing envelope unwrapping ({status:'success',data:[...]}), causing all 43 API calls to fail silently
- ChartPanel.jsx: Same fix — replace raw fetch() calls with api.fetchDistrictEstimates(), fix national_rates.json path (remove hardcoded /crdc-demo/ prefix that breaks in dev), parallelize wave/model fetches with Promise.all for faster loading

Root causes fixed:
1. Navigation button linked to site root instead of app subdirectory → users lost their way after selecting a district
2. Dual API client implementations — inline fetch helpers didn't unwrap the JSON envelope structure, causing data parsing failures that made the page appear stuck on 'Organizing data...'
3. Sequential fetches in ChartPanel caused slow loading; parallelized for better UX
2026-08-10 11:42:31 -04:00
jared c475bfe50b Add handoff notes for CORS deployment fix 2026-08-10 11:27:03 -04:00
jared 4ce64c7839 Fix CORS: add proxy support for browser-based API calls on git-pages
Deploy to git-pages / deploy (push) Successful in 20s
- Add VITE_PROXY_URL env var to route requests through a CORS proxy when
  deployed as static site (CRDC API doesn't send Access-Control-Allow-Origin)
- Update useApi.js, LoadingAnimation.jsx, and ChartPanel.jsx to respect proxy
- Add nginx.conf with /api/v1/ reverse proxy + CORS headers for Docker deployment
- Include proxy.php — simple PHP CORS proxy for git-pages hosts that support PHP
- Document CORS workaround in README

Without this fix, browser fetch() calls are blocked by same-origin policy and
the app shows a blank white screen despite serving correct HTML.
2026-08-10 11:22:44 -04:00
jared 194208d817 Fix base URL for subdirectory deployment on pages.civilytics.org
Deploy to git-pages / deploy (push) Successful in 10s
- Set vite.config.mjs base to '/crdc-demo/' so asset paths resolve correctly
  when served from a subdirectory (not root)
- Add favicon.svg in public/ folder
- Update index.html with correct absolute paths including /crdc-demo/ prefix
2026-08-10 11:13:16 -04:00
jared ff4b2ac690 Update README: document Gitea Actions pages workflow
Replaces the GitHub Actions reference with .gitea/workflows/pages.yml,
following the sln-school-comparison deployment pattern. Documents the
git-pages.civilytics.org/crdc-demo/ URL and token configuration.
2026-08-10 11:08:00 -04:00
jared 7cf9777673 Add Gitea Actions workflow for git-pages deployment
Deploy to git-pages / deploy (push) Successful in 10s
Adapts the sln-school-comparison pattern: builds Vite static site in CI,
then copies dist/ to a pages branch served by the git-pages webhook at
pages.civilytics.org/crdc-demo/
2026-08-10 10:57:00 -04:00
jared 52a0c77e31 Initial commit: CRDC Arrests API demo app
Deploy to Git Pages / build-and-deploy (push) Failing after 19s
React + Vite static site demonstrating the CRDC School Arrest Rate API.
Features 6 interactive charts comparing observed arrest data against Bayesian
model estimates across U.S. school districts, with Civilytics visual identity.

- State selector and district search with 'interesting' suggestions (top arrests)
- Animated histogram loading grid showing posterior draw progress
- Charts: time series, rate by group, district vs national, model comparison quadrants, density proxy, exceedance probability
- Static JSON fixture for national rates (no API changes needed)
- Dockerfile + docker-compose.yml for self-hosted deployment
- GitHub Actions workflow and _config.yml for Git Pages
2026-08-10 10:54:51 -04:00
55 changed files with 7282 additions and 4761 deletions
+62
View File
@@ -0,0 +1,62 @@
name: Deploy to git-pages
# Builds the Vite static site and pushes it to the `pages` branch, which
# the git-pages webhook receiver at pages.civilytics.org serves.
# URL: https://pages.civilytics.org/crdc-demo/
# Same pattern as Civilytics/sln-school-comparison.
on:
push:
branches: [main]
paths:
- 'src/**'
- 'index.html'
- 'vite.config.mjs'
- 'package.json'
- '.gitea/workflows/pages.yml'
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies & build static site
run: |
npm ci
# VITE_API_BASE defaults to the production API; override if needed via secrets.
# For git-pages deployment, CORS is handled by setting up a proxy endpoint on the server.
npm run build
- name: Publish dist/ to pages branch
run: |
git config user.name "Gitea Actions"
git config user.email "actions@gitea.civilytics.org"
# Use the GITHUB_TOKEN for authentication (works with Gitea token auth)
TOKEN="${{ secrets.GITHUB_TOKEN }}"
if [ -z "$TOKEN" ]; then
echo "GITHUB_TOKEN not available; skipping deploy."
exit 0
fi
git remote set-url origin "https://token:${TOKEN}@gitea.civilytics.org/Civilytics/crdc-demo.git"
# Create orphan pages branch with only the built files
rm -rf /tmp/pages-branch
git worktree add --orphan -B pages /tmp/pages-branch
cp -r dist/. /tmp/pages-branch/
echo "${GITHUB_SHA}" > /tmp/pages-branch/deploy-sha.txt
cd /tmp/pages-branch
git add --all
git commit -m "Deploy from ${GITHUB_SHA}" || echo "No changes to deploy"
git push origin pages --force
+39
View File
@@ -0,0 +1,39 @@
name: Deploy to Git Pages
on:
push:
branches: [main, master]
workflow_dispatch:
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Build static site
run: |
# VITE_API_BASE defaults to the production API; override if needed via secrets.
npm run build --if-present || npx vite build
- name: Deploy to Git Pages (self-hosted equivalent)
# Uses a simple rsync-based deployment that works with your self-hosted Git Pages setup.
# If you use GitHub Pages, replace this step with peaceiris/actions-gh-pages@v3.
run: |
echo "Build complete — dist/ ready for upload to your Git Pages host."
ls -la dist/
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: crdc-demo-dist
path: dist/
+8
View File
@@ -0,0 +1,8 @@
node_modules/
dist/
*.local
*.log
.DS_Store
.env
.vscode/
.idea/
+154
View File
@@ -0,0 +1,154 @@
# Agent Guide — CRDC Demo App
This document captures context, decisions, and guidance for agents working on this codebase. It is the primary source of truth for how to make changes safely.
## Project Overview
A React + Vite static web app demonstrating the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/). Visitors select a state, search for a school district, and see 6 charts comparing observed arrest data against Bayesian model estimates. Deployed via Gitea Actions to `pages.civilytics.org/crdc-demo/`.
## Architecture Summary
- **Frontend**: React 19 + Vite (static site generation)
- **Styling**: Plain CSS custom properties matching Civilytics design tokens (`src/styles/tokens.css`)
- **Charts**: Mixed approach:
- Charts 1–3 use inline SVG with manual scales (no D3 dependency for these)
- Chart 4 (ModelDrawsComparison) uses D3.js v7 for data-driven rendering of quadrant comparisons
- Chart 5 (RateDensityRidgeline) uses D3.js v7 for density ridge visualizations
- **API**: Calls public read-only API directly from browser; no backend required
- **Deployment**: Static site deployed via Gitea Actions (`.gitea/workflows/pages.yml`)
## Key Components and Data Flow
```
App.jsx (router)
→ StateSelector (landing screen: state dropdown/grid)
→ DistrictSearch (search + "interesting" suggestions from /estimates?state=&year=)
→ LoadingAnimation (fetches all data in parallel, shows animated histogram grid)
→ ChartPanel (receives district object, fetches structured estimates for 6 charts)
├── ArrestsOverTime — SVG line chart by wave (3 years)
├── RateByGroupBar — SVG bar chart: observed vs modeled per group
├── DistrictVsNational — SVG comparison to national average
├── ModelDrawsComparison — D3 quadrant charts (4 model types × 1 year)
├── RateDensityRidgeline — D3 density ridges per race×sex group
└── ExceedanceProbability — P(district > national) per student group
```
### Data Fetching Strategy (`ChartPanel.jsx`)
`ChartPanel` fetches all chart data on mount (after `LoadingAnimation` pre-fetched via batch calls):
1. **Wave data** for Charts 1–3: Fetches `unified_m3_mod` model estimates for years `['21-22', '17-18', '15-16']`.
- Uses three-year models because they return observed arrest counts across all waves (one-year models only have data for the most recent wave).
2. **Quad data** for Charts 4–6: Fetches estimates from all four quadrant models (`unified_m1_mod`, `unified_m2_mod`, `unified_m3_mod`, `unified_m4_mod`) for year `21-22` only (one year, as the most recent wave).
3. **National rates**: Loaded once from a static JSON fixture or cached by `LoadingAnimation`.
### Error Bar Convention: 90% Intervals
The API returns 95% HPD intervals (`count_lower`, `count_upper`). However, all chart labels and calculations in this app use **90% intervals**.
The 90% convention is baked into the analytic fallback in `src/utils/distributionApprox.js` (the path used when real posterior draws can't be fetched — see the duckdb-wasm section below). That fallback generates **no draws**: `fitSkewedInterval` fits a two-piece normal directly from `{median, lower, upper}`, converting each half-interval to its own sigma with `z = probit((1 + intervalMass) / 2)` and `intervalMass` defaulting to `0.90` (so z ≈ 1.645). For a symmetric interval that is equivalent to the old fixed "full width / 3.29" divisor, but it is computed from `intervalMass` rather than hardcoded, and each side gets its own sigma so the fitted shape stays skewed.
If you change this convention, update:
- `distributionApprox.js` — the `intervalMass = 0.90` default in `fitSkewedInterval`, and its JSDoc claim that the API's bounds are a 90% interval
- `RateByGroupBar.jsx` — the caption and component JSDoc, both of which say "the model's reported 90% interval"
- `ArrestsOverTime.jsx` — the legend label "Modeled (median + 90% interval)"
- Any documentation referencing confidence/credible intervals
## Common Pitfalls & Gotchas
### 1. Null Safety in Chart Components
Several API responses may return empty arrays or missing fields for districts with no arrests:
```javascript
// Always use optional chaining and defaults:
const yearRow = (quadModels[q.key] || []).find(r => r.year === year)
const predMedian = yearRow?.count_median || 0
const enroll = row.stu_enroll || 1 // Prevent division by zero
```
### 2. D3 useEffect Dependency Arrays
When using `useEffect` for D3 rendering, always include all data dependencies to prevent stale renders:
```javascript
// Correct — includes all props used inside the effect
}, [quadData, selectedModel, rateByGroup])
```
### 3. SVG Dimensions and Responsiveness
Charts use fixed dimensions with responsive containers (`overflowX: 'auto'` for wide content):
- Chart cards have `max-width: 70rem` in `ChartPanel.jsx` (vs the default text width of ~60rem)
- SVG elements should set both `width="100%"` and a fixed `viewBox` for proper scaling
### 4. API Endpoint Availability
Not all endpoints are available to browser-based clients:
- `/api/v1/estimates/{leaid}` — ✅ Returns estimates summary (median, lower, upper bounds)
- `/api/v1/draws?...` — Returns a Parquet shard URL + DuckDB SQL, not draw data itself. The app **does** use the real draws in the shard it points to — see below.
### 5. Real posterior draws via duckdb-wasm
Charts 2 (`RateByGroupBar`) and 3 (`RateDensityRidgeline`) fetch the actual
500-draw-per-group posterior from the public Hugging Face parquet dataset
(`civilytics/crdc-school-arrest-rates`), queried client-side with
`@duckdb/duckdb-wasm` (`src/utils/duckdbClient.js` +
`src/hooks/useDrawDistribution.js`). No server-side draws endpoint is
involved. If that fetch fails (network, unsupported browser, HF outage),
both charts fall back to the `distributionApprox.js` analytic approximation
and show the "estimated shape" note — **do not delete `distributionApprox.js`
or `ApproxNote.jsx`**, they're the fallback path, not dead code.
See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md` for
the full design.
The duckdb-wasm engine itself is ~39MB uncompressed / ~8.86MB gzipped (confirmed against the shipped `@duckdb/duckdb-wasm` package, not the design spec's original ~3-5MB estimate, which was wrong). It's loaded via dynamic `import()` only once a district is selected — never on initial page load — and cached by the browser thereafter, but it's a real one-time cost worth knowing about before touching this code path.
### 6. CORS Configuration
The CRDC API does not send CORS headers. When deployed to git-pages (static hosting), requests are blocked by same-origin policy unless a proxy is configured:
- The app auto-detects proxy availability via `VITE_PROXY_URL` environment variable
- If unset, the app attempts direct fetch — works when served from Docker/nginx or same-origin
## Deployment Checklist
Before pushing to production:
1. **Build succeeds**: `npm run build` (check for new errors)2. **No console errors in browser** after hard refresh3. **All 6 charts render** with sample districts (test "Denver", "Mobile County")
4. **Loading animation** appears briefly, then transitions to ChartPanel
### Commit Message Convention
Use descriptive commit messages that explain the *why*, not just the what:
```bash
# Good
git commit -m "Fix: use three-year model for wave data (returns observed counts across all years)"
git commit -m "Add D3 density ridges to Chart 5 with diamond markers for observed rates"
# Avoid vague messages
git commit -m "Fix charts" # Too generic
git commit -m "Update code" # No context
```
## Testing Strategy
There are no automated tests in this project. Manual verification is required:1. **Visual check**: Load a district and verify all 6 charts render correctly2. **Error console**: Check browser DevTools for JavaScript errors3. **Data accuracy**: Compare observed values against API response (check Network tab)
4. **Responsiveness**: Resize window to ensure layout adapts
## Style Guide References
- R code style: Follows tidyverse principles (`r-style-guide` skill in agent knowledge base)- Chart aesthetic decisions should match patterns from `social_media_posts.md` and `white_paper.qmd`
- Colors, typography, spacing are defined as CSS custom properties in `src/styles/tokens.css`
## Related Repositories
- **crdc-arrests** — The API server (Plumber/R) at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/`
- **civilyticsR** — R package with wordmark and visualization functions at `/home/jared/Nextcloud/Civilytics/Code/Civilytics/civilyticsR/`
## Git Conventions
- Remote: `https://gitea.civilytics.org/Civilytics/crdc-demo.git`
- Default branch: `main` (not `master`)
- Gitea Actions workflow auto-deploys on push to `main` via `.gitea/workflows/pages.yml`
- Always pull before making changes: `git pull origin main`
+27
View File
@@ -0,0 +1,27 @@
# Multi-stage build for the CRDC Arrests API Demo app.
# Stage 1: Build static files with Vite + React
FROM node:22-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --prefer-offline
COPY . .
ARG VITE_API_BASE=https://crdc-api.civilytics.org/api/v1
ENV VITE_API_BASE=${VITE_API_BASE}
ARG VITE_PROXY_URL=""
ENV VITE_PROXY_URL=${VITE_PROXY_URL}
RUN npm run build
# Stage 2: Serve with nginx (tiny image, ~5MB) + reverse proxy for API CORS bypass
FROM nginx:alpine AS runtime
# Copy static site files
COPY --from=builder /app/dist/ /usr/share/nginx/html/
# Custom nginx config — serves static files AND proxies /api/v1/ to the CRDC API
# with CORS headers, so browser-based requests work without preflight issues.
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
+52
View File
@@ -0,0 +1,52 @@
# Handoff: Fix CORS for CRDC Arrests API Demo App
## Current Status
**RESOLVED**: The demo app at `https://pages.civilytics.org/crdc-demo/` is fully functional. All charts render correctly and there are no blocking errors in the console.
The original CORS issue was resolved by deploying updated `plumber.R` with CORS headers to the running API server (`crdc-api.civilytics.org`).
## Historical Context (resolved)
### Original Problem
The demo app loaded HTML correctly but showed a blank white screen because browser-based fetch requests to the CRDC API were blocked by CORS (the API didn't send `Access-Control-Allow-Origin` headers).
### Resolution
The CORS issue was fixed at the API source:
- Updated `crdc-arrests/api/plumber.R` with CORS headers (`Access-Control-Allow-Origin: *`) in the existing `cacheHeaders` filter and OPTIONS preflight handling.
- Deployed to the running API server so it took effect at https://crdc-api.civilytics.org/.
The app now calls the public read-only API directly from the browser without requiring a proxy.
## Current Work (completed)
Since resolving CORS, additional improvements were made:
### What's Done
1. ✅ **CORS resolved**: API server (`crdc-api.civilytics.org`) now sends `Access-Control-Allow-Origin: *` headers — deployed via updated `plumber.R`
2. ✅ **App code**: React + Vite app fully built with 6 charts, Civilytics visual identity (wordmark from civilyticsR package), loading animation, district search
3. ✅ **Git Pages deployment**: Gitea Actions workflow (`.gitea/workflows/pages.yml`) successfully builds `dist/` and deploys to a `pages` branch on `Civilytics/crdc-demo` repo at gitea.civilytics.org
4. ✅ **Chart 5 rewrite**: Replaced interval-proxy ridgeline with proper D3 density ridges (`RateDensityRidgeline.jsx`) using synthetic draws from normal approximation of posterior intervals, smooth `curveBasis` rendering, and diamond markers for observed rates
5. ✅ **Chart 4 enhancement**: Added D3-based quadrant visualization in `ModelDrawsComparison.jsx` showing error bars with proper null safety
### Remaining Proxy Code (optional)
The CORS proxy fallback code (`proxy.php`, nginx reverse proxy config, `VITE_PROXY_URL` support) was added but is **not needed** since the API now sends CORS headers. It remains as a safety net for environments where the API can't be modified.
## What Needs to Be Done Next
### Future Enhancements
- ~~**Raw posterior draws**~~ — Done (2026-08-11). Charts 2 and 3 now fetch real posterior draws client-side via `@duckdb/duckdb-wasm` against the public Hugging Face parquet dataset. See `docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md`.
- **Automated testing**: No test suite exists; consider adding basic tests for chart rendering and API error handling.
## Key Files
- **API fix**: `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-arrests/api/plumber.R` (CORS headers added to cacheHeaders filter)
- **App code**: `/home/jared/Nextcloud/Civilytics/Code/Civilytics/crdc-demo/src/` — fully built, deployed to pages branch
- **Proxy fallback**: `proxy.php`, nginx.conf with reverse proxy config
## Verification Steps After Fix
1. Visit https://pages.civilytics.org/crdc-demo/
2. Select a state → search for "Denver" → should see district results
3. Click a district → loading animation runs, then 6 charts appear
4. Check browser dev tools → no CORS errors in console
## Git Status
- `crdc-arrests` repo: plumber.R modified (CORS headers added), not yet committed/pushed to API deployment
- `crdc-demo` repo: All fixes pushed and deployed via Gitea Actions, awaiting CORS resolution at API or server level
+360
View File
@@ -0,0 +1,360 @@
# CRDC Arrests API Demonstration App — Design Proposal
## 1. Overview
A web application hosted at `civilytics.org` that demonstrates the [CRDC School
Arrest Rate API](https://crdc-api.civilytics.org/api/v1/) by letting any visitor
explore school-based arrest estimates for any U.S. school district, with a strong
visual narrative built around Bayesian model comparisons.
The app is a **frontend-only static application** that calls the public read-only
API directly from the browser. No server-side code, no database, no auth — just
HTML/CSS/JS (with optional Docker container for local dev or self-hosting behind
a reverse proxy).
---
## 2. User Flow
```
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ State │ → │ District Search │ → │ Loading (anim) │ → ╔══════════════╗
│ selector │ │ w/ suggestions │ │ patience msg │ ║ Chart panel ║
└─────────────┘ └──────────────────┘ └─────────────────┘ ║ 6 panels ║
║ see §4 ║
╚══════════════╝
```
### Step 1 — State prompt
- Full-screen landing card with Civilytics branding.
- Dropdown / typeahead for U.S. state (50 states + DC). Two-letter codes map to
the API `state` parameter directly.
- On select → slide to Step 2.
### Step 2 — District search
- Search-as-you-type input, hitting `/api/v1/districts?q=<partial>&state=XX`.
- **Suggested districts** appear as a horizontal carousel of "interesting" picks:
- Top N by total arrest count in the most recent wave (2021-22).
- Fetched once per state via `limit=500` from `/estimates?state=XX&year=21-22`
sorted client-side by `observed_arrests`.
- Each suggestion card shows: district name, enrollment (in small), observed
arrests. Clicking a suggestion jumps straight to loading + charts.
- Typing filters the list live; selecting an item from either source proceeds.
### Step 3 — Loading / patience animation
- A full-screen overlay with an animated visual that conveys "data is being
fetched across multiple endpoints, please be patient."
- The animation should reflect the **multi-model comparison** theme: e.g., a grid
of small histogram-like bars (representing posterior draws) that animate in and
out row by row as each model's endpoint responds.
- Estimated total API calls per district lookup: ~12–15 (one per model × race ×
sex combination, filtered to the selected district). The animation should scale
visually with progress.
### Step 4 — Chart panel (6 charts in a 3×2 grid)
All charts follow Civilytics visual identity: warm paper background (`#FAF7F2`),
civic navy text (`#0E1A2B`), ember accent (`#C25311`). Data-viz palette uses the
supporting colors from `_tokens.scss` (teal, plum, moss, brass).
#### Row 1 — Observed & descriptive
**Chart 1: Arrests over time by CRDC wave (line chart)**
- X-axis: CRDC waves (`2015-16`, `2017-18`, `2021-22`).
- Y-axis: total observed arrest count.
- Line + points; each point labeled with the rate per 1,000 students.
- Source: `/estimates?leaid=XXXXX&year=...` across all three years (or a single
call to `/districts/{leaid}` which returns all demographics for one district).
**Chart 2: Arrest rate by student group — most recent year (bar chart)**
- Bars for each race×sex combination in `AM/BL/HI/WH × F/M`.
- Y-axis: arrests per 1,000 students.
- Color-coded by the supporting palette; legend shows full labels.
**Chart 3: District vs. national — top student group (comparison chart)**
- Identify the district's highest-arrest-rate group.
- Side-by-side bars or a small multiples comparison against the corresponding
**national** rate for that same group.
- National rates fetched via `/states?state=XX&race=&sex=&year=21-22` aggregated,
or more precisely from the national summary (which may need to be computed as
an aggregate across all states).
#### Row 2 — Bayesian model distribution comparisons
All four quadrants show results for the **selected district**, comparing:
- **Column A**: One-year models (`unified_m1_mod`, `unified_m2_mod`) vs.
- **Column B**: Three-year models (`unified_m3_mod` through `unified_m5_mod`).
- Within each column, rows differentiate baseline (no covariate) from covariate
models.
**Chart 4: Predicted draws by year vs. observed (scatter / point-range)**
- For three-year models only: for each of the 3 waves, show the model's predicted
median and 95% interval alongside the observed value as a separate marker.
- Layout: x-axis = wave; y-axis = arrest count; points dodge left (model) vs.
right (observed).
**Chart 5: Observed rate per group against model distribution (density / ridge)**
- For each student group, show the observed rate and overlay the posterior draw
distributions from all four model types as ridgeline or violin plots.
- Mirrors `wp_fig_group_density()` in the white paper.
**Chart 6: Probability district exceeds national rate per group (bar chart)**
- For each race×sex group, compute P(district rate > national rate) using the
posterior draws from a chosen model (e.g., the default `unified_m2_mod`).
- Bars colored by threshold crossing; annotated with exact probability.
---
## 3. Data Sources & API Endpoints Used
| Purpose | Endpoint | Frequency |
|---|---|---|
| State list / validation | Hardcoded enum (`ALLOWED_STATES`) | Once |
| District search | `/api/v1/districts?q=&state=` | On keystroke (debounced) |
| "Interesting" suggestions | `/api/v1/estimates?state=XX&year=21-22` sorted by `observed_arrests desc` | Once per state |
| Single district, all demographics | `/api/v1/distimates/{leaid}` or `/estimates?leaid=` with year/model filters | ~6 calls × 3 years = 18+ |
| National comparison rates | `/api/v1/states?state=&race=&sex=&year=21-22` aggregated across states, OR a dedicated national endpoint if available | Once per group |
| Model metadata | `/api/v1/models` | Once (cache) |
**Key data structures returned by the API:**
The `/estimates/{leaid}` endpoint returns one row per `race × sex × year × model` with:
```json
{
"leaid": "...",
"lea_name": "...",
"state": "TX",
"race": "BL",
"sex": "M",
"year": "21-22",
"model": "unified_m2_mod",
"stu_enroll": 35963,
"observed_arrests": 1,
"rate_median": 0.42,
"rate_lower": 0.18,
"rate_upper": 0.91,
"count_median": 15.3,
"count_lower": 6.7,
"count_upper": 28.9
}
```
The `/draws` endpoint returns a Hugging Face Parquet shard URL + DuckDB SQL for
bulk posterior draw access — useful if we need to compute custom quantities like
P(district > national) without round-tripping through multiple API calls. However,
for a demo app that runs in the browser, relying on 10 HF Parquet shards per model
is impractical (80 GB total). The summary endpoints (`/estimates`) provide enough
aggregated information for all six charts using only `rate_median`, `count_median`,
and interval bounds — **no raw draw access needed**.
---
## 4. Visual Design Language (from reference materials)
### Color palette
```css
:root {
--cv-paper: #FAF7F2; /* warm paper background */
--cv-ink: #0E1A2B; /* primary text — civic navy/black */
--cv-navy-600:#22406A; /* links, accents */
--cv-accent: #C25311; /* ember — alerts, highlights, eyebrows */
/* Data-viz supporting colors (from _tokens.scss) */
--teal-600: #1F6F70;
--plum-600: #6B3A5E;
--moss-600: #4A6B2F;
--brass-600: #B8751C;
}
```
### Typography
- **Display**: Libre Franklin / Inter — headings, stat callouts.
- **Body**: Inter — UI text, labels, captions.
- **Mono**: JetBrains Mono — code snippets, API URLs in footers.
### Chart patterns (from social media posts & white paper)
1. **Ridgeline density plots** for posterior draws (`geom_density_ridges`).
2. **Pointrange / error bar charts** comparing model intervals to observed values.
3. **Faceted small multiples** — always split by `covariate × time` (baseline vs.
covariate; one-year vs. three-year).
4. **Transparent PNG export with watermark logo** in bottom-right corner.
5. **"Stat callout" hero numbers** for key metrics (e.g., total arrests, rate per 1k).
---
## 5. Technology Stack Recommendation
### Recommended: React + Vite + vanilla CSS (static site)
| Layer | Choice | Rationale |
|---|---|---|
| Framework | **React 19** (no framework overhead; Vite dev server) | Component model for charts, built-in state management via hooks |
| Build tool | **Vite** | Fast HMR, native ES modules, trivial static export (`npm run build`) |
| Charting | **`@visx/visx`** or plain SVG/CSS animations | Lightweight, no heavy deps; we control every pixel to match Civilytics style |
| HTTP client | `fetch` with AbortController + exponential backoff | No extra dependency; browser-native |
| Styling | Plain CSS custom properties (no Tailwind) | Zero-runtime, matches the existing SCSS token system exactly |
| Deployment | Static site on any host (GitHub Pages, Vercel, Netlify) or Docker nginx container | Self-hostable; no backend required |
#### Why not Shiny / Quarto?
The API returns JSON and we need a dynamic SPA with loading states, debounced search, and animated transitions. A static React app is the most natural fit. R/Shiny would add unnecessary server-side complexity for what is fundamentally a browser-based data visualization demo. The existing `crdc-arrests` project uses Quarto + ggplot2 for reports; this demo app is a different artifact with different requirements.
#### Why not Svelte or Vue?
React has the largest ecosystem, best tooling (Vite), and most team familiarity. For a 6-chart SPA it's more than sufficient without being overkill.
### Docker option
A minimal `nginx:alpine` container serves the built static files. ~20 MB image. Can be run on your self-hosted fleet (`efron`, `maxwell`, etc.) behind Caddy/Traefik with TLS via Let's Encrypt — consistent with how you host other civilytics.org services.
---
## 6. Build & Deployment Strategy
### Local development
```bash
npm install # one-time: installs React, Vite, dev deps
npm run dev # starts Vite on localhost:5173
# Edit src/App.jsx / src/components/*.jsx — hot reload
```
### Production build (static)
```bash
npm run build # outputs dist/ with index.html + assets
# Upload dist/ to any static host, or:
docker build -t crdc-demo . && docker run -p 8080:80 crdc-demo
```
### Docker deployment (self-hosted on your fleet)
1. Build image locally or via CI: `docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo:latest .`
2. Push to local registry on `maxwell`.
3. Deploy via docker-compose or a simple systemd service with nginx container + Caddy reverse proxy for TLS termination.
```yaml
# docker-compose.yml (minimal)
services:
app:
image: crdc-demo:latest
ports: ["8080:80"]
restart: unless-stopped
```
### CI/CD (optional, GitHub Actions)
- On push to `main`: run lint + build, publish Docker image to local registry.
- No automated deployment — you control when new versions go live on the fleet.
---
## 7. File Structure
```
crdc-demo/
├── public/ # Static assets (logo, favicon)
│ └── civilytics-logo.svg
├── src/
│ ├── components/ # Reusable UI pieces
│ │ ├── StateSelector.jsx
│ │ ├── DistrictSearch.jsx
│ │ ├── LoadingAnimation.jsx
│ │ └── StatCallout.jsx
│ ├── charts/ # The 6 chart components
│ │ ├── ArrestsOverTime.jsx
│ │ ├── RateByGroupBar.jsx
│ │ ├── DistrictVsNational.jsx
│ │ ├── ModelDrawsComparison.jsx
│ │ ├── ObservedRateDensity.jsx
│ │ └── ExceedanceProbability.jsx
│ ├── hooks/
│ │ ├── useApi.js # fetch wrapper with retry/backoff
│ │ └── useDistrictData.js # orchestrates all 6 charts' data needs
│ ├── styles/
│ │ ├── tokens.css # Civilytics design tokens
│ │ └── main.css
│ ├── App.jsx # Main router: state → search → loading → charts
│ └── main.jsx # React entry point
├── Dockerfile
├── vite.config.js
└── package.json
```
---
## 8. Key Implementation Notes & Risks
### Risk 1 — API response time for multi-model data
Fetching all models × race×sex combinations for a single district requires ~40
individual `/estimates` calls (5 unified + 5 stratified models × 8 groups). Each
API call may take 200–500ms. **Mitigation**: use `Promise.allSettled()` to fire
all requests in parallel; show progress as batches resolve. The loading animation
should reflect this batching visually (e.g., rows of bars filling left-to-right).
### Risk 2 — National rate computation
The `/states` endpoint returns per-state aggregates, not a national total. To get
national arrest rates by student group for Chart 3 and Chart 6:
- **Option A**: Fetch all states (`limit=100`, iterate through ~50 pages) and sum.
Too slow for a browser app.
- **Option B**: Add a `/api/v1/national` endpoint to the API (server-side aggregate).
Requires modifying `crdc-arrests/api/` — quick R/SQL change but needs your approval.
- **Recommended short-term**: Cache national rates as a static JSON file generated
during data release and committed alongside this app. The white paper already
computes these values; we can extract them into a small fixture.
### Risk 3 — Chart complexity (Chart 5 & 6)
Charts 5 (density/ridge) and 6 (exceedance probability) require either raw posterior
draws or sufficient summary statistics to reconstruct distributions. The `/estimates`
endpoint provides `count_median`, `count_lower`, `count_upper` but not the full draw
distribution. **Mitigation**: Use interval bounds + median as a proxy for ridge plots
(showing just the 95% HPD region), and compute exceedance probability using a normal
approximation to the posterior (mean=median, sd derived from interval width). This is
a reasonable approximation for demonstration purposes but should be clearly labeled.
### Risk 4 — Mobile responsiveness
The social media figures are desktop-first PNGs. The web app must work on mobile:
- Use CSS Grid with `auto-fit` for the chart panel (1 column on mobile, 3 on desktop).
- Make the loading animation responsive.
- Ensure touch targets in district search are ≥44px.
---
## 9. Open Questions for You
Before implementation begins, I need your input on these decisions:
### Q1 — National rates source
Chart 3 and Chart 6 compare a district's arrest rate to the **national average**
for each student group. The API doesn't have a `/national` endpoint. How should we handle this?
- **(A)** Cache national rates as a static JSON fixture (generated from the white paper / release data). [Recommended — simplest, no API changes]
- **(B)** Add a `/api/v1/national` aggregate endpoint to `crdc-arrests/api/` and have the demo call it live.
- **(C)** Approximate by fetching the top 5 largest states' rates weighted by enrollment (fast but imprecise).
### Q2 — Model selection for distribution charts
Charts 4–6 show results from multiple model types (one-year, three-year, baseline, covariate). Should users be able to toggle which models are displayed, or should we always show all four quadrants as specified in your requirements?
- **(A)** Always show all four quadrants (1yr no-cov, 1yr cov, 3yr no-cov, 3yr cov) — matches the white paper's `wp_fig_district_intervals()` layout. [Recommended]
- **(B)** Let users toggle between unified vs. stratified models via a dropdown.
- **(C)** Default to showing only the recommended model (`unified_m2_mod`) with an "advanced" expando for all four.
### Q3 — Loading animation style
The patience animation should convey that data is being fetched across multiple endpoints/models. What visual approach do you prefer?
- **(A)** Animated grid of histogram bars (one per model×group) that fill sequentially as API calls resolve, with a counter showing "X of ~40 datasets loaded". [Recommended — matches the Bayesian posterior theme]
- **(B)** A simple spinner + text message ("Fetching 12 model results from the CRDC API…").
- **(C)** An abstract animation (e.g., particles converging) that loops while loading, with no per-request feedback.
### Q4 — Deployment target
Where should this be hosted? Your infrastructure is self-hosted (`efron`, `maxwell`, etc.), but you also mentioned civilytics.org. Which deployment approach do you want me to implement first?
- **(A)** Docker container ready for your fleet (nginx + static files) — I'll write the Dockerfile and docker-compose.yml, deploy on a dev host. [Recommended]
- **(B)** Static site optimized for GitHub Pages / Vercel with CI/CD via GitHub Actions.
- **(C)** Both A and B (Docker for self-hosting, plus GH Pages config).
---
## 10. Next Steps
Once you answer the four questions above, I'll begin implementation:
1. **Week 1**: Scaffold project (Vite + React), implement design tokens & typography, build state selector + district search with "interesting" suggestions.
2. **Week 2**: Implement loading animation; fetch all model data in parallel; build Charts 1–3 (observed/descriptive).
3. **Week 3**: Build Charts 4–6 (model comparison distributions); wire up national rate fixture if Q1=A.
4. **Week 4**: Polish transitions, mobile responsiveness, Dockerfile + deployment configs; write README with usage/deployment docs.
The app will be fully self-contained — no external dependencies beyond the public CRDC API and optionally a cached JSON fixture for national rates. All code follows Civilytics visual identity as documented in `theme/_tokens.scss` and demonstrated in `social_media_posts.md`.
+146
View File
@@ -0,0 +1,146 @@
# CRDC Arrests API Demo
A demonstration web application for the [CRDC School Arrest Rate API](https://crdc-api.civilytics.org/api/v1/), showing Bayesian model comparisons of school-based arrest rates across U.S. districts. Built with React + Vite, deployable as a static site or Docker container.
## Quick Start (Development)
```bash
# Install dependencies
npm ci
# Start dev server on localhost:5173
npm run dev
# Build for production
npm run build # outputs to dist/
npm run preview # serve built files locally
```
## What It Does
Visitors select a U.S. state, search for a school district (with suggestions of districts that have the most arrests), and see 3 charts comparing observed data against Bayesian model estimates:
1. **Arrests over time** (Chart 1) — raw counts by CRDC wave with per-1k rate labels, built with inline SVG
2. **Rate by student group** (Chart 2) — box-and-whisker per race×sex group for the most recent year, split into Female/Male panels. The box is the 25th–75th percentile of that group's real 500-draw posterior (same duckdb-wasm fetch as Chart 3), the whisker is the model's reported 90% interval, and a diamond marks the observed rate; individual boxes fall back to an analytic approximation when a group's draws can't be fetched.
3. **Predicted rates by student group** (Chart 3) — density ridges built from each group's real 500-draw posterior (fetched client-side via duckdb-wasm from the public Hugging Face parquet dataset), with a model-selector dropdown for the four Bayesian specifications and diamond markers for observed rates; falls back to an analytic approximation if the draws can't be fetched.
## Architecture
### Tech Stack
- **React 19** + **Vite** (static site generation, no backend required)
- Plain CSS custom properties for styling (matches Civilytics design tokens exactly)
- Inline SVG rendering with hand-rolled scales — no charting library, no D3 dependency
- Embeds **DuckDB-Wasm** (`@duckdb/duckdb-wasm`, ~39MB uncompressed / ~8.8MB gzipped, loaded on demand only after a district is selected) to query real posterior draws client-side from a public Hugging Face Parquet dataset
- Calls the public read-only API directly from the browser
### File Structure
```
crdc-demo/
├── index.html # Entry point
├── vite.config.mjs # Vite build config
├── public/ # Static assets (wordmark, favicon, fixtures)
│ ├── civilytics-wordmark.svg # Civilytics wordmark from civilyticsR package
│ └── data/national_rates.json # Static national rates fixture for comparisons
├── src/
│ ├── main.jsx # React entry
│ ├── App.jsx # Main router (state → search → loading → charts)
│ ├── hooks/
│ │ ├── useApi.js # API client with retry/backoff + endpoint wrappers
│ │ └── useDrawDistribution.js # Fetches real posterior draws (duckdb-wasm + HF parquet)
│ ├── components/
│ │ ├── StateSelector.jsx # Landing screen — state dropdown/grid
│ │ ├── DistrictSearch.jsx # Search + "interesting" district suggestions
│ │ ├── LoadingAnimation.jsx # Animated histogram grid during data fetch
│ │ ├── ChartLegend.jsx # Shared legend row
│ │ ├── ApproxNote.jsx # "shape estimated from interval bounds" caption
│ │ └── ChartPanel.jsx # Orchestrates all 3 charts + data fetching
│ ├── charts/
│ │ ├── ArrestsOverTime.jsx # Chart 1 — counts by wave (SVG)
│ │ ├── RateByGroupBar.jsx # Chart 2 — box-and-whisker by group (SVG)
│ │ └── RateDensityRidgeline.jsx # Chart 3 — density ridges per student group (SVG)
│ ├── utils/
│ │ ├── duckdbClient.js # Lazy duckdb-wasm bundle loader (dynamic import)
│ │ ├── kde.js # Empirical density from real draws (+ kde.test.js)
│ │ ├── drawGroups.js # Draw-map key format + coverage check (+ .test.js)
│ │ └── distributionApprox.js # Analytic fallback when draws are unavailable
│ └── styles/tokens.css # Civilytics design tokens (colors, fonts, spacing)
├── Dockerfile # Multi-stage build → nginx static server
├── docker-compose.yml # Local dev / self-hosted deployment
└── .gitea/workflows/pages.yml # CI/CD — builds and deploys to pages branch
```
### API Endpoints Used
| Endpoint | Purpose | Frequency |
|---|---|---|
| `/api/v1/models` | List available Bayesian model specs | Once (cached) |
| `/api/v1/districts?q=&state=` | District name/geo lookup → LEAID | On keystroke |
| `/api/v1/estimates/{leaid}?model=X&year=Y` | Estimates for one district/model/year/group | ~40 calls per district |
| `/api/v1/draws?...` | Locate raw-posterior Parquet shard | Not called from app — the app fetches shards directly from Hugging Face via duckdb-wasm; see `src/hooks/useDrawDistribution.js` |
| `/data/national_rates.json` | Static national rates fixture (committed) | Once per session |
## Deployment
### Option A: Git Pages via Gitea Actions (recommended)
The app is a static site deployed automatically via Gitea Actions. On every push to `main`, the workflow in
`.gitea/workflows/pages.yml` builds the Vite app and copies `dist/` to a `pages` branch served by the
git-pages webhook receiver.
**URL**: `https://pages.civilytics.org/crdc-demo/`
> **CORS note**: The CRDC API (`crdc-api.civilytics.org`) does not send CORS headers. When deployed as a static site,
> browser-based fetch requests will be blocked by the same-origin policy. To fix this, deploy `proxy.php` to your git-pages host
> and set the `VITE_PROXY_URL` environment variable (e.g., `/crdc-demo/proxy.php`). The app automatically detects
> proxy availability — when unset, it attempts direct API calls (works via Docker/nginx or same-origin setups).
The workflow follows the same pattern as `Civilytics/sln-school-comparison`:
1. Triggers on push to `main` (only when source files change)
2. Builds with Node 22 + Vite (`npm ci && npm run build`)
3. Copies `dist/` contents to an orphan `pages` branch
4. Forces push — the git-pages webhook picks it up automatically
> **Note**: Ensure `${{ secrets.GITHUB_TOKEN }}` is configured in repo settings on gitea.civilytics.org.
### Option A.5: CORS Proxy for Git Pages
If you can't deploy PHP, create a simple reverse proxy in nginx or use an edge function that:
1. Accepts `?target=/api/v1/...` as a query parameter
2. Forwards the request to `https://crdc-api.civilytics.org/api/v1/...
3. Returns the response with `Access-Control-Allow-Origin: *`
4. Set `VITE_PROXY_URL` at build time to point at this endpoint.
### Option B: Docker (self-hosted fleet) — includes CORS proxy nginx config:
```bash
docker compose up -d # builds + serves on localhost:8080
# Or build manually for your registry:
docker buildx build --platform linux/amd64 -t registry.civilytics.org/crdc-demo:latest .
docker push registry.civilytics.org/crdc-demo:latest
```
The container is nginx Alpine (~5MB) plus the built site, which is dominated by the ~39MB DuckDB-Wasm engine. `nginx.conf` serves that engine gzipped (~8.8MB on the wire) with immutable cache headers, so it is fetched once per browser. A `/healthz` endpoint supports container orchestration health checks.
### Environment Variables
| Variable | Default | Description |
|---|---|---|
| `VITE_API_BASE` | `https://crdc-api.civilytics.org/api/v1` | Override to point at a staging API (set in Dockerfile build or docker-compose) |
## Visual Design
This app follows the Civilytics visual identity as defined in:
- `theme/_tokens.scss` — colors, typography, spacing tokens
- `social_media_posts.md` — chart patterns from published social media figures
- `white_paper.qmd` / `R/paper_figures.R` — white paper figure builders
Key design decisions:
- **Color palette**: Warm paper background (`#FAF7F2`), civic navy text (`#0E1A2B`), ember accent (`#C25311`)
- **Data viz colors**: Teal, plum, moss, brass from the supporting palette
- **Fonts**: Libre Franklin/Inter for display and body, JetBrains Mono for code
- **Chart patterns**: Pointrange comparisons (model vs. observed), ridgeline-style density bars, faceted small multiples
## Citation & Attribution
The CRDC School Arrest Rate API: Knowles, J.E., & Miller, H. (2025). *CRDC School Arrest Rate API v1*. Civilytics Consulting. https://crdc-api.civilytics.org/api/v1/
Data source: US Department of Education, Office for Civil Rights. Civil Rights Data Collection (2021-22).
This research was supported by a grant from the American Educational Research Association which receives funds for its "AERA Grants Program" from the National Science Foundation under NSF award NSF-DRL #1749275. Opinions reflect those of the author and do not necessarily reflect those AERA or NSF.
+7
View File
@@ -0,0 +1,7 @@
# Git Pages configuration — for self-hosted static site deployment.
# This file is informational; the actual build + deploy happens via .gitea/workflows/pages.yml.
# The workflow builds the Vite app, then copies dist/ to a `pages` branch served by git-pages.
title: CRDC Arrests API Demo
description: "Explore school-based arrest rates by U.S. district with Bayesian model comparisons."
author: Civilytics Consulting LLC
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
var e=`/crdc-demo/assets/duckdb-browser-mvp.worker-C9hF7LGh.js`;export{e as default};
File diff suppressed because one or more lines are too long
-1
View File
@@ -1 +0,0 @@
var e=`/crdc-demo/assets/duckdb-mvp-BP0pRkMH.wasm`;export{e as default};
Binary file not shown.
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large Load Diff
-1
View File
@@ -1 +0,0 @@
654b42ca71f3654156e9b7a5e669dfa3c2fff41f
+12
View File
@@ -0,0 +1,12 @@
# Docker Compose for local development or self-hosted deployment on your fleet.
# Usage: docker compose up -d
services:
crdc-demo:
build: .
ports:
- "8080:80" # Map host port 8080 → container port 80 (nginx)
restart: unless-stopped
environment:
# Override to point at a staging API if needed; defaults to production.
VITE_API_BASE: https://crdc-api.civilytics.org/api/v1
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,187 @@
# Empirical draw distributions via DuckDB-Wasm — Design Spec
**Date:** 2026-08-11
**Status:** Draft for review
---
## 1. Purpose & context
Two of this app's three charts currently show a *modeled* distribution shape that is
not the real posterior — `src/utils/distributionApprox.js` fits a two-piece-normal
curve to each group's `(median, lower, upper)` summary stats returned by the
`/estimates` API, because the API's raw posterior draws are only available in bulk
as Hive-partitioned Parquet on Hugging Face
(`civilytics/crdc-school-arrest-rates`), meant for DuckDB/bulk consumption, not
browser fetches (see `AGENTS.md` §"API Endpoint Availability" and
`HANDOFF.md` §"Future Enhancements").
This spec replaces the approximation with the **real** empirical draws (500 per
group), fetched client-side using `@duckdb/duckdb-wasm` to query the actual Parquet
shard for the district's state directly from Hugging Face — no new backend
endpoint, no change to the existing summary API calls.
**Verified feasibility (2026-08-11):**
- The HF dataset is public, non-gated. Each `(model_id, YEAR, LEA_STATE)` partition
is a single file (`data_0.parquet`).
- File sizes range from ~130KB (DC) to ~6.4MB (CA, the largest state) — confirmed
by resolving the `resolve/main/...` redirect to the actual CDN blob.
- The redirect target sends `access-control-allow-origin: *` and
`accept-ranges: bytes` — browser `fetch()` works directly, no proxy needed.
- Schema (from `crdc-arrests/R/postprocess.R` + `R/export_parquet.R`):
`LEAID, LEA_STATE, YEAR, RACE, SEX, model_id, subgroup_id, draw_id, pred`, sorted
within each shard by `(LEAID, RACE, SEX)`. `LEA_STATE`/`YEAR`/`model_id` are
Hive-partition columns (encoded in the path, not repeated in every row).
`stu_enroll` is **not** in the draws table — it's already available in this app
from the existing `/estimates` summary call.
---
## 2. Architecture
```
district selected (leaid, state)
│
▼
resolve HF parquet URL for (model_id, YEAR=21-22, LEA_STATE=state)
e.g. https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/
resolve/main/parquet/model_id=unified_m4_mod/YEAR=21-22/LEA_STATE=CO/data_0.parquet
│
▼
fetch() the shard (native fetch, follows the HF→CDN redirect automatically)
│
▼
duckdb-wasm: registerFileBuffer + query
SELECT RACE, SEX, pred FROM shard WHERE LEAID = '<leaid>'
│
▼
join `pred` (posterior count draws) against stu_enroll already in app state
(from the existing /estimates summary call) → rate-per-1000 draws per group
│
▼
KDE per race×sex group → smooth density curve, same shape the charts draw today
```
Given verified shard sizes (≤6.4MB), the design fetches the **whole shard** with a
plain `fetch()` and queries it in-memory via duckdb-wasm, rather than relying on
fine-grained HTTP range / row-group pruning. This is simpler and more robust than
depending on duckdb-wasm's HTTP virtual filesystem correctly handling the HF→CDN
redirect chain under partial-range requests — an unverified behavior — for a
saving that wouldn't matter at these file sizes.
`@duckdb/duckdb-wasm` is MIT-licensed and runs entirely client-side in a Web
Worker. It introduces no new server dependency and no new hosted service beyond
the Hugging Face dataset the `crdc-arrests` project's `/draws` endpoint already
points to (per `2026-05-30-draws-api-design.md`, decision #5) — this spec doesn't
introduce that dependency, it makes the demo app actually use data that was
already published there for exactly this purpose.
**Deployment risk:** duckdb-wasm's threaded ("eh") bundle requires
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` response headers
(for `SharedArrayBuffer`), which the git-pages static host does not send today.
This design uses the **single-threaded ("mvp") bundle** instead — at these file
sizes threading has no meaningful benefit, and it avoids needing new headers on
both the git-pages and Docker/nginx deploy paths.
---
## 3. File-level changes
### New files
- **`src/utils/duckdbClient.js`** — lazy-initialized singleton. Dynamic-imports
`@duckdb/duckdb-wasm`, selects the MVP (non-threaded) bundle, starts the worker
once. Dynamic `import()` keeps the ~3–5MB wasm payload out of the main bundle;
it only loads when a chart actually needs draws.
- **`src/utils/kde.js`** — Gaussian KDE over an array of numbers (Silverman
bandwidth). Takes over the role `distributionApprox.js`'s `densityCurve` plays
today, fed real empirical draws instead of a parametric fit.
- **`src/hooks/useDrawDistribution.js`** — given
`{ leaid, state, model, year, groups }` (groups = race/sex + `stu_enroll` already
in app state), resolves the HF URL, fetches, registers the buffer with
duckdb-wasm, runs the query, joins enrollment, and returns
`{ status: 'loading' | 'ready' | 'error', drawsByGroup }`. Owns an in-memory
`Map` cache keyed by `model+state+year` so re-selecting a model in Chart 3's
dropdown, or viewing another district in the same state, reuses the shard
already fetched.
### Modified files
- **`RateDensityRidgeline.jsx`** — on model-dropdown change, calls
`useDrawDistribution` for the selected model; replaces
`fitSkewedInterval`/`densityCurve` with the hook's real draws → `kde.js`. Shows
an inline spinner in the ridge area while that model's shard is loading (Charts
1–2 aren't blocked).
- **`RateByGroupBar.jsx`** — fetches draws for `unified_m3_mod` (the one model
this chart uses) alongside its existing data fetch. `q1`/`q3` become exact
empirical quantiles from the 500 real draws — removes this chart's use of
`fitSkewedInterval`'s fitted quantile function entirely.
- **`ChartPanel.jsx`** — passes `district.leaid`, `state`, and each group's
`stu_enroll` (already fetched) down to the two charts above.
- **`ApproxNote.jsx`** — becomes conditional: renders the "estimated shape" note
only when a chart is in fallback mode; charts backed by real draws show no note
(or a neutral "500 posterior draws" caption).
- **`package.json` / `vite.config.mjs`** — add `@duckdb/duckdb-wasm`; wasm/worker
assets are pulled in via Vite's native `?url` imports, which already respect the
`/crdc-demo/` `base` path — no bundler plugin needed.
### Unchanged
- **`ArrestsOverTime.jsx`** — already uses real summary stats (point-range from
`/estimates`), no approximation involved; out of scope.
- **`useApi.js`** — still the source for medians, intervals, and enrollment.
- **`distributionApprox.js`** — kept as the fallback path (see §4).
---
## 4. Error handling & caching
**Fallback:** `distributionApprox.js` is retained. `useDrawDistribution` catches
fetch/wasm/query failures and returns `status: 'error'`; both chart components
branch on that to render the current analytic-approximation path with
`<ApproxNote />` visible. A Hugging Face outage, a network failure, or an
unsupported browser degrades to today's behavior rather than breaking the chart.
**Caching:** in-memory only (a `Map` inside the hook), scoped to the browser
session. No IndexedDB/persistent cache in this iteration — a demo session
typically covers one or two districts, and shards are cheap enough to refetch on
reload.
---
## 5. Testing
This project has no automated test suite (per `AGENTS.md`); this follows the
existing manual-verification convention:
1. `npm run dev`; walk a small state (DC or WY, ~130–200KB shard) and a large one
(CA or TX, several MB) through the full district-search flow.
2. Confirm both charts render from real draws; confirm the Network tab shows the
expected parquet fetch(es) and sizes.
3. Simulate failure (block the `huggingface.co` / CDN domain in devtools) and
confirm both charts fall back to the analytic approximation with the note
visible, rather than breaking.
4. Confirm the model dropdown in Chart 3 re-fetches on first selection and is
instant on re-selection (cache hit).
---
## 6. Open risk to de-risk first
Before wiring up the full UI, spike: does duckdb-wasm's MVP bundle load and query
correctly when deployed under the `/crdc-demo/` subpath on git-pages, and does
nginx/git-pages serve `.wasm` with a usable content type? Everything else in this
design is standard Vite asset handling already exercised elsewhere in the app, but
this specific combination (wasm worker + subpath base + static host) hasn't been
verified end-to-end and should be checked with a throwaway spike rather than
assumed.
---
## 7. Explicitly out of scope
- `ArrestsOverTime.jsx` (Chart 1) — no approximation to replace.
- A new server-side `/draws`-streaming API endpoint — explicitly rejected in favor
of client-side wasm access, per the brainstorming decision that led to this spec.
- Persistent (IndexedDB) caching of fetched shards.
- Fine-grained HTTP range / row-group-level partial reads — shard sizes are small
enough that whole-file fetch is simpler and sufficiently fast.
- Extending empirical draws to national/exceedance-probability views — those
charts aren't part of the current 3-chart demo.
+1 -2
View File
@@ -5,11 +5,10 @@
<link rel="icon" type="image/svg+xml" href="/crdc-demo/favicon.svg" /> <link rel="icon" type="image/svg+xml" href="/crdc-demo/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>CRDC Arrests API Demo — Civilytics</title> <title>CRDC Arrests API Demo — Civilytics</title>
<script type="module" crossorigin src="/crdc-demo/assets/index-naTE57ov.js"></script>
<link rel="stylesheet" crossorigin href="/crdc-demo/assets/index-DtZIgSE6.css">
</head> </head>
<body> <body>
<div id="root"></div> <div id="root"></div>
<!-- Vite injects the correct script tag with base-relative path --> <!-- Vite injects the correct script tag with base-relative path -->
<script type="module" src="/src/main.jsx"></script>
</body> </body>
</html> </html>
+77
View File
@@ -0,0 +1,77 @@
# nginx config for CRDC Arrests Demo — static SPA + API reverse proxy with CORS.
# When served via Docker, this proxies /api/v1/ requests to the CRDC API server-side,
# adding Access-Control-Allow-Origin: * so browser-based fetch works without issues.
server {
listen 80;
server_name _;
# Proxy all /api/v1/ requests to the CRDC Arrest Rate API with CORS headers
location /api/v1/ {
proxy_pass https://crdc-api.civilytics.org/api/v1/;
proxy_set_header Host crdc-api.civilytics.org;
proxy_ssl_verify off;
# Add CORS headers so browser-based requests work
add_header Access-Control-Allow-Origin "*" always;
add_header Access-Control-Allow-Methods "GET, OPTIONS" always;
add_header Access-Control-Allow-Headers "*" always;
if ($request_method = 'OPTIONS') {
return 204;
}
}
# Also support a simple proxy endpoint for git-pages-style deployments
location /crdc-demo/proxy/ {
# Extract the target path from query string and forward to CRDC API
proxy_pass https://crdc-api.civilytics.org$arg_target;
proxy_set_header Host crdc-api.civilytics.org;
proxy_ssl_verify off;
add_header Access-Control-Allow-Origin "*" always;
add_header Access-Control-Allow-Methods "GET, OPTIONS" always;
}
# Serve static files from the Vite build output
root /usr/share/nginx/html/crdc-demo/; # Adjust if base is different
index index.html;
# Compress text assets and — most importantly — the duckdb-wasm engine,
# which is ~39MB uncompressed and ~8.8MB gzipped. Without this it ships
# uncompressed on every cold load. Dynamic gzip rather than gzip_static
# because the Vite build emits no pre-compressed .gz files.
gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_proxied any;
gzip_comp_level 6;
gzip_types
text/plain
text/css
application/javascript
text/javascript
application/json
image/svg+xml
application/wasm;
location / {
try_files $uri $uri/ /index.html;
}
# Civilytics design tokens: immutable cache headers for static assets.
# `wasm` belongs here too — the duckdb engine is content-hashed by Vite and
# is by far the largest asset in the build, so it must not be re-fetched.
location ~* \.(js|css|wasm|png|jpg|jpeg|gif|svg|woff2|ttf)$ {
expires 1y;
add_header Cache-Control "public, max-age=31536000, immutable";
try_files $uri =404;
}
# Health check endpoint for container orchestration
location /healthz {
access_log off;
return 200 "ok";
add_header Content-Type text/plain;
}
}
+2586
View File
File diff suppressed because it is too large Load Diff
+31
View File
@@ -0,0 +1,31 @@
{
"name": "crdc-arrests-demo",
"version": "0.1.0",
"description": "CRDC School Arrest Rate API demonstration app — shows Bayesian model comparisons for school-based arrest rates across U.S. districts.",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview",
"test": "node --test 'src/**/*.test.js'",
"lint": "eslint src/ --ext .js,.jsx,.ts,.tsx",
"format": "prettier --write \"src/**/*.{js,jsx,css}\""
},
"keywords": [
"crdc",
"school-arrests",
"api-demo",
"civilytics"
],
"author": "Civilytics Consulting LLC",
"license": "MIT",
"devDependencies": {
"@duckdb/duckdb-wasm": "^1.32.0",
"@vitejs/plugin-react-swc": "^4.3.3",
"eslint": "^8.57.1",
"prettier": "^3.9.6",
"react": "^19.2.8",
"react-dom": "^19.2.8",
"vite": "^8.2.1"
},
"type": "module"
}
+57
View File
@@ -0,0 +1,57 @@
<?php
/**
* Simple CORS proxy for CRDC Arrest Rate API.
*
* Deploy this file to your git-pages host alongside the static site (e.g., at /crdc-demo/proxy.php).
* Then set VITE_PROXY_URL=/crdc-demo/proxy.php in your build environment.
*
* Usage: GET /proxy.php?target=/api/v1/districts?q=Denver&state=CO
*/
// Allow cross-origin requests from any origin
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Methods: GET, OPTIONS');
header('Access-Control-Allow-Headers: *');
header('Content-Type: application/json');
if ($_SERVER['REQUEST_METHOD'] === 'OPTIONS') {
http_response_code(204);
exit;
}
// Extract and validate the target path from query string
$target = isset($_GET['target']) ? $_GET['target'] : '';
if (empty($target)) {
http_response_code(400);
echo json_encode(['error' => 'Missing "target" parameter']);
exit;
}
// Construct the full API URL — only allow requests to crdc-api.civilytics.org for security
$apiBase = 'https://crdc-api.civilytics.org';
$url = $apiBase . $target;
// Validate that we're not proxying arbitrary URLs (prevent SSRF)
if (!str_starts_with($url, $apiBase . '/api/v1/')) {
http_response_code(403);
echo json_encode(['error' => 'Invalid target — must be under /api/v1/']);
exit;
}
// Fetch the API response and return it with CORS headers
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_TIMEOUT => 30,
CURLOPT_SSL_VERIFYPEER => false,
]);
$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
// Forward the response with appropriate status code and CORS headers
http_response_code($httpCode);
header('Access-Control-Allow-Origin: *');
echo $response;

Before

Width:  |  Height:  |  Size: 3.0 KiB

After

Width:  |  Height:  |  Size: 3.0 KiB

Before

Width:  |  Height:  |  Size: 5.9 KiB

After

Width:  |  Height:  |  Size: 5.9 KiB

View File

Before

Width:  |  Height:  |  Size: 128 B

After

Width:  |  Height:  |  Size: 128 B

+122
View File
@@ -0,0 +1,122 @@
import { useState, useEffect } from 'react'
import StateSelector from './components/StateSelector.jsx'
import DistrictSearch from './components/DistrictSearch.jsx'
import LoadingAnimation from './components/LoadingAnimation.jsx'
import ChartPanel from './components/ChartPanel.jsx'
import Footer from './components/Footer.jsx'
import { fetchDistrictEstimates } from './hooks/useApi.js'
// Two-letter USPS state code. The deep-link value flows into API request URLs
// and into the Hugging Face parquet shard path duckdb-wasm reads, so it gets
// validated here at the boundary rather than propagated verbatim.
const STATE_CODE_PATTERN = /^[A-Z]{2}$/
/**
* CRDC Arrests API Demo App — main router.
* Flow: state → district search (with interesting suggestions) → loading animation → charts
*/
export default function App() {
const [step, setStep] = useState('state') // 'state' | 'search' | 'loading' | 'results'
const [selectedState, setSelectedState] = useState(null)
const [district, setDistrict] = useState(null)
// On first load: check URL for ?leaid= and state to allow deep-linking
useEffect(() => {
const params = new URLSearchParams(window.location.search)
const leaid = params.get('leaid')
const rawState = params.get('state')
const stateParam = rawState ? rawState.trim().toUpperCase() : null
if (rawState && !STATE_CODE_PATTERN.test(stateParam)) {
// Malformed deep link — drop it and start at the state selector rather
// than passing an arbitrary string into fetch URLs and shard paths.
console.warn('Ignoring deep link: `state` is not a two-letter state code.')
return
}
if (leaid && stateParam) {
// Deep link (including browser back/forward landing on this URL): resolve
// the district name via a real estimates row, keyed by LEAID. The previous
// version called searchDistricts(leaid, ...), a name/text search — passing
// an LEAID as search text never matches, so it always fell back to "Unknown
// District".
setSelectedState(stateParam)
fetchDistrictEstimates(leaid, { model: 'unified_m2_mod', year: '21-22' }).then(rows => {
const row = rows?.[0]
setDistrict(row
? { leaid: row.leaid, lea_name: row.lea_name, state: row.state }
: { leaid, lea_name: 'Unknown District', state: stateParam })
setStep('loading')
}).catch(() => {
// Fallback on error
setDistrict({ leaid, lea_name: 'Unknown District', state: stateParam })
setStep('loading')
})
}
}, [])
const handleStateChange = (stateCode) => {
setSelectedState(stateCode)
setStep('search')
}
const handleDistrictSelect = (dist) => {
// Update URL for shareability
const params = new URLSearchParams({ leaid: dist.leaid, state: selectedState })
window.history.replaceState(null, '', `?${params}`)
setDistrict(dist)
setStep('loading')
}
return (
<div className="cv-app">
{/* Header */}
<header className="cv-header cv-wrap">
<a href="/crdc-demo" aria-label="Civilytics" style={{ display: 'block' }}>
<img
src={import.meta.env.BASE_URL + 'civilytics-wordmark.svg'}
alt="Civilytics — social science for the public good"
style={{ height: '2.5rem', width: 'auto' }}
/>
</a>
</header>
{/* Main content */}
<main style={{ maxWidth: '60rem', margin: '0 auto', padding: '0 var(--space-3)' }}>
{step === 'state' && (
<StateSelector onNext={handleStateChange} />
)}
{step === 'search' && selectedState && (
<DistrictSearch
state={selectedState}
onSelect={handleDistrictSelect}
onBack={() => setStep('state')}
/>
)}
{step === 'loading' && district && (
<LoadingAnimation district={district} state={selectedState} />
)}
{/* ChartPanel listens for data-ready event from LoadingAnimation */}
{step === 'results' && district && (
<ChartPanel district={district} state={selectedState} />
)}
{(step === 'loading' || step === 'results') && (
<button
className="btn-outline"
style={{ margin: 'var(--space-3) 0', display: 'block' }}
onClick={() => { window.location.href = '/crdc-demo/' }}
>
← Search another district
</button>
)}
</main>
<Footer />
</div>
)
}
+142
View File
@@ -0,0 +1,142 @@
import ChartLegend from '../components/ChartLegend.jsx'
import { OBSERVED_MARK_COLOR, MODELED_AGGREGATE_COLOR } from '../utils/colors.js'
import { niceTicks } from '../utils/niceTicks.js'
import { MODEL_QUADRANT_LABEL } from '../utils/labels.js'
/**
* Chart 1: Observed arrests by CRDC wave (line, diamond markers — the same
* "diamond = observed" convention as every other chart), overlaid with one
* model's predicted total (point-range) per wave, aligned to the same x
* position as its observed point rather than dodged aside, so the two are
* read as a pair.
*/
const WAVE_LABELS = { '15-16': '2015–16', '17-18': '2017–18', '21-22': '2021–22' }
const HEADROOM = 26 // px reserved at the top of the plot so a point's rate label has room to sit above it
export default function ArrestsOverTime({ data, modelId }) {
const maxArrests = Math.max(
...data.map((d) => Math.max(d.arrests, d.modeledUpper ?? 0)),
1
)
const { ticks, niceMax } = niceTicks(maxArrests)
const modelLabel = MODEL_QUADRANT_LABEL[modelId] || modelId
const width = 360
const height = 280
const margin = { top: 40, right: 40, bottom: 60, left: 65 }
const innerWidth = width - margin.left - margin.right
const innerHeight = height - margin.top - margin.bottom
const xScale = (i) => (i / Math.max(data.length - 1, 1)) * innerWidth
const yScale = (val) => HEADROOM + (innerHeight - HEADROOM) * (1 - val / niceMax)
return (
<div className="cv-card" style={{ padding: 'var(--space-2)' }}>
<h3 style={{ fontSize: '0.85rem', marginBottom: 0, color: 'var(--cv-ink-2)' }}>
Arrests over time — observed vs. modeled
</h3>
{modelLabel && (
<p style={{ fontSize: '0.78rem', fontStyle: 'italic', color: 'var(--cv-ink-3)', margin: '0.15rem 0 0' }}>
Modeled intervals use the {modelLabel.toLowerCase()} model.
</p>
)}
<svg width="100%" viewBox={`0 0 ${width} ${height}`} style={{ maxWidth: '100%', marginTop: 'var(--space-1)' }}>
<rect x={margin.left} y={margin.top} width={innerWidth} height={innerHeight}
fill="var(--cv-paper-2)" rx={4} />
{ticks.map((val) => {
const y = margin.top + yScale(val)
return (
<g key={`y-${val}`}>
<line x1={margin.left} y1={y} x2={margin.left + innerWidth} y2={y}
stroke="var(--cv-rule)" strokeWidth={1} />
<text x={margin.left - 8} y={y + 4} textAnchor="end"
fontSize="0.7rem" fill="var(--cv-ink-3)">{val.toLocaleString()}</text>
</g>
)
})}
<text x={15} y={margin.top + innerHeight / 2} textAnchor="middle"
fontSize="0.7rem" fill="var(--cv-ink-3)" transform={`rotate(-90 15 ${margin.top + innerHeight / 2})`}>
Total arrests (count)
</text>
{data.map((d, i) => (
<text key={d.year} x={margin.left + xScale(i)} y={height - margin.bottom + 15}
textAnchor="middle" fontSize="0.7rem" fill="var(--cv-ink)">
{WAVE_LABELS[d.year] || d.label}
</text>
))}
<text x={margin.left + innerWidth / 2} y={height - 5} textAnchor="middle"
fontSize="0.7rem" fill="var(--cv-ink-3)">CRDC wave</text>
{/* Modeled point-range per wave — drawn first, directly under the observed marks it pairs with */}
{data.map((d, i) => {
if (d.modeledMedian == null) return null
const cx = margin.left + xScale(i)
const cyMedian = margin.top + yScale(d.modeledMedian)
const cyLower = margin.top + yScale(d.modeledLower ?? d.modeledMedian)
const cyUpper = margin.top + yScale(d.modeledUpper ?? d.modeledMedian)
return (
<g key={`modeled-${d.year}`}>
<line x1={cx} y1={cyLower} x2={cx} y2={cyUpper} stroke={MODELED_AGGREGATE_COLOR} strokeWidth={2} />
<circle cx={cx} cy={cyMedian} r={4.5} fill={MODELED_AGGREGATE_COLOR} stroke="#fff" strokeWidth={1.25} />
</g>
)
})}
{/* Observed line */}
{data.length > 1 && (
<polyline
points={data.map((d, i) => `${margin.left + xScale(i)},${margin.top + yScale(d.arrests)}`).join(' ')}
fill="none" stroke={OBSERVED_MARK_COLOR} strokeWidth={2.5}
strokeLinejoin="round" strokeLinecap="round"
/>
)}
{/* Observed diamonds + rate-per-1k labels (halo behind the text keeps
it legible over the modeled whisker sharing the same column) */}
{data.map((d, i) => {
const cx = margin.left + xScale(i)
const cy = margin.top + yScale(d.arrests)
const ratePerK = d.enroll > 0 ? (d.arrests / (d.enroll / 1000)).toFixed(2) : '0.0'
// The first/last points sit flush on the plot's left/right edge, so a
// center-anchored label above them would overflow into the y-axis
// ticks or off the right edge — anchor those two outward instead.
const isFirst = i === 0
const isLast = i === data.length - 1
const anchor = isFirst ? 'start' : isLast ? 'end' : 'middle'
const labelX = isFirst ? cx + 7 : isLast ? cx - 7 : cx
return (
<g key={d.year}>
<rect x={cx - 5} y={cy - 5} width={10} height={10}
fill={OBSERVED_MARK_COLOR} stroke="#fff" strokeWidth={1.5}
transform={`rotate(45 ${cx} ${cy})`} />
<text x={labelX} y={cy - 16} textAnchor={anchor} fontSize="0.7rem" fontWeight={600}
fill="var(--cv-ink)" stroke="var(--cv-paper-2)" strokeWidth={3}
strokeLinejoin="round" paintOrder="stroke">{ratePerK}</text>
<text x={labelX} y={cy - 7} textAnchor={anchor} fontSize="0.6rem"
fill="var(--cv-ink-3)" stroke="var(--cv-paper-2)" strokeWidth={3}
strokeLinejoin="round" paintOrder="stroke">per 1k</text>
</g>
)
})}
</svg>
<ChartLegend items={[
{ shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
{ shape: 'dot', color: MODELED_AGGREGATE_COLOR, label: 'Modeled (median + 90% interval)' },
]} />
<p style={{ fontSize: '0.75rem', color: 'var(--cv-ink-3)', marginTop: 'var(--space-1)' }}>
Rate per 1,000 students labeled above each observed point. The modeled total sums each
selected group's median/interval independently, which approximates but is not exactly the
median of the combined total. Data from CRDC waves 2015–16 through 2021–22.
</p>
</div>
)
}
+162
View File
@@ -0,0 +1,162 @@
import ChartLegend from '../components/ChartLegend.jsx'
import ApproxNote from '../components/ApproxNote.jsx'
import { raceColor, OBSERVED_MARK_COLOR, SHORT_RACE_LABEL } from '../utils/colors.js'
import { fitSkewedInterval } from '../utils/distributionApprox.js'
import { groupKey, hasDrawsForAll } from '../utils/drawGroups.js'
import { quantile } from '../utils/kde.js'
import { niceTicks } from '../utils/niceTicks.js'
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
/**
* Arrest rate by student group, most recent year, disaggregated into two
* panels (Female / Male), each a horizontal box-and-whisker across the 4
* race categories. Whisker = the model's reported 90% interval; box = the
* 25th-75th percentile of the group's 500 real posterior draws (or, if draws
* are unavailable, the fitSkewedInterval approximation); white tick =
* median; dark diamond = observed rate.
*/
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
const SEX_PANELS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
const ROW_HEIGHT = 34
const MODEL = 'unified_m3_mod' // matches ChartPanel's WAVE_MODEL, which built `data`
function buildBox(d, draws) {
const median = Math.max(d.modeledMedian || 0, 0)
const lower = Math.max(Math.min(d.rateLower ?? median, median), 0)
const upper = Math.max(d.rateUpper ?? median, median)
if (draws && draws.length > 0) {
return { lower, upper, median, q1: Math.max(quantile(draws, 0.25), 0), q3: Math.max(quantile(draws, 0.75), median) }
}
const fit = fitSkewedInterval({ median, lower, upper })
return { lower, upper, median, q1: Math.max(fit.quantile(0.25), 0), q3: Math.max(fit.quantile(0.75), median) }
}
export default function RateByGroupBar({ data, leaid, state }) {
const groups = data.map((d) => ({ race: d.race, sex: d.sex, stuEnroll: d.enrollment }))
const { drawsByGroup } = useDrawDistribution({ leaid, state, model: MODEL, year: '21-22', groups })
// Only the RACE_ORDER × SEX_PANELS cells are actually drawn, so coverage is
// judged against those — not against everything the API returned. The hook's
// 'ready' status is an any-group signal and would over-claim here: individual
// boxes still fall back to fitSkewedInterval whenever their group is missing
// from the shard. A null map (loading, or the whole fetch failed) fails this
// check too, so the error path still shows the note.
const renderedGroups = data.filter(
(d) => RACE_ORDER.includes(d.race) && SEX_PANELS.some((p) => p.sex === d.sex),
)
const allGroupsHaveDraws = hasDrawsForAll(drawsByGroup, renderedGroups)
const maxRate = Math.max(
...data.map((d) => Math.max(d.observedRate, d.rateUpper ?? d.modeledMedian ?? 0)),
0.5
)
const { ticks, niceMax } = niceTicks(maxRate, 4)
return (
<div className="cv-card" style={{ padding: 'var(--space-2)' }}>
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
Arrest rate by student group — 2021–22 (per 1,000)
</h3>
{!allGroupsHaveDraws && <ApproxNote />}
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
{SEX_PANELS.map(({ sex, label }) => (
<SexPanel
key={sex}
label={label}
rows={data.filter((d) => d.sex === sex)}
drawsByGroup={drawsByGroup}
ticks={ticks}
niceMax={niceMax}
/>
))}
</div>
<ChartLegend items={[
...RACE_ORDER.filter((race) => data.some((d) => d.race === race)).map((race) => ({
shape: 'swatch', color: raceColor(race), label: SHORT_RACE_LABEL[race],
})),
{ shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
]} />
<p style={{ fontSize: '0.72rem', color: 'var(--cv-ink-3)', marginTop: 'var(--space-1)' }}>
{allGroupsHaveDraws
? "Box = 25th–75th percentile of 500 real posterior draws; whisker = the model's reported 90% interval; white tick = median."
: "Box = modeled 25th–75th percentile (fitted approximation); whisker = the model's reported 90% interval; white tick = median."}
{' '}The dark diamond is the observed rate.
</p>
</div>
)
}
function SexPanel({ label, rows, drawsByGroup, ticks, niceMax }) {
const width = 300
const margin = { top: 30, right: 16, bottom: 34, left: 66 }
const innerWidth = width - margin.left - margin.right
const byRace = RACE_ORDER.map((race) => rows.find((d) => d.race === race)).filter(Boolean)
const bodyHeight = byRace.length * ROW_HEIGHT
const height = margin.top + bodyHeight + margin.bottom
const xScale = (val) => margin.left + (val / niceMax) * innerWidth
return (
<div style={{ border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)', padding: 'var(--space-1)' }}>
<svg width="100%" viewBox={`0 0 ${width} ${height}`} style={{ maxWidth: '100%' }}>
<text x={width / 2} y={16} textAnchor="middle" fontSize="0.78rem" fontWeight={700} fill="var(--cv-ink-2)">
{label}
</text>
<rect x={margin.left} y={margin.top} width={innerWidth} height={bodyHeight} fill="var(--cv-paper-2)" rx={4} />
{ticks.map((val) => {
const x = xScale(val)
return (
<g key={val}>
<line x1={x} y1={margin.top} x2={x} y2={margin.top + bodyHeight} stroke="var(--cv-rule)" strokeWidth={1} />
<text x={x} y={margin.top + bodyHeight + 14} textAnchor="middle" fontSize="0.6rem" fill="var(--cv-ink-3)">
{val}
</text>
</g>
)
})}
{byRace.map((d, i) => {
const rowY = margin.top + i * ROW_HEIGHT
const midY = rowY + ROW_HEIGHT / 2
const boxTop = midY - ROW_HEIGHT * 0.26
const boxBottom = midY + ROW_HEIGHT * 0.26
const color = raceColor(d.race)
const draws = drawsByGroup?.[groupKey(d.race, d.sex)]
const box = buildBox(d, draws)
const observedX = xScale(d.observedRate)
return (
<g key={d.race}>
<line x1={xScale(box.lower)} y1={midY} x2={xScale(box.upper)} y2={midY} stroke={color} strokeWidth={1.5} />
<line x1={xScale(box.lower)} y1={boxTop} x2={xScale(box.lower)} y2={boxBottom} stroke={color} strokeWidth={1.5} />
<line x1={xScale(box.upper)} y1={boxTop} x2={xScale(box.upper)} y2={boxBottom} stroke={color} strokeWidth={1.5} />
<rect
x={xScale(box.q1)} y={boxTop} width={Math.max(xScale(box.q3) - xScale(box.q1), 1)} height={boxBottom - boxTop}
fill={color} fillOpacity={0.55} stroke={color} strokeWidth={1} rx={1.5}
/>
<line x1={xScale(box.median)} y1={boxTop} x2={xScale(box.median)} y2={boxBottom} stroke="#fff" strokeWidth={1.5} />
<rect
x={observedX - 4} y={midY - 4} width={8} height={8}
fill={OBSERVED_MARK_COLOR} stroke="#fff" strokeWidth={1}
transform={`rotate(45 ${observedX} ${midY})`}
/>
<text x={margin.left - 8} y={midY + 4} textAnchor="end" fontSize="0.72rem" fill="var(--cv-ink)">
{SHORT_RACE_LABEL[d.race] || d.race}
</text>
</g>
)
})}
<text x={margin.left + innerWidth / 2} y={height - 6} textAnchor="middle" fontSize="0.62rem" fill="var(--cv-ink-3)">
Rate per 1,000 students
</text>
</svg>
</div>
)
}
+204
View File
@@ -0,0 +1,204 @@
import { useState } from 'react'
import { MODEL_QUADRANTS } from '../hooks/useApi.js'
import { raceColor, OBSERVED_MARK_COLOR, RACE_LABELS, SHORT_RACE_LABEL } from '../utils/colors.js'
import { fitSkewedInterval, densityCurve } from '../utils/distributionApprox.js'
import { groupKey, hasDrawsForAll } from '../utils/drawGroups.js'
import { kdeCurve } from '../utils/kde.js'
import { niceTicks } from '../utils/niceTicks.js'
import { useDrawDistribution } from '../hooks/useDrawDistribution.js'
import ChartLegend from '../components/ChartLegend.jsx'
import ApproxNote from '../components/ApproxNote.jsx'
/**
* Modeled posterior density per race×sex group, for one selected model
* (dropdown, default three-year + referral rate), split into Female/Male
* columns. Each ridge is drawn from that group's 500 real posterior draws
* (Gaussian KDE) when available, falling back per-group to the analytic
* fitSkewedInterval/densityCurve approximation otherwise.
*/
const RACE_ORDER = ['WH', 'BL', 'HI', 'AM']
const SEX_COLUMNS = [{ sex: 'F', label: 'Female' }, { sex: 'M', label: 'Male' }]
const DEFAULT_MODEL = 'unified_m4_mod' // Three-year + referral rate
function buildGroupRow(row, draws) {
const enroll = row.stu_enroll || 0
const observedRate = enroll > 0 ? ((row.observed_arrests || 0) / enroll) * 1000 : 0
const rateMedian = (row.rate_median || 0) * 1000
const rateLower = (row.rate_lower || 0) * 1000
const rateUpper = (row.rate_upper || 0) * 1000
return {
race: row.race,
sex: row.sex,
observedRate,
draws,
fit: fitSkewedInterval({ median: rateMedian, lower: rateLower, upper: rateUpper }),
}
}
export default function RateDensityRidgeline({ quadData, leaid, state }) {
const [selectedModel, setSelectedModel] = useState(DEFAULT_MODEL)
const rows = (quadData && quadData[selectedModel]) || []
const groups = rows.map((r) => ({ race: r.race, sex: r.sex, stuEnroll: r.stu_enroll || 0 }))
const { drawsByGroup } = useDrawDistribution({ leaid, state, model: selectedModel, year: '21-22', groups })
// Only the RACE_ORDER × SEX_COLUMNS cells get a ridge, so coverage is judged
// against those. The hook's 'ready' status is an any-group signal: individual
// ridges still fall back to densityCurve when their group is missing from the
// shard, so gating the note on `status` alone would hide it while part of the
// chart is an approximation. A null map (loading or a failed fetch) also fails
// this check, so the error path still shows the note.
const renderedGroups = rows.filter(
(r) => RACE_ORDER.includes(r.race) && SEX_COLUMNS.some((c) => c.sex === r.sex),
)
const allGroupsHaveDraws = hasDrawsForAll(drawsByGroup, renderedGroups)
const modelSelect = (
<select
value={selectedModel}
onChange={(e) => setSelectedModel(e.target.value)}
style={{ padding: '0.25rem 0.5rem', fontFamily: 'var(--font-sans)', fontSize: '0.8rem' }}
>
{MODEL_QUADRANTS.map((q) => (
<option key={q.model} value={q.model}>{q.label}</option>
))}
</select>
)
return (
<div className="cv-card" style={{ padding: 'var(--space-2)' }}>
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'flex-start', flexWrap: 'wrap', gap: 'var(--space-2)' }}>
<div>
<h3 style={{ fontSize: '0.85rem', marginBottom: 'var(--space-1)', color: 'var(--cv-ink-2)' }}>
Predicted arrest rates by student group
</h3>
{!allGroupsHaveDraws && <ApproxNote />}
</div>
{modelSelect}
</div>
{rows.length === 0 ? (
<p style={{ color: 'var(--cv-ink-3)', marginTop: 'var(--space-2)' }}>No model data available.</p>
) : (
<>
<RidgeColumns rows={rows} drawsByGroup={drawsByGroup} />
<ChartLegend items={[
...RACE_ORDER.map((race) => ({ shape: 'swatch', color: raceColor(race), label: RACE_LABELS[race] })),
{ shape: 'diamond', color: OBSERVED_MARK_COLOR, label: 'Observed' },
]} />
</>
)}
</div>
)
}
function RidgeColumns({ rows, drawsByGroup }) {
// Shared x-domain across both columns, so Female/Male are directly comparable.
const allUpper = rows.map((r) => (r.rate_upper || 0) * 1000)
const allObserved = rows
.filter((r) => (r.stu_enroll || 0) > 0)
.map((r) => ((r.observed_arrests || 0) / r.stu_enroll) * 1000)
const rawMax = Math.min(Math.max(...allUpper, ...allObserved, 1) * 1.15, 30)
const { ticks, niceMax } = niceTicks(rawMax, 5)
return (
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 'var(--space-3)', marginTop: 'var(--space-2)' }}>
{SEX_COLUMNS.map(({ sex, label }) => (
<SexRidgeColumn
key={sex}
label={label}
rows={rows.filter((r) => r.sex === sex)}
drawsByGroup={drawsByGroup}
ticks={ticks}
maxRate={niceMax}
/>
))}
</div>
)
}
function SexRidgeColumn({ label, rows, drawsByGroup, ticks, maxRate }) {
const groups = RACE_ORDER
.map((race) => rows.find((r) => r.race === race))
.filter(Boolean)
.map((row) => buildGroupRow(row, drawsByGroup?.[groupKey(row.race, row.sex)]))
const width = 300
const rowHeight = 58
const margin = { top: 26, right: 16, bottom: 26, left: 56 }
const innerWidth = width - margin.left - margin.right
const height = margin.top + Math.max(groups.length, 1) * rowHeight + margin.bottom
const xScale = (val) => margin.left + (val / maxRate) * innerWidth
const curves = groups.map((g) =>
g.draws && g.draws.length > 0
? kdeCurve(g.draws, { min: 0, max: maxRate, n: 60 })
: densityCurve(g.fit, { min: 0, max: maxRate, n: 60 }),
)
const maxPdf = Math.max(...curves.flatMap((c) => c.map((p) => p.y)), 1e-9)
const peakHeight = rowHeight * 0.82
return (
<div style={{ border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)', padding: 'var(--space-1)' }}>
<svg width="100%" viewBox={`0 0 ${width} ${height}`} style={{ maxWidth: '100%' }}>
<text x={width / 2} y={16} textAnchor="middle" fontSize="0.78rem" fontWeight={700} fill="var(--cv-ink-2)">
{label}
</text>
{groups.length === 0 ? (
<text x={width / 2} y={height / 2} textAnchor="middle" fontSize="0.65rem" fill="var(--cv-ink-3)">
No data for this group
</text>
) : (
<>
{ticks.map((val) => {
const x = xScale(val)
return (
<g key={val}>
<line x1={x} y1={margin.top} x2={x} y2={margin.top + groups.length * rowHeight} stroke="var(--cv-rule)" strokeWidth={1} />
<text x={x} y={margin.top + groups.length * rowHeight + 14} textAnchor="middle" fontSize="0.58rem" fill="var(--cv-ink-3)">
{val}
</text>
</g>
)
})}
{groups.map((g, i) => {
const rowTop = margin.top + i * rowHeight
const baselineY = rowTop + rowHeight * 0.9
const curve = curves[i]
const color = raceColor(g.race)
const topPath = curve
.map((p, j) => `${j === 0 ? 'M' : 'L'}${xScale(p.x)},${baselineY - (p.y / maxPdf) * peakHeight}`)
.join(' ')
const areaPath = `${topPath} L${xScale(maxRate)},${baselineY} L${xScale(0)},${baselineY} Z`
const obsX = xScale(Math.min(g.observedRate, maxRate))
const obsY = baselineY - peakHeight * 0.15
return (
<g key={g.race}>
<text x={margin.left - 8} y={rowTop + rowHeight / 2 + 4} textAnchor="end" fontSize="0.68rem" fill="var(--cv-ink)">
{SHORT_RACE_LABEL[g.race] || g.race}
</text>
<path d={areaPath} fill={color} opacity={0.6} stroke="#fff" strokeWidth={0.5} />
<rect
x={obsX - 4} y={obsY - 4} width={8} height={8}
fill={OBSERVED_MARK_COLOR} stroke="#fff" strokeWidth={1}
transform={`rotate(45 ${obsX} ${obsY})`}
/>
</g>
)
})}
<text x={width / 2} y={height - 6} textAnchor="middle" fontSize="0.58rem" fill="var(--cv-ink-3)">
Arrests per 1,000 students
</text>
</>
)}
</svg>
</div>
)
}
+10
View File
@@ -0,0 +1,10 @@
import { DISTRIBUTION_APPROX_NOTE } from '../utils/distributionApprox.js'
/** Consistent caption for any chart that renders an approximated distribution shape. */
export default function ApproxNote() {
return (
<p style={{ fontSize: '0.78rem', fontStyle: 'italic', color: 'var(--cv-ink-3)', margin: '0.25rem 0 0' }}>
{DISTRIBUTION_APPROX_NOTE}
</p>
)
}
+43
View File
@@ -0,0 +1,43 @@
/**
* Plain-HTML chart legend — replaces every chart's hand-rolled SVG legend.
* Both of the app's stray-`}` CSS bugs lived inside hand-written SVG legend
* `fill="..."` strings; keeping legends out of SVG removes that bug class.
*
* items: [{ color, label, shape: 'swatch' | 'diamond' | 'line' | 'dot' }]
*/
export default function ChartLegend({ items }) {
return (
<div style={{ display: 'flex', flexWrap: 'wrap', gap: '1.25rem', marginTop: 'var(--space-2)' }}>
{items.map((item) => (
<div key={item.label} style={{ display: 'flex', alignItems: 'center', gap: '0.4rem' }}>
<LegendMark shape={item.shape} color={item.color} />
<span style={{ fontSize: '0.8rem', color: 'var(--cv-ink-2)' }}>{item.label}</span>
</div>
))}
</div>
)
}
function LegendMark({ shape, color }) {
if (shape === 'diamond') {
return (
<span
style={{
display: 'inline-block',
width: '9px',
height: '9px',
background: color,
transform: 'rotate(45deg)',
flexShrink: 0,
}}
/>
)
}
if (shape === 'line') {
return <span style={{ display: 'inline-block', width: '16px', height: '3px', background: color, borderRadius: '2px', flexShrink: 0 }} />
}
if (shape === 'dot') {
return <span style={{ display: 'inline-block', width: '10px', height: '10px', background: color, borderRadius: '50%', flexShrink: 0 }} />
}
return <span style={{ display: 'inline-block', width: '11px', height: '11px', background: color, borderRadius: '3px', flexShrink: 0 }} />
}
+124
View File
@@ -0,0 +1,124 @@
import { useState, useEffect } from 'react'
import * as api from '../hooks/useApi.js'
import { groupLabel } from '../utils/colors.js'
import ArrestsOverTime from '../charts/ArrestsOverTime.jsx'
import RateByGroupBar from '../charts/RateByGroupBar.jsx'
import RateDensityRidgeline from '../charts/RateDensityRidgeline.jsx'
const ALL_WAVES = ['15-16', '17-18', '21-22']
const QUADRANT_MODELS = ['unified_m1_mod', 'unified_m2_mod', 'unified_m3_mod', 'unified_m4_mod']
const WAVE_MODEL = 'unified_m3_mod' // three-year, no covariate — powers Charts 1 & 2
/**
* ChartPanel — 3 charts: arrests over time (observed vs. modeled), arrest
* rate by student group (Female/Male panels), and the model-selectable
* posterior density ridge chart.
*/
export default function ChartPanel({ district, state }) {
const [data, setData] = useState(null) // all fetched estimate rows keyed by year/model
const [loading, setLoading] = useState(true)
useEffect(() => {
async function fetchData() {
try {
// Chart 1 + 2: all 3 waves × three-year model (unified_m3_mod)
const waveData = {}
await Promise.all(ALL_WAVES.map(async (year) => {
try { waveData[year] = await api.fetchDistrictEstimates(district.leaid, { model: WAVE_MODEL, year }) } catch (e) {}
}))
// Chart 3: all 4 quadrant models, so the dropdown can switch between them
const quadData = {}
await Promise.all(QUADRANT_MODELS.map(async (model) => {
try { quadData[model] = await api.fetchDistrictEstimates(district.leaid, { model, year: '21-22' }) } catch (e) {}
}))
setData({ waveData, quadData })
} catch (err) {
console.error('ChartPanel data fetch failed:', err)
} finally {
setLoading(false)
}
}
fetchData()
}, [district.leaid])
if (loading || !data) {
return <LoadingCharts />
}
const mostRecent = data.waveData['21-22'] || []
// Chart 1: Arrests over time by wave — observed total + modeled (three-year model) point-range
const timeSeriesData = ALL_WAVES.map((year) => {
const yearRows = data.waveData[year] || []
return {
year,
label: `20${year.replace('-', '-')}`,
arrests: yearRows.reduce((sum, r) => sum + (r.observed_arrests || 0), 0),
enroll: yearRows.reduce((sum, r) => sum + (r.stu_enroll || 0), 0),
modeledMedian: yearRows.reduce((sum, r) => sum + (r.count_median || 0), 0),
modeledLower: yearRows.reduce((sum, r) => sum + (r.count_lower || 0), 0),
modeledUpper: yearRows.reduce((sum, r) => sum + (r.count_upper || 0), 0),
}
})
// Chart 2: rate by student group (most recent wave, per 1k), with modeled + observed + interval
const rateByGroup = mostRecent.map((r) => ({
race: r.race, sex: r.sex, label: groupLabel(r.race, r.sex),
observedRate: (r.observed_arrests || 0) / ((r.stu_enroll || 1) / 1000),
modeledMedian: (r.count_median || 0) / ((r.stu_enroll || 1) / 1000),
rateLower: (r.rate_lower || 0) * 1000,
rateUpper: (r.rate_upper || 0) * 1000,
observedArrests: r.observed_arrests || 0,
enrollment: r.stu_enroll || 0,
}))
return (
<div style={{ padding: 'var(--space-3) 0 var(--space-7)' }}>
{/* Chart panel header */}
<div style={{ marginBottom: 'var(--space-4)' }}>
<span className="eyebrow">District estimates — Bayesian model comparison</span>
<h2 style={{ marginTop: 'var(--space-1)', marginBottom: 0 }}>
{district.lea_name} ({state}) — School-based arrest rates, 2021–22 CRDC
</h2>
</div>
<div style={{
display: 'flex',
flexDirection: 'column',
gap: 'var(--space-5)',
maxWidth: '70rem',
marginLeft: 'auto',
marginRight: 'auto'
}}>
<ArrestsOverTime data={timeSeriesData} modelId={WAVE_MODEL} />
<RateByGroupBar data={rateByGroup} leaid={district.leaid} state={state} />
<RateDensityRidgeline quadData={data.quadData} leaid={district.leaid} state={state} />
</div>
{/* Methodology footer */}
<div style={{ marginTop: 'var(--space-6)', padding: 'var(--space-3) 0', borderTop: '1px solid var(--cv-rule)' }}>
<span className="eyebrow">Methodology</span>
<p style={{ fontSize: '0.85rem', color: 'var(--cv-ink-2)', marginTop: 'var(--space-1)' }}>
Estimates are from the CRDC School Arrest Rate API (Knowles & Miller 2025). Data shown spans three waves of
the Civil Rights Data Collection (2015–16, 2017–18, 2021–22) and lets you explore four Bayesian model
specifications: one-year vs. three-year models with and without referral-rate covariates. All rates are
per 1,000 students.
</p>
</div>
</div>
)
}
function LoadingCharts() {
return (
<div style={{ padding: 'var(--space-5) 0 var(--space-7)', textAlign: 'center' }}>
<span className="eyebrow">Preparing charts</span>
<p style={{ color: 'var(--cv-ink-3)' }}>Organizing data across all model specifications…</p>
</div>
)
}
+175
View File
@@ -0,0 +1,175 @@
import { useState, useEffect } from 'react'
import { searchDistricts, fetchStateDistricts } from '../hooks/useApi.js'
/**
* District search screen with:
* 1. "Interesting" suggestions — districts with the most arrests in the selected state (fetched once)
* 2. Live-search as you type → /api/v1/districts?q=...&state=XX
*/
export default function DistrictSearch({ state, onSelect, onBack }) {
const [query, setQuery] = useState('')
const [searchResults, setSearchResults] = useState([])
const [suggestions, setSuggestions] = useState([]) // "interesting" districts sorted by arrests desc
const [loadingSugg, setLoadingSugg] = useState(true)
const [loadingSearch, setLoadingSearch] = useState(false)
// ——— Fetch interesting suggestions (top arrests in this state) once on mount ———
useEffect(() => {
let cancelled = false
async function loadSuggestions() {
try {
// /estimates?state=XX&year=21-22 returns all districts; sort by observed_arrests desc client-side
const data = await fetchStateDistricts(state, '21-22', 500)
// Aggregate arrests per district (sum across race×sex groups), then sort
const byLeaid = {}
for (const row of data) {
if (!byLeaid[row.leaid]) {
byLeaid[row.leaid] = { leaid: row.leaid, lea_name: row.lea_name, state: row.state, observed_arrests: 0 }
}
byLeaid[row.leaid].observed_arrests += (row.observed_arrests || 0)
}
const sorted = Object.values(byLeaid).sort((a, b) => b.observed_arrests - a.observed_arrests)
if (!cancelled) {
// Take top 8 for suggestions; filter to those with >0 arrests
setSuggestions(sorted.filter(d => d.observed_arrests > 0).slice(0, 8))
}
} catch (err) {
console.error('Failed to load interesting districts:', err)
if (!cancelled) setSuggestions([]) // degrade gracefully — just show search box
} finally {
if (!cancelled) setLoadingSugg(false)
}
}
loadSuggestions()
return () => { cancelled = true }
}, [state])
// ——— Live search as user types (debounced by React's natural event batching) ———
useEffect(() => {
if (!query.trim()) { setSearchResults([]); return }
const timer = setTimeout(async () => {
setLoadingSearch(true)
try {
const data = await searchDistricts(query, state)
setSearchResults(data || [])
} catch (err) {
console.error('District search failed:', err)
setSearchResults([])
} finally {
setLoadingSearch(false)
}
}, 300)
return () => clearTimeout(timer)
}, [query, state])
// ——— Render a district card (used for both suggestions and search results) ———
const renderDistrictCard = (dist) => (
<button
key={dist.leaid}
className="cv-card"
style={{ textAlign: 'left', padding: 'var(--space-2)' }}
onClick={() => onSelect(dist)}
>
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'baseline' }}>
<span style={{ fontWeight: 600, color: 'var(--cv-navy-700)', fontSize: '0.95rem' }}>
{dist.lea_name || dist.name}
</span>
{dist.observed_arrests !== undefined && (
<span className="stat-label accent" style={{ marginTop: 0 }}>
{dist.observed_arrests.toLocaleString()} arrests in 2021–22
</span>
)}
</div>
</button>
)
return (
<div style={{ padding: 'var(--space-5) 0 var(--space-7)' }}>
{/* Back button */}
<button className="btn-outline" onClick={onBack} style={{ marginBottom: 'var(--space-3)' }}>
← Change state
</button>
<h2 style={{ marginTop: 0, marginBottom: 'var(--space-1)' }}>
{state === 'DC' ? "District of Columbia" : `Find a district in ${getStateName(state)}`}
</h2>
<p style={{ color: 'var(--cv-ink-2)', marginBottom: 'var(--space-3)' }}>
Search by district name, or try one of the suggested districts with the highest arrest counts.
</p>
{/* "Interesting" suggestions */}
{!query && (
<div style={{ marginTop: 'var(--space-4)' }}>
<span className="eyebrow">Suggested districts — most arrests in 2021–22</span>
{loadingSugg ? (
<p style={{ color: 'var(--cv-ink-3)', padding: 'var(--space-2) 0' }}>Loading suggestions…</p>
) : suggestions.length > 0 ? (
<div style={{ display: 'grid', gridTemplateColumns: 'repeat(auto-fill, minmax(280px, 1fr))', gap: 'var(--space-1)' }}>
{suggestions.map(renderDistrictCard)}
</div>
) : (
<p style={{ color: 'var(--cv-ink-3)', padding: 'var(--space-2) 0' }}>
No districts with reported arrests found in this state. Try searching by name below.
</p>
)}
</div>
)}
{/* Live search */}
<div style={{ marginTop: query ? 0 : 'var(--space-5)' }}>
<label htmlFor="district-search" className="stat-label">Search district name:</label>
<input
id="district-search"
type="text"
placeholder="Type a district name… (e.g., Madison, Derby, Paterson)"
value={query}
onChange={(e) => setQuery(e.target.value)}
style={{
width: '100%', padding: 'var(--space-2)', fontSize: '1rem',
border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)',
fontFamily: 'var(--font-sans)'
}}
/>
{query && (
<div style={{ marginTop: 'var(--space-2)' }}>
{loadingSearch ? (
<p style={{ color: 'var(--cv-ink-3)' }}>Searching…</p>
) : searchResults.length > 0 ? (
<div style={{ display: 'grid', gap: 'var(--space-1)' }}>
{searchResults.slice(0, 20).map(renderDistrictCard)}
</div>
) : (
<p style={{ color: 'var(--cv-ink-3)' }}>No districts found matching "{query}".</p>
)}
{/* Back to suggestions */}
{searchResults.length > 0 && (
<button className="btn-outline" onClick={() => setQuery('')} style={{ marginTop: 'var(--space-2)' }}>
← Show suggested districts again
</button>
)}
</div>
)}
</div>
{/* Hint */}
<p style={{ fontSize: '0.85rem', color: 'var(--cv-ink-3)', marginTop: 'var(--space-4)' }}>
Tip: Try districts like Derby (KS), Paterson (NJ), or Mobile County (AL) — they have notable arrest rates.
</p>
</div>
)
}
function getStateName(code) {
const names = {
AL:'Alabama',AK:'Alaska',AZ:'Arizona',AR:'Arkansas',CA:'California',CO:'Colorado',CT:'Connecticut',DE:'Delaware',DC:'District of Columbia',FL:'Florida',GA:'Georgia',HI:'Hawaii',ID:'Idaho',IL:'Illinois',IN:'Indiana',IA:'Iowa',KS:'Kansas',KY:'Kentucky',LA:'Louisiana',ME:'Maine',MD:'Maryland',MA:'Massachusetts',MI:'Michigan',MN:'Minnesota',MS:'Mississippi',MO:'Missouri',MT:'Montana',NE:'Nebraska',NV:'Nevada',NH:'New Hampshire',NJ:'New Jersey',NM:'New Mexico',NY:'New York',NC:'North Carolina',ND:'North Dakota',OH:'Ohio',OK:'Oklahoma',OR:'Oregon',PA:'Pennsylvania',RI:'Rhode Island',SC:'South Carolina',SD:'South Dakota',TN:'Tennessee',TX:'Texas',UT:'Utah',VT:'Vermont',VA:'Virginia',WA:'Washington',WV:'West Virginia',WI:'Wisconsin',WY:'Wyoming'
}
return names[code] || code
}
+24
View File
@@ -0,0 +1,24 @@
/**
* Footer — matches the Civilytics visual style from theme/civilytics.scss.
*/
export default function Footer() {
return (
<footer className="cv-footer cv-wrap" style={{ marginTop: 'var(--space-5)' }}>
<span>© Civilytics — social science for the public good.</span>
<div style={{ display: 'flex', gap: 'var(--space-2)', flexWrap: 'wrap' }}>
<a href="https://civilytics.com">civilytics.com</a>
<a href="https://github.com/civilytics/crdc-arrests" target="_blank" rel="noopener noreferrer">Source code</a>
<a href="https://crdc-api.civilytics.org/api/v1/" target="_blank" rel="noopener noreferrer">API docs</a>
</div>
{/* Methodology footnote */}
<div style={{ width: '100%', marginTop: 'var(--space-2)', padding: 'var(--space-1) 0', borderTop: '1px solid var(--cv-rule)' }}>
<span className="stat-label">About this demo</span>
<p style={{ fontSize: '0.8rem', color: 'var(--cv-ink-2)', marginTop: 'var(--space-0.5)' }}>
This application demonstrates the CRDC School Arrest Rate API, which provides Bayesian small-area estimates of school-based arrest rates from the US Department of Education Civil Rights Data Collection. Estimates use hierarchical models that improve precision for rare events — enabling meaningful comparisons across districts, student groups, and time. Cite: Knowles & Miller 2025.
</p>
</div>
</footer>
)
}
+209
View File
@@ -0,0 +1,209 @@
import { useState, useEffect } from 'react'
import ChartPanel from './ChartPanel.jsx'
import * as api from '../hooks/useApi.js'
import { STUDENT_GROUPS, MODEL_QUADRANTS } from '../hooks/useApi.js'
import { shortGroupLabel } from '../utils/colors.js'
// CRDC waves to fetch (for Chart 1 — time series)
const WAVES = ['21-22', '17-18', '15-16']
/**
* Loading screen with animated histogram grid.
* Fetches all required data in parallel, shows progress as bars fill.
* When complete → renders ChartPanel directly (no transition needed).
*/
export default function LoadingAnimation({ district, state }) {
const [loadedCount, setLoadedCount] = useState(0)
const [totalCalls, setTotalCalls] = useState(0)
const [error, setError] = useState(null)
// Build the full list of API calls needed for all 3 charts
useEffect(() => {
let cancelled = false
async function loadData() {
try {
// ——— Chart 1: Arrests over time (3 waves × default model) ———
const wavePromises = WAVES.map((year) =>
fetchDistrictEstimatesBatch(district.leaid, year)
.then(() => !cancelled && setLoadedCount(c => c + 1))
.catch(() => { /* individual failure doesn't block */ })
)
// ——— Chart 2: Rate by group (8 groups × default model, most recent wave) ———
const groupPromises = STUDENT_GROUPS.map((sg) =>
fetchDistrictEstimatesBatch(district.leaid, '21-22', sg.race, sg.sex)
.then(() => !cancelled && setLoadedCount(c => c + 1))
.catch(() => {})
)
// ——— Chart 3: posterior density, all 4 quadrant models so the dropdown can switch (8 groups × 4 = 32 calls) ———
const modelPromises = MODEL_QUADRANTS.flatMap((quad) =>
STUDENT_GROUPS.map((sg) =>
fetchDistrictEstimatesBatch(district.leaid, '21-22', sg.race, sg.sex, quad.model)
.then(() => !cancelled && setLoadedCount(c => c + 1))
.catch(() => {})
)
)
// Set total before starting (for progress bar)
const allPromises = [...wavePromises, ...groupPromises, ...modelPromises]
if (!cancelled) setTotalCalls(allPromises.length)
await Promise.all(allPromises)
if (!cancelled) {
// All data loaded — ChartPanel renders in place of this component
// We use a render prop pattern: return <ChartPanel /> when done
setLoadedCount(allPromises.length)
}
} catch (err) {
if (!cancelled) setError(err.message || 'Failed to load data')
}
}
loadData()
return () => { cancelled = true }
}, [district.leaid])
// ——— Render the histogram grid animation or ChartPanel when done ———
const isComplete = loadedCount >= totalCalls && totalCalls > 0
if (isComplete) {
return <ChartPanel district={district} state={state} />
}
if (error) {
return renderError(error, district)
}
// Build the grid of "bars" — one per API call needed.
// Total = WAVES (3) + STUDENT_GROUPS (8) + MODEL_QUADRANTS×STUDENT_GROUPS (4×8=32)
const TOTAL_BARS = WAVES.length + STUDENT_GROUPS.length + (MODEL_QUADRANTS.length * STUDENT_GROUPS.length)
// Build bar metadata for rendering
const allBars = Array.from({ length: TOTAL_BARS }, (_, i) => {
if (i < WAVES.length) return { subLabel: 'Time series', group: 'waves' }
if (i < WAVES.length + STUDENT_GROUPS.length) {
const sg = STUDENT_GROUPS[i - WAVES.length]
return { subLabel: shortGroupLabel(sg.race, sg.sex), group: 'groups' }
}
const quadIdx = Math.floor((i - WAVES.length - STUDENT_GROUPS.length) / STUDENT_GROUPS.length)
const sgIdx = (i - WAVES.length - STUDENT_GROUPS.length) % STUDENT_GROUPS.length
const sg = STUDENT_GROUPS[sgIdx]
return { subLabel: shortGroupLabel(sg.race, sg.sex), group: 'models', modelIdx: quadIdx }
})
return (
<div style={{ padding: 'var(--space-5) 0 var(--space-7)' }}>
{/* Header */}
<span className="eyebrow">Fetching data from the CRDC Arrest Rate API</span>
<h2 style={{ marginTop: 'var(--space-2)', marginBottom: 'var(--space-3)' }}>
Building estimates for {district.lea_name} in {state === 'DC' ? "District of Columbia" : getStateName(state)}
</h2>
{/* Progress counter */}
<div style={{ display: 'flex', justifyContent: 'space-between', marginBottom: 'var(--space-3)' }}>
<span style={{ color: 'var(--cv-navy-600)', fontWeight: 600, fontSize: '1.1rem' }}>
{loadedCount} of {totalCalls || allBars.length} datasets loaded
</span>
<span className="stat-label">
{Math.round((loadedCount / (totalCalls || allBars.length)) * 100)}% complete
</span>
</div>
{/* Animated histogram grid — mirrors the Bayesian posterior draw theme */}
<div style={{
display: 'grid',
gridTemplateColumns: `repeat(${3}, 1fr)`,
gap: 'var(--space-2)',
marginTop: 'var(--space-4)'
}}>
{/* Column headers */}
{['Time series (Chart 1)', 'By student group (Chart 2)', 'Posterior density (Chart 3)'].map((h, i) => (
<div key={i} style={{ textAlign: 'center', paddingBottom: 'var(--space-1)' }}>
<span className="stat-label" style={{ display: 'block' }}>{h}</span>
</div>
))}
{/* Grid of animated bars — each represents one API call */}
{allBars.map((bar, i) => {
const filled = i < loadedCount
// Random height for "data viz" aesthetic (deterministic via seed = index)
const heightSeed = ((i * 37) % 100) + 20 // 20–120px range
return (
<div key={i} style={{ display: 'flex', flexDirection: 'column', alignItems: 'center' }}>
{/* The animated bar */}
<div style={{
width: '100%', height: `${heightSeed}px`,
background: filled
? `var(--cv-navy-${600 - (i % 2) * 100})`
: 'var(--paper-3)',
border: '1px solid var(--cv-rule)',
borderRadius: '4px 4px 0 0',
transition: 'height 300ms ease, background 300ms ease',
position: 'relative',
overflow: 'hidden'
}}>
{filled && (
<div style={{
position: 'absolute', bottom: 0, left: 0, right: 0, height: '100%',
background: `linear-gradient(to top, var(--teal-600) 0%, transparent ${70 + (i % 30)}%)`,
opacity: 0.4, transition: 'opacity 500ms ease'
}} />
)}
</div>
{/* Label below */}
<span style={{ fontSize: '0.65rem', color: filled ? 'var(--cv-ink)' : 'var(--cv-ink-4)', marginTop: '0.25rem', textAlign: 'center' }}>
{bar.subLabel}
</span>
</div>
)
})}
</div>
{/* Context message */}
<p style={{ fontSize: '0.9rem', color: 'var(--cv-ink-2)', marginTop: 'var(--space-4)' }}>
Fetching {totalCalls || allBars.length} datasets across 3 CRDC waves, 8 student groups, and 4 Bayesian model specifications...
</p>
{/* Model legend */}
<div style={{ display: 'flex', gap: 'var(--space-2)', marginTop: 'var(--space-3)', flexWrap: 'wrap' }}>
{MODEL_QUADRANTS.map((q, i) => (
<span key={i} className="stat-label" style={{ display: 'inline-flex', alignItems: 'center', gap: '0.4rem' }}>
<span style={{ width: 12, height: 12, background: `var(--cv-navy-${600 - i * 150})`, borderRadius: 2 }} />
{q.label}
</span>
))}
</div>
</div>
)
}
// ——— API helper using the centralized client from useApi.js ———
// This ensures consistent error handling, envelope unwrapping, and CORS proxy support.
async function fetchDistrictEstimatesBatch(leaid, year, race = null, sex = null, model = 'unified_m2_mod') {
return api.fetchDistrictEstimates(leaid, { year, model, race, sex })
}
function getStateName(code) {
const names = {
AL:'Alabama',AK:'Alaska',AZ:'Arizona',AR:'Arkansas',CA:'California',CO:'Colorado',CT:'Connecticut',DE:'Delaware',DC:'District of Columbia',FL:'Florida',GA:'Georgia',HI:'Hawaii',ID:'Idaho',IL:'Illinois',IN:'Indiana',IA:'Iowa',KS:'Kansas',KY:'Kentucky',LA:'Louisiana',ME:'Maine',MD:'Maryland',MA:'Massachusetts',MI:'Michigan',MN:'Minnesota',MS:'Mississippi',MO:'Missouri',MT:'Montana',NE:'Nebraska',NV:'Nevada',NH:'New Hampshire',NJ:'New Jersey',NM:'New Mexico',NY:'New York',NC:'North Carolina',ND:'North Dakota',OH:'Ohio',OK:'Oklahoma',OR:'Oregon',PA:'Pennsylvania',RI:'Rhode Island',SC:'South Carolina',SD:'South Dakota',TN:'Tennessee',TX:'Texas',UT:'Utah',VT:'Vermont',VA:'Virginia',WA:'Washington',WV:'West Virginia',WI:'Wisconsin',WY:'Wyoming'
}
return names[code] || code
}
function renderError(error, district) {
return (
<div style={{ padding: 'var(--space-5) 0 var(--space-7)', textAlign: 'center' }}>
<span className="stat-label" style={{ color: 'var(--cv-danger)' }}>ERROR</span>
<h3 style={{ marginTop: 'var(--space-2)' }}>Failed to load data for {district.lea_name}</h3>
<p style={{ color: 'var(--cv-ink-2)', maxWidth: '40rem', margin: '0 auto var(--space-3)' }}>
{error}. This may be a temporary API issue or the district may not have available estimates.
</p>
<button className="btn-outline" onClick={() => window.location.href = '/'}>
← Try another district
</button>
</div>
)
}
+137
View File
@@ -0,0 +1,137 @@
import { useState } from 'react'
// All 50 states + DC, matching the API's ALLOWED_STATES
const STATES = [
'AL','AK','AZ','AR','CA','CO','CT','DE','DC','FL','GA','HI',
'ID','IL','IN','IA','KS','KY','LA','ME','MD','MA','MI','MN','MS','MO','MT',
'NE','NV','NH','NJ','NM','NY','NC','ND','OH','OK','OR','PA','RI','SC','SD',
'TN','TX','UT','VT','VA','WA','WV','WI','WY'
]
const STATE_NAMES = {
AL: 'Alabama', AK: 'Alaska', AZ: 'Arizona', AR: 'Arkansas', CA: 'California',
CO: 'Colorado', CT: 'Connecticut', DE: 'Delaware', DC: 'District of Columbia', FL: 'Florida',
GA: 'Georgia', HI: 'Hawaii', ID: 'Idaho', IL: 'Illinois', IN: 'Indiana', IA: 'Iowa',
KS: 'Kansas', KY: 'Kentucky', LA: 'Louisiana', ME: 'Maine', MD: 'Maryland', MA: 'Massachusetts',
MI: 'Michigan', MN: 'Minnesota', MS: 'Mississippi', MO: 'Missouri', MT: 'Montana',
NE: 'Nebraska', NV: 'Nevada', NH: 'New Hampshire', NJ: 'New Jersey', NM: 'New Mexico',
NY: 'New York', NC: 'North Carolina', ND: 'North Dakota', OH: 'Ohio', OK: 'Oklahoma',
OR: 'Oregon', PA: 'Pennsylvania', RI: 'Rhode Island', SC: 'South Carolina', SD: 'South Dakota',
TN: 'Tennessee', TX: 'Texas', UT: 'Utah', VT: 'Vermont', VA: 'Virginia',
WA: 'Washington', WV: 'West Virginia', WI: 'Wisconsin', WY: 'Wyoming'
}
/**
* Landing screen — prompts user to select a state.
* Matches the Civilytics visual style with eyebrow label, stat callout, and clean card layout.
*/
export default function StateSelector({ onNext }) {
const [query, setQuery] = useState('')
// Filter states by code or name as user types
const filtered = STATES.filter(
(code) =>
code.toLowerCase().includes(query.toLowerCase()) ||
STATE_NAMES[code].toLowerCase().includes(query.toLowerCase())
)
return (
<div style={{ padding: 'var(--space-5) 0 var(--space-7)' }}>
{/* Eyebrow */}
<span className="eyebrow">US Department of Education Civil Rights Data Collection</span>
{/* Hero */}
<h1 style={{ marginTop: 'var(--space-2)', marginBottom: 'var(--space-3)' }}>
Explore school-based arrest rates by district
</h1>
<p style={{ fontSize: '1.1rem', lineHeight: 1.7, maxWidth: '48rem' }}>
The Civil Rights Data Collection (CRDC) is the only source of data on school-related
arrests for all U.S. public schools and districts. Enter a state to see estimates for
any district — with Bayesian model comparisons that reveal how confident we can be in
each rate, even when arrests are rare events.
</p>
{/* Stat callout — key context */}
<div className="stat-callout" style={{ marginTop: 'var(--space-4)' }}>
<div>
<span className="stat-num">34,846</span>
<span className="stat-label">students arrested in 2021–22</span>
</div>
<div>
<span className="stat-num">0.72</span>
<span className="stat-label accent">arrests per 1,000 students (national rate)</span>
</div>
<div>
<span className="stat-num">11.6%</span>
<span className="stat-label">of districts reported &gt;0 arrests</span>
</div>
</div>
{/* Search input */}
<div style={{ marginTop: 'var(--space-5)' }}>
<label htmlFor="state-search" style={{ display: 'block', marginBottom: 'var(--space-2)', fontWeight: 600 }}>
Select a state to begin:
</label>
<input
id="state-search"
type="text"
placeholder="Type a state name or code (e.g., Texas, TX)…"
value={query}
onChange={(e) => setQuery(e.target.value)}
style={{
width: '100%', padding: 'var(--space-2)', fontSize: '1rem',
border: '1px solid var(--cv-rule)', borderRadius: 'var(--radius-md)',
fontFamily: 'var(--font-sans)'
}}
autoFocus
/>
{/* Filtered grid of states */}
{filtered.length > 0 && (
<div style={{
display: 'grid',
gridTemplateColumns: 'repeat(auto-fill, minmax(120px, 1fr))',
gap: 'var(--space-1)',
marginTop: 'var(--space-3)'
}}>
{filtered.map((code) => (
<button
key={code}
className="cv-card"
style={{ textAlign: 'center', padding: 'var(--space-2)' }}
onClick={() => onNext(code)}
>
<span style={{ fontSize: '1.25rem', fontWeight: 700, color: 'var(--cv-navy-600)' }}>
{code}
</span>
<span className="stat-label" style={{ marginTop: '0.25rem' }}>
{STATE_NAMES[code]}
</span>
</button>
))}
</div>
)}
{filtered.length === 0 && (
<p style={{ color: 'var(--cv-ink-3)', marginTop: 'var(--space-2)' }}>
No states match "{query}". Try a full state name or two-letter code.
</p>
)}
</div>
{/* Context footer */}
<div style={{ marginTop: 'var(--space-5)', paddingBottom: 'var(--space-3)', borderBottom: '1px solid var(--cv-rule)' }}>
<span className="eyebrow">How it works</span>
<p style={{ fontSize: '0.9rem', color: 'var(--cv-ink-2)', marginTop: 'var(--space-1)' }}>
After selecting a state, you'll search for a school district and see arrest estimates from 10 Bayesian models —
including one-year vs. three-year specifications and baseline vs. referral-rate covariate models.
The app fetches data directly from the{' '}
<a href="https://crdc-api.civilytics.org/api/v1/" target="_blank" rel="noopener noreferrer">
CRDC Arrest Rate API
</a>. No account or key required.
</p>
</div>
</div>
)
}
+148
View File
@@ -0,0 +1,148 @@
/**
* Civilytics CRDC Arrest Rate API client.
* Wraps fetch() with retry, backoff, and typed error handling.
* Base URL: https://crdc-api.civilytics.org/api/v1
*/
// The git-pages static host serves from pages.civilytics.org/crdc-demo/ but the CRDC API
// at crdc-api.civilytics.org does not send CORS headers. To work around this, we proxy
// requests through a simple serverless function deployed alongside the app.
// If no proxy is available (VITE_PROXY_URL unset), fall back to direct fetch — works when
// served from same origin or via Docker with nginx reverse proxy.
const BASE_URL = import.meta.env.VITE_API_BASE || 'https://crdc-api.civilytics.org/api/v1'
const PROXY_URL = import.meta.env.VITE_PROXY_URL // e.g., https://pages.civilytics.org/crdc-demo/proxy/
// Student group labels matching the API enum (race=AM|BL|HI|WH; sex=F|M)
export const STUDENT_GROUPS = [
{ race: 'WH', sex: 'F', label: 'White Female' },
{ race: 'WH', sex: 'M', label: 'White Male' },
{ race: 'BL', sex: 'F', label: 'Black Female' },
{ race: 'BL', sex: 'M', label: 'Black Male' },
{ race: 'HI', sex: 'F', label: 'Hispanic Female' },
{ race: 'HI', sex: 'M', label: 'Hispanic Male' },
{ race: 'AM', sex: 'F', label: 'American Indian\nFemale' },
{ race: 'AM', sex: 'M', label: 'American Indian\nMale' },
]
// CRDC wave labels (year=15-16|17-18|21-22)
export const CRDC_WAVES = [
{ value: '15-16', label: '2015–16' },
{ value: '17-18', label: '2017–18' },
{ value: '21-22', label: '2021–22' },
]
// Model IDs returned by /models endpoint
export const MODEL_IDS = [
'unified_m1_mod', 'unified_m2_mod', 'unified_m3_mod', 'unified_m4_mod', 'unified_m5_mod',
'stratified_m1_mod', 'stratified_m2_mod', 'stratified_m3_mod', 'stratified_m4_mod', 'stratified_m5_mod',
]
// The "four quadrants" for distribution charts (Charts 4–6)
export const MODEL_QUADRANTS = [
{ model: 'unified_m1_mod', label: 'One-year, no covariate' },
{ model: 'unified_m2_mod', label: 'One-year + referral rate' },
{ model: 'unified_m3_mod', label: 'Three-year, no covariate' },
{ model: 'unified_m4_mod', label: 'Three-year + referral rate' },
]
// Default/recommended model (from validate.R)
export const DEFAULT_MODEL = 'unified_m2_mod'
/** Exponential backoff fetch wrapper */
async function apiFetch(path, { retries = 3, delay = 500 } = {}) {
// If a proxy URL is configured, route through it to bypass CORS restrictions.
// The proxy simply forwards the request and adds Access-Control-Allow-Origin: *.
const url = PROXY_URL ? `${PROXY_URL}?target=${encodeURIComponent(path)}` : `${BASE_URL}${path}`
let lastError
for (let attempt = 0; attempt <= retries; attempt++) {
try {
const res = await fetch(url, { signal: AbortSignal.timeout(30000) })
if (!res.ok) throw new Error(`HTTP ${res.status}: ${res.statusText}`)
/** @type {{status:string,data:any,error?:string,meta:object}} */
const envelope = PROXY_URL ? await res.json() : (await res.json())
// When using a proxy that returns the raw API response, unwrap it.
if (PROXY_URL && envelope.status) return envelope.data
if (!PROXY_URL && envelope.status === 'success') return envelope.data
throw new Error(envelope.error || 'Unknown API error')
} catch (err) {
lastError = err
if (attempt < retries && !(err instanceof DOMException)) {
const wait = delay * Math.pow(2, attempt) + Math.random() * 100
await new Promise((r) => setTimeout(r, wait))
} else {
throw lastError
}
}
}
throw lastError
}
// ——— API endpoint wrappers (return the `data` array from the envelope) ———
/** GET /models — static list of available Bayesian model specifications */
export async function fetchModels() {
return apiFetch('/models')
}
/** GET /districts?q=<partial>&state=XX — name/geo lookup → LEAID */
export async function searchDistricts(query, state) {
const params = new URLSearchParams({ q: query, state })
return apiFetch(`/districts?${params}`)
}
/** GET /estimates/{leaid}?model=X&year=Y — all demographics for one district/year/model */
export async function fetchDistrictEstimates(leaid, options = {}) {
const params = new URLSearchParams()
if (options.model) params.set('model', options.model)
if (options.year) params.set('year', options.year)
if (options.race) params.set('race', options.race)
if (options.sex) params.set('sex', options.sex)
const qs = params.toString()
return apiFetch(`/estimates/${leaid}${qs ? `?${qs}` : ''}`)
}
/** GET /states/{state}?race=X&sex=Y&year=Z — state-level aggregate for one group */
export async function fetchStateEstimates(state, options = {}) {
const params = new URLSearchParams()
if (options.race) params.set('race', options.race)
if (options.sex) params.set('sex', options.sex)
if (options.year) params.set('year', options.year)
const qs = params.toString()
return apiFetch(`/states/${state}${qs ? `?${qs}` : ''}`)
}
/** GET /estimates?state=XX&year=Y — all districts in a state for "interesting" suggestions */
export async function fetchStateDistricts(state, year = '21-22', limit = 500) {
return apiFetch(`/estimates?state=${state}&year=${year}&limit=${limit}`)
}
/** GET /draws?... — returns HF parquet shard URL + DuckDB SQL (bulk only, not used in browser app) */
export async function fetchDrawLocation(state, race, sex, year, model) {
const params = new URLSearchParams({ state })
if (race) params.set('race', race)
if (sex) params.set('sex', sex)
if (year) params.set('year', year)
if (model) params.set('model', model)
return apiFetch(`/draws?${params}`)
}
/** GET /estimates/{leaid}/draws?model=X&year=Y — fetch posterior draws for a district */
export async function fetchDistrictDraws(leaid, options = {}) {
const params = new URLSearchParams()
if (options.model) params.set('model', options.model)
if (options.year) params.set('year', options.year)
if (options.race) params.set('race', options.race)
if (options.sex) params.set('sex', options.sex)
const qs = params.toString()
return apiFetch(`/estimates/${leaid}/draws${qs ? `?${qs}` : ''}`)
}
+150
View File
@@ -0,0 +1,150 @@
import { useEffect, useState } from 'react'
import { getDb } from '../utils/duckdbClient.js'
import { groupKey } from '../utils/drawGroups.js'
const HF_BASE = 'https://huggingface.co/datasets/civilytics/crdc-school-arrest-rates/resolve/main/parquet'
// Module-level cache: one registered duckdb-wasm file buffer per
// (model, year, state) shard, shared across every component instance and
// district navigated to in this browser session. See Global Constraints —
// in-memory only, no persistence across page loads.
const shardCache = new Map()
function shardKey(model, year, state) {
return `${model}__${year}__${state}`
}
function ensureShardRegistered(db, model, year, state) {
const key = shardKey(model, year, state)
if (!shardCache.has(key)) {
shardCache.set(
key,
(async () => {
try {
const url = `${HF_BASE}/model_id=${model}/YEAR=${year}/LEA_STATE=${state}/data_0.parquet`
const res = await fetch(url)
if (!res.ok) throw new Error(`Failed to fetch draw shard: HTTP ${res.status}`)
const buffer = new Uint8Array(await res.arrayBuffer())
const fileName = `${key}.parquet`
await db.registerFileBuffer(fileName, buffer)
return fileName
} catch (err) {
// Don't let a transient failure (network blip, HF outage) poison the
// cache forever — remove the rejected entry so the next caller for
// this shard gets a fresh attempt instead of the same dead promise.
shardCache.delete(key)
throw err
}
})(),
)
}
return shardCache.get(key)
}
/**
* Fetches real posterior draws for one district/model/year from the Hugging
* Face parquet dataset via duckdb-wasm, converts predicted counts to
* rate-per-1,000 using each group's stu_enroll (not present in the draws
* table itself — joined here from data this app already has), and returns
* them keyed by "RACE_SEX" (see `groupKey` in utils/drawGroups.js).
*
* `status` is an ANY-group signal: 'ready' means at least one group came back
* with real draws, not that every requested group did. Groups can be missing
* individually (falsy enrollment, or absent from the parquet shard), so a caller
* that wants to claim "these are all real draws" must check its own rendered
* groups against `drawsByGroup` — use `hasDrawsForAll` from utils/drawGroups.js.
*
* @param {{leaid: string, state: string, model: string, year: string,
* groups: Array<{race: string, sex: string, stuEnroll: number}>}} params
* @returns {{status: 'loading'|'ready'|'error', drawsByGroup: Record<string, number[]> | null}}
*/
export function useDrawDistribution({ leaid, state, model, year, groups }) {
const [status, setStatus] = useState('loading')
const [drawsByGroup, setDrawsByGroup] = useState(null)
// groups is typically a fresh array literal every render; derive a stable
// primitive so the effect only re-runs when its actual content changes.
const groupsSignature = (groups || []).map((g) => `${g.race}:${g.sex}:${g.stuEnroll}`).join(',')
useEffect(() => {
let cancelled = false
// Reset BEFORE the input guard below, not after. Clearing any previous
// model/district's draws has to happen on every input change, including the
// ones that have nothing to fetch. Without this ordering, a chart that
// varies `model` across renders (e.g. RateDensityRidgeline's dropdown) and
// lands on a model whose `groups` is empty (that model's upstream fetch
// failed) would keep reporting 'ready' and keep handing back the *previous*
// model's real draws — keyed by the same RACE_SEX strings — under the newly
// selected model's summary stats, silently mixing two models' data.
setStatus('loading')
setDrawsByGroup(null)
// Nothing to fetch: stay in 'loading' with no draws, which every consumer
// already treats as "fall back to the approximation". No cleanup needed —
// nothing async was started.
if (!leaid || !state || !model || !year || !groups?.length) return
async function run() {
let conn
try {
const db = await getDb()
const fileName = await ensureShardRegistered(db, model, year, state)
conn = await db.connect()
const stmt = await conn.prepare(`SELECT RACE, SEX, pred FROM read_parquet('${fileName}') WHERE LEAID = ?`)
const table = await stmt.query(leaid)
await stmt.close()
const rows = table.toArray().map((r) => r.toJSON())
const enrollByGroup = {}
for (const g of groups) enrollByGroup[groupKey(g.race, g.sex)] = g.stuEnroll || 0
const byGroup = {}
for (const row of rows) {
const key = groupKey(row.RACE, row.SEX)
const enroll = enrollByGroup[key]
if (!enroll) continue
const rate = (Number(row.pred) / enroll) * 1000
;(byGroup[key] ??= []).push(rate)
}
// A successful query with zero matching rows (this district isn't in
// the draws shard, or none of its rows matched a known group) is not
// "ready" — there's no real data to show, so treat it like a failure
// and let the caller fall back, rather than silently claiming real
// draws while every group actually uses the fitted approximation.
const hasDraws = Object.values(byGroup).some((draws) => draws.length > 0)
if (!cancelled) {
if (hasDraws) {
setDrawsByGroup(byGroup)
setStatus('ready')
} else {
console.warn('useDrawDistribution: query succeeded but returned no usable draws for', { leaid, model, year, state })
setDrawsByGroup(null)
setStatus('error')
}
}
} catch (err) {
console.error('useDrawDistribution failed:', err)
if (!cancelled) {
// Belt-and-suspenders alongside the setDrawsByGroup(null) at the
// top of this effect: a failed fetch/query must never leave a
// *previous* model's real draws in place under the newly-selected
// model's status/summary stats.
setDrawsByGroup(null)
setStatus('error')
}
} finally {
if (conn) await conn.close()
}
}
run()
return () => {
cancelled = true
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [leaid, state, model, year, groupsSignature])
return { status, drawsByGroup }
}
+10
View File
@@ -0,0 +1,10 @@
import React from 'react'
import ReactDOM from 'react-dom/client'
import App from './App.jsx'
import './styles/tokens.css'
ReactDOM.createRoot(document.getElementById('root')).render(
<React.StrictMode>
<App />
</React.StrictMode>,
)
+272
View File
@@ -0,0 +1,272 @@
/* =============================================================
Civilytics Design Tokens — single source of truth.
Mirrored from crdc-arrests/theme/_tokens.scss so this demo app
stays in sync with the white paper / social media visual style.
Neutrals: warm paper → civic ink
Brand: Civic Navy (primary) + Ember (accent)
VIZ: Supporting palette for data series
============================================================ */
:root {
/* — Paper (backgrounds) — */
--cv-paper: #FAF7F2;
--cv-paper-2: #F2EDE4;
--cv-paper-3: #E6DFD1;
/* — Rule / borders — */
--cv-rule: #D6CEBD;
--cv-rule-strong: #B8AE97;
/* — Ink (text) — */
--cv-ink: #0E1A2B;
--cv-ink-2: #2B3A52;
--cv-ink-3: #5A6A82;
--cv-ink-4: #8C97AB;
/* — Civic Navy (primary brand) — */
--cv-navy-50: #EEF3FA;
--cv-navy-100: #DDE6F2;
--cv-navy-200: #B3C6E0;
--cv-navy-300: #7A9BCA;
--cv-navy-400: #4A74B0;
--cv-navy-500: #2E5590;
--cv-navy-600: #22406A; /* links, primary buttons */
--cv-navy-700: #1A2E4A; /* hover state */
/* — Ember (accent) — */
--cv-accent: #C25311; /* main accent color */
--cv-accent-text: #A04400;
--cv-accent-dark: #923D00;
--cv-accent-hover: #7A3600;
/* — Supporting data-viz palette (from _tokens.scss) — */
--teal-600: #1F6F70;
--plum-600: #6B3A5E;
--moss-600: #4A6B2F;
--brass-600: #B8751C;
/* — Semantic status colors — */
--cv-success: #4A6B2F; /* moss */
--cv-warning: #9A5F18;
--cv-danger: #A6271D;
/* — Font stacks (from _tokens.scss) — */
--font-display: 'Libre Franklin', 'Franklin Gothic', 'Inter', system-ui, sans-serif;
--font-sans: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif;
--font-mono: 'JetBrains Mono', 'SF Mono', Menlo, Consolas, monospace;
/* — Spacing scale (from _tokens.scss) — */
--space-1: 0.5rem;
--space-2: 1rem;
--space-3: 1.5rem;
--space-4: 2rem;
--space-5: 3rem;
--space-6: 4rem;
--space-7: 6rem;
/* — Border radius (from _tokens.scss) — */
--radius-sm: 4px;
--radius-md: 6px;
--radius-lg: 12px;
/* — Race categorical palette (validated, see src/utils/colors.js) — */
--race-wh: #3D6FC4;
--race-bl: #C98A2A;
--race-hi: #9C3F86;
--race-am: #3D9A6B;
/* — Chart-specific colors — aliased onto the tokens charts actually use */
--chart-modeled: var(--cv-navy-600);
--chart-frequentist: var(--cv-ink-4);
--chart-observed: var(--cv-ink);
/* — Model quadrant colors — */
--model-one-year-baseline: var(--cv-navy-600);
--model-three-year-baseline: var(--teal-600);
}
/* =============================================================
Base reset + typography (from crdc-arrests/theme/civilytics.scss)
============================================================ */
* { box-sizing: border-box; }
html { font-size: 100%; -webkit-font-smoothing: antialiased; text-rendering: optimizeLegibility; }
body {
margin: 0;
background: var(--cv-paper);
color: var(--cv-ink);
font-family: var(--font-sans);
line-height: 1.65;
}
h1, h2, h3, h4 {
font-family: var(--font-display);
font-weight: 800;
letter-spacing: -0.025em;
line-height: 1.08;
color: var(--cv-ink);
text-wrap: pretty;
margin: 0 0 .4em;
}
h1 { font-size: clamp(2.25rem, 4.2vw, 3.75rem); font-weight: 900; letter-spacing: -0.035em; line-height: 1.0; }
h2 { font-size: clamp(1.75rem, 2.8vw, 2.375rem); }
h3 { font-size: 1.5rem; font-weight: 700; letter-spacing: -0.022em; color: var(--cv-ink-2); }
p, li { line-height: 1.7; text-wrap: pretty; }
a {
color: var(--cv-navy-600);
text-decoration: underline;
text-decoration-thickness: 1px;
text-underline-offset: 2px;
transition: color 120ms ease;
}
a:hover { color: var(--cv-navy-700); text-decoration-thickness: 2px; }
code, .mono { font-family: var(--font-mono); font-feature-settings: "tnum" 1, "zero" 1; }
:focus-visible { outline: 2px solid var(--cv-accent); outline-offset: 2px; }
/* =============================================================
Layout helpers (from civilytics.scss)
============================================================ */
.cv-wrap { max-width: 60rem; margin: 0 auto; padding: 0 var(--space-3); }
.cv-header {
display: flex; align-items: center; justify-content: space-between;
padding: var(--space-3) 0; border-bottom: 1px solid var(--cv-rule);
}
.cv-header .cv-logo { height: 28px; display: block; }
/* Eyebrow / section label (from civilytics.scss + extras.css) */
.eyebrow, .cv-eyebrow {
font-family: var(--font-sans);
font-size: 0.75rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.1em;
color: var(--cv-accent-text);
margin: 0 0 0.5rem;
display: inline-flex;
align-items: center;
gap: 0.5rem;
}
.eyebrow::before, .cv-eyebrow::before {
content: "";
display: inline-block;
width: 24px; height: 2px;
background: var(--cv-accent);
}
/* Stat callout — hero numbers (from extras.css) */
.stat-callout {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
gap: 2rem;
margin: 2rem 0 2.5rem;
padding: 1.5rem 0;
border-top: 3px double var(--cv-ink);
border-bottom: 1px solid var(--cv-rule);
}
.stat-callout .stat-num {
font-family: 'Source Serif 4', Georgia, serif;
font-weight: 700;
font-size: 3rem;
line-height: 1;
letter-spacing: -0.02em;
color: var(--cv-ink);
font-variant-numeric: tabular-nums;
display: block;
}
.stat-callout .stat-label {
font-family: var(--font-sans);
font-size: 0.75rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.08em;
color: var(--cv-ink-3);
margin-top: 0.4rem;
}
.stat-callout .stat-label.accent { color: var(--cv-accent); }
/* Card component (used for district suggestions, model cards) */
.cv-card {
display: flex;
flex-direction: column;
gap: 0.25rem;
background: #fff;
border: 1px solid var(--cv-rule);
border-top: 3px solid var(--cv-accent);
border-radius: var(--radius-lg);
padding: var(--space-2) var(--space-3);
text-decoration: none;
transition: border-color 120ms ease, box-shadow 120ms ease;
}
.cv-card:hover {
border-color: var(--cv-ink-3);
box-shadow: 0 2px 8px rgba(0,0,0,0.04);
}
/* Button (primary) */
.btn-primary {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 0.5rem;
padding: 0.625rem 1.25rem;
background: var(--cv-navy-600);
color: #fff;
border: none;
border-radius: var(--radius-md);
font-family: var(--font-sans);
font-size: 0.9rem;
font-weight: 600;
cursor: pointer;
transition: background 120ms ease, transform 20ms ease;
}
.btn-primary:hover { background: var(--cv-navy-700); }
.btn-primary:active { transform: scale(0.98); }
.btn-primary:disabled { opacity: 0.5; cursor: not-allowed; }
/* Button (secondary / outline) */
.btn-outline {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 0.5rem;
padding: 0.5rem 1rem;
background: transparent;
color: var(--cv-navy-600);
border: 1px solid var(--cv-rule);
border-radius: var(--radius-md);
font-family: var(--font-sans);
font-size: 0.85rem;
font-weight: 500;
cursor: pointer;
transition: background 120ms ease, color 120ms ease;
}
.btn-outline:hover {
background: var(--cv-paper-2);
color: var(--cv-navy-700);
}
/* Footer */
.cv-footer {
border-top: 1px solid var(--cv-rule);
padding: var(--space-4) 0;
color: var(--cv-ink-3);
font-size: 0.9rem;
display: flex;
flex-wrap: wrap;
gap: var(--space-2);
justify-content: space-between;
align-items: center;
}
/* Responsive */
@media (max-width: 680px) { .cv-footer { flex-direction: column; text-align: center; } }
+40
View File
@@ -0,0 +1,40 @@
/**
* Fixed categorical race palette — validated with the dataviz skill's
* validate_palette.js (passes lightness band, chroma floor, CVD separation
* ΔE 12.8, normal-vision floor ΔE 26.4; gold's sub-3:1 contrast WARN is
* mitigated by always-visible direct labels/legend in every chart that uses
* it). Order is fixed (WH, BL, HI, AM) and must not be re-derived from array
* position — re-run the validator before changing any of these hex values:
* node scripts/validate_palette.js "#3D6FC4,#C98A2A,#9C3F86,#3D9A6B" --mode light
*
* Sex (F/M) is deliberately NOT a second hue — it's encoded by row/facet
* position in every chart. Observed-vs-modeled is deliberately NOT a second
* hue either — it's encoded by mark type (diamond vs. filled bar/density).
*/
export const RACE_COLORS = { WH: '#3D6FC4', BL: '#C98A2A', HI: '#9C3F86', AM: '#3D9A6B' }
export const RACE_LABELS = { WH: 'White', BL: 'Black', HI: 'Hispanic', AM: 'American Indian / Alaska Native' }
// Compact form for axis ticks and other space-constrained labels, where the
// full RACE_LABELS text would overflow its container and get clipped by the
// SVG viewBox.
export const SHORT_RACE_LABEL = { WH: 'White', BL: 'Black', HI: 'Hispanic', AM: 'AI/AN' }
// Mark-type colors for the observed-vs-modeled convention. Used where a
// chart has no race facet of its own (aggregate totals).
export const OBSERVED_MARK_COLOR = 'var(--cv-ink)'
export const MODELED_AGGREGATE_COLOR = 'var(--cv-navy-600)'
// Fallback for an unrecognized race key — kept neutral rather than defaulting to a real race hue.
export const REFERENCE_GRAY = 'var(--cv-ink-4)'
export function raceColor(race) {
return RACE_COLORS[race] || REFERENCE_GRAY
}
export function groupLabel(race, sex) {
return `${RACE_LABELS[race] || race} ${sex === 'F' ? 'Female' : 'Male'}`
}
export function shortGroupLabel(race, sex) {
return `${SHORT_RACE_LABEL[race] || race} ${sex}`
}
+123
View File
@@ -0,0 +1,123 @@
/**
* Approximates a distribution shape from summary statistics only
* (median + interval bounds) when raw posterior draws aren't available
* client-side. This is NOT the true posterior — always pair its use with
* DISTRIBUTION_APPROX_NOTE (rendered via <ApproxNote />).
*
* Arrest rates/counts are right-skewed, not normal, so a single symmetric
* normal (the old approach) systematically misrepresents the shape. This
* fits two normal halves — one on each side of the median, each sized to
* its own interval bound — and splices their CDFs at the median. Because
* each half's CDF independently spans [0, 0.5] or [0.5, 1], the join is
* exactly the median by construction (cdf(median) === 0.5 always), unlike
* the classical two-piece-normal parameterization, which biases the median
* away from the split point whenever the two sigmas differ. Do not "fix"
* this toward that textbook formula — losing median-exactness is the bug,
* not a missing feature.
*/
const SQRT_2PI = Math.sqrt(2 * Math.PI)
const MIN_ABS_SIGMA = 1e-3
function standardNormalPdf(z) {
return Math.exp(-0.5 * z * z) / SQRT_2PI
}
// Abramowitz & Stegun 7.1.26 approximation of erf, ~1.5e-7 max error.
function erf(x) {
const sign = x < 0 ? -1 : 1
const ax = Math.abs(x)
const a1 = 0.254829592
const a2 = -0.284496736
const a3 = 1.421413741
const a4 = -1.453152027
const a5 = 1.061405429
const p = 0.3275911
const t = 1 / (1 + p * ax)
const y = 1 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-ax * ax)
return sign * y
}
function standardNormalCdf(z) {
return 0.5 * (1 + erf(z / Math.SQRT2))
}
// Peter Acklam's rational approximation of the inverse standard normal CDF
// (probit), relative error < 1.15e-9. Supports arbitrary intervalMass.
function probit(p) {
const a = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02, 1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00]
const b = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02, 6.680131188771972e+01, -1.328068155288572e+01]
const c = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00, -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00]
const d = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00, 3.754408661907416e+00]
const pLow = 0.02425
const pHigh = 1 - pLow
if (p < pLow) {
const q = Math.sqrt(-2 * Math.log(p))
return (((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5]) /
((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1)
}
if (p <= pHigh) {
const q = p - 0.5
const r = q * q
return (((((a[0] * r + a[1]) * r + a[2]) * r + a[3]) * r + a[4]) * r + a[5]) * q /
(((((b[0] * r + b[1]) * r + b[2]) * r + b[3]) * r + b[4]) * r + 1)
}
const q = Math.sqrt(-2 * Math.log(1 - p))
return -(((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5]) /
((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1)
}
/**
* @param {{median:number, lower:number, upper:number, intervalMass?:number, floorAtZero?:boolean}} p
* intervalMass: fraction of probability covered by [lower, upper] — the
* API's rate/count interval bounds are a 90% interval, so default 0.90.
* @returns {{ median:number, sigmaLeft:number, sigmaRight:number, pdf:(x:number)=>number, cdf:(x:number)=>number }}
*/
export function fitSkewedInterval({ median, lower, upper, intervalMass = 0.90, floorAtZero = true }) {
const z = probit((1 + intervalMass) / 2)
let sigmaLeft = (median - lower) / z
let sigmaRight = (upper - median) / z
if (!(sigmaLeft > 0)) sigmaLeft = Math.max(sigmaRight * 0.15, MIN_ABS_SIGMA)
if (!(sigmaRight > 0)) sigmaRight = Math.max(sigmaLeft * 0.15, MIN_ABS_SIGMA)
sigmaLeft = Math.max(sigmaLeft, MIN_ABS_SIGMA)
sigmaRight = Math.max(sigmaRight, MIN_ABS_SIGMA)
function pdf(x) {
if (floorAtZero && x < 0) return 0
const sigma = x < median ? sigmaLeft : sigmaRight
return standardNormalPdf((x - median) / sigma) / sigma
}
function cdf(x) {
if (floorAtZero && x < 0) return 0
const sigma = x <= median ? sigmaLeft : sigmaRight
return standardNormalCdf((x - median) / sigma)
}
/** Inverse cdf — exact round-trip of fitSkewedInterval's own construction (quantile(fit, 0.05) === lower, etc). */
function quantile(p) {
const z = probit(p)
return median + (p <= 0.5 ? sigmaLeft : sigmaRight) * z
}
return { median, sigmaLeft, sigmaRight, pdf, cdf, quantile }
}
/** n evenly spaced {x,y} points of the fitted pdf, for drawing a smooth curve. */
export function densityCurve(fit, { min = 0, max, n = 60 } = {}) {
const hi = max ?? fit.median + fit.sigmaRight * 3.5
const lo = Math.min(min, fit.median - fit.sigmaLeft * 0.1)
const step = (hi - lo) / (n - 1)
const points = []
for (let i = 0; i < n; i++) {
const x = lo + step * i
points.push({ x, y: fit.pdf(x) })
}
return points
}
export const DISTRIBUTION_APPROX_NOTE =
'Distribution shape estimated from interval bounds — not raw posterior draws.'
+33
View File
@@ -0,0 +1,33 @@
/**
* Shared key format and coverage check for the per-group posterior draws
* returned by `useDrawDistribution`. Both the hook (which builds the map) and
* the charts (which read it, and decide whether to claim "real draws") go
* through here so the key format lives in exactly one place.
*/
/** Canonical key for one race×sex group in a `drawsByGroup` map. */
export function groupKey(race, sex) {
return `${race}_${sex}`
}
/**
* True only when *every* group in `groups` has a non-empty draw array.
*
* `useDrawDistribution`'s `status` is an any-group signal: it reports 'ready'
* as soon as one group has real draws. Charts fall back per-group, so the
* chart-wide "these are real posterior draws" note/caption must be gated on
* complete coverage instead — a group can be missing because its enrollment is
* falsy or because its (LEAID, RACE, SEX) isn't in the parquet shard.
*
* Returns false for an empty group list (nothing rendered means nothing to
* claim) and for a null map (loading, or the fetch failed outright).
*
* @param {Record<string, number[]> | null | undefined} drawsByGroup
* @param {Array<{race: string, sex: string}>} groups - the groups a chart is
* actually rendering, not everything the API returned.
* @returns {boolean}
*/
export function hasDrawsForAll(drawsByGroup, groups) {
if (!drawsByGroup || !groups?.length) return false
return groups.every((g) => (drawsByGroup[groupKey(g.race, g.sex)]?.length ?? 0) > 0)
}
+41
View File
@@ -0,0 +1,41 @@
import { test } from 'node:test'
import assert from 'node:assert/strict'
import { groupKey, hasDrawsForAll } from './drawGroups.js'
test('groupKey: joins race and sex with an underscore', () => {
assert.equal(groupKey('BL', 'F'), 'BL_F')
})
test('hasDrawsForAll: true when every rendered group has draws', () => {
const map = { WH_F: [1, 2], BL_F: [3] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), true)
})
test('hasDrawsForAll: false when one rendered group is missing', () => {
// The any-group 'ready' status would still be true here — this is exactly the
// case where the chart must keep showing the approximation note.
const map = { WH_F: [1, 2] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), false)
})
test('hasDrawsForAll: false when a rendered group has an empty draw array', () => {
const map = { WH_F: [1, 2], BL_F: [] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), false)
})
test('hasDrawsForAll: false for a null map (loading or failed fetch)', () => {
assert.equal(hasDrawsForAll(null, [{ race: 'WH', sex: 'F' }]), false)
assert.equal(hasDrawsForAll(undefined, [{ race: 'WH', sex: 'F' }]), false)
})
test('hasDrawsForAll: false for an empty or missing group list', () => {
assert.equal(hasDrawsForAll({ WH_F: [1] }, []), false)
assert.equal(hasDrawsForAll({ WH_F: [1] }, undefined), false)
})
test('hasDrawsForAll: ignores groups the chart is not rendering', () => {
// Extra keys in the map (e.g. a race outside RACE_ORDER) must not block the
// claim for the groups actually on screen.
const map = { WH_F: [1], BL_F: [2], AS_F: [3] }
assert.equal(hasDrawsForAll(map, [{ race: 'WH', sex: 'F' }, { race: 'BL', sex: 'F' }]), true)
})
+26
View File
@@ -0,0 +1,26 @@
/**
* Lazy-initialized singleton AsyncDuckDB instance running the single-threaded
* MVP wasm bundle only — never eh/coi, which need Cross-Origin-Opener-Policy /
* Cross-Origin-Embedder-Policy response headers this static host doesn't send.
* See docs/superpowers/specs/2026-08-11-empirical-draws-wasm-design.md.
*/
let dbPromise = null
/** @returns {Promise<import('@duckdb/duckdb-wasm').AsyncDuckDB>} */
export function getDb() {
if (!dbPromise) dbPromise = initDb()
return dbPromise
}
async function initDb() {
const duckdb = await import('@duckdb/duckdb-wasm')
const mvpWorkerUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js?url')).default
const mvpWasmUrl = (await import('@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm?url')).default
const worker = new Worker(mvpWorkerUrl)
const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING)
const db = new duckdb.AsyncDuckDB(logger, worker)
await db.instantiate(mvpWasmUrl, null)
return db
}
+110
View File
@@ -0,0 +1,110 @@
/**
* Empirical density utilities for posterior draw arrays — Gaussian KDE with
* Silverman's rule-of-thumb bandwidth, plus a linear-interpolated quantile.
* Used in place of distributionApprox.js's analytic fitSkewedInterval/
* densityCurve approximation whenever real posterior draws are available.
*/
const SQRT_2PI = Math.sqrt(2 * Math.PI)
// Absolute last-resort floor, used only when no plotting domain is known.
const MIN_BANDWIDTH = 1e-3
// Bandwidth is clamped relative to the plotting domain, not to absolute units,
// because these draws are rate-per-1,000 values whose scale varies by orders of
// magnitude between districts. domainWidth/50 is slightly wider than one render
// step at the charts' n = 60 (step = domainWidth/59), so a degenerate draw set
// (e.g. 500 identical zeros, common for small districts) resolves as a narrow
// bump instead of a delta spike that flattens every other ridge sharing the
// column's maxPdf. It is also loose enough not to bind on an ordinary posterior:
// a spread wider than ~8% of the domain keeps its own Silverman bandwidth.
// domainWidth/6 stops a handful of extreme draws from inflating sd until the
// curve is a flat line.
const BANDWIDTH_FLOOR_DIVISOR = 50
const BANDWIDTH_CEILING_DIVISOR = 6
/**
* Linear-interpolated quantile (R type-7). Does not mutate `draws`.
* @param {number[]} draws
* @param {number} p - probability in [0, 1]
* @returns {number}
*/
export function quantile(draws, p) {
const sorted = [...draws].sort((a, b) => a - b)
const idx = p * (sorted.length - 1)
const lo = Math.floor(idx)
const hi = Math.ceil(idx)
if (lo === hi) return sorted[lo]
const frac = idx - lo
return sorted[lo] * (1 - frac) + sorted[hi] * frac
}
function standardDeviation(draws) {
const n = draws.length
const mean = draws.reduce((sum, d) => sum + d, 0) / n
const variance = draws.reduce((sum, d) => sum + (d - mean) ** 2, 0) / (n - 1)
return Math.sqrt(variance)
}
/**
* Silverman's rule-of-thumb bandwidth (robust variant using the smallest
* *positive* spread estimate among sd and IQR/1.34), clamped to a fraction of
* the plotting domain.
*
* Both ends of the clamp matter for the zero-inflated count posteriors small
* districts produce. Without the floor, an all-identical draw set (sd = IQR = 0)
* collapses to a delta-function spike. Without the ceiling, a group whose IQR is
* 0 (the normal case when most draws are 0) falls back to raw sd, which a
* handful of extreme draws inflates until the ridge is a featureless flat line.
*
* @param {number[]} draws
* @param {number} [domainWidth] - width of the x-range the curve will be drawn
* over. Omit only when no domain is known; the clamp then degrades to the
* absolute MIN_BANDWIDTH floor.
* @returns {number}
*/
export function silvermanBandwidth(draws, domainWidth = 0) {
const width = domainWidth > 0 ? domainWidth : 0
const floor = width ? width / BANDWIDTH_FLOOR_DIVISOR : MIN_BANDWIDTH
const ceiling = width ? width / BANDWIDTH_CEILING_DIVISOR : Infinity
const n = draws.length
if (n < 2) return floor
const sd = standardDeviation(draws)
const iqr = quantile(draws, 0.75) - quantile(draws, 0.25)
// Degrade gracefully: keep the robust rule when IQR is informative, use sd
// when it isn't, and let the floor handle a fully degenerate draw set —
// rather than treating a zero spread as "no estimate available".
const candidates = [sd, iqr / 1.34].filter((v) => v > 0)
const spread = candidates.length ? Math.min(...candidates) : 0
const raw = 0.9 * spread * Math.pow(n, -0.2)
return Math.min(Math.max(raw, floor), ceiling)
}
/**
* n evenly spaced {x, y} points of a Gaussian KDE over `draws` — same shape
* contract as distributionApprox.js's densityCurve, so chart code can switch
* between the two without changing its rendering path.
* @param {number[]} draws
* @param {{min?: number, max?: number, n?: number}} [options]
* @returns {Array<{x: number, y: number}>}
*/
export function kdeCurve(draws, { min = 0, max, n = 60 } = {}) {
const hi = max ?? Math.max(...draws) * 1.1
// Guarded against a degenerate/inverted domain so the bandwidth clamp can
// never be handed a negative width.
const domainWidth = Math.max(hi - min, 0)
const h = silvermanBandwidth(draws, domainWidth)
const step = (hi - min) / (n - 1)
const points = []
for (let i = 0; i < n; i++) {
const x = min + step * i
let sum = 0
for (const d of draws) {
const z = (x - d) / h
sum += Math.exp(-0.5 * z * z) / SQRT_2PI
}
points.push({ x, y: sum / (draws.length * h) })
}
return points
}
+107
View File
@@ -0,0 +1,107 @@
import { test } from 'node:test'
import assert from 'node:assert/strict'
import { quantile, silvermanBandwidth, kdeCurve } from './kde.js'
test('quantile: median of an odd-length array', () => {
assert.equal(quantile([3, 1, 2], 0.5), 2)
})
test('quantile: linear interpolation between two ranks', () => {
// sorted: [10, 20, 30, 40] — p=0.25 -> index 0.75 -> interpolate 10..20
assert.equal(quantile([40, 10, 30, 20], 0.25), 17.5)
})
test('quantile: does not mutate its input array', () => {
const input = [5, 3, 4, 1, 2]
quantile(input, 0.5)
assert.deepEqual(input, [5, 3, 4, 1, 2])
})
test('silvermanBandwidth: positive, finite floor for identical draws', () => {
const h = silvermanBandwidth([7, 7, 7, 7, 7], 10)
assert.ok(h > 0 && Number.isFinite(h))
assert.equal(h, 10 / 50, 'degenerate spread falls back to the domain-relative floor')
})
test('silvermanBandwidth: positive, finite floor for a single draw', () => {
const h = silvermanBandwidth([7], 10)
assert.ok(h > 0 && Number.isFinite(h))
assert.equal(h, 10 / 50)
})
test('silvermanBandwidth: positive, finite floor when no domain is supplied', () => {
assert.ok(silvermanBandwidth([7, 7, 7, 7, 7]) > 0)
assert.ok(silvermanBandwidth([7]) > 0)
})
test('silvermanBandwidth: clamped to [domainWidth/50, domainWidth/6]', () => {
const domainWidth = 30
// sd is inflated by 50 extreme draws; without the ceiling this is ~15.6.
const outlierDraws = [...Array(450).fill(0), ...Array(50).fill(200)]
assert.equal(silvermanBandwidth(outlierDraws, domainWidth), domainWidth / 6)
// Fully degenerate: sd = IQR = 0.
assert.equal(silvermanBandwidth(Array(500).fill(0), domainWidth), domainWidth / 50)
})
test('silvermanBandwidth: leaves an ordinary spread untouched by the clamp', () => {
const draws = Array.from({ length: 500 }, (_, i) => 3 + Math.sin(i) * 1.2 + (i % 7) * 0.15)
const h = silvermanBandwidth(draws, 10)
assert.ok(h > 10 / 50 && h < 10 / 6, `expected an unclamped bandwidth, got ${h}`)
})
test('kdeCurve: all-identical draws do not produce a delta-function spike', () => {
// 500 identical zeros is the common case for a small district's rare-event
// count posterior. The old absolute 1e-3 floor gave maxY ~399 here (peak
// ~3989x a uniform density over the same domain), which flattened every other
// ridge sharing the column's maxPdf to sub-pixel height.
const domainWidth = 10
const curve = kdeCurve(Array(500).fill(0), { min: 0, max: domainWidth, n: 60 })
const maxY = Math.max(...curve.map((p) => p.y))
assert.ok(Number.isFinite(maxY) && maxY > 0)
assert.ok(
maxY * domainWidth < 25,
`peak density should stay within ~25x a uniform density over the domain, got ${maxY * domainWidth}x`,
)
})
test('kdeCurve: zero-inflated draws keep their shape (not over-smoothed to flat)', () => {
const domainWidth = 10
const draws = [...Array(450).fill(0), ...Array(50).fill(1)]
const curve = kdeCurve(draws, { min: 0, max: domainWidth, n: 60 })
const ys = curve.map((p) => p.y)
const maxY = Math.max(...ys)
const meanY = ys.reduce((sum, y) => sum + y, 0) / ys.length
assert.ok(maxY / meanY > 3, `expected a peaked curve, got peak/mean ${maxY / meanY}`)
})
test('kdeCurve: outlier-inflated sd does not flatten the curve', () => {
// IQR is 0 here (most draws are 0), so the rule falls back to sd — which these
// 50 extreme draws inflate to ~60. Unclamped that gives h ~15.6 on a domain of
// 30, i.e. peak/mean ~1.6: a near-flat line claiming maximal uncertainty.
const domainWidth = 30
const draws = [...Array(450).fill(0), ...Array(50).fill(200)]
const curve = kdeCurve(draws, { min: 0, max: domainWidth, n: 60 })
const ys = curve.map((p) => p.y)
const maxY = Math.max(...ys)
const meanY = ys.reduce((sum, y) => sum + y, 0) / ys.length
assert.ok(maxY / meanY > 3, `expected a peaked curve, got peak/mean ${maxY / meanY}`)
})
test('kdeCurve: returns n points spanning [min, max]', () => {
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
const curve = kdeCurve(draws, { min: 0, max: 10, n: 60 })
assert.equal(curve.length, 60)
assert.equal(curve[0].x, 0)
assert.ok(Math.abs(curve[curve.length - 1].x - 10) < 1e-9)
})
test('kdeCurve: density integrates to ~1 over a wide domain (trapezoidal check)', () => {
const draws = [1, 2, 2, 3, 4, 5, 5, 5, 6, 8]
const curve = kdeCurve(draws, { min: -20, max: 30, n: 2000 })
let area = 0
for (let i = 1; i < curve.length; i++) {
const dx = curve[i].x - curve[i - 1].x
area += (dx * (curve[i].y + curve[i - 1].y)) / 2
}
assert.ok(Math.abs(area - 1) < 0.01, `expected area ~1, got ${area}`)
})
+6
View File
@@ -0,0 +1,6 @@
import { MODEL_QUADRANTS } from '../hooks/useApi.js'
/** model id -> display label, derived from the single source of truth in useApi.js. */
export const MODEL_QUADRANT_LABEL = Object.fromEntries(
MODEL_QUADRANTS.map((q) => [q.model, q.label])
)
+40
View File
@@ -0,0 +1,40 @@
/**
* "Nice numbers" axis tick generator (Heckbert 1990) — produces clean,
* human-friendly tick values (0 / 50 / 100 / 150 rather than 0 / 37 / 74 /
* 111, an artifact of slicing frac*max into equal fractions) for any linear
* axis, from just the data's max value.
*/
function niceNumber(range, round) {
const exponent = Math.floor(Math.log10(range))
const fraction = range / 10 ** exponent
let niceFraction
if (round) {
if (fraction < 1.5) niceFraction = 1
else if (fraction < 3) niceFraction = 2
else if (fraction < 7) niceFraction = 5
else niceFraction = 10
} else {
if (fraction <= 1) niceFraction = 1
else if (fraction <= 2) niceFraction = 2
else if (fraction <= 5) niceFraction = 5
else niceFraction = 10
}
return niceFraction * 10 ** exponent
}
/**
* @param {number} maxValue - the largest value the axis must cover
* @param {number} tickCount - approximate desired number of ticks
* @returns {{ ticks: number[], niceMax: number }}
*/
export function niceTicks(maxValue, tickCount = 5) {
if (!(maxValue > 0)) return { ticks: [0], niceMax: 1 }
const niceRange = niceNumber(maxValue, false)
const niceStep = niceNumber(niceRange / (tickCount - 1), true)
const niceMax = Math.ceil(maxValue / niceStep) * niceStep
const ticks = []
for (let v = 0; v <= niceMax + niceStep / 2; v += niceStep) {
ticks.push(Math.round(v * 1e6) / 1e6)
}
return { ticks, niceMax }
}
+17
View File
@@ -0,0 +1,17 @@
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react-swc'
// The app is served at /crdc-demo/ on pages.civilytics.org.
// Set base to the subdirectory so asset paths resolve correctly.
export default defineConfig({
plugins: [react()],
server: { port: 5173 },
build: { outDir: 'dist' },
base: '/crdc-demo/', // Required for subdirectory deployment on git-pages
optimizeDeps: {
// duckdb-wasm ships its own worker + wasm binaries resolved via `?url`
// imports; esbuild's dev-server pre-bundling can rewrite those import
// paths and break worker instantiation. Exclude it from pre-bundling.
exclude: ['@duckdb/duckdb-wasm'],
},
})