jared
c1920c11fb
Ports Figs 6 and 7 from crdc-arrests/R/paper_figures.R (wp_fig_group_density and
wp_fig_group_difference) so someone who has read the paper sees the paper. The
three old charts become a summary table and two charts; ArrestsOverTime stays.
Draws pipeline. useDrawDistribution now selects draw_id and returns predicted
counts indexed by draw rather than pre-divided rates. That one column is what
unlocks the rest: pooling has to sum numerators and denominators separately, and
a between-group difference has to be taken at a common draw index. Indexing by
draw_id rather than push order makes DuckDB's row ordering irrelevant and turns
a missing draw into a hole, which isCompleteDrawSet then rejects — a group
present for 300 of 500 draws would otherwise get an interval computed off a
biased subsample that looks identical on screen to a complete one. It also takes
models[] instead of a single model, so "compare all four" needs no conditional
hooks. The reset-before-guard ordering is preserved.
Also fixes a pre-existing bug in that pipeline: a state's draws are split across
data_0.parquet, data_1.parquet, … and the part count varies by state. The app
only ever fetched data_0. Nevada has one part, so this was invisible in every
Nevada test; California has eight, totalling 6.2MB, of which data_0 is 37KB and
holds 11 of California's 1,715 districts. Every other CA district looked absent
from the published data and silently fell back to the approximation. Parts are
now discovered from the Hugging Face tree listing API and fetched in parallel —
listing rather than probing data_N until a 404, because the browser logs a 404
as a console error however cleanly the fetch handles it, and a red error on
every load is indistinguishable from a real one. HEAD probing is the fallback.
New pure utils, all written against tests first:
agrestiCoull faithful port incl. the zero-numerator rule of three and the
negative lower bound at (1, 53); pinned to five R outputs
pooling sex pooling for sparse districts; numerator and denominator
are always drawn from the same set of groups
densityProfile discrete probability mass below 12 distinct values, KDE above;
KDE delegates to kde.js, whose bandwidth clamp is untouched
districtGroups display-row derivation, defaults, pooled vs unpooled keys
groupDifference per-draw delta; refuses to pair mismatched draw sets
rateDomain shared x-axis, with a clip flag so the cap is never silent
Chart A keeps the palette contract: race is hue, sex is position. Female and
Male are stacked panels sharing one axis, collapsing to one panel when pooled —
never a second hue. Chart B uses a diverging ramp centred at zero rather than
the paper's sequential YlOrRd, because delta is signed and a sequential ramp
encodes "more" where the data means "which direction".
Captions say "posterior predictive draws", never "paired parameter draws":
draw_id is renumbered per write batch upstream and a district's groups land in
different batches, so cross-group pairing is effectively independent (measured
cor ~= 0.02). The published figure has the same property; what neither can claim
is a paired-parameter contrast.
Sparse districts (under 20 arrests district-wide) pool Female and Male within
each race, with a banner stating the rule and a switch to override it. A pooled
group carries no modelled interval — summing two groups' interval bounds is not
a pooled interval, and there is no honest way to fake one without the draws.
React still owns the DOM. d3-scale/shape/array/interpolate supply scales, path
generators and colour interpolation; no selections, no useEffect DOM mutation.
Deletes RateByGroupBar and RateDensityRidgeline. Keeps distributionApprox.js and
ApproxNote.jsx — still the per-group fallback when draws can't be fetched.
112 tests pass; npm run build clean.
115 lines
4.6 KiB
JavaScript
115 lines
4.6 KiB
JavaScript
/**
|
||
* Chooses an honest visual profile for one group's posterior predictive draws.
|
||
*
|
||
* In a sparse district the posterior predictive is a *discrete count*
|
||
* distribution, not a smooth one. Carson City NV (3200390) has a 53-student
|
||
* AI/AN female cell where a single arrest is 18.9 per 1,000: the draws take
|
||
* four distinct values, and a Gaussian KDE renders those four spikes as a lumpy
|
||
* smear that reads as a rendering bug rather than as a finding about the data.
|
||
*
|
||
* So: few distinct values → draw the actual probability mass at each achievable
|
||
* rate. Many → the smooth KDE, delegated to `kde.js` unchanged (its bandwidth
|
||
* clamp was tuned for exactly these zero-inflated posteriors — see
|
||
* BANDWIDTH_FLOOR_DIVISOR there; do not re-derive it here).
|
||
*/
|
||
|
||
import { kdeCurve } from './kde.js'
|
||
import { toRates } from './pooling.js'
|
||
|
||
/**
|
||
* At or below this many distinct predicted counts, the draws are shown as
|
||
* discrete mass rather than smoothed.
|
||
*
|
||
* 12 is comfortably above the 3–5 distinct values a genuinely sparse cell
|
||
* produces and comfortably below the ~40+ a district with real arrest volume
|
||
* produces, so the switch happens well away from either regime rather than
|
||
* flickering at the boundary.
|
||
*/
|
||
export const MASS_MAX_DISTINCT = 12
|
||
|
||
/**
|
||
* @param {number[] | null | undefined} counts - predicted counts, indexed by draw
|
||
* @param {number | null | undefined} enroll - students in the group
|
||
* @param {{min?: number, max?: number, n?: number}} [domain] - the x-range the
|
||
* profile will be drawn over, in rate per 1,000. Passed straight to
|
||
* `kdeCurve`, whose bandwidth clamp is relative to this width.
|
||
* @returns {{kind: 'kde'|'mass', points: Array<{x: number, y: number}>,
|
||
* maxY: number, step: number} | null}
|
||
* `null` when there is nothing to draw (no draws, or no denominator).
|
||
* For `kind: 'mass'`, `y` is a probability and `step` is the spacing between
|
||
* achievable rates — the bar width for a filled staircase. For `kind: 'kde'`,
|
||
* `y` is a density and `step` is 0.
|
||
* `maxY` is the profile's own peak: because the two kinds carry different y
|
||
* units, a chart overlaying several groups must normalize each profile by its
|
||
* own `maxY` rather than by a shared maximum.
|
||
*/
|
||
export function densityProfile(counts, enroll, domain = {}) {
|
||
const rates = toRates(counts, enroll)
|
||
if (!rates.length) return null
|
||
// Counts are integers, so consecutive achievable rates are exactly one
|
||
// student-rate apart — pass that explicitly rather than inferring it from the
|
||
// gaps actually observed, which overstates the bar width when (say) only
|
||
// counts 0 and 3 appear.
|
||
return rateProfile(rates, domain, 1000 / enroll)
|
||
}
|
||
|
||
/**
|
||
* The same choice made directly on a set of rates, for quantities that are
|
||
* already differences rather than count/denominator pairs (see
|
||
* `utils/groupDifference.js`). A difference of two discrete posteriors is
|
||
* itself discrete, and deserves the same honesty.
|
||
*
|
||
* @param {number[] | null | undefined} rates - values on the plotted scale
|
||
* @param {{min?: number, max?: number, n?: number}} [domain]
|
||
* @param {number} [explicitStep] - known spacing between achievable values;
|
||
* inferred from the smallest observed gap when omitted.
|
||
* @returns {{kind: 'kde'|'mass', points: Array<{x: number, y: number}>,
|
||
* maxY: number, step: number} | null}
|
||
*/
|
||
export function rateProfile(rates, domain = {}, explicitStep = 0) {
|
||
if (!rates?.length) return null
|
||
|
||
const { min = 0, max, n = 60 } = domain
|
||
|
||
// Round before tallying so two draws that differ only in floating-point noise
|
||
// count as one achievable value rather than two.
|
||
const tally = new Map()
|
||
for (const r of rates) {
|
||
const key = Math.round(r * 1e9) / 1e9
|
||
tally.set(key, (tally.get(key) || 0) + 1)
|
||
}
|
||
|
||
if (tally.size <= MASS_MAX_DISTINCT) {
|
||
const total = rates.length
|
||
const points = [...tally.entries()]
|
||
.map(([x, freq]) => ({ x, y: freq / total }))
|
||
.sort((a, b) => a.x - b.x)
|
||
return {
|
||
kind: 'mass',
|
||
points,
|
||
maxY: Math.max(...points.map((p) => p.y)),
|
||
step: explicitStep > 0 ? explicitStep : inferStep(points, max, min),
|
||
}
|
||
}
|
||
|
||
const points = kdeCurve(rates, { min, max, n })
|
||
return {
|
||
kind: 'kde',
|
||
points,
|
||
maxY: Math.max(...points.map((p) => p.y)),
|
||
step: 0,
|
||
}
|
||
}
|
||
|
||
/** Smallest gap between achievable values; a visible default for a single point. */
|
||
function inferStep(points, max, min) {
|
||
let smallest = Infinity
|
||
for (let i = 1; i < points.length; i++) {
|
||
const gap = points[i].x - points[i - 1].x
|
||
if (gap > 0 && gap < smallest) smallest = gap
|
||
}
|
||
if (Number.isFinite(smallest)) return smallest
|
||
const width = Number.isFinite(max) && max > min ? max - min : 1
|
||
return width / 40
|
||
}
|