jared
15f8a86409
Deploy to git-pages / deploy (push) Successful in 16s
The band on the time-series chart summed each student group's own
count_lower/count_upper. That is not the interval of the district total: the sum
of eight 97.5th percentiles is the case where every group lands at its extreme
in the same draw, which is far less likely than any one group doing so. The
chart was overstating uncertainty by roughly a factor of two.
Now summed within each draw and summarized across draws — build_state_summary()'s
method from crdc-arrests/R/summarize_draws.R on a different axis, the same
operation poolBySex already performs. Clark County NV, 95% interval of the total:
wave draws-based summed bounds change
2015-16 215-276 (w 61) 172-322 (w 150) -60%
2017-18 175-230 (w 55) 134-269 (w 135) -60%
2021-22 116-164 (w 48) 78-188 (w 110) -57%
Medians barely move (245->246, 200->202, 137->138), which is the expected
signature: the sum of medians already approximates the median of the sum. It is
the tails that were wrong. Values verified against an offline DuckDB computation
over the same shards and then read back off the rendered SVG geometry.
Cost is very uneven, so it is not unconditional. Three waves is 0.31MB for
Nevada but 13.9MB for California and 14.3MB for Texas. Directory listings
(~1KB/year, and already fetched for part discovery) carry each part's byte size,
so useDistrictTotalDraws probes the total up front: under 3MB it fetches
automatically, above it the chart offers a button naming the real size
("Compute the exact interval from the draws (13.9 MB)"). Measured: Nevada adds
no perceptible time at all — the totals resolve in the same frame as the density
panel, since both wait on the duckdb-wasm engine that dominates a cold load —
and California's opt-in completes in 1.3s.
totalPerDraw refuses to sum unless every group has a complete draw set, and
fetchModelCounts now reports how many groups it dropped: for a per-group density
a missing group is a gap, but for a total it is an undercount that would shift
the whole interval down while looking entirely plausible.
The caption states which interval is on screen in either case, and names the
assumption behind the draws-based one — draw_id is renumbered per write batch
upstream, so summing at a draw index convolves what are effectively independent
draws. That is defensible for a posterior *predictive* total, whose observation
noise is independent across groups by construction and dominates the
parameter-level covariance, but it should be read as a predictive total rather
than a correlation-preserving contrast.
71 lines
3.0 KiB
JavaScript
71 lines
3.0 KiB
JavaScript
/**
|
|
* The district-wide arrest total, taken from the posterior predictive draws.
|
|
*
|
|
* Why this exists: the time-series chart used to draw its band by summing each
|
|
* student group's own `count_lower`/`count_upper`. That is not the interval of
|
|
* the total. The sum of per-group 97.5th percentiles is the value you would see
|
|
* if *every* group simultaneously landed at its own extreme in the same draw,
|
|
* which is far less likely than any one group doing so — so the band came out
|
|
* systematically too wide. Summing within each draw and taking quantiles of the
|
|
* resulting totals answers the actual question: how many arrests does the model
|
|
* think this district had?
|
|
*
|
|
* This is `build_state_summary()`'s method from crdc-arrests/R/summarize_draws.R
|
|
* applied on a different axis — sum inside the draw, summarize across draws —
|
|
* the same operation `poolBySex` performs for sex pooling.
|
|
*
|
|
* One caveat to carry into any caption. Upstream, `draw_id` is renumbered per
|
|
* write batch, so draw *k* of one group is not the same posterior sample as
|
|
* draw *k* of another; measured cross-group correlation is ≈ 0.02. Summing at a
|
|
* draw index therefore convolves what are effectively independent draws. That
|
|
* is a reasonable model here — these are *posterior predictive* draws, whose
|
|
* observation noise is independent across groups by construction and dominates
|
|
* the parameter-level covariance — but it does mean the result should be read
|
|
* as a predictive total, not as a contrast that preserves parameter
|
|
* correlation. It is still much closer to the truth than summing bounds.
|
|
*/
|
|
|
|
import { quantile } from './kde.js'
|
|
import { isCompleteDrawSet } from './drawGroups.js'
|
|
|
|
/** Matches the API's default interval, so the two are directly comparable. */
|
|
export const TOTAL_INTERVAL_MASS = 0.95
|
|
|
|
/**
|
|
* District total at each draw index: the sum across every student group.
|
|
*
|
|
* Returns null unless *every* group has a complete draw set. A group silently
|
|
* missing from the sum would undercount the total at every draw and shift the
|
|
* whole interval down, which is indistinguishable on screen from a real result.
|
|
*
|
|
* @param {Record<string, number[]> | null | undefined} countsByGroup
|
|
* @param {number} nDraws
|
|
* @returns {number[] | null}
|
|
*/
|
|
export function totalPerDraw(countsByGroup, nDraws) {
|
|
const groups = Object.values(countsByGroup || {})
|
|
if (!groups.length || !(nDraws > 0)) return null
|
|
if (!groups.every((counts) => isCompleteDrawSet(counts, nDraws))) return null
|
|
|
|
const totals = new Array(nDraws).fill(0)
|
|
for (const counts of groups) {
|
|
for (let i = 0; i < nDraws; i++) totals[i] += counts[i]
|
|
}
|
|
return totals
|
|
}
|
|
|
|
/**
|
|
* @param {number[] | null | undefined} totals - district total per draw
|
|
* @returns {{lower: number, median: number, upper: number, nDraws: number} | null}
|
|
*/
|
|
export function totalInterval(totals) {
|
|
if (!totals?.length) return null
|
|
const tail = (1 - TOTAL_INTERVAL_MASS) / 2
|
|
return {
|
|
lower: quantile(totals, tail),
|
|
median: quantile(totals, 0.5),
|
|
upper: quantile(totals, 1 - tail),
|
|
nDraws: totals.length,
|
|
}
|
|
}
|