Filed from the Madison walkthrough audit (cog_explorer/docs/walkthroughs/FINDINGS.md, 2026-07-28). Verdict: definitional — the function does exactly what its per-cell computation implies; the documented shape just isn't documented.
Root cause
.peer_summary_rows() (R/peers.R) computes stats::quantile()separately inside each (year, spend_subtype, category) cell across the peer set. So a summary_p50 row is "the median peer's value in that one category", not "the value of the median peer's total."cog_peer_compare()'s roxygen says the summary rows exist "so the result can be faceted by role in a single ggplot call" and that they retain the same columns as everything else — including spend_subtype and category — but never states that the quantile is computed per cell, and never warns that the rows are not additive across category.
Findings resolved
Finding
Severity
Summary
F-021
high
cog_peer_compare()'s built-in summary_p25/summary_p50/summary_p75 rows are quantiles within each (year, spend_subtype, category) cell, not quantiles of each peer government's grand total — summing them naively misrepresents a "total spending" band by -33% to +251%
Reproduction (verbatim from FINDINGS.md, verified against the live corpus)
peers<-cog_find_peers(mad_id,year=2023L,max_peers=15)peer_cmp<-cog_peer_compare(target_govid=mad_id,peers=peers,category=NULL,years=2000:2023,per_capita=TRUE,adjust_to_year=2023L)table(peer_cmp$role)#> peer: 11580 summary_p25/p50/p75: 947 each target: 794# The naive move: sum the pre-built summary_p50 rows for a year.peer_cmp|>dplyr::filter(role=="summary_p50",year%in%c(2000,2012))|>dplyr::group_by(year)|>dplyr::summarise(builtin_sum=sum(amt_per_capita_real,na.rm=TRUE))#> 2000: $1,533 2012: $6,434# The correct move: sum each peer's OWN total first, then take quantiles.peer_totals<-peer_cmp|>dplyr::filter(role%in%c("target","peer"))|>dplyr::group_by(year,role,canonical_govid)|>dplyr::summarise(pc=sum(amt_per_capita_real,na.rm=TRUE),.groups="drop")peer_totals|>dplyr::filter(role=="peer",year%in%c(2000,2012))|>dplyr::group_by(year)|>dplyr::summarise(p50=stats::quantile(pc,0.5,na.rm=TRUE))#> 2000: $1,929 2012: $1,833
The mechanism reproduces on the bundled fixture corpus (Madison, 10 peers, category = NULL, FY2020, nominal per-capita): naive sum(summary_p50) = $6,180 against a correct per-government median of $2,043 — a +202% overstatement, the same direction and rough magnitude as the live corpus's post-FY2012 range.
Why it matters
Wrong in every one of the 24 years tested (2000-2023): -20.5% to -32.7% before FY2012, then a sign flip to +189% to +251% from FY2012 onward. That exceeds the 7.1% understatement in the direct-excludes-interest finding, which is itself rated high. And the trap is not exotic — it bites exactly the caller who wants a single "peer median total spending" line and infers from the column names alone that summing the summary rows gets it. The documented primary use case (faceting by roleandcategory) is internally consistent and unaffected; the failure is entirely in what a reader can reasonably infer from the return shape.
Definition of done
cog_peer_compare()'s @return documentation states explicitly that summary_p25/summary_p50/summary_p75 rows are quantiles computed within each (year, spend_subtype, category) cell — per-category quantiles — and are not additive across category into a total-spending quantile band.
The documentation shows the correct computation for a total-spending band in one or two lines: sum each peer government's own value across category/subtype first, then take quantiles of those per-government totals.
Consider (design, not required here) marking the rows in the data itself — e.g. a non-empty notes value on every summary_* row — so the warning travels with the object rather than living only in ?cog_peer_compare. The API surface needs exactly this; see the cross-reference.
Test goes green: tests/testthat/test-peer-summary-scope.R → test_that("cog_peer_compare() documents that summary_* rows are per-category quantiles", ...). It asserts the rendered help (man/cog_peer_compare.Rd) contains a statement matching /per-category|not additive|within each/i next to the summary_ rows, and pins the mechanism numerically against the fixture (naive sum $6,180 vs. correct median $2,043 for Madison FY2020) so a future refactor that quietly changes the quantile grouping fails here. Remove the skip() on line 1 of the test body to activate.
Cross-reference
cog-api carries the same rows with the same silence, and rates it a defect there rather than definitional: /governments/{govid}/peer-comparison requires category and accepts exactly one value, so a caller wanting a multi-category comparison has no option but to make N calls and combine per-category quantile rows — the API pushes every such caller straight into this trap. Tracked there as finding F-033.
Severity: high. Verdict: definitional.
Filed from the **Madison walkthrough audit** (`cog_explorer/docs/walkthroughs/FINDINGS.md`, 2026-07-28). Verdict: **definitional** — the function does exactly what its per-cell computation implies; the documented shape just isn't documented.
## Root cause
`.peer_summary_rows()` (`R/peers.R`) computes `stats::quantile()` **separately inside each `(year, spend_subtype, category)` cell** across the peer set. So a `summary_p50` row is *"the median peer's value in that one category"*, not *"the value of the median peer's total."* `cog_peer_compare()`'s roxygen says the summary rows exist "so the result can be faceted by `role` in a single ggplot call" and that they retain the same columns as everything else — including `spend_subtype` and `category` — but never states that the quantile is computed per cell, and never warns that the rows are not additive across `category`.
## Findings resolved
| Finding | Severity | Summary |
|---|---|---|
| **F-021** | high | `cog_peer_compare()`'s built-in `summary_p25`/`summary_p50`/`summary_p75` rows are quantiles within each (year, spend_subtype, category) cell, not quantiles of each peer government's grand total — summing them naively misrepresents a "total spending" band by -33% to +251% |
## Reproduction (verbatim from FINDINGS.md, verified against the live corpus)
```r
peers <- cog_find_peers(mad_id, year = 2023L, max_peers = 15)
peer_cmp <- cog_peer_compare(target_govid = mad_id, peers = peers,
category = NULL, years = 2000:2023,
per_capita = TRUE, adjust_to_year = 2023L)
table(peer_cmp$role)
#> peer: 11580 summary_p25/p50/p75: 947 each target: 794
# The naive move: sum the pre-built summary_p50 rows for a year.
peer_cmp |> dplyr::filter(role == "summary_p50", year %in% c(2000, 2012)) |>
dplyr::group_by(year) |>
dplyr::summarise(builtin_sum = sum(amt_per_capita_real, na.rm = TRUE))
#> 2000: $1,533 2012: $6,434
# The correct move: sum each peer's OWN total first, then take quantiles.
peer_totals <- peer_cmp |> dplyr::filter(role %in% c("target","peer")) |>
dplyr::group_by(year, role, canonical_govid) |>
dplyr::summarise(pc = sum(amt_per_capita_real, na.rm = TRUE), .groups = "drop")
peer_totals |> dplyr::filter(role == "peer", year %in% c(2000, 2012)) |>
dplyr::group_by(year) |>
dplyr::summarise(p50 = stats::quantile(pc, 0.5, na.rm = TRUE))
#> 2000: $1,929 2012: $1,833
```
The mechanism reproduces on the **bundled fixture corpus** (Madison, 10 peers, `category = NULL`, FY2020, nominal per-capita): naive `sum(summary_p50)` = **$6,180** against a correct per-government median of **$2,043** — a +202% overstatement, the same direction and rough magnitude as the live corpus's post-FY2012 range.
## Why it matters
Wrong in every one of the 24 years tested (2000-2023): **-20.5% to -32.7% before FY2012, then a sign flip to +189% to +251% from FY2012 onward**. That exceeds the 7.1% understatement in the `direct`-excludes-interest finding, which is itself rated high. And the trap is not exotic — it bites exactly the caller who wants a single "peer median total spending" line and infers from the column names alone that summing the summary rows gets it. The documented primary use case (faceting by `role` *and* `category`) is internally consistent and unaffected; the failure is entirely in what a reader can reasonably infer from the return shape.
## Definition of done
1. `cog_peer_compare()`'s `@return` documentation states explicitly that `summary_p25`/`summary_p50`/`summary_p75` rows are quantiles computed **within each `(year, spend_subtype, category)` cell** — per-category quantiles — and are **not additive across `category`** into a total-spending quantile band.
2. The documentation shows the correct computation for a total-spending band in one or two lines: sum each peer government's own value across category/subtype first, then take quantiles of those per-government totals.
3. Consider (design, not required here) marking the rows in the data itself — e.g. a non-empty `notes` value on every `summary_*` row — so the warning travels with the object rather than living only in `?cog_peer_compare`. The API surface needs exactly this; see the cross-reference.
4. Test goes green: **`tests/testthat/test-peer-summary-scope.R`** → `test_that("cog_peer_compare() documents that summary_* rows are per-category quantiles", ...)`. It asserts the rendered help (`man/cog_peer_compare.Rd`) contains a statement matching `/per-category|not additive|within each/i` next to the `summary_` rows, and pins the mechanism numerically against the fixture (naive sum $6,180 vs. correct median $2,043 for Madison FY2020) so a future refactor that quietly changes the quantile grouping fails here. Remove the `skip()` on line 1 of the test body to activate.
## Cross-reference
**`cog-api`** carries the same rows with the same silence, and rates it a **defect** there rather than definitional: `/governments/{govid}/peer-comparison` requires `category` and accepts exactly one value, so a caller wanting a multi-category comparison has no option but to make N calls and combine per-category quantile rows — the API pushes every such caller straight into this trap. Tracked there as finding F-033.
**Severity: high. Verdict: definitional.**
Taking this over. No kodor activity since assignment on 2026-07-29 — no branch, no PR, no comment — so I am picking it up rather than leaving it queued. Reassign back if kodor is simply slow and you would rather it land there; I will not push until this repo is otherwise quiet.
Being fixed together with the other two kodor/fix issues in one PR: they are all single-file, all have committed acceptance tests, and two of them touch the same documentation surfaces.
Taking this over. No kodor activity since assignment on 2026-07-29 — no branch, no PR, no comment — so I am picking it up rather than leaving it queued. Reassign back if kodor is simply slow and you would rather it land there; I will not push until this repo is otherwise quiet.
Being fixed together with the other two `kodor/fix` issues in one PR: they are all single-file, all have committed acceptance tests, and two of them touch the same documentation surfaces.
Fixed and merged in PR #22 (8db944e on main). Closing manually: that PR said "Closes #16, #15, #14", and Gitea only parsed the first reference, so #16 auto-closed while this one stayed open. The keyword needs repeating per issue.
Verified present on main, not just claimed.
Fixed and merged in PR #22 (`8db944e` on `main`). Closing manually: that PR said "Closes #16, #15, #14", and Gitea only parsed the first reference, so #16 auto-closed while this one stayed open. The keyword needs repeating per issue.
Verified present on `main`, not just claimed.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Filed from the Madison walkthrough audit (
cog_explorer/docs/walkthroughs/FINDINGS.md, 2026-07-28). Verdict: definitional — the function does exactly what its per-cell computation implies; the documented shape just isn't documented.Root cause
.peer_summary_rows()(R/peers.R) computesstats::quantile()separately inside each(year, spend_subtype, category)cell across the peer set. So asummary_p50row is "the median peer's value in that one category", not "the value of the median peer's total."cog_peer_compare()'s roxygen says the summary rows exist "so the result can be faceted byrolein a single ggplot call" and that they retain the same columns as everything else — includingspend_subtypeandcategory— but never states that the quantile is computed per cell, and never warns that the rows are not additive acrosscategory.Findings resolved
cog_peer_compare()'s built-insummary_p25/summary_p50/summary_p75rows are quantiles within each (year, spend_subtype, category) cell, not quantiles of each peer government's grand total — summing them naively misrepresents a "total spending" band by -33% to +251%Reproduction (verbatim from FINDINGS.md, verified against the live corpus)
The mechanism reproduces on the bundled fixture corpus (Madison, 10 peers,
category = NULL, FY2020, nominal per-capita): naivesum(summary_p50)= $6,180 against a correct per-government median of $2,043 — a +202% overstatement, the same direction and rough magnitude as the live corpus's post-FY2012 range.Why it matters
Wrong in every one of the 24 years tested (2000-2023): -20.5% to -32.7% before FY2012, then a sign flip to +189% to +251% from FY2012 onward. That exceeds the 7.1% understatement in the
direct-excludes-interest finding, which is itself rated high. And the trap is not exotic — it bites exactly the caller who wants a single "peer median total spending" line and infers from the column names alone that summing the summary rows gets it. The documented primary use case (faceting byroleandcategory) is internally consistent and unaffected; the failure is entirely in what a reader can reasonably infer from the return shape.Definition of done
cog_peer_compare()'s@returndocumentation states explicitly thatsummary_p25/summary_p50/summary_p75rows are quantiles computed within each(year, spend_subtype, category)cell — per-category quantiles — and are not additive acrosscategoryinto a total-spending quantile band.notesvalue on everysummary_*row — so the warning travels with the object rather than living only in?cog_peer_compare. The API surface needs exactly this; see the cross-reference.tests/testthat/test-peer-summary-scope.R→test_that("cog_peer_compare() documents that summary_* rows are per-category quantiles", ...). It asserts the rendered help (man/cog_peer_compare.Rd) contains a statement matching/per-category|not additive|within each/inext to thesummary_rows, and pins the mechanism numerically against the fixture (naive sum $6,180 vs. correct median $2,043 for Madison FY2020) so a future refactor that quietly changes the quantile grouping fails here. Remove theskip()on line 1 of the test body to activate.Cross-reference
cog-apicarries the same rows with the same silence, and rates it a defect there rather than definitional:/governments/{govid}/peer-comparisonrequirescategoryand accepts exactly one value, so a caller wanting a multi-category comparison has no option but to make N calls and combine per-category quantile rows — the API pushes every such caller straight into this trap. Tracked there as finding F-033.Severity: high. Verdict: definitional.
Taking this over. No kodor activity since assignment on 2026-07-29 — no branch, no PR, no comment — so I am picking it up rather than leaving it queued. Reassign back if kodor is simply slow and you would rather it land there; I will not push until this repo is otherwise quiet.
Being fixed together with the other two
kodor/fixissues in one PR: they are all single-file, all have committed acceptance tests, and two of them touch the same documentation surfaces.Fixed and merged in PR #22 (
8db944eonmain). Closing manually: that PR said "Closes #16, #15, #14", and Gitea only parsed the first reference, so #16 auto-closed while this one stayed open. The keyword needs repeating per issue.Verified present on
main, not just claimed.