jared
d006dea6e4
The three kodor/fix issues, taken over after a day with no branch, PR or comment on any of them. Batched because each is single-file with a committed acceptance test, and two share documentation surfaces. #16 (F-025) -- cog_gov_search() utility mode interpolated `name` straight into regexp_matches() unescaped, while basket mode in the same file already routed it through .escape_regex() with the comment "so `name` is treated as a literal substring". Two failure modes, both HTTP 200 through the API: a government could not be found by its own complete name when that name contains a metacharacter (FREDONIA (BRISCOE) CITY returned nothing), and a bare "." matched all 608 Wisconsin cities. Malformed pattern text reached the engine as an error, which cog-api surfaced as a 500 -- reachable by typing a real name one character at a time ("Athens-Clarke County (bal"). Utility mode now calls the escaper that already existed. Roxygen updated: utility mode is documented as a literal case-insensitive substring match, and the basket-mode "substring fallback" step no longer describes itself as a regex either. BEHAVIOUR CHANGE worth flagging: anchored exact-match searches stop working, because there is no regex left to anchor. Two existing tests used "^BROWARD COUNTY$" and "^FLORIDA$" as their exact-match idiom; both now search for those characters literally. Updated to the bare names, which still resolve to exactly one row each once scoped by state/type (verified, not assumed). There is no exact-match option in utility mode any more -- noted on the issue, since that is a real if small capability loss. #15 (F-004) -- the raw Census files report thousands of dollars; this package multiplies by 1000 and returns full US dollars. Correct, and already stated in ?cog_spending / ?cog_revenue @return, in provenance, and in cog-api's data-dictionary. Absent from every surface a reader meets FIRST. Added to README.md as its own section and to both vignettes' openings. The dangerous one is cog_explorer/CLAUDE.md, which states the opposite rule ("All raw `amt` values are in $1,000s") without scoping it to the raw column -- a reader applying that to amt_nominal overstates by 1000x and gets a plausible-looking number rather than an obvious error. Fixed there too; that directory has no git remote, so it rides in no PR and is left uncommitted for the owner. #14 (F-021) -- .peer_summary_rows() computes stats::quantile() separately inside each (year, spend_subtype, category) cell, so a summary_p50 row is "the median peer's value in that one category", never "the value of the median peer's total" -- the median peer for Police and for Fire are usually different governments. Summing them across categories misstated a total-spending band by -32.7% to +251.0% across 24 years, with a sign flip at FY2012. The verb is right and its documented use (facet by role AND category) is unaffected, so the fix is @return prose plus a worked snippet showing the correct computation: sum each peer's own categories first, then take the quantile of those per-government totals. This is the R-side counterpart of cog-api#9, fixed on the API surface earlier today; the wording is deliberately consistent across the two. Note the phrase "not additive" has to stay on one roxygen source line -- the test greps the generated Rd, where a line wrap turns it into "not additive" and stops matching. Cost one red run to find. man/ regenerated with roxygen 8.0.0 against a repo built with 7.3.3, so cog_spending.Rd and DESCRIPTION were reverted -- their entire diff was version churn (reindentation, RoxygenNote -> Config/roxygen2/version) with no content change. The two Rd files kept carry only the edits above. Suite: 629 pass / 0 fail / 3 skip (was 606/0/6). The three remaining skips are #11, #12 and #13.
46 lines
2.4 KiB
R
46 lines
2.4 KiB
R
# Madison walkthrough audit -- finding F-004. Tracked as uscogdata#15.
|
|
# See docs/walkthroughs/FINDINGS.md in cog_explorer.
|
|
#
|
|
# The raw Census files report thousands of dollars; this package multiplies by
|
|
# 1000 and returns full US dollars. That is the friendlier choice and is not
|
|
# wrong -- but cog_explorer's CLAUDE.md states "All raw `amt` values are in
|
|
# $1,000s", so a reader who applies that rule to amt_nominal overstates every
|
|
# figure by 1000x, and gets a plausible-looking number rather than an obvious
|
|
# error. The audit rates this the highest-consequence definitional gap it found.
|
|
#
|
|
# Deliberately NOT asserted here: man/cog_spending.Rd and man/cog_revenue.Rd,
|
|
# which ALREADY carry the statement in their @return sections (verified
|
|
# 2026-07-29), as does cog-api's data-dictionary.md (since 2b71b41). The gap is
|
|
# in the surfaces a reader meets first and in cog_explorer's own conventions
|
|
# doc -- see uscogdata#15 for the full surface-by-surface table and for the two
|
|
# secondary tasks (cog_explorer/CLAUDE.md, which has no git remote, and
|
|
# cog-api's llms.txt, which is silent on units).
|
|
|
|
test_that("returned amounts are documented as full US dollars where readers meet the package", {
|
|
|
|
says_units <- function(path) {
|
|
txt <- paste(readLines(path, warn = FALSE), collapse = " ")
|
|
grepl("full US dollars|full U\\.S\\. dollars", txt, ignore.case = TRUE) &&
|
|
grepl("\\$1,000s|thousands of dollars", txt, ignore.case = TRUE)
|
|
}
|
|
|
|
expect_true(says_units(testthat::test_path("..", "..", "README.md")))
|
|
expect_true(says_units(testthat::test_path("..", "..", "vignettes", "total-spending.Rmd")))
|
|
expect_true(says_units(testthat::test_path("..", "..", "vignettes",
|
|
"population-denominators.Rmd")))
|
|
|
|
# Pin the documented claim to the actual behaviour, so the two cannot drift.
|
|
# The expected raw amount is read straight from the corpus's parquet
|
|
# partitions -- never through cog_spending(), which is the thing being
|
|
# described. Madison FY2020: E/F/G = 623,347 ($1,000s) -> $623,347,000.
|
|
raw_thousands <- wt_raw_amt("552025209777", 2020L, prefixes = c("E", "F", "G"))
|
|
expect_equal(raw_thousands, 623347)
|
|
|
|
returned <- cog_spending(govid = "552025209777", years = 2020L)
|
|
expect_equal(sum(returned$amt_nominal), raw_thousands * 1000)
|
|
|
|
units <- attr(returned, "provenance")$transformations$units_conversion
|
|
expect_true(units$applied)
|
|
expect_equal(units$multiplier, 1000)
|
|
})
|