Compare commits
30
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2e8383b098
|
||
|
|
d006dea6e4
|
||
|
|
ebac39e6de | ||
|
|
47dc08c4b0 | ||
|
|
1d553a788f
|
||
|
|
c375c55da7
|
||
|
|
82acda6f93 | ||
|
|
9233c3d18e
|
||
|
|
1f257812b6 | ||
|
|
d258cef8c5 | ||
|
|
e7d3a7a310 | ||
|
|
a4eb80d823 | ||
|
|
aba7ffbac2 | ||
|
|
c1c6b5a6ba | ||
|
|
c7260cb20c | ||
|
|
54ece11867 | ||
|
|
3bd9b1f011 | ||
|
|
24b2ff7d8c | ||
|
|
c28712f62f | ||
|
|
7913b0f664 | ||
|
|
887acf7e81 | ||
|
|
81fd1a5279 | ||
|
|
e2088458e1 | ||
|
|
fefd4fe969 | ||
|
|
7ed1da9b79 | ||
|
|
9240a18ea3 | ||
|
|
46fed3a241 | ||
|
|
c9d1a05d4f | ||
|
|
fcecd62a03 | ||
|
|
e581e7360c |
@@ -16,3 +16,4 @@
|
||||
^Meta$
|
||||
^\.gitea$
|
||||
^CLAUDE\.md$
|
||||
^\.superpowers$
|
||||
|
||||
@@ -9,3 +9,6 @@ docs/
|
||||
/Meta/
|
||||
.DS_Store
|
||||
/.quarto/
|
||||
|
||||
# SDD working artifacts (ledger, briefs, review packages) — plans/ stays tracked
|
||||
.superpowers/sdd/
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,5 +1,51 @@
|
||||
# uscogdata 0.1.0 (development)
|
||||
|
||||
## Corpus-wide series breaks now reach users (`corpus_break_refs`)
|
||||
|
||||
* Four catalogued series breaks carry `fin_code = "ALL"` — caveats about the
|
||||
corpus as a whole rather than about one item code. `series_break_refs` is
|
||||
built by matching `fin_code` against the item codes in the result, and no
|
||||
row's `item_code` is ever the literal `"ALL"`, so **none of them could ever
|
||||
be surfaced**: `SB085` (dollar precision across the 1976/1977 boundary),
|
||||
`SB087` (imputation exclusion from FY2002), `SB194` (the dense → sparse
|
||||
representation change at FY2012) and `SB086` (the government id scheme
|
||||
change at FY2017).
|
||||
* Provenance gains `corpus_break_refs`, selected on the break-year window
|
||||
alone and disjoint from `series_break_refs` by construction, so a consumer
|
||||
can tell a whole-result caveat from a break in one series. `cog_explain()`
|
||||
prints them under their own "Corpus-wide caveats" heading. cog-api passes
|
||||
provenance through verbatim, so the field appears there without an API
|
||||
change.
|
||||
* `SB194` is the one that made this urgent: a query spanning FY2011 → FY2012
|
||||
crosses the boundary where an absent cell stops meaning "Census published
|
||||
`$0`" and starts meaning "not reported", and until now nothing said so.
|
||||
|
||||
## Bundled fixture regenerated against the sparsified corpus
|
||||
|
||||
* `inst/extdata/fixture_corpus/` now tracks the corpus published on
|
||||
2026-07-29 (`pipeline_commit 83f9715`, schema v6). The wide era no longer
|
||||
stores explicit zeros: FY2011 fell from 2,864,212 rows to 496,004, of
|
||||
which none are `$0`. **Absence now means two different things** — in a
|
||||
`dense_source` year (≤ FY2011) an absent cell means Census published `$0`;
|
||||
in a `sparse_source` year (≥ FY2012) it means not reported. The corpus
|
||||
carries that rule in two new tables the fixture now ships,
|
||||
`representation.parquet` and `code_set.parquet`, alongside
|
||||
`census_collection_coverage.parquet` and `lineage_events.parquet`
|
||||
(all ten publish-tree metadata tables, up from six). Catalogued upstream
|
||||
as series break `SB194`.
|
||||
* `cog_categories()` gains an `assistance` spending subtype: the J-prefix
|
||||
aid/benefit codes (`J19`, `J67`, `J68`, `J85`) are categorised now that
|
||||
the upstream crosswalk covers every flow code carrying dollars.
|
||||
* Two consequences worth knowing about, both visible in provenance rather
|
||||
than in returned dollars. The harmonization block's `na_rows_excluded`
|
||||
counts only rows that exist, so wide-era codes that were zero-padded no
|
||||
longer appear there. Coverage-gap `suggestions` are presence-based for the
|
||||
same reason, so a recipe whose component codes were all `$0` for a given
|
||||
government-year is no longer suggested for it.
|
||||
* `tests/testthat/test-fixture-vintage.R` pins these structural facts, so a
|
||||
fixture left behind by a future publish fails loudly instead of letting the
|
||||
suite pass against a corpus that no longer exists.
|
||||
|
||||
## Breaking: corpus schema_version 4 (Phase P canonical ids)
|
||||
|
||||
* The package now requires corpus `schema_version = 4` (`MinCorpusSchema` /
|
||||
|
||||
+23
@@ -60,6 +60,21 @@ cog_explain <- function(result, format = c("print", "list")) {
|
||||
cli::cli_text("Basis: {prov$basis}{note}")
|
||||
}
|
||||
|
||||
if (!is.null(prov$expenditure_concept)) {
|
||||
concept_note <- if (!is.null(prov$expenditure_concept_note) &&
|
||||
!is.na(prov$expenditure_concept_note)) {
|
||||
sprintf(" (%s)", prov$expenditure_concept_note)
|
||||
} else {
|
||||
""
|
||||
}
|
||||
cli::cli_text("Concept: {prov$expenditure_concept}{concept_note}")
|
||||
if (isTRUE(prov$expenditure_concept_direct_suppressed)) {
|
||||
cli::cli_alert_warning(
|
||||
"Direct leg unavailable for at least one requested (year, category) -- affected rows report intergovernmental dollars alone, not Direct + IG. See each row's notes."
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
cli::cli_h2("Codes observed")
|
||||
codes <- prov$codes_summed$observed
|
||||
if (length(codes) == 0L) {
|
||||
@@ -108,6 +123,14 @@ cog_explain <- function(result, format = c("print", "list")) {
|
||||
cli::cli_ul(.series_break_story_lines(prov$series_break_refs))
|
||||
}
|
||||
|
||||
# Kept in a section of its own: these qualify the whole result, so folding
|
||||
# them in with the per-code breaks above would invite reading them as a
|
||||
# caveat about one series.
|
||||
if (length(prov$corpus_break_refs) > 0L) {
|
||||
cli::cli_h2("Corpus-wide caveats")
|
||||
cli::cli_ul(.series_break_story_lines(prov$corpus_break_refs))
|
||||
}
|
||||
|
||||
cli::cli_h2("Transformations")
|
||||
uc <- prov$transformations$units_conversion
|
||||
if (isTRUE(uc$applied)) {
|
||||
|
||||
@@ -133,7 +133,9 @@ cog_find_peers <- function(target_govid,
|
||||
#' [cog_find_peers()] result or a character vector of `canonical_govid`) and
|
||||
#' appends peer-distribution summary rows (`summary_p25`, `summary_p50`,
|
||||
#' `summary_p75`) so the result can be faceted by `role` in a single ggplot
|
||||
#' call.
|
||||
#' call. Those summary rows are quantiles **within each category**, not
|
||||
#' quantiles of each peer's total — see the `@return` section before summing
|
||||
#' them.
|
||||
#'
|
||||
#' @param target_govid Character scalar.
|
||||
#' @param peers A tibble from [cog_find_peers()] or a character vector of
|
||||
@@ -143,6 +145,10 @@ cog_find_peers <- function(target_govid,
|
||||
#' @param per_capita Default `TRUE` — peer compare usually normalizes by
|
||||
#' population.
|
||||
#' @param adjust_to_year Integer base year for CPI-U conversion or `NULL`.
|
||||
#' @param expenditure_concept `"direct"` (default) or `"total"`. Currently only
|
||||
#' `"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
|
||||
#' single-government queries but cannot be used here because combining Total
|
||||
#' across peer sets counts intergovernmental transfers twice.
|
||||
#' @return Tibble matching [cog_spending()]'s columns, plus a `role`
|
||||
#' column taking values `"target"`, `"peer"`, `"summary_p25"`,
|
||||
#' `"summary_p50"`, or `"summary_p75"`, `target_rank` (target's rank
|
||||
@@ -151,10 +157,41 @@ cog_find_peers <- function(target_govid,
|
||||
#' `attr(peers, "cohort_year")`; `NA` when `peers` was a bare character
|
||||
#' vector). Provenance reports `verb = "cog_peer_compare"`, `peer_count`,
|
||||
#' `cohort_year`, and `cohort_govids`.
|
||||
#'
|
||||
#' **The `summary_*` rows are per-category quantiles: they are not additive.**
|
||||
#' Each one is computed **within each `(year, spend_subtype,
|
||||
#' category)` cell** across the peer set, so a `summary_p50` row is *the
|
||||
#' median peer's value in that one category*, not *the value of the median
|
||||
#' peer's total*. The median peer for Police and the median peer for Fire
|
||||
#' are usually different governments, so summing `summary_*` rows across
|
||||
#' categories does not give any peer's total and misstates the band it
|
||||
#' appears to describe — measured at −32.7% to +251.0% across 24 years on
|
||||
#' one cohort, with a sign flip at FY2012.
|
||||
#'
|
||||
#' Facet by `role` **and** `category` (the documented use, and what the
|
||||
#' rows are built for). For a genuine "median peer's total spending" line,
|
||||
#' sum each peer's own categories first and take the quantile of those
|
||||
#' per-government totals:
|
||||
#'
|
||||
#' ```r
|
||||
#' library(dplyr)
|
||||
#' cmp |>
|
||||
#' filter(role %in% c("target", "peer")) |>
|
||||
#' group_by(year, role, canonical_govid) |>
|
||||
#' summarise(total = sum(amt_per_capita_real, na.rm = TRUE), .groups = "drop") |>
|
||||
#' filter(role == "peer") |>
|
||||
#' group_by(year) |>
|
||||
#' summarise(p50 = quantile(total, 0.5, na.rm = TRUE))
|
||||
#' ```
|
||||
#' @export
|
||||
cog_peer_compare <- function(target_govid, peers, category, years,
|
||||
per_capita = TRUE, adjust_to_year = NULL) {
|
||||
per_capita = TRUE, adjust_to_year = NULL,
|
||||
expenditure_concept = c("direct", "total")) {
|
||||
call <- match.call()
|
||||
expenditure_concept <- match.arg(expenditure_concept)
|
||||
if (identical(expenditure_concept, "total")) {
|
||||
.abort_concept_not_aggregatable("cog_peer_compare")
|
||||
}
|
||||
if (!is.character(target_govid) || length(target_govid) != 1L) {
|
||||
cli::cli_abort("`target_govid` must be a length-1 character string.")
|
||||
}
|
||||
|
||||
+17
-1
@@ -6,6 +6,9 @@
|
||||
per_capita, adjust_to_year, result, sql,
|
||||
subtype_col, basis = NA_character_,
|
||||
basis_note = NA_character_,
|
||||
expenditure_concept = "direct",
|
||||
expenditure_concept_note = NA_character_,
|
||||
expenditure_concept_direct_suppressed = FALSE,
|
||||
harmonization = NULL, recipe = NULL,
|
||||
suggestions = list()) {
|
||||
manifest <- .uscogdata_env$manifest
|
||||
@@ -34,11 +37,20 @@
|
||||
|
||||
schema_version <- suppressWarnings(as.integer(manifest$schema_version %||% 0L))
|
||||
con <- .uscogdata_env$con
|
||||
break_refs <- if (!is.null(con) && DBI::dbIsValid(con)) {
|
||||
have_con <- !is.null(con) && DBI::dbIsValid(con)
|
||||
break_refs <- if (have_con) {
|
||||
.build_series_break_refs(con, codes_observed, years, schema_version)
|
||||
} else {
|
||||
character(0)
|
||||
}
|
||||
# Corpus-wide caveats travel separately: they qualify the whole result
|
||||
# rather than one series, and they do not depend on codes_observed (see
|
||||
# .build_corpus_break_refs()).
|
||||
corpus_refs <- if (have_con) {
|
||||
.build_corpus_break_refs(con, years, schema_version)
|
||||
} else {
|
||||
character(0)
|
||||
}
|
||||
|
||||
list(
|
||||
verb = verb,
|
||||
@@ -51,6 +63,9 @@
|
||||
category = category,
|
||||
basis = basis,
|
||||
basis_note = basis_note,
|
||||
expenditure_concept = expenditure_concept,
|
||||
expenditure_concept_note = expenditure_concept_note,
|
||||
expenditure_concept_direct_suppressed = isTRUE(expenditure_concept_direct_suppressed),
|
||||
harmonization = harmonization %||% list(
|
||||
applied = FALSE, na_rows_excluded = 0L, na_amount_excluded = 0,
|
||||
note = NA_character_
|
||||
@@ -110,6 +125,7 @@
|
||||
)
|
||||
),
|
||||
series_break_refs = break_refs,
|
||||
corpus_break_refs = corpus_refs,
|
||||
manifest = list(
|
||||
schema_version = as.integer(manifest$schema_version),
|
||||
pipeline_commit = manifest$pipeline_commit %||% NA_character_,
|
||||
|
||||
+12
-1
@@ -25,6 +25,12 @@
|
||||
#' population from `gov_population_yearly`. Govs with missing population
|
||||
#' are excluded from the result.
|
||||
#' @param adjust_to_year Integer base year for CPI-U conversion, or `NULL`.
|
||||
#' @param expenditure_concept `"direct"` (default) or `"total"`. Currently only
|
||||
#' `"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
|
||||
#' single-government queries but cannot be used here because combining Total
|
||||
#' across multiple layers of government double-counts intergovernmental
|
||||
#' transfers (a state's payment to a school district is the same dollar the
|
||||
#' district reports as its own Direct spending).
|
||||
#' @return Tibble with columns `year`, `layer`, `canonical_govid`, `gov_name`,
|
||||
#' `spend_subtype`, `category`, `amt_nominal`, optional `amt_real` /
|
||||
#' `amt_per_capita_nominal` / `amt_per_capita_real`, optional `pop_source`,
|
||||
@@ -33,8 +39,13 @@
|
||||
#' and `rollup$included_govids` / `rollup$excluded_govids`.
|
||||
#' @export
|
||||
cog_geographic_rollup <- function(govids, category, years,
|
||||
per_capita = FALSE, adjust_to_year = NULL) {
|
||||
per_capita = FALSE, adjust_to_year = NULL,
|
||||
expenditure_concept = c("direct", "total")) {
|
||||
call <- match.call()
|
||||
expenditure_concept <- match.arg(expenditure_concept)
|
||||
if (identical(expenditure_concept, "total")) {
|
||||
.abort_concept_not_aggregatable("cog_geographic_rollup")
|
||||
}
|
||||
.validate_rollup_layers(govids)
|
||||
|
||||
govids <- lapply(govids, .coerce_govid_input, arg = "govids[[layer]]")
|
||||
|
||||
+19
-7
@@ -6,8 +6,11 @@
|
||||
#' the cross-vintage canonical-government registry. Operates in two modes:
|
||||
#'
|
||||
#' * **Utility mode** (single `name`, the original behavior): returns all
|
||||
#' rows whose `gov_name` matches the regex case-insensitively, sorted by
|
||||
#' `population_acs` descending. Useful for exploratory lookups.
|
||||
#' rows whose `gov_name` contains `name` as a **literal, case-insensitive
|
||||
#' substring**, sorted by `population_acs` descending. Useful for
|
||||
#' exploratory lookups. Regex metacharacters in `name` are escaped, so a
|
||||
#' government is findable by its own complete name even when that name
|
||||
#' contains parentheses or a period.
|
||||
#' * **Basket mode** (`length(name) > 1`): resolves each input row to a
|
||||
#' single canonical govid and returns a tibble in input order, suitable
|
||||
#' for piping straight into [cog_spending()] / [cog_revenue()] /
|
||||
@@ -19,7 +22,8 @@
|
||||
#' 1. Filter `canonical_fips_xwalk` by `state` and (if non-NA) `type`.
|
||||
#' 2. **Exact pass:** case-insensitive equality against `gov_name`.
|
||||
#' Single hit -> resolved. Multiple -> step 4.
|
||||
#' 3. **Substring fallback:** case-insensitive regex against `gov_name`.
|
||||
#' 3. **Substring fallback:** case-insensitive literal substring against
|
||||
#' `gov_name` (metacharacters escaped).
|
||||
#' Single hit -> resolved (`match_method = "substring"`). Zero hits ->
|
||||
#' `status = "no_match"`. Multiple hits -> step 4.
|
||||
#' 4. **Disambiguation:** if matches share one `govs_type`, pick the
|
||||
@@ -48,7 +52,7 @@
|
||||
#' [cog_spending()], [cog_revenue()].
|
||||
#' @examples
|
||||
#' \dontrun{
|
||||
#' # Utility mode — exploratory regex lookup
|
||||
#' # Utility mode — exploratory substring lookup
|
||||
#' cog_gov_search("broward", state = "FL")
|
||||
#'
|
||||
#' # Basket mode — resolve a known cohort
|
||||
@@ -98,9 +102,16 @@ cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
|
||||
if (!is.character(name) || length(name) != 1L) {
|
||||
cli::cli_abort("`name` must be a length-1 character string.")
|
||||
}
|
||||
# Escaped, so `name` is a literal case-insensitive substring -- the same
|
||||
# treatment basket mode has always given it. Interpolating it raw made a
|
||||
# government unfindable by its own name whenever that name contains a
|
||||
# metacharacter (FREDONIA (BRISCOE) CITY), turned a bare "." into a
|
||||
# match-everything wildcard, and let malformed pattern text reach the
|
||||
# engine as an error -- which cog-api surfaced as a 500, reachable by
|
||||
# typing a real name one character at a time (uscogdata#16, F-025).
|
||||
preds <- c(preds,
|
||||
sprintf("regexp_matches(gov_name, %s, 'i')",
|
||||
.sql_lit_chr(name)))
|
||||
.sql_lit_chr(.escape_regex(name))))
|
||||
}
|
||||
if (!is.null(state)) {
|
||||
st_fips <- .coerce_state_to_fips(state)
|
||||
@@ -136,8 +147,9 @@ cog_gov_search <- function(name = NULL, state = NULL, type = NULL) {
|
||||
#' @noRd
|
||||
.escape_regex <- function(x) {
|
||||
# Backslash-escape POSIX regex metacharacters so `name` is treated as a
|
||||
# literal substring in the DuckDB regexp_matches call (substring fallback
|
||||
# only; utility-mode intentionally preserves regex behavior).
|
||||
# literal substring in the DuckDB regexp_matches call. Used by BOTH modes:
|
||||
# utility mode used to interpolate raw, which was a defect rather than a
|
||||
# feature -- see the call site and uscogdata#16.
|
||||
gsub("([\\^$.|?*+(){}\\[\\]])", "\\\\\\1", x, perl = TRUE)
|
||||
}
|
||||
|
||||
|
||||
+33
-1
@@ -14,9 +14,41 @@
|
||||
sql <- sprintf(
|
||||
"SELECT DISTINCT break_id
|
||||
FROM series_breaks_pq
|
||||
WHERE fin_code IN (%s) AND break_year BETWEEN %d AND %d
|
||||
WHERE fin_code IN (%s) AND fin_code <> 'ALL'
|
||||
AND break_year BETWEEN %d AND %d
|
||||
ORDER BY break_id",
|
||||
.sql_lit_chr(codes_observed), min(as.integer(years)), max(as.integer(years))
|
||||
)
|
||||
DBI::dbGetQuery(con, sql)$break_id
|
||||
}
|
||||
|
||||
#' Corpus-wide caveats: catalogued breaks whose `fin_code` is the literal
|
||||
#' `"ALL"` rather than an item code. They qualify the whole result, so they
|
||||
#' cannot be matched the way `.build_series_break_refs()` matches -- no row's
|
||||
#' `item_code` is ever `"ALL"`, which is exactly why they reached no user
|
||||
#' before uscogdata#19. Selection is on the break_year window alone: which
|
||||
#' codes a result happens to contain is irrelevant to a caveat about the
|
||||
#' corpus.
|
||||
#'
|
||||
#' All four catalogued entries are *boundary* caveats (dollar precision
|
||||
#' across 1976/1977, imputation exclusion from 2002, the dense -> sparse
|
||||
#' representation change at 2012, the id scheme change at 2017), so the same
|
||||
#' `break_year BETWEEN min(years) AND max(years)` rule the code-specific
|
||||
#' path uses is the right one -- a request that never crosses the boundary
|
||||
#' is not affected by it.
|
||||
#'
|
||||
#' Returned separately from `series_break_refs` so a consumer can tell a
|
||||
#' whole-result caveat from a break in one series; the two are disjoint by
|
||||
#' construction.
|
||||
#' @noRd
|
||||
.build_corpus_break_refs <- function(con, years, schema_version) {
|
||||
if (schema_version < 5L || length(years) == 0L) return(character(0))
|
||||
sql <- sprintf(
|
||||
"SELECT DISTINCT break_id
|
||||
FROM series_breaks_pq
|
||||
WHERE fin_code = 'ALL' AND break_year BETWEEN %d AND %d
|
||||
ORDER BY break_id",
|
||||
min(as.integer(years)), max(as.integer(years))
|
||||
)
|
||||
DBI::dbGetQuery(con, sql)$break_id
|
||||
}
|
||||
|
||||
+346
-13
@@ -42,6 +42,34 @@
|
||||
#' `basis = "recipe"` with an inert `harmonization` block (`applied =
|
||||
#' FALSE`, pointing at the `recipe` block instead) rather than a
|
||||
#' possibly-misleading `"harmonized"`/`"raw"` value.
|
||||
#' @param expenditure_concept `"direct"` (default) returns only the
|
||||
#' government's own direct spending (item codes `E`/`F`/`G`), unchanged
|
||||
#' from prior releases. `"total"` additionally UNIONs in the
|
||||
#' intergovernmental leg -- payments to local governments (`M` codes) and
|
||||
#' to the state government (`L` codes, excluding the `L--` family-total
|
||||
#' rollup) -- so results gain rows with `spend_subtype ==
|
||||
#' "intergovernmental"`. Requires the active corpus's `summary_categories`
|
||||
#' to carry M/L rows (added by cog_pipeline PR #59); aborts with class
|
||||
#' `uscogdata_ig_categories_unsupported` on an older corpus rather than
|
||||
#' silently under-reporting. Mutually exclusive with `recipe` (a recipe
|
||||
#' already defines its own component codes). **Do not sum `"total"`
|
||||
#' results across levels of government** (e.g. state + county + city):
|
||||
#' a state's `M12` payment to a school district is the same dollar the
|
||||
#' district reports as its own direct `E12`, so summing both double-counts
|
||||
#' it. This matters in particular with [cog_geographic_rollup()], which
|
||||
#' sums across exactly that kind of multi-layer government set.
|
||||
#'
|
||||
#' In the legacy wide era (<= FY2011), some functions are published ONLY
|
||||
#' as an aggregate-flagged family total (e.g. Corrections' `E04`/`E05`
|
||||
#' split), which the Direct leg excludes by construction but the IG leg
|
||||
#' deliberately keeps (see `inst/sql/24-ig_long.sql`). For a `"total"`
|
||||
#' query, any (year, category) where this leaves intergovernmental rows
|
||||
#' with NO Direct counterpart is flagged: the affected rows' `notes`
|
||||
#' name the harmonization recipe that recovers the missing Direct
|
||||
#' component (when one exists), and
|
||||
#' `provenance$expenditure_concept_direct_suppressed` is `TRUE` -- the
|
||||
#' figure in those rows is the intergovernmental leg alone, not Direct +
|
||||
#' IG.
|
||||
#' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
|
||||
#' `spend_subtype`, `category`, `amt_nominal`, optional `amt_real`,
|
||||
#' optional `amt_per_capita_nominal`, optional `amt_per_capita_real`,
|
||||
@@ -50,12 +78,13 @@
|
||||
#' @export
|
||||
cog_spending <- function(govid, years, category = NULL,
|
||||
per_capita = FALSE, adjust_to_year = NULL,
|
||||
basis = c("harmonized", "raw"), recipe = NULL) {
|
||||
basis = c("harmonized", "raw"), recipe = NULL,
|
||||
expenditure_concept = c("direct", "total")) {
|
||||
.verb_spendrev(
|
||||
verb = "cog_spending",
|
||||
view_base = "spending_annotated",
|
||||
subtype_col = "spend_subtype",
|
||||
flow_prefixes = c("E", "F", "G", "K"),
|
||||
flow_prefixes = c("E", "F", "G"),
|
||||
call = match.call(),
|
||||
govid = govid,
|
||||
years = years,
|
||||
@@ -63,22 +92,79 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
per_capita = per_capita,
|
||||
adjust_to_year = adjust_to_year,
|
||||
basis = basis,
|
||||
recipe = recipe
|
||||
recipe = recipe,
|
||||
expenditure_concept = expenditure_concept
|
||||
)
|
||||
}
|
||||
|
||||
#' @noRd
|
||||
.abort_concept_not_aggregatable <- function(verb) {
|
||||
cli::cli_abort(c(
|
||||
"{.code expenditure_concept = \"total\"} cannot be used in {.fn {verb}}.",
|
||||
"*" = "Use {.code expenditure_concept = \"direct\"} (the default) for any \\
|
||||
comparison or sum that spans more than one government.",
|
||||
"i" = "Why: Census \"Total\" is a government's own Direct spending PLUS the \\
|
||||
money it hands to other governments. The receiving government reports \\
|
||||
that same dollar again as its own Direct when it actually spends it, \\
|
||||
so combining Total across governments double-counts intergovernmental \\
|
||||
transfers.",
|
||||
"i" = "For one government's own Total, use \\
|
||||
{.code cog_spending(expenditure_concept = \"total\")}."
|
||||
), class = "uscogdata_concept_not_aggregatable")
|
||||
}
|
||||
|
||||
#' @noRd
|
||||
.verb_spendrev <- function(verb, view_base, subtype_col, flow_prefixes, call,
|
||||
govid, years, category,
|
||||
per_capita, adjust_to_year,
|
||||
basis = c("harmonized", "raw"), recipe = NULL) {
|
||||
basis = c("harmonized", "raw"), recipe = NULL,
|
||||
expenditure_concept = c("direct", "total")) {
|
||||
basis_explicit <- length(basis) == 1L
|
||||
basis <- match.arg(basis, c("harmonized", "raw"))
|
||||
# match.arg() itself throws a base `simpleError`, not an rlang-classed
|
||||
# condition; wrap it so an invalid expenditure_concept aborts consistently
|
||||
# with the rest of this package's validation (cli::cli_abort -> rlang_error).
|
||||
expenditure_concept <- tryCatch(
|
||||
match.arg(expenditure_concept, c("direct", "total")),
|
||||
error = function(e) {
|
||||
cli::cli_abort(
|
||||
"`expenditure_concept` must be one of {.val direct} or {.val total}.",
|
||||
class = "uscogdata_invalid_expenditure_concept",
|
||||
parent = e
|
||||
)
|
||||
}
|
||||
)
|
||||
|
||||
govid <- .coerce_govid_input(govid, arg = "govid")
|
||||
.validate_verb_inputs(govid, years, category, per_capita, adjust_to_year,
|
||||
recipe)
|
||||
|
||||
if (!is.null(recipe) && identical(expenditure_concept, "total")) {
|
||||
cli::cli_abort(c(
|
||||
"`recipe` and `expenditure_concept = \"total\"` are mutually exclusive.",
|
||||
i = "A recipe defines its own component codes; pass one or the other.",
|
||||
i = "For a recipe's intergovernmental counterpart, use the matching IG recipe (e.g. `corrections_ig_local_combined`)."
|
||||
), class = "uscogdata_recipe_concept_conflict")
|
||||
}
|
||||
|
||||
# .verb_spendrev() is shared with cog_revenue(), which never exposes
|
||||
# expenditure_concept and always resolves it to "direct" -- so nothing on
|
||||
# the public API can reach this today. But it's a cheap guard against a
|
||||
# future call (direct or via a modified cog_revenue()) that would UNION
|
||||
# the IG leg's expenditure M/L rows into a revenue result, which has no
|
||||
# matching IG view and no sensible meaning.
|
||||
if (identical(expenditure_concept, "total") &&
|
||||
!identical(view_base, "spending_annotated")) {
|
||||
cli::cli_abort(
|
||||
paste0(
|
||||
"`expenditure_concept = \"total\"` is only supported for spending ",
|
||||
"(view_base = \"spending_annotated\"); got view_base = ",
|
||||
"{.val {view_base}}."
|
||||
),
|
||||
class = "uscogdata_expenditure_concept_unsupported"
|
||||
)
|
||||
}
|
||||
|
||||
years <- as.integer(years)
|
||||
if (!is.null(adjust_to_year)) adjust_to_year <- as.integer(adjust_to_year)
|
||||
|
||||
@@ -105,7 +191,13 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
category_for_prov <- recipe_label
|
||||
} else {
|
||||
view <- .select_view(view_base, resolved$basis)
|
||||
sql <- .build_verb_sql(view, subtype_col, govid, years, category)
|
||||
ig_view <- if (identical(expenditure_concept, "total")) {
|
||||
.require_ig_categories(con)
|
||||
.select_ig_view(resolved$basis)
|
||||
} else {
|
||||
NULL
|
||||
}
|
||||
sql <- .build_verb_sql(view, subtype_col, govid, years, category, ig_view)
|
||||
result <- tibble::as_tibble(DBI::dbGetQuery(con, sql))
|
||||
}
|
||||
|
||||
@@ -114,8 +206,6 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
result <- .attach_real_dollars(result, adjust_to_year, per_capita)
|
||||
}
|
||||
|
||||
result$notes <- .notes_column(result)
|
||||
|
||||
# A recipe result doesn't go through spending_annotated(_harmonized) /
|
||||
# revenue_annotated(_harmonized) at all -- .run_recipe()'s generic join
|
||||
# reads `long` directly -- so `basis` and the `harmonization` exclusion
|
||||
@@ -141,7 +231,64 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
harmonization <- .build_harmonization_block(
|
||||
con, govid, years, resolved, flow_prefixes
|
||||
)
|
||||
suggestions <- .build_suggestions(con, govid, years, category, result, resolved$basis)
|
||||
# C1(a): gap detection must run against the Direct leg alone. `result`
|
||||
# can also carry UNION'd intergovernmental rows (expenditure_concept =
|
||||
# "total"), and the wide era (<= FY2011) routinely has legacy IG dollars
|
||||
# surviving (ig_long deliberately keeps aggregate rows) for a
|
||||
# (year, category) whose legacy Direct dollars were suppressed (spending_
|
||||
# long/spending_long_harmonized both filter NOT is_aggregate). Passing
|
||||
# the UNION'd result here would let a surviving IG row count as coverage
|
||||
# and silently cancel the recipe-hint suggestion that should fire.
|
||||
direct_leg_result <- if (identical(expenditure_concept, "total")) {
|
||||
result[!(result[[subtype_col]] %in% "intergovernmental"), , drop = FALSE]
|
||||
} else {
|
||||
result
|
||||
}
|
||||
suggestions <- .build_suggestions(con, govid, years, category,
|
||||
direct_leg_result,
|
||||
resolved$basis, flow_prefixes)
|
||||
}
|
||||
|
||||
# C1(b): when expenditure_concept = "total", flag any row where the IG
|
||||
# leg has dollars but the Direct leg has none for that same (year,
|
||||
# canonical_govid, category) AND a harmonization recipe actually recovers
|
||||
# the missing Direct dollars for that exact triple -- see
|
||||
# .detect_direct_suppressed() for why bare Direct-row absence alone is NOT
|
||||
# sufficient (the dominant real cause is a government that simply has no
|
||||
# direct spending in that category, which is correct, ordinary data). When
|
||||
# a covering recipe is found, both the row-level notes and the provenance
|
||||
# say so rather than pass silently as a plausible Total.
|
||||
direct_suppressed_info <- if (identical(expenditure_concept, "total")) {
|
||||
.detect_direct_suppressed(con, result, subtype_col)
|
||||
} else {
|
||||
list(flag = rep(FALSE, nrow(result)), notes = rep(NA_character_, nrow(result)))
|
||||
}
|
||||
direct_suppressed <- direct_suppressed_info$flag
|
||||
direct_suppressed_flag <- isTRUE(any(direct_suppressed))
|
||||
|
||||
result$notes <- .notes_column(result, direct_suppressed_info$notes)
|
||||
|
||||
# Determine expenditure_concept_note: only non-empty for "total", explains
|
||||
# how the IG leg was assembled from legacy-era aggregates. When the Direct
|
||||
# leg is suppressed for at least one requested (year, category), append an
|
||||
# explicit warning rather than let the base note's "Total = Direct + IG"
|
||||
# framing stand unqualified for rows where that arithmetic didn't happen.
|
||||
expenditure_concept_note_for_prov <- if (identical(expenditure_concept, "total")) {
|
||||
base_note <- "Total = Direct + intergovernmental (M to local govts + L to state govts). Legacy-era IG is assembled from aggregate-flagged rows, which are year-disjoint from their modern leaf components; the L-- family total is excluded."
|
||||
if (direct_suppressed_flag) {
|
||||
paste0(
|
||||
base_note,
|
||||
" NOTE: for at least one requested (year, category) the Direct leg ",
|
||||
"has NO rows in this corpus (a legacy aggregate-only family) -- the ",
|
||||
"affected result rows report the intergovernmental leg alone, not ",
|
||||
"Direct + IG. See `expenditure_concept_direct_suppressed` and each ",
|
||||
"affected row's `notes`."
|
||||
)
|
||||
} else {
|
||||
base_note
|
||||
}
|
||||
} else {
|
||||
NA_character_
|
||||
}
|
||||
|
||||
prov <- .build_provenance(
|
||||
@@ -157,6 +304,9 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
subtype_col = subtype_col,
|
||||
basis = basis_for_prov,
|
||||
basis_note = basis_note_for_prov,
|
||||
expenditure_concept = expenditure_concept,
|
||||
expenditure_concept_note = expenditure_concept_note_for_prov,
|
||||
expenditure_concept_direct_suppressed = direct_suppressed_flag,
|
||||
harmonization = harmonization,
|
||||
recipe = recipe_block,
|
||||
suggestions = suggestions
|
||||
@@ -211,6 +361,43 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
if (identical(basis, "harmonized")) paste0(view_base, "_harmonized") else view_base
|
||||
}
|
||||
|
||||
#' @noRd
|
||||
.select_ig_view <- function(basis) {
|
||||
if (identical(basis, "harmonized")) "ig_annotated_harmonized" else "ig_annotated"
|
||||
}
|
||||
|
||||
#' Abort unless the active corpus's `summary_categories` actually carries
|
||||
#' intergovernmental (M/L) rows.
|
||||
#'
|
||||
#' The 66 M/L category rows arrived via cog_pipeline PR #59 with NO
|
||||
#' `schema_version` bump (`DESCRIPTION` still declares `MinCorpusSchema: 4`),
|
||||
#' so `schema_version` alone cannot gate `expenditure_concept = "total"` --
|
||||
#' a pre-#59 corpus can validly report schema_version 4, 5, or 6 and still
|
||||
#' have zero M/L rows in `summary_categories`. Against such a corpus,
|
||||
#' `ig_annotated`'s LEFT JOIN to `summary_categories` silently produces NA
|
||||
#' `category`/`spend_subtype` for every IG row: with a `category` filter
|
||||
#' this returns 0 rows (reads as "no intergovernmental spending" rather than
|
||||
#' "can't tell"), and with `category = NULL` every IG dollar collapses into
|
||||
#' one NA-subtype group that is invisible to the `spend_subtype ==
|
||||
#' "intergovernmental"` filter this package's own tests, roxygen, and
|
||||
#' vignette all rely on. Checking the data directly (rather than
|
||||
#' schema_version) is the only reliable gate.
|
||||
#' @noRd
|
||||
.require_ig_categories <- function(con, what = "expenditure_concept = \"total\"") {
|
||||
n <- DBI::dbGetQuery(con,
|
||||
"SELECT COUNT(*) AS n FROM summary_categories WHERE LEFT(item_code, 1) IN ('M', 'L')"
|
||||
)$n
|
||||
if (identical(as.integer(n), 0L)) {
|
||||
cli::cli_abort(c(
|
||||
sprintf("%s requires a corpus with intergovernmental category rows.", what),
|
||||
x = "The active corpus's `summary_categories` has no M/L (intergovernmental) rows.",
|
||||
i = "This corpus predates the intergovernmental category rows added by cog_pipeline PR #59.",
|
||||
i = "Point USCOGDATA_URL at a newer corpus that includes the M/L summary_categories rows."
|
||||
), class = "uscogdata_ig_categories_unsupported")
|
||||
}
|
||||
invisible(TRUE)
|
||||
}
|
||||
|
||||
#' @noRd
|
||||
.sql_lit_chr <- function(x) {
|
||||
safe <- gsub("'", "''", x, fixed = TRUE)
|
||||
@@ -218,7 +405,8 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
}
|
||||
|
||||
#' @noRd
|
||||
.build_verb_sql <- function(view, subtype_col, govid, years, category) {
|
||||
.build_verb_sql <- function(view, subtype_col, govid, years, category,
|
||||
ig_view = NULL) {
|
||||
govid_lit <- .sql_lit_chr(govid)
|
||||
years_lit <- paste(as.integer(years), collapse = ",")
|
||||
category_pred <- if (is.null(category)) {
|
||||
@@ -227,6 +415,26 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
sprintf("AND category IN (%s)", .sql_lit_chr(category))
|
||||
}
|
||||
|
||||
# expenditure_concept = "total" adds the intergovernmental leg. UNION ALL,
|
||||
# never UNION: the two legs are disjoint by item_code prefix (E/F/G vs M/L),
|
||||
# so de-duplication would be pure cost, and a silent row-drop if two
|
||||
# governments ever reported identical values.
|
||||
source_expr <- if (is.null(ig_view)) {
|
||||
view
|
||||
} else {
|
||||
sprintf("(SELECT * FROM %s UNION ALL SELECT * FROM %s)", view, ig_view)
|
||||
}
|
||||
|
||||
# bool_or(), not bool_and(): a no-op for the Direct/revenue legs (those
|
||||
# views filter NOT is_aggregate, so no row in any group is ever aggregate),
|
||||
# but load-bearing for the IG leg, which deliberately keeps aggregate rows
|
||||
# (see inst/sql/24-ig_long.sql). The wide era is dense -- every government
|
||||
# has a row for every code in a family, most of them $0 -- so a $0 leaf
|
||||
# commonly lands in the same (year, gov, subtype, category) group as the
|
||||
# real aggregate row. bool_and() would then read FALSE for that group even
|
||||
# though its dollars came entirely from an aggregate row, silently
|
||||
# suppressing the "Aggregate fallback applied" note on exactly the rows
|
||||
# this feature exists to surface.
|
||||
sprintf(
|
||||
"SELECT
|
||||
year,
|
||||
@@ -236,14 +444,14 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
category,
|
||||
SUM(amt) * 1000.0 AS amt_nominal,
|
||||
string_agg(DISTINCT item_code, ',' ORDER BY item_code) AS codes_included,
|
||||
bool_and(is_aggregate) AS aggregate_fallback
|
||||
bool_or(is_aggregate) AS aggregate_fallback
|
||||
FROM %2$s
|
||||
WHERE canonical_govid IN (%3$s)
|
||||
AND year IN (%4$s)
|
||||
%5$s
|
||||
GROUP BY year, canonical_govid, gov_name, xwalk_gov_name, %1$s, category
|
||||
ORDER BY year, canonical_govid, %1$s, category",
|
||||
subtype_col, view, govid_lit, years_lit, category_pred
|
||||
subtype_col, source_expr, govid_lit, years_lit, category_pred
|
||||
)
|
||||
}
|
||||
|
||||
@@ -296,11 +504,131 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
result
|
||||
}
|
||||
|
||||
#' Detect rows where expenditure_concept = "total" is reporting the
|
||||
#' intergovernmental leg with NO Direct counterpart in the same (year,
|
||||
#' canonical_govid, category) group AND a harmonization recipe actually
|
||||
#' recovers the missing Direct dollars for that exact (year, canonical_govid,
|
||||
#' category) triple.
|
||||
#'
|
||||
#' Bare Direct-row absence is deliberately NOT sufficient on its own: the
|
||||
#' dominant real cause of "no Direct sibling row" is a government that simply
|
||||
#' has no direct spending in that category (e.g. a state that funds K-12
|
||||
#' entirely through school districts), which is correct, ordinary data, not
|
||||
#' suppression. Genuine suppression -- a legacy aggregate-only family whose
|
||||
#' Direct-leg basis query excludes it by construction (spending_long/
|
||||
#' spending_long_harmonized both filter NOT is_aggregate) -- always has a
|
||||
#' covering harmonization recipe, because that is exactly what the recipe
|
||||
#' catalog exists to recover (see R/suggestions.R and `cog_recipes()`). So
|
||||
#' checking "does a recipe actually cover this triple" cleanly separates the
|
||||
#' two cases instead of conflating them.
|
||||
#'
|
||||
#' Returns `list(flag, notes)`, both the same length as `result`: `flag` is
|
||||
#' `TRUE` only for the `spend_subtype == "intergovernmental"` row(s) in a
|
||||
#' suppressed group, and `notes` names the recovering recipe(s) for those
|
||||
#' rows (`NA` everywhere else).
|
||||
#' @noRd
|
||||
.notes_column <- function(result) {
|
||||
.detect_direct_suppressed <- function(con, result, subtype_col) {
|
||||
n <- nrow(result)
|
||||
empty_notes <- rep(NA_character_, n)
|
||||
if (n == 0L) return(list(flag = logical(0), notes = character(0)))
|
||||
is_ig <- result[[subtype_col]] %in% "intergovernmental"
|
||||
if (!any(is_ig)) return(list(flag = rep(FALSE, n), notes = empty_notes))
|
||||
|
||||
key <- paste(result$year, result$canonical_govid, result$category, sep = "\r")
|
||||
has_direct <- key %in% unique(key[!is_ig])
|
||||
candidate <- is_ig & !has_direct
|
||||
|
||||
flag <- rep(FALSE, n)
|
||||
notes <- empty_notes
|
||||
if (!any(candidate)) return(list(flag = flag, notes = notes))
|
||||
|
||||
idx <- which(candidate)
|
||||
rows <- unique(result[idx, c("year", "canonical_govid", "category")])
|
||||
covering <- .covering_recipes(con, rows)
|
||||
cov_key <- paste(covering$year, covering$canonical_govid, covering$category,
|
||||
sep = "\r")
|
||||
|
||||
for (i in idx) {
|
||||
k <- paste(result$year[i], result$canonical_govid[i], result$category[i],
|
||||
sep = "\r")
|
||||
m <- match(k, cov_key)
|
||||
if (is.na(m)) next
|
||||
ids <- covering$recipe_ids[[m]]
|
||||
if (length(ids) == 0L) next
|
||||
flag[i] <- TRUE
|
||||
notes[i] <- sprintf(
|
||||
"Direct component is unavailable through this basis for this year; recover it via recipe = '%s' (see cog_recipes()).",
|
||||
paste(sort(unique(ids)), collapse = "', '")
|
||||
)
|
||||
}
|
||||
list(flag = flag, notes = notes)
|
||||
}
|
||||
|
||||
#' For each (year, canonical_govid, category) triple potentially affected by
|
||||
#' a suppressed Direct leg, find the harmonization recipe(s) that (a) cover
|
||||
#' this `category` (share a component item_code via `summary_categories`,
|
||||
#' excluding any recipe that is itself entirely intergovernmental M/L -- the
|
||||
#' same exclusion `.build_suggestions()` applies, see I2) and (b) actually
|
||||
#' produce a `long` row for this exact (canonical_govid, year) via the same
|
||||
#' generic join `.run_recipe()` uses (component year_min/year_max +
|
||||
#' gov_type_scope, no is_aggregate filter -- a recipe's whole point is to
|
||||
#' recover data that's aggregate-only). Adds a list-column `recipe_ids`
|
||||
#' (possibly length-0) to `rows`.
|
||||
#' @noRd
|
||||
.covering_recipes <- function(con, rows) {
|
||||
rows$recipe_ids <- vector("list", nrow(rows))
|
||||
cats <- unique(rows$category[!is.na(rows$category)])
|
||||
if (length(cats) == 0L) return(rows)
|
||||
|
||||
cand <- DBI::dbGetQuery(con, sprintf(
|
||||
"SELECT DISTINCT sc.category, r.recipe_id
|
||||
FROM harmonization_recipes r
|
||||
JOIN summary_categories sc ON sc.item_code = r.component_code
|
||||
WHERE sc.category IN (%s)
|
||||
AND r.recipe_id NOT IN (
|
||||
SELECT DISTINCT recipe_id FROM harmonization_recipes
|
||||
WHERE LEFT(component_code, 1) IN ('M', 'L')
|
||||
)",
|
||||
.sql_lit_chr(cats)
|
||||
))
|
||||
if (nrow(cand) == 0L) return(rows)
|
||||
|
||||
recipe_ids_all <- unique(cand$recipe_id)
|
||||
govids <- unique(rows$canonical_govid)
|
||||
years <- unique(rows$year)
|
||||
covered <- DBI::dbGetQuery(con, sprintf(
|
||||
"SELECT DISTINCT r.recipe_id, l.canonical_govid, l.year
|
||||
FROM long l
|
||||
JOIN harmonization_recipes r
|
||||
ON l.item_code = r.component_code
|
||||
AND l.year BETWEEN r.year_min AND r.year_max
|
||||
AND (r.gov_type_scope = 'all'
|
||||
OR (r.gov_type_scope = 'state' AND l.type = 0)
|
||||
OR (r.gov_type_scope = 'local' AND l.type BETWEEN 1 AND 3))
|
||||
WHERE r.recipe_id IN (%s)
|
||||
AND l.canonical_govid IN (%s)
|
||||
AND l.year IN (%s)",
|
||||
.sql_lit_chr(recipe_ids_all), .sql_lit_chr(govids), paste(years, collapse = ",")
|
||||
))
|
||||
|
||||
for (i in seq_len(nrow(rows))) {
|
||||
cat_i <- rows$category[i]
|
||||
if (is.na(cat_i)) next
|
||||
cat_recipe_ids <- cand$recipe_id[cand$category == cat_i]
|
||||
if (length(cat_recipe_ids) == 0L) next
|
||||
sub <- covered[covered$canonical_govid == rows$canonical_govid[i] &
|
||||
covered$year == rows$year[i] &
|
||||
covered$recipe_id %in% cat_recipe_ids, ]
|
||||
rows$recipe_ids[[i]] <- sort(unique(sub$recipe_id))
|
||||
}
|
||||
rows
|
||||
}
|
||||
|
||||
#' @noRd
|
||||
.notes_column <- function(result, direct_suppressed_notes = NULL) {
|
||||
n <- nrow(result)
|
||||
if (n == 0L) return(character(0))
|
||||
parts <- vector("list", 2L)
|
||||
parts <- vector("list", 3L)
|
||||
agg <- result[["aggregate_fallback"]]
|
||||
parts[[1]] <- if (!is.null(agg)) {
|
||||
ifelse(agg %in% TRUE,
|
||||
@@ -317,6 +645,11 @@ cog_spending <- function(govid, years, category = NULL,
|
||||
} else {
|
||||
rep(NA_character_, n)
|
||||
}
|
||||
parts[[3]] <- if (!is.null(direct_suppressed_notes)) {
|
||||
direct_suppressed_notes
|
||||
} else {
|
||||
rep(NA_character_, n)
|
||||
}
|
||||
out <- character(n)
|
||||
for (i in seq_len(n)) {
|
||||
pieces <- vapply(parts, `[[`, character(1), i)
|
||||
|
||||
+139
-7
@@ -22,6 +22,13 @@
|
||||
# positives from ordinary reporting variance -- most governments don't use
|
||||
# every sibling code in a multi-code category every year, and that is not
|
||||
# a format-boundary gap worth signposting.
|
||||
#
|
||||
# C1(a): for expenditure_concept = "total" callers, `result` here must
|
||||
# already be the Direct-leg subset (the caller filters out
|
||||
# spend_subtype == "intergovernmental" rows before calling in). A gap year
|
||||
# is "the requested year has no Direct rows", never "no rows at all" --
|
||||
# an IG row surviving on a legacy aggregate that Direct excludes must not
|
||||
# read as coverage and cancel the very suggestion that would recover it.
|
||||
|
||||
#' Build the `prov$suggestions` list for a (non-recipe) basis = "harmonized"
|
||||
#' verb call: recipes whose generic join would fill a real gap in `result`.
|
||||
@@ -33,18 +40,41 @@
|
||||
#' @param category `category` argument as passed to the verb (character
|
||||
#' vector or `NULL`; suggestions are only computed when non-NULL).
|
||||
#' @param result The verb's already-computed result tibble (post basis
|
||||
#' query, pre per_capita/adjust_to_year).
|
||||
#' query, pre per_capita/adjust_to_year), pre-filtered to the Direct leg
|
||||
#' only when the caller's `expenditure_concept = "total"` (see C1(a)).
|
||||
#' @param basis The *resolved* basis (`"harmonized"` or `"raw"`).
|
||||
#' @return List of `list(recipe_id, label, available_years, hint)`, possibly
|
||||
#' empty.
|
||||
#' @param flow_prefixes The calling verb's own flow-type prefixes (e.g.
|
||||
#' `c("E", "F", "G")` for `cog_spending()`, `c("T", "A", "U", "B", "C",
|
||||
#' "D")` for `cog_revenue()` -- see `.verb_spendrev()`). Passed through to
|
||||
#' `.attach_ig_counterparts()` to keep the intergovernmental-counterpart
|
||||
#' lookup scoped to the calling verb's own flow family.
|
||||
#' @return List of `list(recipe_id, label, available_years, hint,
|
||||
#' ig_recipe_id)`, possibly empty.
|
||||
#' @noRd
|
||||
.build_suggestions <- function(con, govid, years, category, result, basis) {
|
||||
.build_suggestions <- function(con, govid, years, category, result, basis,
|
||||
flow_prefixes) {
|
||||
if (!identical(basis, "harmonized") || is.null(category)) return(list())
|
||||
|
||||
# Exclude any recipe that is ITSELF an intergovernmental (M/L) recipe --
|
||||
# i.e. every one of its own component codes is M/L-prefixed. Without this,
|
||||
# a category whose summary_categories rows span both a Direct family
|
||||
# (e.g. E04/E05, "Corrections") and its M/L counterpart (M04/M05, same
|
||||
# category since Task 1) makes the M/L recipe itself (e.g.
|
||||
# `corrections_ig_local_combined`) a raw top-level candidate for a plain
|
||||
# (Direct) cog_spending() call -- following that hint would silently
|
||||
# return intergovernmental dollars under `expenditure_concept = "direct"`
|
||||
# provenance. This is a stronger, unconditional exclusion than the
|
||||
# flow-prefix gate below/in `.attach_ig_counterparts()`: an M/L recipe
|
||||
# should never be suggested as a coverage-gap filler for EITHER verb, not
|
||||
# just kept from being named as the *counterpart* of another suggestion.
|
||||
candidates <- DBI::dbGetQuery(con, sprintf(
|
||||
"SELECT DISTINCT recipe_id FROM harmonization_recipes
|
||||
WHERE component_code IN (
|
||||
SELECT DISTINCT item_code FROM summary_categories WHERE category IN (%s)
|
||||
)
|
||||
AND recipe_id NOT IN (
|
||||
SELECT DISTINCT recipe_id FROM harmonization_recipes
|
||||
WHERE LEFT(component_code, 1) IN ('M', 'L')
|
||||
)",
|
||||
.sql_lit_chr(category)
|
||||
))$recipe_id
|
||||
@@ -98,19 +128,121 @@
|
||||
hint = sprintf("re-run with recipe = '%s'", rid)
|
||||
)
|
||||
}
|
||||
suggestions
|
||||
.attach_ig_counterparts(con, suggestions, flow_prefixes)
|
||||
}
|
||||
|
||||
#' Attach `ig_recipe_id` to each suggestion: the intergovernmental-expenditure
|
||||
#' recipe (an M-to-local or L-to-state recipe) whose component codes cover
|
||||
#' exactly the same set of function suffixes as the firing recipe's own
|
||||
#' components, e.g. `corrections_combined`'s {E04, E05} -> suffixes {"04",
|
||||
#' "05"} matches `corrections_ig_local_combined`'s {M04, M05} -> the same
|
||||
#' {"04", "05"}. `NULL` when no such recipe exists, which also covers the
|
||||
#' case where the firing recipe already IS the IG recipe (self-matches are
|
||||
#' excluded, so an IG recipe never names itself as its own counterpart).
|
||||
#'
|
||||
#' Matching is deliberately an exact set match, not "any suffix in common":
|
||||
#' the two-digit suffix only means the same "function" across recipes that
|
||||
#' share the underlying Census functional-classification scheme (E/F/G/L/M
|
||||
#' all use "04"/"05" for corrections). M/L "combined other" codes (47/89/
|
||||
#' 91-94) reuse digits for an unrelated catch-all construct, so e.g.
|
||||
#' `general_gov_e89_wide`'s {E85, E89} -> {"85", "89"} must NOT match
|
||||
#' `ige_local_m89_wide`'s {"89", "91", "92", "93"} on the shared "89" alone.
|
||||
#' Checked by hand against the full harmonization_recipes catalog: only the
|
||||
#' corrections family (E/F/G/M, suffixes 04/05) has an exact-set match in
|
||||
#' this corpus.
|
||||
#'
|
||||
#' Exact-set suffix matching is NOT enough on its own, though: the same
|
||||
#' reused-digit problem exists ACROSS the revenue-side IG families too.
|
||||
#' `ig_local_d47_wide` (D47/D94, suffixes {"47","94"}) is an exact-set match
|
||||
#' for `ige_local_m47_wide` (M47/M94, same suffixes) even though one is
|
||||
#' intergovernmental REVENUE received from local governments and the other is
|
||||
#' intergovernmental EXPENDITURE paid to local governments -- unrelated flows
|
||||
#' that happen to reuse "47"/"94" for their own "transit/utilities" and
|
||||
#' "other/combined" catch-alls. `ig_federal_b47_wide`, `ig_state_c47_wide`,
|
||||
#' and their `*_89` siblings all collide the same way. None of this is
|
||||
#' reachable via `cog_revenue()` in the bundled fixture today (its B/C/D
|
||||
#' recipes never happen to have a covered gap year for any fixture govid),
|
||||
#' but it IS reachable via a mis-scoped `cog_spending()` call on a
|
||||
#' revenue-only category, e.g. `cog_spending(gov, category = "IG Federal")`
|
||||
#' fires `ig_federal_b47_wide`/`ig_federal_b89_wide` for real in the fixture
|
||||
#' -- so this is a live, not merely theoretical, gap.
|
||||
#'
|
||||
#' Two flow-family checks close this, both required (see
|
||||
#' `tests/testthat/test-expenditure-concept.R`, "revenue-flavored ... never
|
||||
#' receives an M/L counterpart" tests, for the pairwise verification):
|
||||
#' 1. `own_prefix %in% flow_prefixes`: the firing recipe's own component
|
||||
#' codes must belong to the calling verb's own flow family (the same
|
||||
#' `flow_prefixes` `.build_harmonization_block()` uses, see
|
||||
#' `R/basis.R`). This blocks a recipe surfaced through a mis-scoped
|
||||
#' category from ever reaching the M/L search, e.g. `cog_spending()`'s
|
||||
#' flow_prefixes are `c("E","F","G")`, which `ig_federal_b47_wide`'s own
|
||||
#' `"B"` is not part of.
|
||||
#' 2. `own_prefix %in% c("E","F","G")`: M/L only ever pairs with the
|
||||
#' DIRECT-expenditure family, never with revenue (`cog_revenue()`'s
|
||||
#' flow_prefixes already fold B/C/D in as ordinary revenue -- there is
|
||||
#' no separate "Total" bolt-on for revenue the way `expenditure_concept`
|
||||
#' adds one for spending) and never with ANOTHER M/L recipe (without
|
||||
#' this check, `ige_local_m47_wide` would wrongly match sibling
|
||||
#' `ige_state_l47_wide` on their shared {"47","94"} suffix set).
|
||||
#' Condition 1 alone does not catch this: under `cog_revenue()`,
|
||||
#' `ig_federal_b47_wide`'s own `"B"` IS inside revenue's own
|
||||
#' `flow_prefixes`, so only this second, family-specific check blocks
|
||||
#' the search.
|
||||
#' @noRd
|
||||
.attach_ig_counterparts <- function(con, suggestions, flow_prefixes) {
|
||||
if (length(suggestions) == 0L) return(suggestions)
|
||||
|
||||
comp <- DBI::dbGetQuery(con,
|
||||
"SELECT recipe_id, component_code FROM harmonization_recipes")
|
||||
comp$prefix <- substr(comp$component_code, 1L, 1L)
|
||||
comp$suffix <- substr(comp$component_code, 2L, nchar(comp$component_code))
|
||||
suffix_sets <- lapply(split(comp$suffix, comp$recipe_id), function(x) sort(unique(x)))
|
||||
prefix_sets <- lapply(split(comp$prefix, comp$recipe_id), function(x) sort(unique(x)))
|
||||
|
||||
ig_recipe_ids <- unique(comp$recipe_id[comp$prefix %in% c("M", "L")])
|
||||
|
||||
find_counterpart <- function(rid) {
|
||||
own_prefix <- prefix_sets[[rid]]
|
||||
own_suffix <- suffix_sets[[rid]]
|
||||
if (is.null(own_prefix) || is.null(own_suffix)) return(NULL)
|
||||
if (!all(own_prefix %in% flow_prefixes)) return(NULL)
|
||||
if (!all(own_prefix %in% c("E", "F", "G"))) return(NULL)
|
||||
for (cand in ig_recipe_ids) {
|
||||
if (identical(cand, rid)) next
|
||||
if (setequal(suffix_sets[[cand]], own_suffix)) return(cand)
|
||||
}
|
||||
NULL
|
||||
}
|
||||
|
||||
lapply(suggestions, function(s) {
|
||||
# `s$ig_recipe_id <- NULL` would DELETE the element rather than set it
|
||||
# (standard R list-assignment gotcha), leaving no-match entries missing
|
||||
# the key entirely instead of carrying it as NULL. Single-bracket
|
||||
# assignment with a wrapped list preserves a NULL-valued element so the
|
||||
# field is always present, per the brief's "NULL when there is none".
|
||||
s["ig_recipe_id"] <- list(find_counterpart(s$recipe_id))
|
||||
s
|
||||
})
|
||||
}
|
||||
|
||||
#' Emit the single cli::cli_inform() message summarizing all suggestions
|
||||
#' for a verb call (the brief's "one message", not one per suggestion).
|
||||
#' Bullet text is pre-formatted plain text (no cli/glue `{}` markup) since
|
||||
#' recipe ids/labels are untrusted-ish data values, not literal call-site
|
||||
#' expressions.
|
||||
#' expressions. When a suggestion has an `ig_recipe_id`, one indented
|
||||
#' continuation line is appended naming the intergovernmental counterpart
|
||||
#' recipe (embedded `\n` renders as a hanging-indent continuation of the
|
||||
#' same bullet under cli, not a new bullet).
|
||||
#' @noRd
|
||||
.inform_suggestions <- function(suggestions) {
|
||||
bullets <- vapply(suggestions, function(s) {
|
||||
sprintf("%s (%d-%d): %s", s$recipe_id,
|
||||
bullet <- sprintf("%s (%d-%d): %s", s$recipe_id,
|
||||
s$available_years[1], s$available_years[2], s$hint)
|
||||
if (!is.null(s$ig_recipe_id)) {
|
||||
bullet <- paste0(bullet, sprintf(
|
||||
"\n intergovernmental counterpart: recipe = '%s'", s$ig_recipe_id))
|
||||
}
|
||||
bullet
|
||||
}, character(1))
|
||||
cli::cli_inform(c(
|
||||
i = "Coverage gap detected for the requested years; a harmonization recipe may fill it:",
|
||||
|
||||
@@ -1,23 +1,35 @@
|
||||
# R/views.R
|
||||
|
||||
# SQL files whose view definitions read schema-v5-only parquet tables
|
||||
# SQL files that cannot be registered unconditionally against a v4 corpus,
|
||||
# for one of two distinct reasons -- both fail at CREATE VIEW time (DuckDB
|
||||
# resolves a view's source schema eagerly, even though it defers execution),
|
||||
# so a v4 corpus can't tolerate either unconditionally:
|
||||
#
|
||||
# (a) Missing FILE. 33-/34-/35- read_parquet() a v5-only parquet table
|
||||
# (harmonization_map.parquet, harmonization_recipes.parquet,
|
||||
# series_breaks.parquet) or select from views built on top of them. DuckDB's
|
||||
# read_parquet() resolves the file at CREATE VIEW time (even for a view, it
|
||||
# still needs the source schema) and errors immediately -- "IO Error: No
|
||||
# files found" -- if the path doesn't exist, so these cannot be registered
|
||||
# unconditionally against a v4 corpus the way the rest of inst/sql/ is.
|
||||
# Registration is therefore gated on manifest$schema_version >= 5; verb-level
|
||||
# *usage* of the resulting views is separately gated by .resolve_basis() /
|
||||
# .require_schema_v5().
|
||||
# series_breaks.parquet) that doesn't exist at all on a v4 corpus --
|
||||
# "IO Error: No files found".
|
||||
#
|
||||
# (b) Missing COLUMN. 22-/23-/25- reference `long.harmonized_code`, a
|
||||
# column that does not exist on a v4 corpus's `long` table (harmonized
|
||||
# space was introduced in schema v5) -- "Binder Error: Referenced
|
||||
# column harmonized_code not found". 42-/43-/45- are on this list only
|
||||
# because they SELECT s.* FROM the (a)/(b) views above, so they'd fail
|
||||
# to resolve their own source view if it weren't already skipped.
|
||||
#
|
||||
# Registration is therefore gated on manifest$schema_version >= 5 for all of
|
||||
# them; verb-level *usage* of the resulting views is separately gated by
|
||||
# .resolve_basis() / .require_schema_v5().
|
||||
.harmonization_view_files <- c(
|
||||
"22-spending_long_harmonized.sql",
|
||||
"23-revenue_long_harmonized.sql",
|
||||
"25-ig_long_harmonized.sql",
|
||||
"33-harmonization_map.sql",
|
||||
"34-harmonization_recipes.sql",
|
||||
"35-series_breaks_pq.sql",
|
||||
"42-spending_annotated_harmonized.sql",
|
||||
"43-revenue_annotated_harmonized.sql"
|
||||
"43-revenue_annotated_harmonized.sql",
|
||||
"45-ig_annotated_harmonized.sql"
|
||||
)
|
||||
|
||||
#' Register DuckDB views from inst/sql/ SQL files
|
||||
|
||||
@@ -19,20 +19,59 @@ package implements.
|
||||
# pak::pkg_install("gitea.civilytics.org/Civilytics/uscogdata")
|
||||
```
|
||||
|
||||
## Amounts are in full US dollars
|
||||
|
||||
Every amount column this package returns — `amt_nominal`, `amt_real`,
|
||||
`amt_per_capita_nominal`, `amt_per_capita_real` — is in **full US dollars**.
|
||||
|
||||
The raw Census source files report **thousands of dollars**, and the corpus's
|
||||
own `amt` column preserves that. The verbs multiply by 1000 on the way out, so
|
||||
you never have to. The conversion is recorded in every result:
|
||||
|
||||
```r
|
||||
r <- cog_spending("552025209777", 2020L)
|
||||
attr(r, "provenance")$transformations$units_conversion
|
||||
#> $applied TRUE $source_unit "$1,000s (raw Census)" $target_unit "$USD" $multiplier 1000
|
||||
```
|
||||
|
||||
**Do not multiply again.** If you have read elsewhere that COG amounts are in
|
||||
`$1,000s` — true of the raw corpus, and of `cog_explorer`'s conventions doc —
|
||||
that rule does not apply to anything a `cog_*()` verb hands you. Applying it
|
||||
twice overstates every figure by 1000x, and the result looks plausible rather
|
||||
than obviously wrong.
|
||||
|
||||
## Configuration
|
||||
|
||||
- `USCOGDATA_URL` — corpus root URL (public Nextcloud share, trailing slash)
|
||||
- `USCOGDATA_CACHE_DIR` — optional override for the manifest cache directory
|
||||
- `USCOGDATA_MANIFEST_TTL_SECS` — optional manifest re-fetch TTL (default 3600)
|
||||
|
||||
## Direct vs Total spending
|
||||
|
||||
`cog_spending(..., expenditure_concept = c("direct", "total"))` controls
|
||||
whose spending a result counts. `"direct"` (the default) is a government's
|
||||
own current operations, capital outlay, and other direct spending. `"total"`
|
||||
additionally adds in the intergovernmental legs — money it hands to other
|
||||
governments to spend on its behalf — which is meaningful for describing one
|
||||
government's own budget over time, but double-counts when summed across
|
||||
governments (a state's payment to a county is the same dollar the county
|
||||
reports as its own direct spending).
|
||||
|
||||
**Rule of thumb: any figure that spans more than one government uses
|
||||
`direct`.** `cog_geographic_rollup()` and `cog_peer_compare()` enforce this
|
||||
by refusing `expenditure_concept = "total"`. See
|
||||
`vignette("total-spending", package = "uscogdata")` for the full
|
||||
explanation with worked examples.
|
||||
|
||||
## Developer notes
|
||||
|
||||
### Testing
|
||||
|
||||
The package ships a bundled fixture corpus at `inst/extdata/fixture_corpus/` —
|
||||
a 3.6 MB two-year slice (2019 + 2020) of the full corpus covering all 50
|
||||
states. `tests/testthat/setup.R` automatically points `USCOGDATA_URL` at this
|
||||
fixture, so the full test suite runs offline with no network dependency:
|
||||
a 15 MB four-year slice (2011, 2012, 2019, 2020) of the full corpus covering
|
||||
all 50 states. `tests/testthat/setup.R` automatically points `USCOGDATA_URL`
|
||||
at this fixture, so the full test suite runs offline with no network
|
||||
dependency:
|
||||
|
||||
```r
|
||||
devtools::test() # uses bundled fixture, no credentials required
|
||||
|
||||
@@ -11,12 +11,14 @@
|
||||
# Each partition is a full year (all states/govs) as published, so
|
||||
# Broward County FL and every other previously-pinned government stay
|
||||
# covered without any per-gov slicing logic.
|
||||
# 2. Copies the full canonical_fips_xwalk.parquet, canonical_alias.parquet,
|
||||
# summary_categories.parquet, harmonization_map.parquet,
|
||||
# harmonization_recipes.parquet, and series_breaks.parquet metadata
|
||||
# tables as-is (these are small cross-vintage registries, not
|
||||
# partitioned by year, so the fixture ships the complete tables rather
|
||||
# than a year-scoped subset).
|
||||
# 2. Copies every metadata parquet the publish tree ships (see
|
||||
# .FIXTURE_METADATA_FILES) as-is. These are small cross-vintage
|
||||
# registries, not partitioned by year, so the fixture ships the complete
|
||||
# tables rather than a year-scoped subset. representation.parquet and
|
||||
# code_set.parquet are what make the sparse wide era interpretable --
|
||||
# absence means "$0" in a dense_source year and "not reported" in a
|
||||
# sparse_source one -- so a fixture without them cannot represent the
|
||||
# published corpus.
|
||||
# 3. Resyncs the four reference docs (data_dictionary.md,
|
||||
# reader-specification.md, README.md, series_breaks.md) from the
|
||||
# publish tree's docs/.
|
||||
@@ -38,6 +40,22 @@
|
||||
# source("data-raw/regenerate_fixture_corpus.R")
|
||||
# regenerate_fixture_corpus(publish_cache_dir = "/path/to/publish_cache")
|
||||
|
||||
# Every metadata parquet the publish tree ships, in the order they appear in
|
||||
# the corpus manifest. Single source of truth for both the copy step and the
|
||||
# fixture manifest, so the two can never drift apart.
|
||||
.FIXTURE_METADATA_FILES <- c(
|
||||
"canonical_alias.parquet",
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"census_collection_coverage.parquet",
|
||||
"code_set.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"lineage_events.parquet",
|
||||
"representation.parquet",
|
||||
"series_breaks.parquet",
|
||||
"summary_categories.parquet"
|
||||
)
|
||||
|
||||
regenerate_fixture_corpus <- function(
|
||||
publish_cache_dir = file.path(
|
||||
"..", "cog_pipeline", "_targets", "publish_cache"
|
||||
@@ -100,20 +118,11 @@ regenerate_fixture_corpus <- function(
|
||||
invisible(NULL)
|
||||
}
|
||||
|
||||
# Copy the full (not year-scoped) canonical_fips_xwalk, canonical_alias,
|
||||
# summary_categories, and (schema v5+) harmonization_map/
|
||||
# harmonization_recipes/series_breaks parquet tables.
|
||||
# Copy the full (not year-scoped) metadata tables listed in
|
||||
# .FIXTURE_METADATA_FILES.
|
||||
#' @noRd
|
||||
.copy_metadata_parquets <- function(publish_cache_dir, fixture_dir) {
|
||||
files <- c(
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"canonical_alias.parquet",
|
||||
"summary_categories.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"series_breaks.parquet"
|
||||
)
|
||||
for (f in files) {
|
||||
for (f in .FIXTURE_METADATA_FILES) {
|
||||
src <- file.path(publish_cache_dir, "data", f)
|
||||
dst <- file.path(fixture_dir, "data", f)
|
||||
if (!file.exists(src)) {
|
||||
@@ -179,15 +188,7 @@ regenerate_fixture_corpus <- function(
|
||||
)
|
||||
})
|
||||
|
||||
metadata_files <- c(
|
||||
"canonical_alias.parquet",
|
||||
"canonical_fips_xwalk.parquet",
|
||||
"summary_categories.parquet",
|
||||
"harmonization_map.parquet",
|
||||
"harmonization_recipes.parquet",
|
||||
"series_breaks.parquet"
|
||||
)
|
||||
metadata <- lapply(metadata_files, function(f) {
|
||||
metadata <- lapply(.FIXTURE_METADATA_FILES, function(f) {
|
||||
rel <- file.path("data", f)
|
||||
path <- file.path(fixture_dir, rel)
|
||||
list(
|
||||
@@ -203,13 +204,16 @@ regenerate_fixture_corpus <- function(
|
||||
pipeline_commit = source_manifest$pipeline_commit,
|
||||
fixture_note = paste(
|
||||
"Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full",
|
||||
"corpus available via USCOGDATA_URL. Regenerated for Phase R2",
|
||||
"(schema_version 5, harmonization_map/harmonization_recipes/",
|
||||
"series_breaks parquet tables added). 2011/2012 straddle the",
|
||||
"wide-aggregate -> modern-leaf format boundary exercised by basis=",
|
||||
"\"harmonized\" and recipe= queries; 2019/2020 retain the prior",
|
||||
"per-capita/CPI regression anchors. Full canonical_fips_xwalk master",
|
||||
"and canonical_alias lookup table included via",
|
||||
"corpus available via USCOGDATA_URL. Regenerated from the sparsified",
|
||||
"schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit",
|
||||
"zeros, so FY2011 absence means Census published $0 while FY2012+",
|
||||
"absence means not reported. representation.parquet and",
|
||||
"code_set.parquet carry that rule and ship in full, as do every other",
|
||||
"metadata table in the publish tree. 2011/2012 straddle both the",
|
||||
"wide-aggregate -> modern-leaf format boundary (exercised by",
|
||||
"basis=\"harmonized\" and recipe= queries) and the dense -> sparse",
|
||||
"representation boundary (SB194); 2019/2020 retain the prior",
|
||||
"per-capita/CPI regression anchors. Regenerated via",
|
||||
"data-raw/regenerate_fixture_corpus.R."
|
||||
),
|
||||
data_vintage = source_manifest$data_vintage,
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+30
-10
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"schema_version": 6,
|
||||
"built_at": "2026-07-23T16:14:30Z",
|
||||
"pipeline_commit": "4f992a0",
|
||||
"fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated for Phase R2 (schema_version 5, harmonization_map/harmonization_recipes/ series_breaks parquet tables added). 2011/2012 straddle the wide-aggregate -> modern-leaf format boundary exercised by basis= \"harmonized\" and recipe= queries; 2019/2020 retain the prior per-capita/CPI regression anchors. Full canonical_fips_xwalk master and canonical_alias lookup table included via data-raw/regenerate_fixture_corpus.R.",
|
||||
"built_at": "2026-07-30T14:07:36Z",
|
||||
"pipeline_commit": "83f9715",
|
||||
"fixture_note": "Four-year (2011, 2012, 2019, 2020) fixture for uscogdata tests. Full corpus available via USCOGDATA_URL. Regenerated from the sparsified schema-v6 corpus: the wide era (<= FY2011) no longer stores explicit zeros, so FY2011 absence means Census published $0 while FY2012+ absence means not reported. representation.parquet and code_set.parquet carry that rule and ship in full, as do every other metadata table in the publish tree. 2011/2012 straddle both the wide-aggregate -> modern-leaf format boundary (exercised by basis=\"harmonized\" and recipe= queries) and the dense -> sparse representation boundary (SB194); 2019/2020 retain the prior per-capita/CPI regression anchors. Regenerated via data-raw/regenerate_fixture_corpus.R.",
|
||||
"data_vintage": {
|
||||
"source_vintages": {
|
||||
"2012": "10162019",
|
||||
@@ -36,9 +36,9 @@
|
||||
{
|
||||
"year": 2011,
|
||||
"path": "data/long/year=2011/part-0.parquet",
|
||||
"sha256": "84302ab364dc9fc3b3fbbc3c3f8b826e3508b4d73ff7c42d094d3863cd1e37b5",
|
||||
"row_count": 2864212,
|
||||
"size_bytes": 3845911
|
||||
"sha256": "7848e18497080c8980a4f89c5b386205b2c5bc90db6773827ea01ab3943d16b1",
|
||||
"row_count": 496004,
|
||||
"size_bytes": 2202455
|
||||
},
|
||||
{
|
||||
"year": 2012,
|
||||
@@ -74,9 +74,14 @@
|
||||
"description": "canonical_fips_xwalk.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/summary_categories.parquet",
|
||||
"sha256": "8e6fcd4dd9bb4723841a67233b19388c9762dfc23b4479501183cebf7ea3c1b5",
|
||||
"description": "summary_categories.parquet"
|
||||
"path": "data/census_collection_coverage.parquet",
|
||||
"sha256": "143e025616cde684da7c4442bc00d07fbd1556fabb0ea96223931b737e5d10a4",
|
||||
"description": "census_collection_coverage.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/code_set.parquet",
|
||||
"sha256": "4cffcb0198dd51e4ff2b694050bb371a5f9965cdac12f25521cb628fb8e118a9",
|
||||
"description": "code_set.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/harmonization_map.parquet",
|
||||
@@ -88,10 +93,25 @@
|
||||
"sha256": "1133e9a0b02f8f34f5f936e55c5ecd596bb8a55d8425dcce76767f0f3203581c",
|
||||
"description": "harmonization_recipes.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/lineage_events.parquet",
|
||||
"sha256": "36c16acfbe621d61010984767f1c566993b8a5f481a2c1e134c4c0a600e4502f",
|
||||
"description": "lineage_events.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/representation.parquet",
|
||||
"sha256": "31ec328a7dd505a321b45f97aafff12e53d68a1a986f63509863035b22a4360d",
|
||||
"description": "representation.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/series_breaks.parquet",
|
||||
"sha256": "b0b6794b6887a4f300079adfa10029c2a77109faa4952fbff1c5a270793cc02b",
|
||||
"sha256": "5ae050dd7a76c4d25e5f99e7c2e81c1896482e3504e0443b47ab5d78ba148953",
|
||||
"description": "series_breaks.parquet"
|
||||
},
|
||||
{
|
||||
"path": "data/summary_categories.parquet",
|
||||
"sha256": "e71d6d70d767c26c983fe56213baf204355f879582aa94841e62d9aea1877f83",
|
||||
"description": "summary_categories.parquet"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
@@ -12,6 +12,19 @@
|
||||
"category": { "type": ["string", "array", "null"] },
|
||||
"basis": { "type": ["string", "null"] },
|
||||
"basis_note": { "type": ["string", "null"] },
|
||||
"expenditure_concept": {
|
||||
"type": "string",
|
||||
"enum": ["direct", "total"],
|
||||
"description": "Which spending concept produced this result. 'direct' is the government's own E/F/G spending; 'total' adds its intergovernmental payments (M to local governments, L to state governments). Only 'direct' is valid for results combined across governments."
|
||||
},
|
||||
"expenditure_concept_note": {
|
||||
"type": ["string", "null"],
|
||||
"description": "How the intergovernmental leg was assembled; null for 'direct'."
|
||||
},
|
||||
"expenditure_concept_direct_suppressed": {
|
||||
"type": "boolean",
|
||||
"description": "TRUE when expenditure_concept = 'total' and at least one requested (year, category) has intergovernmental rows but NO Direct rows in this corpus (typically a legacy aggregate-only family) -- those result rows report the intergovernmental leg alone, not Direct + IG. Always FALSE for expenditure_concept = 'direct'. See the affected rows' `notes` for the recovering recipe, if any."
|
||||
},
|
||||
"harmonization": { "type": "object" },
|
||||
"recipe": { "type": ["object", "null"] },
|
||||
"suggestions": { "type": "array" },
|
||||
@@ -20,6 +33,11 @@
|
||||
"aggregate_fallback": { "type": ["object", "null"] },
|
||||
"transformations":{ "type": "object" },
|
||||
"series_break_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"corpus_break_refs": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" },
|
||||
"description": "Ids of catalogued series breaks whose fin_code is the literal 'ALL' -- caveats about the corpus as a whole (dollar precision across 1976/1977, imputation exclusion from 2002, the dense -> sparse representation change at 2012, the government id scheme change at 2017) rather than about one item code. Selected on the break_year window alone, so they do not depend on which codes a result contains. Disjoint from series_break_refs by construction: an entry qualifies the whole result, not one series."
|
||||
},
|
||||
"manifest": { "type": "object" },
|
||||
"sql_query": { "type": "string" }
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
CREATE OR REPLACE VIEW spending_long AS
|
||||
SELECT *
|
||||
FROM long
|
||||
WHERE LEFT(item_code, 1) IN ('E', 'F', 'G', 'K')
|
||||
WHERE LEFT(item_code, 1) IN ('E', 'F', 'G')
|
||||
AND NOT is_aggregate;
|
||||
|
||||
@@ -3,4 +3,4 @@ SELECT * REPLACE (harmonized_code AS item_code)
|
||||
FROM long
|
||||
WHERE NOT is_aggregate
|
||||
AND harmonized_code IS NOT NULL
|
||||
AND LEFT(harmonized_code, 1) IN ('E', 'F', 'G', 'K');
|
||||
AND LEFT(harmonized_code, 1) IN ('E', 'F', 'G');
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
-- Intergovernmental expenditure rows (M = to local govts, L = to state govts).
|
||||
--
|
||||
-- Deliberately does NOT filter `NOT is_aggregate`, unlike spending_long. In the
|
||||
-- wide era (<= FY2011) the IG families M05/M12/M47/M89/L47/L89 are published
|
||||
-- ONLY as aggregate-flagged rows -- filtering them would hide ~70% of legacy IG
|
||||
-- dollars and make Total silently collapse to Direct. This is safe because the
|
||||
-- aggregate codes and their modern leaf components are strictly year-disjoint
|
||||
-- (M47 ends 2011 / M94 starts 2012; M89 is aggregate only <= 2011 and a leaf
|
||||
-- from 2012 alongside M91-93), so no row is ever counted twice. Same argument
|
||||
-- the pipeline's recipe joins use.
|
||||
--
|
||||
-- `L--` IS excluded: it is the IG-to-state FAMILY TOTAL and genuinely rolls up
|
||||
-- the L-NN codes, so including it would double-count.
|
||||
CREATE OR REPLACE VIEW ig_long AS
|
||||
SELECT *
|
||||
FROM long
|
||||
WHERE LEFT(item_code, 1) IN ('M', 'L')
|
||||
AND item_code NOT LIKE '%--';
|
||||
@@ -0,0 +1,15 @@
|
||||
-- Harmonized-basis IG rows. Uses COALESCE(harmonized_code, item_code) rather
|
||||
-- than harmonized_code alone: aggregate rows carry NO harmonized_code by
|
||||
-- construction (harmonized space is leaf-only), so a plain
|
||||
-- `harmonized_code IS NOT NULL` filter would drop every legacy IG aggregate --
|
||||
-- in the bundled fixture corpus (year 2011; 2012+ all carry a harmonized_code)
|
||||
-- that is $379,016,063k across 25,688 M rows and $2,277,458k across 19,266 L
|
||||
-- rows (`SELECT year, LEFT(item_code,1), SUM(amt), COUNT(*) FROM ig_long
|
||||
-- WHERE harmonized_code IS NULL GROUP BY 1, 2`). COALESCE keeps the one real
|
||||
-- IG collapse rule (M38 -> M36, SB012, year-disjoint 1967-2011 vs 2012+)
|
||||
-- while never dropping a row.
|
||||
CREATE OR REPLACE VIEW ig_long_harmonized AS
|
||||
SELECT * REPLACE (COALESCE(harmonized_code, item_code) AS item_code)
|
||||
FROM long
|
||||
WHERE LEFT(item_code, 1) IN ('M', 'L')
|
||||
AND item_code NOT LIKE '%--';
|
||||
@@ -0,0 +1,16 @@
|
||||
CREATE OR REPLACE VIEW ig_annotated AS
|
||||
SELECT
|
||||
s.*,
|
||||
x.gov_name AS xwalk_gov_name,
|
||||
x.govs_type,
|
||||
x.type_label,
|
||||
x.fips_state AS xwalk_fips_state,
|
||||
x.fips_county AS xwalk_fips_county,
|
||||
x.fips_place,
|
||||
x.population_acs,
|
||||
c.category,
|
||||
c.category_type,
|
||||
c.spend_subtype
|
||||
FROM ig_long s
|
||||
LEFT JOIN canonical_fips_xwalk x USING (canonical_govid)
|
||||
LEFT JOIN summary_categories c USING (item_code);
|
||||
@@ -0,0 +1,16 @@
|
||||
CREATE OR REPLACE VIEW ig_annotated_harmonized AS
|
||||
SELECT
|
||||
s.*,
|
||||
x.gov_name AS xwalk_gov_name,
|
||||
x.govs_type,
|
||||
x.type_label,
|
||||
x.fips_state AS xwalk_fips_state,
|
||||
x.fips_county AS xwalk_fips_county,
|
||||
x.fips_place,
|
||||
x.population_acs,
|
||||
c.category,
|
||||
c.category_type,
|
||||
c.spend_subtype
|
||||
FROM ig_long_harmonized s
|
||||
LEFT JOIN canonical_fips_xwalk x USING (canonical_govid)
|
||||
LEFT JOIN summary_categories c USING (item_code);
|
||||
@@ -9,7 +9,8 @@ cog_geographic_rollup(
|
||||
category,
|
||||
years,
|
||||
per_capita = FALSE,
|
||||
adjust_to_year = NULL
|
||||
adjust_to_year = NULL,
|
||||
expenditure_concept = c("direct", "total")
|
||||
)
|
||||
}
|
||||
\arguments{
|
||||
@@ -27,6 +28,13 @@ population from `gov_population_yearly`. Govs with missing population
|
||||
are excluded from the result.}
|
||||
|
||||
\item{adjust_to_year}{Integer base year for CPI-U conversion, or `NULL`.}
|
||||
|
||||
\item{expenditure_concept}{`"direct"` (default) or `"total"`. Currently only
|
||||
`"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
|
||||
single-government queries but cannot be used here because combining Total
|
||||
across multiple layers of government double-counts intergovernmental
|
||||
transfers (a state's payment to a school district is the same dollar the
|
||||
district reports as its own Direct spending).}
|
||||
}
|
||||
\value{
|
||||
Tibble with columns `year`, `layer`, `canonical_govid`, `gov_name`,
|
||||
|
||||
@@ -32,8 +32,11 @@ the cross-vintage canonical-government registry. Operates in two modes:
|
||||
}
|
||||
\details{
|
||||
* **Utility mode** (single `name`, the original behavior): returns all
|
||||
rows whose `gov_name` matches the regex case-insensitively, sorted by
|
||||
`population_acs` descending. Useful for exploratory lookups.
|
||||
rows whose `gov_name` contains `name` as a **literal, case-insensitive
|
||||
substring**, sorted by `population_acs` descending. Useful for
|
||||
exploratory lookups. Regex metacharacters in `name` are escaped, so a
|
||||
government is findable by its own complete name even when that name
|
||||
contains parentheses or a period.
|
||||
* **Basket mode** (`length(name) > 1`): resolves each input row to a
|
||||
single canonical govid and returns a tibble in input order, suitable
|
||||
for piping straight into [cog_spending()] / [cog_revenue()] /
|
||||
@@ -45,7 +48,8 @@ the cross-vintage canonical-government registry. Operates in two modes:
|
||||
1. Filter `canonical_fips_xwalk` by `state` and (if non-NA) `type`.
|
||||
2. **Exact pass:** case-insensitive equality against `gov_name`.
|
||||
Single hit -> resolved. Multiple -> step 4.
|
||||
3. **Substring fallback:** case-insensitive regex against `gov_name`.
|
||||
3. **Substring fallback:** case-insensitive literal substring against
|
||||
`gov_name` (metacharacters escaped).
|
||||
Single hit -> resolved (`match_method = "substring"`). Zero hits ->
|
||||
`status = "no_match"`. Multiple hits -> step 4.
|
||||
4. **Disambiguation:** if matches share one `govs_type`, pick the
|
||||
@@ -58,7 +62,7 @@ inputs (`ambiguous` / `no_match`) appear only in the sidecar.
|
||||
}
|
||||
\examples{
|
||||
\dontrun{
|
||||
# Utility mode — exploratory regex lookup
|
||||
# Utility mode — exploratory substring lookup
|
||||
cog_gov_search("broward", state = "FL")
|
||||
|
||||
# Basket mode — resolve a known cohort
|
||||
|
||||
+36
-2
@@ -10,7 +10,8 @@ cog_peer_compare(
|
||||
category,
|
||||
years,
|
||||
per_capita = TRUE,
|
||||
adjust_to_year = NULL
|
||||
adjust_to_year = NULL,
|
||||
expenditure_concept = c("direct", "total")
|
||||
)
|
||||
}
|
||||
\arguments{
|
||||
@@ -27,6 +28,11 @@ cog_peer_compare(
|
||||
population.}
|
||||
|
||||
\item{adjust_to_year}{Integer base year for CPI-U conversion or `NULL`.}
|
||||
|
||||
\item{expenditure_concept}{`"direct"` (default) or `"total"`. Currently only
|
||||
`"direct"` is accepted; the `"total"` option exists in [cog_spending()] for
|
||||
single-government queries but cannot be used here because combining Total
|
||||
across peer sets counts intergovernmental transfers twice.}
|
||||
}
|
||||
\value{
|
||||
Tibble matching [cog_spending()]'s columns, plus a `role`
|
||||
@@ -37,11 +43,39 @@ Tibble matching [cog_spending()]'s columns, plus a `role`
|
||||
`attr(peers, "cohort_year")`; `NA` when `peers` was a bare character
|
||||
vector). Provenance reports `verb = "cog_peer_compare"`, `peer_count`,
|
||||
`cohort_year`, and `cohort_govids`.
|
||||
|
||||
**The `summary_*` rows are per-category quantiles: they are not additive.**
|
||||
Each one is computed **within each `(year, spend_subtype,
|
||||
category)` cell** across the peer set, so a `summary_p50` row is *the
|
||||
median peer's value in that one category*, not *the value of the median
|
||||
peer's total*. The median peer for Police and the median peer for Fire
|
||||
are usually different governments, so summing `summary_*` rows across
|
||||
categories does not give any peer's total and misstates the band it
|
||||
appears to describe — measured at −32.7% to +251.0% across 24 years on
|
||||
one cohort, with a sign flip at FY2012.
|
||||
|
||||
Facet by `role` **and** `category` (the documented use, and what the
|
||||
rows are built for). For a genuine "median peer's total spending" line,
|
||||
sum each peer's own categories first and take the quantile of those
|
||||
per-government totals:
|
||||
|
||||
```r
|
||||
library(dplyr)
|
||||
cmp |>
|
||||
filter(role %in% c("target", "peer")) |>
|
||||
group_by(year, role, canonical_govid) |>
|
||||
summarise(total = sum(amt_per_capita_real, na.rm = TRUE), .groups = "drop") |>
|
||||
filter(role == "peer") |>
|
||||
group_by(year) |>
|
||||
summarise(p50 = quantile(total, 0.5, na.rm = TRUE))
|
||||
```
|
||||
}
|
||||
\description{
|
||||
Pulls spending for the target plus a peer set (either a
|
||||
[cog_find_peers()] result or a character vector of `canonical_govid`) and
|
||||
appends peer-distribution summary rows (`summary_p25`, `summary_p50`,
|
||||
`summary_p75`) so the result can be faceted by `role` in a single ggplot
|
||||
call.
|
||||
call. Those summary rows are quantiles **within each category**, not
|
||||
quantiles of each peer's total — see the `@return` section before summing
|
||||
them.
|
||||
}
|
||||
|
||||
+31
-1
@@ -11,7 +11,8 @@ cog_spending(
|
||||
per_capita = FALSE,
|
||||
adjust_to_year = NULL,
|
||||
basis = c("harmonized", "raw"),
|
||||
recipe = NULL
|
||||
recipe = NULL,
|
||||
expenditure_concept = c("direct", "total")
|
||||
)
|
||||
}
|
||||
\arguments{
|
||||
@@ -55,6 +56,35 @@ argument is ignored and the result's provenance reports
|
||||
`basis = "recipe"` with an inert `harmonization` block (`applied =
|
||||
FALSE`, pointing at the `recipe` block instead) rather than a
|
||||
possibly-misleading `"harmonized"`/`"raw"` value.}
|
||||
|
||||
\item{expenditure_concept}{`"direct"` (default) returns only the
|
||||
government's own direct spending (item codes `E`/`F`/`G`), unchanged
|
||||
from prior releases. `"total"` additionally UNIONs in the
|
||||
intergovernmental leg -- payments to local governments (`M` codes) and
|
||||
to the state government (`L` codes, excluding the `L--` family-total
|
||||
rollup) -- so results gain rows with `spend_subtype ==
|
||||
"intergovernmental"`. Requires the active corpus's `summary_categories`
|
||||
to carry M/L rows (added by cog_pipeline PR #59); aborts with class
|
||||
`uscogdata_ig_categories_unsupported` on an older corpus rather than
|
||||
silently under-reporting. Mutually exclusive with `recipe` (a recipe
|
||||
already defines its own component codes). **Do not sum `"total"`
|
||||
results across levels of government** (e.g. state + county + city):
|
||||
a state's `M12` payment to a school district is the same dollar the
|
||||
district reports as its own direct `E12`, so summing both double-counts
|
||||
it. This matters in particular with [cog_geographic_rollup()], which
|
||||
sums across exactly that kind of multi-layer government set.
|
||||
|
||||
In the legacy wide era (<= FY2011), some functions are published ONLY
|
||||
as an aggregate-flagged family total (e.g. Corrections' `E04`/`E05`
|
||||
split), which the Direct leg excludes by construction but the IG leg
|
||||
deliberately keeps (see `inst/sql/24-ig_long.sql`). For a `"total"`
|
||||
query, any (year, category) where this leaves intergovernmental rows
|
||||
with NO Direct counterpart is flagged: the affected rows' `notes`
|
||||
name the harmonization recipe that recovers the missing Direct
|
||||
component (when one exists), and
|
||||
`provenance$expenditure_concept_direct_suppressed` is `TRUE` -- the
|
||||
figure in those rows is the intergovernmental leg alone, not Direct +
|
||||
IG.}
|
||||
}
|
||||
\value{
|
||||
Tibble with columns `year`, `canonical_govid`, `gov_name`,
|
||||
|
||||
@@ -6,6 +6,33 @@ fixture_corpus_path <- function() {
|
||||
if (nzchar(p)) paste0(p, "/") else ""
|
||||
}
|
||||
|
||||
# Path to a file in the SOURCE tree (README.md, man/*.Rd, vignettes/*.Rmd),
|
||||
# or "" when it isn't there.
|
||||
#
|
||||
# Tests that assert on documentation content have to read the sources, and the
|
||||
# sources only exist when the suite runs from a checkout. Under R CMD check the
|
||||
# suite runs from the INSTALLED package, where man/ and vignettes/ are not
|
||||
# shipped and `../../README.md` does not resolve -- so those tests must skip
|
||||
# rather than error. CI runs testthat::test_local() from the checkout BEFORE
|
||||
# rcmdcheck, so the assertions are still enforced on every push; this only
|
||||
# stops them from failing a context that structurally cannot satisfy them.
|
||||
source_tree_path <- function(...) {
|
||||
p <- testthat::test_path("..", "..", ...)
|
||||
if (file.exists(p)) p else ""
|
||||
}
|
||||
|
||||
# Skip unless every named source file is present (see source_tree_path()).
|
||||
skip_if_no_source_tree <- function(...) {
|
||||
paths <- vapply(list(...), function(rel) do.call(source_tree_path, as.list(rel)),
|
||||
character(1))
|
||||
missing <- vapply(paths, function(p) !nzchar(p), logical(1))
|
||||
testthat::skip_if(
|
||||
any(missing),
|
||||
"package source tree not available (running against the installed package)"
|
||||
)
|
||||
invisible(paths)
|
||||
}
|
||||
|
||||
# Skip a test if no corpus is reachable (bundled fixture or explicit remote URL).
|
||||
skip_if_no_corpus <- function() {
|
||||
p <- fixture_corpus_path()
|
||||
@@ -57,3 +84,37 @@ with_doctored_schema_version <- function(version, code) {
|
||||
}, add = TRUE)
|
||||
force(code)
|
||||
}
|
||||
|
||||
# Copy the bundled fixture to a temp dir with summary_categories.parquet
|
||||
# rewritten to drop every M/L (intergovernmental) row, then run `code`
|
||||
# against it with a clean session (mirrors with_fixture_corpus()/
|
||||
# with_doctored_schema_version()). Models a real pre-cog_pipeline-PR#59
|
||||
# corpus: the 66 M/L category rows shipped with NO schema_version bump (see
|
||||
# C2 in the expenditure-concept review), so schema_version is left
|
||||
# untouched here -- only the category data itself is rolled back.
|
||||
with_corpus_missing_ig_categories <- function(code) {
|
||||
src <- fixture_corpus_path()
|
||||
tmp <- withr::local_tempdir(.local_envir = parent.frame())
|
||||
file.copy(list.files(src, full.names = TRUE), tmp, recursive = TRUE)
|
||||
|
||||
cats_path <- file.path(tmp, "data", "summary_categories.parquet")
|
||||
filtered_path <- file.path(tmp, "data", "summary_categories_filtered.parquet")
|
||||
write_con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
|
||||
DBI::dbExecute(write_con, sprintf(
|
||||
"COPY (SELECT * FROM read_parquet(%s) WHERE LEFT(item_code, 1) NOT IN ('M', 'L'))
|
||||
TO %s (FORMAT PARQUET)",
|
||||
uscogdata:::.sql_lit_chr(cats_path), uscogdata:::.sql_lit_chr(filtered_path)
|
||||
))
|
||||
file.remove(cats_path)
|
||||
file.rename(filtered_path, cats_path)
|
||||
|
||||
old_url <- Sys.getenv("USCOGDATA_URL", unset = NA)
|
||||
uscogdata:::cog_close()
|
||||
Sys.setenv(USCOGDATA_URL = paste0(tmp, "/"))
|
||||
on.exit({
|
||||
uscogdata:::cog_close()
|
||||
if (is.na(old_url)) Sys.unsetenv("USCOGDATA_URL") else Sys.setenv(USCOGDATA_URL = old_url)
|
||||
}, add = TRUE)
|
||||
force(code)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Helper for the Madison-walkthrough finding tests (uscogdata #11-#16).
|
||||
#
|
||||
# Those tests all assert something about what a `cog_*` verb includes or
|
||||
# excludes. The expected amounts must therefore come from the RAW corpus, never
|
||||
# from the verb under test: verifying an absence through the filter that creates
|
||||
# it proves nothing. `wt_raw_*()` opens its own DuckDB connection straight onto
|
||||
# the corpus's `long` parquet partitions, bypassing uscogdata's SQL views (and
|
||||
# therefore its `flow_prefixes` filtering) entirely.
|
||||
|
||||
wt_corpus_glob <- function() {
|
||||
url <- Sys.getenv("USCOGDATA_URL")
|
||||
if (!nzchar(url)) testthat::skip("USCOGDATA_URL is not set")
|
||||
paste0(sub("/$", "", url), "/data/long/**/*.parquet")
|
||||
}
|
||||
|
||||
wt_raw_query <- function(sql) {
|
||||
con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
|
||||
DBI::dbGetQuery(con, sql)
|
||||
}
|
||||
|
||||
# Sum of `amt` (in $1,000s, as the corpus stores it) for one government-year,
|
||||
# restricted either to an explicit set of item codes or to a set of first-letter
|
||||
# prefixes. Aggregate rows are excluded, matching every published verb.
|
||||
wt_raw_amt <- function(govid, year, codes = NULL, prefixes = NULL) {
|
||||
stopifnot(xor(is.null(codes), is.null(prefixes)))
|
||||
filter_sql <- if (!is.null(codes)) {
|
||||
paste0("item_code IN (", paste0("'", codes, "'", collapse = ", "), ")")
|
||||
} else {
|
||||
paste0("LEFT(item_code, 1) IN (", paste0("'", prefixes, "'", collapse = ", "), ")")
|
||||
}
|
||||
out <- wt_raw_query(paste0(
|
||||
"SELECT COALESCE(SUM(amt), 0) AS amt FROM read_parquet('", wt_corpus_glob(), "') ",
|
||||
"WHERE canonical_govid = '", govid, "' AND year = ", year,
|
||||
" AND NOT is_aggregate AND ", filter_sql
|
||||
))
|
||||
out$amt[[1]]
|
||||
}
|
||||
|
||||
# The item codes a verb reports having summed, flattened out of the
|
||||
# comma-separated `codes_included` column.
|
||||
wt_codes_included <- function(df) {
|
||||
sort(unique(trimws(unlist(strsplit(stats::na.omit(df$codes_included), ",")))))
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
# Madison walkthrough audit -- finding F-004. Tracked as uscogdata#15.
|
||||
# See docs/walkthroughs/FINDINGS.md in cog_explorer.
|
||||
#
|
||||
# The raw Census files report thousands of dollars; this package multiplies by
|
||||
# 1000 and returns full US dollars. That is the friendlier choice and is not
|
||||
# wrong -- but cog_explorer's CLAUDE.md states "All raw `amt` values are in
|
||||
# $1,000s", so a reader who applies that rule to amt_nominal overstates every
|
||||
# figure by 1000x, and gets a plausible-looking number rather than an obvious
|
||||
# error. The audit rates this the highest-consequence definitional gap it found.
|
||||
#
|
||||
# Deliberately NOT asserted here: man/cog_spending.Rd and man/cog_revenue.Rd,
|
||||
# which ALREADY carry the statement in their @return sections (verified
|
||||
# 2026-07-29), as does cog-api's data-dictionary.md (since 2b71b41). The gap is
|
||||
# in the surfaces a reader meets first and in cog_explorer's own conventions
|
||||
# doc -- see uscogdata#15 for the full surface-by-surface table and for the two
|
||||
# secondary tasks (cog_explorer/CLAUDE.md, which has no git remote, and
|
||||
# cog-api's llms.txt, which is silent on units).
|
||||
|
||||
test_that("returned amounts are documented as full US dollars where readers meet the package", {
|
||||
|
||||
# README and vignettes ship only in the source tree, not in the installed
|
||||
# package, so these assertions cannot run under R CMD check -- CI's earlier
|
||||
# testthat::test_local() step is what enforces them. See
|
||||
# skip_if_no_source_tree() in helper-fixture.R.
|
||||
docs <- skip_if_no_source_tree(
|
||||
"README.md",
|
||||
c("vignettes", "total-spending.Rmd"),
|
||||
c("vignettes", "population-denominators.Rmd")
|
||||
)
|
||||
|
||||
says_units <- function(path) {
|
||||
txt <- paste(readLines(path, warn = FALSE), collapse = " ")
|
||||
grepl("full US dollars|full U\\.S\\. dollars", txt, ignore.case = TRUE) &&
|
||||
grepl("\\$1,000s|thousands of dollars", txt, ignore.case = TRUE)
|
||||
}
|
||||
|
||||
for (path in docs) expect_true(says_units(path))
|
||||
|
||||
# Pin the documented claim to the actual behaviour, so the two cannot drift.
|
||||
# The expected raw amount is read straight from the corpus's parquet
|
||||
# partitions -- never through cog_spending(), which is the thing being
|
||||
# described. Madison FY2020: E/F/G = 623,347 ($1,000s) -> $623,347,000.
|
||||
raw_thousands <- wt_raw_amt("552025209777", 2020L, prefixes = c("E", "F", "G"))
|
||||
expect_equal(raw_thousands, 623347)
|
||||
|
||||
returned <- cog_spending(govid = "552025209777", years = 2020L)
|
||||
expect_equal(sum(returned$amt_nominal), raw_thousands * 1000)
|
||||
|
||||
units <- attr(returned, "provenance")$transformations$units_conversion
|
||||
expect_true(units$applied)
|
||||
expect_equal(units$multiplier, 1000)
|
||||
})
|
||||
@@ -15,7 +15,22 @@ test_that("cog_categories(type = 'spending') returns only expenditure rows", {
|
||||
skip_if_no_corpus()
|
||||
r <- cog_categories(type = "spending")
|
||||
expect_true(all(r$category_type == "expenditure"))
|
||||
expect_true(all(r$subtype %in% c("operations", "capital")))
|
||||
# "assistance" (the J-prefix aid/benefit codes) joined the vocabulary with
|
||||
# the crosswalk completion in cog_pipeline#60/#65 -- every flow code
|
||||
# carrying dollars now maps to a category.
|
||||
expect_true(all(r$subtype %in%
|
||||
c("operations", "capital", "intergovernmental", "assistance")))
|
||||
})
|
||||
|
||||
test_that("cog_categories surfaces the intergovernmental spending subtype", {
|
||||
skip_if_no_corpus()
|
||||
r <- cog_categories(type = "spending")
|
||||
expect_true("intergovernmental" %in% r$subtype)
|
||||
# IG rows reuse the existing functional categories -- they add a subtype,
|
||||
# not new category values.
|
||||
ig_cats <- sort(unique(r$category[r$subtype == "intergovernmental"]))
|
||||
direct_cats <- sort(unique(r$category[r$subtype != "intergovernmental"]))
|
||||
expect_true(all(ig_cats %in% c(direct_cats, "Other Education")))
|
||||
})
|
||||
|
||||
test_that("cog_categories(type = 'revenue') returns only revenue rows", {
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# tests/testthat/test-corpus-breaks.R
|
||||
#
|
||||
# uscogdata#19. Four catalogued series breaks carry fin_code = "ALL" -- they
|
||||
# are caveats about the corpus itself rather than about one item code:
|
||||
#
|
||||
# SB085 1977 dollar precision across the 1976/1977 boundary
|
||||
# SB087 2002 imputation exclusion FY2002-2006
|
||||
# SB194 2012 dense -> sparse representation change
|
||||
# SB086 2017 government ID scheme change
|
||||
#
|
||||
# .build_series_break_refs() matches `fin_code IN (<codes in the result>)`,
|
||||
# and no row's item_code is ever the literal "ALL", so none of them could
|
||||
# ever reach a user. They now travel in their own provenance field,
|
||||
# `corpus_break_refs`, which keeps them distinguishable from the
|
||||
# code-specific `series_break_refs` (an ALL caveat qualifies the whole
|
||||
# result, not one series).
|
||||
|
||||
test_that("corpus_break_refs surfaces an ALL-scoped break the year range spans", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
# SB194 sits at FY2012 -- the dense/sparse boundary. A query spanning
|
||||
# 2011 -> 2012 straddles it, and this is the case cog_pipeline#64's
|
||||
# DoD 4 intended to reach users.
|
||||
r <- cog_spending("121011212191", 2011:2012, "Police")
|
||||
prov <- attr(r, "provenance")
|
||||
expect_true("SB194" %in% prov$corpus_break_refs)
|
||||
})
|
||||
})
|
||||
|
||||
test_that("corpus_break_refs stays empty when no ALL break falls in the range", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
# 2019-2020 spans no catalogued corpus-wide break.
|
||||
r <- cog_spending("121011212191", 2019:2020, "Police")
|
||||
expect_equal(attr(r, "provenance")$corpus_break_refs, character(0))
|
||||
})
|
||||
})
|
||||
|
||||
test_that("corpus_break_refs and series_break_refs stay disjoint", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
r <- cog_spending("121011212191", 2011:2012, "Police")
|
||||
prov <- attr(r, "provenance")
|
||||
expect_type(prov$series_break_refs, "character")
|
||||
expect_type(prov$corpus_break_refs, "character")
|
||||
# An ALL caveat must never masquerade as a break in a specific series.
|
||||
expect_length(intersect(prov$series_break_refs, prov$corpus_break_refs), 0L)
|
||||
expect_false("SB194" %in% prov$series_break_refs)
|
||||
})
|
||||
})
|
||||
|
||||
test_that(".build_corpus_break_refs matches on the break_year window alone", {
|
||||
skip_if_no_corpus()
|
||||
con <- cog_open()
|
||||
on.exit(cog_close())
|
||||
|
||||
# SB085's boundary is 1976/1977, outside the fixture's partitions -- the
|
||||
# series_breaks table is a full cross-vintage registry, so the matching
|
||||
# logic is testable there even though no long partition covers it.
|
||||
expect_true("SB085" %in% uscogdata:::.build_corpus_break_refs(
|
||||
con, years = 1975:1980, schema_version = 6L
|
||||
))
|
||||
# ... and does not fire for a range that misses it, unlike a filter keyed
|
||||
# on the era rather than the boundary.
|
||||
expect_false("SB085" %in% uscogdata:::.build_corpus_break_refs(
|
||||
con, years = 1978:1980, schema_version = 6L
|
||||
))
|
||||
|
||||
# Unlike code-specific refs, these do not depend on which codes a result
|
||||
# happens to contain -- that dependency is the whole defect.
|
||||
expect_setequal(
|
||||
uscogdata:::.build_corpus_break_refs(con, years = 2001:2003, schema_version = 6L),
|
||||
"SB087"
|
||||
)
|
||||
|
||||
# Gated on schema_version >= 5: series_breaks_pq is not registered below it.
|
||||
expect_equal(
|
||||
uscogdata:::.build_corpus_break_refs(con, years = 2011:2012, schema_version = 4L),
|
||||
character(0)
|
||||
)
|
||||
})
|
||||
|
||||
test_that("cog_explain() prints corpus-wide caveats under their own heading", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
r <- cog_spending("121011212191", 2011:2012, "Police")
|
||||
out <- paste(c(
|
||||
capture.output(cog_explain(r)),
|
||||
capture.output(cog_explain(r), type = "message")
|
||||
), collapse = "\n")
|
||||
expect_match(out, "Corpus-wide caveats", fixed = TRUE)
|
||||
expect_match(out, "SB194", fixed = TRUE)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,92 @@
|
||||
# Madison walkthrough audit -- findings F-020 and F-023. Tracked as uscogdata#13.
|
||||
# See docs/walkthroughs/FINDINGS.md in cog_explorer.
|
||||
#
|
||||
# The owner's settled design (2026-07-28): a `coverage` argument on
|
||||
# cog_geographic_rollup(), cog_find_peers()/cog_peer_compare() and their
|
||||
# cog-api equivalents --
|
||||
# "all" every unit that reported that year (today's behaviour, DEFAULT)
|
||||
# "census" census years only (years ending 2 or 7)
|
||||
# "consistent" only units reporting in every requested year (balanced panel)
|
||||
# -- PLUS always-on coverage metadata on every result regardless of mode:
|
||||
# n_units_reporting, n_units_expected, is_census_year.
|
||||
#
|
||||
# Motivating principle: using these verbs correctly must not require the user to
|
||||
# know that the Census of Governments is a complete census only in years ending
|
||||
# in 2 and 7.
|
||||
#
|
||||
# The helper below accepts that metadata either as columns on the returned
|
||||
# tibble or as a per-year table in provenance$coverage -- the design fixes the
|
||||
# three field names and that they reach the caller, not the container.
|
||||
|
||||
wt_coverage <- function(x) {
|
||||
prov <- attr(x, "provenance")
|
||||
cov <- prov$coverage
|
||||
if (is.null(cov)) {
|
||||
needed <- c("year", "n_units_reporting", "n_units_expected", "is_census_year")
|
||||
expect_true(all(needed %in% names(x)))
|
||||
cov <- unique(x[, needed])
|
||||
}
|
||||
cov[order(cov$year), ]
|
||||
}
|
||||
|
||||
test_that("multi-government aggregates disclose reporting coverage on every result", {
|
||||
testthat::skip("Blocked on uscogdata#13 (findings F-020, F-023)")
|
||||
|
||||
# -- F-020: geographic rollups -------------------------------------------
|
||||
# Wisconsin's city/village universe is 608 governments. On the bundled
|
||||
# fixture, FY2012 (a census year) has 597 of them reporting while FY2019 and
|
||||
# FY2020 (sample years) have 112 and 114 -- an 18%-98% swing that today's
|
||||
# return value says nothing about. Counts cross-checked against the raw
|
||||
# corpus, not through cog_geographic_rollup(), which is under test.
|
||||
wi <- cog_gov_search(name = NULL, state = "WI", type = "city")
|
||||
expect_equal(nrow(wi), 608L)
|
||||
|
||||
roll <- cog_geographic_rollup(govids = list(city = wi$canonical_govid),
|
||||
category = NULL, years = c(2011L, 2012L, 2019L, 2020L))
|
||||
cov <- wt_coverage(roll)
|
||||
|
||||
expect_equal(cov$n_units_expected, rep(608L, 4L))
|
||||
expect_equal(cov$n_units_reporting, c(152L, 597L, 112L, 114L))
|
||||
expect_equal(cov$is_census_year, c(FALSE, TRUE, FALSE, FALSE))
|
||||
|
||||
raw_2012 <- wt_raw_query(paste0(
|
||||
"SELECT COUNT(DISTINCT canonical_govid) n FROM read_parquet('", wt_corpus_glob(), "') ",
|
||||
"WHERE type = 2 AND fips_state = 55 AND year = 2012 ",
|
||||
"AND LEFT(item_code, 1) IN ('E','F','G') AND NOT is_aggregate"))
|
||||
expect_equal(cov$n_units_reporting[cov$year == 2012], as.integer(raw_2012$n[[1]]))
|
||||
|
||||
# -- F-023: peer cohorts --------------------------------------------------
|
||||
# CHILTON CITY, WI (ACS population 4,017): a 15-peer cohort fixed at FY2012
|
||||
# reports 15 of 15 in FY2012 and only 3 of 15 in FY2019 and FY2020. Nothing
|
||||
# in cog_peer_compare()'s return distinguishes those years today.
|
||||
chilton <- "552015177095"
|
||||
peers <- cog_find_peers(chilton, year = 2012L, max_peers = 15L)
|
||||
expect_equal(nrow(peers), 15L)
|
||||
|
||||
cmp <- cog_peer_compare(target_govid = chilton, peers = peers, category = NULL,
|
||||
years = c(2012L, 2019L, 2020L), per_capita = TRUE)
|
||||
cov_peers <- wt_coverage(cmp)
|
||||
expect_equal(cov_peers$n_units_expected, rep(15L, 3L))
|
||||
expect_equal(cov_peers$n_units_reporting, c(15L, 3L, 3L))
|
||||
expect_equal(cov_peers$is_census_year, c(TRUE, FALSE, FALSE))
|
||||
|
||||
# -- the three coverage modes --------------------------------------------
|
||||
expect_equal(attr(cog_peer_compare(target_govid = chilton, peers = peers,
|
||||
category = NULL, years = c(2012L, 2019L, 2020L),
|
||||
per_capita = TRUE),
|
||||
"provenance")$coverage_mode, "all") # unchanged default
|
||||
|
||||
consistent <- cog_peer_compare(target_govid = chilton, peers = peers,
|
||||
category = NULL, years = c(2012L, 2019L, 2020L),
|
||||
per_capita = TRUE, coverage = "consistent")
|
||||
n_by_year <- tapply(consistent$canonical_govid[consistent$role == "peer"],
|
||||
consistent$year[consistent$role == "peer"],
|
||||
function(g) length(unique(g)))
|
||||
expect_equal(unname(as.integer(n_by_year)), c(3L, 3L, 3L)) # balanced panel
|
||||
|
||||
census_only <- cog_geographic_rollup(govids = list(city = wi$canonical_govid),
|
||||
category = NULL,
|
||||
years = c(2011L, 2012L, 2019L, 2020L),
|
||||
coverage = "census")
|
||||
expect_equal(sort(unique(census_only$year)), 2012)
|
||||
})
|
||||
@@ -0,0 +1,544 @@
|
||||
test_that("the corpus contains no K-prefix rows, so the Direct leg omits K", {
|
||||
con <- .ensure_session()
|
||||
n <- DBI::dbGetQuery(con,
|
||||
"SELECT COUNT(*) AS n FROM long WHERE LEFT(item_code, 1) = 'K'")$n
|
||||
expect_equal(n, 0)
|
||||
|
||||
sql_files <- c("20-spending_long.sql", "22-spending_long_harmonized.sql")
|
||||
for (f in sql_files) {
|
||||
txt <- paste(readLines(system.file("sql", f, package = "uscogdata")),
|
||||
collapse = " ")
|
||||
expect_false(grepl("'K'", txt, fixed = TRUE),
|
||||
label = paste(f, "must not reference the inert K prefix"))
|
||||
}
|
||||
})
|
||||
|
||||
test_that("expenditure_concept defaults to direct and preserves today's numbers", {
|
||||
gov <- "010000226085" # Alabama state government
|
||||
base <- cog_spending(gov, years = 2019, category = "Police")
|
||||
expl <- cog_spending(gov, years = 2019, category = "Police",
|
||||
expenditure_concept = "direct")
|
||||
expect_equal(base$amt_nominal, expl$amt_nominal)
|
||||
expect_false("intergovernmental" %in% base$spend_subtype)
|
||||
})
|
||||
|
||||
test_that("expenditure_concept = 'total' adds an intergovernmental subtype", {
|
||||
gov <- "010000226085"
|
||||
d <- cog_spending(gov, years = 2019, category = "Police",
|
||||
expenditure_concept = "direct")
|
||||
t <- cog_spending(gov, years = 2019, category = "Police",
|
||||
expenditure_concept = "total")
|
||||
expect_true("intergovernmental" %in% t$spend_subtype)
|
||||
# Direct rows are untouched; Total only ever ADDS. Use %in% rather than
|
||||
# != : a category = NULL result can contain a NULL-subtype group (codes
|
||||
# with no summary_categories row, e.g. E16/E21/E85/F16/F85/G16/G21/G85),
|
||||
# and `NA != "intergovernmental"` is NA, not TRUE, which would silently
|
||||
# smuggle an all-NA phantom row into dt.
|
||||
dt <- t[!(t$spend_subtype %in% "intergovernmental"), ]
|
||||
expect_equal(sort(dt$amt_nominal), sort(d$amt_nominal))
|
||||
expect_gt(sum(t$amt_nominal), sum(d$amt_nominal))
|
||||
})
|
||||
|
||||
test_that("legacy-era Total does not collapse to Direct (the is_aggregate trap)", {
|
||||
# In the wide era the IG dollars live almost entirely on aggregate-flagged
|
||||
# rows. A Total leg that inherited the Direct leg's NOT is_aggregate filter
|
||||
# would silently return Total == Direct here.
|
||||
gov <- "010000226085"
|
||||
d <- cog_spending(gov, years = 2011, category = "Education K-12",
|
||||
expenditure_concept = "direct")
|
||||
t <- cog_spending(gov, years = 2011, category = "Education K-12",
|
||||
expenditure_concept = "total")
|
||||
expect_true("intergovernmental" %in% t$spend_subtype)
|
||||
ig <- sum(t$amt_nominal[t$spend_subtype == "intergovernmental"])
|
||||
expect_gt(ig, 0)
|
||||
expect_gt(sum(t$amt_nominal), sum(d$amt_nominal))
|
||||
})
|
||||
|
||||
test_that("the IG leg never includes the L-- family total", {
|
||||
con <- .ensure_session()
|
||||
codes <- DBI::dbGetQuery(con,
|
||||
"SELECT DISTINCT item_code FROM ig_long")$item_code
|
||||
expect_false(any(grepl("--$", codes)))
|
||||
expect_true(all(substr(codes, 1, 1) %in% c("M", "L")))
|
||||
})
|
||||
|
||||
test_that("expenditure_concept rejects unknown values", {
|
||||
expect_error(
|
||||
cog_spending("010000226085", years = 2019, expenditure_concept = "gross"),
|
||||
class = "rlang_error"
|
||||
)
|
||||
})
|
||||
|
||||
test_that("total composes with basis = 'raw' and basis = 'harmonized'", {
|
||||
gov <- "010000226085"
|
||||
h <- cog_spending(gov, years = 2011, category = "Education K-12",
|
||||
expenditure_concept = "total", basis = "harmonized")
|
||||
r <- cog_spending(gov, years = 2011, category = "Education K-12",
|
||||
expenditure_concept = "total", basis = "raw")
|
||||
ig_h <- sum(h$amt_nominal[h$spend_subtype == "intergovernmental"])
|
||||
ig_r <- sum(r$amt_nominal[r$spend_subtype == "intergovernmental"])
|
||||
# The only IG harmonization rule is M38 -> M36 (year-disjoint), so the IG
|
||||
# total must agree between bases even though the code labels may differ.
|
||||
expect_equal(ig_h, ig_r)
|
||||
})
|
||||
|
||||
test_that("recipe = and expenditure_concept = 'total' together aborts", {
|
||||
expect_error(
|
||||
cog_spending("121011212191", 2020L, recipe = "corrections_combined",
|
||||
expenditure_concept = "total"),
|
||||
class = "uscogdata_recipe_concept_conflict"
|
||||
)
|
||||
})
|
||||
|
||||
test_that("aggregate-sourced IG dollars are flagged aggregate_fallback = TRUE (bool_or, not bool_and)", {
|
||||
# Regression test: .build_verb_sql() originally used bool_and(is_aggregate)
|
||||
# for aggregate_fallback, which is correct for the Direct leg (a group can
|
||||
# never mix aggregate and non-aggregate rows there -- spending_long filters
|
||||
# NOT is_aggregate) but wrong for the IG leg. The wide era is dense -- every
|
||||
# government has a $0 row for every code in a family -- so a $0 leaf sits in
|
||||
# the same (year, gov, subtype, category) group as the real aggregate row
|
||||
# and flips bool_and() to FALSE. Measured: AL state 2011 had $5,740,775,000
|
||||
# of aggregate-sourced IG dollars (Corrections $31,358,000 + Education K-12
|
||||
# $5,152,385,000 + General Government $557,032,000) reporting
|
||||
# aggregate_fallback = FALSE under bool_and(), with the only TRUE row being
|
||||
# Transit Utilities at $0. bool_or() reports all of them correctly.
|
||||
gov <- "010000226085"
|
||||
t <- cog_spending(gov, years = 2011, category = "Education K-12",
|
||||
expenditure_concept = "total")
|
||||
ig <- t[t$spend_subtype == "intergovernmental", ]
|
||||
expect_equal(nrow(ig), 1L)
|
||||
expect_true(ig$aggregate_fallback)
|
||||
expect_true(nzchar(ig$notes))
|
||||
expect_match(ig$notes, "Aggregate fallback applied", fixed = TRUE)
|
||||
})
|
||||
|
||||
test_that("legacy aggregate IG codes are year-disjoint from their modern leaf components", {
|
||||
# The safety of ig_long's deliberate omission of `NOT is_aggregate` (see
|
||||
# inst/sql/24-ig_long.sql) rests entirely on each legacy code's AGGREGATE
|
||||
# instance being year-disjoint from the modern leaf codes it rolls up --
|
||||
# if a future corpus rebuild ever back-filled a leaf into a year where the
|
||||
# code is still flagged aggregate, `total` would silently double-count and
|
||||
# this suite would still pass. This test fails loudly if that ever
|
||||
# happens.
|
||||
#
|
||||
# Note the invariant is scoped to the AGGREGATE flag, not bare code
|
||||
# presence: M89/L89 do NOT disappear after the wide era the way M47/L47
|
||||
# do -- they continue past 2011 as their OWN independent leaf line item
|
||||
# (is_aggregate = FALSE) alongside M91-93/L91-93, which is fine because a
|
||||
# non-aggregate M89/L89 no longer represents a rollup of those codes.
|
||||
# (Verified in the fixture: M89/L89 are is_aggregate = TRUE only in 2011,
|
||||
# when M91-93/L91-93 don't exist yet; from 2012 on M89/L89 are
|
||||
# is_aggregate = FALSE leaves coexisting with M91-93/L91-93.)
|
||||
#
|
||||
# Pairs are the M/L-prefixed components (this package's ig_long only
|
||||
# covers M/L; other prefixes in the same rollup, e.g. N/O/P/Q/R, fall
|
||||
# outside its domain and are irrelevant here) enumerated in
|
||||
# cog_pipeline's data/wide_to_long_xwalk.csv `full_desc` column (read
|
||||
# once at authoring time, not at test time -- this test stays offline):
|
||||
# M47 "To local governments, total (includes N47, O47, P47, R47, and M94)"
|
||||
# M89 "To local governments, total (incl N89, O89, P89, R89, M91, M92, and M93)"
|
||||
# L47 "To state government (includes L94)"
|
||||
# L89 "To state government (includes L91, L92, and L93)"
|
||||
con <- .ensure_session()
|
||||
pairs <- list(
|
||||
list(aggregate = "M47", components = "M94"),
|
||||
list(aggregate = "M89", components = c("M91", "M92", "M93")),
|
||||
list(aggregate = "L47", components = "L94"),
|
||||
list(aggregate = "L89", components = c("L91", "L92", "L93"))
|
||||
)
|
||||
agg_years_by_code <- DBI::dbGetQuery(con,
|
||||
"SELECT DISTINCT year, item_code FROM ig_long WHERE is_aggregate")
|
||||
codes_by_year <- DBI::dbGetQuery(con, "SELECT DISTINCT year, item_code FROM ig_long")
|
||||
|
||||
for (p in pairs) {
|
||||
agg_years <- agg_years_by_code$year[agg_years_by_code$item_code == p$aggregate]
|
||||
for (yr in agg_years) {
|
||||
codes_yr <- codes_by_year$item_code[codes_by_year$year == yr]
|
||||
has_component <- any(p$components %in% codes_yr)
|
||||
expect_false(
|
||||
has_component,
|
||||
label = sprintf(
|
||||
"year %s has aggregate-flagged %s co-occurring with a modern component (%s)",
|
||||
yr, p$aggregate, paste(p$components, collapse = ",")
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test_that(".verb_spendrev rejects expenditure_concept = 'total' for a non-spending view_base", {
|
||||
# cog_revenue() never exposes expenditure_concept and always resolves it
|
||||
# to the "direct" default, so there is no revenue codepath that reaches
|
||||
# this today -- but .verb_spendrev() is shared, and nothing else stops a
|
||||
# future caller from passing expenditure_concept = "total" alongside
|
||||
# view_base = "revenue_annotated", which would UNION expenditure M/L rows
|
||||
# into a revenue result. Exercise the internal helper directly.
|
||||
expect_error(
|
||||
uscogdata:::.verb_spendrev(
|
||||
verb = "cog_revenue_test", view_base = "revenue_annotated",
|
||||
subtype_col = "revenue_subtype",
|
||||
flow_prefixes = c("T", "A", "U", "B", "C", "D"),
|
||||
call = quote(cog_revenue_test()),
|
||||
govid = "010000226085", years = 2019L, category = NULL,
|
||||
per_capita = FALSE, adjust_to_year = NULL, basis = "raw",
|
||||
recipe = NULL, expenditure_concept = "total"
|
||||
),
|
||||
class = "uscogdata_expenditure_concept_unsupported"
|
||||
)
|
||||
})
|
||||
|
||||
test_that("cog_geographic_rollup refuses expenditure_concept = 'total'", {
|
||||
expect_error(
|
||||
cog_geographic_rollup(
|
||||
govids = list(state = "010000226085"),
|
||||
category = "Police", years = 2019,
|
||||
expenditure_concept = "total"
|
||||
),
|
||||
class = "uscogdata_concept_not_aggregatable"
|
||||
)
|
||||
})
|
||||
|
||||
test_that("cog_peer_compare refuses expenditure_concept = 'total'", {
|
||||
expect_error(
|
||||
cog_peer_compare(
|
||||
target_govid = "010000226085", peers = "010000226085",
|
||||
category = "Police", years = 2019,
|
||||
expenditure_concept = "total"
|
||||
),
|
||||
class = "uscogdata_concept_not_aggregatable"
|
||||
)
|
||||
})
|
||||
|
||||
test_that("the refusal message names the fix and the reason", {
|
||||
err <- tryCatch(
|
||||
cog_geographic_rollup(govids = list(state = "010000226085"),
|
||||
category = "Police", years = 2019,
|
||||
expenditure_concept = "total"),
|
||||
condition = function(e) e
|
||||
)
|
||||
msg <- paste(conditionMessage(err), collapse = " ")
|
||||
expect_match(msg, "direct")
|
||||
expect_match(msg, "double-count|double count")
|
||||
expect_match(msg, "cog_geographic_rollup")
|
||||
|
||||
# Test that cog_peer_compare's message names its own function
|
||||
err2 <- tryCatch(
|
||||
cog_peer_compare(target_govid = "010000226085", peers = "010000226085",
|
||||
category = "Police", years = 2019,
|
||||
expenditure_concept = "total"),
|
||||
condition = function(e) e
|
||||
)
|
||||
msg2 <- paste(conditionMessage(err2), collapse = " ")
|
||||
expect_match(msg2, "direct")
|
||||
expect_match(msg2, "double-count|double count")
|
||||
expect_match(msg2, "cog_peer_compare")
|
||||
})
|
||||
|
||||
test_that("both cross-government verbs still accept the direct default", {
|
||||
expect_no_error(
|
||||
cog_geographic_rollup(govids = list(state = "010000226085"),
|
||||
category = "Police", years = 2019)
|
||||
)
|
||||
expect_no_error(
|
||||
cog_peer_compare(target_govid = "010000226085", peers = "010000226085",
|
||||
category = "Police", years = 2019)
|
||||
)
|
||||
})
|
||||
|
||||
test_that("provenance always records the expenditure concept", {
|
||||
d <- cog_spending("010000226085", years = 2019, category = "Police")
|
||||
t <- cog_spending("010000226085", years = 2019, category = "Police",
|
||||
expenditure_concept = "total")
|
||||
expect_equal(attr(d, "provenance")$expenditure_concept, "direct")
|
||||
expect_equal(attr(t, "provenance")$expenditure_concept, "total")
|
||||
# The note explains the non-obvious part: how legacy IG was assembled.
|
||||
expect_true(nzchar(attr(t, "provenance")$expenditure_concept_note))
|
||||
expect_true(is.na(attr(d, "provenance")$expenditure_concept_note) ||
|
||||
!nzchar(attr(d, "provenance")$expenditure_concept_note))
|
||||
})
|
||||
|
||||
test_that("the provenance schema documents expenditure_concept", {
|
||||
sch <- jsonlite::fromJSON(
|
||||
system.file("schemas", "provenance-v1.json", package = "uscogdata"),
|
||||
simplifyVector = FALSE
|
||||
)
|
||||
expect_true("expenditure_concept" %in% names(sch$properties))
|
||||
})
|
||||
|
||||
test_that("a firing suggestion names the intergovernmental counterpart recipe", {
|
||||
# Corrections has no legacy leaf rows, so the coverage-gap suggestion fires;
|
||||
# corrections_ig_local_combined is its IG counterpart.
|
||||
r <- suppressMessages(
|
||||
cog_spending("010000226085", years = c(2005, 2011), category = "Corrections")
|
||||
)
|
||||
sugg <- attr(r, "provenance")$suggestions
|
||||
expect_gt(length(sugg), 0L)
|
||||
ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
|
||||
expect_true("corrections_combined" %in% ids)
|
||||
ig <- unlist(lapply(sugg, function(s) s$ig_recipe_id))
|
||||
expect_true("corrections_ig_local_combined" %in% ig)
|
||||
})
|
||||
|
||||
test_that("no suggestion fires for a healthy query", {
|
||||
r <- cog_spending("010000226085", years = 2019, category = "Police")
|
||||
expect_length(attr(r, "provenance")$suggestions, 0L)
|
||||
})
|
||||
|
||||
test_that("a mis-scoped cog_spending() call never attaches an M/L counterpart to a revenue-flavored recipe", {
|
||||
# "IG Federal" is a revenue-only category (summary_categories maps it to
|
||||
# B-prefixed component codes only; its recipes are ig_federal_b47_wide /
|
||||
# ig_federal_b89_wide). A cog_spending() call scoped to it returns zero
|
||||
# spending rows for every requested year -- there is no spending
|
||||
# component in this category at all -- so the coverage-gap machinery
|
||||
# fires for real (not hypothetically) even though this isn't the kind of
|
||||
# format-boundary gap the recipe catalog is meant to signpost. This is
|
||||
# exactly the live-corpus risk flagged in review: ig_federal_b47_wide's
|
||||
# own component codes (B47/B94, suffixes {"47","94"}) are an EXACT
|
||||
# suffix-set match for the expenditure recipe ige_local_m47_wide
|
||||
# (M47/M94, same suffixes) -- a coincidence of reused digits, not a real
|
||||
# Direct/Total pairing. The flow-family gate in
|
||||
# .attach_ig_counterparts() must keep ig_recipe_id NULL here.
|
||||
#
|
||||
# Anchored on FL state government, not AL. Coverage is presence-based: a
|
||||
# recipe is only suggested when its component codes have rows for the
|
||||
# requested government-year. AL state's only FY2011 B47 cell was an
|
||||
# explicit zero, which the corpus no longer stores after sparsification
|
||||
# (SB194, cog_pipeline#64), so the recipe stopped being a candidate there.
|
||||
# FL state carries a real FY2011 B47 amount, so this exercises the guard
|
||||
# against a suggestion that genuinely fires.
|
||||
r <- suppressMessages(
|
||||
cog_spending("120000226351", years = c(2005, 2011), category = "IG Federal")
|
||||
)
|
||||
sugg <- attr(r, "provenance")$suggestions
|
||||
expect_gt(length(sugg), 0L)
|
||||
ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
|
||||
expect_true("ig_federal_b47_wide" %in% ids)
|
||||
ig <- unlist(lapply(sugg, function(s) s$ig_recipe_id))
|
||||
expect_length(ig, 0L)
|
||||
})
|
||||
|
||||
test_that("C1: 'total' on a legacy aggregate-only family reports the IG-only figure honestly, not as Direct + IG", {
|
||||
# AL state government, Corrections, 2011. Measured pre-fix: 'total'
|
||||
# returned $31,358,000 (the IG leg alone, on an aggregate-flagged M04/M05
|
||||
# row) with 0 suggestions (the surviving IG row made the gap-detection
|
||||
# machinery think the Direct leg was covered) and a note asserting
|
||||
# "Total = Direct + intergovernmental" with no caveat. True Direct (via
|
||||
# recipe = "corrections_combined") is $521,651,000 -- the IG-only figure
|
||||
# is ~6% of it.
|
||||
gov <- "010000226085"
|
||||
|
||||
d <- cog_spending(gov, years = 2011, category = "Corrections",
|
||||
expenditure_concept = "direct")
|
||||
expect_equal(nrow(d), 0L)
|
||||
|
||||
t <- suppressMessages(cog_spending(
|
||||
gov, years = 2011, category = "Corrections", expenditure_concept = "total"
|
||||
))
|
||||
expect_equal(nrow(t), 1L)
|
||||
expect_equal(t$spend_subtype, "intergovernmental")
|
||||
expect_equal(t$amt_nominal, 31358000)
|
||||
|
||||
r <- cog_spending(gov, years = 2011, recipe = "corrections_combined")
|
||||
expect_equal(r$amt_nominal, 521651000)
|
||||
|
||||
# C1(a): the recipe hints must fire for "total" exactly as they do for
|
||||
# "direct" -- the surviving IG row must not be mistaken for Direct
|
||||
# coverage.
|
||||
prov <- attr(t, "provenance")
|
||||
expect_gt(length(prov$suggestions), 0L)
|
||||
ids <- vapply(prov$suggestions, function(s) s$recipe_id %||% "", character(1))
|
||||
expect_true("corrections_combined" %in% ids)
|
||||
|
||||
# C1(b): the affected row's notes name a recovering recipe rather than
|
||||
# staying silent, and the provenance carries a flag a downstream consumer
|
||||
# (e.g. cog-api, which passes provenance through verbatim) can test.
|
||||
expect_true(nzchar(t$notes))
|
||||
expect_match(t$notes, "unavailable", fixed = TRUE)
|
||||
expect_match(t$notes, "corrections_combined", fixed = TRUE)
|
||||
expect_true(prov$expenditure_concept_direct_suppressed)
|
||||
|
||||
# The base "Total = Direct + IG" note must NOT stand unqualified when that
|
||||
# arithmetic didn't actually happen for this row.
|
||||
expect_match(prov$expenditure_concept_note, "NOTE", fixed = TRUE)
|
||||
expect_match(prov$expenditure_concept_note,
|
||||
"expenditure_concept_direct_suppressed", fixed = TRUE)
|
||||
})
|
||||
|
||||
test_that("C1(b): expenditure_concept_direct_suppressed is FALSE when the Direct leg is present", {
|
||||
d <- cog_spending("010000226085", years = 2019, category = "Police",
|
||||
expenditure_concept = "direct")
|
||||
t <- cog_spending("010000226085", years = 2019, category = "Police",
|
||||
expenditure_concept = "total")
|
||||
expect_false(isTRUE(attr(d, "provenance")$expenditure_concept_direct_suppressed))
|
||||
expect_false(isTRUE(attr(t, "provenance")$expenditure_concept_direct_suppressed))
|
||||
expect_false(any(nzchar(t$notes[t$spend_subtype == "intergovernmental"]) &
|
||||
grepl("unavailable", t$notes[t$spend_subtype == "intergovernmental"])))
|
||||
})
|
||||
|
||||
# M/I fix: .detect_direct_suppressed() was equating "no Direct sibling row"
|
||||
# with "Direct was suppressed", but the dominant real cause is a government
|
||||
# that simply has no direct spending in that category -- correct, ordinary
|
||||
# data. The fix gates the flag (and its row note) on a harmonization recipe
|
||||
# ACTUALLY covering that exact (year, canonical_govid, category) triple.
|
||||
|
||||
test_that("M/I: true positive, category supplied explicitly (unchanged behavior)", {
|
||||
al <- "010000226085"
|
||||
t_cat <- suppressMessages(cog_spending(
|
||||
al, years = 2011, category = "Corrections", expenditure_concept = "total"
|
||||
))
|
||||
expect_true(attr(t_cat, "provenance")$expenditure_concept_direct_suppressed)
|
||||
expect_match(t_cat$notes, "corrections_combined", fixed = TRUE)
|
||||
expect_match(t_cat$notes, "unavailable", fixed = TRUE)
|
||||
})
|
||||
|
||||
test_that("M/I: true positive, category = NULL now also names the recipe (was the fallback bug)", {
|
||||
# Root bug: .build_suggestions() short-circuits to list() when category is
|
||||
# NULL, so the note previously always hit its "no covering recipe found"
|
||||
# fallback here even though corrections_combined genuinely covers this row.
|
||||
al <- "010000226085"
|
||||
t_null <- suppressMessages(cog_spending(
|
||||
al, years = 2011, category = NULL, expenditure_concept = "total"
|
||||
))
|
||||
corr_row <- t_null[t_null$category %in% "Corrections", ]
|
||||
expect_equal(nrow(corr_row), 1L)
|
||||
expect_true(attr(t_null, "provenance")$expenditure_concept_direct_suppressed)
|
||||
expect_match(corr_row$notes, "corrections_combined", fixed = TRUE)
|
||||
expect_match(corr_row$notes, "unavailable", fixed = TRUE)
|
||||
expect_false(grepl("no covering recipe found", corr_row$notes, fixed = TRUE))
|
||||
})
|
||||
|
||||
test_that("M/I: false positive -- Virginia Education K-12 FY2019 total is NOT flagged", {
|
||||
# States fund K-12 through school districts, so the Direct leg (E12/F12/
|
||||
# G12) is genuinely, correctly zero -- not suppressed. Must not be flagged
|
||||
# and must carry no suppression note.
|
||||
va <- "510000227542"
|
||||
t_va <- suppressMessages(cog_spending(
|
||||
va, years = 2019, category = "Education K-12", expenditure_concept = "total"
|
||||
))
|
||||
expect_equal(nrow(t_va), 1L)
|
||||
expect_equal(t_va$spend_subtype, "intergovernmental")
|
||||
expect_equal(t_va$amt_nominal, 8028179000)
|
||||
expect_false(isTRUE(attr(t_va, "provenance")$expenditure_concept_direct_suppressed))
|
||||
expect_false(nzchar(t_va$notes) && grepl("unavailable", t_va$notes))
|
||||
})
|
||||
|
||||
test_that("M/I: false positive by construction -- 'Other Education' has no E/F/G code, never flagged", {
|
||||
# "Other Education" maps only to M21/L21 in summary_categories -- there is
|
||||
# no E/F/G code for it in this corpus at all, so no Direct-recovering
|
||||
# recipe can exist and it must never be flagged, in any fixture year.
|
||||
con <- uscogdata:::.ensure_session()
|
||||
years_all <- DBI::dbGetQuery(con, "SELECT DISTINCT year FROM long ORDER BY year")$year
|
||||
states <- DBI::dbGetQuery(con,
|
||||
"SELECT DISTINCT canonical_govid FROM long WHERE type = 0")$canonical_govid
|
||||
oe <- suppressMessages(cog_spending(
|
||||
states, years = years_all, category = "Other Education",
|
||||
expenditure_concept = "total"
|
||||
))
|
||||
expect_false(isTRUE(attr(oe, "provenance")$expenditure_concept_direct_suppressed))
|
||||
expect_false(any(nzchar(oe$notes) & grepl("unavailable", oe$notes)))
|
||||
})
|
||||
|
||||
test_that("M/I: a clean FY2019 category = NULL total query flags far fewer than the pre-fix 32/50 states", {
|
||||
con <- uscogdata:::.ensure_session()
|
||||
states <- DBI::dbGetQuery(con,
|
||||
"SELECT DISTINCT canonical_govid FROM long WHERE type = 0")$canonical_govid
|
||||
r <- suppressMessages(cog_spending(
|
||||
states, years = 2019, category = NULL, expenditure_concept = "total"
|
||||
))
|
||||
ig <- r[r$spend_subtype == "intergovernmental", ]
|
||||
flagged <- ig[nzchar(ig$notes) & grepl("unavailable", ig$notes), ]
|
||||
expect_lt(length(unique(flagged$canonical_govid)), 32L)
|
||||
# Every remaining flagged row must actually name a covering recipe --
|
||||
# never the old no-recipe-found fallback.
|
||||
expect_true(all(grepl("recipe = '", flagged$notes, fixed = TRUE)))
|
||||
expect_false(any(grepl("no covering recipe found", flagged$notes, fixed = TRUE)))
|
||||
})
|
||||
|
||||
test_that("C2: expenditure_concept = 'total' aborts on a corpus with no intergovernmental category rows", {
|
||||
with_corpus_missing_ig_categories({
|
||||
con <- uscogdata:::.ensure_session()
|
||||
n <- DBI::dbGetQuery(con,
|
||||
"SELECT COUNT(*) AS n FROM summary_categories WHERE LEFT(item_code, 1) IN ('M', 'L')"
|
||||
)$n
|
||||
expect_equal(n, 0)
|
||||
|
||||
err <- tryCatch(
|
||||
cog_spending("010000226085", years = 2019, category = "Police",
|
||||
expenditure_concept = "total"),
|
||||
condition = function(e) e
|
||||
)
|
||||
expect_s3_class(err, "uscogdata_ig_categories_unsupported")
|
||||
msg <- conditionMessage(err)
|
||||
expect_match(msg, "PR #59|predates", perl = TRUE)
|
||||
})
|
||||
|
||||
# 'direct' is unaffected on the same corpus -- the guard is scoped to
|
||||
# expenditure_concept = "total" only.
|
||||
with_corpus_missing_ig_categories({
|
||||
expect_no_error(
|
||||
cog_spending("010000226085", years = 2019, category = "Police",
|
||||
expenditure_concept = "direct")
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
test_that("C2: expenditure_concept = 'total' still works on a corpus that DOES carry M/L category rows", {
|
||||
expect_no_error(
|
||||
cog_spending("010000226085", years = 2019, category = "Police",
|
||||
expenditure_concept = "total")
|
||||
)
|
||||
})
|
||||
|
||||
test_that("I2: an intergovernmental (M/L) recipe never appears as its own top-level suggestion", {
|
||||
# Task 1's M04/M05 category rows share the "Corrections" summary_categories
|
||||
# category with the Direct-flavored E04/E05, so `corrections_ig_local_
|
||||
# combined` (entirely M-prefixed) becomes a raw *candidate* in
|
||||
# .build_suggestions()'s component_code-driven query. Following a
|
||||
# "re-run with recipe = 'corrections_ig_local_combined'" hint on a plain
|
||||
# cog_spending() call would silently return intergovernmental dollars
|
||||
# under provenance$expenditure_concept = "direct". Task 6's gate
|
||||
# (.attach_ig_counterparts()) already protects the *counterpart* lookup;
|
||||
# this exercises that the candidate list itself is filtered too.
|
||||
r <- suppressMessages(
|
||||
cog_spending("010000226085", years = c(2005, 2011), category = "Corrections")
|
||||
)
|
||||
sugg <- attr(r, "provenance")$suggestions
|
||||
ids <- vapply(sugg, function(s) s$recipe_id %||% "", character(1))
|
||||
expect_true("corrections_combined" %in% ids)
|
||||
expect_false("corrections_ig_local_combined" %in% ids)
|
||||
})
|
||||
|
||||
test_that(".attach_ig_counterparts() never pairs a revenue-side recipe with its coincidental M/L suffix twin", {
|
||||
# Broader version of the case above, run at the matching-helper level
|
||||
# (the same level code review's pairwise enumeration was done at) rather
|
||||
# than end-to-end: the fixture has no (govid, year) combination where
|
||||
# cog_revenue() itself produces a covered gap for any B/C/D recipe, so an
|
||||
# end-to-end repro for THIS specific set of recipes isn't reachable
|
||||
# today. Each of these six recipes shares an exact suffix set with an
|
||||
# M/L expenditure recipe purely by reused-digit coincidence:
|
||||
# ig_federal_b47_wide {"47","94"} == ige_local_m47_wide / ige_state_l47_wide
|
||||
# ig_federal_b89_wide {"89","91","92","93"} == ige_local_m89_wide / ige_state_l89_wide
|
||||
# ig_state_c47_wide {"47","94"} == ige_local_m47_wide / ige_state_l47_wide
|
||||
# ig_state_c89_wide {"89","91","92","93"} == ige_local_m89_wide / ige_state_l89_wide
|
||||
# ig_local_d47_wide {"47","94"} == ige_local_m47_wide / ige_state_l47_wide
|
||||
# ig_local_d89_wide {"89","91","92","93"} == ige_local_m89_wide / ige_state_l89_wide
|
||||
# None of them may receive an ig_recipe_id under cog_revenue()'s own
|
||||
# flow_prefixes, since M/L only ever pairs with the direct-expenditure
|
||||
# (E/F/G) family.
|
||||
con <- uscogdata:::.ensure_session()
|
||||
fake_suggestion <- function(rid) {
|
||||
list(recipe_id = rid, label = "x", available_years = c(1967L, 2023L),
|
||||
hint = "h")
|
||||
}
|
||||
fake_suggestions <- lapply(
|
||||
c("ig_federal_b47_wide", "ig_federal_b89_wide",
|
||||
"ig_state_c47_wide", "ig_state_c89_wide",
|
||||
"ig_local_d47_wide", "ig_local_d89_wide"),
|
||||
fake_suggestion
|
||||
)
|
||||
out <- uscogdata:::.attach_ig_counterparts(
|
||||
con, fake_suggestions, c("T", "A", "U", "B", "C", "D")
|
||||
)
|
||||
ig <- unlist(lapply(out, function(s) s$ig_recipe_id))
|
||||
expect_length(ig, 0L)
|
||||
})
|
||||
@@ -0,0 +1,75 @@
|
||||
# Madison walkthrough audit -- findings F-012, F-017, F-018.
|
||||
# Tracked as uscogdata#11. See docs/walkthroughs/FINDINGS.md in cog_explorer.
|
||||
#
|
||||
# The owner's settled three-concept model (2026-07-28):
|
||||
# total = primary + interest + intergovernmental transfers
|
||||
# direct = primary + interest (Census's published Direct Expenditure)
|
||||
# primary = direct minus debt service (the NEW DEFAULT)
|
||||
# implemented by reclassifying on the crosswalk's `spend_type` column, NOT on
|
||||
# item-code first letters -- F-018 shows prefix `Y` carries both revenue
|
||||
# (Y01/Y02) and expenditure (Y05/Y06) codes, so no first-letter allowlist can
|
||||
# route them correctly.
|
||||
#
|
||||
# Fixture reproducibility: the finding's headline reconciliation is Madison
|
||||
# FY2022, where the corpus carries I89 = 46,609 (thousands) and Census's
|
||||
# published Direct Expenditure is $654,893,000 against cog_spending()'s
|
||||
# $608,284,000 (-7.1%). FY2022 is outside the bundled fixture's year window
|
||||
# (2011/2012/2019/2020), so the same invariant is asserted on FY2020, where the
|
||||
# fixture carries I89 = 27,704. Anyone running against the full corpus should
|
||||
# also check the FY2022 numbers above.
|
||||
|
||||
test_that("expenditure concepts classify on spend_type, not item-code prefix", {
|
||||
testthat::skip("Blocked on uscogdata#11 (findings F-012, F-017, F-018)")
|
||||
|
||||
mad <- "552025209777" # MADISON CITY, WI
|
||||
wi_state <- "550000227544" # WISCONSIN (state government)
|
||||
|
||||
# -- F-012: `primary` is the new default, and equals today's E/F/G figure ---
|
||||
primary <- cog_spending(govid = mad, years = 2020L)
|
||||
expect_equal(attr(primary, "provenance")$expenditure_concept, "primary")
|
||||
expect_equal(sum(primary$amt_nominal), 623347000)
|
||||
|
||||
# -- F-012: `direct` adds interest on long-term debt ------------------------
|
||||
# Expected interest read from the RAW corpus, never through cog_spending(),
|
||||
# which is the filter under test.
|
||||
interest <- wt_raw_amt(mad, 2020L, prefixes = "I")
|
||||
expect_equal(interest, 27704) # I89, in $1,000s
|
||||
|
||||
direct <- cog_spending(govid = mad, years = 2020L, expenditure_concept = "direct")
|
||||
expect_equal(sum(direct$amt_nominal), 651051000) # 623,347 + 27,704 thousands
|
||||
expect_equal(sum(direct$amt_nominal) - sum(primary$amt_nominal), interest * 1000)
|
||||
expect_true("I89" %in% wt_codes_included(direct))
|
||||
|
||||
# -- F-017: `total` carries Q12/Q18, state IG transfers to school districts --
|
||||
# Wisconsin FY2019: Q12 = 6,431,530 and Q18 = 533,391 (thousands). Today
|
||||
# neither verb's flow_prefixes contains "Q", so both are dropped from the one
|
||||
# concept that is supposed to include intergovernmental transfers.
|
||||
ig_expected <- wt_raw_amt(wi_state, 2019L, prefixes = c("M", "L", "Q"))
|
||||
expect_equal(ig_expected, 11609814) # M 4,644,893 + Q 6,964,921
|
||||
|
||||
wi_direct <- cog_spending(govid = wi_state, years = 2019L,
|
||||
expenditure_concept = "direct")
|
||||
wi_total <- cog_spending(govid = wi_state, years = 2019L,
|
||||
expenditure_concept = "total")
|
||||
|
||||
# total - direct is exactly the intergovernmental component. Asserted as a
|
||||
# delta rather than a grand total so this stays correct however the J and Y
|
||||
# families land inside `primary`.
|
||||
expect_equal(sum(wi_total$amt_nominal) - sum(wi_direct$amt_nominal),
|
||||
ig_expected * 1000)
|
||||
expect_true(all(c("Q12", "Q18") %in% wt_codes_included(wi_total)))
|
||||
|
||||
# -- F-018: prefix Y splits revenue from expenditure, by spend_type ---------
|
||||
# Y01/Y02 are Insurance Trust revenue; Y05/Y06 are Insurance Trust benefit
|
||||
# payments. All four share the first letter `Y` and the spend_type
|
||||
# "Insurance Trust", so this pair of assertions is the concrete proof that
|
||||
# classification is no longer keyed on the first letter.
|
||||
wi_revenue <- cog_revenue(govid = wi_state, years = 2019L)
|
||||
spend_codes <- wt_codes_included(wi_total)
|
||||
rev_codes <- wt_codes_included(wi_revenue)
|
||||
|
||||
expect_true("Y05" %in% spend_codes)
|
||||
expect_false("Y05" %in% rev_codes)
|
||||
expect_true("Y01" %in% rev_codes)
|
||||
expect_false("Y01" %in% spend_codes)
|
||||
})
|
||||
@@ -70,6 +70,37 @@ test_that("cog_explain prints a Suggestions section when the provenance has one"
|
||||
expect_true(grepl("re-run with recipe", txt))
|
||||
})
|
||||
|
||||
test_that("cog_explain prints the expenditure concept (I1)", {
|
||||
skip_if_no_corpus()
|
||||
d <- cog_spending("010000226085", years = 2019, category = "Police")
|
||||
t <- cog_spending("010000226085", years = 2019, category = "Police",
|
||||
expenditure_concept = "total")
|
||||
txt_d <- paste(c(
|
||||
capture.output(cog_explain(d)),
|
||||
capture.output(cog_explain(d), type = "message")
|
||||
), collapse = "\n")
|
||||
txt_t <- paste(c(
|
||||
capture.output(cog_explain(t)),
|
||||
capture.output(cog_explain(t), type = "message")
|
||||
), collapse = "\n")
|
||||
expect_true(grepl("Concept: direct", txt_d))
|
||||
expect_true(grepl("Concept: total", txt_t))
|
||||
})
|
||||
|
||||
test_that("cog_explain surfaces the C1(b) direct-suppressed flag as a warning", {
|
||||
skip_if_no_corpus()
|
||||
t <- suppressMessages(cog_spending(
|
||||
"010000226085", years = 2011, category = "Corrections",
|
||||
expenditure_concept = "total"
|
||||
))
|
||||
expect_true(attr(t, "provenance")$expenditure_concept_direct_suppressed)
|
||||
txt <- paste(c(
|
||||
capture.output(cog_explain(t)),
|
||||
capture.output(cog_explain(t), type = "message")
|
||||
), collapse = "\n")
|
||||
expect_true(grepl("Direct leg unavailable", txt))
|
||||
})
|
||||
|
||||
test_that("cog_explain prints denominator + popyear_range + counts", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# tests/testthat/test-fixture-vintage.R
|
||||
#
|
||||
# The bundled fixture is a slice of a real cog_pipeline publish tree, and
|
||||
# every test in this package -- plus the whole cog-api suite -- runs against
|
||||
# it. When the published corpus changes shape and the fixture does not, both
|
||||
# suites stay green against a corpus that no longer exists (uscogdata#18).
|
||||
#
|
||||
# These tests pin the structural facts that distinguish the current published
|
||||
# vintage from its predecessor, so a stale fixture fails loudly instead of
|
||||
# passing quietly. They assert shape, never dollar values: re-running
|
||||
# data-raw/regenerate_fixture_corpus.R against a newer publish tree should
|
||||
# keep them green.
|
||||
|
||||
# Open a bare DuckDB connection on the fixture's parquet files. Deliberately
|
||||
# not the package session: these assertions are about what the fixture
|
||||
# CONTAINS, and routing them through the reader's own views would let a
|
||||
# filter hide the very absence being checked.
|
||||
fixture_query <- function(sql, ...) {
|
||||
con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
|
||||
path <- function(rel) {
|
||||
sprintf("read_parquet(%s)",
|
||||
DBI::dbQuoteString(con, file.path(fixture_corpus_path(), rel)))
|
||||
}
|
||||
DBI::dbGetQuery(con, do.call(sprintf, c(list(sql), lapply(c(...), path))))
|
||||
}
|
||||
|
||||
test_that("fixture ships every metadata table the publish tree does", {
|
||||
skip_if_no_corpus()
|
||||
# representation/code_set are what make a sparse corpus interpretable; a
|
||||
# fixture without them predates sparsification (cog_pipeline#64).
|
||||
expected <- c(
|
||||
"canonical_alias.parquet", "canonical_fips_xwalk.parquet",
|
||||
"census_collection_coverage.parquet", "code_set.parquet",
|
||||
"harmonization_map.parquet", "harmonization_recipes.parquet",
|
||||
"lineage_events.parquet", "representation.parquet",
|
||||
"series_breaks.parquet", "summary_categories.parquet"
|
||||
)
|
||||
on_disk <- basename(list.files(
|
||||
file.path(fixture_corpus_path(), "data"), pattern = "\\.parquet$"
|
||||
))
|
||||
expect_true(all(expected %in% on_disk))
|
||||
|
||||
# The manifest must list them too -- consumers read the manifest, not ls().
|
||||
in_manifest <- with_fixture_corpus(
|
||||
basename(vapply(cog_manifest()$files$metadata, function(f) f$path, character(1)))
|
||||
)
|
||||
expect_true(all(expected %in% in_manifest))
|
||||
})
|
||||
|
||||
test_that("fixture carries the dense/sparse representation contract", {
|
||||
skip_if_no_corpus()
|
||||
rep <- fixture_query(
|
||||
"SELECT year, representation, absence_means FROM %s
|
||||
WHERE year IN (2011, 2012, 2019, 2020) ORDER BY year",
|
||||
"data/representation.parquet"
|
||||
)
|
||||
expect_equal(nrow(rep), 4L)
|
||||
expect_equal(rep$representation, c("dense_source", rep("sparse_source", 3L)))
|
||||
expect_equal(rep$absence_means, c("census_zero", rep("not_reported", 3L)))
|
||||
})
|
||||
|
||||
test_that("the fixture's wide era is sparse, not zero-padded", {
|
||||
skip_if_no_corpus()
|
||||
# FY2011 is a dense_source year: the corpus publishes only the cells Census
|
||||
# reported non-zero, and an absent cell means Census published $0. Before
|
||||
# sparsification this partition was 2,864,212 rows, ~83% of them explicit
|
||||
# zeros. A single explicit zero here means the fixture predates the change.
|
||||
zeros_2011 <- fixture_query(
|
||||
"SELECT COUNT(*) AS n FROM %s WHERE amt = 0",
|
||||
"data/long/year=2011/part-0.parquet"
|
||||
)$n
|
||||
expect_equal(zeros_2011, 0L)
|
||||
|
||||
# The modern era is a different regime: a reported zero there is real data
|
||||
# (the government filed $0), so zeros legitimately survive and must not be
|
||||
# asserted away.
|
||||
expect_gt(
|
||||
fixture_query("SELECT COUNT(*) AS n FROM %s", "data/long/year=2012/part-0.parquet")$n,
|
||||
0L
|
||||
)
|
||||
})
|
||||
|
||||
test_that("code_set covers every fixture year with the reader-spec columns", {
|
||||
skip_if_no_corpus()
|
||||
cs <- fixture_query(
|
||||
"SELECT * FROM %s WHERE year IN (2011, 2012, 2019, 2020)",
|
||||
"data/code_set.parquet"
|
||||
)
|
||||
expect_true(all(
|
||||
c("code_set_id", "year", "type", "item_code", "is_aggregate", "n_units")
|
||||
%in% names(cs)
|
||||
))
|
||||
expect_setequal(unique(cs$year), c(2011L, 2012L, 2019L, 2020L))
|
||||
})
|
||||
|
||||
test_that("every flow code carrying dollars has a category, J-prefix included", {
|
||||
skip_if_no_corpus()
|
||||
# The J (assistance/benefit) codes were uncategorised until the crosswalk
|
||||
# completion shipped (cog_pipeline#60/#65, J19 held back until #64's
|
||||
# duplication fix landed). Their absence is how a pre-crosswalk fixture
|
||||
# gives itself away.
|
||||
j <- fixture_query(
|
||||
"SELECT item_code, category, category_type, spend_subtype FROM %s
|
||||
WHERE LEFT(item_code, 1) = 'J' ORDER BY item_code",
|
||||
"data/summary_categories.parquet"
|
||||
)
|
||||
expect_true("J19" %in% j$item_code)
|
||||
expect_true(all(j$category_type == "expenditure"))
|
||||
expect_true(all(j$spend_subtype == "assistance"))
|
||||
expect_false(any(is.na(j$category)))
|
||||
})
|
||||
@@ -0,0 +1,57 @@
|
||||
# Madison walkthrough audit -- finding F-025. Tracked as uscogdata#16.
|
||||
# See docs/walkthroughs/FINDINGS.md in cog_explorer.
|
||||
#
|
||||
# cog_gov_search()'s UTILITY mode interpolates `name` into
|
||||
# regexp_matches(gov_name, <name>, 'i')
|
||||
# unescaped (R/search.R:102), while BASKET mode in the same file already routes
|
||||
# it through .escape_regex() (R/search.R:307) with the comment "so `name` is
|
||||
# treated as a literal substring". Two failure modes result:
|
||||
# correctness -- a real government cannot be found by its own exact name, and
|
||||
# a single "." matches everything (HTTP 200 both ways via the API);
|
||||
# robustness -- malformed regex reaches the engine and errors, which cog-api
|
||||
# surfaces as a 500, reachable by typing a real name one
|
||||
# character at a time.
|
||||
#
|
||||
# NOT asserted here: the finding's `q=St. Louis` example. Under correct literal
|
||||
# matching that search still returns 0 rows, because the stored name is
|
||||
# "ST LOUIS CITY" with no period -- it demonstrates today's over-matching
|
||||
# semantics, not a row the fix makes findable.
|
||||
|
||||
test_that("cog_gov_search() matches name literally, not as an unescaped regex", {
|
||||
|
||||
# -- correctness (1): a government must be findable by its own exact name ---
|
||||
# FREDONIA (BRISCOE) CITY is real; today the parentheses are read as regex
|
||||
# grouping, so its own complete name matches nothing.
|
||||
fredonia <- cog_gov_search(name = "FREDONIA (BRISCOE) CITY")
|
||||
expect_equal(nrow(fredonia), 1L)
|
||||
expect_equal(fredonia$canonical_govid, "052117184386")
|
||||
expect_equal(cog_gov_search(name = "FREDONIA (BRISCOE)")$canonical_govid,
|
||||
"052117184386")
|
||||
|
||||
# -- correctness (2): a metacharacter must not become a wildcard ------------
|
||||
# No Wisconsin city or village name contains a literal period -- established
|
||||
# against the raw registry below, NOT through the verb under test. A literal
|
||||
# search for "." must therefore return nothing; today it returns all 608.
|
||||
con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
|
||||
xwalk <- paste0(sub("/$", "", Sys.getenv("USCOGDATA_URL")),
|
||||
"/data/canonical_fips_xwalk.parquet")
|
||||
with_dot <- DBI::dbGetQuery(con, paste0(
|
||||
"SELECT COUNT(*) n FROM read_parquet('", xwalk, "') ",
|
||||
"WHERE fips_state = '55' AND govs_type = 2 AND gov_name LIKE '%.%'"))
|
||||
expect_equal(as.integer(with_dot$n[[1]]), 0L)
|
||||
|
||||
expect_equal(nrow(cog_gov_search(name = ".", state = "WI", type = "city")), 0L)
|
||||
expect_equal(nrow(cog_gov_search(name = "M.dison", state = "WI", type = "city")), 0L)
|
||||
expect_equal(nrow(cog_gov_search(name = "Mad(i|o)son", state = "WI", type = "city")), 0L)
|
||||
|
||||
# A metacharacter-free name still resolves exactly as before.
|
||||
expect_equal(nrow(cog_gov_search(name = "Madison", state = "WI", type = "city")), 1L)
|
||||
|
||||
# -- robustness: malformed pattern text returns no rows, and does not error --
|
||||
# "[" alone, and "Athens-Clarke County (bal" -- an in-progress substring of
|
||||
# ATHENS-CLARKE COUNTY (BALANCE), a real government -- both currently raise
|
||||
# (DuckDB: "Invalid Input Error: missing ]").
|
||||
expect_equal(nrow(cog_gov_search(name = "[")), 0L)
|
||||
expect_equal(nrow(cog_gov_search(name = "Athens-Clarke County (bal")), 0L)
|
||||
})
|
||||
@@ -0,0 +1,51 @@
|
||||
# Madison walkthrough audit -- finding F-021. Tracked as uscogdata#14.
|
||||
# See docs/walkthroughs/FINDINGS.md in cog_explorer.
|
||||
#
|
||||
# .peer_summary_rows() computes stats::quantile() separately INSIDE each
|
||||
# (year, spend_subtype, category) cell. A summary_p50 row is therefore "the
|
||||
# median peer's value in that one category", not "the value of the median
|
||||
# peer's total". Summing those rows across categories -- the obvious move for a
|
||||
# caller who wants one peer-median total line and reads only the column names --
|
||||
# misstated a total-spending band by -32.7% to +251.0% across the 24 years the
|
||||
# audit tested, with a sign flip at FY2012.
|
||||
#
|
||||
# The verb is not wrong and its documented use (faceting by role AND category)
|
||||
# is unaffected, so the fix is documentation: one sentence in @return.
|
||||
|
||||
test_that("cog_peer_compare() documents that summary_* rows are per-category quantiles", {
|
||||
|
||||
# man/ ships only in the source tree (the installed package carries a
|
||||
# compiled help database instead), so the prose assertions below cannot run
|
||||
# under R CMD check -- CI's earlier testthat::test_local() step enforces
|
||||
# them. The numeric pin further down needs only the corpus, but it lives in
|
||||
# the same test_that() as the sentence it protects, deliberately: they are
|
||||
# one claim, and splitting them would let the prose drift while a separate
|
||||
# test kept passing.
|
||||
rd_path <- skip_if_no_source_tree(c("man", "cog_peer_compare.Rd"))
|
||||
rd <- paste(readLines(rd_path, warn = FALSE), collapse = " ")
|
||||
|
||||
# The @return section must say the quantile is computed within each cell...
|
||||
expect_match(rd, "within each|per-category|per category", ignore.case = TRUE)
|
||||
# ...and must warn that the rows are not additive across category.
|
||||
expect_match(rd, "not additive|do(es)? not sum|cannot be summed", ignore.case = TRUE)
|
||||
# ...naming the grouping explicitly.
|
||||
expect_match(rd, "spend_subtype", fixed = TRUE)
|
||||
|
||||
# Pin the mechanism numerically so a future refactor that quietly changes the
|
||||
# quantile grouping fails here rather than silently invalidating the sentence
|
||||
# above. Fixture: Madison, 10 peers found at FY2020, category = NULL.
|
||||
peers <- cog_find_peers("552025209777", year = 2020L, max_peers = 10L)
|
||||
cmp <- cog_peer_compare(target_govid = "552025209777", peers = peers,
|
||||
category = NULL, years = 2020L, per_capita = TRUE)
|
||||
|
||||
naive <- sum(cmp$amt_per_capita_nominal[cmp$role == "summary_p50"], na.rm = TRUE)
|
||||
|
||||
peer_rows <- cmp[cmp$role == "peer", ]
|
||||
per_gov <- tapply(peer_rows$amt_per_capita_nominal, peer_rows$canonical_govid,
|
||||
sum, na.rm = TRUE)
|
||||
correct <- unname(stats::quantile(per_gov, 0.5, na.rm = TRUE))
|
||||
|
||||
expect_equal(round(naive), 6180) # summing the built-in summary rows
|
||||
expect_equal(round(correct), 2043) # quantile of each peer's OWN total
|
||||
expect_gt(naive / correct, 2) # a +200% misstatement on this cohort
|
||||
})
|
||||
@@ -0,0 +1,55 @@
|
||||
# Madison walkthrough audit -- finding F-014. Tracked as uscogdata#12.
|
||||
# See docs/walkthroughs/FINDINGS.md in cog_explorer.
|
||||
#
|
||||
# cog_revenue()'s flow_prefixes = c("T","A","U","B","C","D") never returns
|
||||
# item-code prefix X (Employee Retirement) or Y (other Insurance Trust). Per
|
||||
# Census's standard identity, Total Revenue = General + Utility + Liquor Store +
|
||||
# Insurance Trust Revenue, and Employee Retirement System contributions and
|
||||
# earnings ARE the Insurance Trust Revenue component -- so prefix X sits inside
|
||||
# a published Census revenue concept exactly the way I89 sits inside Census's
|
||||
# Direct Expenditure concept (finding F-012).
|
||||
#
|
||||
# CAVEAT FOR WHOEVER PICKS THIS UP: the argument name below (`revenue_concept =
|
||||
# "total"`) is this test's *proposal*, not a settled decision. The owner's
|
||||
# 2026-07-28 resolution covers expenditure concepts only; no revenue-side
|
||||
# naming has been ruled on. If the eventual argument is named differently,
|
||||
# change the two calls here -- the asserted dollar invariants are what matter
|
||||
# and are independent of the naming.
|
||||
#
|
||||
# Fixture reproducibility: Madison's own X-prefix revenue (FY1970-FY1986,
|
||||
# $15,098,000 nominal, $0 thereafter) is outside the bundled fixture's year
|
||||
# window (2011/2012/2019/2020), so the same invariant is asserted on Wisconsin
|
||||
# state government FY2012, where the fixture carries nonzero X01/X05/X08.
|
||||
|
||||
test_that("cog_revenue() can return Census Total Revenue including Insurance Trust (prefix X)", {
|
||||
testthat::skip("Blocked on uscogdata#12 (finding F-014)")
|
||||
|
||||
wi_state <- "550000227544" # WISCONSIN (state government)
|
||||
|
||||
# Revenue-shaped Employee Retirement codes, read from the RAW corpus rather
|
||||
# than through cog_revenue(), which is the filter under test:
|
||||
# X01 local employee contribution, X04/X05 contributions and transfers from
|
||||
# other governments, X08 earnings on investments.
|
||||
x_revenue <- wt_raw_amt(wi_state, 2012L, codes = c("X01", "X04", "X05", "X08"))
|
||||
expect_equal(x_revenue, 2038800) # 615,835 + 0 + 560,382 + 862,583 ($1,000s)
|
||||
|
||||
general <- cog_revenue(govid = wi_state, years = 2012L)
|
||||
expect_equal(sum(general$amt_nominal), 31338293000)
|
||||
|
||||
total <- cog_revenue(govid = wi_state, years = 2012L, revenue_concept = "total")
|
||||
expect_equal(sum(total$amt_nominal) - sum(general$amt_nominal), x_revenue * 1000)
|
||||
expect_equal(sum(total$amt_nominal), 33377093000)
|
||||
expect_true(all(c("X01", "X05", "X08") %in% wt_codes_included(total)))
|
||||
|
||||
# Sibling codes under the SAME first letter must stay out: X11/X12 are
|
||||
# benefit payments (an expenditure) and X21/X30/X47 are cash and securities
|
||||
# holdings (a balance-sheet stock). This is the F-018 point restated on the
|
||||
# revenue side -- the split has to come from the crosswalk's spend_type, not
|
||||
# from the letter X.
|
||||
expect_false(any(c("X11", "X12", "X21", "X30", "X47") %in% wt_codes_included(total)))
|
||||
|
||||
# Every returned row still resolves to a category. summary_categories has
|
||||
# zero rows for prefix X today, so relaxing the prefix filter alone would
|
||||
# produce category = NA rows -- see census_of_governments_finance_pipeline#60.
|
||||
expect_false(any(is.na(total$category)))
|
||||
})
|
||||
@@ -80,8 +80,11 @@ test_that("cog_geographic_rollup provenance reports the outer verb", {
|
||||
|
||||
test_that("cog_geographic_rollup accepts data.frames per layer", {
|
||||
skip_if_no_corpus()
|
||||
fl_state <- cog_gov_search("^FLORIDA$", type = "state")
|
||||
broward <- cog_gov_search("^BROWARD COUNTY$", state = "FL", type = "county")
|
||||
# Unanchored: utility mode matches literally now, so "^...$" would be
|
||||
# searched for as characters rather than read as anchors (uscogdata#16).
|
||||
# Both still resolve to exactly one row once scoped by type/state.
|
||||
fl_state <- cog_gov_search("FLORIDA", type = "state")
|
||||
broward <- cog_gov_search("BROWARD COUNTY", state = "FL", type = "county")
|
||||
r <- cog_geographic_rollup(
|
||||
govids = list(state = fl_state, county = broward),
|
||||
category = "Police", years = 2020L
|
||||
|
||||
@@ -98,7 +98,11 @@ test_that("cog_spending rejects invalid inputs", {
|
||||
|
||||
test_that("cog_spending accepts a cog_gov_search result directly", {
|
||||
skip_if_no_corpus()
|
||||
picks <- cog_gov_search("^BROWARD COUNTY$", state = "FL", type = "county")
|
||||
# Unanchored: utility mode matches `name` as a literal substring now, so
|
||||
# "^...$" would be searched for as those characters rather than read as
|
||||
# anchors (uscogdata#16). Scoped by state and type, the bare name still
|
||||
# resolves to exactly one row.
|
||||
picks <- cog_gov_search("BROWARD COUNTY", state = "FL", type = "county")
|
||||
expect_gt(nrow(picks), 0L)
|
||||
r <- cog_spending(picks, 2020L, "Corrections")
|
||||
expect_equal(unique(r$canonical_govid), "121011212191")
|
||||
@@ -244,24 +248,42 @@ test_that("basis defaults to 'harmonized' when not passed", {
|
||||
test_that("provenance carries basis + harmonization block with na_rows_excluded", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
r <- cog_spending("121011212191", 2011:2012, "Corrections")
|
||||
# FL state government. The harmonization block is scoped by government,
|
||||
# year and flow prefix -- NOT by category -- so a Corrections query still
|
||||
# counts every E/F/G-prefixed row the harmonized basis drops for having
|
||||
# no harmonized_code. The three that apply here are E21/F21/G21
|
||||
# (Education NEC, SB184-186, "discontinued_na", wide-era window ending
|
||||
# FY2011); the other discontinued_na rulings live outside E/F/G.
|
||||
# See docs/phase_r_harmonization_review.md § 1.3/1.4 and cog_pipeline
|
||||
# data/harmonization_map.csv.
|
||||
r <- cog_spending("120000226351", 2011:2012, "Corrections")
|
||||
prov <- attr(r, "provenance")
|
||||
expect_equal(prov$basis, "harmonized")
|
||||
expect_true(prov$harmonization$applied)
|
||||
expect_true(prov$harmonization$na_rows_excluded >= 0L)
|
||||
expect_true(prov$harmonization$na_amount_excluded >= 0)
|
||||
# Data-verified for the v6 fixture (corpus 2026-07-22). The Task 18 map
|
||||
# extension added E/F/G-prefix discontinued_na rulings the earlier pin's
|
||||
# comment predated: E21/F21/G21 (Education NEC local, SB184-186,
|
||||
# "trivial; explicit-NA, full wide-era window"). Broward's 2011 legacy
|
||||
# partition zero-pads exactly those three codes, so this query now
|
||||
# excludes 3 NA-harmonized rows -- all with amt = 0, hence the excluded
|
||||
# AMOUNT stays exactly zero. (The other discontinued_na rulings -- S74,
|
||||
# Z61, X04, X06, the debt-detail family, L24 -- remain outside the
|
||||
# E/F/G/K prefixes.) See docs/phase_r_harmonization_review.md § 1.3/1.4
|
||||
# and cog_pipeline data/harmonization_map.csv E21/F21/G21 rows.
|
||||
expect_equal(prov$harmonization$na_rows_excluded, 3L)
|
||||
expect_equal(prov$harmonization$na_amount_excluded, 0)
|
||||
# $2,825,439 thousands of FY2011 E21 + F21 + G21, reported in full USD.
|
||||
# Pinning a non-zero amount is the point: the earlier Broward anchor's
|
||||
# three rows were all explicit zeros, so the AMOUNT accounting was
|
||||
# asserted only against 0 and could not have caught a bug.
|
||||
expect_equal(prov$harmonization$na_amount_excluded, 2825439 * 1000)
|
||||
})
|
||||
})
|
||||
|
||||
test_that("sparsification removed the wide era's zero-pads from the exclusion count", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
# Broward County FY2011 used to carry E21/F21/G21 rows of exactly $0 --
|
||||
# the wide era stored every government x every code, zeros included. The
|
||||
# published corpus no longer does (SB194, cog_pipeline#64), so there is
|
||||
# now nothing for the harmonized basis to exclude. Absence in a
|
||||
# dense_source year means Census published $0; it does not mean the
|
||||
# exclusion machinery stopped working, which the FL state anchor above
|
||||
# proves independently.
|
||||
r <- cog_spending("121011212191", 2011:2012, "Corrections")
|
||||
h <- attr(r, "provenance")$harmonization
|
||||
expect_true(h$applied)
|
||||
expect_equal(h$na_rows_excluded, 0L)
|
||||
expect_equal(h$na_amount_excluded, 0)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -303,11 +325,13 @@ test_that("provenance$series_break_refs is a populated-when-applicable character
|
||||
r <- cog_spending("121011212191", 2020L, "Corrections")
|
||||
refs <- attr(r, "provenance")$series_break_refs
|
||||
expect_type(refs, "character")
|
||||
# No catalogued series_breaks_pq row falls inside this fixture's
|
||||
# 2011/2012/2019/2020 window for the codes this query touches (E04/G04)
|
||||
# -- data-verified; the mechanism itself is what's under test here, via
|
||||
# a query-shaped unit test in test-views.R since the fixture has no
|
||||
# positive case to pin against.
|
||||
# No catalogued code-specific series_breaks_pq row falls inside this
|
||||
# fixture's 2011/2012/2019/2020 window for the codes this query touches
|
||||
# (E04/G04) -- data-verified; the mechanism itself is what's under test
|
||||
# here, via a query-shaped unit test in test-views.R since the fixture
|
||||
# has no positive case to pin against. Corpus-wide ("ALL") entries never
|
||||
# appear in this field by construction -- they travel in
|
||||
# corpus_break_refs; see test-corpus-breaks.R.
|
||||
expect_equal(refs, character(0))
|
||||
})
|
||||
})
|
||||
|
||||
+174
-4
@@ -9,7 +9,9 @@ test_that("all expected views register on session open", {
|
||||
expected <- c(
|
||||
"long", "spending_long", "revenue_long",
|
||||
"canonical_fips_xwalk", "summary_categories",
|
||||
"spending_annotated", "revenue_annotated"
|
||||
"spending_annotated", "revenue_annotated",
|
||||
"ig_long", "ig_annotated",
|
||||
"ig_long_harmonized", "ig_annotated_harmonized"
|
||||
)
|
||||
expect_true(all(expected %in% views$table_name))
|
||||
})
|
||||
@@ -110,12 +112,78 @@ test_that("inst/sql/22- and 23- harmonized views enforce every WHERE predicate (
|
||||
expect_equal(rev$amt, 225)
|
||||
})
|
||||
|
||||
test_that("inst/sql/24- and 25- IG views retain aggregates, COALESCE NULL harmonized_code, and exclude the L-- family total (real SQL text, synthetic parquet)", {
|
||||
# ig_long / ig_long_harmonized have the subtlest predicates in the package:
|
||||
# a deliberately ABSENT `NOT is_aggregate` (unlike every other *_long view),
|
||||
# and COALESCE(harmonized_code, item_code) instead of a plain
|
||||
# `harmonized_code IS NOT NULL` filter. The only end-to-end guard on this
|
||||
# today is bound to AL state / 2011 / Education K-12, where M12 happens to
|
||||
# be the sole IG code present -- regenerate the fixture without that one
|
||||
# row and the guard would die silently while staying green. As with the
|
||||
# 22-/23- test above, this reads the real inst/sql/24-/25- text off disk
|
||||
# and executes it against a synthetic hive-partitioned parquet tree, so a
|
||||
# regression in either predicate changes which rows survive.
|
||||
skip_if_no_corpus()
|
||||
|
||||
tmp <- withr::local_tempdir()
|
||||
part_dir <- file.path(tmp, "data", "long", "year=2004")
|
||||
dir.create(part_dir, recursive = TRUE)
|
||||
part_path <- file.path(part_dir, "part-0.parquet")
|
||||
|
||||
write_con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
|
||||
DBI::dbExecute(write_con, sprintf("
|
||||
COPY (
|
||||
SELECT * FROM (VALUES
|
||||
('ig-A', 'M04', 100, false, 'M04'), -- control: passes through as-is
|
||||
('ig-B', 'M38', 50, false, 'M36'), -- fold control: real SB012 rule, renamed to M36 under harmonized basis
|
||||
('ig-C', 'M47', 99999, true, NULL), -- legacy aggregate, NO harmonized_code: must survive BOTH views
|
||||
('ig-D', 'L--', 55555, false, 'L--'), -- family total: excluded from BOTH views
|
||||
('ig-E', 'T29', 44444, false, 'T29') -- wrong prefix (revenue, not M/L): excluded from BOTH views
|
||||
) AS t(canonical_govid, item_code, amt, is_aggregate, harmonized_code)
|
||||
) TO %s (FORMAT PARQUET)
|
||||
", uscogdata:::.sql_lit_chr(part_path)))
|
||||
|
||||
sql_dir <- system.file("sql", package = "uscogdata")
|
||||
.read_view_sql <- function(filename) {
|
||||
txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
|
||||
gsub("\\{url\\}", paste0(tmp, "/"), txt, fixed = FALSE)
|
||||
}
|
||||
|
||||
con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
|
||||
DBI::dbExecute(con, .read_view_sql("10-long.sql"))
|
||||
DBI::dbExecute(con, .read_view_sql("24-ig_long.sql"))
|
||||
DBI::dbExecute(con, .read_view_sql("25-ig_long_harmonized.sql"))
|
||||
|
||||
raw <- DBI::dbGetQuery(con,
|
||||
"SELECT item_code, SUM(amt) AS amt FROM ig_long
|
||||
GROUP BY item_code ORDER BY item_code"
|
||||
)
|
||||
# L-- (family total) and T29 (wrong prefix) are gone; the aggregate row
|
||||
# M47 survives -- proof `NOT is_aggregate` is absent from ig_long.
|
||||
expect_equal(raw$item_code, c("M04", "M38", "M47"))
|
||||
expect_equal(raw$amt, c(100, 50, 99999))
|
||||
|
||||
harmonized <- DBI::dbGetQuery(con,
|
||||
"SELECT item_code, SUM(amt) AS amt FROM ig_long_harmonized
|
||||
GROUP BY item_code ORDER BY item_code"
|
||||
)
|
||||
# M38 folds to M36 (real harmonized_code present); M47 keeps its raw code
|
||||
# via COALESCE(NULL, 'M47') -- proof the aggregate row is NOT dropped by
|
||||
# a plain `harmonized_code IS NOT NULL` filter. L-- and T29 stay excluded.
|
||||
expect_equal(harmonized$item_code, c("M04", "M36", "M47"))
|
||||
expect_equal(harmonized$amt, c(100, 50, 99999))
|
||||
})
|
||||
|
||||
test_that(".build_series_break_refs matches fin_code + break_year window", {
|
||||
# No series_breaks_pq row falls inside the bundled fixture's 2011-2020
|
||||
# window (data-verified; see the "series_break_refs" test in
|
||||
# No CODE-SPECIFIC series_breaks_pq row falls inside the bundled fixture's
|
||||
# 2011-2020 window (data-verified; see the "series_break_refs" test in
|
||||
# test-spending.R), so this proves the matching logic itself against the
|
||||
# live view + a synthetic year window that DOES hit a cataloged break
|
||||
# (SB075, fin_code E62, break_year 2005).
|
||||
# (SB075, fin_code E62, break_year 2005). The corpus-wide entries are a
|
||||
# separate path with its own coverage -- SB194 does sit at 2012, inside
|
||||
# the fixture window; see test-corpus-breaks.R.
|
||||
skip_if_no_corpus()
|
||||
con <- cog_open()
|
||||
on.exit(cog_close())
|
||||
@@ -152,6 +220,108 @@ test_that("schema v5 harmonization views register when the corpus supports them"
|
||||
expect_true(all(expected_v5 %in% views$table_name))
|
||||
})
|
||||
|
||||
test_that(".harmonization_view_files guard is necessary: registration against a v4-shaped corpus (no harmonized_code column at all) succeeds only because the harmonized views are skipped", {
|
||||
# with_doctored_schema_version() (used elsewhere in this suite) only
|
||||
# rewrites manifest.json's schema_version -- the underlying `long` parquet
|
||||
# is still the bundled v6 fixture, which DOES have a harmonized_code
|
||||
# column, so it only proves the skip *happens*, not that it is *required*.
|
||||
# This test builds a genuinely v4-shaped corpus: `long` has no
|
||||
# harmonized_code column at all, matching a real pre-Phase-R2 publish
|
||||
# tree, and then shows two things: (1) the real .register_views(), gated
|
||||
# on manifest$schema_version, registers cleanly against it; (2) the exact
|
||||
# SQL text of a gated file (25-ig_long_harmonized.sql), executed directly
|
||||
# against the same corpus without the gate, fails -- proving the gate is
|
||||
# load-bearing, not incidental.
|
||||
tmp <- withr::local_tempdir()
|
||||
part_dir <- file.path(tmp, "data", "long", "year=2004")
|
||||
dir.create(part_dir, recursive = TRUE)
|
||||
part_path <- file.path(part_dir, "part-0.parquet")
|
||||
|
||||
write_con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(write_con, shutdown = TRUE), add = TRUE)
|
||||
DBI::dbExecute(write_con, sprintf("
|
||||
COPY (
|
||||
SELECT * FROM (VALUES
|
||||
('gov-1', 'E36', 100, false, 500000, 2020)
|
||||
) AS t(canonical_govid, item_code, amt, is_aggregate, population, popyear)
|
||||
) TO %s (FORMAT PARQUET)
|
||||
", uscogdata:::.sql_lit_chr(part_path)))
|
||||
|
||||
xwalk_path <- file.path(tmp, "data", "canonical_fips_xwalk.parquet")
|
||||
DBI::dbExecute(write_con, sprintf("
|
||||
COPY (
|
||||
SELECT * FROM (VALUES
|
||||
('gov-1', 'Test Gov', 1, 'County', '01', '001', NULL, 500000)
|
||||
) AS t(canonical_govid, gov_name, govs_type, type_label, fips_state,
|
||||
fips_county, fips_place, population_acs)
|
||||
) TO %s (FORMAT PARQUET)
|
||||
", uscogdata:::.sql_lit_chr(xwalk_path)))
|
||||
|
||||
cats_path <- file.path(tmp, "data", "summary_categories.parquet")
|
||||
DBI::dbExecute(write_con, sprintf("
|
||||
COPY (
|
||||
SELECT * FROM (VALUES
|
||||
('E36', 'Test Category', 'expenditure', 'direct', NULL)
|
||||
) AS t(item_code, category, category_type, spend_subtype, revenue_subtype)
|
||||
) TO %s (FORMAT PARQUET)
|
||||
", uscogdata:::.sql_lit_chr(cats_path)))
|
||||
|
||||
# Confirm the synthetic `long` genuinely lacks harmonized_code (not just
|
||||
# NULL values -- the column itself must be absent) before trusting the
|
||||
# rest of this test.
|
||||
cols <- DBI::dbGetQuery(write_con, sprintf(
|
||||
"DESCRIBE SELECT * FROM read_parquet(%s)", uscogdata:::.sql_lit_chr(part_path)
|
||||
))$column_name
|
||||
expect_false("harmonized_code" %in% cols)
|
||||
|
||||
url <- paste0(tmp, "/")
|
||||
|
||||
# (1) Full .register_views() against this v4-shaped corpus must succeed --
|
||||
# this is the behavior the guard exists to protect.
|
||||
con <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(con, shutdown = TRUE), add = TRUE)
|
||||
expect_no_error(
|
||||
uscogdata:::.register_views(con, url, manifest = list(schema_version = 4L))
|
||||
)
|
||||
views <- DBI::dbGetQuery(con,
|
||||
"SELECT table_name FROM information_schema.tables
|
||||
WHERE table_schema = 'main' AND table_type = 'VIEW'")$table_name
|
||||
expect_true(all(c("ig_long", "ig_annotated", "spending_annotated") %in% views))
|
||||
expect_false(any(c("ig_long_harmonized", "ig_annotated_harmonized",
|
||||
"spending_long_harmonized") %in% views))
|
||||
|
||||
# (2) Prove the gate is load-bearing: the exact SQL text of the skipped
|
||||
# file, executed directly (bypassing .register_views()'s schema_version
|
||||
# check) against the SAME corpus, fails because it references
|
||||
# long.harmonized_code, a column this corpus's `long` does not have.
|
||||
sql_dir <- system.file("sql", package = "uscogdata")
|
||||
.read_view_sql <- function(filename) {
|
||||
txt <- paste(readLines(file.path(sql_dir, filename), warn = FALSE), collapse = "\n")
|
||||
gsub("\\{url\\}", url, txt, fixed = FALSE)
|
||||
}
|
||||
con2 <- DBI::dbConnect(duckdb::duckdb())
|
||||
on.exit(DBI::dbDisconnect(con2, shutdown = TRUE), add = TRUE)
|
||||
DBI::dbExecute(con2, .read_view_sql("10-long.sql"))
|
||||
expect_error(DBI::dbExecute(con2, .read_view_sql("25-ig_long_harmonized.sql")))
|
||||
|
||||
# Reconciling this test with the C2 guard (expenditure-concept review):
|
||||
# `ig_annotated`/`spending_annotated` registering cleanly above proves
|
||||
# only that CREATE VIEW binds against a `summary_categories` with no M/L
|
||||
# rows at all (this synthetic corpus's own summary_categories has a
|
||||
# single E36 row, see the COPY above) -- a LEFT JOIN never fails to
|
||||
# resolve regardless of what the joined-to table contains. It does NOT
|
||||
# mean querying expenditure_concept = "total" against this shape is safe:
|
||||
# exactly this corpus (schema_version reported as supported, but
|
||||
# summary_categories predates the M/L rows cog_pipeline PR #59 added) is
|
||||
# what .require_ig_categories() exists to catch at the *verb* level,
|
||||
# since PR #59 shipped those rows with no schema_version bump. Confirm
|
||||
# the new runtime guard actually fires against this same `con`.
|
||||
expect_error(
|
||||
uscogdata:::.require_ig_categories(con),
|
||||
class = "uscogdata_ig_categories_unsupported"
|
||||
)
|
||||
})
|
||||
|
||||
test_that("spending_long filters to E/F/G/K prefixes and excludes aggregates", {
|
||||
skip_if_no_corpus()
|
||||
con <- cog_open()
|
||||
|
||||
@@ -13,6 +13,8 @@ knitr::opts_chunk$set(eval = FALSE, collapse = TRUE, comment = "#>")
|
||||
|
||||
# Why per-year population matters
|
||||
|
||||
A note on units first, since every figure below is a rate: the numerator is in **full US dollars**. The raw Census files report **thousands of dollars** and the corpus keeps them that way in its own `amt` column, but `cog_spending()` and `cog_revenue()` multiply by 1000 on the way out, so `amt_per_capita_nominal` is already dollars per person. Do not scale it again.
|
||||
|
||||
Per-capita finance numbers divide each year's spending or revenue by a population denominator. The choice of denominator is a research decision, not an implementation detail: a 24-year corpus paired with a single 5-year ACS estimate produces biased per-capita values whose magnitude scales with each government's population change.
|
||||
|
||||
`uscogdata` defaults to the **Census F-33 population value Census itself uses to compute its published per-capita tables.** That value is recorded on every COG row as `population`, with `popyear` indicating the vintage. For a city that grew from 200,000 to 300,000 between 2000 and 2023, this default reproduces the per-capita value Census published. A static ACS denominator would have understated 2000 per-capita by ~33%.
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
---
|
||||
title: "Total spending: Direct, Total, and when each is right"
|
||||
output: rmarkdown::html_vignette
|
||||
vignette: >
|
||||
%\VignetteIndexEntry{Total spending: Direct, Total, and when each is right}
|
||||
%\VignetteEngine{knitr::rmarkdown}
|
||||
%\VignetteEncoding{UTF-8}
|
||||
---
|
||||
|
||||
```{r setup, include = FALSE}
|
||||
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
|
||||
```
|
||||
|
||||
# Two questions that sound the same but aren't
|
||||
|
||||
"Total spending" means two different things depending on whether the question
|
||||
is about one government or several:
|
||||
|
||||
1. **"What did my county spend in total, a decade ago vs today?"** — one
|
||||
government, tracked over time. Either `direct` or `total` spending answers
|
||||
this correctly, as long as the same concept is used for both years.
|
||||
2. **"How do all the counties in my state compare, a decade ago vs today,
|
||||
against the neighboring state?"** — several governments, summed together.
|
||||
Here only `direct` gives the right answer; summing `total` across
|
||||
governments double-counts money that passes between them.
|
||||
|
||||
`cog_spending()`'s `expenditure_concept` argument (`"direct"` or `"total"`)
|
||||
controls which of these a query answers. This vignette walks through both
|
||||
questions with code that actually runs against the package's bundled fixture
|
||||
corpus, then explains why the second question refuses `"total"` outright.
|
||||
|
||||
Before any of the numbers below: every amount column here — `amt_nominal`,
|
||||
`amt_real`, and their `amt_per_capita_*` counterparts — is in **full US
|
||||
dollars**. The raw Census files report **thousands of dollars** and the
|
||||
corpus preserves that in its own `amt` column, but the verbs multiply by 1000
|
||||
on the way out. So `amt_nominal = 1317000` means $1.317 million, not $1.317
|
||||
billion. Do not scale it again.
|
||||
|
||||
```{r}
|
||||
library(uscogdata)
|
||||
|
||||
# Point at the bundled offline fixture (years 2011, 2012, 2019, 2020, all 50
|
||||
# states) so this vignette knits without network access. In real use,
|
||||
# USCOGDATA_URL is instead set to the published corpus URL -- see README.md.
|
||||
Sys.setenv(USCOGDATA_URL = paste0(
|
||||
system.file("extdata/fixture_corpus", package = "uscogdata"), "/"
|
||||
))
|
||||
```
|
||||
|
||||
The fixture doesn't carry 2017 or the present year, so the examples below use
|
||||
the closest years it does ship -- **2012 and 2020** -- in place of "2017 vs
|
||||
today" / "ten years ago vs today". Point `USCOGDATA_URL` at the published
|
||||
corpus and swap in real years; the mechanics are identical.
|
||||
|
||||
# Archetype 1: one government's own trend
|
||||
|
||||
For a single government, `total` is a legitimate way to describe "everything
|
||||
this government spent, including money it handed to other governments to
|
||||
spend on its behalf":
|
||||
|
||||
```{r}
|
||||
al_total <- cog_spending(
|
||||
"010000226085", # Alabama, the state government
|
||||
years = c(2012, 2020),
|
||||
category = "Highways",
|
||||
expenditure_concept = "total"
|
||||
)
|
||||
al_total
|
||||
```
|
||||
|
||||
The `intergovernmental` rows are what `"total"` adds on top of `"direct"`
|
||||
(`capital` + `operations`): Alabama's own payments out to counties and
|
||||
cities for highway work. Because this query only ever concerns Alabama,
|
||||
including that piece is safe -- there's no other government's number it
|
||||
could be double-counted against.
|
||||
|
||||
`"direct"` (the default) answers the same trend question just as validly:
|
||||
|
||||
```{r}
|
||||
al_direct <- cog_spending(
|
||||
"010000226085", years = c(2012, 2020), category = "Highways"
|
||||
# expenditure_concept = "direct" is the default; shown here for contrast
|
||||
)
|
||||
al_direct
|
||||
```
|
||||
|
||||
Both are internally consistent series. What breaks the comparison is
|
||||
**switching concepts between the two years being compared** -- e.g. `direct`
|
||||
for 2012 and `total` for 2020 -- which manufactures a trend that isn't
|
||||
really there. Pick one concept for a given question and hold it fixed across
|
||||
every year in the series.
|
||||
|
||||
# Archetype 2: a cross-government rollup
|
||||
|
||||
`cog_geographic_rollup()` sums spending across state/county/city layers for
|
||||
a place. Its default -- and, as shown below, its *only* accepted value for
|
||||
`expenditure_concept` -- is `"direct"`:
|
||||
|
||||
```{r}
|
||||
fl_rollup <- cog_geographic_rollup(
|
||||
govids = list(
|
||||
state = "120000226351", # Florida
|
||||
county = c("121011212191", "121099101897") # Broward + Palm Beach
|
||||
),
|
||||
category = "Highways",
|
||||
years = c(2012, 2020)
|
||||
)
|
||||
fl_rollup
|
||||
```
|
||||
|
||||
For the neighboring state, the comparison is a single government, so it's a
|
||||
plain `cog_spending()` call rather than a rollup:
|
||||
|
||||
```{r}
|
||||
ga_state <- cog_spending(
|
||||
"130000226087", years = c(2012, 2020), category = "Highways" # Georgia
|
||||
)
|
||||
ga_state
|
||||
```
|
||||
|
||||
Now the same rollup, but asking for `expenditure_concept = "total"`:
|
||||
|
||||
```{r, error = TRUE}
|
||||
cog_geographic_rollup(
|
||||
govids = list(state = "120000226351", county = "121011212191"),
|
||||
category = "Highways",
|
||||
years = 2020,
|
||||
expenditure_concept = "total"
|
||||
)
|
||||
```
|
||||
|
||||
`cog_geographic_rollup()` (and `cog_peer_compare()`, for the same reason)
|
||||
refuses `"total"` outright rather than silently returning an inflated
|
||||
number. The next section is why.
|
||||
|
||||
# The mechanism
|
||||
|
||||
Suppose Alabama gives a county $10M toward a highway project. That $10M
|
||||
shows up **twice** in the underlying corpus:
|
||||
|
||||
- Once on Alabama's own record, coded `M44` ("to local governments,
|
||||
Highways") -- Alabama's intergovernmental leg.
|
||||
- Again on the county's record, coded `E44` / `F44` ("Highways, current
|
||||
operations" / "capital outlay") -- the county's direct spending, because
|
||||
the county is the government that actually lets the contract and pays the
|
||||
paving crew.
|
||||
|
||||
`direct` (item codes `E`/`F`/`G`) only ever counts the second of those --
|
||||
the government that actually did the spending. `total` (Direct plus the
|
||||
`M`/`L` intergovernmental legs) counts the first one *as well*, which is
|
||||
exactly right for describing Alabama's own budget: Alabama's `total`
|
||||
genuinely includes the $10M it committed to highways, whether it built the
|
||||
road itself or paid the county to. But sum `total` across Alabama **and**
|
||||
the county, and that $10M is counted twice -- once as Alabama's payment out,
|
||||
once as the county's spending in -- reporting $20M of highway work for $10M
|
||||
actually spent.
|
||||
|
||||
This is exactly the shape of query `cog_geographic_rollup()` exists to run
|
||||
(summing across layers of government), so it refuses `"total"` rather than
|
||||
silently overstating every multi-layer figure it produces.
|
||||
|
||||
# How big is the risk in practice
|
||||
|
||||
Intergovernmental transfers aren't evenly distributed by government type.
|
||||
Measured against the bundled fixture corpus (all 50 states, each of its
|
||||
four years -- 2011, 2012, 2019, 2020), intergovernmental spending as a
|
||||
share of a government's own Direct spending is:
|
||||
|
||||
| Government type | Intergovernmental / Direct |
|
||||
|---|---|
|
||||
| State | 16.7%-48.4% (varies by year; 24.0% pooled across all four) |
|
||||
| County | 3.4%-5.1% (varies by year) |
|
||||
| City | 2.6%-3.1% (varies by year) |
|
||||
|
||||
So the Direct/Total choice matters overwhelmingly for **state** governments
|
||||
-- a state's Total genuinely differs from its Direct by a meaningful margin,
|
||||
while for a county or city the two are close. The state range is also far
|
||||
wider than a single flat figure would suggest: legacy wide-era years (2011:
|
||||
48.4%) carry proportionally more intergovernmental spending than the modern
|
||||
era (2019-2020: 16.7%-17.0%), so a state's Direct/Total gap can be nearly
|
||||
3x larger a decade earlier than it is today. That's also why the mistake
|
||||
this vignette warns about is easy to make unnoticed at the county/city level
|
||||
and costly at the state level: rolling up every government in a state using
|
||||
`total` instead of `direct` overstates the true figure -- measured at 7.6%
|
||||
for Alabama in FY2019, and 11.6% nationally.
|
||||
|
||||
# Why Total = Direct + M + L, not Direct + M
|
||||
|
||||
It's tempting to assume `total` only needs to add `M`. But `M` and `L` are
|
||||
both money the queried government itself pays **out** -- they're not two
|
||||
different accounts of a receiving government's revenue. `M` is what it
|
||||
pays to other **local** governments (e.g. a county paying a city for a
|
||||
shared paving contract); `L` is what it pays **up** to its **state**
|
||||
government (e.g. a county's contribution to a state-administered program).
|
||||
A local government's Total genuinely includes both legs, because both are
|
||||
its own spending, just routed to a different kind of recipient. On the
|
||||
bundled fixture corpus (all 50 states, 2011/2012/2019/2020), `L` is 0 for
|
||||
state governments (a state has no "payments to the state government" leg of
|
||||
its own) but is 43%-51% the size of `M` for counties (varies by year) and
|
||||
144%-189% the size of `M` for cities (varies by year; 166% pooled across
|
||||
all four) -- so a `total` that omitted `L` would silently undercount Total
|
||||
specifically for local governments, and for cities `L` is often the
|
||||
*larger* of the two legs.
|
||||
`cog_spending(expenditure_concept = "total")` includes both legs (excluding
|
||||
the `L--` family-total rollup row, which would double-count its own
|
||||
components).
|
||||
|
||||
# Composition rules
|
||||
|
||||
- `expenditure_concept` (whose spending counts -- Direct vs Direct plus
|
||||
intergovernmental) is **orthogonal** to `basis` (which vintage of the
|
||||
item-code space a query resolves against -- `"harmonized"` vs `"raw"`).
|
||||
They combine freely: `expenditure_concept = "total", basis = "raw"` is a
|
||||
valid, meaningful query, and so is every other pairing.
|
||||
- `expenditure_concept = "total"` is **mutually exclusive** with `recipe`: a
|
||||
recipe already defines its own component codes (some recipes have their
|
||||
own matching intergovernmental counterpart recipe instead -- see
|
||||
`cog_recipes()` and the "firing suggestion" notes surfaced in
|
||||
`cog_spending()`'s provenance), so layering a second, generic `total`
|
||||
union on top of a recipe query has no well-defined meaning. Passing both
|
||||
together aborts with an error naming the conflict.
|
||||
- `expenditure_concept` is a **spending-only** concept: `cog_revenue()`
|
||||
doesn't expose it (revenue's own intergovernmental codes are a different
|
||||
axis -- see `?cog_revenue`).
|
||||
|
||||
# Summary
|
||||
|
||||
- Comparing one government to itself over time: `"direct"` or `"total"`
|
||||
both work -- pick one and hold it fixed across every year compared.
|
||||
- Comparing or summing across governments -- counties within a state, a
|
||||
state against its neighbor, cities against counties: use `"direct"`.
|
||||
`cog_geographic_rollup()` and `cog_peer_compare()` enforce this by
|
||||
refusing `"total"`.
|
||||
- `"total"` = Direct (`E`/`F`/`G`) + intergovernmental (`M` to local
|
||||
governments + `L` to the state government, excluding the `L--`
|
||||
family-total row).
|
||||
Reference in New Issue
Block a user