feat: cog_balances(), a reader surface for cash and security holdings (#25) #28
@@ -28,8 +28,23 @@ USCOGDATA_URL (local path or https://)
|
||||
- `R/session.R` — `cog_open()`, `cog_close()`, `.ensure_session()`, `.coerce_govid_input()`
|
||||
- `R/manifest.R` — `.fetch_or_cache_manifest()`, `.is_local_path()` (local paths bypass HTTP/cache)
|
||||
- `R/views.R` — `.register_views()` (substitutes `{url}` into SQL files at `inst/sql/`)
|
||||
- `inst/sql/` — 7 SQL view definitions: `long`, `spending_long`, `revenue_long`, `canonical_fips_xwalk`, `summary_categories`, `spending_annotated`, `revenue_annotated`
|
||||
- `inst/sql/` — **23** SQL view definitions (measured), numbered by load order
|
||||
(`10-` through `46-`): the `*_long` layer (`long`, `spending_long`,
|
||||
`revenue_long`, `ig_long`, `balance_long`, plus `_harmonized` variants of
|
||||
`spending_long`/`revenue_long`/`ig_long`), the `*_annotated` layer
|
||||
(`spending_annotated`, `revenue_annotated`, `ig_annotated`,
|
||||
`balance_annotated`, plus `_harmonized` variants of `spending_annotated`/
|
||||
`revenue_annotated`/`ig_annotated`), and metadata views
|
||||
(`canonical_fips_xwalk`, `summary_categories`, `gov_population_yearly`,
|
||||
`harmonization_map`, `harmonization_recipes`, `series_breaks_pq`,
|
||||
`representation`, `code_set`)
|
||||
- `R/spending.R` / `R/revenue.R` — `cog_spending()` / `cog_revenue()` via shared `.verb_spendrev()`
|
||||
- `R/balances.R` — `cog_balances()`. A third money-adjacent verb, but returns a
|
||||
**stock** (a balance at a point in time) rather than a **flow** (activity
|
||||
over a fiscal year), so it does NOT route through `.verb_spendrev()` and has
|
||||
no `expenditure_concept`/`revenue_concept`/`complete`/`subtype` arguments.
|
||||
`R/balance_caveats.R` attaches `provenance$balance_caveats` (GAAP-vs-gross
|
||||
disclosure + measured per-subtype coverage windows).
|
||||
- `R/rollup.R` — `cog_geographic_rollup()` (accepts named list of govids by layer)
|
||||
- `R/peers.R` — `cog_find_peers()` + `cog_peer_compare()`
|
||||
- `R/search.R` — `cog_gov_search()` (name pattern, state, type filters)
|
||||
@@ -49,18 +64,19 @@ Any value without `://` is treated as a local path by `.is_local_path()` and rea
|
||||
|
||||
**Version:** 0.1.0 (pre-release)
|
||||
**Branch:** `main`, commit `d65e9fe`
|
||||
**Tests:** 181 PASS / 0 FAIL / 0 SKIP
|
||||
**Tests:** 763 PASS / 0 FAIL / 0 SKIP (measured `testthat::test_local()`, 2026-08-03)
|
||||
**CI:** Gitea Actions green (`.gitea/workflows/ci.yml`)
|
||||
|
||||
### Completed (Tasks 2.1–2.7)
|
||||
|
||||
All 8 exported verbs implemented and tested:
|
||||
`cog_spending`, `cog_revenue`, `cog_explain`, `cog_geographic_rollup`,
|
||||
`cog_find_peers`, `cog_peer_compare`, `cog_gov_search`, `cog_mirror`,
|
||||
plus `cog_categories`.
|
||||
All 10 exported verbs implemented and tested:
|
||||
`cog_spending`, `cog_revenue`, `cog_balances`, `cog_explain`,
|
||||
`cog_geographic_rollup`, `cog_find_peers`, `cog_peer_compare`,
|
||||
`cog_gov_search`, `cog_mirror`, plus `cog_categories`.
|
||||
|
||||
Bundled fixture corpus at `inst/extdata/fixture_corpus/` (3.6 MB, years
|
||||
2019+2020, all 50 states). Tests run fully offline — no credentials needed.
|
||||
Bundled fixture corpus at `inst/extdata/fixture_corpus/` (years
|
||||
2011, 2012, 2019, 2020 — measured via DuckDB `read_parquet(hive_partitioning=1)`,
|
||||
2026-08-03; all 50 states). Tests run fully offline — no credentials needed.
|
||||
|
||||
### Remaining to v0.1 release
|
||||
|
||||
@@ -99,6 +115,10 @@ devtools::test()
|
||||
- All verbs call `.ensure_session()` first, then query via `DBI::dbGetQuery()`
|
||||
- Return value is always a `tbl_df` with a `provenance` attribute
|
||||
- govid inputs always go through `.coerce_govid_input()` (accepts character or data frame)
|
||||
- SQL lives in `inst/sql/` — never inline SQL strings in R files
|
||||
- SQL has two layers. **View definitions** live in `inst/sql/` and are
|
||||
registered by `.register_views()`, which globs the directory in sorted order
|
||||
and substitutes `{url}`. **Query construction** is inline `sprintf()` in R
|
||||
(`.build_verb_sql()`, `.run_recipe()`, `.attach_per_capita()`). Add a view as
|
||||
a numbered `.sql` file; build a query in R.
|
||||
- No arrow dependency — DuckDB reads parquet natively
|
||||
- `withr` is a Suggests-only dep; only used in tests
|
||||
|
||||
@@ -1,5 +1,18 @@
|
||||
# uscogdata 0.1.0 (development)
|
||||
|
||||
## New: `cog_balances()` for cash-and-security holdings
|
||||
|
||||
* New `cog_balances()` exposes the 14 cash-and-security holding codes
|
||||
(`category_type = "balance"`): fund balances, retirement system holdings and
|
||||
insurance trust balances (#25). Holdings are a stock, not a flow, so the verb
|
||||
has no `expenditure_concept` / `revenue_concept` / `complete` arguments, and
|
||||
no `subtype` argument either -- for holdings, `category` is a strict
|
||||
coarsening of `balance_subtype`, so `category = "Fund Balances"` is exactly
|
||||
the `general` family (`W01`/`W31`/`W61`).
|
||||
* `cog_balances()` results carry `provenance$balance_caveats`, recording that
|
||||
Census holdings are gross rather than GAAP fund balance, and the measured
|
||||
coverage window of each subtype family.
|
||||
|
||||
## Multi-government aggregates now disclose their reporting coverage
|
||||
|
||||
* The Census of Governments is a **complete census only in years ending in 2
|
||||
|
||||
+13
-2
@@ -41,8 +41,19 @@
|
||||
#' `"cash_securities_z77_wide"` and `"cash_securities_z78_wide"` bridge the
|
||||
#' wide era to the modern one.
|
||||
#'
|
||||
#' @return A `tbl_df` with a `provenance` attribute. Amounts are full US
|
||||
#' dollars.
|
||||
#' @return Tibble with columns `year`, `canonical_govid`, `gov_name`,
|
||||
#' `balance_subtype`, `category`, `amt_nominal`, optional
|
||||
#' `amt_per_capita_nominal` and `pop_source` (when `per_capita = TRUE`),
|
||||
#' optional `amt_real` and `amt_per_capita_real` (when `adjust_to_year` is
|
||||
#' set), `codes_included`, `aggregate_fallback`, `notes`. Amounts are full
|
||||
#' US dollars.
|
||||
#'
|
||||
#' Carries a `provenance` attribute matching
|
||||
#' `inst/schemas/provenance-v1.json`, whose `balance_caveats` block reports
|
||||
#' `not_gaap`, `not_gaap_note`, `coverage_window` (measured per-subtype year
|
||||
#' extents) and `truncated` (subtypes whose coverage falls short of the
|
||||
#' requested years). `expenditure_concept`/`revenue_concept` are `NA` --
|
||||
#' holdings are a stock, not a flow, so neither concept vocabulary applies.
|
||||
#' @export
|
||||
cog_balances <- function(govid, years, category = NULL,
|
||||
per_capita = FALSE, adjust_to_year = NULL,
|
||||
|
||||
@@ -3,6 +3,12 @@ template:
|
||||
bootstrap: 5
|
||||
|
||||
reference:
|
||||
- title: Financial data
|
||||
desc: Spending, revenue and balance-sheet holdings for one or more governments.
|
||||
contents:
|
||||
- cog_spending
|
||||
- cog_revenue
|
||||
- cog_balances
|
||||
- title: Search & basket
|
||||
desc: Resolve place names into canonical govids.
|
||||
contents:
|
||||
|
||||
@@ -53,6 +53,16 @@
|
||||
"items": { "type": "string" },
|
||||
"description": "Ids of catalogued series breaks whose fin_code is the literal 'ALL' -- caveats about the corpus as a whole (dollar precision across 1976/1977, imputation exclusion from 2002, the dense -> sparse representation change at 2012, the government id scheme change at 2017) rather than about one item code. Selected on the break_year window alone, so they do not depend on which codes a result contains. Disjoint from series_break_refs by construction: an entry qualifies the whole result, not one series."
|
||||
},
|
||||
"balance_caveats": {
|
||||
"type": ["object", "null"],
|
||||
"description": "Present only on cog_balances() results (null/absent for cog_spending()/cog_revenue()). `not_gaap` is always TRUE and `not_gaap_note` explains that Census holdings are gross -- no liabilities are netted -- so they are NOT comparable to a GAAP fund balance. `coverage_window` maps each observed balance_subtype to its measured [min year, max year] in the mounted corpus (never hardcoded). `truncated` lists the subtypes whose coverage_window does not fully span the requested years.",
|
||||
"properties": {
|
||||
"not_gaap": { "type": "boolean" },
|
||||
"not_gaap_note": { "type": "string" },
|
||||
"coverage_window": { "type": "object" },
|
||||
"truncated": { "type": "array", "items": { "type": "string" } }
|
||||
}
|
||||
},
|
||||
"manifest": { "type": "object" },
|
||||
"sql_query": { "type": "string" }
|
||||
}
|
||||
|
||||
+13
-2
@@ -46,8 +46,19 @@ and raw space are identical for holdings. Reported in
|
||||
wide era to the modern one.}
|
||||
}
|
||||
\value{
|
||||
A `tbl_df` with a `provenance` attribute. Amounts are full US
|
||||
dollars.
|
||||
Tibble with columns `year`, `canonical_govid`, `gov_name`,
|
||||
`balance_subtype`, `category`, `amt_nominal`, optional
|
||||
`amt_per_capita_nominal` and `pop_source` (when `per_capita = TRUE`),
|
||||
optional `amt_real` and `amt_per_capita_real` (when `adjust_to_year` is
|
||||
set), `codes_included`, `aggregate_fallback`, `notes`. Amounts are full
|
||||
US dollars.
|
||||
|
||||
Carries a `provenance` attribute matching
|
||||
`inst/schemas/provenance-v1.json`, whose `balance_caveats` block reports
|
||||
`not_gaap`, `not_gaap_note`, `coverage_window` (measured per-subtype year
|
||||
extents) and `truncated` (subtypes whose coverage falls short of the
|
||||
requested years). `expenditure_concept`/`revenue_concept` are `NA` --
|
||||
holdings are a stock, not a flow, so neither concept vocabulary applies.
|
||||
}
|
||||
\description{
|
||||
Returns Census cash-and-security holdings (`category_type = "balance"`):
|
||||
|
||||
@@ -359,6 +359,14 @@ test_that("a request past a family's coverage window is flagged", {
|
||||
})
|
||||
})
|
||||
|
||||
test_that("the provenance schema documents balance_caveats", {
|
||||
sch <- jsonlite::fromJSON(
|
||||
system.file("schemas", "provenance-v1.json", package = "uscogdata"),
|
||||
simplifyVector = FALSE
|
||||
)
|
||||
expect_true("balance_caveats" %in% names(sch$properties))
|
||||
})
|
||||
|
||||
test_that("the caveat message fires once per session", {
|
||||
skip_if_no_corpus()
|
||||
with_fixture_corpus({
|
||||
|
||||
Reference in New Issue
Block a user