# uscogdata 0.3.0

First public release.

`uscogdata` provides curated R verbs over the Civilytics US Census of
Governments finance corpus: unit-level financial profiles, geographic rollups
and peer comparisons, with auditable provenance on every result.

## What it covers

Government types 0-3 (state, county, municipality, township), FY1967-FY2024 --
56 fiscal years, 46,148,034 rows, 190.6 MB. There is no source data for FY1968
or FY1969. Special districts (type 4) and school districts (type 5) are out of
scope pending validation.

## The verbs

`cog_spending()`, `cog_revenue()` and `cog_balances()` for flows and holdings;
`cog_gov_search()` to resolve place names (including basket mode for many at
once); `cog_find_peers()` and `cog_peer_compare()` for cohorts;
`cog_geographic_rollup()` for aggregates; `cog_categories()`, `cog_recipes()`,
`cog_manifest()` and `cog_explain()` for metadata and provenance; and
`cog_mirror()` for a local copy of the corpus.

## Reading the corpus now works out of the box

* The package reads the published corpus over HTTPS **with no configuration**.
  Previously the default was a placeholder sentinel and no document in the
  package supplied a working URL, so a new user had no path to a session.
* Remote reads work at all. The partitioned view used a glob, and DuckDB
  cannot expand a glob over generic HTTP -- there is no directory listing to
  expand against. Partition paths are now enumerated from the corpus manifest,
  which is host-agnostic: an HTTPS mirror, a Nextcloud share and a local
  `cog_mirror()` copy all take the same path.
* Nothing is written to disk in remote mode; DuckDB fetches only the row
  groups a query needs.

## Four things to know before your first query

* **Amounts are in full US dollars.** The raw Census files report thousands;
  the verbs multiply by 1000 on the way out. Do not multiply again.
* **Multi-government aggregates disclose their coverage.** The Census is a
  complete enumeration only in years ending in 2 and 7; every other year is a
  sample. Every such result carries `provenance$coverage` with per-year
  `n_units_reporting`.
* **Absence means two different things.** Before FY2012 an absent cell means
  Census published $0; from FY2012 it means not reported. `complete = TRUE`
  labels which.
* **Series breaks reach you unasked.** Catalogued breaks intersecting your
  query appear in provenance and in `cog_explain()`.

## Known limits

* Special districts (type 4) and school districts (type 5) are out of scope.
* Per-capita rollups exclude governments with no F-33 population, which is by
  design but does silently narrow a rollup.
* `n_units_reporting` is category-conditional and is not a response rate.
* Employee-retirement (`X`) codes stop at FY2016, when those systems moved to
  the Annual Survey of Public Pensions.

# uscogdata 0.2.0

## New features

* `cog_spending()` and `cog_revenue()` accept the reserved category
  `"All Categories"`, returning one summed row per
  `(year, canonical_govid, subtype)` across every category inside the
  requested concept's subtype scope. Filtering the result to
  `spend_subtype == "operations"` gives an operating-expenditure total.
  `cog_geographic_rollup()` inherits it,
  which is the efficient way to build a geographic total — previously a
  caller had to issue one rollup per category and sum the results
  (cog-api#37).

  `"All Categories"` is not the same thing as `expenditure_concept = "total"`.
  The concept chooses which subtypes are in scope; `"All Categories"` chooses
  whether the rows inside that scope are broken out or summed.

* `cog_categories()` advertises `"All Categories"` for the expenditure and
  revenue vocabularies, so the reserved value is discoverable.

* Coverage signposting (see "Signposting now catches partially-suppressed
  categories" below) now also works in `category = "All Categories"` mode.
  The recipe-suggestion candidate query used to be scoped by `category`,
  which is never a match for the reserved `"All Categories"` value, so
  `provenance$suggestions` always came back empty there — the one mode whose
  whole point is "you cannot sum the wrong scope" was silently unable to
  signal a wrong scope. The candidate query is now scoped by the concept's
  subtype allowlist instead, symmetric with how `.build_verb_sql()` itself
  scopes the summed total: Los Angeles County FY2011, `category = "All
  Categories"` still excludes $271,589,000 of aggregate-published Public
  Welfare (`E68`), but now names `recipe = "welfare_cash_e68_wide"` to
  recover it instead of reporting zero suggestions.

## Documentation

* `cog_geographic_rollup()` and `cog_peer_compare()` now document that
  `provenance$coverage`'s `n_units_reporting` is **category-conditional** and
  is not a response rate: a government that was surveyed and genuinely spends
  nothing in the requested category is indistinguishable from one never
  surveyed (uscogdata#36).