Final-review followups:
1. cog_explain rendered the popyear range as raw 2-digit values
("popyear range: 19-20"), which a user could read as the years 19-20.
Added .expand_popyear() helper to format as 4-digit calendar years
(2019-2020). Pivot at 70 to handle pre-2000 vintages if the corpus
ever extends backward.
2. cog_peer_compare provenance was missing pop_range and is_ratio,
omitted from the spec-required reproducibility metadata.
cog_find_peers now stamps both as tibble attributes; cog_peer_compare
reads them through to provenance$pop_range and provenance$is_ratio.
Test coverage extended: explain test asserts the 4-digit format and
rejects the old 2-digit form; peer-compare test asserts pop_range +
is_ratio propagate end-to-end.
326 PASS / 0 FAIL.
Updates transformations\$per_capita with the new denominator_source string,
popyear_range, and pop_source_counts. .attach_per_capita stashes
popyear_range on the result; .verb_spendrev strips the helper attr after
provenance is built.
Reads attr(peers, 'cohort_year') when the caller passed a cog_find_peers()
tibble; NA when the caller passed a bare character vector. Stamped as a
constant column on the result and recorded in provenance alongside the
cohort govids.
Adds optional 'year' argument (defaults to most recent observed year for
the target). Filters and ranks candidates by gov_population_yearly.population
at that year. Returned column renamed population_acs -> population.
Cohort year attached as attr(x, 'cohort_year').
Adds .resolve_cohort_year() helper. Updates test assertions to use
'population' column name. Regenerates man/cog_find_peers.Rd.
Three RED tests that drive Task 6's cog_find_peers() rewrite:
- defaults year to most recent observed (expects cohort_year attr + population column)
- honors explicit year= argument (expects cohort_year attr)
- errors with "no observed population" for unobserved year
.notes_column now joins multiple per-row notes with '; '. Adds the
'No population denominator available for this gov type' note when
pop_source is 'unavailable'. Gracefully handles absent pop_source
(per_capita = FALSE). Two new tests: one corpus-level (census_f33
branch) and one synthetic unit test covering multi-note concatenation.
Adds a RED test asserting that cog_spending(per_capita = TRUE) divides
by the per-year F-33 population (Broward 2019: 1,935,878; 2020: 1,952,778)
rather than the static ACS value (1,940,907). Uses absolute-tolerance
expect_true(abs(...) < 1) instead of expect_equal(tolerance=1) because
testthat 3 treats the tolerance argument as relative.
Cross-task review found two edge cases that violated the basket-mode
soft-fail contract:
- Per-row excluded type (e.g. type = c(NA, "special_district")) hit
.coerce_type()'s abort inside the per-row resolver, killing the
whole basket call. Now treated as no_match in the sidecar.
- Malformed regex in the substring fallback (e.g. name = "San(Diego")
propagated DuckDB engine errors. .escape_regex() now backslash-
escapes meta characters before the regexp_matches call. Utility-
mode regex behavior is unchanged.
Plus a new public-surface test for the all-no-match case.
Sidecar accessors for basket-mode results. cog_basket_resolution()
returns the full resolution tibble (drops candidates list-col by
default for readable printing). cog_basket_unresolved() filters to
ambiguous/no_match rows for iterative refinement.
Vector name + state + type arguments dispatch to a per-row resolver
that produces a basket tibble with a 'resolution' sidecar attribute.
Utility mode (length-1 name) is unchanged.
Multi-row matches within a single govs_type pick the largest-population
row (status=largest_pop). Multi-row matches spanning >=2 types return no
basket row (status=ambiguous) with all candidates preserved for the
sidecar.
.resolve_basket_row() now falls back to case-insensitive substring
match when no exact match is found, and short-circuits empty/whitespace
input to no_match. Disambiguation stub raises pending Task 5.
Adds inst/extdata/fixture_corpus/ — a 3.6 MB two-year (2019/2020) slice
of the published corpus (OH+VT+WY fixture from cog_pipeline test profile
plus all 50 states). Includes canonical_fips_xwalk.parquet,
summary_categories.parquet, docs/, and a trimmed manifest.json.
setup.R now points USCOGDATA_URL at the bundled fixture automatically,
bypassing HTTP / Nextcloud entirely. DuckDB reads local parquet via the
existing .is_local_path() fast-path in manifest.R; no httpfs required.
session is reset between test files via withr::defer(cog_close()).
helper-fixture.R gains fixture_corpus_path(), a richer skip_if_no_corpus()
that checks the bundled fixture first, and with_fixture_corpus() for
tests that need explicit session isolation.
test-spending.R: adjust the inflate-column test to use 2019 (fixture year)
instead of 2015 (absent from fixture).
Result: 181 PASS / 0 FAIL / 0 SKIP — all tests run against real parquet
data with real DuckDB queries and no network dependency.
Discovery verb over the summary_categories view, grouped one row per
(category, subtype). Parallels cog_gov_search: analysts use it to
find the valid `category` values to pass into cog_spending(),
cog_revenue(), cog_geographic_rollup().
Columns: category, category_type, subtype, n_codes, item_codes
(comma-separated, alphabetical). Optional filters:
type = NULL | "spending" | "revenue"
pattern = regex matched case-insensitively on category
The user-facing "spending" alias is translated internally to the
corpus-native "expenditure" so callers don't have to learn Census
vocabulary, while the returned category_type column preserves the
native value for auditability.
Also: fix @noRd placement in session.R so devtools::document() stops
warning.
Tests: +16 new / 181 total pass. check 0E/0W/0N.
Two UX fixes surfaced by first real-user use:
1. cog_spending / cog_revenue / cog_geographic_rollup now accept either
a character vector OR a data.frame with a canonical_govid column
(e.g. output of cog_gov_search() or cog_find_peers()). Shared
.coerce_govid_input() helper in session.R. This lets the natural
pipe work:
cog_gov_search('MIAMI', state='FL', type='city') |>
cog_spending(years=2022, category='Police')
cog_peer_compare already accepted a data.frame for the peer arg;
behavior there is unchanged.
2. .check_govids_in_scope() message reworded. The old text led with
'v0.1 covers gov_types 0-3' which falsely implied the missing govids
were scope-excluded types when the more common real cause is a typo
or a guessed value. New message leads with typo + pre-2017 PID,
mentions scope exclusion as one possibility, and points at
cog_gov_search() as the recovery path.
Tests: 165 pass / 0 fail. check 0E/0W/0N.
Three pieces:
1. cog_gov_search: name/state/type search over canonical_fips_xwalk
for resolving human-readable place names into canonical_govids.
Accepts USPS abbrev ('FL') or FIPS int (12) for state; integer
0-3 or name ('state','county','city','township') for type. Types
4/5 emit an explanatory cli message and return an empty tibble
(v0.1 corpus excludes them). USPS<->FIPS table hardcoded with
50 states + DC + territories; FIPS 66 = GU (not GA).
2. cog_mirror: downloads manifest-listed files to a local directory
with SHA-256 idempotency (files with matching hash return status
'cached'). Supports HTTP and local-path fixture URLs. Round-trip
test: mirror + re-open against the mirror + query Broward 2020
returns identical results.
3. Scope-aware verbs: .check_govids_in_scope() helper in session.R
queries canonical_fips_xwalk for the requested govids, emits a
cli_inform listing any missing ones, and records the found/missing
sets under provenance$scope. Wired into cog_spending (and
transitively into cog_revenue, cog_geographic_rollup,
cog_peer_compare via their cog_spending calls).
Also: dropped dbplyr from Imports (unused).
Tests: +29 (22 search + 12 mirror - 5 refactored) / 159 total pass.
devtools::check() now clean: 0E / 0W / 0N.
cog_find_peers selects peers from canonical_fips_xwalk by same-type,
same-state, and population-range (ratio or absolute) criteria,
ordered by |log(pop_ratio)| ascending.
cog_peer_compare accepts the find_peers result (or a plain character
vector of govids), pulls spending for target + peers via cog_spending,
and appends summary rows (summary_p25/p50/p75) so the whole result
can be faceted by `role` in a single ggplot call. Summary rows honor
per_capita + adjust_to_year by picking the right value column.
target_rank reports the target's rank among target+peers at max(years).
Provenance is rewritten with verb = cog_peer_compare and peer_count.
Also: globalVariables('.data') in zzz.R to silence R CMD check on
tidy-eval pronouns.
Tests: 21 new / 120 total pass. devtools::check() 0E/0W/2N.
Wraps cog_spending across a named list of state/county/city layers,
tagging each row with its `layer` and attaching a scope_note that
documents geographic-scope caveats (state totals are statewide, county
totals include areas outside a listed city, city proper excludes
special districts). Per-capita uses each layer's own population from
the canonical_fips_xwalk.
Provenance is inherited from cog_spending but rewritten to reflect
the outer verb (verb, call, layers).
Tests: 19 new / 99 total pass. devtools::check() 0E/0W/2N.
Three core query verbs over the spending_annotated / revenue_annotated
DuckDB views. Each verb accepts vector govid, vector years, optional
category filter, per_capita flag, and adjust_to_year for CPI-U
real-dollar conversion (bundled index).
Amounts are returned in full USD (SUM(amt) * 1000) so callers can
freely rescale to millions/billions. The $1,000s -> $USD conversion
is recorded in provenance$transformations$units_conversion.
Every result carries an attr(., "provenance") list matching
inst/schemas/provenance-v1.json. cog_explain() prints the structured
form via cli or returns the raw list for MCP/JSON consumers.
Also: .fetch_or_cache_manifest() now handles local fixture paths so
tests can point USCOGDATA_FIXTURE_URL at the pipeline publish_cache/
without a working HTTP server.
Tests: 80 pass / 0 fail. devtools::check() 0E/0W/2N (both notes
pre-existing / environmental).
Adds R/sysdata.rda with the annual-average CPIAUCSL index (1947-2026,
80 years, sourced from FRED) and an internal .inflate() helper that
converts nominal amounts between two years via the ratio of CPI values.
Bundling CPI in the package (rather than publishing a cpi_annual.parquet
in the corpus) matches the reader-specification intent: real-dollar
conversion is a verb-level option, not a corpus-level artifact, so the
target_year stays flexible at query time.
data-raw/cpi_annual.R carries the one-shot FRED refresh used to build
sysdata.rda. Re-run when the CPI series needs to roll forward.
Verified: CPI(2000)/CPI(2021) ≈ 0.635, matching the ~0.63 sanity
anchor in cog_explorer's existing inflation logic.
Tests: tests/testthat/test-adjust.R covers the known 2000→2021 ≈ 1.574x
anchor, vectorized from_year, error cases for out-of-range years, and
NA-amount preservation. 14 pass / 0 fail.
Seven DuckDB views register on session open: long (raw), spending_long
and revenue_long (prefix-filtered, NOT is_aggregate per reader-spec §4),
canonical_fips_xwalk and summary_categories (identity), spending_annotated
and revenue_annotated (LEFT JOIN xwalk + categories for verb composition).
File prefix `NN-` enforces creation order so *_annotated views resolve
their *_long dependencies.
Deviation from plan: series_breaks/cpi_annual/legacy_aggregate_map and
*_with_transforms views are deferred — their backing parquets are not
in the v0.1 corpus (manifest.files.metadata only lists canonical_fips_xwalk
and summary_categories). The verbs will compute CPI adjustment verb-side
against a bundled cpi table in a later task.
jared