3 Commits

Include renames

Author	SHA1	Message	Date
jared	0fbae00e27	feat: name a cohort by state/type predicate instead of a 40k-id IN list ... cog_spending(), cog_revenue() and cog_balances() gain optional state/type arguments. Both default to NULL, so every existing govid-based call is unchanged. The verbs took a cohort only as a govid vector, which .sql_lit_chr() rendered into a quoted IN list and .verb_spendrev() embedded into 5-8 separate statements per call: the scope check, the main aggregate, the per-capita join, the harmonization block, and the suggestion and suppression queries. For type = "city" that list is 301,589 characters, parsed and planned from scratch every time it appears. Passing state/type instead expresses the cohort as a subquery against canonical_fips_xwalk, so its size never enters the SQL string at all. Measured on the production corpus, same FY2022 aggregate over the 20,106-government city cohort, DUCKDB_THREADS=2, median of 5: IN (20,106 literals) -- 0.3.0 432 ms join against a temp cohort table 132 ms predicate on canonical_fips_xwalk 102 ms no cohort filter at all (the floor) 105 ms The predicate reaches the no-filter floor: the cohort restriction is now free. End to end through cog_spending(category = "Police"), 1080 ms -> 271 ms, 3.99x -- larger than the single-query saving, because the repetition across statements is what actually cost. Design decisions, both made explicitly rather than left implicit: - govid AND state/type INTERSECT. "These ids, narrowed to that state/type" is a real query, and an error here could never be relaxed later without breaking callers. - A predicate cohort has no id list to report, so provenance$scope$govids_found/govids_missing stay empty and a new scope$cohort block carries state, type and n_governments. Resolving the ids just to report them would put 20,000 govids in every fleet-scale response body -- the cost this change removes. A govid-named cohort's provenance is untouched. state/type are coerced with .coerce_state_to_fips()/.coerce_type(), the same helpers cog_gov_search() uses. That is load-bearing: the argument is a postal abbreviation ("WI") while fips_state holds a FIPS code ("55"), and a predicate on the raw parameter matches nothing and returns an empty result indistinguishable from "reported nothing". cog-api hit exactly this trap optimizing the same path. .attach_per_capita() now keys its population lookup on the govids present in the result rather than the requested cohort. Those are the only ones its LEFT JOIN can match, so the output is identical -- but it needs no id list, and on a paginated call it looks up one page instead of the fleet. Fixes uscogdata#58.	2026-08-09 14:15:21 -04:00
jared	93300ae0c1	feat: three-concept expenditure model classified by crosswalk membership (#11) ... Rewrites expenditure/revenue classification off item-code first-letter prefixes and onto summary_categories membership (F-018: prefix Y spans revenue, expenditure, and balance codes), and exposes expenditure_concept = c("primary", "direct", "total") with primary as the new default: primary = operations + capital + assistance direct = primary + interest + insurance_benefits (Census Direct) total = direct + intergovernmental (M/L/Q via ig views) - inst/sql: flow views (20-25) select by crosswalk membership; summary_categories moves to 11- so it registers before them (DuckDB binds view sources eagerly). The IG leg gains Q11/Q12/Q18 state school-system payments (F-017). - R: one subtype scope per verb call drives the verb SQL, the harmonization exclusion count, and the complete = TRUE grid; flow_prefixes survives only to scope recipe suggestions. cog_geographic_rollup/cog_peer_compare accept primary|direct, still refuse total, and now actually pass the concept through. - Balance codes can never reach a spending or revenue result (uscogdata#25), asserted at both view and verb level. - Deletes the #11 skip; per the 2026-07-30 owner ruling the F-018 Y01 proof is asserted against the crosswalk, not the default cog_revenue() call (which stays General Revenue pending #12). Suite: 696 pass / 0 fail / 1 skip (#12, expected). Closes #11	2026-07-30 16:56:50 -04:00
jared	af85a23ea7	feat: complete = TRUE fills absent cells with their meaning (#18) ... Sparsification (cog_pipeline#64, SB194) stopped the corpus storing the wide era's explicit zeros, which made absence ambiguous: <= FY2011 dense_source absent => Census published $0 >= FY2012 sparse_source absent => not reported, unknown A wide-era query whose cells were all $0 had begun returning nothing at all, with no way to get them back -- strictly less than the reader exposed before, which is why #64 filed this follow-on. complete = TRUE fills the requested grid from `code_set` and stamps every row with value_source: "reported", "census_zero" (amt 0), or "not_reported" (amt NA). The NA is the point. Filling a modern absence with 0 would invent data, which is exactly the error the representation contract exists to prevent -- and it makes this strictly MORE informative than the pre-sparsification corpus, which could not tell a published zero from an unreported cell either. Measured on the fixture, Broward County: FY2011 returns 28 reported + 16 census_zero; FY2019 returns 30 reported + 14 not_reported. The five categories that walkthrough finding F-006 read as "retired at FY2012" now report themselves correctly as census_zero before and not_reported after. Scoping decisions, each of which would invent rows if taken loosely: - The grid is per government TYPE (code_set.type). Filling against the union of all types would give a county cells like "state IG transfer to school districts", indistinguishable from real census zeros. - NOT is_aggregate, mirroring spending_long/revenue_long. Without it the grid offers cells those views never return, so each would fill as a phantom $0. - Filling happens BEFORE per_capita and inflation, so a census_zero stays 0 through both and a not_reported stays NA rather than becoming 0. Two new views (36-representation, 37-code_set) are gated on the manifest LISTING those tables, not on schema_version. Sparsification did not bump the version -- the fixture this package shipped against until 2026-07-30 was already v6 and carried neither table -- so a version gate would register a view over a missing file and fail at CREATE VIEW time on exactly the corpora the check exists to tolerate. with_corpus_missing_representation() models that corpus and asserts the abort. Refused where the fill would be guesswork, both classed uscogdata_complete_unsupported: a recipe defines its own component codes and never touches summary_categories; the intergovernmental leg deliberately keeps aggregate rows (inst/sql/24-ig_long.sql) so its cells are not the ones code_set describes. Expected cell sets in the tests are computed from the corpus parquet directly, never through the verb -- verifying what a filter does through that same filter proves nothing. Closes DoD 2, 3 and 4 of #18. DoD 5 (the cog-api follow-on) is filed separately. Suite: 658 pass / 0 fail / 3 skip (was 629/0/3). rcmdcheck clean.	2026-07-30 11:47:51 -04:00