Jared Knowles jared
  • Joined on 2020-12-20
jared merged pull request Civilytics/uscogdata#61 2026-08-09 14:31:54 -04:00
feat: cohort predicates instead of a 40k-id IN list (#58)
jared closed issue Civilytics/uscogdata#58 2026-08-09 14:31:54 -04:00
Accept state=/type= predicates on the money verbs instead of a 40k-id IN list
jared pushed to master at Civilytics/civilyticsR 2026-08-09 14:21:12 -04:00
e43c4229c3 Fix ggplot2 imports and polish utility functions for publication
jared commented on issue Civilytics/uscogdata#58 2026-08-09 14:16:37 -04:00
Accept state=/type= predicates on the money verbs instead of a 40k-id IN list

Implemented in #61 — measured, and it reaches the floor

Same four-way comparison re-run against the production corpus with the shipped implementation as one of the arms. DUCKDB_THREADS=2,…

jared created pull request Civilytics/uscogdata#61 2026-08-09 14:16:07 -04:00
feat: cohort predicates instead of a 40k-id IN list (#58)
jared created branch feat/cohort-predicates-58 in Civilytics/uscogdata 2026-08-09 14:15:38 -04:00
jared pushed to feat/cohort-predicates-58 at Civilytics/uscogdata 2026-08-09 14:15:38 -04:00
0fbae00e27 feat: name a cohort by state/type predicate instead of a 40k-id IN list
jared pushed to master at Civilytics/civilyticsR 2026-08-09 13:59:23 -04:00
023aedf83a Add NEWS.md, rename LICENSE.md to LICENSE, add utility tests
jared pushed to master at Civilytics/civilyticsR 2026-08-09 13:53:16 -04:00
6051e4bb5c Remove vendored headshots and add proportion CI tests
jared commented on issue Civilytics/uscogdata#58 2026-08-09 13:49:19 -04:00
Accept state=/type= predicates on the money verbs instead of a 40k-id IN list

Implementation notes (from the profiling above)

Where the IN list is built today

  • .sql_lit_chr() — R/spending.R:737. Renders 'a','b',...; harmless in itself.
  • .build_verb_sql()…
jared closed issue Civilytics/uscogdata#59 2026-08-09 13:48:35 -04:00
cog_geographic_rollup(): push aggregation and pagination into SQL
jared commented on issue Civilytics/uscogdata#59 2026-08-09 13:48:34 -04:00
cog_geographic_rollup(): push aggregation and pagination into SQL

Closing in favour of #58, which is where the rollup win actually is.

Profiling (detail in the comment above) showed this issue targets ~1% of the runtime: the many-to-many left_join is 6 ms…

jared commented on issue Civilytics/uscogdata#58 2026-08-09 13:46:20 -04:00
Accept state=/type= predicates on the money verbs instead of a 40k-id IN list

Measured: this is worth more than the issue claims, and it is the fix for #59 too

Same aggregate over the 20,106-government type = city cohort (FY2022), with the cohort expressed four ways…

jared commented on issue Civilytics/uscogdata#59 2026-08-09 13:46:20 -04:00
cog_geographic_rollup(): push aggregation and pagination into SQL

Measured before implementing, and the premise here does not hold

This issue proposes moving the aggregation and crosswalk join into SQL, on the theory that "the many-to-many join is the part…

jared opened issue Civilytics/uscogdata#60 2026-08-09 11:21:17 -04:00
Let cog_open() cap DuckDB threads (USCOGDATA_DUCKDB_THREADS)
jared opened issue Civilytics/uscogdata#59 2026-08-09 11:21:16 -04:00
cog_geographic_rollup(): push aggregation and pagination into SQL
jared opened issue Civilytics/uscogdata#58 2026-08-09 11:20:44 -04:00
Accept state=/type= predicates on the money verbs instead of a 40k-id IN list
jared opened issue Civilytics/uscogdata#57 2026-08-09 11:20:43 -04:00
Add limit/offset to cog_gov_search() and cog_balances()
jared commented on issue Civilytics/uscogdata#56 2026-08-09 11:20:11 -04:00
Performance pass over the reader before final release

Evidence from the cog-api side: the overhead is remote I/O, not compute

While doing a performance pass on cog-api (which is a thin wrapper over these verbs), I measured the same call shapes…

jared commented on issue Civilytics/uscogdata#46 2026-08-09 09:20:27 -04:00
Release 6/11: multi-OS R CMD check on GitHub Actions

@/tmp/claude-1000/-home-jared-Nextcloud-Civilytics-Code-Civilytics-cog-explorer/4bbb145c-ac06-4d88-84be-1e596655c718/scratchpad/c46.md