with the comment "Backslash-escape POSIX regex metacharacters so name is treated as a literal substring." The fix already exists in this file; utility mode just doesn't call it.
Two distinct failure modes result, and they must not be conflated:
Correctness — a real government cannot be found by its own exact name, and a wildcard matches everything. HTTP 200 in both directions, silently wrong.
Robustness — syntactically malformed regex reaches the engine and errors, which the API surfaces as a 500.
Findings resolved
Finding
Severity
Summary
F-025
high
q is interpolated unescaped into a regex match, not a substring match; a real government cannot be found by its own exact name, and ordinary typing can 500 the endpoint
Reproduction (verbatim from FINDINGS.md, verified against the deployed API)
B=https://cog-api.civilytics.org/api/v1
# Regex semantics, not substring semantics
curl -s -G "$B/governments" --data-urlencode "q=Madison"\
--data-urlencode "state=WI" --data-urlencode "type=city"| jq -c '.meta.total'# 1 (baseline)
curl -s -G "$B/governments" --data-urlencode "q=."\
--data-urlencode "state=WI" --data-urlencode "type=city"| jq -c '.meta.total'# 608 -- a single "." matches every WI city, the same failure shape as F-001# Silent wrongness on a real, unscoped name containing "."
curl -s -G "$B/governments" --data-urlencode "q=St. Louis"| jq -c '.meta.total'# 0 -- HTTP 200, not a crash; St. Louis exists in the corpus# A real government whose own name contains parentheses cannot be found by it
curl -s -G "$B/governments" --data-urlencode "q=FREDONIA (BRISCOE) CITY"| jq -c '.meta.total'# 0 -- the government's exact, complete name, typed verbatim
curl -s -G "$B/governments" --data-urlencode "q=FREDONIA"| jq -c '.data[] | select(.gov_name | contains("BRISCOE"))'# {"gov_name":"FREDONIA (BRISCOE) CITY","canonical_govid":"052117184386","fips_state":"05", ...}# -- found ONLY by deleting the parenthesized part of its own name# Malformed regex reached through ordinary typing, not just a bare "["
curl -s -G "$B/governments" --data-urlencode "q=[" -w '\n%{http_code}\n'# {"status":"error","error":"Internal server error."} 500
curl -s -G "$B/governments" --data-urlencode "q=Athens-Clarke County (bal" -w '\n%{http_code}\n'# {"status":"error","error":"Internal server error."} 500 -- reached# partway through typing ATHENS-CLARKE COUNTY (BALANCE), a real government's name
Six probes against state=WI&type=city pin the semantics as regex, not substring: q=. = 608; q=M.dison = 1; q=^MADISON = 1; q=MADISON$ = 0 (the stored name is MADISON CITY); q=Mad(i|o)son = 1.
Reproduces directly on the bundled fixture corpus through the R verb, which is what the test asserts against:
cog_gov_search(name=".",state="WI",type="city")# 608 rows; no WI city name contains a literal "."cog_gov_search(name="FREDONIA (BRISCOE) CITY")# 0 rows; the government is 052117184386cog_gov_search(name="[")# Error: Invalid Input Error: missing ]: [
Why it matters
cog_gov_search() is step one of essentially every workflow in this package and in the API built on it. The Fredonia case is the strongest evidence in the finding, and needs no regex literacy to read: a database returning zero results for a government's own on-file name, typed verbatim, is broken. The robustness half is reachable by nothing more adversarial than typing a real government's name one character at a time — a type-ahead client querying per keystroke would 500 on every parenthesized name, then find nothing once the name is complete.
Unvalidated input reaching a regex engine is also a plausible catastrophic-backtracking (ReDoS) vector. That was deliberately not tested against the live production API to avoid degrading service — recorded as an unverified concern for maintainers to assess, not a confirmed finding.
Definition of done
Utility mode stops interpolating name into a regex engine unescaped. Either route it through the existing .escape_regex() (one-line change, matching basket mode) or switch to a literal substring match (parameterised LIKE/ILIKE with %-wrapping and proper escaping of %/_).
A malformed or exotic search string returns zero rows, not an error — nothing a user can type into a name box should reach the matching engine as a pattern.
?cog_gov_search's documentation is updated: its @details currently promises "matches the regex case-insensitively," which will no longer be true, and the basket-mode step-3 description ("Substring fallback: case-insensitive regex") should be restated in terms of the literal semantics both modes then share.
Test goes green: tests/testthat/test-gov-search-literal-match.R → test_that("cog_gov_search() matches name literally, not as an unescaped regex", ...). Against the fixture it asserts cog_gov_search("FREDONIA (BRISCOE) CITY") returns exactly one row with canonical_govid == "052117184386"; that cog_gov_search(".", state = "WI", type = "city") returns 0 rows (no Wisconsin city name contains a literal period — verified against the raw registry, not through the verb under test); and that cog_gov_search("[") returns 0 rows without raising. Remove the skip() on line 1 of the test body to activate.
(The test does not assert on q=St. Louis. Under correct literal matching that search still returns 0 rows, because the stored name is ST LOUIS CITY with no period — the finding's St. Louis example demonstrates today's over-matching semantics, not a row the fix will make findable. Noted in the test file.)
Cross-reference
cog-api owns two follow-ups: documenting q (name and matching semantics) on both self-description surfaces — finding F-002 — and mapping any residual malformed-input error to a 400 rather than a 500, as defence in depth once this fix removes the cause.
Severity: high. Verdict: defect.
Filed from the **Madison walkthrough audit** (`cog_explorer/docs/walkthroughs/FINDINGS.md`, 2026-07-28). Verdict: **defect**.
## Root cause
`cog_gov_search()`'s **utility mode** interpolates the caller's `name` straight into a DuckDB regex match, unescaped (`R/search.R:102`):
```r
sprintf("regexp_matches(gov_name, %s, 'i')", .sql_lit_chr(name))
```
The **basket mode** in the same file already does the right thing (`R/search.R:307`):
```r
sprintf("regexp_matches(gov_name, %s, 'i')", .sql_lit_chr(.escape_regex(name)))
```
with the comment *"Backslash-escape POSIX regex metacharacters so `name` is treated as a literal substring."* The fix already exists in this file; utility mode just doesn't call it.
Two distinct failure modes result, and they must not be conflated:
* **Correctness** — a real government cannot be found by its own exact name, and a wildcard matches everything. HTTP 200 in both directions, silently wrong.
* **Robustness** — syntactically malformed regex reaches the engine and errors, which the API surfaces as a 500.
## Findings resolved
| Finding | Severity | Summary |
|---|---|---|
| **F-025** | high | `q` is interpolated unescaped into a regex match, not a substring match; a real government cannot be found by its own exact name, and ordinary typing can 500 the endpoint |
## Reproduction (verbatim from FINDINGS.md, verified against the deployed API)
```bash
B=https://cog-api.civilytics.org/api/v1
# Regex semantics, not substring semantics
curl -s -G "$B/governments" --data-urlencode "q=Madison" \
--data-urlencode "state=WI" --data-urlencode "type=city" | jq -c '.meta.total'
# 1 (baseline)
curl -s -G "$B/governments" --data-urlencode "q=." \
--data-urlencode "state=WI" --data-urlencode "type=city" | jq -c '.meta.total'
# 608 -- a single "." matches every WI city, the same failure shape as F-001
# Silent wrongness on a real, unscoped name containing "."
curl -s -G "$B/governments" --data-urlencode "q=St. Louis" | jq -c '.meta.total'
# 0 -- HTTP 200, not a crash; St. Louis exists in the corpus
# A real government whose own name contains parentheses cannot be found by it
curl -s -G "$B/governments" --data-urlencode "q=FREDONIA (BRISCOE) CITY" | jq -c '.meta.total'
# 0 -- the government's exact, complete name, typed verbatim
curl -s -G "$B/governments" --data-urlencode "q=FREDONIA" | jq -c '.data[] | select(.gov_name | contains("BRISCOE"))'
# {"gov_name":"FREDONIA (BRISCOE) CITY","canonical_govid":"052117184386","fips_state":"05", ...}
# -- found ONLY by deleting the parenthesized part of its own name
# Malformed regex reached through ordinary typing, not just a bare "["
curl -s -G "$B/governments" --data-urlencode "q=[" -w '\n%{http_code}\n'
# {"status":"error","error":"Internal server error."} 500
curl -s -G "$B/governments" --data-urlencode "q=Athens-Clarke County (bal" -w '\n%{http_code}\n'
# {"status":"error","error":"Internal server error."} 500 -- reached
# partway through typing ATHENS-CLARKE COUNTY (BALANCE), a real government's name
```
Six probes against `state=WI&type=city` pin the semantics as regex, not substring: `q=.` = 608; `q=M.dison` = 1; `q=^MADISON` = 1; `q=MADISON$` = **0** (the stored name is `MADISON CITY`); `q=Mad(i|o)son` = 1.
**Reproduces directly on the bundled fixture corpus** through the R verb, which is what the test asserts against:
```r
cog_gov_search(name = ".", state = "WI", type = "city") # 608 rows; no WI city name contains a literal "."
cog_gov_search(name = "FREDONIA (BRISCOE) CITY") # 0 rows; the government is 052117184386
cog_gov_search(name = "[") # Error: Invalid Input Error: missing ]: [
```
## Why it matters
`cog_gov_search()` is step one of essentially every workflow in this package and in the API built on it. The Fredonia case is the strongest evidence in the finding, and needs no regex literacy to read: **a database returning zero results for a government's own on-file name, typed verbatim, is broken.** The robustness half is reachable by nothing more adversarial than typing a real government's name one character at a time — a type-ahead client querying per keystroke would 500 on every parenthesized name, then find nothing once the name is complete.
Unvalidated input reaching a regex engine is also a plausible catastrophic-backtracking (ReDoS) vector. That was **deliberately not tested** against the live production API to avoid degrading service — recorded as an unverified concern for maintainers to assess, not a confirmed finding.
## Definition of done
1. Utility mode stops interpolating `name` into a regex engine unescaped. Either route it through the existing `.escape_regex()` (one-line change, matching basket mode) or switch to a literal substring match (parameterised `LIKE`/`ILIKE` with `%`-wrapping and proper escaping of `%`/`_`).
2. A malformed or exotic search string returns **zero rows, not an error** — nothing a user can type into a name box should reach the matching engine as a pattern.
3. `?cog_gov_search`'s documentation is updated: its `@details` currently promises "matches the regex case-insensitively," which will no longer be true, and the basket-mode step-3 description ("Substring fallback: case-insensitive regex") should be restated in terms of the literal semantics both modes then share.
4. Test goes green: **`tests/testthat/test-gov-search-literal-match.R`** → `test_that("cog_gov_search() matches name literally, not as an unescaped regex", ...)`. Against the fixture it asserts `cog_gov_search("FREDONIA (BRISCOE) CITY")` returns exactly one row with `canonical_govid == "052117184386"`; that `cog_gov_search(".", state = "WI", type = "city")` returns **0** rows (no Wisconsin city name contains a literal period — verified against the raw registry, not through the verb under test); and that `cog_gov_search("[")` returns 0 rows without raising. Remove the `skip()` on line 1 of the test body to activate.
*(The test does **not** assert on `q=St. Louis`. Under correct literal matching that search still returns 0 rows, because the stored name is `ST LOUIS CITY` with no period — the finding's `St. Louis` example demonstrates today's over-matching semantics, not a row the fix will make findable. Noted in the test file.)*
## Cross-reference
**`cog-api`** owns two follow-ups: documenting `q` (name and matching semantics) on both self-description surfaces — finding F-002 — and mapping any residual malformed-input error to a 400 rather than a 500, as defence in depth once this fix removes the cause.
**Severity: high. Verdict: defect.**
Cross-repo note: when this lands, two API documentation surfaces need a one-sentence deletion.
cog-api PR #18 documents q= on /governments for the first time (cog-api#11). Since cog_gov_search() utility mode still matches gov_name as an unescaped regex, both surfaces carry a caveat:
Caveat: the value is currently interpreted as a regular expression, so metacharacters ((, ), [, ., |, *) are significant and a name containing them may over-match or error; prefer a plain word until that is fixed.
Documenting the post-fix semantics before the fix would have made the docs lie about the surface a caller actually meets, so the caveat is deliberate and temporary.
When this issue closes, delete that sentence from both:
api/data-dictionary.md — the q row in ### Filters
api/llms.txt — the START HERE block
The rest of the wording (case-insensitive substring against gov_name, not anchored, not exact, ordered by ACS population descending) already describes the post-fix behaviour and needs no change.
Cross-repo note: when this lands, two API documentation surfaces need a one-sentence deletion.
cog-api PR #18 documents `q=` on `/governments` for the first time (cog-api#11). Since `cog_gov_search()` utility mode still matches `gov_name` as an **unescaped regex**, both surfaces carry a caveat:
> **Caveat:** the value is currently interpreted as a regular expression, so metacharacters (`(`, `)`, `[`, `.`, `|`, `*`) are significant and a name containing them may over-match or error; prefer a plain word until that is fixed.
Documenting the post-fix semantics before the fix would have made the docs lie about the surface a caller actually meets, so the caveat is deliberate and temporary.
**When this issue closes, delete that sentence from both:**
- `api/data-dictionary.md` — the `q` row in `### Filters`
- `api/llms.txt` — the `START HERE` block
The rest of the wording (case-insensitive substring against `gov_name`, not anchored, not exact, ordered by ACS population descending) already describes the post-fix behaviour and needs no change.
Taking this over. No kodor activity since assignment on 2026-07-29 — no branch, no PR, no comment — so I am picking it up rather than leaving it queued. Reassign back if kodor is simply slow and you would rather it land there; I will not push until this repo is otherwise quiet.
Being fixed together with the other two kodor/fix issues in one PR: they are all single-file, all have committed acceptance tests, and two of them touch the same documentation surfaces.
Taking this over. No kodor activity since assignment on 2026-07-29 — no branch, no PR, no comment — so I am picking it up rather than leaving it queued. Reassign back if kodor is simply slow and you would rather it land there; I will not push until this repo is otherwise quiet.
Being fixed together with the other two `kodor/fix` issues in one PR: they are all single-file, all have committed acceptance tests, and two of them touch the same documentation surfaces.
With name escaped, those now search for the literal characters ^ and $ and return nothing. Both were updated to the bare names, which still resolve to exactly one row each once scoped by state/type — verified against the fixture, not assumed.
That is fine for those two call sites, but the general capability is gone: utility mode is now substring-only, so cog_gov_search("BROWARD COUNTY") would also match a hypothetical NORTH BROWARD COUNTY .... Basket mode is unaffected — it still does an exact case-insensitive equality pass first, then falls back to substring.
If exact matching should stay reachable from utility mode, the clean shape is an explicit argument (match = c("substring", "exact")) rather than a return to regex, which is what produced the 500s and the silent over-matching in the first place. Say the word and I will add it; leaving it out for now, since the issue asked for literal matching and nothing in the package depends on the anchors any more.
Fixed in PR #22. One consequence the issue did not anticipate, flagged rather than buried:
**Anchored exact-match searches stop working, and utility mode no longer offers any exact-match option.**
Two existing tests used regex anchors as their exact-match idiom:
```r
cog_gov_search("^BROWARD COUNTY$", state = "FL", type = "county")
cog_gov_search("^FLORIDA$", type = "state")
```
With `name` escaped, those now search for the literal characters `^` and `$` and return nothing. Both were updated to the bare names, which still resolve to exactly one row each once scoped by `state`/`type` — verified against the fixture, not assumed.
That is fine for those two call sites, but the general capability is gone: utility mode is now substring-only, so `cog_gov_search("BROWARD COUNTY")` would also match a hypothetical `NORTH BROWARD COUNTY ...`. Basket mode is unaffected — it still does an exact case-insensitive equality pass first, then falls back to substring.
If exact matching should stay reachable from utility mode, the clean shape is an explicit argument (`match = c("substring", "exact")`) rather than a return to regex, which is what produced the 500s and the silent over-matching in the first place. Say the word and I will add it; leaving it out for now, since the issue asked for literal matching and nothing in the package depends on the anchors any more.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Filed from the Madison walkthrough audit (
cog_explorer/docs/walkthroughs/FINDINGS.md, 2026-07-28). Verdict: defect.Root cause
cog_gov_search()'s utility mode interpolates the caller'snamestraight into a DuckDB regex match, unescaped (R/search.R:102):The basket mode in the same file already does the right thing (
R/search.R:307):with the comment "Backslash-escape POSIX regex metacharacters so
nameis treated as a literal substring." The fix already exists in this file; utility mode just doesn't call it.Two distinct failure modes result, and they must not be conflated:
Findings resolved
qis interpolated unescaped into a regex match, not a substring match; a real government cannot be found by its own exact name, and ordinary typing can 500 the endpointReproduction (verbatim from FINDINGS.md, verified against the deployed API)
Six probes against
state=WI&type=citypin the semantics as regex, not substring:q=.= 608;q=M.dison= 1;q=^MADISON= 1;q=MADISON$= 0 (the stored name isMADISON CITY);q=Mad(i|o)son= 1.Reproduces directly on the bundled fixture corpus through the R verb, which is what the test asserts against:
Why it matters
cog_gov_search()is step one of essentially every workflow in this package and in the API built on it. The Fredonia case is the strongest evidence in the finding, and needs no regex literacy to read: a database returning zero results for a government's own on-file name, typed verbatim, is broken. The robustness half is reachable by nothing more adversarial than typing a real government's name one character at a time — a type-ahead client querying per keystroke would 500 on every parenthesized name, then find nothing once the name is complete.Unvalidated input reaching a regex engine is also a plausible catastrophic-backtracking (ReDoS) vector. That was deliberately not tested against the live production API to avoid degrading service — recorded as an unverified concern for maintainers to assess, not a confirmed finding.
Definition of done
nameinto a regex engine unescaped. Either route it through the existing.escape_regex()(one-line change, matching basket mode) or switch to a literal substring match (parameterisedLIKE/ILIKEwith%-wrapping and proper escaping of%/_).?cog_gov_search's documentation is updated: its@detailscurrently promises "matches the regex case-insensitively," which will no longer be true, and the basket-mode step-3 description ("Substring fallback: case-insensitive regex") should be restated in terms of the literal semantics both modes then share.tests/testthat/test-gov-search-literal-match.R→test_that("cog_gov_search() matches name literally, not as an unescaped regex", ...). Against the fixture it assertscog_gov_search("FREDONIA (BRISCOE) CITY")returns exactly one row withcanonical_govid == "052117184386"; thatcog_gov_search(".", state = "WI", type = "city")returns 0 rows (no Wisconsin city name contains a literal period — verified against the raw registry, not through the verb under test); and thatcog_gov_search("[")returns 0 rows without raising. Remove theskip()on line 1 of the test body to activate.(The test does not assert on
q=St. Louis. Under correct literal matching that search still returns 0 rows, because the stored name isST LOUIS CITYwith no period — the finding'sSt. Louisexample demonstrates today's over-matching semantics, not a row the fix will make findable. Noted in the test file.)Cross-reference
cog-apiowns two follow-ups: documentingq(name and matching semantics) on both self-description surfaces — finding F-002 — and mapping any residual malformed-input error to a 400 rather than a 500, as defence in depth once this fix removes the cause.Severity: high. Verdict: defect.
Cross-repo note: when this lands, two API documentation surfaces need a one-sentence deletion.
cog-api PR #18 documents
q=on/governmentsfor the first time (cog-api#11). Sincecog_gov_search()utility mode still matchesgov_nameas an unescaped regex, both surfaces carry a caveat:Documenting the post-fix semantics before the fix would have made the docs lie about the surface a caller actually meets, so the caveat is deliberate and temporary.
When this issue closes, delete that sentence from both:
api/data-dictionary.md— theqrow in### Filtersapi/llms.txt— theSTART HEREblockThe rest of the wording (case-insensitive substring against
gov_name, not anchored, not exact, ordered by ACS population descending) already describes the post-fix behaviour and needs no change.Taking this over. No kodor activity since assignment on 2026-07-29 — no branch, no PR, no comment — so I am picking it up rather than leaving it queued. Reassign back if kodor is simply slow and you would rather it land there; I will not push until this repo is otherwise quiet.
Being fixed together with the other two
kodor/fixissues in one PR: they are all single-file, all have committed acceptance tests, and two of them touch the same documentation surfaces.Fixed in PR #22. One consequence the issue did not anticipate, flagged rather than buried:
Anchored exact-match searches stop working, and utility mode no longer offers any exact-match option.
Two existing tests used regex anchors as their exact-match idiom:
With
nameescaped, those now search for the literal characters^and$and return nothing. Both were updated to the bare names, which still resolve to exactly one row each once scoped bystate/type— verified against the fixture, not assumed.That is fine for those two call sites, but the general capability is gone: utility mode is now substring-only, so
cog_gov_search("BROWARD COUNTY")would also match a hypotheticalNORTH BROWARD COUNTY .... Basket mode is unaffected — it still does an exact case-insensitive equality pass first, then falls back to substring.If exact matching should stay reachable from utility mode, the clean shape is an explicit argument (
match = c("substring", "exact")) rather than a return to regex, which is what produced the 500s and the silent over-matching in the first place. Say the word and I will add it; leaving it out for now, since the issue asked for literal matching and nothing in the package depends on the anchors any more.