cog_open() connected with a bare dbConnect() and set no resource pragmas, so DuckDB
claimed every visible core. Right for one interactive session on a dedicated machine;
wrong for a server, where cog-api runs two replicas on an 8-core host budgeted 4 and each
replica independently claims all 8.
What changed
USCOGDATA_DUCKDB_THREADS and USCOGDATA_DUCKDB_MEMORY_LIMIT resolve through .cfg(),
inheriting the env var > option > default precedence USCOGDATA_URL already had, and are
applied as pragmas when the connection is created.
Unset issues no pragma at all, so an unconfigured session is byte-identical to before.
That negative property is the load-bearing one, so it is asserted directly against a
connection opened the pre-change way rather than against a hardcoded core count — a
literal would fail on a machine with a different core count and pass for the wrong reason
on this one.
Two things worth knowing
.cfg() returns an environment variable as character. Both resolvers coerce and
validate rather than trusting the type: sprintf("SET threads TO %d", "4") would abort
inside the connection path with an error naming the pragma instead of the setting the
operator got wrong.
memory_limit is shape-checked, not parsed. DuckDB owns the unit vocabulary, and
re-implementing that parse would be a second definition free to drift from the engine's.
The check exists to stop a SQL fragment reaching the connection as one; an unrecognised
unit surfaces as DuckDB's own error, which names the setting correctly.
What this replaces
cog-api reaches into this namespace at boot:
con<-getFromNamespace(".ensure_session","uscogdata")()DBI::dbExecute(con,sprintf("SET threads TO %d",n))
That depends on a private name and on the session already being open. A consumer
depending on an internal is exactly what should not be frozen at a public release, which
is why this lands before the tag rather than after.
The workaround in cog-api can be deleted once this is merged and the reader is
reinstalled there — note install_local() no-ops on a version match, so that needs force = TRUE.
Verification
Full suite: 1054 passed, 0 failed, 0 warnings, 2 pre-existing test-live-corpus.R
skips (the suite runs against the bundled fixture).
R CMD check --as-cran: 0 errors, 0 warnings, 1 NOTE — the pre-existing one
(custom MinCorpusSchema/MaxCorpusSchema fields, and the r-universe URL 404ing
because registration is #47 itself).
Measured cost of capping, from the issue: ~5% (502 ms at 2 threads vs 475 ms uncapped on
16 cores) — cheap enough that a server should always cap.
Closes #60.
`cog_open()` connected with a bare `dbConnect()` and set no resource pragmas, so DuckDB
claimed every visible core. Right for one interactive session on a dedicated machine;
wrong for a server, where cog-api runs two replicas on an 8-core host budgeted 4 and each
replica independently claims all 8.
## What changed
`USCOGDATA_DUCKDB_THREADS` and `USCOGDATA_DUCKDB_MEMORY_LIMIT` resolve through `.cfg()`,
inheriting the env var > option > default precedence `USCOGDATA_URL` already had, and are
applied as pragmas when the connection is created.
**Unset issues no pragma at all**, so an unconfigured session is byte-identical to before.
That negative property is the load-bearing one, so it is asserted directly against a
connection opened the pre-change way rather than against a hardcoded core count — a
literal would fail on a machine with a different core count and pass for the wrong reason
on this one.
## Two things worth knowing
**`.cfg()` returns an environment variable as character.** Both resolvers coerce and
validate rather than trusting the type: `sprintf("SET threads TO %d", "4")` would abort
inside the connection path with an error naming the pragma instead of the setting the
operator got wrong.
**`memory_limit` is shape-checked, not parsed.** DuckDB owns the unit vocabulary, and
re-implementing that parse would be a second definition free to drift from the engine's.
The check exists to stop a SQL fragment reaching the connection as one; an unrecognised
unit surfaces as DuckDB's own error, which names the setting correctly.
## What this replaces
cog-api reaches into this namespace at boot:
```r
con <- getFromNamespace(".ensure_session", "uscogdata")()
DBI::dbExecute(con, sprintf("SET threads TO %d", n))
```
That depends on a private name **and** on the session already being open. A consumer
depending on an internal is exactly what should not be frozen at a public release, which
is why this lands before the tag rather than after.
The workaround in cog-api can be deleted once this is merged and the reader is
reinstalled there — note `install_local()` no-ops on a version match, so that needs
`force = TRUE`.
## Verification
- Full suite: **1054 passed, 0 failed, 0 warnings**, 2 pre-existing `test-live-corpus.R`
skips (the suite runs against the bundled fixture).
- `R CMD check --as-cran`: **0 errors, 0 warnings, 1 NOTE** — the pre-existing one
(custom `MinCorpusSchema`/`MaxCorpusSchema` fields, and the r-universe URL 404ing
because registration is #47 itself).
Measured cost of capping, from the issue: ~5% (502 ms at 2 threads vs 475 ms uncapped on
16 cores) — cheap enough that a server should always cap.
cog_open() connected with a bare dbConnect() and set no resource pragmas, so
DuckDB claimed every visible core. Right for one interactive session on a
dedicated machine; wrong for a server, where cog-api runs two replicas on an
8-core host budgeted 4 and each replica independently claims all 8.
USCOGDATA_DUCKDB_THREADS and USCOGDATA_DUCKDB_MEMORY_LIMIT now resolve through
.cfg() -- inheriting the env var > option > default precedence USCOGDATA_URL
already had -- and are applied as pragmas when the connection is created.
Unset issues NO pragma, so an unconfigured session is byte-identical to before.
That negative property is asserted directly against a connection opened the
pre-change way rather than against a hardcoded core count.
.cfg() returns an env var as character, so both resolvers coerce and validate
rather than trusting the type: sprintf("SET threads TO %d", "4") would
otherwise abort inside the connection path with an error naming the pragma
instead of the setting the operator got wrong.
Replaces cog-api's getFromNamespace(".ensure_session", "uscogdata") workaround,
which depended on a private name and on the session already being open.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #60.
cog_open()connected with a baredbConnect()and set no resource pragmas, so DuckDBclaimed every visible core. Right for one interactive session on a dedicated machine;
wrong for a server, where cog-api runs two replicas on an 8-core host budgeted 4 and each
replica independently claims all 8.
What changed
USCOGDATA_DUCKDB_THREADSandUSCOGDATA_DUCKDB_MEMORY_LIMITresolve through.cfg(),inheriting the env var > option > default precedence
USCOGDATA_URLalready had, and areapplied as pragmas when the connection is created.
Unset issues no pragma at all, so an unconfigured session is byte-identical to before.
That negative property is the load-bearing one, so it is asserted directly against a
connection opened the pre-change way rather than against a hardcoded core count — a
literal would fail on a machine with a different core count and pass for the wrong reason
on this one.
Two things worth knowing
.cfg()returns an environment variable as character. Both resolvers coerce andvalidate rather than trusting the type:
sprintf("SET threads TO %d", "4")would abortinside the connection path with an error naming the pragma instead of the setting the
operator got wrong.
memory_limitis shape-checked, not parsed. DuckDB owns the unit vocabulary, andre-implementing that parse would be a second definition free to drift from the engine's.
The check exists to stop a SQL fragment reaching the connection as one; an unrecognised
unit surfaces as DuckDB's own error, which names the setting correctly.
What this replaces
cog-api reaches into this namespace at boot:
That depends on a private name and on the session already being open. A consumer
depending on an internal is exactly what should not be frozen at a public release, which
is why this lands before the tag rather than after.
The workaround in cog-api can be deleted once this is merged and the reader is
reinstalled there — note
install_local()no-ops on a version match, so that needsforce = TRUE.Verification
test-live-corpus.Rskips (the suite runs against the bundled fixture).
R CMD check --as-cran: 0 errors, 0 warnings, 1 NOTE — the pre-existing one(custom
MinCorpusSchema/MaxCorpusSchemafields, and the r-universe URL 404ingbecause registration is #47 itself).
Measured cost of capping, from the issue: ~5% (502 ms at 2 threads vs 475 ms uncapped on
16 cores) — cheap enough that a server should always cap.
cog_open() connected with a bare dbConnect() and set no resource pragmas, so DuckDB claimed every visible core. Right for one interactive session on a dedicated machine; wrong for a server, where cog-api runs two replicas on an 8-core host budgeted 4 and each replica independently claims all 8. USCOGDATA_DUCKDB_THREADS and USCOGDATA_DUCKDB_MEMORY_LIMIT now resolve through .cfg() -- inheriting the env var > option > default precedence USCOGDATA_URL already had -- and are applied as pragmas when the connection is created. Unset issues NO pragma, so an unconfigured session is byte-identical to before. That negative property is asserted directly against a connection opened the pre-change way rather than against a hardcoded core count. .cfg() returns an env var as character, so both resolvers coerce and validate rather than trusting the type: sprintf("SET threads TO %d", "4") would otherwise abort inside the connection path with an error naming the pragma instead of the setting the operator got wrong. Replaces cog-api's getFromNamespace(".ensure_session", "uscogdata") workaround, which depended on a private name and on the session already being open.