feat: cog_open() honours a DuckDB thread and memory budget (#60)
cog_open() connected with a bare dbConnect() and set no resource pragmas, so
DuckDB claimed every visible core. Right for one interactive session on a
dedicated machine; wrong for a server, where cog-api runs two replicas on an
8-core host budgeted 4 and each replica independently claims all 8.
USCOGDATA_DUCKDB_THREADS and USCOGDATA_DUCKDB_MEMORY_LIMIT now resolve through
.cfg() -- inheriting the env var > option > default precedence USCOGDATA_URL
already had -- and are applied as pragmas when the connection is created.
Unset issues NO pragma, so an unconfigured session is byte-identical to before.
That negative property is asserted directly against a connection opened the
pre-change way rather than against a hardcoded core count.
.cfg() returns an env var as character, so both resolvers coerce and validate
rather than trusting the type: sprintf("SET threads TO %d", "4") would
otherwise abort inside the connection path with an error naming the pragma
instead of the setting the operator got wrong.
Replaces cog-api's getFromNamespace(".ensure_session", "uscogdata") workaround,
which depended on a private name and on the session already being open.
This commit is contained in:
@@ -110,6 +110,17 @@ After that, nothing in your analysis touches an external service.
|
||||
- `USCOGDATA_URL` — corpus root: an HTTPS URL or a local path, **trailing slash required**
|
||||
- `USCOGDATA_CACHE_DIR` — where the manifest is cached (default: user cache dir)
|
||||
- `USCOGDATA_MANIFEST_TTL_SECS` — manifest re-fetch interval (default 3600)
|
||||
- `USCOGDATA_DUCKDB_THREADS` — cap DuckDB's thread count (default: every visible core)
|
||||
- `USCOGDATA_DUCKDB_MEMORY_LIMIT` — cap DuckDB's memory, e.g. `"4GB"` (default: DuckDB's own)
|
||||
|
||||
Each also has an `options()` spelling — `uscogdata.url`, `uscogdata.duckdb_threads`,
|
||||
and so on — and the environment variable wins where both are set.
|
||||
|
||||
The two DuckDB caps exist for **servers**, not laptops. Unset, DuckDB claims every
|
||||
core it can see, which is right for one interactive session on your own machine and
|
||||
wrong when several readers share a box: each claims the whole machine and they fight.
|
||||
Capping costs roughly 5% on a single query and is worth it anywhere the process is
|
||||
sharing hardware.
|
||||
|
||||
## Amounts are in full US dollars
|
||||
|
||||
|
||||
Reference in New Issue
Block a user