docs: re-measure the corpus-access table against the published corpus (#56)

Conversation 0 Commits 2 Files Changed 2

+94 -46

2 Commits

Author	SHA1	Message	Date
jared	303aa59b07	docs: order the 0.4.0 NEWS sections by user impact, not merge order R-CMD-check / check (push) Successful in 3m58s Details R-CMD-check / check (pull_request) Successful in 3m38s Details The three 0.4.0 features landed in the order their PRs merged, which buried the headline change (cohort predicates, 4.8x) below an operator config knob and a documentation note. Reordered to: cohorts, pagination, DuckDB budget, docs, Fixes -- with Fixes last, where it was already. The stacked PRs each appended their own '## Fixes' heading, so resolving the conflicts also folded two of them into the single section that belongs there. Content is byte-identical to what merged; only section order changed. Verified by diffing the sorted non-blank lines against the previous commit.	2026-08-10 19:29:43 -04:00
jared	81f72321ee	docs: re-measure the corpus-access table against the published corpus (#56 ) figures predate the row-group rechunk (cog_pipeline#93, published 2026-08-09) and reported the mirrored column as 'local speed' with no number -- hiding the largest difference available to a user. Measured 2026-08-10, fresh R session per arm, against the live corpus at pipeline_commit 3d28ddd. Madison WI, 16-core Linux workstation. Three findings the old table could not express: - A local mirror is 60-80x faster. A one-off question is ~12 s end to end remotely against ~0.15 s mirrored. Stated outright now, because it is a bigger and cheaper win for users than anything in the R code. - Opening the session is the LARGEST remote cost (~7.5 s), bigger than any individual query, and it lands on the user's first query rather than on library(). The old table accounted for it nowhere, so every per-query figure was quietly missing it. - The remote cost is round-trips, not scanning: a repeat query over already-touched partitions is ~1.5 s against ~4 s cold, and a full-history query costs ~7 s whether it runs first or last (verified by running the arms in both orders). This is why #93's 1.4-1.7x, measured through cog-api against a local mount, does not show up on the remote path -- there, network latency swamps scan time. Corpus size corrected to ~201 MB: row-group chunking added ~3.4%, and 190.6 was ambiguous between MB and MiB besides. Measured from the manifest and on disk. The 0.3.0 NEWS section keeps 190.6 -- it was correct for that release. Also documented HTTP 429: a burst of remote queries gets rate-limited by the host. Hit while taking these measurements.	2026-08-10 19:29:05 -04:00

Author

SHA1

Message

Date

jared

303aa59b07

docs: order the 0.4.0 NEWS sections by user impact, not merge order

R-CMD-check / check (push) Successful in 3m58s

Details

R-CMD-check / check (pull_request) Successful in 3m38s

Details

The three 0.4.0 features landed in the order their PRs merged, which buried the
headline change (cohort predicates, 4.8x) below an operator config knob and a
documentation note. Reordered to: cohorts, pagination, DuckDB budget, docs,
Fixes -- with Fixes last, where it was already.

The stacked PRs each appended their own '## Fixes' heading, so resolving the
conflicts also folded two of them into the single section that belongs there.

Content is byte-identical to what merged; only section order changed. Verified
by diffing the sorted non-blank lines against the previous commit.

2026-08-10 19:29:43 -04:00

jared

81f72321ee

docs: re-measure the corpus-access table against the published corpus (#56 )

figures predate the row-group rechunk (cog_pipeline#93, published 2026-08-09)
and reported the mirrored column as 'local speed' with no number -- hiding the
largest difference available to a user.

Measured 2026-08-10, fresh R session per arm, against the live corpus at
pipeline_commit 3d28ddd. Madison WI, 16-core Linux workstation.

Three findings the old table could not express:

- A local mirror is 60-80x faster. A one-off question is ~12 s end to end
  remotely against ~0.15 s mirrored. Stated outright now, because it is a
  bigger and cheaper win for users than anything in the R code.

- Opening the session is the LARGEST remote cost (~7.5 s), bigger than any
  individual query, and it lands on the user's first query rather than on
  library(). The old table accounted for it nowhere, so every per-query figure
  was quietly missing it.

- The remote cost is round-trips, not scanning: a repeat query over
  already-touched partitions is ~1.5 s against ~4 s cold, and a full-history
  query costs ~7 s whether it runs first or last (verified by running the arms
  in both orders). This is why #93's 1.4-1.7x, measured through cog-api against
  a local mount, does not show up on the remote path -- there, network latency
  swamps scan time.

Corpus size corrected to ~201 MB: row-group chunking added ~3.4%, and 190.6 was
ambiguous between MB and MiB besides. Measured from the manifest and on disk.
The 0.3.0 NEWS section keeps 190.6 -- it was correct for that release.

Also documented HTTP 429: a burst of remote queries gets rate-limited by the
host. Hit while taking these measurements.

2026-08-10 19:29:05 -04:00

docs: re-measure the corpus-access table against the published corpus (#56) #65