ci: fetch apt indexes over HTTPS so the install step stops hanging #31

2026-08-04T12:02:16-04:00

jared commented

2026-08-04 12:02:16 -04:00

The "Install system libraries" step was stalling indefinitely.

It was not a config prompt and not slow-but-progressing. Measured inside the live runner container: /var/cache/apt/archives held 0 .deb files after 3+ minutes, with apt's http workers parked in S state waiting on the network, and no process in D state.

Root cause

The http:// mirror path is pathologically slow from this runner — not broken. Measured 2026-08-04 from inside the CI container, same index file, back to back:

Scheme	Time
`http://archive.ubuntu.com/ubuntu/dists/noble/Release`	20.1s
`https://archive.ubuntu.com/ubuntu/dists/noble/Release`	3.1s

apt fetches many indexes serially, so ~20s apiece compounds into what presents as a hang.

Verified before committing

All checked inside the running rocker/r-ver:4.4 CI container, not assumed:

ca-certificates present and apt 2.8.3 ships the https method built in — no bootstrap problem, nothing needs installing over http first to enable TLS
the rewrite hits both archive and security URIs; the only remaining http:// is an inert comment line
# is the sed delimiter deliberately — | collides with the alternation and fails with unknown option to \s'` (hit and fixed during testing)
the regex survives YAML block-scalar parsing with backslashes intact
sh -n syntax-checks clean, including the missing-file path

|| true guards each sed because the step runs under sh -e, so a missing sources file on some other base image must not kill the job.

Note

The identical apt step exists in cog-api's ci.yml and will hit the same stall. Deliberately not patched here — worth proving this fix on one repo first.

The "Install system libraries" step was stalling indefinitely. **It was not a config prompt and not slow-but-progressing.** Measured inside the live runner container: `/var/cache/apt/archives` held **0 `.deb` files after 3+ minutes**, with apt's http workers parked in `S` state waiting on the network, and no process in `D` state. ## Root cause The `http://` mirror path is pathologically slow from this runner — not broken. Measured 2026-08-04 from inside the CI container, same index file, back to back: | Scheme | Time | |---|---| | `http://archive.ubuntu.com/ubuntu/dists/noble/Release` | **20.1s** | | `https://archive.ubuntu.com/ubuntu/dists/noble/Release` | **3.1s** | apt fetches many indexes serially, so ~20s apiece compounds into what presents as a hang. ## Verified before committing All checked inside the running `rocker/r-ver:4.4` CI container, not assumed: - `ca-certificates` present and apt 2.8.3 ships the https method built in — **no bootstrap problem**, nothing needs installing over http first to enable TLS - the rewrite hits both `archive` and `security` URIs; the only remaining `http://` is an inert comment line - `#` is the sed delimiter **deliberately** — `|` collides with the alternation and fails with `unknown option to \`s'` (hit and fixed during testing) - the regex survives YAML block-scalar parsing with backslashes intact - `sh -n` syntax-checks clean, including the missing-file path `|| true` guards each sed because the step runs under `sh -e`, so a missing sources file on some other base image must not kill the job. ## Note The identical apt step exists in **cog-api**'s `ci.yml` and will hit the same stall. Deliberately not patched here — worth proving this fix on one repo first.

jared added 1 commit 2026-08-04 12:02:17 -04:00

ci: fetch apt indexes over HTTPS so the install step stops hanging

R-CMD-check / check (push) Has been cancelled

Details

R-CMD-check / check (pull_request) Has been cancelled

Details

342debaefa

The "Install system libraries" step was stalling indefinitely. It was not
deadlocked on a config prompt and not slow-but-progressing: measured inside the
live runner container, /var/cache/apt/archives stayed at 0 .deb files after 3+
minutes, with apt's http workers parked in S state waiting on the network.

Root cause is the http:// mirror path being pathologically slow from this
runner, not broken. Measured 2026-08-04 from inside the CI container, same
index file, back to back:

  http://archive.ubuntu.com/ubuntu/dists/noble/Release   20.1s
  https://archive.ubuntu.com/ubuntu/dists/noble/Release    3.1s

apt fetches many indexes serially, so ~20s apiece compounds into what looks
like a hang. Rewriting the deb822 sources to https makes the step complete.

Verified before committing, in the running CI container (rocker/r-ver:4.4):
- ca-certificates present and apt 2.8.3 ships the https method built in, so
  nothing has to be installed over http first to bootstrap TLS
- the sed rewrites both URIs (archive + security); the only remaining http://
  is an inert comment line
- '#' is used as the sed delimiter deliberately: '|' collides with the
  alternation and fails with "unknown option to `s'"
- the regex survives YAML block-scalar parsing with backslashes intact

`|| true` guards each sed because the step runs under `sh -e`, so a
missing-sources-file on some other base image must not kill the job.

jared added 1 commit 2026-08-04 12:08:02 -04:00

test: accept schema_version 7, and keep the upper bound enforced

R-CMD-check / check (pull_request) Successful in 6m3s

Details

R-CMD-check / check (push) Has been cancelled

Details

e067a5930f

The merged schema-v7 fix (b59b79b) widened .validate_schema()'s allow-list but
left this test asserting that 7 is REJECTED, so main went red. CI had been
hanging on the apt step before ever reaching the suite, which is why the
failure only surfaced once the HTTPS fix let the job get that far.

Flips 7L from expect_error to expect_silent, and ADDS an 8L rejection case.
That second part is the point: simply deleting the 7L expectation would have
left the test unable to prove any upper bound is enforced at all, so a future
v8 corpus with a genuinely breaking change would pass validation silently. The
test should assert the boundary moved, not that it disappeared.

Suite: 796 PASS, 0 FAIL, 0 WARN, 0 SKIP.

jared merged commit 6cd219a291 into main

2026-08-04 12:28:49 -04:00

kodor referenced this pull request

2026-08-05 01:14:24 -04:00

fix: signpost partially-suppressed categories (#9) #32

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Civilytics/uscogdata#31