A shared cluster (shared_cluster(), shared_cluster_handle() and the
fixtures built on them) remembers a failed bootstrap for the rest of the test
process, so every later test in that binary reports the same failure. Under
cargo nextest, each test binary runs in its own process and starts its own
shared cluster. Two measures stop one network hiccup from failing a whole
binary.
Bounded retry
In releases after 0.6.0, a shared cluster retries a bootstrap that fails for a plainly transient reason:
- an I/O or repository failure while downloading or extracting the PostgreSQL archive;
- a lifecycle timeout, such as a slow first start under load;
- an extension archive that could not be downloaded
(
BootstrapErrorKind::ExtensionArchiveUnavailable).
It makes three attempts in all, one and then two seconds apart. A retry is
logged as a warn event. If every attempt fails, the error keeps its
BootstrapErrorKind and gains bootstrap failed after 3 attempts, and that
failure is the one remembered. Any other failure, such as a configuration error
or a version that does not exist, is returned at once without a retry.
Root-privileged bootstraps report worker failures as text, so they are never
retried. TestCluster::new() and TestCluster::new_split() called directly do
not retry.
A failure on binaries copied from the shared binary cache is treated as
deterministic, apart from a timeout. The cache holds an extracted installation
tree rather than an archive, so a retry would copy the same tree again rather
than download it afresh. Such an error says that the binaries came from the
cache. Remove that version's directory from the cache (see
cache.md) to force a fresh download on the next run. Extension
failures are not affected by the cache hit: they come from the extension hook,
not the cached tree, so they carry no such advice, and an extension archive
that could not be downloaded is still retried.
Warm the caches before the tests run
Run the pg_embedded_setup_unpriv binary as a CI step before the test runner.
It downloads and verifies the PostgreSQL archive and fills the shared binary
cache (see cache.md). When PG_EXTENSIONS is set, it also fills
the extension cache, so the test processes then need no network. Give it
throwaway installation and data directories, and the same cache, version and
extension settings as the test step:
- name: Warm the PostgreSQL caches
env:
PG_VERSION_REQ: "=17.4.0"
PG_BINARY_CACHE_DIR: ${{ runner.temp }}/pg-binaries
PG_RUNTIME_DIR: ${{ runner.temp }}/pg-warm/install
PG_DATA_DIR: ${{ runner.temp }}/pg-warm/data
run: pg_embedded_setup_unpriv
- name: Test
env:
PG_VERSION_REQ: "=17.4.0"
PG_BINARY_CACHE_DIR: ${{ runner.temp }}/pg-binaries
run: cargo nextest run
Install the binary at the version of the library the tests use, for example with
cargo binstall pg-embed-setup-unpriv@<version>. Cache population is best
effort: it is skipped when the cache lock cannot be taken. To prove the tests
will not download, assert that the cache directory holds a complete entry after
the warm step.