Skip to content
GitHub Get started
Engineering plans · Internals

Internals

Catalogue and builders construct; policy authorizes; the runtime executes; observers report. This is how Cuprum keeps each of those boundaries legible.

Describes Cuprum 0.2.0-beta2. Internal module names can change without notice; the public API is in the documentation.

Inspect the lifecycle Show me the failure path The Rust extension
Plate 04 In the drafting room, Copper checks a tee on the plan. The figures below are the plan; the drawing is only a drawing.

01 — Division of labour

Four jobs, four owners

Catalogue and builders construct. Policy authorizes. The runtime executes. Observers report. No component does another’s job.

Fig. 01

Who constructs, who authorizes, who runs, and who reports?

  1. Catalogue ProgramCatalogue Which programs exist, grouped by project, with noise rules and documentation links.
    passes Program to the next stage
  2. Command or builder sh.make → SafeCmd An immutable command: program, argv tuple, project metadata. The catalogue is the one named, else the scope’s, else the default.
    passes SafeCmd to the next stage
  3. Authorization boundary CuprumContext scope The allowlist in force for this thread or task, plus hooks, environment policy, executable bindings, and defaults.
    passes authorized SafeCmd to the next stage
  4. Runtime run() / run_sync() Spawns without a shell, pumps streams, enforces timeouts, tears down.
    passes status, bytes to the next stage
  5. Result CommandResult ok, exit code, captured output, pid, timing, resource use, and the path that ran if a binding chose it.
  • Observers, throughout Before and after hooks, sh.observe event hooks, control and echo observers, telemetry adapters, and presentation sinks receive events from the runtime. They report; they do not route.
Construction and authorization are separate: the catalogue decides what can be built, and the scope in force decides what may run here. Source: docs/cuprum-design.md §§4–6 at v0.2.0-beta2

Stated invariants

What holds, every time

Each line names where it is enforced. If the code and this list ever disagree, the code is right and this page is out of date.

  1. Nothing reaches a shell. Every spawn passes an argv tuple with shell=False. cuprum/_wait4_process.py, cuprum/_pipeline_spawn.py
  2. A command exists only for a catalogued program. sh.make resolves an explicit catalogue, else the scope’s, else the default, and raises UnknownProgramError when it does not list the program. cuprum/sh/factory.py, cuprum/catalogue.py
  3. Authorization happens at run time, before any spawn. The scope in force is checked for a command, and for every stage of a pipeline before the first stage starts. cuprum/context/, cuprum/_pipeline_internals.py
  4. Scopes only narrow. A nested scoped() intersects its allowlist with its parent’s. Widening is a separate, explicit call to allow(). cuprum/context/
  5. A binding never widens the allowlist. bind_executable decides only what a permitted program executes; resolution happens after the allowlist check, and a refused program never calls its resolver. cuprum/context/, cuprum/executable_binding.py
  6. Every spawned child is reaped. Cancellation, timeouts, and terminate() on a handle escalate from SIGTERM to SIGKILL and wait for the exit, even under repeated cancellation. Only a run that opts into OWN_GROUP signals a process group, and only the group its spawn created. cuprum/_process_lifecycle.py, cuprum/_group_teardown.py
  7. Capture is complete. Truncating an echoed line, or disabling echo for a sink that cannot encode the text, never removes bytes from the result. cuprum/_stream_echo.py
  8. Python is the behavioural reference. The optional Rust pump must produce identical bytes, and a hop it cannot take runs on the Python pump instead. cuprum/_pipeline_streams.py

02 — Execution lifecycle

One command, start to finish

What happens, in order, between calling a builder and holding a CommandResult — and what happens when the run is cut short.

Fig. 02

What happens between a builder call and a result?

  1. 1 · Construction sh.make(PROGRAM)(*args, **flags) UnknownProgramError if the resolved catalogue does not list the program. Arguments must be ArgValue: str, int, float, bool, or a path; keywords become --flag=value; None, bytes, and a keyword naming an execution option are rejected.
    passes SafeCmd to the next stage
  2. 2 · Authorization scope check ForbiddenProgramError if the allowlist in force excludes the program. Nothing has been spawned and no hook has run.
    passes authorized to the next stage
  3. 3 · Plan plan event, then before hooks Observers see the resolved argv, cwd, environment, and environment mode; before hooks then run in registration order. Overlays apply over the live os.environ unless the mode is REPLACE, and an explicit timeout beats ExecutionContext, which beats the scope default.
    passes argv, env, cwd to the next stage
  4. 4 · Spawn exec, no shell The argv tuple goes straight to the operating system, with the executable an active binding resolved for the program, if any. On Linux and macOS Cuprum owns the child’s reap so it can report its CPU time and peak memory; under OWN_GROUP the child leads its own session and process group.
    passes start event, pid to the next stage
  5. 5 · Streams capture, echo, line events stdout and stderr are drained by independent tasks. Echo mirrors bytes to sinks; capture keeps everything; line events are decoded once.
    passes stdout / stderr events to the next stage
  6. 6 · Completion exit event, after hooks, settled, result After hooks run in reverse registration order, so the innermost cleans up first. One settled event then closes the execution with its terminal outcome, whatever happened.
  • start() instead of run() The caller holds a RunningCommand after step 4 and decides when to wait, signal, terminate, or kill. The handle settles once, and leaving its async with block tears a live child down the same way.
  • Task cancelled SIGTERM, a grace period (cancel_grace, 0.5 s by default), then SIGKILL and reap. Repeated cancellation does not interrupt teardown; the first cancellation is re-raised afterwards.
  • Timeout expires The same teardown, then TimeoutExpired carrying whatever output was captured, a timeout event, and a WARNING on the cuprum.timeout logger. With process-group ownership the group is signalled, so a descendant holding a pipe open is reclaimed too.
  • An observe hook raises The run fails with the hook’s own exception type. A hook that must never affect a run catches its own errors.
A command that was never authorized is never spawned, and a spawned command is always reaped, however the run ends. Source: cuprum/sh.py, cuprum/_command_internals.py, and docs/users-guide.md at v0.2.0-beta2

03 — Pipelines and streams

Pipes, pumps, and fail-fast

How stages are joined, where bytes are copied, how backpressure works, and which stage is blamed when one fails.

Fig. 03

Which bytes go where in a three-stage pipeline?

  1. Stage 0 producer stdout → OS pipe. stderr → caller (echo, capture).
    passes bytes, with backpressure to the next stage
  2. Stage 1 filter Reads stage 0, writes stage 2. Its stdout result is None.
    passes bytes, with backpressure to the next stage
  3. Stage 2 (final) consumer Its stdout is the pipeline’s output.
    passes final stdout to the next stage
  4. Result PipelineResult stdout, stages[], failure, failure_index.
  • First non-final stage exits non-zero Every other still-running stage is terminated, upstream and downstream. The failing stage owns its own exit and is reported as failure_index.
  • A downstream stage stops reading The pump drains the upstream reader to EOF or a short timeout, so the producer is not left blocked on a full pipe.
  • Deadline reached One deadline covers the whole pipeline; each stage still gets its own timeout event and log record.
Only the final stage’s stdout reaches the caller; every stage’s stderr does. Blame goes to the first stage to fail, not the last. Source: cuprum/_pipeline_internals.py, cuprum/_pipeline_wait.py, and docs/users-guide.md at v0.2.0-beta2

Backpressure

Stages are joined by operating-system pipes. Between two stages a pump reads from one and writes to the next, awaiting drain() on each write, so a slow consumer slows its producer instead of filling memory.

Where Rust fits

That inter-stage pump is the one place the optional Rust extension is used: when it is installed and selected, the copy runs outside the GIL, using splice() on Linux. Capturing the final output always uses the Python path.

Diagnosing a fail-fast

When the first non-final failure still has stages to stop, observe hooks receive one pipeline_fail_fast event before termination begins, carrying the stage index, pipeline width, exit code, and duration. It omits argv and tags, so command-line secrets do not reach the warning it produces.

A final-stage failure, or one whose siblings have already exited, emits no such event: there was nothing left to terminate. It is still reported as failure_index.

04 — Policy and observation

Scopes, hooks, and sinks

Where the allowlist lives, how it narrows, and the channels that report on a run without changing what it does.

Where the policy lives

CuprumContext is an immutable value held in a ContextVar. Each thread gets its own; each async task inherits its creator’s and may change it independently. Entering scoped() replaces the value, and leaving the block restores the previous one, even on an exception.

At the root, with no scope established, every catalogued program may run. That permissive default eases adoption; narrowing it is how code states its policy. An explicitly empty allowlist inside a restricted scope permits nothing. A scope made from a catalogue also makes that catalogue the one sh.make consults, while a scope that only narrows the allowlist leaves it alone.

Two more policies ride on the same context. EnvMode decides whether a child inherits, overlays, or replaces the parent's environment, and an executable binding decides which file a permitted program runs, without changing the name the allowlist, metadata, and telemetry carry.

Hooks

Before hooks run in registration order, parent first. After hooks run in reverse, child first, so cleanup nests like context managers. logging_hook() pairs the two to log start and exit.

Observation

sh.observe() delivers structured ExecEvent values — plan, start, stdout, stderr, exit, one settled event with the bounded terminal outcome, and the ancillary timeout, teardown, and group-teardown events — each correlated by an exec_id rather than a pid the system may recycle. Narrower facts travel on channels of their own: what a control operation did to a child, an echo that fell back, a line stream's lifecycle, and the Rust pump's routing decisions.

The adapters in cuprum.adapters turn those events into structured logs, Prometheus-style metrics, and OpenTelemetry-style spans. They define protocols rather than importing a telemetry library, so Cuprum keeps its zero-dependency install.

Presentation sinks

An opt-in sink reframes a run’s echoed output without changing capture or the result. GitHubActionsSink wraps a run in a collapsible log group and turns a failure into one annotation; it activates only when GITHUB_ACTIONS is true, unless forced.

05 — Sources

Read the plans

Plan

Cuprum design

The design document: principles, conceptual model, async execution, pipelines, error model, and the Rust extension.

Manual

Users' guide

The public API, with every option and its failure behaviour.