Circuit breakers

Retries help with brief failures. When a dependency is down for longer, retrying every call only adds load and latency. A circuit breaker notices repeated failures, rejects calls immediately for a while, then lets a single probe call test whether the dependency has recovered.

Usage#

from relprim import CircuitBreaker, resilient

provider_breaker = CircuitBreaker(
    name="primary_provider",
    failure_threshold=5,
    recovery_timeout_seconds=30,
)


@resilient(retries=2, timeout=10, circuit_breaker=provider_breaker, fallback=call_backup)
async def call_provider(prompt: str) -> str:
    return await provider.generate(prompt)

In the builder API, use .with_circuit_breaker(provider_breaker).

A breaker is a shared, stateful object. Create one per dependency and pass the same instance to every operation that should share failure accounting. Breakers compare by identity.

Field	Default	Rule
name	"default"	Non-blank; appears in errors and events.
failure_threshold	5	int of at least 1.
recovery_timeout_seconds	30.0	Finite, greater than 0.
record_failure_on	(Exception,)	Exception subclasses that count as failures.
clock	time.monotonic	Injectable for tests.

States#

State	Calls	Leaves the state when
closed	Allowed. Recorded failures are counted; a success resets the count.	The count reaches failure_threshold: the breaker opens.
open	Rejected immediately with CircuitBreakerOpenError.	recovery_timeout_seconds have passed since it opened: the next call becomes the probe.
half_open	One probe call is admitted; others are rejected while it runs.	The probe succeeds (closed) or fails with a recorded failure (open again, with a new open time).

1. Closed. Each recorded failure increments the failure count; each success resets it to zero. At failure_threshold consecutive failures the breaker opens.
2. Open. Every call fails immediately. The error’s retry_after_seconds says how long until the recovery timeout elapses.
3. Half-open. The first call after the recovery timeout is the probe. While it runs, other calls are rejected with retry_after_seconds set to None.
4. Probe outcome. Success closes the breaker and clears the count. A recorded failure opens it again.

Which failures count#

Inside an operation, each primary attempt is one breaker call, including its timeout and validation. So an OperationTimeoutError or a ValidationFailedError counts as a failure, as long as it matches record_failure_on.

Exceptions that do not match record_failure_on are neither failures nor successes: they leave the count unchanged. If the probe raises such an exception, the probe slot is released and the next call probes again.

Fallback candidates run outside the breaker and never affect it.

Rejected calls#

A rejected attempt fails with CircuitBreakerOpenError:

Attribute	Meaning
breaker_name	The breaker’s name.
state	A CircuitBreakerState, open or half_open. It is a string enum, so state == "open" works.
retry_after_seconds	Seconds until the recovery timeout elapses, or None while a probe is in flight.

The rejection is an ordinary attempt failure. With the default retry_on it is retried, and with the default FallbackPolicy it triggers the fallback. That makes an open breaker plus a fallback a cheap way to route around an outage without waiting for timeouts. To fail fast instead, leave CircuitBreakerOpenError out of retry_on.

With structured events, each rejection emits circuit_breaker.rejected, and report metadata sets circuit_breaker_open when the final attempt was rejected.

Cancellation#

A cancelled call is neither a success nor a failure. If a half-open probe is cancelled, its probe slot is released immediately, so the breaker cannot be left waiting for a probe that will never finish.

Snapshots and reset#

snapshot = await provider_breaker.snapshot()

snapshot.state          # CircuitBreakerState.CLOSED
snapshot.failure_count  # 0
snapshot.open           # False

await provider_breaker.reset()   # back to closed, count cleared

snapshot() returns an immutable, consistent CircuitBreakerSnapshot with name, state, failure_count, failure_threshold, opened_at, recovery_timeout_seconds and half_open_probe_in_flight, plus the open, closed and half_open properties. Both methods are coroutines.

Scope#

The breaker’s state lives in one process and one event loop. It protects that process from a failing dependency; it does not coordinate breakers across processes or hosts.

await breaker.run_async(operation, *args) runs a call through the breaker outside an operation, raising CircuitBreakerOpenError when it is rejected.