RelPrim: Reliability Primitives for Async Python

External calls are easy to write and difficult to operate.

A provider integration often begins with one asynchronous function. Production requirements then accumulate around it. The call needs a timeout. Some failures should be retried. The result may need validation. A fallback may become necessary. Repeated failures should affect whether more work is admitted. Duplicate requests should not repeat the work. When the operation ends, the team needs enough evidence to understand what happened.

RelPrim is an open-source Python library I created to give those concerns a small, explicit execution model.

It provides reusable reliability mechanics without pretending that infrastructure can decide whether a business operation is safe to repeat or whether a degraded result remains valid for the product.

RelPrim 1.0 freezes the public reliability model after the pre-stable releases were used to build and harden the core primitives. This case study explains why the library exists and how it was designed. The documentation explains how to use it.

Problem

Calls that cross a process, network, or provider boundary fail in more ways than their function signatures suggest.

A remote operation may:

exceed the time the caller is willing to wait;
fail temporarily and succeed on another attempt;
be rejected by a rate limit that says when to try again;
return a response that is structurally valid but unusable;
leave the primary provider unavailable while an alternative still works;
trigger repeated failures against a dependency that is already unhealthy;
be requested twice for the same logical piece of work;
succeed only after several attempts;
produce a value without preserving any record of the path that produced it.

Applications usually address these problems gradually.

A timeout is added inside one client. Retry logic appears in a decorator elsewhere. A fallback is implemented with a broad exception handler. Validation lives in application code after the provider call. Logging records the final exception but loses the intermediate decisions.

The result is not simply duplicated code. It is duplicated reliability semantics.

Two operations that look similar may retry different exceptions, use different timeout meanings, or report failures in incompatible ways. The policy is spread across control flow, helper functions, SDK configuration, and operational conventions.

That makes the system difficult to review before failure and difficult to explain after it.

Why I built RelPrim

I created RelPrim after repeatedly seeing external integrations follow the same path.

The first implementation is usually direct:

response = await provider.generate(prompt)

That is a reasonable place to start.

The problem appears when reliability requirements are added one at a time. Individual mechanisms are not especially difficult to implement. The harder part is keeping their behavior coherent as the integration grows.

A retry should know which failures are temporary. A timeout should have a clear scope. A fallback should be identifiable in the final result. Invalid responses should not be accepted as successful merely because the transport returned 200. Operational data should describe the complete execution rather than one isolated request.

Copying these mechanics into each integration creates drift.

Hiding them inside a generic client creates a different risk. The code may claim that an operation is resilient without showing what that resilience means.

RelPrim grew from the tension between those two outcomes.

I wanted the mechanics to be reusable while keeping the policy visible in the application. A reviewer should be able to see that an operation retries, times out, validates its result, uses a fallback, or participates in circuit protection. The library should provide the execution model, but the application should remain responsible for the decisions that depend on business meaning.

The design question

The central design question was not:

How can another retry decorator be implemented?

It was:

What should an external operation return when its execution path matters as much as its value?

A normal function returns a value or raises an exception.

That interface is often too narrow for a reliability layer. Two executions can produce the same value after taking very different paths.

One may succeed on the first attempt.

Another may time out, retry, reject an invalid response, switch to a fallback, and finally produce a usable result.

Both may satisfy the immediate caller. They are not operationally equivalent.

RelPrim therefore treats the business value and the execution history as two parts of the same result.

Solution

RelPrim wraps an existing asynchronous operation rather than replacing the provider client or the application workflow around it.

Every operation runs its primitives in one fixed, documented order:

Application operation
        |
        v
Named RelPrim execution
        |
        +--> idempotency      joins or replays duplicate calls
        |
        |    for each attempt:
        +--> circuit breaker  admits or rejects the attempt
        +--> timeout          bounds the call
        +--> your callable
        +--> validation       accepts or rejects the value
        |
        +--> retry delay      backoff, or the provider's retry-after hint
        |
        +--> fallback chain   after the primary path is exhausted
        |
        +--> structured events, throughout
        |
        v
OperationResult[T]
        |
        +--> business value
        |
        +--> execution report

The application still owns the callable and its domain semantics.

RelPrim owns the reusable mechanics around its execution and preserves a structured account of what happened. Because the order never changes, a reviewer can reason about how the primitives interact without reading the library’s source. The core concepts page documents the order in detail.

This boundary keeps the library useful across provider SDKs, HTTP clients, workers, services, and orchestration code without turning it into another integration framework.

Two levels of adoption

RelPrim provides two public entry points built on the same underlying primitives.

The decorator API is intended for integrations where a concise policy is enough.

from relprim import resilient


@resilient(
    retries=3,
    timeout=10,
)
async def call_provider(prompt: str) -> str:
    return await provider.generate(prompt)


result = await call_provider("Write a short product summary")

The policy remains visible at the function boundary. Adoption does not require rewriting the provider integration around a new client abstraction, and the decorated function keeps its parameters for type checkers.

For operations that need more deliberate composition, RelPrim exposes the async_operation(...) builder.

from relprim import RetryPolicy, TimeoutPolicy, async_operation


result = await (
    async_operation("generate_response", provider.generate)
    .with_retry(RetryPolicy(max_attempts=4))
    .with_timeout(TimeoutPolicy(seconds=10))
    .run("Write a short product summary")
)

The builder gives the operation an explicit name and keeps its selected reliability mechanisms in one definition.

This is useful when a policy is assembled dynamically, uses several primitives, or needs structured lifecycle events and reporting.

Every decorator option is a shortcut for one builder method. The two APIs serve different levels of complexity without creating two separate execution models.

Named operations

The builder requires an operation name:

async_operation("generate_response", provider.generate)

This looks like a small API choice, but it reflects a broader design principle.

Reliability behavior needs a stable operational identity.

Names appear in execution reports and structured events. They allow metrics, logs, tests, and diagnostics to describe a capability rather than an anonymous callable.

A name such as:

generate_response

communicates intent.

A name such as:

call

or:

invoke

describes implementation mechanics and provides little operational value.

The operation name is not intended to contain request-specific data. It should remain stable across executions so that behavior can be aggregated and compared.

Operation results

A decorated function or builder execution returns an OperationResult[T] instead of the raw business value.

result = await call_provider(prompt)

value = result.value
report = result.report

This makes execution metadata part of the contract rather than a side effect hidden in logs.

The caller can use the business value normally while retaining access to the report for diagnostics, testing, persistence, or observability. When every recovery path fails, the same report is attached to the raised OperationExecutionError, so the failure path is as explainable as the success path.

That distinction also prevents a successful fallback from erasing the fact that the primary operation failed.

From the product perspective, the returned value may be acceptable.

From the engineering perspective, the execution path still matters. A growing fallback rate may indicate that the primary dependency is degrading long before the user-visible operation begins to fail.

The result model preserves both truths.

Execution reports

RelPrim produces structured execution reports that can be inspected or serialized.

print(result.report.to_dict())

The report provides one place to understand how the operation was executed: every attempt, its duration and outcome, whether it was a fallback, and the metadata recorded by each primitive along the way.

Its purpose is not to replace logs, metrics, or tracing. It is to preserve execution-level facts before an observability backend is chosen.

This keeps the core model transport-agnostic.

An application may:

inspect reports in tests;
attach them to structured logs;
persist selected information;
translate them into metrics;
expose them through internal diagnostic tooling;
correlate them with higher-level workflows.

The library does not require every consumer to adopt the same telemetry stack.

Structured lifecycle events

Reports describe the completed execution. Some systems also need to observe the operation while it is running.

RelPrim can emit structured lifecycle events through an opt-in event emitter.

from relprim import EventEmitter, InMemoryEventSink


event_sink = InMemoryEventSink()
event_emitter = EventEmitter(sinks=(event_sink,))

result = await (
    async_operation("generate_response", provider.generate)
    .with_events(event_emitter)
    .with_retry(RetryPolicy(max_attempts=3))
    .run(prompt)
)

Events are disabled by default.

This is deliberate. A reusable library should not silently produce logs, emit telemetry, or introduce operational cost without the application choosing that behavior.

The event model is independent of one logging or tracing product. A sink translates events into whatever the application already uses. RelPrim ships three:

an in-memory sink for tests and diagnostics;
an OpenTelemetry sink that adds events to the active span, installed as an optional extra;
a SQLite event store built on the standard library, for durable local history with basic queries and retention.

The event types, their payload keys, and the report metadata keys are a documented contract. They are part of what 1.0 keeps stable: minor releases may add to them but do not rename or remove them, so dashboards and alerts built on them keep working. The structured events page lists them.

Retry policies

RelPrim supports explicit retry policies, including exponential backoff with jitter.

The important design choice is not the presence of retrying itself. It is that retry behavior can be defined independently from the operation and kept visible at the boundary.

For simple adoption, the decorator accepts a retry count:

@resilient(retries=3, timeout=10)
async def call_provider(prompt: str) -> str:
    return await provider.generate(prompt)

retries=3 means three retries after the first attempt, four attempts in total. The builder states the same thing as RetryPolicy(max_attempts=4). The distinction sounds pedantic until it is wrong in a dashboard, so the documentation states it explicitly.

For more precise behavior, the caller can classify which exceptions are safe to retry.

class TemporaryProviderError(Exception):
    pass


@resilient(
    retries=3,
    retry_on=(TemporaryProviderError,),
    timeout=10,
)
async def call_provider(prompt: str) -> str:
    return await provider.generate(prompt)

Production integrations should prefer narrow retry classification.

Catching every exception may look resilient while repeating failures that are permanent, caused by invalid input, or unsafe to execute again.

Cancellation is never treated as a failure to recover from. If the caller cancels the operation, RelPrim does not retry it, does not fall back, and does not count it against a circuit breaker. Cancellation belongs to the caller.

RelPrim provides the retry mechanism. The application decides which failures have the semantics required to use it. See retry for the complete policy.

Rate-limit recovery

A rate limit is a special kind of temporary failure: the provider often says when to try again.

RelPrim can use that hint for the next retry, fall back to normal backoff when there is none, and refuse to wait longer than the operation can afford.

@resilient(
    retries=3,
    timeout=10,
    rate_limit_on=(ProviderRateLimitError,),
    retry_after=provider_retry_after,
    max_rate_limit_wait=30,
    fallback=call_backup_provider,
)
async def call_provider(prompt: str) -> str:
    return await provider.generate(prompt)

A short delay leads to another primary attempt. A delay beyond the maximum wait sends the execution straight to the fallback instead of blocking the caller.

This is recovery, not throttling. RelPrim reacts after a provider has rejected a call. Proactive quotas, token buckets, and distributed rate limiting remain the job of the application, gateway, or provider SDK. The rate limits page describes how the delay is selected.

Timeout enforcement

Timeouts bound how long an asynchronous attempt is allowed to run.

They prevent a slow or stalled dependency from holding application resources indefinitely.

The simple API accepts a timeout directly. The advanced API uses an explicit TimeoutPolicy.

result = await (
    async_operation("generate_response", provider.generate)
    .with_timeout(TimeoutPolicy(seconds=10))
    .run(prompt)
)

Each attempt has its own deadline. Only that deadline produces OperationTimeoutError. A TimeoutError raised by the operation itself, such as a client socket timeout, is reported as the ordinary failure it is. Two different timeouts should not be reported as one.

A timeout still cannot prove that a remote side effect did not occur.

That distinction belongs to the operation’s domain semantics. The library can stop waiting and report what happened locally. It cannot infer the final state of a remote system it does not control.

This is one example of a boundary RelPrim is designed not to hide.

Fallback chains

A fallback is not merely another exception handler.

It is an explicit decision that another operation can produce an acceptable result when the primary path fails.

RelPrim supports named fallback candidates:

from relprim import fallback_chain


result = await (
    async_operation("generate_response", call_primary_provider)
    .with_retry(RetryPolicy(max_attempts=2))
    .with_fallbacks(
        fallback_chain(
            ("backup_provider", call_backup_provider),
        )
    )
    .run(prompt)
)

The candidate name appears in reports and structured events, and the fallback attempt is marked as such. Retry counts describe only the primary path.

That makes degraded execution observable. The application can distinguish a normal primary success from a result produced by a backup path.

Each candidate’s result goes through the same validation as the primary one. An unusable backup response is a failed candidate, not a successful fallback.

RelPrim does not decide whether using a fallback is correct.

Switching between two read providers may be safe. Switching providers after an ambiguous side-effecting operation may duplicate work. A fallback policy can restrict which failures fall back at all, so the policy reflects the meaning of the operation rather than treating every provider as interchangeable.

Circuit breakers

RelPrim includes asynchronous circuit breakers for protecting the application and an unhealthy downstream dependency.

from relprim import CircuitBreaker


circuit_breaker = CircuitBreaker(
    name="primary_provider",
    failure_threshold=3,
    recovery_timeout_seconds=30,
)

result = await (
    async_operation("generate_response", call_primary_provider)
    .with_circuit_breaker(circuit_breaker)
    .with_retry(RetryPolicy(max_attempts=3))
    .run(prompt)
)

The breaker has an explicit name because its state represents an operational boundary that should be identifiable. It is shared deliberately between the operations that call the same dependency.

It does not repair failed work. It decides whether new attempts should continue reaching a dependency after repeated failure. After the recovery timeout, one probe is let through; a probe whose caller is cancelled releases its slot rather than leaving the breaker stuck.

The surrounding application still needs to decide what happens when the circuit is open. It may use a fallback, defer durable work, return a degraded response, or fail immediately.

Result validation

Transport success is not always application success.

A provider may return an empty response, malformed content, an incomplete payload, or a value that violates the caller’s expectations while still using a successful protocol status.

RelPrim supports validation policies that run before a result is accepted as successful.

from relprim import validation_policy, validator


response_validation = validation_policy(
    validator(
        "non_empty_response",
        lambda value: bool(value.strip()),
        message="Response must not be empty.",
    )
)

result = await (
    async_operation("generate_response", provider.generate)
    .with_validation(response_validation)
    .run(prompt)
)

A failed validation is an ordinary attempt failure. Retries, fallbacks, and the circuit breaker treat it like any other failure the application has classified, and the reason is captured in the execution report.

When responses already have a JSON Schema or a Pydantic model, optional JSON Schema and Pydantic adapters plug into the same policy. Validation checks values; it never transforms them. The caller receives exactly what the operation returned.

Reliability is not only the ability to obtain a response. It is the ability to obtain a result that satisfies the operation’s contract.

Idempotency

Callers retry requests, queues redeliver messages, and concurrent workers pick up the same job. Without coordination, each copy repeats the work.

RelPrim gives one logical operation a key. While an execution for that key is running, duplicate calls join it. After it succeeds, later calls replay its result instead of running the operation again.

def payment_key(request_id: str, amount_cents: int) -> str:
    return f"create-payment:{request_id}"


@resilient(retries=2, timeout=10, idempotency_key=payment_key, idempotency_ttl=3600)
async def create_payment(request_id: str, amount_cents: int) -> str:
    return await payment_gateway.create(request_id=request_id, amount_cents=amount_cents)

Every call reports whether it executed, joined, or replayed, so duplicate traffic is visible rather than silently absorbed.

This is the primitive where an honest boundary matters most. RelPrim idempotency is not an exactly-once guarantee for external side effects. If a process stops after a payment provider accepted a charge but before the result was stored, the next caller cannot know that. Payments, order creation, and other external mutations still need the provider’s own idempotency mechanism. RelPrim coordinates callers; the application and the provider own the invariant. The idempotency page describes exactly what is and is not guaranteed.

Composition without hidden policy

RelPrim brings the selected mechanisms into one operation definition.

result = await (
    async_operation("generate_response", call_primary_provider)
    .with_retry(RetryPolicy(max_attempts=3))
    .with_timeout(TimeoutPolicy(seconds=10))
    .with_validation(response_validation)
    .with_circuit_breaker(circuit_breaker)
    .with_fallbacks(
        fallback_chain(
            ("backup_provider", call_backup_provider),
        )
    )
    .with_events(event_emitter)
    .run(prompt)
)

The value of this API is not that every external call should enable every primitive.

Most should not.

Its value is that the operation’s reliability behavior becomes visible in one reviewable place. A reviewer can see which mechanisms participate and ask whether they match the semantics of the operation. The order in which the builder methods are called does not matter; the execution order is always the documented one.

The library remains intentionally small enough that this definition does not become a separate workflow language.

A stable contract

The pre-stable releases added primitives one at a time: idempotency, rate-limit recovery, durable and OpenTelemetry events, schema validation. Each release was used to learn where the semantics were unclear.

Before 1.0, the work shifted from adding features to removing ambiguity:

An explicit public API. Every public name is listed in the API reference. Anything not listed there is internal and may change.
Types that survive strict checking. The package ships inline type information (PEP 561), and @resilient preserves the decorated function’s parameters and return type, including the union with a fallback’s type.
Semantics that match their names. Cancellation is never retried; a timeout reports only its own deadline; fallback attempts are not retries; a cancelled circuit-breaker probe cannot block the breaker.
Observable behavior as contract. Event types, payload keys, and report metadata keys are never renamed or removed outside a major release.
Verified distributions. CI tests every supported Python version, from 3.11 to 3.14, and checks the built wheel and source distribution in clean environments.

The stability policy defines what 1.x keeps compatible, and the migration guide lists every change from 0.11. Stability here means a compatibility promise for the documented API. It is not a claim about where or how widely the library is used.

Important engineering decisions

Wrap integrations instead of replacing them

RelPrim accepts existing callables.

It does not require applications to abandon provider-native SDKs, introduce a new HTTP client, or move domain behavior into the library.

This keeps the integration boundary narrow and makes incremental adoption possible.

Keep the business value and execution history together

OperationResult[T] prevents a usable return value from erasing the path that produced it.

This is particularly important for retries, fallbacks, validation, and operational diagnosis.

Provide simple and explicit APIs

The decorator covers common cases without requiring builder configuration.

The builder exposes composition when the operation needs more control.

Both use the same primitives, which reduces the risk that convenience and advanced usage develop different semantics.

Fix the execution order

Primitives always run in the same order, regardless of how they were configured. Composition is predictable, and interactions such as “does a timeout count as a breaker failure?” have one documented answer.

Make observability part of execution

Reports and lifecycle events are generated from the operation model.

Observability is not reconstructed later from a collection of unrelated log messages.

Keep events opt-in

Libraries should not create hidden telemetry behavior.

Applications choose whether events are emitted and where they are sent.

Keep optional integrations optional

The core package has no runtime dependencies. OpenTelemetry, JSON Schema, and Pydantic support are installed as extras and never imported unless used.

Prefer named operations and candidates

Stable names make reports and events useful across executions.

They also force the application to describe what capability is being protected.

Do not claim domain safety

A timeout, retry, circuit breaker, fallback, or idempotency key can be executed correctly while the overall business behavior remains wrong.

RelPrim exposes mechanics. The application owns the invariant.

What RelPrim deliberately does not replace

RelPrim is not:

an HTTP client;
an AI provider SDK;
a task queue;
a workflow engine;
an observability backend;
a distributed rate limiter;
a replacement for provider-native clients;
a replacement for orchestration systems.

It is a reliability layer used inside an application, service, worker, or orchestration workflow.

Keeping this boundary narrow is an architectural decision.

A library that attempts to own transport, orchestration, persistence, observability, and business recovery eventually becomes another platform that the application must work around.

RelPrim focuses on the execution boundary around a callable.

Scope at 1.0

The stable API covers:

the @resilient decorator and the async_operation(...) builder;
retry policies with exponential backoff and jitter;
per-attempt timeouts;
named fallback chains;
asynchronous circuit breakers;
validation policies, with optional JSON Schema and Pydantic adapters;
idempotency with joined and replayed calls;
provider-aware rate-limit recovery with bounded waiting;
typed operation results, execution reports, and execution errors;
structured events, with in-memory, OpenTelemetry, and SQLite sinks.

What remains the application’s responsibility is just as deliberate: the idempotency key and its durability beyond the process, whether waiting, failing, or falling back is correct for a given operation, and the reconciliation of outcomes that RelPrim cannot observe.

New capabilities can still arrive in minor releases, but they extend the 1.0 model rather than replace it.

Challenges and trade-offs

A small API versus complete flexibility

Reliability libraries can accumulate configuration quickly.

Every provider exposes different behavior. Every application has different failure semantics. Supporting every variation directly would make RelPrim as difficult to understand as the ad hoc code it is intended to replace.

The current direction favors a focused set of primitives with clear responsibilities.

Unusual domain behavior can remain in application code rather than becoming another generic option.

Convenience versus visible semantics

A decorator makes adoption easy, but excessive convenience can hide important policy decisions.

The builder exists for operations where the mechanisms need to be more visible. The simple API should remain a concise entry point rather than grow into a long list of opaque keyword arguments.

Useful reports versus coupling to observability vendors

Execution reports and events need enough structure to support diagnostics without making the core library depend on one telemetry product.

The transport-agnostic model adds some integration work for applications, but it keeps RelPrim usable across different operational environments.

Stability versus continued change

Freezing an API makes some early decisions permanent for the 1.x line. The hardening work before 1.0 therefore fixed inconsistencies rather than adding features, and several of those fixes were deliberately breaking. Breaking them once, with a migration guide, was preferable to carrying ambiguous semantics into a stable release.

Reusable mechanics versus domain correctness

The library can classify exceptions, execute retries, enforce timeouts, and record outcomes.

It cannot determine whether repeating an external side effect is safe.

Refusing to blur that boundary makes the library less magical, but more honest.

Lessons

Building RelPrim reinforced that reliability is not a collection of defensive wrappers.

The difficult part is preserving the meaning of an operation while its execution passes through retries, timeouts, validation, fallback behavior, and dependency health decisions.

The second lesson is that observability should begin inside the execution model.

Once a function returns a raw value or raises a generic exception, much of the useful context may already be gone. Keeping the value, report, operation name, candidate identity, and lifecycle events connected makes later diagnosis more reliable.

The third lesson came from preparing 1.0: most reliability bugs live in the interactions between mechanisms, not in the mechanisms themselves. What happens when a fallback’s result is invalid, when a probe is cancelled, or when a client’s own timeout looks like the policy’s timeout? Those questions only have good answers once the behavior is written down as a contract.

The last lesson is that reusable infrastructure should make decisions visible without pretending to make every decision itself.

RelPrim provides a vocabulary and execution layer for operations at unreliable boundaries. The application still decides what is safe, what is acceptable, and what recovery means.

That separation is not a limitation to work around.

It is the central design constraint that keeps the library useful without creating false guarantees.

RelPrim

Case study