Reliable External Operations Are More Than Retries

The smallest distributed system

Calling an external system is one of the smallest distributed systems we build.

The code may look local:

response = client.create_resource(payload)

The operation is not local.

Between the call and its outcome sit a network, a remote load balancer, an application you do not deploy, storage you cannot inspect, queues you cannot observe, and failure handling you did not design. The remote system may accept the request before your client times out. It may reject the request temporarily, process it twice, respond late, or return an answer that is technically valid but unusable for your product.

A retry addresses one narrow part of that problem. It can repeat an attempt after a failure that is both temporary and safe to repeat.

It does not tell you how long the operation may take, whether the first attempt already changed remote state, how much additional load you are allowed to create, what should happen during a prolonged outage, or how an operator can recover when the final outcome is unknown.

Reliable external operations need more than retries. They need an explicit execution policy.

That policy defines how the system behaves before the first request is sent, while attempts are in progress, and after the available evidence is no longer sufficient to produce a trustworthy answer.

Why retries are not enough

Retries are attractive because they often produce an immediate improvement.

A dependency occasionally returns 503. The client tries again. The second attempt succeeds. The incident disappears from the user’s perspective.

That success can hide the assumptions underneath it.

The first assumption is that the failure is temporary. Retrying a validation error, an invalid credential, or an unsupported operation will not produce a different result.

The second assumption is that another attempt is safe. Repeating a read is usually harmless. Repeating a command that creates a shipment, charges a card, sends a message, or provisions infrastructure may duplicate a business effect.

The third assumption is that the dependency has enough capacity to recover while clients continue adding work. During a widespread outage, retries can multiply traffic precisely when the remote system is least capable of handling it.

The fourth assumption is that failure is known. A timeout tells the caller that it stopped waiting. It does not prove that the remote system stopped working.

Consider a request that reaches a provider and commits successfully. The response is lost on the way back. From the caller’s perspective, the attempt timed out. A blind retry can now repeat an operation that already happened.

The problem is no longer transient failure. It is uncertainty.

Retries are useful only after the system has answered three questions:

1. What kind of failure occurred?
2. Is another attempt likely to help?
3. Is repeating the operation safe?

Without those answers, retrying is not a reliability strategy. It is repetition under incomplete information.

The failure modes at an external boundary

External operations fail in different ways, and those differences matter.

A useful policy starts by separating transport symptoms from business outcomes.

This distinction is important because application code often collapses all unsuccessful calls into one exception path.

try:
    return provider.call(payload)
except Exception:
    return retry()

That code has no model of what happened. It knows only that the expected return value was not produced.

A production system needs a richer vocabulary.

An operation may have:

• succeeded;
• failed definitively;
• failed in a way that is safe to retry;
• completed with a degraded but valid result;
• ended with an unknown outcome.

The unknown outcome deserves special attention. It is not a temporary error and it is not a definitive failure. It means the caller lacks enough evidence to choose either one.

A reliable system preserves that state instead of converting it into whichever boolean is easier to return.

Reliability as one execution policy

Timeouts, retries, idempotency, circuit breakers, rate limits, fallbacks, and observability are often introduced independently.

One team adds a retry decorator. Another sets an HTTP timeout. A third adds a circuit breaker around the client. Individual call sites begin to implement their own fallbacks. Metrics count requests and failures, but not the decisions made between them.

Each mechanism may be reasonable on its own. The result can still be incoherent.

The mechanisms interact.

A timeout may apply to each attempt or to the entire operation. A circuit breaker may observe individual attempts or only final operation failures. A rate limiter may count the original call or every retry. A fallback may execute after a definitive failure, after exhausted retries, or after an ambiguous timeout. Idempotency may protect the inbound API while leaving the real side effect unprotected.

The correct behavior cannot be inferred from the presence of the mechanisms. It comes from how they are composed.

An execution policy should answer questions such as:

• What is the identity of the logical operation?
• How long may the complete operation continue?
• How long may one attempt run?
• Which outcomes are safe to retry?
• How many attempts may be made?
• How much concurrency may reach the dependency?
• What should happen when the dependency is already known to be unhealthy?
• Which business effect must be idempotent?
• When is a fallback semantically valid?
• What evidence must be recorded for recovery?
• Who owns an operation whose outcome remains unknown?

These are architectural decisions. Hiding them in a chain of decorators does not make them less important.

A useful distinction is the difference between an operation and an attempt.

The operation represents the business intention:

Create one shipment for order 1842.

An attempt represents one interaction with the remote provider:

Send the third HTTP request for that shipment operation.

Retries create new attempts. They must not create new business intentions.

That distinction should be visible in identity, telemetry, persistence, and recovery tooling.

The mechanisms inside the policy

Each reliability mechanism solves a different problem.

The value comes from giving it a precise responsibility and defining how it interacts with the others.

Timeouts

A timeout bounds waiting.

It does not cancel time itself, and it does not prove that remote work has stopped.

There are usually at least two useful limits:

• a per-attempt timeout;
• an end-to-end deadline for the complete logical operation.

Suppose one attempt may take two seconds and the policy allows three attempts. Without an end-to-end deadline, backoff, scheduling delay, connection setup, and queueing can make the total operation last much longer than six seconds.

The operation deadline is the real contract with the caller.

operation deadline
    |
    +-- attempt 1 timeout
    |
    +-- backoff
    |
    +-- attempt 2 timeout
    |
    +-- backoff
    |
    +-- attempt 3 timeout

Every stage should consume from the same remaining budget.

When only 400 milliseconds remain, starting another two-second attempt is not meaningful. The policy should stop before the deadline is exceeded, not discover afterwards that the final attempt could never have completed in time.

Timeouts also need semantic classification.

A timeout before a connection is established carries different information from a timeout after the request body was transmitted. The first often suggests that the provider never received the operation. The second may leave the outcome unknown.

The client library may expose both as timeout exceptions. The reliability policy should not assume they mean the same thing.

Retries

A retry is a new attempt to complete the same operation.

It is appropriate when the failure is temporary, another attempt has a realistic chance of succeeding, and repeating the action does not violate a business invariant.

The retry classifier should be based on semantics, not only exception types.

A 429 response may be retryable after the provider’s Retry-After period. A 503 may be retryable within the remaining deadline. A malformed request is not retryable. A timeout may require reconciliation instead of another attempt.

Backoff reduces immediate pressure. Jitter prevents many clients from retrying in lockstep.

Neither one solves unlimited amplification.

If one user request triggers three service calls and every layer retries three times, a single initial operation can produce dozens of attempts. A retry policy therefore needs a budget, not only a maximum configured on each client.

That budget may be expressed as:

• maximum attempts per operation;
• maximum additional traffic as a percentage of normal traffic;
• maximum retry time inside the operation deadline;
• a shared retry allowance across service layers.

The exact model depends on the system. The underlying principle does not.

Retries spend capacity. They should be treated as an investment made when the probability of recovery justifies the cost.

Idempotency

Idempotency protects the business effect from repetition.

It is not the same as ignoring duplicate HTTP requests.

A request may be deduplicated at the edge and still produce duplicate work downstream. A worker may crash after applying the effect but before acknowledging the message. A queue may redeliver. An operator may replay an event. A retry may reach another service instance.

The idempotency boundary must surround the effect that cannot safely happen twice.

A robust idempotency record often needs more than a key:

operation key
request fingerprint
current state
result or external reference
creation time
completion time
failure classification
recovery status

The request fingerprint protects against accidental reuse of the same key for a different operation.

The current state distinguishes an operation that is running from one that completed.

The stored result lets duplicate callers receive the same outcome instead of executing the effect again.

An idempotency record may move through states such as:

started
in_progress
completed
failed_definitively
outcome_unknown

The last state matters. Removing the record after a timeout would allow a later request to recreate the operation even though the provider may already have completed it.

Idempotency cannot manufacture certainty, but it can stop uncertainty from turning into duplication.

When the external provider supports its own idempotency key, the same logical operation identity should usually cross that boundary. Local idempotency and provider-side idempotency then protect different failure windows.

Local storage prevents duplicate internal execution. The provider key prevents repeated attempts from creating repeated remote effects.

Circuit breakers

A circuit breaker is a decision to stop.

It protects the caller from continuing to spend resources on a dependency that is unlikely to respond successfully. It may also protect the dependency from additional load while it is recovering.

The breaker needs a meaningful failure signal.

Validation errors should not open a circuit. Neither should failures caused by the caller’s own malformed requests. Provider timeouts and temporary server failures may count. Rate-limit responses may need separate treatment because they indicate capacity policy rather than general health.

The breaker’s scope also matters.

One global breaker for an entire provider may stop unrelated operations because one endpoint is failing. A breaker scoped too narrowly may fail to notice that the provider is broadly unavailable.

Useful dimensions can include:

• provider;
• endpoint or operation class;
• region;
• credential or tenant;
• dependency capability.

Half-open probes should be limited. Allowing normal traffic through as soon as the open period ends can create another burst against a dependency that is only beginning to recover.

A breaker does not complete failed work. It only decides whether new attempts should be admitted.

Recovery still requires a queue, a retry schedule, a degraded response, reconciliation, or a clear failure returned to the caller.

Rate limits and concurrency limits

Rate and concurrency solve related but different problems.

A rate limit controls how many operations may begin during a period.

A concurrency limit controls how many operations may be in flight at the same time.

A provider may allow 100 requests per second while becoming unstable when 100 slow requests remain open concurrently. Respecting the rate limit alone will not protect either side.

The policy should also distinguish original operations from retry attempts.

If retries bypass the limiter, an outage can create traffic beyond the level the client was designed to send. If every retry waits behind the same limiter, the operation deadline must account for that waiting time.

Backpressure is the behavior that follows when capacity has been consumed.

The system can:

• wait within a bounded deadline;
• queue durable work;
• reject new work;
• reduce lower-priority traffic;
• return a degraded response;
• shed load before the dependency becomes the bottleneck.

The choice belongs to the product and the operation.

A background synchronization job can often wait. An interactive request may need to fail quickly. A safety-critical action may need reserved capacity.

The limiter should express those priorities instead of treating all work as equal.

Fallbacks

A fallback provides a different result when the preferred path cannot produce one.

The fallback must still satisfy a real product contract.

Returning stale profile data may be acceptable when a personalization service is unavailable. Returning stale authorization data may be dangerous. Using a second provider for a read operation may be straightforward. Using a second provider after an ambiguous command could duplicate the effect.

Fallbacks can include:

• cached data;
• stale but bounded data;
• reduced functionality;
• a secondary provider;
• a locally computed approximation;
• a clear unavailable state.

The fallback should reveal degradation when that information matters.

A result produced from yesterday’s cache is not equivalent to a fresh provider response. A heuristic score is not equivalent to a model-backed decision. Hiding that distinction creates a reliable transport path that delivers an unreliable product meaning.

Fail-open and fail-closed decisions are business decisions.

When a fraud provider is unavailable, accepting every transaction may protect conversion while increasing financial risk. Rejecting every transaction may protect risk while blocking legitimate customers.

Infrastructure cannot decide which outcome is correct without domain context.

A good policy makes that decision explicit and gives the caller enough information to communicate it honestly.

Observability and recovery

Most external-client telemetry is request-centric.

It records an HTTP span, status code, duration, and perhaps an exception.

The logical operation may contain several attempts, waiting periods, breaker decisions, idempotency checks, and a final reconciliation step. Looking at one request does not explain the operation.

Observability should preserve both levels.

An operation report may include:

operation_id
operation_type
dependency
started_at
completed_at
final_outcome
attempt_count
attempt_outcomes
time_waiting_for_capacity
time_spent_in_backoff
deadline_remaining
breaker_state
fallback_used
idempotency_state
external_reference
recovery_required

This data is useful for more than dashboards.

It lets an operator answer practical questions:

• Did the provider receive the request?
• How many attempts were made?
• Why was another attempt not allowed?
• Did the circuit breaker reject execution?
• Was the result returned from a fallback?
• Is the operation safe to replay?
• Does it require reconciliation?
• Which external reference can be used to check provider state?

The final outcome should be visible as a first-class signal.

Counting timeout exceptions is useful. Counting operations that ended in an unknown state is more valuable.

The latter tells the team that work exists which cannot be resolved automatically from the evidence currently available.

Why composition order changes behavior

There is no universally correct wrapper order for every external operation.

There is, however, a wrong assumption that the order does not matter.

Consider two simplified compositions:

with_timeout(
    with_retries(call_provider, attempts=3),
    seconds=2,
)

with_retries(
    with_timeout(call_provider, seconds=2),
    attempts=3,
)

In the first version, two seconds may bound the complete retry loop.

In the second, each attempt may consume two seconds before backoff and scheduling delay are considered.

Both compositions contain a timeout and retries. They implement different contracts.

The same issue appears elsewhere.

When a circuit breaker wraps the full retry loop, it may observe one final operation failure. When it wraps each attempt, three failed attempts may contribute three observations.

When rate limiting happens outside the retry loop, one admitted operation may create several outbound requests. When it happens inside, each attempt consumes provider capacity but may also wait long enough to exhaust the deadline.

When a fallback wraps an idempotent side effect, it may run after an outcome that is still unknown. That can create a second effect rather than a degraded response.

A practical execution order for a side-effecting external operation may look like this:

1. Identify the logical operation
2. Load or create its idempotency record
3. Establish the end-to-end deadline
4. Check whether new work may be admitted
5. Enter the attempt loop
6. Wait for rate and concurrency capacity
7. Check the circuit breaker
8. Execute one attempt within its timeout
9. Classify the result
10. Record attempt telemetry and breaker outcome
11. Retry only when safe and within budget
12. Persist the final operation state
13. Apply a fallback only when its semantics are valid
14. Route unknown outcomes to reconciliation

This is not a universal template. Read-only operations can often use a simpler policy. A command without provider-side idempotency may require more conservative handling. A durable background job may wait where an interactive API request would return immediately.

The important part is that the order is chosen and reviewable.

A named policy is easier to reason about than a stack of defensive mechanisms assembled independently at every call site.

A practical external operation

Consider a service that creates shipments through a carrier API.

The product requirement is simple:

One customer order should produce one shipment.

The carrier API may be slow, rate limited, temporarily unavailable, or successful without returning a response to the client. Creating the shipment twice can produce duplicate labels, duplicate collection requests, or conflicting tracking state.

The logical operation receives a stable identity:

create-shipment:order-1842

The application stores an idempotency record before contacting the carrier.

operation_id: create-shipment:order-1842
state: in_progress
request_fingerprint: 8f2c...
external_reference: null
result: null

The policy establishes a total deadline of five seconds. Individual attempts may use shorter timeouts so that a retry remains possible within that budget.

Before an attempt begins, the operation waits for both rate and concurrency capacity. Waiting consumes the same deadline.

The circuit breaker then decides whether the carrier is currently eligible to receive traffic. If the breaker is open, the operation does not send a request merely to rediscover a known outage.

The request carries the same logical idempotency key to the carrier when the provider supports it.

Several outcomes are possible.

The carrier returns a temporary error

The policy classifies the response as retryable.

It records the attempt, applies bounded backoff with jitter, checks the remaining deadline, and sends another attempt only when enough time remains.

The carrier rejects the address

This is a definitive business failure.

Another identical attempt will not help. The policy stores the rejection and completes the operation without retrying.

The request times out before a connection is established

The carrier probably did not receive it.

A retry may be safe, subject to the operation budget and provider policy.

The request times out after the body was sent

The carrier may have created the shipment.

The operation enters an unknown state.

operation_id: create-shipment:order-1842
state: outcome_unknown
external_reference: null
last_attempt: timed_out_after_send
recovery_required: true

The system does not immediately send the same shipment to another carrier. It does not delete the idempotency record. It does not tell the caller that the shipment definitively failed.

Instead, a reconciliation process checks the carrier using the idempotency key, order reference, or another supported lookup mechanism.

If the shipment exists, the operation records the carrier reference and completes successfully.

If the carrier confirms that no shipment was created, the operation may safely resume according to policy.

If the provider cannot answer, the uncertainty remains visible for operational review.

The same client request arrives again

The application finds the existing idempotency record.

If the operation completed, it returns the stored result.

If it is still running, it returns an in-progress status or waits according to the API contract.

If the outcome is unknown, it returns that state rather than starting another independent shipment creation.

The interesting property is not that every failure disappears.

The property is that each failure leads to a defined state with a known next action.

A result needs more than success or failure

A traditional client often returns a value or raises an exception.

That interface is convenient, but it can discard information the application needs.

A richer operation result can preserve the final semantics:

class OperationResult:
    outcome: Literal[
        "succeeded",
        "failed",
        "degraded",
        "unknown",
    ]
    value: object | None
    error: Exception | None
    external_reference: str | None
    attempts: list[AttemptReport]
    recovery_required: bool

The application may still expose a simpler interface to its caller.

The execution layer should not lose the distinction internally.

A definitive failure can be shown to the user and corrected.

A degraded result can be used with an appropriate product signal.

An unknown outcome may require a pending state, reconciliation, or human intervention.

Converting all three into the same exception makes recovery harder because the most important evidence has already been erased.

Policy design belongs close to the domain boundary

Reliability mechanisms are often placed in shared infrastructure because they are technically reusable.

The policy itself should still be defined close to the domain boundary.

A generic HTTP client can implement timeout enforcement, backoff, rate limiting, or circuit-breaker mechanics. It cannot know whether retrying POST /shipments is safe. It cannot decide whether stale pricing data is acceptable. It cannot determine whether a fallback should fail open.

Those choices depend on the operation.

The reusable layer should provide mechanisms and require the caller to provide semantics:

policy = ExternalOperationPolicy(
    operation_name="create_shipment",
    total_deadline=5.0,
    attempt_timeout=1.5,
    retry_classifier=classify_carrier_failure,
    idempotency=shipment_idempotency,
    fallback=None,
    on_unknown=reconcile_shipment,
)

The syntax is illustrative. The shape is what matters.

A policy should be named after the operation or capability it protects. It should not be an anonymous stack of wrappers called default_retry_policy.

Names create review pressure.

create_shipment_policy invites questions about shipment semantics.

standard_resilience encourages reuse without understanding.

That leaves a recurring engineering gap. The domain must define what is safe, but every external integration still needs many of the same execution mechanics: retry classification, time bounds, fallback behavior, result validation, circuit state, and a record of what actually happened.

Rebuilding those mechanics at every boundary creates drift. Moving them into a generic client can create the opposite problem by hiding decisions that should remain visible.

That tension is why I built RelPrim.

Why I built RelPrim

I created RelPrim after seeing the same reliability concerns reappear around external integrations.

A provider call usually begins as a small piece of application code. Production requirements then accumulate around it. The call needs a timeout. Some failures should be retried. A fallback may be useful. The response may need validation. Repeated failures should affect whether more work is admitted. When the operation finishes, the team needs enough evidence to understand what happened.

These concerns are often implemented in one of two ways.

The first is repetition. Each integration builds its own combination of helper functions, decorators, exception handlers, and logging. Similar operations slowly develop different behavior.

The second is concealment. Reliability is moved into a generic wrapper that makes a call appear safe without showing what that safety means. The code says that an operation is resilient, but not which failures are retried, how long the operation may continue, what happens after exhaustion, or what evidence remains for recovery.

RelPrim is my attempt to make the mechanics reusable without making the policy invisible.

It provides two levels of adoption. The @resilient decorator covers straightforward integrations where a small, explicit policy is enough. The lower-level async_operation(...) builder supports cases where the operation needs more deliberate composition.

from relprim import RetryPolicy, TimeoutPolicy, async_operation


result = await (
    async_operation("generate_response", call_provider)
    .with_retry(RetryPolicy(max_attempts=3))
    .with_timeout(TimeoutPolicy(seconds=10))
    .run(prompt)
)

The operation name is explicit because it becomes part of the execution report and any structured lifecycle events emitted by the policy. A stable name such as generate_response says more operationally than a generic name such as call or invoke.

The result also contains more than the business value:

value = result.value
report = result.report.to_dict()

RelPrim returns an OperationResult[T] so that execution metadata is preserved instead of being discarded once a value is produced.

That distinction matters. Two operations may return the same business value after taking very different paths. One may succeed immediately. Another may time out, retry twice, reject an invalid response, and finally use a fallback. From the caller’s product perspective, both may be usable. From an operational perspective, they are not equivalent.

The execution report makes that history available to application code, diagnostics, tests, and observability systems.

For more advanced operations, the same builder can compose mechanisms such as:

• retry policies;
• async timeout enforcement;
• fallback chains;
• circuit breakers;
• result validation;
• structured execution events;
• idempotency policies;
• provider-aware rate-limit recovery.

Idempotency support can prevent duplicate execution for repeated calls, allow concurrent callers to join the same in-progress operation, and replay a previously completed result. The default in-memory store is intentionally limited to a single process, while the storage interface allows applications to provide persistence appropriate to their deployment model.

Rate-limit recovery can use provider-supplied retry delays, including values exposed through Retry-After or provider-specific exceptions. Waiting remains bounded by the operation policy: when the requested delay exceeds the configured maximum, RelPrim continues to a fallback or final failure instead of waiting indefinitely.

These capabilities still do not make an operation automatically safe. The application must define a stable idempotency key, decide which result may be replayed, translate provider-specific rate-limit information, and determine whether waiting, failing, or falling back remains correct for the business operation.

The list is intentionally narrower than the complete reliability model described in this series. RelPrim provides reusable execution mechanics, but it does not replace durable workflow state, distributed coordination, reconciliation logic, or domain-specific recovery.

That boundary is worth stating plainly. A reliability library should not create confidence by claiming guarantees it does not yet provide.

RelPrim is also deliberately not an HTTP client, provider SDK, task queue, workflow engine, or observability backend. It wraps an operation that already exists and gives its reliability behavior a visible structure.

Most importantly, it does not decide domain semantics.

It can enforce an attempt timeout. It cannot know whether a timed-out payment, shipment, or infrastructure command is safe to repeat.

It can execute a fallback. It cannot decide whether the degraded result remains honest for the product.

It can record how an operation was executed. It cannot determine what reconciliation means for the business.

Those decisions remain in the application, where they can be reviewed alongside the invariant they protect.

That division of responsibility is the point. RelPrim provides explicit, composable execution mechanics and a structured account of what happened. The application remains responsible for correctness.

The remaining articles in this series examine the individual mechanisms in more depth, then return to the harder question: how to compose them without hiding the real behavior of the system.

The complete series

This article is the entry point to Reliability at System Boundaries, a practical series about external operations that must remain understandable and recoverable under failure.

The following parts explore timeouts, retries, idempotency, ambiguous outcomes, circuit breakers, backpressure, fallbacks, observability, policy composition, testing, and the path from recurring patterns to reusable reliability primitives.

The complete, current series index is rendered below. Published parts are available as links, while upcoming entries remain visible as the editorial roadmap.

Takeaway

An external operation is not reliable because it retries.

It is reliable when the system knows how long it may wait, which failures may be repeated, how duplicate effects are prevented, when pressure must be reduced, what degradation is acceptable, and how uncertain outcomes will be recovered.

Those decisions should not be scattered across client code and rediscovered during incidents.

They should form one explicit execution policy with a clear operation identity, a deliberate composition order, and enough evidence to explain the final outcome.

The network will always leave gaps in what the caller can know.

Good reliability engineering does not deny those gaps. It gives them names, states, limits, and recovery paths.