Retrying Without Making Things Worse

What a retry can and cannot fix

A retry gives an operation another chance. It also gives the dependency another piece of work.

For an isolated failure, that trade-off is usually easy to justify. A connection drops while a service is fetching a shipment’s tracking status. The next attempt reaches the carrier and returns a useful answer. The user never needs to know that the first attempt failed.

Now suppose the carrier is overloaded. Responses slow down, callers reach their timeouts, and every caller makes two more attempts. The carrier receives more work while it is already struggling to finish the work it has. Some original requests may still be running when their replacements arrive.

The retry code can behave exactly as configured while making the incident worse.

That is the central problem with retries in distributed systems: a decision that helps one caller can hurt the system when every caller makes it at once.

The first article in this series separated a logical operation from its attempts. A retry creates another attempt to complete the same intention. It should not create a new payment, shipment, or generation job merely because the response to the previous attempt was inconvenient.

The previous article added the time boundary. A retry must fit inside the useful lifetime of the operation, including waiting and cleanup. Reaching a timeout does not give the caller a fresh deadline.

There is another constraint: something must be capable of changing between attempts.

A new connection may avoid a broken socket. Another backend instance may have capacity. A temporary lock may be released. A provider may recover during the delay. Those are plausible reasons to try again.

Sending the same invalid address to the same validator has no such reason. Neither does repeating a request with an expired credential. Waiting longer does not repair the input.

Overload needs more care. Retrying against a healthy instance can help when only part of a fleet is affected. Repeating work against a saturated dependency can prevent recovery. The relevant question is whether the next attempt has a credible path to a different outcome, at a cost the system can afford.

Retryable and terminal failures

An exception type is useful evidence, but it is rarely the whole retry policy.

Consider an HTTP 503. A carrier might use it to reject a request before processing begins. A gateway might return it after losing contact with a backend that already accepted the command. The status looks the same to the client. The safety of repeating a tracking lookup and repeating shipment creation is different.

A useful classification combines the observed failure with the meaning of the operation:

The third row is easy to mishandle. A temporary communication failure can leave a lasting business effect. Classifying the network problem as transient does not establish that another attempt is safe.

HTTP method semantics help, but the endpoint must uphold them. RFC 9110 describes why idempotent requests can be repeated after communication failures and why automatic retries of non-idempotent requests need stronger evidence. Adding an Idempotency-Key header to a request does not create that evidence unless the receiving system implements the contract.

Response validation needs the same discipline. A provider returning malformed data may be experiencing a temporary problem. It may also be returning a new schema that the integration no longer understands. Retrying the same deterministic mismatch several times adds latency before producing the same failure.

The integration should translate provider-specific evidence into a small set of meaningful outcomes. A retry layer can then act on those outcomes without trying to infer business meaning from every SDK exception.

I prefer a narrow retry classifier for that reason. A new programming error should escape visibly, rather than quietly becoming three executions of the same bug. Broader policies need a deliberate justification and a way to distinguish repeated transient failures from repeated mistakes.

Cancellation belongs outside that classifier. A cancelled request should not return as another provider attempt because a wrapper caught too much.

Backoff and jitter

Once another attempt is justified, the next question is when to send it.

Immediate retries can work when failure is isolated and recovery does not require waiting. They are a poor default for a population of clients observing the same outage. Each client sees a local failure and reacts at roughly the same time.

A fixed delay moves that reaction into the future. It can leave the clients just as synchronized.

Exponential backoff increases the waiting interval after successive failures. A cap prevents the delay from growing without bound. Full jitter chooses a random delay within that capped interval:

ceiling = min(max_delay, base_delay * multiplier ** (retry_number - 1))
delay = random.uniform(0, ceiling)

Here, retry_number=1 means the first retry after the initial attempt. With a base of 250 milliseconds and a multiplier of two, the first three ceilings are 250 milliseconds, 500 milliseconds, and one second. The actual waits vary within those ranges.

Plain exponential backoff still groups clients around the same retry times. Jitter spreads them out. AWS’s comparison of exponential backoff and jitter illustrates this behavior with a contention simulation. Its results describe that workload, not a universal performance improvement for every dependency.

Jitter is not a minimum recovery period. Full jitter can select a delay close to zero. If a provider requires a minimum wait, that requirement takes precedence over sampling from a range that includes earlier times.

The HTTP Retry-After field can express either a delay in seconds or an HTTP date. When the policy honors that guidance, it should not shorten the wait to squeeze another attempt into the caller’s deadline. If the provider asks for ten seconds and only two seconds remain, the inline retry no longer fits. Adding randomized delay after the provider’s requested wait may be appropriate; randomizing below it is not.

A maximum backoff and a maximum acceptable provider wait therefore serve different purposes. The first caps a locally calculated delay. The second can make the operation stop or defer when the provider’s requested delay is too long.

There is no need to sleep after the last allowed attempt. There is also no value in sleeping through the remaining deadline before admitting that another useful attempt cannot fit.

For the implementation details and alternative jitter strategies, see the Python exponential backoff guide. The architectural constraint is broader: waiting must reduce pressure without silently extending the operation’s contract.

Retry budgets

max_attempts=3 bounds one execution path. It says little about the volume of retries a busy service can produce.

If 1,000 operations all use their full allowance, each making one initial attempt and two retries, the dependency receives 3,000 attempts. Each operation respected its limit. Retry traffic still doubled the initial load.

A useful policy needs several budgets with different scopes:

A shared allowance might permit at most 100 retry attempts for 1,000 initial attempts admitted during a defined accounting window. That is a retry-to-initial ratio of 10 percent, or about 9.1 percent of the resulting 1,100 total attempts. The denominator matters when comparing policies and dashboards.

This is an example budget, not a recommended universal percentage. A cheap metadata lookup and a costly inference request should not automatically receive the same allowance.

One possible implementation replenishes retry tokens from initial traffic and spends one token before each additional attempt. The bucket needs a cap so a long healthy period cannot accumulate an enormous outage allowance. Its accounting must not let retries replenish the capacity they consume.

Low-volume operations may need a small minimum allowance to recover from an isolated failure. That allowance needs a defined scope as well. A fixed allowance in every process grows as the application scales. A process-local budget can be useful without being a fleet-wide guarantee.

Google’s Handling Overload describes combining per-request limits with a per-client retry budget. The useful distinction is between limiting one caller’s persistence and limiting the additional work generated by a population of callers.

The scope should follow the constrained dependency. An allowance shared by unrelated providers can let one outage consume another integration’s recovery capacity. Per-operation limits may also be needed so one failing request cannot monopolize the shared allowance.

An empty retry budget should produce a decision immediately: fail, return an acceptable degraded result, or durably defer the work. Starting a new queue wait without a limit only moves the unbounded behavior elsewhere. A deferred attempt must still be counted when it is eventually dispatched.

Counts are not always enough. Retrying a large report, an image generation, or a transaction with several downstream calls can cost much more than retrying a small read. Where that difference matters, account for the constrained resource as well as the number of requests.

The budget also needs feedback. Record additional attempts and how many logical operations they actually recover. A rising attempt count with little recovered work is a reason to reconsider the policy, even if the configured cap has not been reached. A successful retry should not erase the failed attempt from the operational picture.

None of these limits makes initial traffic safe. A dependency can still be overloaded before the first retry arrives. Admission control and backpressure address that broader problem; a retry budget limits the extra pressure caused by recovery attempts.

Retry amplification

Retry ownership becomes harder to see when several layers make the decision independently.

A user request reaches an application. The application calls an integration service, which uses a provider SDK. If every layer allows three total attempts, persistent failure at the provider can produce up to 27 provider attempts for one user operation.

The counts multiply. They do not add up to nine.

The following small experiment isolates that multiplication. Each layer repeats the callable below it, and the final dependency always fails. It uses only the Python standard library.

from functools import partial


class TemporaryError(Exception):
    pass


def retry(call, max_attempts):
    for attempt in range(1, max_attempts + 1):
        try:
            return call()
        except TemporaryError:
            if attempt == max_attempts:
                raise


def count_provider_calls(attempt_limits):
    calls = 0

    def provider():
        nonlocal calls
        calls += 1
        raise TemporaryError("Provider unavailable")

    call = provider
    for limit in attempt_limits:
        call = partial(retry, call, max_attempts=limit)

    try:
        call()
    except TemporaryError:
        return calls


for limits in ((1, 1, 1), (3, 1, 1), (3, 3, 3)):
    print(limits, count_provider_calls(limits))

The output is:

(1, 1, 1) 1
(3, 1, 1) 3
(3, 3, 3) 27

This is an attempt-count experiment, not a load test. There is no concurrency, backoff, queueing, deadline, or capacity model. In this persistent-failure scenario, adding jitter alone would change the timing without changing the maximum count. A deadline or shared allowance could stop execution earlier.

The experiment also assumes that every failed inner call remains retryable at the next layer. That assumption is exactly what an integration needs to examine. If an inner layer exhausts its recovery policy and the outer layer treats the resulting error as permission to start over, the outer layer has silently renewed the allowance.

Choose the retry owner deliberately. The layer immediately above a failing dependency can often retry without repeating successful upstream work. A higher workflow layer may have better knowledge of the business operation. Either choice has consequences, especially when one operation fans out into several calls.

Other layers must respect that decision. A flag saying “SDK retries disabled” is insufficient if an HTTP transport or service mesh still repeats requests underneath it. Inventory the actual execution path, including worker redelivery and operator replay.

Giving each hop a percentage budget helps, but does not automatically create one end-to-end budget. Each hop can still amplify the work it receives. Uber’s discussion of retry storms and error ownership examines this distinction in deep service call chains. The general lesson is that a propagated failure needs context: it may describe recovery that has already been attempted elsewhere.

For smaller systems, that context may be an explicit integration error that is excluded from upstream retry rules. Larger systems may propagate retry state through shared infrastructure. The important property is that passing an error upward does not erase the decisions already made below.

Retrying requests versus retrying operations

An operation often contains several steps, and only one of them may need another attempt.

Consider a generated shipping label. The application requests generation, receives a provider job ID, waits for completion, downloads the file, and stores it in its own object storage. If the download fails, repeating the complete function may request another label instead of retrieving the one that already exists.

The useful retry boundary is the failed step, with the completed steps preserved.

Each row still needs its own safety contract. An upload retry, for example, must not accidentally create another product asset because the code generated a new identity on every execution.

The same reasoning applies to AI products. A model provider may have completed billable work even if saving the result fails. Retrying persistence is different from paying for another generation. A retry decorator around the entire workflow cannot discover that distinction on its own.

A queue changes the timing, but not the problem. Returning a failed job to the queue may rerun every step unless the worker can load durable progress. A new worker does not make the old provider job disappear, and a new delivery should not automatically receive a fresh operation budget.

Long-running work therefore needs a distinction between the deadline for an interactive response and the recovery window for the durable operation. Returning “pending” can end the user’s wait while recovery continues under a separate, explicit policy. It should not reset the operation identity or authorize indefinite retries.

These are reasons to choose narrow execution boundaries before adding retry machinery. The later parts of this series will examine durable state and recovery in more detail. For now, the practical test is straightforward: identify which effects have already happened before deciding how much code to execute again.

The relationship with idempotency

Safe repetition and useful repetition are separate properties.

An idempotent operation can still overload a provider. Looking up the same tracking status 27 times may leave business state unchanged while consuming connections, CPU time, and quota. Conversely, a carefully delayed retry can still create a duplicate shipment if the endpoint has no replay contract.

The policies need to answer both questions: what prevents another business effect, and what limits the work spent trying to obtain the result?

For a command, one logical identity should survive retries, worker handoffs, and recovery. If the carrier supports idempotency, the integration must follow that provider’s rules for key scope, payload matching, and retention. A locally stable identifier does not prove that the remote system uses it for deduplication.

The request also has to be physically replayable. A consumed upload stream cannot necessarily be sent again. Credentials, signatures, and temporary URLs may need refreshing according to their protocols while the business intention remains the same. Reusing an operation identity does not mean replaying every byte unchanged.

There is also a point where local evidence is insufficient. After an ambiguous command timeout, another attempt may be safe under a documented provider replay contract. Without that protection, the next action may need to be a status lookup or reconciliation rather than another write.

The next article, Idempotency Is a Business Invariant, will examine the ownership, storage, and concurrency rules behind that guarantee. The REST API idempotency guide covers a concrete request lifecycle. Here, idempotency is a prerequisite for some retry decisions, not a reason to keep retrying without limits.

How this maps to RelPrim

RelPrim gives the attempt policy an explicit place around an existing callable. The integration still decides which failures are temporary and which operations can be repeated.

The following example wraps a tracking-status lookup. fetch_once is an existing asynchronous integration function that performs one provider attempt and translates selected temporary failures into TemporaryTrackingError. Retries in its underlying client are disabled.

import asyncio
from collections.abc import Awaitable, Callable

from relprim import (
    ExponentialBackoff,
    OperationResult,
    OperationTimeoutError,
    RetryPolicy,
    TimeoutPolicy,
    async_operation,
)


class TemporaryTrackingError(Exception):
    pass


async def load_tracking_status(
    fetch_once: Callable[[str], Awaitable[str]],
    tracking_id: str,
) -> OperationResult[str]:
    async with asyncio.timeout(2.0):
        return await (
            async_operation("load_tracking_status", fetch_once)
            .with_retry(
                RetryPolicy(
                    max_attempts=3,
                    retry_on=(TemporaryTrackingError, OperationTimeoutError),
                    backoff=ExponentialBackoff(
                        base_delay_seconds=0.1,
                        max_delay_seconds=0.5,
                        jitter=True,
                    ),
                )
            )
            .with_timeout(TimeoutPolicy(seconds=0.5))
            .run(tracking_id)
        )

There are at most three primary attempts, including the first. Each has a 500-millisecond timeout. The outer asyncio.timeout() bounds local waiting for the complete execution, including backoff, subject to cooperative cancellation. The tracking lookup is assumed to be safe to repeat; a shipment-creation command would need its own replay contract.

OperationTimeoutError is explicitly retryable here because the per-attempt timeout can expire on a repeatable read. The outer deadline ends the operation rather than beginning another retry cycle. Authentication failures and other exceptions outside the classifier do not enter this retry loop.

The returned OperationResult retains the value and execution report. It can be consumed without discarding the path that produced the value:

status = result.value
execution = result.report.to_dict()

The example was executed against RelPrim 0.11.0 with Python 3.12.14 on October 24, 2026. RetryPolicy.max_attempts counts all attempts. The shorthand @resilient(retries=3) instead allows three retries after the first attempt. That difference matters when calculating downstream traffic.

This configuration provides an attempt limit and an outer deadline. It does not implement a shared retry allowance across callers or coordinate budgets across service layers. Those decisions remain in the surrounding application or infrastructure.

RelPrim also supports provider-aware rate-limit recovery through a separate RateLimitPolicy. That policy can select a provider-supplied delay and reject a wait above a configured maximum. It is not a fleet-wide quota manager. The rate-limit handling documentation describes that boundary.

The general retry policy should remain understandable without the library. RelPrim supplies execution mechanics; the operation definition supplies their meaning.

Takeaway

A retry policy is incomplete if the only question it answers is how many times to call a function.

Before the next attempt, the system needs a reason to expect a different result, a contract that makes repetition safe, and enough remaining time and capacity to justify the work. Those conditions have to survive the layers between the user and the failing dependency.

Backoff and jitter shape when attempts arrive. Attempt limits and shared budgets constrain how much recovery work is admitted. Choosing the right execution boundary prevents a failed step from repeating effects that already completed.

Sometimes the useful outcome is another successful attempt. Sometimes it is an explicit decision to stop, preserve what is known, and recover through a different path. A system needs both.