Skip to content
Bart Rozycki
RelPrim docsRate-limit recovery

Examples

Example: Retry-After recovery

A complete, runnable example of provider-aware rate-limit recovery. A 429 response with Retry-After is retried after the provider's delay, and a delay beyond max_rate_limit_wait is rejected.

Many providers reject excess calls with 429 Too Many Requests and say when to try again. This example uses rate-limit recovery to wait for the delay the provider asked for, within a budget you choose.

The problem

A support assistant generates replies with a hosted model. Under load the provider rejects a call with a Retry-After value. A fixed backoff either retries too early and is rejected again, or waits longer than necessary. And some requests should not wait at all if the provider asks for minutes.

Implementation

The demo provider rejects the first call of each request. record_wait replaces the sleep so the example runs instantly and shows the delays RelPrim selected.

import asyncio

from relprim import OperationExecutionError, RetryPolicy, resilient


class ProviderRateLimitError(Exception):
    """HTTP 429 from the provider, with its Retry-After value in seconds."""

    def __init__(self, retry_after_seconds: float | None) -> None:
        super().__init__("429 Too Many Requests")
        self.retry_after_seconds = retry_after_seconds


class DemoProvider:
    """Rejects the first call of each request with a Retry-After hint."""

    def __init__(self) -> None:
        self.retry_after_seconds = 2.0
        self.calls = 0

    async def generate(self, prompt: str) -> str:
        self.calls += 1
        if self.calls == 1:
            raise ProviderRateLimitError(self.retry_after_seconds)
        return f"Reply to: {prompt}"


def provider_retry_after(exception: Exception) -> float | None:
    if isinstance(exception, ProviderRateLimitError):
        return exception.retry_after_seconds
    return None


waits: list[float] = []


async def record_wait(delay: float) -> None:
    """Record the delay instead of sleeping, so the example runs instantly."""
    waits.append(delay)


provider = DemoProvider()


@resilient(
    name="generate_reply",
    retry=RetryPolicy(max_attempts=3, async_sleeper=record_wait),
    rate_limit_on=(ProviderRateLimitError,),
    retry_after=provider_retry_after,
    max_rate_limit_wait=30,
)
async def generate_reply(prompt: str) -> str:
    return await provider.generate(prompt)


async def main() -> None:
    result = await generate_reply("Where is my order?")
    metadata = result.report.metadata
    print(result.value)
    print(f"waited: {waits}  provider calls: {provider.calls}")
    print(f"rate_limit_wait_seconds={metadata['rate_limit_wait_seconds']}")

    # The provider now asks for two minutes, beyond the 30 second budget.
    waits.clear()
    provider.calls = 0
    provider.retry_after_seconds = 120.0
    try:
        await generate_reply("Where is my order?")
    except OperationExecutionError as error:
        metadata = error.report.metadata
        print(f"failed with {type(error.cause).__name__}")
        print(f"waited: {waits}  provider calls: {provider.calls}")
        print(f"rate_limit_wait_exceeded={metadata['rate_limit_wait_exceeded']}")


asyncio.run(main())

It prints:

Reply to: Where is my order?
waited: [2.0]  provider calls: 2
rate_limit_wait_seconds=2.0
failed with ProviderRateLimitError
waited: []  provider calls: 1
rate_limit_wait_exceeded=True

What happens

First request. The provider raises ProviderRateLimitError with retry_after_seconds=2.0. It matches rate_limit_on, so provider_retry_after extracts the hint. Two seconds is within max_rate_limit_wait=30, so RelPrim waits exactly that long (the provider’s delay replaces the backoff) and retries. The second call succeeds. rate_limit_wait_seconds in the report metadata records the time spent waiting.

Second request. The provider asks for 120 seconds. That exceeds the 30-second budget, so RelPrim neither sleeps nor retries the primary operation. With no fallback configured, the operation fails with OperationExecutionError, whose cause is the provider’s exception and whose report has rate_limit_wait_exceeded=True. With a fallback, the execution would continue into it instead.

max_rate_limit_wait applies to whichever delay was selected: the provider’s hint, or the retry policy’s backoff when the provider gives none. Rate-limit exceptions do not need to be listed in retry_on.

Why this matters

The provider’s own signal decides the delay, the caller’s latency budget decides whether waiting is acceptable, and both are written next to the operation. A long rate-limit delay becomes a fast failure (or a fallback) instead of a request that silently hangs for minutes.

Production note

Pick max_rate_limit_wait from the caller’s point of view: an interactive request may accept a few seconds, a background job much more. Each rate-limited attempt still uses one of the retry policy’s max_attempts.