Rate limits

When a provider rejects a call because of a rate limit, it often says when to try again. RelPrim can use that hint for the next retry, fall back to the normal backoff when there is none, and refuse to wait longer than the operation can afford.

Identify rate limits and their delay#

Provider SDKs expose rate-limit information differently, so you supply two things: the exception types that mean “rate limited”, and an optional extractor that reads the suggested delay in seconds from such an exception.

class ProviderRateLimitError(Exception):
    def __init__(self, message: str, *, retry_after_seconds: float | None = None) -> None:
        super().__init__(message)
        self.retry_after_seconds = retry_after_seconds


def provider_retry_after(exception: Exception) -> float | None:
    if isinstance(exception, ProviderRateLimitError):
        return exception.retry_after_seconds
    return None

The extractor returns a number of seconds, 0 to allow an immediate retry, or None when the provider gave no usable hint.

Decorator usage#

from relprim import resilient


@resilient(
    retries=3,
    timeout=10,
    rate_limit_on=(ProviderRateLimitError,),
    retry_after=provider_retry_after,
    max_rate_limit_wait=30,
    fallback=call_backup_provider,
)
async def call_provider(prompt: str) -> str:
    return await provider.generate(prompt)

Option	Default	Meaning
rate_limit_on	(required)	Tuple of Exception subclasses that represent a rate limit.
retry_after	None	Extractor returning the provider delay in seconds, or None.
max_rate_limit_wait	60	Longest acceptable delay before the next attempt, in seconds.

retry_after and max_rate_limit_wait require rate_limit_on.

How the delay is selected#

When an attempt fails with a configured rate-limit exception and another attempt is available, RelPrim selects the delay before it:

1. the provider’s delay, when the extractor returns one;
2. otherwise the retry policy’s backoff delay, with its jitter.

Then it applies the maximum wait to the selected delay, whichever source it came from:

• a delay up to and including max_rate_limit_wait is awaited, and the next attempt runs;
• a larger delay is rejected. RelPrim does not sleep and does not retry the primary operation; the execution continues into the fallback or fails.

A provider delay of 0 retries immediately. max_rate_limit_wait=0 accepts only immediate retries.

On the final attempt no delay is selected, because no attempt follows, so the maximum wait is not evaluated either.

Rate-limit exceptions are classified before retry_on, so they do not need to be listed there. A long provider delay can therefore send the call straight to a backup provider, while a short one leads to another primary attempt.

Policy objects#

RateLimitPolicy is the advanced form, for the decorator’s rate_limit= option or the builder’s .with_rate_limit(...):

from relprim import RateLimitPolicy, RetryPolicy, async_operation

rate_limits = RateLimitPolicy(
    rate_limit_on=(ProviderRateLimitError,),
    retry_after=provider_retry_after,
    max_wait_seconds=30,
)

result = await (
    async_operation("call_provider", provider.generate)
    .with_retry(RetryPolicy(max_attempts=4))
    .with_rate_limit(rate_limits)
    .run("Summarize this ticket")
)

max_wait_seconds defaults to 60.0 and must be finite and at least 0. rate_limit_policy(...) is an equivalent factory. Do not combine rate_limit= with the simple rate_limit_on, retry_after or max_rate_limit_wait options.

Extractor failures#

The extractor must return an int, a float or None; numbers must be finite and at least 0. If it raises or returns anything else, RelPrim does not retry with an untrusted delay. The attempt fails with RetryAfterExtractionError, whose rate_limit_error is the provider’s exception and whose cause is the extractor’s own exception, if any. The execution continues into the fallback or fails, with that error as OperationExecutionError.cause.

Reports and events#

Each failed primary attempt records the rate-limit decision in its metadata:

{
    "rate_limited": True,
    "rate_limit_retry_after_seconds": 2.0,   # provider hint, or None
    "rate_limit_delay_seconds": 2.0,         # selected delay, or None
    "rate_limit_delay_source": "provider",   # "provider", "backoff" or None
    "rate_limit_wait_exceeded": False,
}

The delay keys are None when no delay was selected: for failures that were not rate-limited, and on the final attempt. When the selected delay is rejected, they describe the rejected delay and rate_limit_wait_exceeded is True.

The report metadata summarizes the execution with rate_limit_encountered, rate_limit_wait_seconds (total time spent waiting because of rate limits) and rate_limit_wait_exceeded.

With structured events, a rate-limited failure emits rate_limit.detected, followed by retry.scheduled with rate_limited=True and the delay source, or by rate_limit.wait_exceeded with the rejected delay.

Cancellation#

Cancelling the caller during a rate-limit wait raises asyncio.CancelledError immediately. No further attempt or fallback runs.