Why provider hints matter

A normal retry policy knows how long the application would like to wait. A rate-limited provider may know when the next request is actually allowed.

RelPrim 0.9.0 adds provider-aware rate-limit recovery for external operations.

Extracting the provider delay

Different SDKs expose rate-limit information in different ways. One may provide a retry_after attribute, another may expose a response header, while another may not return a delay at all.

RelPrim uses a small extractor function to translate that provider-specific exception into a delay expressed in seconds.

def provider_retry_after(
    exception: Exception,
) -> float | None:
    if isinstance(exception, ProviderRateLimitError):
        return exception.retry_after_seconds

    return None

The extractor receives the original provider exception and returns:

  • a number when the provider supplied a retry delay;
  • 0 when an immediate retry is allowed;
  • None when no provider hint is available.

Returning None does not disable retries. RelPrim falls back to the delay calculated by the configured retry policy.

Using retry-after delays

The extractor is passed to the @resilient(...) decorator together with the exception types that represent provider rate limits.

from relprim import resilient


@resilient(
    retries=3,
    timeout=10,
    rate_limit_on=(ProviderRateLimitError,),
    retry_after=provider_retry_after,
    max_rate_limit_wait=30,
)
async def call_provider(prompt: str) -> str:
    return await provider.generate(prompt)

When the provider supplies a retry-after value, RelPrim uses it instead of the normal exponential backoff.

When no retry hint is available, the existing retry backoff is used.

Avoiding excessive waits

Interactive operations should not necessarily wait minutes for a provider quota to recover.

max_rate_limit_wait defines the longest acceptable delay for the current operation.

When the selected delay exceeds that limit, RelPrim skips the retry and enters the normal fallback or failure path.

@resilient(
    retries=3,
    rate_limit_on=(ProviderRateLimitError,),
    retry_after=provider_retry_after,
    max_rate_limit_wait=5,
    fallback=call_backup_provider,
)
async def call_primary_provider(prompt: str) -> str:
    return await primary_provider.generate(prompt)

This makes it possible to wait briefly when recovery is likely, while switching providers when the suggested delay would make the operation unusable.

Reports and events

Rate-limited attempts include the selected delay, its source and whether the maximum acceptable wait was exceeded.

Structured operations may emit:

  • rate_limit.detected
  • retry.scheduled
  • rate_limit.wait_exceeded

Scope

This release handles recovery after a provider rejects an operation because of a rate limit.

It does not add local throttling, token-bucket enforcement or distributed quota coordination.