Example: provider failover

The most common reliability problem: a provider you depend on is briefly unavailable, or stops answering. This example combines a retry, a timeout and a fallback so the caller still gets an answer, and the execution report shows how it was produced.

The problem#

A support tool summarizes tickets with a primary provider. Sometimes the provider answers 503; sometimes it accepts the connection and never responds. A backup provider gives an acceptable answer. Without explicit policy, the caller either fails on the first error or waits indefinitely, and nobody can tell afterwards which provider produced the summary.

Implementation#

The demo providers are local classes. The primary fails on its first call and hangs on its second; the backup always answers.

import asyncio

from relprim import (
    OperationTimeoutError,
    RetryPolicy,
    TimeoutPolicy,
    async_operation,
    fallback_chain,
)


class TemporaryProviderError(Exception):
    """The provider is briefly unavailable; trying again may help."""


class PrimaryProvider:
    def __init__(self) -> None:
        self.calls = 0

    async def summarize(self, ticket: str) -> str:
        self.calls += 1
        if self.calls == 1:
            raise TemporaryProviderError("503 Service Unavailable")
        await asyncio.sleep(10)  # the second call hangs
        return f"Primary summary: {ticket}"


class BackupProvider:
    async def summarize(self, ticket: str) -> str:
        return f"Backup summary: {ticket}"


async def no_sleep(delay: float) -> None:
    """Skip the backoff delay so the example runs instantly."""


async def main() -> None:
    primary = PrimaryProvider()
    backup = BackupProvider()

    result = await (
        async_operation("summarize_ticket", primary.summarize)
        .with_retry(
            RetryPolicy(
                max_attempts=2,
                retry_on=(TemporaryProviderError, OperationTimeoutError),
                async_sleeper=no_sleep,
            )
        )
        .with_timeout(TimeoutPolicy(seconds=0.2))
        .with_fallbacks(fallback_chain(("backup_provider", backup.summarize)))
        .run("Printer on floor 3 is offline")
    )

    report = result.report
    print(result.value)
    print(f"attempt_count={report.attempt_count} retry_count={report.retry_count}")
    for attempt in report.attempts:
        error = f" ({attempt.error.type})" if attempt.error else ""
        role = " [fallback]" if attempt.fallback else ""
        print(f"attempt {attempt.attempt_number}: {attempt.status}{error}{role}")
    print(f"served by: {report.last_attempt.metadata['fallback_candidate_name']}")


asyncio.run(main())

It prints:

Backup summary: Printer on floor 3 is offline
attempt_count=3 retry_count=1
attempt 1: failed (TemporaryProviderError)
attempt 2: timed_out (OperationTimeoutError)
attempt 3: succeeded [fallback]
served by: backup_provider

What happens#

1. Attempt 1 calls the primary, which raises TemporaryProviderError. It is listed in retry_on, so a retry is scheduled.
2. Attempt 2 calls the primary again. It hangs, and after 0.2 seconds the timeout policy cancels it and records OperationTimeoutError. Only the policy’s own deadline produces that error.
3. max_attempts=2 is the total number of primary attempts, so the primary is not called again. The failure matches the fallback policy (by default every Exception), so the chain runs.
4. Attempt 3 is the fallback candidate backup_provider, which succeeds. Fallback candidates run without the timeout policy.

In the report, attempt_count is 3, because the fallback attempt is a recorded attempt. retry_count is 1: it counts primary attempts after the first, and fallback attempts are never retries. The fallback attempt has fallback=True, and its metadata names the candidate that produced the value. report.metadata["fallback_used"] is True.

Why this matters#

The caller writes one call and gets a value. The decisions behind it (which failures are retried, how long an attempt may take, what the alternative is) are visible in one place in the code, and the report records which path was taken. A summary served by the backup can be logged, counted or labelled instead of looking identical to a primary answer.

Production note#

Retry and fallback are only safe when running the operation again, or running an alternative, is acceptable. Summarizing a ticket is read-only. For an operation with external side effects, such as charging a card, a retry or a fallback can repeat the effect; see the idempotent payment example. Use retry_on and fallback_on to list the failures you consider transient, rather than retrying everything.

The default backoff waits between attempts; this example replaces the sleep with no_sleep only so it runs instantly.

Related documentation#

• Retry: max_attempts, retries and backoff.
• Timeouts: what is timed and which error is raised.
• Fallbacks: chains, fallback_on and attempt counting.
• Runnable repository examples: basic_resilience.py and fallback_chain.py.