Building Idempotency in REST APIs with Keys, Concurrency and Safe Retries

The guarantee you actually needx

Idempotency in a REST API means that repeating the same logical operation does not produce the business effect twice.

For a write such as POST /payments, the client creates one idempotency key before the first attempt and reuses it for every retry. The server associates that key with the operation, verifies that later requests carry the same payload, and replays the original result once the operation completes.

The UUID is not the guarantee. The guarantee comes from the protocol around it: atomic ownership, payload matching, durable state, response replay, and a recovery path when the process fails after the side effect has started.

That last case is the one simple examples usually miss. A timeout tells the client that it stopped waiting. It does not prove that the payment, order, or external command failed.

For the broader distinction between an operation and an attempt, see Reliable External Operations Are More Than Retries.

What the server needs to remember

An idempotency record should answer three questions:

1. Does this key already belong to an operation?
2. Is the repeated request the same operation?
3. What should the server return now?

A compact data model is enough:

A practical identity may be scoped as:

tenant-17 + create-payment + checkout-8d32a6

The fingerprint should be calculated from canonical, validated business input. Hashing raw JSON is fragile because whitespace and key order can change without changing the request meaning.

The repository API matters more than the storage engine. The application should work with an atomic claim operation rather than a sequence of SELECT, check, and INSERT calls.

from dataclasses import dataclass
from enum import StrEnum
from typing import Protocol


class ClaimStatus(StrEnum):
    ACQUIRED = "acquired"
    IN_PROGRESS = "in_progress"
    COMPLETED = "completed"
    PAYLOAD_MISMATCH = "payload_mismatch"


@dataclass(frozen=True)
class ClaimResult:
    status: ClaimStatus
    response_code: int | None = None
    response_body: dict[str, object] | None = None


class IdempotencyRepository(Protocol):
    async def claim(
        self,
        *,
        scope: str,
        key: str,
        request_fingerprint: str,
    ) -> ClaimResult: ...

    async def complete(
        self,
        *,
        scope: str,
        key: str,
        response_code: int,
        response_body: dict[str, object],
    ) -> None: ...

The database implementation can use a unique constraint, an atomic insert, or a transactional lock. The service layer should not depend on raw SQL details to understand the operation.

The request lifecycle

The first request claims the key and becomes the owner of the operation.

new key
   |
   v
claim atomically
   |
   +--> different payload: reject
   |
   +--> already completed: replay
   |
   +--> already running: return in-progress contract
   |
   v
execute effect
   |
   v
store response

A small service function can make that contract visible:

async def create_payment(
    *,
    key: str,
    command: CreatePayment,
    idempotency: IdempotencyRepository,
    payments: PaymentService,
) -> ApiResponse:
    fingerprint = command.fingerprint()

    claim = await idempotency.claim(
        scope=f"customer:{command.customer_id}:create-payment",
        key=key,
        request_fingerprint=fingerprint,
    )

    if claim.status is ClaimStatus.PAYLOAD_MISMATCH:
        return ApiResponse.conflict("Key already used with another payload")

    if claim.status is ClaimStatus.COMPLETED:
        return ApiResponse(claim.response_code, claim.response_body)

    if claim.status is ClaimStatus.IN_PROGRESS:
        return ApiResponse.conflict(
            "Operation is still in progress",
            headers={"Retry-After": "1"},
        )

    result = await payments.create(command)

    response = ApiResponse.created(result.to_dict())
    await idempotency.complete(
        scope=f"customer:{command.customer_id}:create-payment",
        key=key,
        response_code=response.status_code,
        response_body=response.body,
    )
    return response

The important part is the state machine. The same key and payload either owns, joins, or replays one logical operation. The same key with another payload is a conflict.

The race most implementations miss

This is not safe:

if not await repository.exists(key):
    await repository.create(key)
    await perform_effect()

Two requests can both pass the check before either creates the record.

The key must be claimed atomically. A unique constraint is a useful enforcement mechanism, but it is only the beginning. The losing request still needs a defined response.

Two common contracts are reasonable:

• return 409 Conflict or 425 Too Early with Retry-After;
• wait for the owner within a bounded deadline, then replay its result.

Waiting can improve the client experience, but it consumes connections and adds coordination. Returning immediately is simpler and works well when the client already has a retry policy.

The crash window

Local effects can often be committed in the same database transaction as the completed idempotency record.

External effects cannot.

Consider:

1. The API claims the key.
2. It sends a request to a payment provider.
3. The provider creates the payment.
4. The API crashes before storing the result.

The local record remains in_progress, but the remote effect may already exist.

Deleting the record and trying again can create a duplicate. Marking it as failed can also be wrong. This is an ambiguous outcome.

Recovery needs evidence from the remote system. Useful mechanisms include:

• passing the same idempotency key to the provider;
• querying by a stable order or request reference;
• consuming a provider webhook;
• moving stale operations into a reconciliation workflow.

The idempotency record may need a state such as recovery_required rather than only failed.

This is why idempotency does not create exactly-once execution across a distributed system. It gives the operation one identity and allows every boundary to protect the effect it owns.

TTL and observability

Keep completed records for at least as long as clients can realistically retry or redeliver the request.

A short TTL reopens the duplicate window. A long TTL increases storage and may retain response data longer than necessary. The correct period follows the business operation, not a universal default.

Stale in_progress records need monitoring. They should not silently expire if the effect may already have happened.

Useful signals include:

• operations executed for the first time;
• completed results replayed;
• payload conflicts;
• concurrent requests joining or receiving in_progress;
• age of the oldest unfinished operation;
• records moved to reconciliation.

Include the scoped key, or a safe hash of it, in logs and traces so the API request, domain operation, queue message, and provider call can be correlated.

Coordinating a client operation with RelPrim

Server-side idempotency protects the API that owns the business effect. Client-side coordination protects repeated execution inside the calling application.

RelPrim can give one logical operation a stable key, coordinate concurrent callers, and replay a successful result.

The following example was verified against the RelPrim 0.8.0 API docs on July 28, 2026.

import asyncio

from relprim import resilient


def payment_key(request_id: str, amount_cents: int) -> str:
    return f"create-payment:{request_id}"


@resilient(
    retries=2,
    timeout=10,
    idempotency_key=payment_key,
    idempotency_ttl=3600,
)
async def create_payment(
    request_id: str,
    amount_cents: int,
) -> str:
    return await payment_gateway.create(
        request_id=request_id,
        amount_cents=amount_cents,
    )


first, second = await asyncio.gather(
    create_payment("request-123", 2499),
    create_payment("request-123", 2499),
)

replayed = await create_payment("request-123", 2499)

print(first.report.metadata["idempotency_status"])
print(second.report.metadata["idempotency_status"])
print(replayed.report.metadata["idempotency_status"])

One concurrent caller receives executed. The other receives joined. A later call receives replayed.

Idempotency wraps the complete RelPrim execution lifecycle, including retries, timeouts, validation, circuit-breaker checks, fallbacks, reports, and structured events. A replay does not start a second retry or fallback flow.

RelPrim also currently assumes that callers do not reuse one key for another payload. An API that owns the effect should still enforce its own request fingerprint, durable state, concurrency rules, and recovery process.

See the RelPrim idempotency guide for execution statuses, policy configuration, concurrency behavior, and store limitations.

When an idempotency key adds no value

Do not add a key to every endpoint by default.

Read-only operations normally do not need one. A correctly designed PUT that replaces a known resource and a DELETE that leaves it absent may already be naturally idempotent.

Keys are most useful when a write creates a new effect and the caller may repeat the request without knowing the first outcome.

They are not a substitute for:

• domain uniqueness constraints;
• transactional state changes;
• consumer deduplication;
• provider-side protection;
• reconciliation after uncertain outcomes.

Before you ship

Before releasing an idempotent endpoint, walk through the complete operation lifecycle rather than checking only that an idempotency key exists.

One key should represent one business intention. Its scope should separate tenants and operation types, and reusing it with another payload should produce a conflict. Ownership must be acquired atomically, concurrent callers need a documented response, and a completed operation should replay the original result instead of executing again.

Where the business effect and the idempotency record share a database, commit them together whenever possible. Where the effect crosses a system boundary, assume that a timeout or crash can leave the outcome unknown. Those operations need a reconciliation path based on a stable external reference, provider-side idempotency, status lookup, or an asynchronous confirmation.

Retention should cover the real retry and redelivery window. Records that remain in_progress beyond the expected operation time should be visible, investigated, and moved through an explicit recovery policy rather than silently expired.

Idempotency protects business effects, not HTTP requests.

Read Reliable External Operations Are More Than Retries for the wider policy around retries, timeouts, fallbacks, and uncertain outcomes.

How long do you retain idempotency records, and what does your system do when an operation remains in_progress beyond its expected completion window?