The problem in one paragraph
HTTP says GET, HEAD, PUT, and DELETE are idempotent: doing them twice has the same effect as doing them once. POST is not. So when a POST request fails — connection reset, gateway timeout, 502 from a load balancer — the client cannot tell whether the server processed it. Retry, and you risk a duplicate side-effect: a double charge, a duplicated order, two messages sent. Don't retry, and you risk losing a request that genuinely failed in flight. Neither is acceptable for any operation that costs money or sends a message.
The contract
The pattern that has emerged in production APIs — and which we recommend for any non-idempotent endpoint — works like this:
- The client generates a unique key for the logical operation it wants to perform. A UUID is fine; the only requirement is uniqueness within a sensible window.
- The client sends the key as the
Idempotency-Keyrequest header. - If the server has not seen the key before, it processes the request normally, records the key together with the response, and returns the response.
- If the server has seen the key before, it returns the original response without re-doing the work.
- If the server has seen the key before but the request body is different, it returns an error (typically 422) so the client knows it has reused a key for a different operation.
The key insight is that the client controls the key, not the server. Only the client knows whether two requests are "the same operation" — the server cannot tell whether two structurally identical requests sent five seconds apart were a retry or a deliberate repeat.
A worked example: charging a card
Consider a payment endpoint:
POST /payments HTTP/1.1
Idempotency-Key: 7c1e4a3a-2b9f-4d2a-9b3a-2c8f1e3d4a5b
Content-Type: application/json
{ "amount": 4500, "currency": "USD", "source": "card_xyz" }First time the server sees this key, it charges the card and returns 201 with a payment record. The key, the request fingerprint (a hash of the body and the relevant headers), and the full response are stored together.
Now suppose the response never reaches the client — TCP reset, client timeout, whatever. The client retries with the same key. The server sees the key, looks up the stored response, and returns it as if the work had just happened. The card is charged once.
Suppose instead the client retries with the same key but a different body — say, accidentally amount: 5400. The server sees the key, sees the stored fingerprint doesn't match the new request, and rejects with 422 and an error explaining the mismatch. The client now knows it has a bug, not a successful retry.
Storing keys: the trade-offs
The store needs to be fast, durable, and shared across every server that can handle the request. The realistic options are:
- Redis or another in-memory store with persistence. Fast lookups, fast writes, easy TTLs. Requires careful capacity planning — every key holds the full response body. For most APIs this is the right answer.
- The same database that stores the actual records. Adds a row per request, but gives you transactional guarantees. Simpler operationally; slower at scale.
- A dedicated key-value store with strong consistency. Worth it once you outgrow Redis but before you want to push idempotency state into your primary database.
Whichever you pick, the lookup has to be transactional with the write. The classic bug is: check whether the key exists, find it doesn't, do the work, then store the key. Two concurrent retries can both pass the check before either writes the key, and both proceed to do the work. Use atomic insert-or-fail semantics (Redis SET key value NX, a database unique constraint) so that only one request can claim the key.
How long to keep keys
There's no single right answer. The window has to cover any retry the client might reasonably make, plus a margin. Common choices:
- 24 hours — the conventional default. Long enough for any client-side retry, including ones that involve a human noticing and pressing the button again. Short enough that storage cost is bounded.
- 7 days — sensible for operations that involve a multi-step external workflow (a payment that takes a day to settle, a webhook that gets retried for a week).
- 30 days or longer — only when the operation has external side effects whose failure window genuinely lasts that long. Comes with real storage cost.
Whichever window you choose, document it. Clients that don't know how long a key is valid will reuse one that has just expired and get a duplicate.
Corner cases that catch implementations out
The in-flight request
What happens when the server receives the same key while it is still processing the original request? Two acceptable answers: queue the second request behind a lock keyed on the idempotency key, or return 409 with a "request in progress" body. Both work. What is not acceptable is to start a second copy of the work — that defeats the entire point.
Partial failure mid-way
If the work fails halfway through, you have a choice. Either store the failure as the canonical response (so retries return the same failure), or don't store anything (so retries try again). The first is correct when the failure is the genuine outcome of the operation — say, "card declined" — because retrying won't change the answer. The second is correct when the failure is transient — say, "downstream service timeout" — because the client should be able to retry and succeed.
The pragmatic rule: store the response if it is a deterministic outcome of the request. Don't store if the failure was internal and might not recur. This means a 4xx response is usually stored; a 5xx response is usually not.
Idempotency on operations that span multiple resources
If the operation creates several records — say, an order with line items — make sure either all of them are created or none of them are, before you record the key as having succeeded. Storing the key under partial success leaves the system in a state where retries return success but the records are incomplete.
Keys that look unique but aren't
Don't accept keys whose uniqueness depends on what the server happens to know. A timestamp alone is not unique across multiple clients. The customer ID is not unique across operations. The order number is not unique if two clients are independently creating orders. UUIDs are unique by construction; require them, or document the format you accept.
Common mistakes
- Treating GET as needing idempotency keys. GET is already idempotent at the protocol level. Asking for keys on GET adds noise without adding safety.
- Validating the key without validating the body. A retried key with a changed body should be rejected, not silently treated as a hit on the original.
- Idempotency on the wrong scope. Per-account, not global. Two different customers must be able to use the same UUID without colliding.
- Forgetting concurrency. Without atomic claim-and-write, two retries that arrive before the first response is stored will both do the work.
- Not telling clients. Idempotency is a contract; if it isn't documented, clients won't use it, and you'll keep getting duplicate-charge bug reports.
Frequently asked questions
What is an idempotency key?
An idempotency key is a client-generated unique value — normally a UUID — sent with a write request so the server can recognise a retry of that exact operation and return the original result instead of doing the work twice. It turns a non-idempotent POST into one that is safe to retry, which is what makes automated retry logic usable on operations that charge money, book inventory, or send messages.
When is an idempotency key needed?
Whenever a request has a side effect that would be harmful if repeated and the client cannot tell whether it succeeded. In practice that means every POST that creates or moves something of value: payments, refunds, orders, bookings, outbound messages, provisioning calls. You do not need one on GET, HEAD, or DELETE, which are already idempotent at the protocol level, and you rarely need one on a PUT that writes a full representation to a known URL.
How do I create idempotency keys for retries?
Generate the key on the client, once, before the first attempt — then reuse the same key for every retry of that logical operation. Generating a fresh key per attempt defeats the whole mechanism. A version 4 UUID is the usual choice; anything with enough entropy to be unique per account works. Do not derive the key from request content alone, or two genuinely distinct requests with identical bodies will collide.
What should the expiration period of an idempotency key be?
Long enough to cover every retry a client will realistically make, short enough that storage stays bounded. Twenty-four hours is the common default and covers connection-level retries, client restarts, and most queued-job backoff schedules. If your clients batch overnight or retry from a durable queue, extend it to seven days. Document the window explicitly: once a key expires, a retry is treated as a brand-new request, and a client that does not know the window cannot reason about that.
What should the API return while a request with the same idempotency key is still in progress?
Return 409 Conflict and say so in the error body — the operation is being processed and the client should retry after a short delay rather than assume failure. This is why the key must be claimed atomically before the work starts: without an atomic claim-and-write, two retries arriving before the first response is stored will both execute. Some APIs return 425 Too Early instead; either is defensible, but pick one and document it.
What should the API return if the same key arrives with a different request body?
Reject it. The key promised "this exact operation", and a different body is a different operation, so returning the stored response would be a lie. 422 Unprocessable Content is the better fit when the request is syntactically valid but conflicts with a stored key; 400 Bad Request is acceptable if you do not distinguish. Store a hash of the request body alongside the key so this check is cheap.
How does this stop double bookings?
It stops the class of double booking caused by retries — the client sends "book seat 14A", the response is lost to a timeout, the client retries, and the second attempt is recognised as the same operation. It does not stop two different users booking the same seat; that needs a uniqueness constraint or a lock on the resource itself. Idempotency keys deduplicate requests, not intentions.
What about a 502 or gateway timeout on the client side?
This is the case the pattern exists for. A 502, 504, or reset connection tells you nothing about whether the server did the work. With an idempotency key the client can retry with confidence and backoff; without one, the only safe options are to give up or to reconcile afterwards. Pair the key with exponential backoff and jitter so a downstream stall does not turn into a retry storm.
How do idempotency keys relate to request IDs and correlation IDs?
They answer different questions and should be separate headers. The idempotency key identifies a logical operation and is deliberately stable across retries. A request ID identifies a single HTTP attempt and must change on every retry, or your logs collapse two attempts into one. A correlation ID identifies a flow across multiple services and is propagated unchanged through the call graph. Reusing one value for all three makes debugging retries considerably harder.
Do idempotency keys apply to retries after 4xx errors?
Mostly they do not help, because a 4xx is usually a deterministic answer: retrying an invalid request produces the same invalid request. Store the response so retries return the same failure quickly rather than re-running validation. The exceptions are 408, 425, and 429, which are transient by definition and should be retried — with the same key.
Where to go next
For how this fits the broader API design picture, see API Design Best Practices. For the rate-limiting algorithms that often run alongside idempotency in retry-heavy clients, see API Rate Limiting Strategies. For the related problem of delivering events reliably to a webhook, see Webhook Design and Delivery. For the status codes and error envelopes that carry these outcomes back to clients, see API Error Handling Conventions, and for keeping the contract stable as it changes, API Versioning and Schema Evolution.