You shipped retry logic on a Friday. By Monday one customer has two identical charges, an angry support ticket with screenshots, and no obvious bug anywhere in the code. The client library did exactly what it was configured to do when the connection dropped after 30 seconds: it tried again.
That gap between a client that retries and a server that can't tell a retry from a new request is what API idempotency exists to close. It's tempting to think of it as a header you add. It isn't. It's a property of your storage layer, and either the storage layer has it or the header is decoration.
What follows is each failure case in the order you'll meet it, from the definition the HTTP specification uses and the ambiguous timeout that starts most incidents. The fixes end up in a database table with a unique constraint, which is less glamorous than the header and considerably more useful.
Table of Contents
- Which HTTP Methods Are Idempotent and Safe
- Why a Timeout Can Charge a Customer Twice
- How Idempotency Keys Make a POST Retryable
- What the Server Must Store and Return
- Pitfalls That Break Idempotency in Production
- How Often to Retry and Which Errors to Retry
- A Working Idempotent POST Example
- Check Your Idempotency Design Before You Ship
Which HTTP Methods Are Idempotent and Safe
Press an elevator call button five times. One elevator still arrives, and the fifth press asked for nothing the first press hadn't already asked for. The button is idempotent.
HTTP behaves the same way. A request method is idempotent when the intended effect on the server of several identical requests equals the effect of one. The HTTP semantics standard RFC 9110 defines it in exactly those terms and lists PUT, DELETE and every safe method as idempotent.
Three questions get tangled together, so keep them apart.
| Method | Safe | Idempotent | Retry without a key |
|---|---|---|---|
| GET | Yes | Yes | Yes |
| HEAD | Yes | Yes | Yes |
| OPTIONS | Yes | Yes | Yes |
| PUT | No | Yes | Yes |
| DELETE | No | Yes | Yes |
| POST | No | No | Only with a key |
| PATCH | No | No | Only with a key |
Safe means the method asks for nothing to change. Idempotent means a repeat lands you in the same place. Retry without a key is the consequence, and it's the column that matters at 2am when a socket closes.
GET, HEAD and OPTIONS change nothing, so they're safe, and safe methods are idempotent by definition. PUT and DELETE do change state. A PUT sets a resource to a known value and a DELETE removes it, so sending either twice leaves the end state you asked for. That's why both tolerate a blind retry.
POST and PATCH are neither. A POST asks the server to create something, and two of them create two things. PATCH isn't defined in RFC 9110 at all; it comes from its own specification, which states plainly that PATCH is neither safe nor idempotent while noting that a PATCH request can be written in a way that happens to be idempotent. The method doesn't decide. Your implementation does.
The trap: idempotent doesn't mean identical responses. A second DELETE often returns 404 where the first returned 200. The state on the server has to match, not the bytes on the wire.
The specification is direct about the consequence. A client shouldn't automatically retry a non-idempotent request unless it has some means to know the semantics are repeatable or that the original request was never applied, and a proxy must not retry a non-idempotent request at all. Which gives the rule the rest of this guide hangs on: before your client retries a POST or a PATCH, it needs either a key the server recognises or proof the first attempt never landed. Anything else is a coin flip with a customer's money.
Why a Timeout Can Charge a Customer Twice
Say a checkout service sends POST /payments. The server validates the request, charges the card, writes the order row, and starts serialising the response. Then a load balancer recycles the connection and the response never arrives. The client waits its configured timeout, sees a socket error, and does the reasonable thing. It retries.

Now there are two charges, two order rows, and one customer who will find the duplicate on a statement three days later. Nothing crashed. Nobody wrote a bad line of code. The retry was correct behaviour by a client that couldn't see the outcome.
The Three Things a Timeout Can Mean
After a timeout, the client has no way to distinguish these:
- The request never arrived. It died in transit and no work happened. A retry is safe and necessary.
- The request was processed and the response was lost. The work happened. A retry duplicates it.
- The request is still running. The server is mid-flight on a slow upstream call, and a retry races the first attempt. Which one wins depends on timing.
Only the first case is genuinely a retry. The other two are a second operation wearing the same name.
The damage isn't limited to money. Duplicate orders, duplicate records, background jobs that each burn credits or quota, notifications that fire twice, and downstream systems that disagree about how many of something exist and take a reconciliation job to untangle.
Removing retries doesn't solve it either. Connections drop on every deploy, every mobile network handoff, every idle-timeout setting between you and the origin. A client that never retries converts transient network noise into user-visible errors for requests that already succeeded.
The scale isn't marginal. A 2026 GitHub engineering document on tool idempotency assumption failures estimates that 20 to 40 percent of tool-call timeouts under load are response-delivery failures in which the underlying action completed. At the top of that range, nearly four timeouts in ten are completed work whose answer went missing.
The picture changes again once you separate synchronous API responses from asynchronous delivery. A synchronous call that times out is an open question. An asynchronous submission that timed out after returning a job identifier is a different one, and the submission endpoint still has to decide whether a repeated call starts a second job or hands back the first.
The goal for the rest of this guide isn't to retry less. It's to retry as freely as the network demands without ever producing a second effect.
How Idempotency Keys Make a POST Retryable
The mechanism fits in one sentence. The client names the operation, and the server remembers the name.
Most explanations stop there and tell you to send the same key again, which skips every interesting part. Here's the full sequence with the reason each step exists.

- Generate a unique key before the first attempt. A UUID version 4 is the usual pick: high entropy, no meaning encoded, nothing sensitive inside. Create it when the logical operation is created, not when the HTTP request is assembled, or a rebuild will produce a different name for the same intent.
- Send it in an Idempotency-Key header on the POST. The name and the work travel in the same message, so the server receives them together and can't act on one without the other.
- Look the key up, scoped to the caller's credential. A bare key is a global namespace. One tenant can collide with another's key, and worse, can probe for another tenant's stored response.
- If the key is new, insert it as PROCESSING and run the operation. The insert comes before the work and has to be atomic, because two retries 40 milliseconds apart will otherwise both see nothing.
- Save the final status code and response body against the key. Failures too. A stored 500 should replay as a 500 instead of running the charge again.
- On any retry with the same key, return the saved result and skip the work. This is the entire point. The client gets an answer, and the effect count stays at one.
- Keep the same key until you have a final answer. A timeout or a 409 isn't final, so the key survives. A 201 or a 422 is final, so it doesn't need to.
AWS calls the same idea a caller-generated request identifier, exposed across many of its APIs as a ClientToken. Its guidance on making retries safe with idempotent APIs explains that requests from the same caller carrying the same identifier are treated as duplicates, and a retry takes on the meaning of the first successful request. Their SDKs and command-line tools generate the token automatically for operations with an idempotent contract and reuse it during retries, which is how the platform offers an at-most-once commitment without the developer thinking about it.
The slip that kills this in practice is regenerating the key inside the retry loop. Every attempt then looks like a new order to the server, and your carefully designed dedupe table dutifully creates three of them. Generate once, pass it down, log it, and print it in the error message so support can search on it.
Here's the flow end to end, including the moment the server refuses to start a second job.
Success looks unremarkable. However many attempts the access log shows, the ledger shows one charge, the payments table shows one row, and the statement shows one line.
What the Server Must Store and Return
None of the above survives without durable storage. An in-memory dictionary of seen keys dies on the next deploy, and deploys are precisely when retries cluster. You need a table in your database, with a unique constraint on (credential_id, key).
Each row carries the key, the caller's credential, a fingerprint (a hash of method, route and normalised body), the state, the saved status code, the saved response body, and timestamps. That row is your idempotency record, and it belongs in the same database, and the same transaction, as the business data it protects.
| Situation | Server response | Why |
|---|---|---|
| First request with a new key | Run the operation, store the key as PROCESSING, return the result | Creates the single logical operation |
| Same key after completion | Return the stored status and body without new work | Replay prevents a duplicate effect |
| Same key after a 500 failure | Return the stored 500 and body without new work | The first attempt may already have had effects, so re-running is unsafe |
| Same key while still processing | Return 409 Conflict, client retries later | Prevents concurrent duplicates |
| Same key with a different body | Return 422 Unprocessable Content and run nothing | Key reused for a different operation is a client bug |
| Same key after retention expires | Treat as a new request | Retention must outlast the replay window |
Why the Unique Constraint Is the Whole Mechanism
The naive version reads the table first and acts second. Two requests arriving simultaneously both find nothing, both proceed, and the card is charged twice. No amount of correct header handling fixes that. The claim and the check have to be one operation, so make the insert the check: ON CONFLICT DO NOTHING RETURNING id returns zero rows when you lost the race. Your database then decides which request proceeds, atomically, without a lock in application code.
A Different Body Under the Same Key
Reusing a key with changed parameters is a client bug, not a retry. Returning the original result would hide the bug and answer a question nobody asked. Hash the parts that define the operation, normalise the body so key order doesn't matter, compare, and reject the mismatch with 422.
Retention and What Expiry Means
The window has to outlive the longest replay any client, queue or background job might attempt. That's why the Stripe idempotency documentation, the closest thing the industry has to a reference implementation, notes that API v1 keys may be removed once they're at least 24 hours old, after which the same key is a new request. Enforce expiry with an index and a background sweep, because an idempotency table that grows forever eventually becomes the slowest write in your system.
Store the failed outcome too. A replay of a request whose first execution returned 500 should return that stored 500 rather than starting over, because the first attempt may have moved money before it broke.
If the endpoint you're protecting is the one that moves payments, the surrounding flow matters as much as the dedupe table. Rite NRG's payment processing integration guidance walks through the pieces around the charge, from tokenisation to reconciliation, and it's worth reading before you decide what a replayed response should contain.
The Test That Proves It
Three calls settle it. Send the same key with the same body twice and compare the responses byte for byte. Then send the same key with one field changed and expect a 422. If either check surprises you, the table isn't doing the work you think it is.
Pitfalls That Break Idempotency in Production
The most common production failure isn't a missing key. It's a key generated inside the retry loop. The developer knows the header is required, so the request builder adds one, and the request builder runs on every attempt. Each retry carries a fresh name, the server sees three unrelated operations, and the safeguard becomes a formality that makes the code look correct in review.
| Pitfall | What goes wrong | Fix |
|---|---|---|
| New key generated on every retry | Each retry looks like a new order | Generate the key once per logical operation and reuse it |
| Key derived from payload hash alone | Two identical legitimate orders collapse into one | Client chooses a key that identifies the intended operation |
| Stored response treated as live data | Stale results returned indefinitely | Replay only within the window, then fetch live |
Hashing the request body and calling the hash a key is the second trap, and it's an attractive one because it removes the need for client discipline. But two legitimately identical requests occur routinely. The same customer buys the same coffee twice in a minute, or a data team submits the same lookup for a profile it already enriched yesterday. Content hashing detects duplicates. An idempotency key identifies one intended operation, and only the client knows which operation it meant. Deriving the key from a stable object the user is already acting on, such as a cart identifier or a draft record, is a different matter and works well, because the object names the operation while the payload merely describes it.
The third trap is confusion about freshness. A stored response is a record of one attempt, not a source of truth, and treating it as a cache turns your dedupe table into a stale-data service. Two different keys can and should return different live results from the same endpoint, because they represent two separate requests. Replay the stored outcome only inside the retention window, and let anything after that fetch live.
Simultaneous Duplicates Are a Different Failure
A late retry meets a finished row and gets the saved answer. A simultaneous duplicate meets a row stuck in PROCESSING and must be told to come back. Both have to end with one effect, and only one of them is solved by a lookup. The other needs the atomic claim.
Test the second case properly: fire two identical requests from two threads with no sleep between them, against a staging database, and assert afterwards that your business table gained exactly one row, one charge, or one background job. Sequential tests pass on broken code, which is why so many broken implementations ship with a green suite.
How Often to Retry and Which Errors to Retry
Start with the response, not the clock. A 400, 401, 403 or 404 means the request itself is wrong, and a retry sends the same wrong request.

| Response or failure | Retry? | Condition |
|---|---|---|
| 400 Bad Request | No | Fix the request first |
| 401 or 403 | No | Fix credentials or permissions |
| 404 Not Found | No | Resource is absent |
| 409 Conflict | Later | Wait, then replay with the same key |
| 429 Too Many Requests | Yes | Wait for Retry-After before retrying |
| Network timeout | Yes | Only with the same idempotency key |
Backoff That Doesn't Stampede
Retry at roughly 200 ms, double the wait each attempt, and cap the delay near 30 seconds. Then add full jitter: pick a random wait between zero and that delay instead of waiting exactly that long. Without jitter, a thousand clients that failed together retry together, and the second wave is as heavy as the first. Cap attempts at about five, honour the Retry-After header on a 429 rather than guessing, and hold a retry budget, keeping retries under roughly 10 percent of total traffic so a misbehaving dependency can't multiply your request volume.
The intuition to keep is that backoff protects the server and jitter protects you from other people's backoff.
When Retries Become the Outage
Idempotency alone doesn't tell you whether a retry is operationally safe. A 2025 analysis of idempotency-safe retries in service meshes makes the case that concurrency, latency tolerance, failure probability and spare capacity all belong in the decision, and warns against disabling retries indiscriminately to avoid a retry storm, since that trades duplicated work for hard failures. During an outage every client retries at once, and the retries become the load keeping the service down. A key prevents a duplicate effect. It doesn't prevent ten thousand clients from each holding a connection slot.
Billing shouldn't be your reason to cut retries short: retries after a 429 or 5xx error are free until the retry succeeds. Rate limits and retry policy are two halves of one control, and our guide to API rate limits covers how limits are counted and what the response headers tell a client about waiting.
When you're mapping codes to decisions, a reference that states each error's meaning saves an argument. Openbase's error response documentation lists the codes alongside the situations that produce them.
The rule that outranks every setting in this section: retry a non-idempotent write after a timeout only when it carries a key. Everything else is tuning.
A Working Idempotent POST Example
A payment request can time out after the server has charged the customer but before the response reaches the client. The client cannot tell whether the charge happened. Reusing one idempotency key lets it retry the same operation and receive the stored result rather than trigger another charge.
import random, time, uuid
import requests
def create_payment(amount_cents, currency, max_attempts=5):
key = str(uuid.uuid4()) # once per logical operation, not per attempt
body = {"amount": amount_cents, "currency": currency}
for attempt in range(max_attempts):
try:
r = requests.post(
"https://api.example.com/v1/payments",
json=body,
headers={"Idempotency-Key": key},
timeout=10,
)
except (requests.Timeout, requests.ConnectionError):
pass # outcome unknown: same key next time
else:
if 200 <= r.status_code < 300:
return r.json()
if r.status_code == 409:
pass # still processing: back off, replay
elif r.status_code == 429:
retry_after = r.headers.get("Retry-After")
if retry_after:
time.sleep(float(retry_after))
continue
elif 400 <= r.status_code < 500:
raise RuntimeError(r.text) # fix the request, don't retry
time.sleep(min(2 ** attempt, 30) + random.uniform(0, 1))
raise RuntimeError(f"payment {key} still unresolved")
The 429 branch waits for the server's Retry-After value, then repeats the request with the same key. A 409 means the first request is still processing, so the client backs off and checks again. Other 4xx responses, such as a validation error, require a corrected request rather than an automatic retry. A timeout remains ambiguous: the server may have completed the charge, or it may not have received the request.
The server must record the key before doing the work. Its fingerprint should cover the method, path, and canonicalized body, so differences in JSON key order or whitespace do not look like different operations.
import hashlib, json
def fingerprint(method, path, body):
canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(f"{method}:{path}:{canonical}".encode()).hexdigest()
@app.post("/v1/payments")
def post_payment():
key = request.headers.get("Idempotency-Key")
if not key:
return jsonify(error="Idempotency-Key required"), 400
fp = fingerprint("POST", "/v1/payments", request.get_json())
claimed = db.execute("""
INSERT INTO idempotency_keys (tenant_id, key, fingerprint, state)
VALUES (:tenant, :key, :fp, 'PROCESSING')
ON CONFLICT (tenant_id, key) DO NOTHING
RETURNING id
""", {"tenant": tenant_id, "key": key, "fp": fp}).fetchone()
if claimed is None: # someone already owns this key
row = db.get_key(tenant_id, key)
if row.fingerprint != fp:
return jsonify(error="key reused with different parameters"), 422
if row.state == "PROCESSING":
return jsonify(error="in progress"), 409, {"Retry-After": "1"}
return jsonify(row.body), row.status_code
result, status = charge(request.get_json())
db.finish_key(tenant_id, key, status, result) # same transaction as the charge
return jsonify(result), status
The insert claims the key before payment work begins. The charge and the saved response must commit together. If they commit separately, a crash after charging but before saving the result leaves the key marked PROCESSING; a retry then cannot reliably tell the client what happened.
Here is the same operation from a terminal:
curl -X POST "$API_HOST/v1/payments" \
-H "Idempotency-Key: 6f1c2a34-8b7e-4a1d-9c2f-2d5b7a9e1c40" \
-H "Content-Type: application/json" \
-d '{"amount": 2500, "currency": "usd"}'
Run the command again with the identical key and body. The second response should contain the same payment identifier and status, with no additional charge in the ledger. Change one field while keeping the key, and the server should return 422: one key identifies one operation. For guidance on shaping routes, headers, and error bodies consistently, see this API design example walkthrough.
Check Your Idempotency Design Before You Ship
A retry can look correct in a sequential test and still create duplicates under load. Review these nine checks with the endpoint change, and assign each one to an owner.
- Every non-idempotent endpoint accepts a key. Owner: API lead.
- The client generates the key once per logical operation. Owner: the team that owns the caller.
- Keys are scoped to the caller's credential, so tenants cannot collide or probe one another.
- The server compares fingerprints and rejects mismatches with 422.
- State is durable and survives a restart mid-request.
- Retention exceeds the longest replay window a client or queue might use.
- Retries cover only 429, 5xx, and timeouts, with backoff and jitter.
- Concurrent duplicates are tested, not just sequential ones.
- Duplicate attempts are logged and monitored, with the key in the log line so support can trace an operation.

Prioritise operations that move money or start billable background work. Duplicates there can charge customers twice or repeat work. Then cover record and order creation, where duplicates create support and reconciliation problems. Read-only GET requests are already idempotent, so adding a key to them usually adds work without changing the outcome.
Failed API calls are never billed. That means an attempted retry that ends in an error does not spend the customer's credits.
Run the concurrency and payload-mismatch tests against staging before the next release. A sequential test alone will not expose races between simultaneous requests.



