Idempotency keys are not optional in a payments system
The fastest way to lose trust in a payments product is to charge someone twice. The second fastest is to pay a seller twice. Both come from the same root cause, a retry that was not idempotent.
Here is the short version of what I have learned putting real money through Stripe.
The bug
Your API call to create a charge succeeds on Stripe’s side, but the response never makes it back to you. A timeout, a dropped connection, a redeploy mid request. Your job queue does what it is supposed to and retries. Now there are two charges and one very unhappy customer.
The network will do this to you. Not often. Just often enough that “it works in testing” guarantees nothing.
The fix, and the part people skip
Stripe, and most serious payment APIs, lets you pass an idempotency key. Same key, same result. The second request returns the first response instead of doing the work again. Easy.
The part people skip is scoping the key correctly.
Derive it from something stable about the intent, not the attempt. A fresh UUID per retry defeats the entire mechanism. That is the most common mistake I see.
Tie it to your own domain object, this order or this payout, so a retry of “pay out invoice 1234” reuses the same key no matter how many times the worker wakes up.
Persist the key before you make the call, in the same transaction that records your intent to charge. If you generate it in memory and the process dies, the retry cannot find it, and you are back to double charges.
The rule
Every state changing call to a payments API needs an idempotency key derived from a persisted intent. No exceptions, no “this endpoint is probably fine.”
It is a few lines of code that prevents the single most damaging class of bug in the whole system. In a payments product, the unglamorous reliability work is the product.
I'm Adam. I architect and rescue Stripe Connect payment platforms. I built a full B2B2C healthcare marketplace on Connect end to end: split payments, payouts, and ledger reconciliation. If you're a founder or eng lead building or fixing one, say hello.