How do you make payment finalization idempotent when two paths can both complete it?

Give both finalization paths — the payment gateway's server-to-server callback and the frontend's own status polling — a shared code path guarded by a single idempotency check, such as a fresh-success flag that only one caller can claim. Whichever path reaches it first performs the downstream work (marking the payment complete, syncing to dependent applications); the other sees the flag already set and does nothing. That way it doesn't matter which path finalizes first, only that exactly one of them does the work.

Common Login Portal

Three clocks racing: a payment reconciliation bug

Ashish Bhagat · 13 Aug 2026

The scenario that worried me most about CLP's payment flow wasn't a bug report. It was a question I made myself answer before it became one: what happens if a user's bank takes its time confirming 3-D Secure, and the frontend gives up waiting before the bank actually says yes?

The payment flow, in brief: submit registration, get an invoice with a 10-minute expiry, send an AES-128-CBC encrypted payment request to a Kale-operated gateway in front of Bank Muscat, the bank hosts its own payment page, and once the user's done there the bank posts an encrypted response back to CLP's backend, which finalizes the payment and redirects the SPA to a status page that polls until it sees a result. Three separate pieces of that flow each keep their own sense of time, and none of them were designed together.

Three clocks racing

None of these three numbers are wrong in isolation, either — that's what makes this a genuinely interesting failure mode rather than an obvious oversight. A 10-minute invoice expiry is a reasonable business window. A 2-minute frontend timeout is a defensible UX call; nobody wants a spinner that could run indefinitely. A 60-second reconciliation cycle is a sensible balance between promptness and load on the backend. Each choice is locally correct. The bug lives entirely in the gap between them, which is exactly the kind of bug that a review of any single component, in isolation, will never surface.

The invoice itself is valid for 10 minutes — a business rule, not a technical one, but it's a clock. The frontend, once it lands on the status page, polls for roughly 2 minutes before it stops and shows the user something. The backend runs its own reconciliation job on a 60-second cycle, independent of anything the frontend is doing. None of those three numbers were chosen with the other two in mind, because they answer different questions: how long is this invoice worth honoring, how long should a user reasonably stare at a spinner, how often should the backend check in on payments nobody's actively watching.

Put them next to each other and there's an obvious gap. A 3-D Secure confirmation that takes a bit longer than usual — nothing exotic, just a slow bank-side round trip — can easily outlast the frontend's 2-minute patience while still comfortably finishing inside the invoice's 10-minute window. The payment succeeds. The user, by then, is looking at a page that already gave up on them.

The payment that succeeds after you've stopped watching

This is the failure mode worth designing around, and it's not really a race condition in the classic shared-mutable-state sense — it's a race between what the user's browser is willing to wait for and what the bank is actually willing to guarantee. The browser can close. The tab can be killed. None of that unwinds the payment; the bank doesn't know or care that the frontend stopped polling. If the only thing tracking payment completion is a live browser tab, then closing that tab — on purpose, by accident, or because a 2-minute timeout gave up on the user's behalf — orphans a payment that went through.

Two finalization paths, one idempotency guard

The fix treats browser presence as optional, not required, for a payment to reach a correct final state. There are two independent ways a payment can get finalized: the bank's own server-to-server callback, which is authoritative and fires whether or not the user's browser is even open anymore, and the SPA's own status polling, which is a convenience for the user who's still there watching, not the source of truth. Either one is allowed to be the one that actually finalizes the payment and triggers downstream app sync.

Having two paths that can both reach the same finalization logic means both paths can, in principle, race each other and try to finalize the same payment twice — which is worse than either failure mode on its own if it means a downstream system gets synced twice for one payment. Both paths funnel through a shared idempotency guard, a freshSuccess check, before any downstream work happens. Whichever path gets there first does the work; the other sees it's already done and stops. It doesn't matter which one wins the race, only that exactly one of them does.

Two schedulers as a safety net

Designing around browser presence being optional isn't just a payments concern — it's a reasonable default for anything where the business outcome (a payment succeeding) and the UI's job (showing the user something reassuring) are allowed to diverge. The moment those two things can legitimately disagree, whichever one you've made authoritative had better not be the one that depends on a tab staying open.

Underneath both of those paths, two backend schedulers run independent of either the bank's callback or the frontend's polling ever showing up at all: PaymentReconciliationSchedulerService, which periodically checks payments that should have resolved and haven't, and PaymentExpirySchedulerService, which enforces the 10-minute invoice expiry from the backend side rather than trusting the frontend to notice and react to it. Between the two callback-driven finalization paths and these two schedulers, a payment's final state doesn't depend on any single client staying connected, polling on schedule, or receiving a callback cleanly. It depends on the backend eventually noticing, on its own clock, regardless of what any browser did.

A requirement that arrived mid-project

Payments were originally mocked whenever the rest of the flow was running against the mocked SAML identity provider, which made sense for local and lower-environment development. Partway through the project, that assumption changed: even the mock-SAML flow now had to process a real payment. That meant the Spring bean wiring that decided which payment adapter got injected — governed by @Primary and @Profile — had to be reworked so the real payment adapter was selected by default in every environment except one explicit mock-payment profile, instead of the mock adapter quietly winning whenever SAML happened to be mocked. Two toggles that used to move together suddenly needed to move independently, and Spring's bean resolution doesn't care that they used to be correlated — it just picks whichever bean actually matches the active profile.

What I'd do differently

The three-clocks problem is the kind of thing that's much cheaper to catch by asking "what if this is slow" on a whiteboard than by finding it in production. I did ask it here, before it became an incident — which is the version of this story I'd want to repeat. The bean-wiring change is the one I'd do differently: I'd decouple "which identity provider" from "which payment adapter" from the start, as two independently-set profile flags, rather than two concerns that happened to travel together until a client requirement proved they shouldn't.