Common Login Portal

It was never the network

Ashish Bhagat · 13 Aug 2026

During UAT, calls from Common Login Portal to the Ministry of Labour's SOAP API started failing intermittently:

500 Internal Server Error
503 Service Unavailable
(occasional timeout)

No pattern anyone could immediately see. Not every request, not every hour, no obvious correlation with load. And there was a ready-made explanation sitting right there: the tester was in India, the government's server was in Oman, and "it's probably the network" is the easiest sentence in distributed systems to say and the hardest one to actually verify before you've wasted a week acting on it.

The India–Oman latency hypothesis

It's a reasonable first guess. Cross-border requests do add latency, government infrastructure isn't always generously provisioned, and "network flakiness between two countries" explains intermittent failures without implicating anyone's code. It's also the kind of explanation that's comfortable precisely because it doesn't require anyone to prove it — you just keep retrying and blaming the wire.

The numbers that killed it

The integration's configured timeout was 60 seconds. Real round-trip latency between India and Oman, for ordinary internet traffic, runs on the order of 100 to 250 milliseconds. That's not a rough approximation that happens to be in the right ballpark — it's roughly 240 times shorter than the window the client was willing to wait. For latency alone to cause a timeout, the network would need to be degraded by two and a half orders of magnitude beyond its normal behavior, consistently, across a mix of requests, without anyone's ordinary web browsing between the same two countries also grinding to a halt. That doesn't happen. Bad network days look like added milliseconds, not added minutes.

Why it couldn't be the network, by definition

There's a second, sharper argument, and it doesn't even need the latency numbers: HTTP 500 and 503 are responses. A server generates a 500 or a 503 after it has already accepted the connection and started handling the request — a network-path problem, by contrast, tends to show up as a connection timeout, a reset, or no response at all, not a status code with a body. Getting a 503 back means the request got there, and the government's own server decided, on its own infrastructure, to answer with "unavailable." That's not something the wire between India and Oman is capable of producing on the client's behalf. The government's servers were unstable. That wasn't a hypothesis anymore, it was the only explanation left standing once the other one had been checked against arithmetic.

Retrying for the right reason

I wrote this up formally, because "the government's systems are intermittently unreliable" is not the kind of conclusion you want resting on a hunch when it's about to justify a design decision — it needed to be something I could show, not just assert. The justified fix was retry logic: up to four attempts with backoff, on the calls to MOL and MOCI specifically. That's the correct mitigation for a flaky upstream you don't control and can't fix — not for a network path, which retries wouldn't meaningfully help if the real cause had been latency-driven timeouts in the first place.

A bug that looked like nothing

Chasing this down turned up something else entirely, sitting quietly in the same integration: MOL's production API returns the string "WORKING" for an active work permit, not "Active". Whatever code was checking that status field against the string it expected was checking against the wrong one, and had been failing closed for every real user trying to register — silently, with no exception, no retry to obscure it, just a status check that never matched. It's the kind of bug that a root-cause investigation into something else entirely is exactly positioned to catch, because you're already deep in the actual response payloads instead of trusting what the field was supposed to contain.

"It's probably the network" earns its popularity honestly — it's rarely anyone's fault, it doesn't implicate a specific piece of code, and cross-border infrastructure genuinely is less predictable than a same-region call. That's exactly why it deserves the same scrutiny as any other explanation before you act on it, rather than less. The fix here wasn't a clever diagnosis technique. It was refusing to let a plausible-sounding story stand in for a number, and then writing the number down before deciding what to build next.

Neither of these was a network problem. One was the government's own infrastructure being unreliable, which retries can absorb. The other was an assumption about a string value that production quietly disagreed with, which no amount of retrying would ever have fixed.


More from this series:

Hand-rolling SOAP clients against undocumented APIs · Mock and real adapters behind every port