The Tech Praxis · Systems & Infrastructure

Confronting the Physical Limits of Distributed Agreement

Inside one data center, Raft and Paxos feel almost mechanical in their reliability. Stretch the same algorithms across New York, London, and Tokyo, and what remains is a confrontation with the speed of light — a limit no amount of engineering cleverness can fully erase.

A local consensus protocol operates in a world of microsecond messages and predictable timeouts. Move the same nodes across continents and that world dissolves into jitter, partial failure, and clocks that no longer agree on what "now" means.

This is not a shortfall in engineering talent — it is physics. And as multi-region and multi-cloud deployments become the default rather than the exception, that physics stops being a textbook footnote and becomes a daily operating constraint for anyone running a global system.

1	The Tyranny of Distance

Light travels through optical fiber at roughly 200,000 kilometers per second — about two-thirds of its speed in a vacuum. Between New York and London, that puts the theoretical floor for a round trip somewhere in the tens of milliseconds; real routing, queuing, and switching push it well beyond that floor. Between New York and Tokyo the round trip stretches past 100 milliseconds. None of this is a bug to be optimized away — it is the actual cost of moving a signal around a curved planet.

Consensus protocols were not designed with that constraint foremost in mind. Raft's leader election and heartbeat timeouts assume a low-latency, mostly reliable link. A 150-millisecond timeout is comfortable inside a data center; across an ocean, that same timeout invites constant leader churn. A leader in Virginia sends a heartbeat to a follower in Tokyo; the follower, hit by ordinary network jitter, times out and calls an election before the original heartbeat even lands. Multiply that across a cluster and you get not a single failed election but a cascade of them — the system spending more time arguing about who is in charge than doing useful work.

This is why protocols built for wide-area networks have become an active research frontier rather than a solved problem. Mysticeti-C, a DAG-based Byzantine consensus protocol from the team behind the Sui blockchain, was the first Byzantine protocol to bring WAN consensus-commit latency down to roughly 0.5 seconds while sustaining over 200,000 transactions per second — cutting the median latency of Sui's prior consensus stack by more than 4x. Half a second sounds slow by data-center standards; at true global scale, it is a genuine breakthrough.

2	Why Consensus Breaks Down at Global Scale

Latency is only the first problem. Three other failure modes compound it once a consensus group spans continents.

Message complexity. Classic Paxos and Raft variants generate communication that scales roughly with the square of the participant count in the worst case. Five nodes in one rack barely notice. Hundreds of nodes spread across three continents spend an increasing share of their capacity exchanging votes and heartbeats rather than doing useful work — and adding Byzantine fault tolerance, with its cryptographic signing and extra rounds, makes the overhead worse still. This is precisely why production systems avoid putting every node into one giant global group and instead partition the problem, an approach covered below.

Network partitions. Submarine cables get cut — by anchors, seismic activity, or fishing trawlers — far more often than most engineers assume. When a cable severs a region from the rest of a cluster, a poorly designed quorum can split into two islands that both believe they hold a majority, each accepting writes the other doesn't know about. Avoiding that split-brain outcome is the entire reason quorum systems require a strict majority rather than "most of the nodes I can currently see."

Clock skew and ordering. Consensus depends on establishing an order of events, but there is no global clock to consult. Lamport timestamps and vector clocks give partial ordering, not total ordering. Google's Spanner solved this for its own infrastructure with TrueTime, an API that exposes clock uncertainty as a bounded interval — backed by GPS receivers and atomic clocks in every data center — and a deliberate "commit wait" that holds a transaction until its timestamp is provably in the past. It is an elegant solution. It is also hardware-dependent, expensive, and effectively available only to an organization with Google's infrastructure budget.

3	How Production Systems Actually Survive

Given limits that no protocol can fully engineer around, real systems don't try to eliminate them — they design to contain the damage.

Sharded consensus groups. Rather than one global group voting on everything, partition state so each shard runs its own independent Raft or Paxos group, ideally with members clustered in the same region. CockroachDB is built exactly this way: each data range has its own Raft group, and cross-range transactions are coordinated separately, on top of — not inside — the consensus layer.

Hybrid consistency. Keep strong, linearizable consistency inside a region where latency is cheap, and fall back to asynchronous replication or conflict-free data types across regions where it isn't. This is a direct application of the CAP theorem as an engineering choice rather than a theoretical constraint: accept that two regions may briefly disagree, in exchange for both staying available during a partition.

Geography-aware placement and routing. Position replicas so that a quorum can typically be reached without crossing an ocean — say, two nodes in US-East, two in US-West, one in Europe — and route users to their nearest region via geo-DNS, reserving genuinely cross-continental consensus rounds for the operations that truly require it, such as a financial ledger write.

Observability as a first-class requirement, not an afterthought. A consensus bug that only appears when cross-region latency exceeds a specific threshold will not reproduce in staging — it will show up in production, intermittently, and be almost impossible to diagnose from a single region's logs. Distributed tracing, correlated logging across regions, and deliberate chaos testing are what turn a mysterious global outage into a diagnosable, fixable one.

Google's Spanner remains the clearest proof that these constraints can be engineered around, not eliminated. By combining Paxos-based replication with TrueTime, Spanner delivers external consistency and full SQL transactions at global scale — a combination the database community had long treated as mutually exclusive with horizontal scalability. The achievement was significant enough that Spanner received the 2025 ACM SIGMOD Systems Award, recognized specifically for reimagining relational data management to enable serializability with external consistency at that scale. It is also, by its own design, not cheap or easily replicated — a demonstration of what becomes possible with specialized hardware and a large, sustained engineering investment, more than a template every organization can copy directly.

None of this is going away. The speed of light is fixed, the CAP theorem is not negotiable, and network partitions and clock skew will keep happening regardless of how carefully a system is built. What can change is the posture engineers take toward these constraints — treating wide-area consensus as its own design problem from the outset, rather than a data-center protocol stretched further than it was built for.

That means sharding instead of one giant global group, hybrid consistency instead of insisting on strong guarantees everywhere, geography-aware placement instead of uniform node distribution, and observability treated as mandatory rather than optional. The physics is unforgiving. The engineering, done with enough humility about what is actually possible, still works.