Chapter 32: Network

The network is the nervous system of the planetary scale computer. It connects servers within a rack, racks within a data center, data centers within a region, and regions across the globe. The characteristics of the network — bandwidth, latency, reliability, and cost — fundamentally shape the design of distributed systems.

Within a data center, networks are typically structured as a multi-tier hierarchy or a spine-leaf topology. Top-of-rack switches connect servers within a rack. Aggregation switches connect racks within a cluster. Core switches connect clusters within a data center. At each tier, bandwidth is aggregated and the failure domain widens.

Between data centers, network connections range from dedicated fiber links (for data centers in the same metropolitan area) to leased circuits and the public Internet (for intercontinental connectivity). Latency between data centers in the same region is typically 1-5 milliseconds; between continents, 50-200 milliseconds. These latency constraints directly affect system design: strong consistency across continents is expensive because every write must wait for a round trip.

Network failures are a major source of outages in distributed systems. Unlike server failures (which are typically independent), network failures can partition large groups of servers simultaneously, creating split-brain scenarios that consensus protocols like our consensus service are designed to handle.

Our three-region deployment uses a WireGuard mesh to create a private overlay network connecting SFO, NYC, and AMS. Chapter 24: Geo Replication describes how this mesh enables federated discovery and cross-region storage replication.