Same /16, the big 3 cloud providers differ on how far it reaches

cloudnetworkawsgcpazure

The anycast post was about one IP living on many machines at once. This one is about a different kind of duplicate: my home router, my AWS account and a friend's GCP project all use 10.0.0.0/16, and none of them clash.

Why don't duplicate addresses cause trouble? And on AWS, Azure and GCP, how much ground does one block cover?

Like office extensions, only valid inside

Every company can have an extension 105, and two companies' 105s never collide, because an extension only means something inside that company's own phone system. Outside calls use a different number.

10.0.0.0/16 works the same way. It is in a private range reserved by RFC 1918, which tells ISPs to filter these ranges, so they are never routed on the internet and anyone can reuse them inside their own network (RFC 1918).

In CIDR (Classless Inter-Domain Routing) notation, the /16 only says how many addresses there are: the first 16 bits are fixed and the last 16 are free, 65536 in total. It says nothing about where the machines are, just as an extension number says nothing about which floor the phone is on.

How far one block reaches in each cloud

In the cloud, the private network belonging to one account is a VPC (Virtual Private Cloud), the equivalent of one company's phone system (Azure calls it a VNet). A VPC is an abstract, logical thing: creating a 10.0.0.0/16 VPC declares that these numbers belong to this company.

A subnet is what maps onto an actual location: in AWS a subnet maps to one AZ, in Azure and GCP to a whole region. It is a slice of the VPC's range, such as 10.0.1.0/24, like assigning a block of extensions to one floor.

A region contains several zones; AWS's us-east-1, for example, has us-east-1a, us-east-1b and more. AWS describes an AZ as "one or more discrete data centers with redundant power, networking, and connectivity" (AWS Regions and Availability Zones). GCP doesn't say how many data centres a zone has; it says zones are designed so that power, cooling or networking outages usually affect only one zone, so if a zone goes down, traffic can move to another (GCP regions and zones). All three clouds have this layer: AWS calls it an AZ (Availability Zone, e.g. us-east-1a), Azure numbers them zone 1, 2, 3, and GCP calls it a zone (e.g. us-west1-a). Below, AWS gets "AZ" and everything else says "zone".

VMs are often spread across two zones so that one keeps running if the other has trouble. Here is how the subnets work out in each cloud:

A subnet maps toVMs in two zones
AWSone AZtwo subnets: zone a uses 10.0.1.x, zone b uses 10.0.2.x
Azureevery zone in the regionone subnet is enough, all in 10.1.1.x
GCPevery zone in the regionone subnet is enough, all in 10.240.0.x

AWS keeps a subnet inside one AZ. To put VMs in two AZs you need two subnets, like a block of extensions that can only be installed on one floor.

Azure's VNet also stays within one region, but one subnet covers every zone in it. You pick the zone when you create the VM.

GCP's VPC is global: one VPC can hold subnets in both us-west1 and asia-east1, like one company's phone system spanning branch offices in several countries. Subnets are still per region.

Two VPCs both have 10.0.0.5: how does a packet reach the right one?

Two companies sit in the same building, share the building's phone lines, and both have extension 105. When a call comes from company A's line, the switchboard knows it is company A's, looks up company A's list, and rings company A's 105.

The cloud does the same. Two accounts each have a VPC with a machine at 10.0.0.5, running on the same physical network underneath:

In the extension pictureIn the cloud
Extension 105The address 10.0.0.5 inside the VPC
Where the phone actually isThe address of the physical machine (host) the VM runs on, e.g. 10.51.0.7
The switchboard's listA mapping table, each entry "this IP in this VPC → on this host"
The switchboardThe part of each host that forwards packets: software in GCP, a hardware card in AWS

A VM's network card is virtual, and its other end plugs into the part of the host that forwards packets, called the switch below. When the cloud creates a VM, it records on the switch which VPC that card belongs to. So as soon as a packet enters the switch, the switch knows its VPC, looks up which host the destination VM is on, and wraps the packet in an outer layer addressed host to host (10.51.0.7 → 10.51.0.9). The outer layer usually also carries a tag saying which VPC the packet is going into. VXLAN, a common wrapping format, has a 24-bit VNI (VXLAN Network Identifier) field in its outer header to tell virtual networks apart (RFC 7348). None of the three clouds publishes its own outer format, though, so they may not use VXLAN. On the other side, that host's switch removes the outer layer and hands the original packet to the VM. The two VMs only ever deal with addresses in their own VPC.

The host addresses here are made up; providers don't publish theirs. In GCP that part is Andromeda's software running on the host; in AWS it is a hardware card, the Nitro Card for VPC, which the whitepaper calls a hardware offload engine. The cloud's control plane maintains the table and pushes it to every host. Google describes its own in the Andromeda paper; AWS's Nitro whitepaper only says the Nitro Controller is the sole path between a host and the VPC control plane, not what the table looks like.

How does GCP's global VPC cross continents? Andromeda

The mapping table has one entry per address, recording which host the VM is on. So within one 10.0.0.0/16, 10.0.1.5 can be on a host in the US and 10.0.2.5 on a host in Taiwan, like dialling extension 105 and having the switchboard put you through to an overseas branch over the company's own line.

In GCP that layer is Andromeda. When a packet goes from us-west1 to asia-east1, Andromeda on the sending host finds which host the destination VM is on and sends the packet over Google's own backbone. The packet still physically crosses the Pacific. Andromeda also does DDoS protection, load balancing, access control lists and firewalls in software, instead of racks of separate appliances (Google's announcement).

Why VPCs with overlapping ranges can't be peered

Peering connects two VPCs directly so their VMs can reach each other on private IPs. AWS, GCP and Azure all forbid overlapping ranges and refuse when the peering is created.

None of them says why. Think of extensions again: two companies both have extension 105, and once their phone systems are joined, which phone should ring when someone dials 105? An AWS route table has the same problem. Every route table starts with a local route covering the VPC's own CIDR. Say VPC A and VPC B are both 10.0.0.0/16, each with a machine at 10.0.0.9. VPC A's route table matches a packet for 10.0.0.9 to the local route and treats it as VPC A's own traffic. A route table uses the most specific matching route, so the obvious fix is to add 10.0.0.9/32 → peering, narrower than local. But AWS only allows routes more specific than local to target things inside the VPC, such as an ENI (Elastic Network Interface, a VM's network card), a NAT gateway or a Gateway Load Balancer endpoint, not a peering connection (AWS route table options). So a packet VPC A sends to 10.0.0.9 can never reach VPC B's machine. This is inferred from AWS's route table rules, not something AWS has published.

Andromeda doesn't get GCP out of this either. The mapping table earlier can tell which VPC a packet came from, because the switch knows which VPC each network card belongs to. But when a VM in VPC A sends to 10.0.0.9, the packet carries only the destination address; it doesn't say whether it means VPC A's own 10.0.0.9 or VPC B's. The VPC tag on the outer layer doesn't help either. With peering, the tag has to name the destination's VPC, and the switch only knows which VPC 10.0.0.9 is in after a table lookup. The table has two entries for 10.0.0.9, one in VPC A and one in VPC B, so the switch can't pick one, and there is nothing to write in the tag. It is like a switchboard that knows the call came from someone at company A, but when that person dials 105, can't tell whether they want their own company's 105 or company B's. This is also an inference; GCP's documentation only says it checks for overlap when the peering is created.

AWS's answer is NAT, the equivalent of each company getting a separate, non-overlapping set of numbers for calling each other. In AWS's example, VPC A adds 100.64.1.0/24 and VPC B adds 100.64.2.0/24, and a transit gateway joins the two VPCs for those two ranges only. When a VM in VPC A calls a service in VPC B, a private NAT gateway first rewrites the source address to 100.64.1.x, then sends the packet to VPC B's load balancer in 100.64.2.0/24. Traffic between the two uses only 100.64 addresses, so the overlapping 10.0.0.0/16 no longer matters.

Why the three are designed this way

None of the three has said outright why its network is designed this way, but what each has said in public lines up with the design.

AWS: the AZ is the unit that fails. AWS treats the AZ as a hard fault boundary. With a subnet locked to one AZ, spanning AZs takes several subnets, so architectures end up spread across AZs, and when one AZ has trouble the subnets in the others keep running (AWS Fault Isolation Boundaries).

GCP: Google already had a global network, so it handed the whole thing over. When it announced Andromeda in 2014, Google called it making "the latest networking technology that powers our internal services" available to Cloud Platform users (Enter the Andromeda zone). Google's own services already ran on a worldwide network; a global VPC gives customers that same network.

Azure: the way enterprise IT already works. Microsoft says it "designed Azure to be hybrid from day one" (Microsoft Azure on hybrid). A VNet can connect back to a company's own data centre over VPN or ExpressRoute, so it feels like the company's internal network with one more piece in the cloud.

References