AWS regions, AZs, and VPCs, NICs, IPs, and performance

Jump to section:

Availability zones and regions

AWS’ primary cloud is available in 15 regions, each with two to six availability zones, not including separately operated regions (with independent identity) for GovCloud and China. Most AWS services operate independently in each region (though identity is shared across regions in the primary cloud), and each service has its own (often region-specific) endpoint (many libraries and the AWS CLI simply insert the region name in the endpoint URL). Though an AWS account is global, most resources are scoped to a specific region, and some resources are tied to specific availability zones.

AWS’ documentation describes:

Regions

Each Amazon EC2 region is designed to be completely isolated from the other Amazon EC2 regions. This achieves the greatest possible fault tolerance and stability.

When you view your resources, you’ll only see the resources tied to the region you’ve specified. This is because regions are isolated from each other, and we don’t replicate resources across regions automatically.

When you launch an instance, you must select an AMI that’s in the same region. If the AMI is in another region, you can copy the AMI to the region you’re using. For more information, see Copying an AMI.

Note that there is a charge for data transfer between regions. […]

Availability Zones

When you launch an instance, you can select an Availability Zone or let us choose one for you. If you distribute your instances across multiple Availability Zones and one instance fails, you can design your application so that an instance in another Availability Zone can handle requests.

You can also use Elastic IP addresses to mask the failure of an instance in one Availability Zone by rapidly remapping the address to an instance in another Availability Zone. For more information, see Elastic IP Addresses.

An Availability Zone is represented by a region code followed by a letter identifier; for example, us-east-1a. To ensure that resources are distributed across the Availability Zones for a region, we independently map Availability Zones to identifiers for each account. For example, your Availability Zone us-east-1a might not be the same location as us-east-1a for another account. There’s no way for you to coordinate Availability Zones between accounts.

As Availability Zones grow over time, our ability to expand them can become constrained. If this happens, we might restrict you from launching an instance in a constrained Availability Zone unless you already have an instance in that Availability Zone. Eventually, we might also remove the constrained Availability Zone from the list of Availability Zones for new customers. Therefore, your account might have a different number of available Availability Zones in a region than another account.

Facilities

Perhaps surprisingly, an individual availability zone can actually be larger than a single physical data center or building. A single AZ can be made up of up to eight physical data centers or facilities, but there does not appear to be any mechanism in AWS to determine locality at that scale.

The only public documentation or mention of this detail is found within S3 materials. S3 may be the only AWS service that utilizes facilities, as explained in this slide from an AWS re:Invent session:

These details about the number of AZs and facilities are significant to S3’s availability and durability:

Amazon S3 is designed to sustain the concurrent loss of data in two facilities. […] Amazon S3 Standard and Standard - IA redundantly stores your objects on multiple devices across multiple facilities in an Amazon S3 Region. The service is designed to sustain concurrent device failures by quickly detecting and repairing any lost redundancy. When processing a request to store data, the service will redundantly store your object across multiple facilities before returning SUCCESS. Amazon S3 also regularly verifies the integrity of your data using checksums.

Again, though S3 appears to take advantage of individual facilities, there is no representation of them in the public APIs and users are likely unable recognize or take advantage of the locality in their own applications.

AZ naming and visibility

As AZs become “full” AWS may remove them from the list of AZs that customers see if those customers don’t already have instances running in them. From AWS’ docs:

As Availability Zones grow over time, our ability to expand them can become constrained. If this happens, we might restrict you from launching an instance in a constrained Availability Zone unless you already have an instance in that Availability Zone. Eventually, we might also remove the constrained Availability Zone from the list of Availability Zones for new customers. Therefore, your account might have a different number of available Availability Zones in a region than another account.

Additionally, the public-facing name of an availability zone is not consistent across AWS accounts. Observed behavior across a number of cloud offerings is that most users provision in the first availability zone most of the time, unless they have specific reason not to. From AWS’ docs:

To ensure that resources are distributed across the Availability Zones for a region, we independently map Availability Zones to identifiers for each account. For example, your Availability Zone us-east-1a might not be the same location as us-east-1a for another account. There’s no way for you to coordinate Availability Zones between accounts.

This is an issue for customers who attempt to peer VPCs between accounts and expect predictable network performance between AZs of the same name. Reports suggest that sufficiently large customers can open support requests to align AZ names between accounts.

How customers work within regions and AZs

AWS’ EC2 docs define the scope of EC2-related resources within global, region, or AZ scopes. However, as a practical matter, the following covers how a typical non-trivial user can interact with those resources.

Users connect to a single EC2 API endpoint to provision instances across all AZs in a single region.

There are some situations where a user selects the AZ by name when creating an instance, and others in which the user selects a VPC subnet (VPC subnets are limited to a single AZ, see below), and the subnet therefor determines which AZ the instance is provisioned in. The web console prevents users from specifying impossible combinations of subnet and AZ when provisioning an instance (actually, it’s smart enough to avoid presenting both approaches in a provisioning flow). The guardrails are lower for API interactions, but conflicting AZ and subnet resources cause an error back to the user.

AWS’ Auto Scaling Groups service makes a best effort to evenly distribute instances among each AZ it is configured for (though this is linked to VPC subnets, see below). Maintaining balance between AZs is the first priority in the default termination policy. (Also note our own project to develop an ASG equivalent, though it only operates within a single Triton DC.)

When a user creates a custom AMI (image), the image is usable in all AZs within the region (they offer tools to copy the image to other regions).

Individual EBS volumes can only be used in the AZ in which they were created, but snapshots of those volumes can be used in any AZ within the region.

More details about how network interactions across AZs within a region are addressed below.

AZ failures

Individual AZs are isolated from each other in many ways, but not so completely that failures in one AZ don’t affect other AZs (though this is rare).

This is a stub and needs expansion.

VPCs

AWS Virtual Private Cloud (VPC) product was first introduced as a limited availability feature to provide separate, isolated network for EC2 customers, but in 2013 the company re-introduced the feature and made it the default for all new customers. The earlier VPC feature was manually configured with VLANs and VRFs, but to scale that model the company created a VXLAN-like solution that predated the formal VXLAN spec.

Background for, and details about the VPC implementation can be found in AWS re:Invent sessions:

• 2015: video, slides, and video about performance optimization
• 2016: video
• 2017: video

At some point AWS started calling non-VPC EC2 networks and usage “EC2-Classic.” The distinction is important because some features are only available in the VPC context, and usage instructions can vary depending on the platform.

It is possible to link EC2-Classic with VPC networks, but that’s not without complication. Instagram’s migration into FB data centers first required migration from EC2-Classic to VPC, a feat that was rare and complicated enough to be worthy of a blog post.

VPC basics

Answering what a VPC is in the FAQ:

Amazon VPC lets you provision a logically isolated section of the Amazon Web Services (AWS) cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including selection of your own IP address ranges, creation of subnets, and configuration of route tables and network gateways. You can also create a hardware Virtual Private Network (VPN) connection between your corporate datacenter and your VPC and leverage the AWS cloud as an extension of your corporate datacenter.

You can easily customize the network configuration for your Amazon VPC. For example, you can create a public-facing subnet for your web servers that have access to the Internet, and place your backend systems such as databases or application servers in a private-facing subnet with no Internet access. You can leverage multiple layers of security, including security groups and network access control lists, to help control access to Amazon EC2 instances in each subnet.

[…]

Amazon VPC enables you to build a virtual network in the AWS cloud - no VPNs, hardware, or physical datacenters required. You can define your own network space, and control how your network and the Amazon EC2 resources inside your network are exposed to the Internet. You can also leverage the enhanced security options in Amazon VPC to provide more granular access to and from the Amazon EC2 instances in your virtual network.

The FAQ also lists VPC components. The basics start with:

• A Virtual Private Cloud: A logically isolated virtual network in the AWS cloud. You define a VPC’s IP address space from ranges you select.
• Subnet: A segment of a VPC’s IP address range where you can place groups of isolated resources.
• Router: Routers interconnect subnets and direct traffic between Internet gateways, virtual private gateways, NAT gateways, and subnets.

Then there are some components that connect VPC networks to the public internet:

• Internet Gateway: The Amazon VPC side of a connection to the public Internet.
• NAT Gateway: A highly available, managed Network Address Translation (NAT) service for your resources in a private subnet to access the Internet.
• Egress-only Internet Gateway: A stateful gateway to provide egress only access for IPv6 traffic from the VPC to the Internet.

A few components offering solutions to connect a VPC to private networks outside AWS:

• Virtual Private Gateway: The Amazon VPC side of a VPN connection.
• Customer Gateway: Your side of a VPN connection.
• Hardware VPN Connection: A hardware-based VPN connection between your Amazon VPC and your datacenter, home network, or co-location facility.

A straightforward component to connect VPCs to each other:

• Peering Connection: A peering connection enables you to route traffic via private IP addresses between two peered VPCs.

And finally a component that allows you to connect your VPC to selected AWS services so traffic never transits public networks (both more secure and less expensive, because you don’t get charged for traffic to S3 over a VPC endpoint in the same region, for example):

• VPC Endpoints: Enables private connectivity to services hosted in AWS, from within your VPC without using an an Internet Gateway, VPN, Network Address Translation (NAT) devices, or firewall proxies.

Elastic IPs and Elastic Network Interfaces appear to be peer objects to VPCs in AWS’ API hierarchy, but they’re limited by VPCs and networking in AWS in general.

VPCs support both IPv4 and IPv6 addressing, though IPv6 is off by default. VPC IPv4 CIDR blocks can be anything from /16 to /28 (actually these two pages conflict on that: one vs. two), and the same is true for individual subnets within a VPC. Subnets within a VPC cannot overlap, but subnets in separate VPC often overlap (use-case: people provision identical VPCs with identical subnets for dev, test, staging, prod, etc.). There are yet more rules about CIDR blocks within and among VPCs, especially those that apply if VPCs are peered, linked via VPNs, or have other links to other networks.

Importantly, VPC subnets can include non-RFC1918 IPs. Consider this note in the docs:

You can create a VPC with a publicly routable CIDR block that falls outside of the private IPv4 address ranges specified in RFC 1918; however, for the purposes of this documentation, we refer to private IP addresses as the IPv4 addresses that are within the CIDR range of your VPC.

A not uncommon scenario is for people who own their own IP space to assign that to VPCs for use in AWS. We also know [redacted customer name and AWS user] assigns public IPs they don’t own to VPCs and interacts with them over a VPN connection.

AWS limits each account to:

• 5 VPCs per region
• 200 subnets per VPC
• 1 Internet Gateway per VPC
• 5 Virtual Private Gateways per region
• 50 Customer Gateways per region
• 10 IPSec VPN connections per Virtual Private Gateway

Though some of the above limits can be increased by opening a ticket with AWS support.

VPCs vs. AZs

VPCs exist at the regional scope across all AZs in a region; however, subnets are specific to a single AZ. From the FAQ:

When you create a subnet you must specify the Availability Zone in which to place the subnet. When using the VPC Wizard, you can select the subnet’s Availability Zone in the wizard confirmation screen. When using the API or the CLI you can specify the Availability Zone for the subnet as you create the subnet. If you don’t specify an Availability Zone, the default “No Preference” option will be selected and the subnet will be created in an available Availability Zone in the region.

Interestingly, the FAQ claims charges are assessed for traffic between AZs, but the VPC pricing page says nothing about such charges.

AWS creates default VPCs for every user in every region. Users can delete the default or create new VPCs as they desire (up to the limits).

Peering VPCs

AWS allows the peering of VPCs in a single region, or across regions, between separate accounts, and other scenarios. There are also some invalid peering scenarios. Some key things that don’t work:

If I peer VPC A to VPC B and I peer VPC B to VPC C, does that mean VPCs A and C are peered?

No. Transitive peering relationships are not supported.

Can I use AWS Direct Connect or hardware VPN connections to access VPCs I’m peered with?

No. “Edge to Edge routing” isn’t supported in Amazon VPC.

Bandwidth between peered VPCs is claimed to be the same as bandwidth between instances in the same VPC. In fact, AWS claims that availability and performance have no single point of failure or constraint:

AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor a VPN connection, and does not rely on a separate piece of physical hardware. There is no single point of failure for communication or a bandwidth bottleneck.

Linking VPCs to external networks

AWS distinguishes between peering between VPCs and peering a VPC to external networks. While they’ve abstracted away most of the details of joining two or more VPCs, wiring up non-VPC networks requires many more steps. AWS provides Customer Gateways that can be used to solve a number of network interconnection challenges, including peering an office network with a VPC. Various combinations of peering and customer gateways can be used to create sophisticated topologies.

Routing in VPCs

AWS’ docs on routing specify:

• Your VPC has an implicit router.
• Your VPC automatically comes with a main route table that you can modify.
• You can create additional custom route tables for your VPC.
• Each subnet must be associated with a route table, which controls the routing for the subnet. If you don’t explicitly associate a subnet with a particular route table, the subnet is implicitly associated with the main route table.
• You cannot delete the main route table, but you can replace the main route table with a custom table that you’ve created (so that this table is the default table each new subnet is associated with).
• Each route in a table specifies a destination CIDR and a target (for example, traffic destined for the external corporate network 172.16.0.0/12 is targeted for the virtual private gateway). We use the most specific route that matches the traffic to determine how to route the traffic.
  • CIDR blocks for IPv4 and IPv6 are treated separately. For example, a route with a destination CIDR of 0.0.0.0/0 (all IPv4 addresses) does not automatically include all IPv6 addresses. You must create a route with a destination CIDR of ::/0 for all IPv6 addresses.
• Every route table contains a local route for communication within the VPC over IPv4. If your VPC has more than one IPv4 CIDR block, your route tables contain a local route for each IPv4 CIDR block. If you’ve associated an IPv6 CIDR block with your VPC, your route tables contain a local route for the IPv6 CIDR block. You cannot modify or delete these routes.
• When you add an Internet gateway, an egress-only Internet gateway, a virtual private gateway, a NAT device, a peering connection, or a VPC endpoint in your VPC, you must update the route table for any subnet that uses these gateways or connections.
• There is a limit on the number of route tables you can create per VPC, and the number of routes you can add per route table. For more information, see Amazon VPC Limits.

AWS has this to say about routing when peering VPCs:

A VPC peering connection is a networking connection between two VPCs that allows you to route traffic between them using private IPv4 addresses. Instances in either VPC can communicate with each other as if they are part of the same network.

To enable the routing of traffic between VPCs in a VPC peering connection, you must add a route to one or more of your VPC route tables that points to the VPC peering connection to access all or part of the CIDR block of the other VPC in the peering connection. Similarly, the owner of the other VPC must add a route to their VPC route table to route traffic back to your VPC.

Securing instances, subnets, and VPCs

AWS offers a mix of instance firewalls (called Security Groups), network firewalls (Network ACLs), user controls (IAM), and logging (VPC Flow Logs) to secure VPC resources.

Security Groups

Security Groups are AWS’ approach to firewalling. They automatically create a default Security Group for each VPC, though users typically create custom Security Groups. A number of Security Group examples are offered.

Security groups have many similarities to Triton CloudFirewall. Differences:

• CloudFirewall rules are loosely associated with individual instances, but Security Groups are directly assigned to individual instances
• CloudFirewall supports rules based on tags, Security Groups don’t
• Security Groups support rules based on the source VPC, CloudFirewall doesn’t
• Security Groups appear to support any IP protocol number, CloudFirewall only supports TCP, UDP, and ICMP

Network ACLs

In addition to Security Groups, AWS offers Network ACLs to secure subnets or entire VPCs. Unlike Security Groups, Network ACLs are stateless: “responses to allowed inbound traffic are subject to the rules for outbound traffic (and vice versa),” though the full comparison highlights more details.

IAM

Beyond firewalls, IAM is used to protect VPC configuration and control who can launch/connect instances to a VPC.

Logging and auditing

Finally, VPC Flow Logs, which can be applied to an entire VPC, a single subnet, or individual network interface, are used to audit traffic in and out. Example uses include both application debugging and alarming on suspicious network activity.

Elastic IPs and Elastic Network Interfaces

In cloud services, the IP address of an instance is typically assigned by the cloud provider. The IP is typically tied to the instance lifecycle and lasts just as long as the instance does. However, failure recovery and some upgrade strategies can require that instances be replaced, typically with a new IP. This can create complexity when wiring application components together, and those problems are magnified when adding or removing instances to scale the application.

Various discovery solutions—active discovery with a service like Consul, DNS, or just load balancing-as-a-service—are all commonly used approaches to wiring together application components even as IPs are changing. However, some applications and use-cases are better served by having predictable, consistent IPs, even as the instances answering to those IPs are changed or replaced.

In cases where a persistent IP is desired, AWS offers two solutions:

1. Elastic IPs (EIPs) are publicly routable IPv4 addresses that can be used in any AZ within a region. EIPs are provisioned or deleted as separate objects from EC2 instances, but they can be assigned and re-assigned to EC2 instances as needed.

2. Elastic Network Interfaces are a “virtual network card” in a VPC that have a one or more private IPv4 addresses within the VPC, optional IPv6 addresses, Elastic IP addresses (no more than one per private IP), one or more security groups, and a single MAC address.

The differences between EIPs and ENIs are subtle and confusing for many, and both are often misused. In fact, most people recommend using ELBs or ALBs for most EIP use-cases.

Capabilities and limitations of EIPs and ENIs

A single Elastic IP can be associated with just one instance at a time (though a single instance can have multiple Elastic IPs associated). There is no support for IP spoofing as may be used for active-active topologies.

Moving an Elastic IP from one instance or ENI to another can take some time. “The remap process currently takes several minutes from when you instruct us to remap the Elastic IP until it fully propagates through our system.”

AWS allows EIP customers to configure reverse DNS for Elastic IP addresses with a support request, even if it’s for the purpose of sending email.

A single Elastic Network Interface can be associated with just one instance at a time (though a single instance can have multiple ENIs). There is no support for MAC spoofing as may be used for active-active topologies.

ENIs are scoped to a single AZ (actually to a specific subnet in the AZ) and can only be used with instances in the same AZ. They cannot be used in or moved to other AZs.

It is possible to attach or detach ENIs on running instances, however: “A warm or hot attach of an additional network interface may require you to manually bring up the second interface, configure the private IPv4 address, and modify the route table accordingly. Instances running Amazon Linux or Windows Server automatically recognize the warm or hot attach and configure themselves.”

Adding ENIs does not increase network throughput of the instance.

Moving an EIP or ENI from one instance to another takes a non-zero amount of time. Connections are reset, traffic targeted at the EIP or ENI during a transition will go unanswered.

And there are yet more rules for instance IPs

AWS explains that an “Elastic IP address is tied to a region and can be associated only with an instance in the same region.” This is technically true for public Elastic IPs, but it gets more complex for private IPs, as explained in AWS’ docs for migrating an instance to a new AZ:

If you need to preserve the private IPv4 address of the instance, you must delete the subnet in the current Availability Zone and then create a subnet in the new Availability Zone with the same IPv4 address range as the original subnet. Note that you must terminate all instances in a subnet before you can delete it. Therefore, you should create AMIs from all the instances in your subnet so that you can move all instances in the current subnet to the new subnet.

Network performance

NICs

AWS has made a number of efforts at improving network performance through its history.

Early on, AWS offered users a choice of full hardware virtualization (HVM) and paravirtualization (PV), though that has been almost completely phased out now and all new instance types since 2014 are HVM-only.

In 2014 AWS introduced Enhanced Networking based on the Intel 82599 Virtual Function interface, supporting up tp 10GBps. Then, in 2017, AWS introduced their Nitro-based hardware and the Elastic Network Adapter (ENA, not to be confused with ENI) that took network virtual function interfaces even further.

On new instance types (c5.* and i3.* as example), the ENA is presented as a PCI device. However, this is not the same as passing through the raw NIC. The Nitro hardware presents a virtualized PCI device with all the necessary security controls (anti-spoofing, etc.) for operation with un-trusted users. It appears the Nitro ENA can be configured out of band, independently from the host hardware, as these are used in AWS’ new bare metal platform.

AWS has had to develop drivers to support the ENA. Linux and FreeBSD drivers are in GitHub. Various Linux vendors have added them to their distros. The Windows driver does not appear to be open sourced.

MTU

Interestingly, the default MTU for VPC networks is 1500. Customers can change the MTU up to 9001.

Resources by scope

Within AWS, this table shows the resources by scope.

Connectivity by scope

• Within an AZ
  • Full connectivity between instances in the same AZ, regardless of which VPC or subnet they’re in if:
    • The security groups on both instances allow connectivity, and
    • If they’re in the same subnet, or
    • Routing policies between those subnets allow it, and
    • Network ACLs for those subnets allow connectivity, and
    • They’re in separate subnets in the same VPC, or
    • They’re in separate VPCs that are peered (in the same account or separate accounts)
• Within a region (between AZs in the same region)
  • Full, transparent connectivity between instances with the same rules as for within an AZ
    • …But expect to pay fees to transit data between AZs in a single region
• Within the cloud (between regions in the same cloud)
  • Full connectivity between instances in separate regions if inter-region peering is enabled, however:
    • “you cannot reference security groups across Inter-Region VPC Peering”
    • “jumbo frames cannot be sent between regions”
      • …And expect to pay fees to transit data between regions
• Globally (between resources in separate clouds)
  • AWS provides Customer Gateways that can be used to solve a number of network interconnection challenges, including peering an office network with a VPC. * …And expect to pay fees to transit data between clouds
  • AWS operates multiple independent “clouds,” including GovCloud, and two isolated clouds in China.