IPv6 adventures 4: AWS

Known problems

In my homelab I basically got IPv6 working on all my (virtual) machines. Now I also wanted to implement it on production in the AWS cloud. AWS, as big as they are, or maybe because they are big, haven't implemented IPv6 on all of their services. This makes it hard to do a full transition. For me I got to work around it, but for bigger implementations it will be a headache. See the links below for more info.

• https://github.com/DuckbillGroup/aws-ipv6-gaps?tab=readme-ov-file
• https://tty.neveragain.de/2024/05/20/aws-ipv6-egress.html

Architecture

AWS recommends to create two distinct subnets: one public and one private. In the public subnet I created a single EC2 instance. This will be my NAT gateway and SSH bastion. Machine in the private subnet won't be accessible from the internet. They can reach out to the internet, though, through an Egress-only gateway.

• https://docs.aws.amazon.com/vpc/latest/userguide/vpc-example-web-database-servers.html
• https://docs.aws.amazon.com/vpc/latest/userguide/vpc-example-private-subnets-nat.html
• https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html

For this blog, let's say we have two websites. One running on a webserver in Linux. Another on a webserver in Windows.

NAT(64) gateway

Egress-only gateways sound perfect, but they have a limitation; they only work with traffic over IPv6. So for IPv4, we need another solution. I also want this NAT to do 6-4 translation (NAT64), so my machines don't need a public IPv4 address.

The easiest way is to use an AWS NAT gateway, they do everything I need, but they are pricy. So I opted to build my own solution, which can do the same for ~10% of the costs. And luckily this is not that hard. There is even a prebuild EC2-image (fck-nat) if you don't want to do this yourself. Currently they don't support NAT64 yet, but this is being developed.

I opted to create a Debian machine, on a t3.nano (I used a t3.micro during installation for the additional RAM). I configured SSH with 2FA and fail2ban. Installed Jool and configured the firewall, see commands below (more info see sources, right below comments).

nat_interface=$(ip link show dev "ens5" | head -n 1 | awk '{print $2}' | sed s/://g)
iptables -t nat -A POSTROUTING -o "$nat_interface" -j MASQUERADE -m comment --comment "NAT routing rule installed by fck-nat"

• https://medium.com/@rakeshkanagaraj1990/aws-nat-instance-bb0911ba19d5
• https://github.com/AndrewGuenther/fck-nat/blob/main/service/fck-nat.sh
• https://linuxconfig.org/how-to-make-iptables-rules-persistent-after-reboot-on-linux

In the AWS EC2 dashboard, select the EC2. Go to Actions – Networking – Change source/destination check – Select "Stop" – Save.

SSH tunnels/proxy

My production machines are now in an "isolated" private subnet. So how can I reach them to manage them. The safest way, I guess?, would be through the AWS EC2 dashboard, or with AWS tools like Session Manager. I don't like that, it feels to complicated.

I opted to use my NAT instance as a SSH bastion as well. As this instance is in the public subnet, it is reachable. Yes, this makes this machine a weak spot, but then again, it is the only one I need to manage.

To make life easier, I locally created fqdn's for each EC2 server and added them in my local DNS (Technitium). On the bastion I added them to the hosts file, so it can resolve them and forward requests. See examples below.

.ssh/config on my local machine:

Host server-01
    HostName 01.joeplaa.com
    User admin
    IdentityFile ~/.ssh/server-01

Host server-02
    HostName server-02.joeplaa.com
    User ubuntu
    IdentityFile ~/.ssh/server-02
    ProxyJump admin@server-01

Host server-03
    HostName server-03.joeplaa.com
    User ec2
    IdentityFile ~/.ssh/server-03
    ProxyJump admin@server-01

/etc/hosts on the bastion (also entered in Technitium DNS):

2001:db8::2    server-02.joeplaa.com
2001:db8::3    server-03.joeplaa.com
2001:db8::4    server-04.joeplaa.com

Connect RDP or database

As our webservers are in a private subnet, a remote desktop session or a connection to a database server is also not directly possible. We have to create a tunnel from our local machine, through the bastion into the server.

We already have an SSH connection, so we can start there. First add the keys:

ssh-add server-02
ssh-add server-03

Then we can create a tunnel to the RDP port (you can use any local port as long as it's free):

ssh server-03 -L 53389:localhost:3389

Now an RDP session can be opened to 127.0.0.1:53389.

A connection to a MySQL/MariaDB instance:

ssh server-0 -L 53306:localhost:3306

Similarly for SQL server:

ssh server-03 -L 51433:localhost:1433

AWS VPC

I have been mentioning public and private subnets and egress gateways already. Here's what you need to configure in AWS VPC.

Go to Your VPC's. There should already be one. Give it a more useful name like default-vpc.

CIDRs

We need multiple CIDRs:

1. IPv4 CIDR for private subnet
2. IPv4 CIDR for public subnet
3. IPv6 CIDR for private subnet
4. IPv6 CIDR for public subnet

First we need to create an IPv6 pool. Go find the "Amazon VPC IP Address Manager".

• Go to "IPAMs" and press "Create IPAM"
  • Select "Free Tier"
  • As description enter something like "joeplaa IPAM production"
  • Select the region
  • "Private IPv6 GUA CIDRs" can be left disabled
• Go to "Scopes" and press "Create scope"
  • Scope type: Public
• Now go to "Pools", the created IPAM should be selected and press "Create pool"
  • Give it a name ipam-pool-joeplaa-01
  • As description enter something like "GUA pool production"
  • Source: "IPAM scope"
  • Address family: "IPv6"
  • Locale: your region
  • Service: 'EC2"
  • Public IP source: "Amazon-owned"
  • CIDRs to provision – Netmask: "/52"
• Next go to VPC – Your VPCs. There you should see default-vpc. Select it and go to Actions – Edit CIDRs
  • Under IPv4 there should already be one, probably: 172.31.0.0/16
  • Add another one: 172.16.0.0/16
  • Under IPv6 add a new CIDR:
    • IPv6 CIDR block: "IPAM-allocated IPv6 CIDR block"
    • IPv6 IPAM pool: select the one created earlier ipam-pool-joeplaa-01
    • CIDR block: "Netmask length" /60

Subnets

We need to create at least two subnets. You can also "upgrade" to 2 subnets per availability zone, in the case of eu-central-1 this means 6 in total.

• Select a public subnet and go to Actions – Edit subnet settings
  • Under "Auto-assign IP settings" select both boxes:
    • Enable auto-assign IPv6 address
    • Enable auto-assign public IPv4 address
  • Under Resource-based name settings select both boxes:
    • Enable resource name DNS A record on launch
    • Enable resource name DNS AAAA record on launch
  • Under DNS64 settings select:
    • Disable DNS64
• Repeat for all public subnets
• Select a private subnet and go to Actions – Edit subnet settings
  • Under "Auto-assign IP settings" select only box:
    • Enable auto-assign IPv6 address
  • Under Resource-based name settings select both boxes:
    • Enable resource name DNS A record on launch
    • Enable resource name DNS AAAA record on launch
  • Under DNS64 settings select:
    • Enable DNS64

Internet gateway

• Go to Internet gateways. There should already be one.
• Give it a name like igw-01.

Egress-only internet gateway

• Go to Egress-only internet gateways – Create egress only internet gateway.
• Give it a name like eigw-01.

Route tables

We need two tables here. A table for the public subnets and one for the private subnets.

• First rename the existing table to rtb-private.
• Go to Actions – Edit subnet associations
  • Select only the private subnets
  • Click "Save associations
• Click "Create route table"
  • Name: rtb-public
  • VPC: default-vpc
• Go to Actions – Edit subnet associations
  • Select only the public subnets
  • Click "Save associations
• Select the private table and go to Actions – Edit routes:
  • You should see your three CIDRs in the Destination column routing to Target: "local". This is all local traffic within the VPC.
  • Add Destination: 0.0.0.0/0, Target: "Instance", select your NAT instance. This is the route for IPv4 traffic to external IPv4 targets; using NAT.
  • Add Destination: 64:ff9b::/96, Target: "Instance", select your NAT instance. This is the route for IPv6 traffic to external IPv4 targets; NAT64.
  • Add Destination: ::/0, Target: "Egress Only internet gateway", select eigw-01. This is the route for IPv6 traffic to external IPv6 targets.
• Select the public table and go to Actions – Edit routes:
  • You should see your three CIDRs in the Destination column routing to Target: "local". This is all local traffic within the VPC.
  • Add Destination: 0.0.0.0/0, Target :"Internet Gateway", select igw-01. This is the route for IPv4 traffic to external IPv4 targets.
  • Add Destination: 64:ff9b::/96, Target: "Instance", select your NAT instance. This is the route for IPv6 traffic to external IPv4 targets; NAT64.
  • Add Destination: ::/0, Target: "Internet gateway", select igw-01. This is the route for IPv6 traffic to external IPv6 targets.

Endpoints/gateways

This one is optional and only useful if you expect a lot of traffic to S3. In the current setup all traffic to S3 has to go through the NAT instance as this traffic is IPv4. We don't want to use the instance if not absolutely necessary as NAT64 is CPU intensive and we have to pay for data traffic.

In VPC you can create "Endpoints" for services in the AWS cloud. Most of them are expensive and only economically viable if you have large amounts of traffic to that service. One exception is an S3 endpoint which is free. So why not use it?

• Go to Endpoints – Create endpoint
  • Name: s3-eu-central-1
  • Type: "AWS services"
  • Services com.amazonaws.eu-central-1.s3 (search for s3) select the gateway
• This should also create entries in your route tables.
  • Destination: pl-6ea54007, Target: the VPC endpoint ID created above

Network ACLs

To protect our VPC, create two Network ACLs: private and public.

• Select private, go to Actions – Edit subnet associations
  • Select only the private subnets
  • Click "Save associations
• Select public, go to Actions – Edit subnet associations
  • Select only the public subnets
  • Click "Save associations

Public inbound

• 10x: Allow internal traffic between subnets
• 11x: Allow SSH from your home/office and deny from anywhere else
• 12x: Allow RDP from your home/office and deny from anywhere else
• 80x: Allow return traffic (network acl's are stateless)
• 90x: Allow Ping from your home/office and allow echo replies (network acl's are stateless)
• *: Deny everthing else (default Deny)

Public outbound

• 10x: Allow SSH to other local machines
• 11x: Allow HTTP to everywhere
• 12x: Allow HTTPS to everywhere
• 13x: Allow SMTP / SMTPS / TLS to everywhere
• 80x: Allow return traffic (network acl's are stateless)
• 90x: Allow Ping to everywhere
• *: Deny everthing else (default Deny)

Private inbound

• 11x: Allow SSH from the bastion (or other local machines) and deny from anywhere else
• 12x: Deny RDP from anywhere
• 80x: Allow return traffic (network acl's are stateless)
• 90x: Allow Ping from the bastion (or other local machines) and allow echo replies (network acl's are stateless)
• *: Deny everthing else (default Deny)

Private outbound

• 11x: Allow HTTP to everywhere
• 12x: Allow HTTPS to everywhere
• 13x: Allow SMTP / SMTPS / TLS to everywhere
• 80x: Allow return traffic (network acl's are stateless)
• 90x: Allow Ping to everywhere
• *: Deny everthing else (default Deny)

Security groups

To protect our individual machines, we create security groups. Before we start we'll make our lives easier by first creating "Managed prefix lists".

1. Prefix list name: vpc-subnets-ipv4, "Prefix list entries": 172.16.0.0/16 and 172.31.0.0/16.
2. Prefix list name: vpc-subnets-ipv6, "Prefix list entries": 2xxx::/60.
3. Prefix list name: office-ipv4, "Prefix list entries": your IPv4 address.
4. Prefix list name: office-ipv6, "Prefix list entries": your IPv6 address(es)/prefix(es).

sg-linux-server-management for each linux machine

Inbound

Outbound

sg-windows-server-management for each windows machine

Inbound

Outbound

sg-bastion for the SSH bastion

Inbound

Outbound

sg-nat64-gateway for the NAT gateway instance

Inbound

Outbound

sg-cloudfront-website as an example for a website accessed through a CloudFront VPC origin (see further down)

Inbound

Outbound

AWS CloudFront

In CloudFront we are going to create a VPC endpoint. This allows CloudFront to connect to a webserver in our private subnet.

Go to CloudFront – VPC origins – Create VPC origin

• Name: server-02-80
• Origin ARN: the arn of the EC2 instance running the webserver
• Protocol: HTTP only
  • HTTP port: 80

This will take some time. After a few minutes, you should see a new security group pop up: CloudFront-VPCOrigins-Service-SG.

AWS EC2

Back to the EC2 instances. Let's say we have three:

1. The public NAT instance / ssh-bastion
2. A private Linux machine with a webserver
3. A private Windows machine with a webserver

server-01

• This will be the NAT instance. Create it in a public subnet. A public IPv4 address should automatically be assigned. If not, do it manually.
• Go to Actions – Networking – Change source/destination check – Select "Stop" – Save.
• Go to Actions – Security – Change security groups – Select sg-nat64-gateway, sg-bastion and sg-linux-server-management – Save.

server-02

• This will be the Linux webserver instance. Create it in a private subnet. No public IPv4 address should be assigned.
• Go to Actions – Security – Change security groups – Select sg-cloudfront-website and sg-linux-server-management – Save.

server-03

• This will be the Windows webserver instance. Create it in a private subnet. No public IPv4 address should be assigned.
• Go to Actions – Security – Change security groups – Select sg-cloudfront-website and sg-windows-server-management – Save.

Result

We should now have a setup with the following features:

• Bastion
  • The Bastion is the only server we can reach directly from our office (ping and ssh only).
    • Login into the Bastion is only possible from our office.
    • Login into the Bastion is only possible with 2FA.
  • SSH-sessions to private instances can only be created through the Bastion (only from our office and only with 2FA).
  • RDP sessions to private instances can only be created by setting up a tunnel through the Bastion (only from our office and only with 2FA).
  • Databases on private instances can only be reached by setting up a tunnel through the Bastion (only from our office and only with 2FA).
  • Our webservers on private instances can only be reached by setting up a tunnel through the Bastion (only from our office and only with 2FA).
    • This of course is silly. So we created a CloudFront endpoint. Visitors can now open our websites without us exposing the servers.
• Internet Access
  • No instance in the private subnet gets a public IPv4 address. This saves money and cuts outside access. Try by pinging the IPv6 address.
  • Each instance can however connect to the internet, either through their own GUA IPv6 address directly, or via the NAT instance if IPv4 is needed. Try by ping your office from an instance.
  • Each instance can reach a IPv4 only website through the NAT64 on the NAT instance. Try by pinging github.com or api.mollie.com.
  • Files can be stored and retrieved in S3 through the S3 endpoint. Harder to verify, you could somehow watch traffic through the NAT.

Non-IPv6 services

I did run into a few particular issues for my own setup.

ECR

For example, ECR is IPv4 only. I could have created two endpoints in my VPC, one to com.amazonaws.eu-central-1.ecr.api and one to com.amazonaws.eu-central-1.ecr.dkr. This did indeed work, but would have cost me $17.28 per month ($0.012 * 24 * 30 * 2).

There are feature requests to implement IPv6 into container related services:

• https://github.com/aws/containers-roadmap/issues/2287
• https://github.com/aws/containers-roadmap/issues/1340

For now I use my own container hosting solution as I don't need to download container images regularly: Nexus OSS.

SES

SES wasn't supported when I was working on it. IPv6 support seems to have been added recently though.

• https://repost.aws/questions/QUDABNdfE3RLCm5QNFYG4A9A/does-aws-ses-support-mail-sending-from-ipv6-instances