IPv6 adventures part 2: more roadblocks

In a previous post I wrote about some things I discovered while implementing IPv6 in my homelab. While testing at home, I became comfortable enough to go for it and implement IPv6 in our AWS production stack. I made a thorough releaseplan, at least I thought, tested a lot in a sandbox, and went for it. Long story short, that was not a fun evening.

Well at least I learned some more. For starters I found some additional services and sites that don't support IPv6. And I now know that I probably need a NAT64 in my networks. More on that in a future post though. For now a quick list of the services that didn't work.

AWS

Eventbridge

I wrote a script to disable cronjobs running from Amazon EventBridge Scheduler. It doesn't support IPv6 or dual-stack, so running the script on my PC (dual stack) didn't work. It only works when setting use_dualstack_endpoint = false in the .aws/config file.

SES

Sending emails using SES is not (yet) possible over IPv6: https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-support.html

S3

I also run a script daily to backup databases and data to S3. The S3 endpoints are dual-stack ready, but you have to explicitly tell the aws-cli to use the dual-stack version. Locally I did that by adding use_dualstack_endpoint = false in the .aws/config file, but that isn't convenient when using a Docker container for the aws-cli. The solution is to pass an environment variable, --env AWS_USE_DUALSTACK_ENDPOINT=true, to the container.

Also Docker needs to know it should create an IPv6 ready network...

Docker

Network

So far I thought configuring Docker for IPv6 was as easy as adding enable_ipv6: true to the network declaration in a compose file:

networks:
  proxy-net:
    name: proxy-net
    enable_ipv6: true
    external: true

Now the network has to be restarted. This is done by stopping all containers connected to that network, deleting the network and than restarting all containers:

• docker compose down
• docker network prune -a
• docker compose up -d

However, when you run a container directly from the command line, without specifying a network, it will connect to the default network. This network is IPv4 only, unless you enable IPv6. This is done in /etc/docker/daemon.json:

{
    "ipv6": true,
    "fixed-cidr-v6": "2001:db8:1::/64"
}

After that, restart docker: sudo systemctl restart docker.

Github Container Repository (GHCR)

If you need to pull an image from ghcr.io, it only works with IPv4.

Mollie

We us Mollie as our payments provider. Their docs are IPv6 ready, however, their API isn't.

joep@PCJOEP:~$ ping -c 1 docs.mollie.com
PING docs.mollie.com(2606:4700::6810:f276 (2606:4700::6810:f276)) 56 data bytes
64 bytes from 2606:4700::6810:f276 (2606:4700::6810:f276): icmp_seq=1 ttl=59 time=4.06 ms

--- docs.mollie.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.060/4.060/4.060/0.000 ms

joep@PCJOEP:~$ ping -c 1 api.mollie.com
PING api.mollie.com (34.128.165.251) 56(84) bytes of data.
64 bytes from 251.165.128.34.bc.googleusercontent.com (34.128.165.251): icmp_seq=1 ttl=119 time=4.33 ms

--- api.mollie.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.331/4.331/4.331/0.000 ms