AWS Gateway Load Balancer

How to inspect & manipulate network packets in AWS

 Posted on 2023-02-18  |  7 minutes  |  Michał Szczygieł

In this post, I would like to walk you through a sample setup of the AWS Gateway Load Balancer. We will provision the infrastructure using Terraform, write a simple virtual appliance application and show it all in action. I don’t want to go into a theory about this AWS service. It’s very well described in the documentation. In short, it allows us to route network traffic through a virtual appliance where each network packet can be inspected, modified, or dropped.

Application

All the code can be found in my aws-gateway-lb GitHub repository.

Note, that running the example will incur some costs. Remember, to destroy the infrastructure after playing around with it!

Infrastructure

Sample infrastructure can be provisioned using Terraform by following the readme in the code repository. The readme also describes the details about what’s provisioned.

Virtual appliance

A virtual appliance is an application that supports Geneve (Generic Network Virtualization Encapsulation) protocol and exposes a health check endpoint. It’s possible to get one from the AWS marketplace, but for the purpose of this post, we will write our own. In short, the appliance has to:

  • receive a UDP packet,
  • swap source and destination IP in the outermost IP layer,
  • optionally modify the packet’s contents and update the checksum,
  • send the packet back or drop the packet.

In this section, I will explain the steps necessary to create a virtual appliance. Full source code (along with instructions on how to run it) is accessible in the GitHub repository mentioned above.

The appliance will handle all Geneve packets, decode them using the gopacket library, and process them according to the following rules.

UDP packets where the source or destination port is 3000 will be handled as follows

  • if the payload contains the string “drop me” the packet will be dropped;
  • string “weakly typed” in the payload will be replaced with string “strongly typed”.

Additionally, every 5th ICMP packet will be dropped.

The reason I’ve chosen ICMP and UDP is that I wanted to show how to handle various protocols and to emphasize that we’re not limited to TCP or UDP only. I’ve chosen UDP over TCP as it’s easier to show dropping packets. By design, if we drop a single TCP packet, we won’t be able to process any subsequent ones. Therefore dropping a subset of packets is much easier to show with UDP.

For brevity, we will support IPv4 only and skip error handling (although the application in the repository handles errors).

Capturing packets

Packets that the virtual appliance has to handle will be encapsulated using Geneve and transferred over UDP. This means, that each received packet will begin with the following layers:

  • outer IP header,
  • outer UDP header,
  • Geneve header.

After these 3 layers, the encapsulated packet’s layers will follow.

The virtual appliance has to swap source and destination IP addresses in the outer IP header and update the checksum. To properly implement this, we need to capture raw UDP packets so we get access to all the layers mentioned above.

To create a socket from which we can read raw packets we would call unix.Socket as follows.

fd, err := unix.Socket(unix.AF_INET, unix.SOCK_RAW, unix.IPPROTO_UDP)

We have to preserve other parts (except the checksum) of the outer layers. By default, when sending a packet the IP header would be generated for us. Since we will provide the outer IP header ourselves we have to set the IP_HDRINCL socket option.

err = unix.SetsockoptInt(fd, unix.IPPROTO_IP, unix.IP_HDRINCL, 1)

We also have to update the outer IP header’s checksum. Fortunately, it’s handled automatically.

Decoding packets

After our socket is created, we can start receiving packets from it.

buffer := make([]byte, 8500)
length, raddr, err := unix.Recvfrom(fd, buffer, 0)

We can then decode the packet using the gopacket library.

p := gopacket.NewPacket(buffer[:length], layers.LayerTypeIPv4, gopacket.Default)
packetLayers := p.Layers()

Finally, we get access to all of the packet’s layers.

// access the outer IP layer
packetLayers[0].(*layers.IPv4)

// access the outer UDP layer
packetLayers[1].(*layers.UDP)

Dropping packets

Dropping packets is super simple. We just don’t send anything back and stop processing the packet.

Modifying packets

Modifying packets requires a little bit more work, as we have to access inner layers. And, if a packet was modified the checksum has to be recalculated.

We will attempt to modify UDP packets only. This means, that packets interesting to us will contain the following layers:

  • outer IP,
  • outer UDP,
  • Geneve,
  • inner IP,
  • inner UDP,
  • payload.

Since the checksums of both inner UDP and IPv4 layers depend on the payload, we can’t just modify the payload. We have to also recalculate the checksums. UDP checksum depends on the IPv4 header, therefore we have to explicitly set the correct IP layer in the UDP layer to the one that will be used later for the checksum calculation.

type PayloadModifyFun func([]byte) []byte

func (p *Packet) ModifyUDP(f PayloadModifyFun) {
	// get the inner layers
	ip := p.packetLayers[3].(*layers.IPv4)
	udp := p.packetLayers[4].(*layers.UDP)
	payload := p.packetLayers[5].(*gopacket.Payload)
	p.modified = true
  
	// udp checksum depends on IPv4 layer. Therefore, we need to provide a layer that will be used for checksum calculation.
	udp.SetNetworkLayerForChecksum(ip)

  	// update the payload
  	p.packetLayers[5] = gopacket.Payload(f(payload.Payload()))
}

Serializing packets and sending them back

Before we send the packet back, we have to swap the source and destination IP in the outer IP layer. This is quite simple.

func (p *Packet) SwapSrcDstIpv4() {
	ip, _ := p.packetLayers[0].(*layers.IPv4)
	dst := ip.DstIP
	ip.DstIP = ip.SrcIP
	ip.SrcIP = dst
}

After IP addresses have been swapped, we are ready to serialize all the layers (in reverse order). In cases where the payload has been modified, we have to additionally recompute checksums as mentioned above.

func (p *Packet) Serialize() []byte {
	buf := gopacket.NewSerializeBuffer()
	for i := len(p.packetLayers) - 1; i >= 0; i-- {
		if layer, ok := p.packetLayers[i].(gopacket.SerializableLayer); ok {
			var opts gopacket.SerializeOptions

			// recompute checksum of inner IP and UDP layers in case the packet was modified
			if p.modified && (i == p.insideUDPLayerIdx() || i == p.insideIPLayerIdx()) {
				opts = gopacket.SerializeOptions{ComputeChecksums: true, FixLengths: true}
			} else {
				opts = gopacket.SerializeOptions{FixLengths: true}
			}

			layer.SerializeTo(buf, opts)
			buf.PushLayer(layer.LayerType())
		} else if layer, ok := p.packetLayers[i].(*layers.Geneve); ok {
			bytes, _ := buf.PrependBytes(len(layer.Contents))
			copy(bytes, layer.Contents)
		} else {
			return nil
		}
	}
  return buf.Bytes()
}

Finally, we can send the packet back.

unix.Sendto(fd, response, 0, raddr)

Demo

At the very beginning, we have to provision the infrastructure.

./deploy_infra.sh
./deploy_censor.sh

After all the required resources have been created, in a new terminal, we need to install the required packages on the provisioned instances.

./init_infra.sh

Next, in two separate terminals, we can connect to instances a and b. On instance a we start to listen on UDP port 3000 and on instance b we connect to instance a (note, that in your case private IP addresses will be different so please adjust the commands below).

./ssh.sh a

[ec2-user@ip-192-168-1-209 ~]$ nc -l -u 3000

./ssh.sh b

[ec2-user@ip-192-168-2-106 ~]$ nc -u 192.168.1.209 3000
test
drop me
weakly typed programming language

In instance a we would receive the following.

test
strongly typed programming language

We expect messages containing “drop me” won’t be delivered and “weakly typed” in the message will be replaced with “strongly typed”. The above example confirms that everything works as expected.

By pinging instance a from instance b we notice that as expected, every 5th ICMP packet is dropped.

[ec2-user@ip-192-168-2-106 ~]$ ping 192.168.1.209
PING 192.168.1.209 (192.168.1.209) 56(84) bytes of data.
64 bytes from 192.168.1.209: icmp_seq=1 ttl=253 time=3.96 ms
64 bytes from 192.168.1.209: icmp_seq=2 ttl=253 time=1.58 ms
64 bytes from 192.168.1.209: icmp_seq=3 ttl=253 time=1.51 ms
64 bytes from 192.168.1.209: icmp_seq=4 ttl=253 time=1.51 ms
64 bytes from 192.168.1.209: icmp_seq=6 ttl=253 time=1.64 ms
64 bytes from 192.168.1.209: icmp_seq=7 ttl=253 time=2.07 ms
64 bytes from 192.168.1.209: icmp_seq=8 ttl=253 time=1.87 ms
64 bytes from 192.168.1.209: icmp_seq=9 ttl=253 time=3.36 ms
64 bytes from 192.168.1.209: icmp_seq=11 ttl=253 time=1.84 ms

Eventually, when we’re done we can destroy the infrastructure so we don’t spend too much $.

./destroy_infra.sh

Summary

I’ve covered quite a bit in this post:

  • what AWS Gateway Load Balancer is,
  • how to capture and process raw network packets,
  • how to implement a virtual appliance,
  • presented how it all works.

While this was a pretty simple example, I believe it’s a valuable starting point for more advanced applications of this AWS service.

Resources

  1. GitHub repository with example code.
  2. AWS Gateway Load Balancer official documentation.
  3. Integrate your custom logic or appliance with AWS Gateway Load Balancer from AWS blog.
  4. Geneve: Generic Network Virtualization Encapsulation RFC8926.