Debug & Chill 2 - Strange Packet Duplication

Debugging a Strange Packet Duplication

Hi everyone, welcome back! A few years ago, a friend of mine who worked for a large company encountered a strange scenario in their data center: they discovered unexpected packet duplication. I was asked to investigate the issue, and in this post, I’ll share what I found.

As promised, we'll also explore some VMware virtualization concepts in more depth. Let’s start by covering the basics before we dive into the specific bug.

---

A Brief Introduction to VMware Virtualization

• ESX (Elastic Sky X): A server running ESXi, which is a type-1 hypervisor developed by VMware.

• Virtual Switch (vSwitch): A virtual switch configured on an ESX host. It allows two virtual machines on the same ESX to communicate, simulating a physical switch’s behavior.

• Port Group (PG): A group of ports on the vSwitch. If two virtual machines’ network interfaces are placed in the same Port Group, they effectively share the same Layer 2 domain and can exchange ARP requests directly.

A common way to manage port groups is by assigning each PG its own VLAN. Here’s a simplified illustration of how a packet travels between two VMs using VLAN tagging:

1. VM1 sends a packet from its NIC.

2. The ESX host encapsulates it with the VLAN header (defined by the PG), acting like an access port.

3. The vSwitch routes the packet using this VLAN ID to VM2, which is also in the same PG.

4. The ESX host removes the VLAN header before delivering the packet to VM2.

5. VM2 receives the original packet.

Trunk Ports

Another useful feature is configuring a Port Group as a trunk. A trunk interface can carry traffic for multiple VLANs and does not strip VLAN tags on ingress. This is often used in scenarios where you need to pass multiple VLANs into a single VM or device:

In larger environments, multiple ESX hosts are managed by vCenter (VC). vCenter enables distributed virtual switches (DVS) and distributed port groups, providing a centralized way to manage networking across all hosts.

---

Background

To illustrate the problem, let’s look at a simplified version of the setup. In production, the environment was much more complex, but the core issue remains the same:

In this staging environment, there are two separate VLANs:

• VLAN 111 for VM1 and VM3

• VLAN 123 for VM2 and VM4

Each vSwitch also connects to the physical switch via its respective ESX host. A typical traffic flow might be:

1. VM1 sends a packet.

2. ESX1 encapsulates it with VLAN 111.

3. VSwitch1 forwards it to the physical switch through ESX1’s uplink.

4. The physical switch sees VLAN 111 and forwards it to ESX2.

5. ESX2’s VSwitch2 receives the packet, still tagged with VLAN 111.

6. ESX2 removes the VLAN header.

7. VM3 receives the packet.

---

The Problem

One of the engineers noticed that VM1 was receiving nearly double the expected traffic. There was no corresponding increase in user activity, so the suspicion was that packets were somehow being duplicated.

---

The Debugging Process

1. Checking for Monitoring Issues

   To rule out any problems with monitoring tools, I SSHed into both VM1 and VM3. I ran tcpdump on each while sending packets through the network. Sure enough, each packet appeared twice:

   $ tcpdump -eni eth0
   10:11:05.483997 eth0 Out ifindex 2 aa:bb:cc:dd:ee:ff ethertype IPv4 (0x0800), length 60: 1.1.1.1.45554 > 1.1.1.22.443: Flags [R], seq 4037004744, win 0, length 0 
   10:11:05.483997 eth0 Out ifindex 2 aa:bb:cc:dd:ee:ff ethertype IPv4 (0x0800), length 60: 1.1.1.1.45554 > 1.1.1.22.443: Flags [R], seq 4037004744, win 0, length 0

   This confirmed the duplication was real and not just a misreading by the monitoring system.

2. Thinking Of Potential Causes

   • VM1 sending the packet twice: Possibly due to an application or OS configuration.

   • Misconfiguration in the DVS: This might cause a forwarding loop.

   • Flooding from another device on the PG: Another VM could be broadcasting or replaying packets.

3. Testing a New VM

   To test the first possibility, I created a new VM (a plain Ubuntu install) in the same Port Group as VM1. This new VM also experienced packet duplication, ruling out any peculiarities on VM1 itself.

4. Testing a Different DVS and VLAN

   I moved VM1 to a different Port Group and VLAN, but the duplication persisted. However, when I moved VM1 to a new DVS that no one else was using, the duplication disappeared. This strongly suggested the root cause was tied to a specific DVS or ESX configuration.

5. Identifying Inbound Duplication

   On VM1, I used tcpdump -P to separate inbound from outbound packets. One of the duplicated packets was marked inbound, indicating it was being sent back from somewhere else.

6. Investigating the Physical Switch

   I spoke with the IT team and learned that this one switch is actually two physical switches that were running in an LACP (Link Aggregation Control Protocol) configuration for high availability. After some research, I discovered that VMware does not recommend certain LACP setups unless both the VMware and switch configurations are precisely aligned.

   I asked the IT team to temporarily disconnect the ESX host from the LACP-enabled switches. Immediately, the duplication disappeared (albeit with the trade-off that the host was no longer reachable via that switch). This confirmed the LACP switch configuration was the culprit.

It turned out that the switch had restarted a few hours earlier, and its running configuration had not been properly saved. Once the IT team reapplied the correct configuration, the duplication issue was resolved.

---

Key Takeaways

1. Reproducibility

   Before diving into a complex debugging process, ensure you can consistently reproduce the problem. This makes diagnosing the underlying cause much easier.

2. Cross-Team Collaboration

   Don’t be afraid to look beyond your immediate scope. Even if the issue seems related to “infra” or another team’s area, you might uncover crucial insights by collaborating with them.

3. Bonus Tip

   If you’re using IPython, you can press F2 to open your default text editor (e.g., Vim) to edit your current cell. Similarly, in a Bash shell, you can press Ctrl+X+E to open an external editor for the command line.

I hope this walkthrough helps you troubleshoot similar issues in VMware environments or elsewhere. Thanks for reading, and stay curious!