[minipost] Capturing bidirectional traffic of virtual machine (VMs) on vmWare ESX 6.x

Here I was getting a trouble with communication between an ESX virtual machine and the nearby switch (Nuage/DCN controller VM talking with a VTEP switch if someone is interested) and because that switch was direct destination of the control plane packets (OVSDB over TCP) I was not having much success creating a mirroring interface on the switch. So I learned how to capture a specific virtual machine traffic directly on the ESXi host’s SSH console and to not forget that, I will document this here.

Step 1 – enable SSH to the ESX host

In most cases this is not running by default, so go to the ESXi server direct terminal or iLo and via “F2″enter system customization and enter troubleshooting section:

ESX host troubleshooting options location

ESX host troubleshooting options location

Right behind this menu should be “SSH Enable” option, and you simply hit that with enter 😉

ALTERNATIVE: If you have vCenter deployed, you can use its GUI to enable SSH on a specific host like this:

Locate ESX Host in vCenter and open its security profile

Locate ESX Host in vCenter and open its security profile

Inside security profiles enable SSH server

Inside security profiles enable SSH server

Step 2 – locating switchport ID

My virtual machine was called “DCN4.0R5_VSC1” and was simply connected to logical vswitch0 in ESX hosts operation. However what we need to find out is the numerical id of the specific switch port. To do this we have two possible paths, first options is to check “net-stats -l” in ESXi hosts SSH console like this:

ALTERNATIVE if you do not know your VMs name or you would like to know what portgroup it is connected to, you can use “esxcli” commands like this:

And then details of the port of your VM with:

Step 3 – running the capture

The funny part is that on ESX there exists a command pktcap-uw to capture traffic on a specific VM’s interfaces, for example for the VMs I was targeting I would run this command (with traffic captured going to /tmp/ ):

However this command is by default a unidirectional and we have to trick our way around to get bidirectional traffic. There is a selector switch in this commend “–dir 1” that captures input packets and “–dir 0” that captures output packets and what we can do this by using the good old “&” run two captures in parallel capturing both directions into separate files like this:

The above will start two captures in the background. To stop this capture, you can either list the processes using “lsof” command and then stop them by killing them using “kill <number>” manually, or using this great trick by parsing all kill processes from lsof to the kill command using awk:

The result isthat you will have two pcap files, one for each direction, but how to put these together for a coherent capture file? Next step is for that 😉

Step 4 – Merging two unidirectional pcap files to one bidirectional

Actually since both pcap files were captured on one system (with one internal clock), the pcap files have a synchronized timestamp marking packets, therefore a simple merge with utility called “mergecap“, which is part of wireshark installation on both linux and windows, is all that we need. For example:

Windows mergecap example:

Linux ergecap example:

And that’s it, you can now open 33554495_merged.pcap in wireshark and see your ESX VMs traffic as a normal bidirection traffic capture. Enjoy.

PS: If you wonder about timestamps inside pcap files, or in the future would have two captured files from two systems that do not have clocks synchronized and would want to merge files like that, have a look on capinfos and editcap utilities from wireshark. You can offset timing with them very easily (editcap -t <offset> A.pcap B.pcap).

Peter Havrila

About Peter Havrila

Peter's Profile Page