XenServer & Port Mirroring

Mirrored kittenBlogs are made to provide valuable content to readers (well, I hope for my readers). This time, nothing related to security though… Recently, I built a new virtualization platform at home based on XenServer 6.2. Why the Citrix solution? Just because the box has 72GB of memory and the free version of VMware cannot address more than 36GB of physical memory so…good bye VMware! After the migration of multiple VMs and labs, the last one to move was my Suricata IDS. To achieve this, A physical interface on the new box was dedicated to receive a mirror of all my traffic. Enabling a promiscuous interface at guest VM should be a piece of cake… Really?

It took me four (4!) days – ok, I’m not a virtualization guru – and a lot of coffee to find the solution! When you google for some help to setup port mirroring on a XenServer, most websites refer to the following document: CTX121729. But it does not work and, according to forums, I was not alone in the same situation. In fact, since version 6, XenServer is using OpenVSwitch as its core network component. Some articles recommend to switch to the old system based on standard bridges as described here. This is not an option because you loose nice replication features. Other articles say that it’s possible to mirror traffic using the RSPAN feature of OpenVSwitch. To achieve this, you need to download and deploy the “VSwitch Controller virtual appliance“. I did this and tried to play with the RSPAN feature. It looks ok when you need to mirror the traffic from one or more guests to another one but, in my case, I needed a complete copy of packets seen from a physical interface!

Finally, I found the solution on the OpenVSwitch man pages and more precisely the ‘ovs-vsctl‘ command. This command is available in your dom0 and allows the configuration of your virtual switch. Example:

[root@dom0 ~]# ovs-vsctl show
69fd7f44-cf16-413f-9d9d-e4cd2f7ef5e1
    Manager "ssl:192.168.x.x:6632"
        is_connected: true
    Bridge "xenbr1"
        Controller "ssl:192.168.x.x:6633"
            is_connected: true
        fail_mode: standalone
        Port "xenbr1"
            Interface "xenbr1"
                type: internal
        Port "vif1.1"
            Interface "vif1.1"
        Port "eth1"
            Interface "eth1"
    ovs_version: "1.4.6"

The command is quite complex and accepts plenty of arguments: Please read the manpage. The mirrored traffic coming from my switch is connected to the dom0’s physical interface ‘eth1‘ and the VM running my IDS has the virtual interface ‘vif1.1‘. What I needed is just a copy of all traffic on ‘eth1‘ to be mirrored to ‘vif1.1‘. No need to use RSPAN, it can configured using the following command:

[root@dom0 ~]# ovs-vsctl -- set Bridge xenbr1 mirrors=@m \
-- --id=@eth1 get Port eth1 \
-- --id=@vif1.1 get Port vif1.1 \
-- --id=@m create Mirror name=eth1-mirror select-dst-port=@eth1 \
select-src-port=@eth1 output-port=@vif1.1

Set the guest interface in promiscuous mode and you will see the network flows!  I just have one remark: I don’t know if the command above was saved somewhere or if I need to execute it at each boot (I still need to expand my OpenVSwitch knowledge). I can’t reboot my box now to test but if somebody knows the answer, share your findings!

15 comments

  1. This post was a life saver, thank you.

    This doesn’t survive reboot because, as far as I can tell, once the VM that owns the VIFs shuts down the VIFs are destroyed, so the correct VIF numbers must be looked up every time this command is run, which should be every time the VM boots.

    My solution for this is a go binary I made (https://gist.github.com/hkparker/c18c35efe5ac14751d86bc47a59741cc) to parse the output of ovs-vsctl and exec the correct commands. When my IDS boots a reboot cron job to sshes into dom0 and executes it. That way I can reboot the VM, or reboot the xen host, and traffic will flow after either.

  2. This method is still working fine, but I have one question:
    The vif number increases with every VM reboot. Will it keep increasing forever, or will it choke at some point?
    Do previous ovs-vsctl commands need to be negated? Should anything ever be reset?
    Thanks.

  3. I suppose a scripting guru could write a startup script to find the new vif number, dump it in a variable, and then run the ovs=vsctl command with that variable.
    Any takers?

  4. Same problem here. I don’t have persistence across reboots.
    I added this step in my emergency reboot procedure to not forget it.

  5. Thank you!
    I was only on day 2 of trying to figure this out, so you saved me a few days!
    Any ideas on how to make this stick after a restart of the machine?
    The vif # keeps changing.

  6. just in case you wanna go back to VMware: the upcoming vSphere 5.5 removes the memory limit from the free hypervisor version. (upcoming=officially announced two weeks ago)

Leave a Reply

Your email address will not be published. Required fields are marked *