NSX-T: BUM Replication with Two-Tier Hierarchical Replication Mode Deep Dive

In this article we shall validate that Hierarchical Replication Mode utilises unicast messaging, vs layer 2 multicast as seen with NSX-V. Read on or click the link in your Alexa app.

As part of my continued learning of NSX-T I wanted to confirm a few things with respect to how BUM replication is handled. There are numerous blogs and articles about head end vs hierarchical replication, the two modes utilised with NSX-T. Yet I couldn’t find any information to demonstrate at a packet level whether hierarchical is unicast or layer-2 multicast in nature. I caught up with some colleagues who were of the opinion that all messaging happens via unicast channels. So I wanted to validate this.

So firstly the environment, I have two ESXi hosts and two KVM hosts on two segments. ESXi hosts are attached to 192.168.140.x and KVM hosts are attached to 192.168.150.x. I attached VMs that reside on both my KVM hosts, and my ESXi host, so that from a TEP perspective, all hosts would be members of the web logical switch. See below.

Now what I’ve highlighted above is that, from the perspective of esxcomp-02a, 192.168.150.152 i.e. kvmcomp-02a is my remote MTEP in my deployment. So what does this mean? Hierarchical Replication results in a remote host being nominated as the MTEP, or replication node, for BUM traffic in the remote TEP segment. So this is very similar to Unicast Mode (UTEP) and Hybrid Mode (MTEP) with NSX-V, but unlike Hybrid Mode in NSX-V the messages are set as unicast messages, and not layer-2 multicast. What was initially throwing me off was the Is MTEP part of the output, in my head when I see MTEP I immediately associated this with hybrid mode, or at least some sort of multicast messaging. Which is not the case, and it seems this is a simply the UI utilising MTEP when in fact we should be talking about UTEPs.

So I went on to validate this through a packet capture on the kvmcomp-02a, which you’ll see below.

In this output I’ve highlighted a few key elements:

  1. Firstly remember GENEVE is UDP 6081
  2. The first and second box show the heirarchical replication mode happening. I had initiated a ping from Web-01 to 172.16.10.14 (web04), where Web-01 resides on ESXi-01 (192.168.140.151). As you can see there’s a message going out over my GENEVE Tunnel to my MTEP (192.168.150.152), who in turn is sending a unicast message to the other KVM host (192.168.150.151) on my segment
  3. Secondly I initiated a ping from Web-01 to Web-03, to demonstrate that traffic goes direct between VMs that the NSX controllers have known MAC/ARP entries.

Finally, it’s always worth referring back to IETF documentation,  https://tools.ietf.org/html/draft-gross-geneve-01.

In the Geneve standard the following statements are made in Section 4.1.3 Broadcast and Multicast:

Hopefully you find this article valuable just to cement understanding and knowledge. If there’s any errors please provide feedback via the comments, and good luck with your NSX-T journey.

Thanks

 

Bal Birdy

Leave a Reply

%d bloggers like this: