In this article we shall discuss logical routing in NSX-T from the perspective of north/south routing and stateful services. Read on, or click on the link in your Alexa app.
In the previous article I discussed how east/west routing was supported, via the logical routing function in NSX-T, but without the need for control-VMs. In this article we’ll expand on the scenario to talk about north/south needs and stateful services. Before we do that we need to talk about Edge Nodes and Service Routers. ESGs … NO NO NO … Edge Nodes, something completely different. This will be discussed at the end of this article.
Firstly I want to close out the Services Router piece. For every T0/T1 router that you have in your environment you can have an associated services router. The services router is sort of like the ESG in the NSX-V world. It’s a centralised routing construct that handles stateful services and north/south traffic. Where it differs from the ESG is in the way it’s instantiated, and the way interacts with the environment. I’m probably not articulating this in the best way, but let’s work through this and then you’ll see what I mean.
The key thing to understand is that each Logical Router consists of both a distributed router construct (DR) and a centralized routing construct (SR). If you look at the image below you’ll see that Tier0 has both a DR and a SR function running on my Edge Node.
I think at this point it’s worth noting terminology, I’m saying DR and not DLR. Remember one logical router, consisting of a distributed and service function.
Now if you are wondering does the SR exist on my compute nodes, no it does not, the SR only exists within the Edge nodes, as it’s a centralized function. As it’s a centralized function, similar to an ESG, it serves a core function of providing physical connectivity and integration. To achieve this the SR is connected to VLAN backed uplinks, to the physical network, and as it’s not distributed, we can support dynamic routing via BGP, without any worries about a need for a control-VM. Happy Days!!
Hopefully you’re already asking the question about how the packet get’s from the VM to the SR, to allow connectivity to the physical network. Remember when we attached a Tier-1 to a Tier-0 router NSX-T did some auto-plumbing, the same is true when an SR is instantiated. So firstly an SR is created when you either configure an uplink on your Tier-0 Logical Router, remember the SR is a construct that’s part of the LR, or configure some stateful services. When the SR is setup NSX-T auto-magically connects the SR to its associated DR, this is utilising a network based on the 169.254.0.0/28 network, and the Management Plane automatically adds the associated static routing. For routing, a default route is added on the DR pointing to the SR, and on the SR side static routes for the DR networks are added. So now when the VM sends a packet destined outbound, a local route lookup on the DR instance on the compute node it resides on, will say send packet to the T0 DR instance on the same node, which in turn will say send the packet to the SR. So all the outbound routing logical happens on the local node/hypervisor. Return traffic happens on the Edge Node where the SR resides.
From traffic flow perspective the following diagram demonstrates how things work at a high level:
So let’s break this down a bit, and validate how the SR is setup for us, and then we’ll close out with a discussion on edge nodes.
- At the of Part 1 we setup east west routing between out Tier1 connected logical subnets, and attached this to our Tier0 router. The goal was to demonstrate that without an uplink or stateful services enabled on Tier0, the edge node was not utilized, i.e. no SR was instantiated. let’s quickly double check this. Below you’ll see that on my Edge Node I have no SR, and in fact no DR either. This I’m expecting as the DR will only existing in the Edge Node once I have an SR
- I’ll now configure an uplink on my Tier0, and I expect an SR to be created, and that 169.254.0.0/28 auto plumbing between SR and DR to happen
- After configuring the uplink we can confirm the t0 DR and SR are setup on the Edge Node, as well as the T1 instance for local route lookups. If we look at the route table we shall also see we have the 169.254.x.x connection as expected.
So on our Edge Node, we have now validated that the SR instance gets setup only ones an uplink, or stateful services is configured. So now let’s close this out with a discussion about the Edge Node.
As always, I hope this helps, any errors please provide feedback and good luck in your NSX-T journey.