VMware NSX Multi-Tenancy – Part 2

In Part 2 of this blog series dedicated to the new VMware NSX 4.0.1.1 multi-tenancy feature, we shall discover more about the Security aspect of the solution.

Note: All JSON files used in the blog are available from my GitHub repo:
https://github.com/nnikodimov/nsx-multi-tenancy

Project default Groups and Policies

As we reviewed in VMware NSX Multi-Tenancy – Part 1, a default group is automatically created per Project to restrict the scope of the rules to a particular Project. The NSX Manager adds all the segments created in a Project to the respective default group. The VMs’ membership to a Project is then based on their segment connectivity.

A default security policy is auto-created to block any traffic (except DHCP) IN and OUT of the Project. Full communication between VMs in the Project is allowed.

I have already created four VMs (2 per Project) and connected them to the respective segment. Because they are connected to Project’s specific segments, these VMs are automatically placed in the default Project group, and we can see their IP addresses.

​Although the Policy UI shows these default policies applied to the DFW, the default project group is used in the apply-to field for the project-specific security policies. This can be verified using the Manager UI.

Testing the access from the management jump box we have, confirms that out-of-the-box, we can’t communicate with the Projects’ VMs.

I don’t want to waste time proving there is no routing issue in the environment, and the communication problem is really related to the default Project policies, so you will likely have to trust me on that. ?

/infra Policy to Projects

In the /infra space, we can apply rules to Projects only on the default group directly or using static membership.
Let’s use that and create Management access from our Linux jump-box to all the workloads running in both projects, using a Policy API and JSON body provided below. Note the scope property, which defines the Project specific default group.

JSON
livefire@ubuntu-utils:~$ cat nsx-multi-tenancy/infra_management_policy.json
{
  "resource_type": "Infra",
  "children": [
    {
      "resource_type": "ChildDomain",
      "Domain": {
        "id": "default",
        "resource_type": "Domain",
        "children": [
          {
            "resource_type": "ChildGroup",
            "Group": {
              "resource_type": "Group",
              "id": "MGMT",
              "display_name": "MGMT",
              "expression": [
                  {
                      "ip_addresses": [
                          "192.168.110.11"
                      ],
                      "resource_type": "IPAddressExpression"
                  }
              ],
               "effective_member_types": [
                  "IPAddress"
              ]
            }
          },
           {
            "resource_type": "ChildSecurityPolicy",
            "SecurityPolicy": {
              "id": "Infra_Management_Policy",
              "resource_type": "SecurityPolicy",
              "display_name": "Infra_Management_Policy",
                          "category": "Emergency",
              "rules": [
                {
                  "resource_type": "Rule",
                  "display_name": "Management_Access",
                  "sequence_number": 1,
                  "source_groups": [
                    "/infra/domains/default/groups/MGMT"
                  ],
                  "destination_groups": [
                    "ANY"
                  ],
                  "services": [
                    "ANY"
                  ],
                  "scope": [
                    "/orgs/default/projects/project-prd/infra/domains/default/groups/default",
                    "/orgs/default/projects/project-dev/infra/domains/default/groups/default"
                  ],
                  "action": "ALLOW"
                }
              ]
            }
          }
        ]
      }
    }
  ]
}

Because this is an /infra API call, we use the Enterprise admin user to run it.

Verifying from the NSX Manager UI we see the policy created the the Emergency category.

We can now access the Projects’ workload and SSH to the web front-end of the application (for example, in our Development Project). We can also test the Web VM access using the DB VM’s IP address, which also works.
However, if we test the access to the DB using its FQDN, this fails, and we see that the failure is because of DNS resolution. This failure is again related to the default Project policy that out-of-the-box does not allow any outbound traffic (except DHCP).

Projects workload micro-segmentation

The VMware NSX multi-tenancy supports creating groups and policies within the Project.
The groups and the policies of a Project apply only to the VMs that are part of the Project. As mentioned above, the VMs to Project belonging are determined by the segments to which the VMs are connected.
The policies and rules within a Project, including those having DFW in the Applied To field, do not impact workloads outside the Project.

Let’s create Projects specific groups and micro-segmentation policies. Below is the JSON body for the API call that we will use to create the groups and policy for the Development project.

JSON
livefire@ubuntu-utils:~$ cat nsx-multi-tenancy/project-dev-2tier-app-micro-segmentation.json
{
    "resource_type":"Infra",
    "children":[
    {
        "resource_type":"ChildResourceReference",
        "id":"default",
        "target_type":"Domain",
        "children":[
        {
          "resource_type":"ChildGroup",
          "Group":{
              "resource_type":"Group",
              "display_name":"WEB",
              "id":"web",
              "expression":[
              {
                "member_type":"VirtualMachine",
                "key":"Tag",
                "operator":"EQUALS",
                "scope_operator": "EQUALS",
                "value": "webapp|web",
                "resource_type":"Condition"
              }
            ]
          }
        },
        {
          "resource_type":"ChildGroup",
          "Group":{
              "resource_type":"Group",
              "display_name":"DB",
              "id":"db",
              "expression":[
              {
                "member_type":"VirtualMachine",
                "key":"Tag",
                "operator":"EQUALS",
                "scope_operator": "EQUALS",
                "value": "webapp|db",
                "resource_type":"Condition"
              }
            ]
          }
        },
        {
          "resource_type":"ChildSecurityPolicy",
          "SecurityPolicy":{
            "id":"project-dev-micro-segmentation",
            "resource_type":"SecurityPolicy",
            "display_name":"Project-Dev Micro-Segmentation",
            "rules":[
              {
                "resource_type":"Rule",
                "id":"https-access",
                "display_name":"HTTPS Access",
                "sequence_number":10,
                "source_groups":[
                  "ANY"
                ],
                "destination_groups":[
                  "/orgs/default/projects/project-dev/infra/domains/default/groups/web"
                ],
                "services":[
                  "/infra/services/HTTPS"
                ],
                "action":"ALLOW"
              },
              {
                "resource_type":"Rule",
                "id":"mysql-access",
                "display_name":"MySQL Access",
                "sequence_number":20,
                "source_groups":[
                  "/orgs/default/projects/project-dev/infra/domains/default/groups/web"
                ],
                "destination_groups":[
                  "/orgs/default/projects/project-dev/infra/domains/default/groups/db"
                ],
                "services":[
                  "/infra/services/MySQL"
                ],
                "action":"ALLOW"
              },
              {
                "resource_type":"Rule",
                "id":"default-drop",
                "display_name":"Default Drop",
                "sequence_number":30,
                "source_groups":[
                  "ANY"
                ],
                "destination_groups":[
                  "ANY"
                ],
                "services":[
                  "ANY"
                ],
                "action":"DROP"
              }
            ]
          }
        }
      ]
    }
  ]
}

Note that we are running the API call authentication with the project-dev Project Admin user – Sally (ref. VMware NSX Multi-Tenancy – Part 1). As expected, the call was successful.

For a test, let’s run the API call to create the micro-segmentation policy and group for the Production project using Sally again. As expected, we received an Error message that the user is not authorized.
Let’s create the objects using the correct Production Project Admin – Bob. This time the call is successful.

In the NSX Manager UI, Inventory > Groups, we see the new objects created per Project. If we View the Members for one of the groups, we see the Group dynamic membership criteria, which is exactly the same for both the Projects’ groups (ref. to the JSON files in the Github repo). However, if we check the Effective members, we have only the VMs that belongs to the respective Project.

We have the two Project policies created in the Distributed Firewall, Application category. Usually, I get nervous ? when I see an ANY, ANY Drop rule in the middle of my policies that is applied to the DFW.
This should affect any Allow rule below it.

But you have to remember that this is a Project-specific policy, and all its rules are applied only to the Project specific workload. This can be confirmed if we switch to the Manager UI, where we can see that the rules are applied to the Project default group.

At this stage, our test 2Tier Application should work. But surprisingly, we have access to the Web frontend, and then the connection to the backend DB is failing.

I will spare the troubleshooting, knowing what the problem is :). It is a simple DNS resolution issue because, if you remember, all Project outbound traffic is blocked by the Project default policy.
To create a rule that allows the DNS traffic, let’s PATCH the current Project micro-segmentation policy using the JSON body below. We already have a Group that includes our DNS server (DNS-Servers) so let’s use it.

JSON
{
  "resource_type": "SecurityPolicy",
    "rules": [
    {
      "resource_type": "Rule",
      "id":"allow-dns",
      "display_name": "Allow DNS",
      "sequence_number":1,
      "source_groups": [
        "ANY"
      ],
      "destination_groups": [
        "/infra/domains/default/groups/DNS-Servers"
      ],
      "services": [
          "/infra/services/DNS",
          "/infra/services/DNS-UDP"
      ],
      "action" : "ALLOW"
    }
  ]
}
livefire@ubuntu-utils:~$

However, when we run the API call, we receive an Error message that the requested object (DNS-Servers group) could not be found. I am pretty sure that the group exists, so what is the issue?

Resource Sharing

The problem is that the DNS-Servers group is an /infra object; before we can use it in a Project-specific policy, this object must be shared from /infra to /orgs. Below is the API call body we will use to share the DNS-Servers group with the Orgs.

Note that when you share an object, you can also choose to share the child objects if required.

JSON
{
    "resource_objects": [
        {
            "resource_path": "/infra/domains/default/groups/DNS-Servers",
            "include_children": false
        }
    ]
}

Let’s do that by running the following API call.

Note: We can use the GET /policy/api/v1/infra/shares/default/resources/default API to view the objects shared by default with all Projects. For example, all NSX Manager Services and ContextProfiles are shared by default. This is why we did not get an Error message for the DNS and DNS-UDP services in the previous API call. In addition to the objects shared by default, we can also share Group; DhcpServiceConfig; DhcpRelayConfig; Service; PolicyContextProfile, and Segment.

Now that we have shared the DNS-Servers group with the Orgs, we can run the policy PATCH API call, which is successful this time.

The NSX Manager UI visualizes the rule in the Project specific micro-segmentation policy.

And luckily, our 2Tier Application is working as expected.

This concludes the 2nd part of this blog series in which we have explored the VMware NSX (v 4.0.1.1) Multi-Tenancy. Although this feature is currently only available through API, it shows that VMware NSX is becoming a complete multi-tenant SDN platform.

Nikodim Nikodimov on LinkedinNikodim Nikodimov on Twitter
Nikodim Nikodimov
Based in Sofia, Bulgaria, Nikodim is a Staff Solutions Architect at VMware, Inc. He has strong background in Network and Security fields, specialized over 12 years in professional services consultancy, scoping experience and cognition to provide high quality, technical, and informed knowledge within large enterprise and datacenter network environments.

Nikodim has broad experience in a number of technology areas including Cisco, Checkpoint, HPN, Juniper, Fortinet, F5, VMware, Microsoft and Linux. He is Certified Cisco Systems Instructor and holds CCIE Security.

His current focus includes large-scale data centers and cloud deployments, software defined networking (SDN) and data centers (SDDC), and network function virtualization (NFV).

Leave a Reply