When is network introspection not network introspection

If you’ve been digging into the guest introspection setup, and spotted the network introspection driver, this article will explain it’s purpose and how it differs to network introspection.

So what am I talking about? Well a colleague of mine was asking about an explanation of guest introspection (GI) vs network introspection (NI), and as they were digging into the intricacies they got stuck on the below image.

So let’s step back, the problem is simply a bit of naming and branding. Network introspection drivers, as installed by VMware Tools, has nothing to do with Network Introspection in the traditional sense. Time for another image:

OK, so from the following image you can see that introspection services have been broken down into two distinct types, guest and network introspection. Now note the services that are provided by each, data security has been deprecated which is why it’s crossed out, let’s focus on GI for now.

So with GI we support these via either VMware Services or 3rd Party Services. Now here’s the thing, with Activity Monitoring and Identity Firewalling we ultimately need to know the IP address of the VM, as we perform actions based on this information. With Agentless AV/File Integrity we look at storage I/O.

So before we delve deeper, another graphic to look at:

GI is made up of three components –

Thin Agent:

Inside the Virtual Machine Guest OS we need to install Thin Agent (This is actually just an add-on to the existing VMTools drivers). This agent will communicate with the “MUX” via Virtual Machine Communication Interface (VMCI) Channel.

MUX Module:

When we enable Guest Introspection on the cluster level, NSX will install a new VIB (vSphere Installed Bundle) on each ESXi host in our cluster. The new VIB is called “epsec-mux” and MUX module acts as a Multiplexer accepting messages from the Thin Agent running in the VM via VMCI channel, then delivers the information to the SVM (Service Virtual Machine) via a TCP session.

As a part of Guest Introspection deployment, a new standard switch called “vmservice-vswitch” is created at each host, and a new port group named “vmservice-vmknic-pg” will be used for the communication.

Service Virtual Machine – SVM:

On each host in the cluster that we’ve enabled “Guest Introspection” on, NSX will deploy a Service Virtual Machine (SVM).  The SVM will offload the activity needs to be done inside the VM Guest OS and communicate with NSX Manager to send updates and to get new configurations. The SVM have two interfaces: one interface is connected to the standard vSwitch for the MUX traffic, and the Second interface to the management IP defined by the admin to enable the communication to the NSX manager. This communication is done via vDS.

Now to start bringing this all together. Have a look at the following overviews for two use cases with Guest Introspection.

General flow of a Guest introspection – Filescan

  • A file needs to have an action performed on it, read, write, create, etc.
  • The Thin Agent sees that there is an action and locks the file, scans the file, and send it up to the SVA for further investigation.
  • The Thin Agent communicates to the epsec-mux driver on the ESXi host through VMCI to pass this information onward.
  • The SVA communicates to the epsec-mux driver on the ESXi host through TCP/IP and scans the file, provides information on the contents, then sends back information.
  • Once information is gathered on the SVA, the SVA tells the Thin Agent to either delete or ignore the file.

General flow of a Guest introspection – Identity Firewall 

  1. User login to a Virtual Machine (The VM can be part of Virtual Desktop Infrastructure-  VDI) with AD credentials.
  2. On each VM the thin agent (This is VMTools with Guest Introspection installed) running inside the guest os, detects the login event and update the Service Virtual Machine (SVM) with a user id and IP address combination.
  3. The SVM update the NSX manager with these details user details plus AD Group Memberships.
  4. The NSX Manager will store the user id and the IP address inside its database.
  5. NSX manager will then translate the Security Group to VMs and understand where the NSX firewall policy needs to be pushed.
  6. NSX will push the firewall policy to the related VM in the relevant security group.
  7. The user will be able to access the application inside the datacenter from his VM.

Ok but still why the network introspection driver? See below.

The vsepflt.sys is the only driver required for normal Antivirus protection. The vnetflt.sys or vnetWFP.sys (for windows 10 and above) driver is called the NSX Network introspection driver. This driver captures networking events such as AD login/logout and all other normal networking traffic. This driver can be safely removed and does not effect AV if the AV is not configured to use network introspection.

So as you can see, what we are capturing is network events within the VM, this is outside of the visibility of the hypervisor, where the other type of Network Introspection is happening. So as a result we need this driver, who in turn needs to communicate this information to the NSX Manager, but ultimately via the GI-SVM.

In the next article, we’ll talk about proper Network Introspection, and how steering of traffic works.

Bal Birdy on LinkedinBal Birdy on Twitter
Bal Birdy
Bal is an Open Group Certified IT Architect, and VCDX #269, specializing in the network and security arena, with over 15 years experience in enterprise level network/system technologies. His goal has always been to maintain a holistic view of the architecture allowing him to understand how various technology streams may impact the networking/infrastructure space.
Bal has a proven record of delivering on enterprise network designs, leading data center and site migrations as a result of business mergers and acquisitions, and vendor migrations e.g. Cisco to Checkpoint/Juniper. As part of this he worked across several business sectors: Utilities, Banking, Retail and Government, and can base designs around sector specific standards e.g. PCI-DSS, DSD and ISM. He is proficient in several technology areas including Cisco, Juniper, F5, VMware, Citrix and Microsoft. These skills are supported by non-technical certifications: Prince2 Project Management Practitioner, ITILv3, TOGAF 9.1 Certified and Open Group Certified IT Architect – Open CA.
In addition to supporting the Livefire Team, Bal leads several innovation efforts within the VMware WRACE organization, including projects investigating the use of Virtual Reality/Augmented Reality, AI/ML and Interactive 360, to support customer and partner enablement.

BSc (Hons) Computer Science
VCDX-NV #269
Open Group Certificated Architect
Member of the Associated of Enterprise Architects

3 thoughts on “When is network introspection not network introspection”

  1. Hi Bal,
    Thanks for the concise and clear post,
    It seems you mean SVM not SVA in the paragraph “General flow of a Guest introspection – Filescan”,, Otherwise please elaborate what is SVA?!

      1. Thank you for quick reply,
        From EPSecLib definition in the link you shared we may understand that SVA (Security Virtual Appliance) is an SVM hosting EPSecLib, be it GI or partner SVM:

        “EPSeclib is a library used by partner solution SVMs to receive events from Thin Agents and perform operations (read/modify) on those files. EPSecLib can also block file operations. The EPSecLib exists as a virtual machine which is deployed by NSX and runs on on each ESXi host that is prepared for Endpoint. This virtual machine is called the SVA (Security Virtual Appliance).”

Leave a Reply