Escaping VMware’s NSX Edge OS Jailed Shell

WMWare-blog-shell-escape

Intro

In this blog post we provide details about a CLI injection vulnerability that we discovered during a penetration testing engagement against VMware Cloud Director. The vulnerability was identified on NSX Edge OS version < 6.4.13.

According to VMware's advisory the following products are affected by this vulnerability:

VMware Cloud Director is a platform that allows the management of large-scale cloud infrastructures. It is possible to create Virtual Data Centers (VDC) hosted on a physical data center or on multiple data centers distributed across different geographical locations. Moreover, it allows to provision network, storage and compute resources in a flexible fashion, allowing easy scalability and resilience.

The following diagram shows how the VMware components work together to provide cloud services and shows where VMware Cloud Director stands with respect to the other components.

vmware-nsx-edge-1

source

From the diagram, vCenter is the component that manages the NSX Data Center for vSphere virtual infrastructure hosts, virtual machines and virtual resources (CPU, memory, storage, and network).

vCloud Director (vCD) sits higher in the cloud infrastructure: it can traverse multiple virtual infrastructures and further abstract virtual infrastructure resources into software-defined storage, networking, security, and availability.

One of the components that allows network management for a VDC is the NSX Edge Router, which is a virtual router running on a Linux kernel that implements all the common features of a real physical router. The purpose of this router is to allow communication between the VDC and the outside world, standing on the edge of the tenant network. In VMWare Cloud Director, users with “Organization Administrator” privileges are allowed to enable SSH on the NSX Edge Router, which would grant them access to a restricted Linux shell that they can use to configure the router’s services.

It is clear that the NSX Edge Router is a key component of the cloud network infrastructure and it is important to harden and protect it from attackers since a compromise of the router might lead to the compromise of internal resources sitting behind it.

NSX Edge Router - jailed shell

The NSX Edge Router runs on NSX Edge OS which implements a jailed shell on top of the standard Linux shell; meaning that it only allows the execution of a subset of standard UNIX commands as well as custom configuration commands that are needed for network management. All other commands are blocked.

A jailed shell is a command shell on a computer system that exists on an isolated subset of the larger system. In essence, it provides a "system within a system" which protects the containing system from being adversely affected by anything that occurs in the subsystem.

source

The NSX Edge shell has different modes that grant different permissions to the operator.

The following table summarises the CLI command modes:

vmware-nsx-edge-2

source

For the purpose of this blog post only the following two modes are of interest.

It must be noted that all the default NSX modes are restricted. They only allow network management commands that interact with the running services using script helpers (special commands).

Other than the default modes, there is also an additional mode called Engineering mode. This mode is used for technical support and is therefore not listed in the command modes. This mode exposes a fully functional UNIX root shell. VMWare advises to “use this mode only in conjunction with a support call to prevent breaking the virtual infrastructure” (https://kb.vmware.com/s/article/2149630). This mode is disabled by default.

If engineering mode is not enabled, users are not permitted to execute any low level UNIX commands/diagnostics on the underlying operating system or to make changes to the appliance.

Jail Escape

During the engagement, we identified that it was possible to escape from a restricted SSH shell provided by the NSX Edge Router to a root shell on the underlying operating system.

This not only escapes the jailed shell but also completely bypasses engineering mode if it is enabled.

To exploit the issue, an attacker would need network access to the NSX Edge router’s SSH service and valid (non-root) credentials. The latter could potentially be retrieved via a brute force or guessing attack, identified in the target network, intercepted, etc.

After authenticating to the SSH server, the user can execute commands in a jailed shell as explained above.

We identified that a number of script helpers (commands) were vulnerable to OS command injection; thus allowing an attacker to obtain a fully interactive shell.

In order to obtain an interactive shell, the attacker would append the following to any of the vulnerable commands:

;sh${IFS}>&2;

The following commands were identified to be vulnerable and could be abused by appending the string above to obtain an interactive root shell:

show statistics dashboard 1 ;sh${IFS}>&2;
show statistics cbm VNIC_1 ;sh${IFS}>&2;
show statistics loadbalancer pool 1 ;sh${IFS}>&2;
show statistics loadbalancer virtual VS_1 ;sh${IFS}>&2;
show nat maps SNAT ;sh${IFS}>&2;

In addition, the following vulnerable command could be abused to obtain an interactive unprivileged (admin – not root) shell:

show ipv6 forwarding ;sh${IFS}>&2;

After running some of the aforementioned commands, the terminal needed to be reset (using the reset command) to allow correct handling of shell input and output and be able to use a new interactive shell.

vmware-nsx-edge-3

If an attacker gains SSH access to a device running a vulnerable version of the NSX Edge Operating System, they could exploit this vulnerability to obtain root access to the underlying operating system.

The previous screenshot showcases the exploitation of the vulnerability from an unprivileged shell with engineering mode disabled.

Impact

As mentioned, for this vulnerability to be exploitable:

Exploitation of this issue could have adverse effects on the confidentiality, integrity and availability of the systems. For example, it could lead to:

Mitigation

Since we reported the vulnerability to VMware, it has been assigned a CVE number (CVE-2022-22945) and VMware advisory (VMSA-2022-0005).

The vulnerability has been patched in version 6.4.13 of VMware NSX Data Center.

Therefore, the best way to remediate the issue is to patch the affected system.

Additionally, it is important not to expose the SSH service running on the NSX Edge router to the Internet. Management ports should only be accessible to staff who require them to administer the device. IP restrictions to only allow trusted addresses should be used if the virtual device needs to be managed over the Internet.

You may also be interested in...

imagensecforcepost.png
Oct. 30, 2008

Penetration testing and risk management

Penetration tests need to put findings into context performing a risk assessment of how a specific security issue affects the business.

See more
imagensecforcepost.png
Nov. 9, 2008

Penetration testing, antivirus, firewalls and false sense of security

False sense of security is an ongoing issue. Fueled by inaccurate marketing strategies promising the ultimate security product and convincing clients that their product will make your system immune to every single attack.

See more