In this blog post we provide details about a CLI injection vulnerability that we discovered during a penetration testing engagement against VMware Cloud Director. The vulnerability was identified on NSX Edge OS version < 6.4.13.
According to VMware's advisory the following products are affected by this vulnerability:
- NSX Data Center for vSphere [version <6.4.13]
- Cloud Foundation (NSX-V) [version 3.x]
VMware Cloud Director is a platform that allows the management of large-scale cloud infrastructures. It is possible to create Virtual Data Centers (VDC) hosted on a physical data center or on multiple data centers distributed across different geographical locations. Moreover, it allows to provision network, storage and compute resources in a flexible fashion, allowing easy scalability and resilience.
The following diagram shows how the VMware components work together to provide cloud services and shows where VMware Cloud Director stands with respect to the other components.
From the diagram, vCenter is the component that manages the NSX Data Center for vSphere virtual infrastructure hosts, virtual machines and virtual resources (CPU, memory, storage, and network).
vCloud Director (vCD) sits higher in the cloud infrastructure: it can traverse multiple virtual infrastructures and further abstract virtual infrastructure resources into software-defined storage, networking, security, and availability.
One of the components that allows network management for a VDC is the NSX Edge Router, which is a virtual router running on a Linux kernel that implements all the common features of a real physical router. The purpose of this router is to allow communication between the VDC and the outside world, standing on the edge of the tenant network. In VMWare Cloud Director, users with “Organization Administrator” privileges are allowed to enable SSH on the NSX Edge Router, which would grant them access to a restricted Linux shell that they can use to configure the router’s services.
It is clear that the NSX Edge Router is a key component of the cloud network infrastructure and it is important to harden and protect it from attackers since a compromise of the router might lead to the compromise of internal resources sitting behind it.
NSX Edge Router - jailed shell
The NSX Edge Router runs on NSX Edge OS which implements a jailed shell on top of the standard Linux shell; meaning that it only allows the execution of a subset of standard UNIX commands as well as custom configuration commands that are needed for network management. All other commands are blocked.
A jailed shell is a command shell on a computer system that exists on an isolated subset of the larger system. In essence, it provides a "system within a system" which protects the containing system from being adversely affected by anything that occurs in the subsystem.
The NSX Edge shell has different modes that grant different permissions to the operator.
The following table summarises the CLI command modes:
For the purpose of this blog post only the following two modes are of interest.
- Basic. Basic mode is a read-only mode. To access configuration commands, you must enter privileged mode.
- Privileged. Privileged mode commands allow support-level options such as debugging and system diagnostics.
It must be noted that all the default NSX modes are restricted. They only allow network management commands that interact with the running services using script helpers (special commands).
Other than the default modes, there is also an additional mode called Engineering mode. This mode is used for technical support and is therefore not listed in the command modes. This mode exposes a fully functional UNIX root shell. VMWare advises to “use this mode only in conjunction with a support call to prevent breaking the virtual infrastructure” (https://kb.vmware.com/s/article/2149630). This mode is disabled by default.
If engineering mode is not enabled, users are not permitted to execute any low level UNIX commands/diagnostics on the underlying operating system or to make changes to the appliance.
During the engagement, we identified that it was possible to escape from a restricted SSH shell provided by the NSX Edge Router to a root shell on the underlying operating system.
This not only escapes the jailed shell but also completely bypasses engineering mode if it is enabled.
To exploit the issue, an attacker would need network access to the NSX Edge router’s SSH service and valid (non-root) credentials. The latter could potentially be retrieved via a brute force or guessing attack, identified in the target network, intercepted, etc.
After authenticating to the SSH server, the user can execute commands in a jailed shell as explained above.
We identified that a number of script helpers (commands) were vulnerable to OS command injection; thus allowing an attacker to obtain a fully interactive shell.
In order to obtain an interactive shell, the attacker would append the following to any of the vulnerable commands: