Defending Container Compromise

Part two in a series.

In part one of the series, I outlined a scenario in which an attacker could start with a container compromise, perform a container breakout, host compromise and move on to Kubernetes and IaaS reconnaissance. The attacker had to achieve multiple objectives (in combination with several developer errors) to make the scenario possible. Despite these conditions, the scenario itself is not only extremely plausible but may have already occurred in the wild.

Let’s break down the five phases of the scenario to analyze how Aqua Security can help security teams prevent the attack.

Poisoned Docker Image

The scenario begins with an image preloaded with malware. So, how can we derail the attack before it even starts? Perhaps you have heard of “Shift Left,” a term that refers to enhancing security as close to the beginning of the CI process as possible. For example, this entire scenario can be averted if an “Aqua Scan” stage is placed in the gitlab-ci file used to build the image and deploy the container. By adding a few simple lines of code, the image can be scanned post-build and pre-deployment to ensure it meets the compliance and security checks imposed by your organization. The code snippet below illustrates an Aqua Scan stage.

If the image doesn’t meet the policies applied during the scan stage, the job fails until the developer takes remedial action. An example is shown here.

The Aqua Cloud Native Security Platform (CSP) provides default Assurance policies for these types of scans and provides the ability to create custom ones. For more granular policies, Aqua even provides a way to create individual compliance checks that can be added to an assurance policy.

Note that the malware for this post was not flagged by Aqua. The same image was scanned by other technologies with the same result. However, the Aqua scan stage would have stopped the scenario before the malicious image was used in a deployment task. The image was not registered (approved) and it was using root privileges. Which brings us to step 2…

Running a Privileged Container

In order for the attacker to break out the container must be running in an elevated state, which means running containers in a privileged security context should be avoided if possible. In an ideal state containers run with the least amount of privilege necessary to perform a task. (See the Kubernetes container security context documentation for more details.) The deployment.yml file indicates the securityContext of the container here is running privileged.

Aqua not only allows us to identify the use of privileged containers, but it can also prevent them from being deployed. Aqua’s Risk Explorer shows us that the privileges used in the container running kiralyd/ubuntu:latest image are running as “Super User” and that the container is privileged.

We also see this status with Risk Explorer on the Risk and Configuration tabs…

…and on the Configuration tab of the container Risk Explorer.

A quick policy change within Aqua CSP can easily prevent users from deploying privileged containers.

Container C2 Connection

After the initial container deployment a connection is made to the C2 sever located in Azure. It isn’t necessary for the attacker to be in Azure, but we decided to test the level of difficulty associated with it. Regardless of where the C2 server is located, application developers should understand the necessary ports and protocols used by each container. This way Aqua firewall policies for each container can be created to prevent any unnecessary inbound, or in our case, outbound network connections.

Even if the developer is unsure of all the needed network connections, Aqua CSP can record the network traffic, affording the administrator the ability to create rules from captured connections.

Once the container firewall policy is applied any connections made to this IP address and/or port number will be blocked and an alert will be created in the Aqua console. Administrators logging into the UI will see there are blocked alerts associated with any connection attempts made to the command and control server.

Greater detail about these events can be found in Aqua’s Audit view. The event is accompanied with information that includes the alert category, action, IP/DNS, port, Aqua response, policy and details.

Container Breakout

The third phase (or requirement) of the scenario is a docker container breakout. We noted earlier that the breakout can’t occure unless the container is running privileged. But what else can an Aqua administrator do to prevent the breakout? Limiting what binaries can be executed in the container and on the host also prevents a breakout. For example, the breakout requires the use of chmod. An Aqua administrator can easily prevent the use of chmod from executing in a container with a File Block Runtime policy. This would prevent the attacker from making the cmd file in the breakout executable.

The image above shows how simple the file block creation is and the result of the policy is shown in the container below.

As shown below, the code used in the breakout also required the host to execute chmod and curl.

Much like the container runtime policy, a host runtime policy can also be created to prevent execution. This policy prevents even root users from executing these commands.

Similar to the container runtime polices, the Audit tab will display the block alerts associated with host runtime events that match policy.

Host Malware Download and C2 Connection

In our scenario the attacker requires the host to make a network connection to pull down a file and once that file is executed it makes a second network connection back to the command and control server. Much like container firewall policies, host firewall policies are created in the same manner and can easily prevent inbound or outbound communication.

Conclusion

There are numerous ways in which an administrator, with a little thought and a few minutes in the Aqua console, can prevent the attack scenario outlined in this series. The prevention methods discussed here are only some of the options offered Aqua Security CSP. We encourage you to explore how Aqua can be best employed safeguarding your development environment.