Container Runtime Protection

October 29, 2021

Are you asking the right questions when it comes to selecting container security tools?

Over the last two years several endpoint security vendors have started offering coverage for container environments. Some of these vendors have internally developed container security features, while others have made acquisitions to get a jump on the space. How mature are these products? And how do they stack up to a purpose-built solution like Palo Alto Networks Prisma Cloud?

Lab Overview

To get an understanding of the current state of these offerings, Optiv devised a quick runtime protection test plan. This included a Kubernetes environment for each endpoint security product tested. These clusters were built in AWS using EKS, and each EKS cluster was running the Weaveworks microservices application Sock Shop.

Image

Note: Not all pods for the Sock Shop microservices application are displayed in the image.

Each product evaluated was installed on its own independent EKS cluster comprised of three ec2 nodes running Amazon Linux 2. Optiv used these environments to understand how far endpoint security vendors have come in providing protection and visibility into container runtime environments. Optiv also provisioned a cluster for Prisma Cloud. During the evaluation we were looking to see if each solution could provide detail around:

Namespace information:

Network connections:

Runtime detection/prevention policy – out of the box

Scenario Overview

To make the attack scenario as level across all products as possible, Optiv focused on container runtime protection. We also took a few liberties when purposely misconfiguring one of the pods for the initial compromise. Note: while many container security products also focus on web application security, Optiv saw it as unfair to perform a web-based attack on the application, given the feature sets available from endpoint security tools.

In past blog posts, we used a well-documented container breakout that leverages cgroups. The cgroups exploit allows the attacker to move from the container to the host node. This time around we took a simpler approach and ran the container as root. Why is that simpler? If a container is compromised when it’s running as root the attacker can mount the host drive as a temp directory. Yep – that’s all it takes. It’s understood that there are multiple ways to stop this scenario – don’t run the pod as root, scan the image in the registry before deploying it, etc. But for this project we focused on an easy-to-replicate runtime protection use case.

Attack Steps:

Listed below are the exact attack commands that are executed through the initial c2 channel on the pod once the reverse shell executes at runtime.

Image

Note: After the commands were executed the attacker receives a connection from the EKS node in three minutes.

Findings

Deployment Architectures

There have been two different approaches when it comes to gaining visibility and providing protections for containers running in Kubernetes environments. Each approach has benefits and downsides.

Option 1 DaemonSet
A DaemonSet deployment guarantees that the specified pod runs on all of the nodes that are part of the Kubernetes cluster. Benefits include ec2 autoscaling scenarios, node metric insight and node log collection. One of the biggest downsides is that DaemonSets will not work in managed container environments (like AWS Fargate).

Option 2 Sidecar
A sidecar deployment injects itself into each pod on the host. This is different because a DaemonSet deployment has a single deployment per host, regardless of the number of pods running on it. With the sidecar approach, injecting into every pod can cause additional overhead.

There was no common deployment theme. The evaluated endpoint vendors employed both approaches, with one even using a DaemonSet with a sidecar. Interestingly, each endpoint vendor evaluated required Helm for deployment; Palo Alto Networks has it as an option, but it isn’t necessary.

Policy Differences

The policy differences between the runtime protection offered by endpoint products versus Prisma Cloud were quite obvious. For example, Optiv found that policies to prevent containers from running as root or with certain allowed capabilities were nonexistent in the endpoint security products tested. The endpoint security products have largely repurposed their existing Linux policies for container environments.

One product evaluated had limited prevention settings for their Linux policy. These preventions were repurposed for their container protections. Attempting to use minimal Linux preventions in container environments only further highlights the endpoint industry’s avoidance of Linux security. Legacy-focused policies don’t always meet the security runtime requirements necessary to compete with advanced attacks in container environments. This high-level evaluation also didn’t dive into infrastructure policy checks, which would only further highlight the problems with an endpoint approach to container security.

The endpoint products evaluated offered the ability to create custom rules for Linux systems, but there are limitations when attempting to port these over to a container environment. For example, these custom policies couldn’t be made namespace-specific. While there are many elements of endpoint security in containerized environments, treating containers as just another endpoint is the wrong approach.

Runtime Protection/Visibility

Initial Execution
Prisma Cloud and one of the endpoint products evaluated detected the malicious elf executing within the pod at runtime. A second endpoint product logged the commands but didn’t alert on the file execution or the c2 connection. We were able to locate the reverse_shell.elf execution along with the ip address of the c2 server in the logs across all the products tested.

Image

Image: Prisma Cloud reverse_shell.elf detection.

Drive Mount
As part of the scenario, the attacker mounts the ec2 node root drive as a temporary drive within the container. None of the products evaluated created an alert when this happened, but the commands were recorded in the logs of each product tested. It is also important to understand that custom detections can be created within each of the products to alert on this behavior.

File Copy and Crontab Edit
Once c2 was established, and the ec2 node’s root drive was mounted, the attack scenario copied the elf file to /bin directory. At this time the contab file of the ec2 node was edited. This edit happened through the container entrypoint to add a cronjob that executed the file as root every three minutes, offering the adversary persistence to the ec2 node. None of the products alerted to the file copy or that the crontab file was edited. Optiv was able to discover that the actions took place, along with the ip address of the c2 server in the logs for each product.

Note: After each product evaluation was completed Optiv shared the findings with the vendor. If any deployment or policy misconfigurations were made during the initial round of testing Optiv worked with each of the vendors to make corrections and retest.

Additional Findings

The endpoint products we tested offered no visual depiction of cluster nodes, namespaces or the relationship between pods within a namespace. Again, this view was only found in Prisma Cloud. Optiv views this graphical depiction as a valuable resource.

Image

Image: Prisma Cloud Radars/Container view of Optiv’s sock-shop namespace.

We were also surprised to discover that each endpoint product lacked a default timeline view of container events – an additional timesaver when investigating a compelling event. Instead, a specific log search would need to be performed to produce a similar view.

Image

Image: Prisma Cloud forensic view of a single lab node.

Prisma was also the only solution to include an ATT&CK for Containers alignment. This demonstrates more alignment with container security compared to endpoint security vendors who have entered the container security space.

Image

Image: Prisma Cloud ATT&CK for containers view.

Final Thoughts

While the endpoint market has begun providing products designed to provide insight and protection in the container space, there’s quite a bit of room for maturation before they can be considered container security solutions. Lack of policy separation between container and host, ability to create policies based on namespace and visualizations are just some of the areas that traditional endpoint vendors need to improve on.

For now, organizations grappling with container security should look toward those solutions that were purpose-built to protect container environments.

Want to know more? Talk to us.