Palo Alto Networks Prisma Compute can help minimize the risk of deploying untrusted images. Rob Brooks explains how.

In the previous installment of this Kubernetes attack blog series, I looked at how deploying an untrusted image to a Kubernetes cluster could lead to a breach. For this blog, we'll follow the same steps taken in the previous blog, and I'll show how using Palo Alto Networks Prisma Compute could help minimize the risk of deploying untrusted images.

To address the risks of deploying untrusted Docker images, one of the things we can do is to configure controls to detect vulnerable Docker images and prevent the deployment of these untrusted or vulnerable images. With Palo Alto Networks Prisma Compute, we can integrate the platform with Docker Hub to scan images for vulnerabilities. With our Docker Hub integration, Palo Alto's Compute returns results that show us the vulnerabilities in the Ubuntu image I previously used. By scanning the image, Prisma detected vulnerabilities in the image's open source components as well as the Monero crypto-mining malware XMRig and a purposeful misconfiguration.

Figure 1: Result of a Registry Scan - Vulnerabilities (Compute -> Monitor -> Vulnerabilities - Images -> Registries -> Vulnerabilities

Figure 2: Result of a Registry Scan - Vulnerabilities (Compute -> Monitor -> Vulnerabilities - Images -> Registries -> Compliance)

With this capability enabled, a step we could take to minimize our risk would be to restrict an image from being deployed if certain factors are met, which could be vulnerabilities and compliance or even a scope definition preventing the deployment to a specific cluster.

To prevent vulnerable images from being deployed to a cluster, Prisma has a concept of "Trusted Images" and "Trusted Groups", both of which can be found at Defend -> Compliance.

Figure 3: Trusted Image Rules (Compute -> Defend -> Compliance -> Trusted Images -> Policy)

Figure 4: Creating/Editing a Trusted Image Policy (Detailed)

The Trusted Images feature allows Prisma users to create sets of rules, e.g., Trusted Image Policy, that control which image sources are trusted and where those images are allowed to run in the cluster. In the Trust Image rules, Prisma users can define the scope of the cluster and where the image is allowed (or denied) deployment to. Trusted Image actions can be allow, alert, or block.

Prisma users can further tailor the rules for deployment through the use of Trust Groups. Trust Groups are used by the Trusted Images Policy to define a scope of objects, including images and registries. Using Trusted Images and Trusted Groups, Prisma users can define what registries are trusted and untrusted and the scope of each cluster. By defining the scope of the cluster, Prisma users can tailor fine-grained deployment rules for their requirements. For example, users can tie specific trusted image registries to specific clusters.

Figure 5: Trusted Image Group (Compute -> Defend -> Compliance -> Trusted Images -> Trust Groups)

Figure 6: Creating the Trusted Image Group (Compute -> Defend -> Compliance -> Trusted Images -> Trust Groups)

Another control that addresses risk around the deployment of vulnerable images is the ability to prevent an image from being deployed to a cluster, based on vulnerabilities detected in vulnerability scans. Prisma can help with this through the use of Compliance policies.

In Prisma, policies can be created and enforced on nodes in the cluster where if a vulnerability is detected in an image, the policy can trigger an alert or block the deployment, depending on the severity of the vulnerability and the threshold desired between alert and failure. In this scenario, DevOps admins could configure a policy where images with low to medium vulnerabilities would trigger an alert but be allowed to deploy and would fail to deploy when a high vulnerability was detected.

There are two options for vulnerability scanning of images in Prisma: Deployed and CI. Deployed allows for more granular scope to be defined. Scope parameters for deployed include containers, images, on hosts, labels, namespaces and account IDs. The second option, CI, offers the scope parameters of images and labels.

Figure 7: Image Policy (Compute -> Defend -> Vulnerabilities -> Images -> CI)

The concept of vulnerability scanning and remediation of images before deployment, combined with restricting the deployment of trusted images to specific clusters, is something that I will be covering in more detail in a future blog.

While these policies would have prevented my malicious image from being deployed, we can also look at some other controls from Prisma that could be used to minimize risk.

The second part of the attack focused on running untrusted code within a container. Fortunately, Prisma has runtime capabilities to help counter the risks from processes and commands associated with malicious binaries and/or untrusted code.

Prisma has the capability to learn container behavior and respond when a container exhibits behavior outside of the normal baseline. Prisma's runtime enforcement capability learns and monitors processes, networking and the file system for anomalies. System events that fall outside of the container baseline can result in either an alert or block, depending on the setting. Each sub-category within the policy (e.g., general, processes, networking, file system) also includes the ability to define a specific scope for the policy.

Figure 8: Container Policy (Compute -> Defend -> Runtime -> Container Policy)

In the general setting for container runtime policies, basic monitoring includes Prisma Cloud Advanced Threat Protection, which is a collection of 30+ different threat feeds that Prisma ingests and utilizes. Other general settings include basic Kubernetes attacks and suspicious queries to cloud provider APIs, which can be turned on if required (currently defaults to off).

Figure 9: Container Policy (Compute -> Defend -> Runtime -> Container Policy -> General)

Default rules for Process Monitoring include monitoring processes started from modified binaries, crypto miners, processes used for lateral movement and child processes targeted by unrecognized parents. Users can also specify a name of a process to deny. Actions (or effects as they're referred to in Prisma Compute) for process monitoring can be allowed and denied (alert and block).

Figure 10: Container Policy (Compute -> Defend -> Runtime -> Container Policy -> Processes)

For Networking, Prisma's learning engine can learn listening ports, outbound ports and outbound IPs. Default rules for container networking include networking activity from modified binaries, port scanning, raw socket detection, listening ports, outbound internet ports and outbound IPs. Listening ports, outbound internet ports and outbound IPs can be added manually as needed. Effects for networking monitoring can be allowed, denied & fallback (alert and block).

Figure 11: Container Policy (Compute -> Defend -> Runtime -> Container Policy -> Networking)

File Integrity Monitoring is another feature available in Prisma's Container runtime protection that can help minimize risks to containers. In this setting, the scope of the runtime policy can be set to containers, images, hosts, labels, namespaces and cloud account IDs. Prisma can learn the container's file system and respond accordingly, based on the configured action/effect settings. In the event of denied & fallback event, Prisma can either alert, block or prevent. With File Integrity Monitoring, Prisma can look for changes to binaries and certificates, changes to SSH and other configuration files, as well as trigger a denied and fallback effect, based on a custom file path configured.

Figure 12: Container Policy (Compute -> Defend -> Runtime -> Container Policy -> File System)

For the third part of the attack, we looked at the persistence and the issue of the backdoored image running in the cluster. We also briefly touched on the permissions in the container and the potential to capture specific files (e.g., certificate authority, secrets) that the container uses to communicate with the Kubernetes API server.

The issue of risks posed by the malicious image was mostly addressed with the policies in the last section, but to improve the risk profile, we can also look at locking down access to secrets in the containers. Additionally, we can monitor and limit Docker commands from being used as well.

There are several ways that Docker secrets can be compromised. One way is that attackers can potentially execute Docker commands directly against the Docker daemon's local UNIX socket and, as a result, be able to expose the secrets.

With Prisma, access to secrets can be controlled and managed, based on a least-privilege basis. Prisma does this by enabling the Defender agent to intercept requests to the Docker socket, and controlling which Docker commands are permitted and which are denied. The Docker rule/policy (e.g., Defend -> Access -> Docker) allows Prisma admins to tailor which users can execute specific Docker commands. In this scenario, all Docker commands must be processed through the Defender agent. An example of a Docker rule that can be enforced is to limit lower-level privileged users to monitoring commands such as docker ps and docker inspect, both of which are view-only permissions. Prisma enforces the least privilege concepts for Docker commands through directory services (Active Directory, OpenLDAP) or SAML Identity Providers integration.

Secrets can be stored in Prisma and the corresponding environment variables injected into the container as requested. Prisma supports built-in orchestrator secrets stores such as Kubernetes and Docker Swarm and integrates with HashiCorp Vault and CyberArk Enterprise Password Vault.

Prisma users can enable the proxying feature of the Defender by enabling it as a TCP Listener. It should be noted that this capability is not available for Defender agents that have been provisioned as a Daemonset.

Figure 13: Prisma Defender agent with TCP Listener setting (Compute -> Manage -> Defenders)

Figure 14: Host compliance policy/rule showing Docker settings

Figure 15: Docker secrets policy/rule (Defend -> Access -> Secrets)

Figure 16: Docker Actions Rule (Defend -> Access -> Docker)

Addressing the container breakout exploit in the last blog, Prisma can protect against this threat through the use of its machine learning capability, which Palo Alto Networks refers to as the "4D Model." By default, Prisma implements its runtime machine learning in all containers and hosts where the Defender agent is deployed.

To develop a working runtime container or host model, Prisma begins the "learning mode" upon the Defender agent's deployment. The learning process takes 48 hours, which takes into account activities such as networking, process execution, file system activity and command execution. In the event that a process being blocked needs to be allowed, Prisma users can either add an "Allow" exception to the runtime rule or re-run the learning process.

Figure 17: Learned Runtime Container Model (Monitor -> Runtime -> Container Models)

Once the model's baseline is established, Prisma can enforce restrictions on anomalous behavior through the use of runtime policies/rules. With these rules, Prisma users can configure how Prisma responds when activity outside of the model is detected. The granularity of the scope feature allows rules to be applied based on container, image, host, label, namespace and cloud account ID.

The runtime container rule response can be configured to either alert, prevent or block on detection of activity that is not within the runtime model. The alert setting registers an alert in Prisma but allows the activity to continue. The prevent setting terminates the anomalous process but allows the container to continue running and the block setting prevents the container from running altogether. Prisma users can be granular with different types of actions for different aspects of the rule, depending on the requirements. For example, a Prisma user could configure a rule that alerted on all process execution violations while blocking all anomalous network activity. This rule could be applied to any combination of hosts, containers or namespaces.

For our testing, we configured a runtime container rule that would detect the container breakout exploit by detecting processes that were not included in the runtime model. As expected, when inside of the container, the initial command for the exploit fails. As can be seen in the code below, when attempting to exec into the Docker image, commands associated with a typical user login are also prevented from executing.

Figure 18: Unauthorized processes attempting to run in a container

Figure 19: Runtime Container Alert (Compute -> Monitor -> Events -> Container Audits)

Hopefully, this blog series was helpful in showing how DevOps/DevSecOps tools like Gitlab, Terraform and Palo Alto Networks Prisma Compute can be integrated to improve the efficiency and security posture of containers and container orchestration using infrastructure as code.

In a future post, I will be looking at how processes can be developed for vulnerability scanning to build, maintain and enforce usage of trusted container image repositories.