Right before the holidays, a vulnerability was fixed in the Cisco CloudCenter Orchestrator. There have been several news articles about the vulnerability and, of course, Cisco’s own security advisory is the authoritative source for details on the problem and to download their fix. In this blog, though, I want to talk specifically about how Twistlock helped protect a few customers who are real world users of Cisco CloudCenter CVE and how our compliance features provide a powerful way to not just check for proper security settings in your container environments, but to actively enforce them.
As described in Cisco’s security advisory, the core of the problem is that the Docker Engine is configured to bind to TCP 2375, without TLS being required and without any proxy in front of it. While it can be completely desirable and safe to configure dockerd to bind to a TCP socket, it should only be done over TLS, as described in the Docker docs.
If you configure dockerd to listen on a TCP port without TLS, anything that can talk to the socket can run any Docker commands and can trivially escalate to root on the host, such as by running a privileged container. That configuration is essentially what the problem is in Cisco CloudCenter CVE. For example, if a bad guy can talk to the socket, he could run something like:
<span style="display: inline-block; font-weight: 400;">docker -H vulnerable-host:2375 run --privileged --entrypoint /bin/bash --pid=host --net=host evil/image </span>
And he’d be instantly acting as part of the host OS.
So what can Twistlock do to protect customers from risks like this? Recall that one of our core feature areas is compliance, which is a policy engine for describing and enforcing configuration settings across your environment. Compliance includes all 90+ recommendations in the Center for Internet Security’s Docker Benchmark and is extensible using NIST’s XCCDF language (part of the SCAP family). This enables Twistlock to check for and report on settings that span images, containers, Docker Engines, and host operating systems and to enforce compliance with those settings. For example, with Twistlock you can create a compliance rule that prohibits containers from being deployed to non-compliant hosts. It’s this compliance enforcement that we used to help our customers protect themselves from the vulnerability.
Soon after Cisco CloudCenter CVE became public, we heard questions from a few of our customers that rely on CloudCenter about how they could protect themselves. Our answer was pretty simple: utilize compliance policies to ensure that dockerd requires TLS (and prevent the deployment of containers to hosts that are not compliant) and to block containers that run as privileged.
These are both simple settings to configure in the policy. To block deployments to hosts not running the daemon with TLS, enable compliance rule #26 (which corresponds to CIS recommendation 2.6):
To prevent execution of privileged containers, enable rule #54 (which corresponds to CIS recommendation 5.4):
If someone attempts to run a container in privileged mode, here’s the detailed ‘access denied’ response they see back from Twistlock when we block it:
With these compliance rules in place, we were able to help customers remediate some of the risk of this vulnerability, even before they were able to fully deploy the patch in their environment.
Of course, compliance is just a part of the defense in depth that Twistlock provides for containers. We also helped customers analyze runtime defense audit data to track (and later implement rules to block) suspicious activities that occurred in these environments, potentially after the exploit had been used to escalate privilege. One clear example of this was detecting the usage of setns system calls that change the namespaces associated with a thread and are often used to disguise attacks. Here’s a real world example of an alert we logged from one of these events (customer specific details redacted):
In summary, helping customers respond to Cisco CloudCenter CVE really illustrated the defense in depth Twistlock provides, not just in reactive monitoring, but in active enforcement and protection. We’d love to show you how we can help protect your containers, too, so please ping us if you’d like to see a demo or do an eval.
Follow us on Twitter
Follow us on Twitter for real time updates on the cloud native ecosystem, Twistlock product, and cloud native security threats.
Taming the Complexity Monster in a Cloud-Native WorldRead the Blog
How My Company (Teckro) Uses ContainersRead the Blog
Mitigating CVE-2019-5736 Impacting RunC and DockerRead the Blog
From Agile to DevSecOps and DevOps SecurityRead the Blog
What’s Next for Cloud-Native Infrastructure Technology?Read the Blog