7 Tips for SecOps Enabling Containers
We published this article at Black Hat 2016, as part of the Twistlock SecOps Pocket Guide to container security.
The practices documented here represent must-have security tips for organizations adopting container technologies. With dozens of companies using Twistlock to secure their containerized applications, we see requirements converging and top practices quickly emerging. This guide is what we distilled from these customer deployments, no hype – just practices that have worked.
The companies we work with in developing this guide include some of the most innovative companies in the world, largest of the enterprises, and operators of some of the largest cloud applications that ever existed.
We will continue to update this guide as the market and technology ecosystem evolve. As far as we know, this is the first guide that is based on actual deployments and actual use cases across hundreds of applications and thousands of servers.
Know The Source And Content Of Your Images
Do you know where your containers come from? Are your developers downloading container images and libraries from unknown and potentially harmful sources? Do the containers use third party library code that is obsolete or vulnerable?
There are utilities such as Docker Notary that you can use to check the authenticity of images, and you can use Docker Security Scan to verify whether images contain vulnerabilities. But as an enterprise, you must go a step further and establish policies as to where your images can come from, what it means to be a trusted image, and how to verify image trust.
Recommendation: Establish trusted sources of container images as a policy. Ensure that you have a runtime gate checker that only permits trusted images and containers to run on your hosts.
Eradicate Vulnerabilities From Your Containers
Container images are rarely built from scratch, they are typically built off some base image, which is itself built on top of other base images – consider a Node.js application built on top of Apache, which could be on top of an Ubuntu base image.
When a developer builds a container image, she typically grabs a base image and other layers from public third party sources. These images and libraries may contain obsolete or vulnerable code, thereby putting you application at risk.
An added complexity is that many existing vulnerability-scanning tools may not work on containers, nor do they support container delivery workflows including registries and CI/CD pipelines.
In addition, you can’t simply scan for vulnerabilities – you must scan, manage vulnerability fixes, and enforce vulnerability-based policies – for instance, you may have a policy that stipulates no image with CVSS score 3 or above can be deployed into production.
Recommendation: Acquire a vulnerability management tool that can parse container image formats and detect the existence of vulnerable libraries inside a container image before it progresses to production. Employ an enforcement function in runtime to ensure that vulnerable images & containers are not deployed.
Harden Your Images, Containers, Daemons And Hosts
The center for Internet Security (CIS) has published a consensus Docker benchmark, which covers configuration and hardening guidelines for containers, images, and also for hosts that run containers.
For example, one of the best practices is to remove non-essential services from the production host to mitigate potential risks. Another example is restricting kernel capabilities within containers. A recommended practice is to audit kernel capabilities associated with your containers and remove the unnecessary ones.
It is imperative that you not only follow but also enforce the hardening practices to mitigate runtime risks. Avoid manual verifications and checks, as that can be labor intensive.
Recommendation: Automate Docker bench hardening checks and enforcement. Make those practices part of your essential development and deployment processes prior to live production.
Leverage Your CI/CD Pipeline Tools
If you use containers, you are probably using Continuous Integration (CI) or Continuous Delivery (CD) pipeline tools. Popular ones include Jenkins, TeamCity, and CircleCI.
The best place to detect and fix security vulnerabilities is during development and as part of the CI/CD workflow. A possible implementation would be for CI/CD tool to initiate security scanning whenever a new image is constructed and consumes the results in the native CI/CD console.
Also important is that the integration should be able to fail builds and force bug fixes before the image progresses in the pipeline.
Recommendation: Ensure your container vulnerability-scanning tool can integrate easily with your CI/CD pipeline tools for both vulnerability detection and also build management.
Enforce Role Segregation & Access Control For Your Docker Environment
Many organizations, when using containers like Docker in production, have a hard time enforcing role-based policies in the environment. For starters, Docker requires root privilege for accessing any Docker command, hence rendering enterprise role-based policies ineffective.
Docker has since amended its privilege management framework with more fine-grained access control capabilities – Twistlock actually contributed that part of the code. To take advantage of that, you will need to write an authorization plugin and add the plugin to the Docker daemon configuration or leverage a commercial tool like Twistlock that provides this capability.
Twistlock has published an open source authorization plugin. You can access it at https://github.com/twistlock/authz
Recommendation: Get familiar with the authorization plugin, but also consider using an access control tool that offer integration with enterprise directories, fine-grained access management, and also logging and auditing for all your Docker daemon accesses.
Automate Anomaly Detection And Threat Defense In Container Runtime
Containers should be minimal, declarative, and immutable. Those characteristics mean that it is actually possible to build a reliable baseline for the containerized application. Using this baseline in runtime you can detect anomalies and active threats much more accurately than with monolithic applications that change frequently.
However, building the baseline is not a trivial task, especially when you have many containers spinning up and down dynamically.
Automating the baselining process, the detection actions, as well as the enforcement, is the only way to scale up runtime security.
Recommendation: Establish automatic behavior profile generation and anomaly detection functions for containers. Ensure that the tool you use do not rely on manual actions, and can correlate, manage, and automate the different controls to give you central visibility, detection, and response.
Perform Regular Audits To Prevent Image And Container Sprawl
Containers are flexible and easy to use, but that also means it’s easy to end up with many instances of containers and images, some of which maybe obsolete and risky. It is recommended that you perform regular audits to identify no-longer-in-use containers and images and eliminate them from your systems.
Docker provides commands for you to determine the number of containers and images on a particular host, and you can remove them if sprawling starts to consume too much system resources and threatens system consistency.
Recommendation: Automate image and container audit and management workflows to eliminate unused images and containers from your registries and hosts.
Twistlock provides dev-to-production security for containerized applications. Our container security suite offers capabilities for SecOps to meet the practices detailed here. To see how Twistlock works, request a free demo of the Twistlock enterprise suite here.
- Container Security