Recently 2 vulnerabilities in Envoy <= 1.9 have been disclosed and caused a lot of chaos because of their direct impact on Istio. In this article, I would like to bring order to the chaos and shed more light on these two issues and how they were fixed.
What is Istio?
As cloud native architectures have become used at large scale its complexity has brought new problems and security issues that developers have never encountered before.
In order to tackle many of those problems, Lyft, Google, and IBM decided to collaborate and develop the Istio project.
In order to understand how Istio works, we need to go over two fundamental concepts
- Service mesh – A service mesh is a network of containers that act as an infrastructure layer built to handle communication within a cluster.
It can monitor the traffic, set restrictions and policies between microservices, optimize communication, and much more.
- Envoy – Envoy in an open source CNCF project created by Lyft.
As its name implies, it acts as a proxy server which runs alongside every pod.
Istio – Istio is an open-source service mesh, which provides monitoring, tracing, access control, security and more. Istio uses Envoy as a sidecar proxy, which means that Istio runs an Envoy proxy server on each pod. With the help of a preconfigured iptables rules, all communication to and from the pod is redirected through the Envoy server, which makes it the “gateway of the pod”.
With multiple pods, this architecture becomes a service mesh and all of the communication in the cluster is transfer only between Envoys, which are managed by Istio.
Security on Istio
Istio has a lot of features like tracing, monitoring, and logging but in this article, we
will primarily focus on its security features which are:
Traffic encryption: The traffic inside the service mesh can be encrypted.
Identity-based authentication: At the beginning of a service-to-service communication, the two parties must exchange credentials with their identity information for mutual authentication purposes.
Authorization: Which is built on Kubernetes role-based access control (RBAC) and provides access control for the services in the mesh based on multiple attributes such as IP, port, or even HTTP headers.
More technical information can be found at – https://github.com/envoyproxy/envoy/issues/6434.
Another attribute that Envoy can use in order to set access control rules is URL.
For example, we can set a policy that denies access to /secret.txt on our HTTP web server.
The vulnerability that was discovered was that Envoy didn’t sanitize URLs, which means that if an attacker will try to access the HTTP server with the URL /public/../secret.txt, then Envoy will forward the request to the HTTP server who will return /secret.txt.
More technical information can be found at – https://github.com/envoyproxy/envoy/issues/6435.
There is a big danger in those vulnerabilities because by exploiting them, an attacker can bypass some access control policies and access places he should not be able to.
This can compromise sensitive data or even help him to escalate his attack.
Consider the following two scenarios, for example:
- From inside of the cluster: If an attacker has already taken over one container in the cluster, he would be able to access pages on a server within the cluster which he is normally not allowed to. He could potentially get sensitive information or, worse, do things like interacting with a forbidden administrative API endpoint.
- From outside of the cluster: An attacker would be able to access normally forbidden URLs on a web server that is externally published through the ingress controller.
How to protect your cluster from those threats?
Envoy released version 1.9.1, which includes security fixes for those problems, and simultaneously Istio released 2 new versions with the security fixes which includes the new Envoy version.
For Istio 1.1.x deployments: Update to Istio 1.1.2.
For Istio 1.0.x deployments: Update to Istio 1.0.7.
Unpacking Envoy Vulnerabilities (CVE-2019-9900 and CVE-2019-9901) and How it Impacts IstioRead the Blog
Disclosing a directory traversal vulnerability in Kubernetes copy – CVE-2019-1002101Read the Blog
Breaking out of Docker via runC – Explaining CVE-2019-5736Read the Blog
T19 Challenge – Twistlock Lab’s first security challenge summary and solutionsRead the Blog