Last week the Kubernetes Security team released CVE-2019-11248 which is about the debugging endpoint “/debug/pprof” being exposed over the unauthenticated Kubelet healthz port.
In this short post I will explain the problem and its cause.
pprof is a tool for visualization and analysis of profiling data and is part of the Go programming language. pprof reads a collection of profiling samples in profile.proto format and generates reports to visualize and help analyze the data. It can generate both text and graphical reports. It is incredibly powerful: it’s great for debugging a running server. It’s also equally easy to accidentally expose your debugging information.
It is extremely easy to use pprof, all you need to do is have a web server with a single import line of “net/http/pprof“.
What can pprof do?
All kinds of profiling capabilities:
How can pprof be abused?
What happens if pprof is left exposed in a production environment, and it is accessible to potentially malicious actors?
Information leaked by pprof alone is limited, an attacker may be able to learn about the target binary internals such as its functions or stack traces which may be helpful when dealing with an unfamiliar binary. Other than that, it may be possible to result in DoS of the target through pprof.
Profiling is a very CPU hungry task. In my testing I was able to get to a CPU usage of over 20% by a single thread sending profiling requests, so theoretically, it may be possible to craft a DoS attack using this technique. In the case of the open source Kubernetes, DoS is probably the greatest risk imposed by this vulnerability.
The kubelet is the primary “node agent” that runs on each node in a Kubernetes cluster. By default, kubelet binds the healthz port to localhost:10248. Kubernetes uses the healthz port to check the node’s health status.
To understand the problem we first need to discuss muxes.
ServeMux is an HTTP request multiplexer. It matches the URL of each incoming request against a list of registered patterns and calls the handler for the pattern that most closely matches the URL. Basically it maps routes to functions.
Back to the problem
Up until now the healthz port was using the default mux, meaning every single route that was registered anywhere in the code (with the default mux, of-course) could be served using this “http.ListenAndServe” object. Importing “net/http/pprof” for profiling registers all the profiling routes to, you guessed correctly, the default mux. By using the health feature, which is enabled by default, every profiling route (that was registered somewhere in the code by that one import line) is exposed through this port.
Nevertheless, even if a pod was compromised the healthz port is not exposed by default because the pod and the kubelet won’t be in the same network namespace. Unless they would be.
There are 2 scenarios in which one will be vulnerable: either the healthz port is binded to 0.0.0.0:10248 (using –healthz-bind-address) which means anyone can connect to it, or, the pod was created with the hostNetwork: true flag, which tells kubelet to create the new pod with the same network namespace of the host.
Both are not likely to happen and that’s one of the reasons this vulnerability is low severity.
The fix simply changes the healthz mux. It was using the default mux which had all the profiling routes (and many other things) and now it uses a new specific one that exposes only the “/healthz” route.
The fixed Kubernetes versions are 1.15.0+, 1.14.4+, 1.13.8+ or 1.12.10+.