Introduction

Last week the Kubernetes Security team released CVE-2019-11248 which is about the debugging endpoint “/debug/pprof” being exposed over the unauthenticated Kubelet healthz port.
In this short post I will explain the problem and its cause.

pprof

pprof is a tool for visualization and analysis of profiling data and is part of the Go programming language. pprof reads a collection of profiling samples in profile.proto format and generates reports to visualize and help analyze the data. It can generate both text and graphical reports. It is incredibly powerful: it’s great for debugging a running server. It’s also equally easy to accidentally expose your debugging information.
It is extremely easy to use pprof, all you need to do is have a web server with a single import line of “net/http/pprof“.

What can pprof do?

All kinds of profiling capabilities:

  • Trace all current goroutines
  • Get x seconds CPU profile
  • Sample all heap allocations
  • Get stack traces of holders of contended mutexes
  • Get stack traces that led to the creation of new OS threads
  • Get stack traces that led to blocking on synchronization primitives
  • and more.

    How can pprof be abused?

    What happens if pprof is left exposed in a production environment, and it is accessible to potentially malicious actors?
    Information leaked by pprof alone is limited, an attacker may be able to learn about the target binary internals such as its functions or stack traces which may be helpful when dealing with an unfamiliar binary. Other than that, it may be possible to result in DoS of the target through pprof.
    Profiling is a very CPU hungry task. In my testing I was able to get to a CPU usage of over 20% by a single thread sending profiling requests, so theoretically, it may be possible to craft a DoS attack using this technique. In the case of the open source Kubernetes, DoS is probably the greatest risk imposed by this vulnerability.

    The Problem

    The kubelet is the primary “node agent” that runs on each node in a Kubernetes cluster. By default, kubelet binds the healthz port to localhost:10248. Kubernetes uses the healthz port to check the node’s health status.
    To understand the problem we first need to discuss muxes.

    mux

    ServeMux is an HTTP request multiplexer. It matches the URL of each incoming request against a list of registered patterns and calls the handler for the pattern that most closely matches the URL. Basically it maps routes to functions.

    Back to the problem

    Up until now the healthz port was using the default mux, meaning every single route that was registered anywhere in the code (with the default mux, of-course) could be served using this “http.ListenAndServe” object. Importing “net/http/pprof” for profiling registers all the profiling routes to, you guessed correctly, the default mux. By using the health feature, which is enabled by default, every profiling route (that was registered somewhere in the code by that one import line) is exposed through this port.
    Nevertheless, even if a pod was compromised the healthz port is not exposed by default because the pod and the kubelet won’t be in the same network namespace. Unless they would be.
    There are 2 scenarios in which one will be vulnerable: either the healthz port is binded to 0.0.0.0:10248 (using –healthz-bind-address) which means anyone can connect to it, or, the pod was created with the hostNetwork: true flag, which tells kubelet to create the new pod with the same network namespace of the host.
    Both are not likely to happen and that’s one of the reasons this vulnerability is low severity.

    The Fix

    The fix simply changes the healthz mux. It was using the default mux which had all the profiling routes (and many other things) and now it uses a new specific one that exposes only the “/healthz” route.
    The fixed Kubernetes versions are 1.15.0+, 1.14.4+, 1.13.8+ or 1.12.10+.