Today I will discuss the process of securely deploying and using Kubernetes Helm, a package manager for  your environment.

Helm is a tool that streamlines installing and managing Kubernetes applications. Think of it like apt/yum/homebrew for Kubernetes. Helm is made of two components: a client – Helm which runs on your laptop, CI/CD, or wherever you want it to run it, and a server – Tiller which runs inside your Kubernetes cluster and manages installations of your Helm packages.

Helm packages are sets of pre-configured Kubernetes resources that are called charts. These  resources  are basically collections of files that the package consists of. A single chart might be used to deploy something simple, (like a memcached pod), or something complex, (like a full web app stack with HTTP servers, databases, caches,and more). You can explore other examples here and at kubeapps.com.

With the Helm client, you can browse package repositories (containing published charts) and deploy those charts on your Kubernetes cluster. Helm will pull the Chart and by working together with Tiller create a Release (an instance of a Chart). The release is a collection of Kubernetes resources deployed to the cluster using Helm.

From a security perspective there are a couple of problems with the default deployment of Helm:

For starters, there are no authorization or authentication mechanisms that you can configure for your charts repository, in other words anyone with access to the repository will have R/W permissions on it and will be able to download or modify charts as he wishes. You can read the open discussion on this issue on Helm’s github.

The second more severe issue is no TLS by default, which makes all the communication traffic between the client and repositories completely open to an attacker. Don’t get me wrong, though: Helm supports TLS, and most of the Chart repositories I observed are running on TLS including most of the charts from kubeapps.com.

From Helm Documentation on chart repositories:

A chart repository is an HTTP server that houses an index.yaml file and optionally some packaged charts. When you’re ready to share your charts, the preferred way to do so is by uploading them to a chart repository.

Note: For Helm 2.0.0, chart repositories do not have any intrinsic authentication. There is an issue tracking progress in GitHub.

The last, third, issue is a flawed package verification process that can be subverted with a simple Man-in-the-Middle attack

Combining these three can lead to devastating scenarios which I will leave to your imagination for now, as this post is meant to focus on securing your Helm repository rather than attacking it.

We will secure our Helm repository and Helm related communication solely with the help of TLS by incorporating client side certificates for authentication and standard TLS for over-the-wire encryption and integrity verification.

Note: the process is dependant on the web server or the cloud infrastructure that you are using for hosting your charts repositories. For example, the configuration for NGINX or Apache web servers, or GCP / AWS might be different.

So let’s get to it!

I will demonstrate the setup with a simple nginx server:

Creating the certificates

NOTE: This is just a quick sample of creating TLS certs. Please obtain your certificates from a trusted authority for production environments.

#create CA ROOT cert
openssl genrsa -des3 -out ca.key 4096   #
openssl req -new -x509 -days 365 -key ca.key -out ca.crt [1]
#create client key & CSR
openssl genrsa -des3 -out client.key 4096     #  [2]
openssl req -new -key client.key -out client.csr
#self-signed
openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt      # [3]

Configuring the web server

After we installed our nginx server that will be used as a chart repo, we will need to enable client side certificates by editing the nginx config file which is usually /etc/nginx/conf.d/default.conf

The default config file looks like this:

server {
listen    443;
server_name  localhost;

location / {
root   /usr/share/nginx/html;
index  index.html index.htm;
}

error_page   500 502 503 504  /50x.html;
location = /50x.html {
root   /usr/share/nginx/html;
}

}

All we need to do is to add 2 lines right beneath the server_name:

ssl_client_certificate /path/to/ca.crt;
ssl_verify_client on;

Note that the ca.crt in nginx’s config must be the same CA that we used previously and marked by [1] in the client generation process.

Using Helm with the certificate

Now all that is left is to supply a client certificate that Helm will use in order to connect to our repo!

This is done when adding a repo to Helm like this:

Helm repo add --key-file client.key --cert-file client.crt [NAME]
[URL]

Where [NAME] is the repository name and [URL] is the repository location.

The client.key and client.crt files are generated in the certificate creation process and marked in the text above by [2] & [3] respectively.

Verifying charts integrity

Now we have a secured Helm repository that no one will be able to connect to unless they have a matching certificate. No one will also be able to sniff the traffic towards and from the repository.

All that’s left to do is to verify the integrity of our charts, and as we already implemented certificates we know that a MitM attack will not be possible, so the flawed mechanism is not a risk anymore! we can simply add the --verify flag to our Helm commands.
And that’s it: We have one less thing to worry about!