Kubernetes Secrets Encryption

Overview

Kubernetes is an open-source framework for automatically deploying, scaling, and managing  containerized applications. etcd is a key-value store that is used by Kubernetes. etcd holds data and configuration which required to keep persistency of the cluster. etcd stores it’s data in plain text, if someone manages to connect to the etcd she can read all the important configuration and data of the cluster. Among these configurations there are Kubernetes secrets and API tokens.

Kubernetes introduced several new security features in the latest release. One of the new features is called ‘encryption at rest’; this feature allows you to encrypt etcd persistent storage.

In this post I will review the encryption at rest mechanism and explain how to keep secrets secured.

Encryption

The encryption is symmetric – the same key used for encryption is later used for decryption. The encryption/decryption takes place in kube-apiserver process, etcd isn’t aware of the encryption at all and treat the values as any other plain text value. The encryption keys are stored on disk on all the api-servers nodes.

Requirements

Check your Kubernetes version:

$ kubectl version --short
Client Version: v1.7.1
Server Version: v1.7.2

Make sure ‘server version’ is equal or greater than 1.7.0

Enabling Secrets Encryption

There are 3 types of encryption providers: aescbc, secretbox, and aesgcm. Kubernetes documentation recommends to use ‘aescbc’, which is basically AES symmetric encryption algorithm (previously known as Rijndael) together with CBC block cipher mode. aesgcm uses AES encryption algorithm as aescbc, but uses GCM mode of operation which is prone to birthday attack. To avoid this, NIST Special Publication 800-38D (section 8.3) suggests to rotate the encryption key every 2^32 writes. Secretbox uses salsa20 encryption algorithm which is a relatively new stream cipher with Poly1305 MAC.

There are 2 steps to enable encryption:

1. Create a configuration file dedicated to encryption
2. Tell kube-apiserver about the configuration file by adding an argument to it

Creating a configuration file

Lets create a configuration file at /var/kubernetes/secrets.conf with the following contents:

kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: 9cI2cvIw+wTxL0DqTIom9Ka+TUJoVJ16tl0lQ795K8I=
- identity: {}

This configuration specifies an encryption key named ‘key1’ with ‘aescbc’ encryption. The encryption key 9cI2cvIw+wTxL0DqTIom9Ka+TUJoVJ16tl0lQ795K8I= is base64’ed encoded string of 32 bytes which are 256 bits. When using aescbc encryption kubernetes supports only 256 bit keys. To generate your own key you can use:
head -c 32 /dev/urandom | base64 -i -

Let’s make sure nobody can read this file because it contains the symmetric encryption key:

$ sudo chmod 600 /etc/kubernetes/secrets.conf
$ ls -la
total 60
drwxr-xr-x   4 root root  4096 Jul 18 14:33 .
drwxr-xr-x 149 root root 12288 Jul 18 16:48 ..
-rw-------   1 root root  5449 Jul 18 12:09 admin.conf
-rw-------   1 root root  5485 Jul 18 12:09 controller-manager.conf
-rw-------   1 root root  5497 Jul 18 12:09 kubelet.conf
drwxr-xr-x   2 root root  4096 Jul 18 14:33 manifests
drwxr-xr-x   2 root root  4096 Jul 18 12:09 pki
-rw-------   1 root root  5437 Jul 18 12:09 scheduler.conf
-rw-------   1 root root   224 Jul 18 12:19 secrets.conf

Telling kube-apiserver about our encryption configuration

Now we need to add the encryption argument to kube-apiserver. Our cluster was initialised with kubeadm tool, and kube-apiserver runs inside a container, so we need to edit /etc/kubernetes/manifests/kube-apiserver.yaml and add this argument:
--experimental-encryption-provider-config=/etc/kubernetes/secrets.conf

After editing the file, kube-apiserver container will be restarted automatically. Note that this may take a few minutes.

Now we can see that kube-apiserver is running with the new encryption argument:

$ ps -ef | grep kube-apiserver
root      2775  2757  1 Jul18 ?        00:08:57 kube-apiserver --requestheader-group-headers=X-Remote-Group --requestheader-allowed-names=front-proxy-client --service-cluster-ip-range=10.96.0.0/12 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --allow-privileged=true --experimental-bootstrap-token-auth=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-extra-headers-prefix=X-Remote-Extra- --service-account-key-file=/etc/kubernetes/pki/sa.pub --client-ca-file=/etc/kubernetes/pki/ca.crt --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --insecure-port=0 --requestheader-username-headers=X-Remote-User --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --secure-port=6443 --authorization-mode=Node,RBAC --advertise-address=192.168.1.155 --etcd-servers=http://127.0.0.1:2379 --experimental-encryption-provider-config=/etc/kubernetes/secrets.conf

Creating Secrets

The kubectl tool let us create new secrets. for example, to create a simple username & password secret:

$ kubectl create secret generic personal-secret --from-literal=username=josh --from-literal=password=supser-strong-pass
secret "personal-secret" created

kubectl sends our command to kube-apiserver using a secure TLS connection, which in turn talks with the etcd instance.

Checking if secrets are encrypted

To see what is hosted inside etcd we will use etcdctl. etcd container contains etcdctl inside, we will use ‘docker exec’ to run etcdctl from within the container and get our new key:

$ docker ps | grep etcd
91cfc070226c        gcr.io/google_containers/etcd-amd64                      "etcd --listen-cli..."   19 hours ago        Up 19 hours                                                                k8s_etcd_etcd-n_kube-system_9fb4ea9ba2043e46f75eec93827c4ce3_2
09ed5a2f62a0        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 22 hours ago        Up 22 hours                                                                  k8s_POD_etcd-n_kube-system_9fb4ea9ba2043e46f75eec93827c4ce3_1

$ docker exec -it 91cfc070226c /bin/sh -c ‘ETCDCTL_API=3 etcdctl get /registry/secrets/default/personal-secret | hexdump -C’
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 70 65 72 73 6f 6e  |s/default/person|
00000020  61 6c 2d 73 65 63 72 65  74 0a 6b 38 73 3a 65 6e  |al-secret.k8s:en|
00000030  63 3a 61 65 73 63 62 63  3a 76 31 3a 6b 65 79 31  |c:aescbc:v1:key1|
00000040  3a 3d f8 88 02 7a 9e b6  b5 29 a3 63 0d 9c 86 98  |:=…z…).c….|
00000050  b0 ab 78 39 10 bc ea 49  ac 28 d6 d0 71 c7 75 af  |..x9…I.(..q.u.|
00000060  5a 9e c7 b3 37 fb 31 d2  2b 42 1b f0 e8 89 95 b7  |Z…7.1.+B……|
00000070  be b2 99 6a e7 9e 0c 7b  0f 83 5a a6 e5 2f 15 85  |…j…{..Z../..|
00000080  75 05 59 67 19 f8 7f c1  2c c9 1a 39 cd cc 2b 76  |u.Yg….,..9..+v|
00000090  19 da bc 56 fb 0c b3 1a  2d 32 9c 0d f4 cf 8d ca  |…V….-2……|
000000a0  90 f0 ff 9a af 88 72 2a  8d 6f 85 ae d6 11 3b 73  |……r*.o….;s|
000000b0  c8 79 f5 35 10 bc 6d 2c  31 dd da 57 f9 7f 82 3c  |.y.5..m,1..W…<|
000000c0  67 db 85 2d 33 a8 55 d0  00 a5 2d 37 4f 28 66 49  |g..-3.U…-7O(fI|
000000d0  a2 7b 29 59 2e 4a 89 c2  5f 17 8c 43 44 54 e4 a4  |.{)Y.J.._..CDT..|
000000e0  12 0c 01 af 25 6d d3 8d  31 94 8a ef 81 b1 b2 40  |….%m..1……@|
000000f0  24 ab b7 09 2a a7 c0 bb  7c 74 e9 f0 17 58 0f dc  |$…*…|t…X..|
00000100  3b 0a                                             |;.|

Note: the etcd container doesn’t necessarily runs on the master node.

As you can see, our secret starts with k8s:enc:aescbc:v1:key1 which means it was encrypted with key1.

Enforce secrets encryption

Assuming you had plain text data in your etcd and you just set encryption, you need to encrypt current data. Because data is encrypted on write, best way to do it is to read all the data, export it as json and then write it back to etcd:

$ kubectl get secrets -o json | kubectl replace -f -
secret "default-token-3jcfr" replaced
secret "personal-secret" replaced

Rotating encryption key

To change encryption key we need to edit the encryption configuration file & make kube-apiserver reload it.

• Add the new encryption key as the *second* key in encryption configuration file for the required provider
• Restart kube-apiserver processes
  • on kubeadm deployment, apiserver will restart automatically once it will see its configuration file was changed. You can open /etc/kubernetes/manifests/kube-apiserver.yaml and save it to restart the process (touch won’t work)
  • On systemd deployments you should use systemdctl to restart the service
• Edit the configuration and move the new key up to be the first key, which will be used for encryption
• Restart kube-apiserver processes again
• Export and import the secrets to encrypt them with the new key: $ kubectl get secrets -o json | kubectl replace -f -
• Edit the configuration and remove the old key

Running Etcd with TLS

When deploying Kubernetes with kubeadm the default connection between the apiserver and etcd isn’t secured. To enable TLS you need to create a key and certificate pair and then edit /etc/kubernetes/manifests/etcd.yaml accordingly. You can find a detailed overview of etcd flags on Kubernetes docs here.

After taking care of the etcd configuration, you need to change kube-apiserver configuration too. It is the only component in Kubernetes that connects to the etcd server. The relevant ‘–etcd-*’ arguments are found on kube-apiserver documentation and contained in /etc/kubernetes/manifests/kube-apiserver.yaml.

Final words

Kubernetes made a big step in persistent storage encryption with the 1.7.0 release. However, it’s important to use it properly.

A small checklist to make sure your kubernetes persistent storage is secure:

• Secrets and configmaps are encrypted at rest with ‘aescbc’
  • If ‘aesgcm’ encryption is used, encryption keys should be rotated frequently
• Secure connection is set between apiserver and etcd
• Only apiserver user can read / edit EncryptionConfig file

The key rotation process is not working fluently on some systems. I opened an issue on Kubernetes and will update this post once it’s solved.

Get a demo today to find out how to get your organization up and running with Twistlock.

• Categories:
• Cloud Native
• Container Security
• Kubernetes
• Security Research
share: