Secrets Management in Your Container Based Applications

In today’s blog I will cover how the Twistlock solution can assist you in keeping your valuable secrets such as passwords, certs, and tokens safe and still available to your running containers.

The worst thing you can do is embed container based application secrets into your image Dockerfile, particularly when you maintain your source code in a public repository on github – you might as well place a “please hack me” advertisement on Craigslist.

Instead you should use tooling, such as Hashicorp Vault to keep your secrets safe and then utilize Twistlock’s integration with Vault to inject those secrets into your Docker containers – either via an environment variables or files.  In this post we’ll go through each step of the process and even show you the necessary scripts.

Environment Setup

Before starting, create and “dot” an environment script or put the following environment variables in your .bashrc file.  You need this environment when testing your vault as well:

export VAULT_HOST=
export VAULT_HOST_PORT=8200
export VAULT_ADDR=https://${VAULT_HOST}:${VAULT_HOST_PORT}

Containerize HashiCorp Vault

To build your Vault image you will need a custom Dockerfile and a config.hcl file with your Hashicorp Vault configuration

config.hcl
storage "file" {
  path = "/vault/data"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/vault/.ssh/server.crt"
  tls_key_file = "/vault/.ssh/server.key" 
}

disable_mlock = true

Hashicorp provides a Dockerfile, but I found it a bit complicated.  Here is a simpler one, based on Alpine, which has the additional advantage of being very lightweight and very secure.

FROM alpine:3.6
MAINTAINER Matthew Barker <matthew@twistlock.com> (@matthewabq)

# This is the release of Vault to pull in.
ENV VAULT_VERSION=0.7.3

# Create a vault user and group first
RUN addgroup vault && \
adduser -S -G vault vault

# Install vault
RUN apk add –no-cache ca-certificates gnupg openssl libcap && \
gpg –keyserver pgp.mit.edu –recv-keys 91A6E7F85D05C65630BEF18951852D87348FFC4C && \
mkdir -p /vault && cd /vault && \
wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
unzip -d /bin vault_${VAULT_VERSION}_linux_amd64.zip

# /vault/logs is made available to use as a location to store audit logs, if
# desired; /vault/data is made available to use as a location with the file
# storage backend, if desired; the server will be started with /vault as
# the configuration directory so you can add additional config files in that
# location.
RUN mkdir -p /vault/logs && mkdir -p /vault/data && /vault/.ssh
COPY config.hcl /vault/
COPY docker-entrypoint.sh /bin
RUN chown -R vault:vault /vault

# Expose the logs and data directory as volumes since there’s potentially long-running
# state in there
VOLUME /vault/logs
VOLUME /vault/data

# 8200 -default port to connect to vault server, ensure this matches your env setup
EXPOSE 8200

# For production derivatives of this container, you should add the IPC_LOCK
# capability so that Vault can mlock memory.

# start vault server with config file in /vault
USER root
ENTRYPOINT [ “docker-entrypoint.sh”]

Be sure to provide the private key for your server and signed certificate for TLS operation of your Vault, put them in a .ssh folder under your Dockerfile folder before running your container (or use soft links to these files):

private key:               $PWD/.ssh/server.key
signed certificate:  $PWD/.ssh/server.crt

Now you can build and launch your custom Hashicorp Vault container, when you run the container map the .ssh folder so the vault has access to the certs at run time, obviously we don’t want to put these sensitive secrets into the Dockerfile.

docker build -t matt/myvault .
docker run -d -p 8200:VAULT_HOST_PORT \
-p $PWD/.ssh:/vault/.ssh \
--name myvault matt/myvault

Initialize and Unseal Your Vault

Here is a script to initialize and unseal your vault, the script below will generate your vault keys and root token and place them in a file called vault.keys. Be sure to store your vault keys in a secure location, you will need the Vault root token to perform a client auth every time you want to access your secrets.  Twistlock also needs this root token to connect to Vault and access your secrets for injection.

#!/bin/bash

# make sure VAULT_ADDR is not empty!
echo "VAULT_ADDR is $VAULT_ADDR"

if [ ! -e vault.keys ]; then
echo "Initialize Vault"
vault init -key-shares=4 -key-threshold=2 > vault.keys
export ROOT_TOKEN=$(cat vault.keys | grep '^Initial' | awk '{print $4}')
chmod 400 vault.keys

echo “root token: $ROOT_TOKEN”
echo “Setup Vault demo”
else
echo “Vault has already been initialized, skipping.”
fi

echo “Unsealing Vault”
cat vault.keys | grep ‘^Unseal’ | awk ‘{print $4}’ | for key in $(cat -); do
vault unseal $key
done

echo “Vault setup complete.”
cat vault.keys
echo “To use vault be sure to run ‘vault auth ‘”

instructions() {
cat <<EOF
We use an instance of HashiCorp Vault for secrets management.
It has been automatically initialized and unsealed once. Future unsealing must
be done manually.
EOF

exit 1
}

Test Your Vault Access

To test your vault, you will need the Vault client in your execution path, this works for Ubuntu 16.04:
wget https://releases.hashicorp.com/vault/0.7.3/vault_0.7.3_linux_amd64.zip
apt-get install -y unzip
unzip vault_0.7.3_linux_amd64.zip
mv vault /usr/local/bin/
rm vault*.zip

Here is my test script:

# make sure VAULT_ADDR is set first
vault auth
vault write secret/mysecret value=’mysecretvalue’
vault write secret/nexus_pw value=’changeme’
vault read secret/mysecret
vault read secret/nexus_pw
vault list secret

Configure Twistlock to Inject Your Secrets into a Containers at Run Time

Of course the first step is to get a trial of Twistlock and install the Twistlock solution in your cluster or set of servers you want to protect.  Twistlock will secure your Docker container deployments across your entire SDLC and from the top to bottom of your deployment stack as well as allow you to inject secrets into your containers.

Once Twistlock is installed, open the Console UI via a web browser, select the sprocket icon at the top right part of the page, and select “Access”, then select the “Secrets” tab.

Now select ‘+’ sign next to “Add new store”, give your secret store a name, fill in the URL for your vault server (must use https) including port number (default is 8200).  Then use your Vault root token in field “Access token”, no need to fill in ‘CA certificate’ field.

How to Inject Secrets Into Your Containers

You can now inject one or more secrets into your containers when you launch them either as environment variables or as files.

In the Twistlock Console interface, select Defend/Access and then select Secrets tab. Finally, select Add new secrets rule.  In the pane that pops up, fill in a name for your secrets rule, select desired injection type, then fill in the fields for new secret.

First line is the name of the created environment variable or file.  The second is a drop down where you select the secret store you created in the previous step. Next line is the secret name that matches the secret name in the store.  The last line, at least for Hashicorp Vault, you type in literal value just like shown below – the select Add Secret.

Lastly, define the scope for your secrets rule, I am choosing to inject this secret into all containers named nexus3, you can use wildcards here or even accept the default which has wild cards for every field resulting in injecting the secret into every container that you run.  You can further constrain you secret injection by image name, host, or container labels; again wildcards are acceptable.

Testing Your Secrets Injection

To test my example, I simply launch a container named nexus3 based on an Ubuntu or another Linux derivative.  With Ubuntu I can perform the following command on my host to “bash” into my nexus3 container:
docker exec –it nexus3 /bin/bash

and confirm the environment variable is created with env.

If you inject a secret as a file, it will show up in /run/secrets/ which is an in memory mount point so no secrets are stored on your host.  If your secrets are injected as environment variables, validate that they are encrypted with docker inspect.

With Twistlock and Hashicorp Vault, your container based secrets remain, ahhh, secret.

← Back to All Posts Next Post →