At Twistlock, we’ve been proud to partner with Docker on many commits to Moby’s security stack, perhaps most notably working together to make authorization pluggable. We recently had a neat opportunity to jointly solve a problem for a real world customer that wanted to integrate secrets stored in CyberArk with apps running in Docker Swarm. Secrets management is a key (pun intended!) part of securing cloud native apps and Swarm delivers a great set of secure by default capabilities for creating and distributing them. However, until now, if you wanted to use this functionality, the secrets themselves had to be stored within the cluster. We were happy to work with the Docker security team to make this functionality pluggable such that you can rely on existing enterprise secrets stores like CyberArk, Vault, or anything else to store those secrets but still take advantage of all the capabilities of Swarm to distribute them. This is particularly useful in regulated environments or other organizations that have already standardized on an enterprise secrets management platform and now want to extend that into their container deployments.

How it works
Docker Swarm is composed from managers (assign work and store metadata about the cluster) and workers (run containers as services). In the swarm, services can consume secrets that are pre-populated to the swarm.  The secret metadata and value is stored securely on managers in an encrypted storage called raft-log, and are assigned on demand to workers. Workers only receive secrets that are associated with a scheduled task on that node on a need-to-know basis.

Until the new pluggable secret backend, only secrets that are stored inside Docker Swarm were supported.

In the new plugin system, the secret metadata is still stored in the swarm manager and managed by standard Docker commands, but the secret value remain empty. In the last mile, prior to assigning the secrets to the worker, the swarm manager will query the third party plugin to fetch the secret values that are required to run containers. You can read the full details in the design doc.

To illustrate how to use the new plugin subsystem, we will create a working plugin from scratch using the new infrastructure.

First, we create a new secret plugin. As part of the plugin work, we extended the go-plugins-helper package, so it will be easy to add more secret backends.

Let’s follow the plugin creation instructions and create a very simple plugin

package main
import (

func main()

// Register the plugin

h := secrets.NewHandler(&samplePlugin{})

type samplePlugin struct {}

// Get is the API to fulfil in order to return secrets from the third party secret library
// the secret request metadata contains additional information relevant
// for generating the secret, such as the service name, network information and more.

func (p *samplePlugin) Get(req secrets.Request) secrets.Response {
if req.SecretName == "" {

// This is how you return errors from the plugin

return secrets.Response{Err: "missing secret name"}

// This is how you return values from the plugin

return secrets.Response{Value: []byte("twistlock")}

We compile the plugin statically, and place it under a folder named rootfs

> mkdir rootfs
> go build -o ./rootfs/sample-secret-plugin --tags static --ldflags "-extldflags \"-static\"" plugin.go

We consume the new plugin

> docker plugin create twistlock/secrets:0.1 ${PWD}
> docker plugin ls

ID      NAME                  DESCRIPTION                             ENABLED
d48f7e8 twistlock/secrets:0.1 A sample secret plugin for Docker false

Enable the new plugin

$ docker plugin enable twistlock/secrets:0.1

Create a new secret (note the new driver parameter)

> docker secret create mysecret --driver twistlock/secrets:0.1
> docker secret ls

ID              NAME     DRIVER                CREATED        UPDATED
f81pxnwpi9jrq4v mysecret twistlock/secrets:0.1 41 seconds ago 41 seconds ago

Consume the secret using a regular swarm service

> docker service create --name twistlock  --secret mysecret alpine sleep 100

Verify that the secret propagated correctly

docker exec twistlock.1.5zz7t0gy3qmes8m3g6du3ehhn
cat /run/secrets/mysecret

Deploying plugins in the Docker Swarm and next steps
There are two important deployment consideration when setting up a secret plugin in your organization. First, it’s cumbersome to manually deploy the plugin on each node manager, so we would like to be able to accomplish this using the standard API. Second, our secret plugin might need pre-populated secrets (passwords or client certificate) to access the third party provider, so we would like to use the standard secured swarm secret storage for that.

In the upcoming Docker release, the first issue is addressed by a mechanism that enables deploying plugins similar to standard Docker services. Currently, this feature is supported via the engine API and when the –experimental flag is set.

We are actively working on adding the secret population issue to the mainstream engine.

What’s Next:

← Back to All Posts Next Post →