As enterprises deploy an increasing number of cloud native applications across their environments, platform operators and security practitioners can quickly lose track of all of the cloud native applications and services that their development teams are running, and where. Even if every team is trying to follow a disciplined approach to management, it’s easy for a simple mistake or quick test to result in a service being created and forgotten about.

If you’re not even aware something is running, there’s no way you can secure it. With many organizations purposefully choosing a multi-provider cloud strategy and those providers constantly launching new services and regions, it’s already impractical to manually monitor for these ‘shadow’ deployments.

We want to make it easy to find every cloud native service in use across every provider, account, and region so you can identify these deployments — that’s why we’re proud to announce the release of an open source tool called Cloud Discovery.

What is Cloud Discovery?

Cloud Discovery is an open source tool that helps infrastructure, operations, and security teams identify all the cloud native platform services, such as container registries, managed Kubernetes platforms, and serverless services used across your cloud providers, accounts, and regions.

Cloud Discovery is a powerful tool for audit and security practitioners that want a simple way to discover all the ‘unknown unknowns’ across environments without having to manually login to multiple provider consoles, click through many pages, and manually export the data.

How it works

Cloud Discovery connects to cloud providers’ native platform APIs to discover services and their metadata and requires only read permissions. Cloud Discovery also has a network discovery option that uses port scanning to sweep IP ranges and discover cloud native infrastructure and apps, such as Docker Registries and Kubernetes API servers, with weak settings or authentication. This capability is useful for discovering ‘self-installed’ cloud native components not provided as a service by a cloud provider, such as a Docker Registry running on an EC2 instance.

Cloud Discovery is provided as a simple Docker container image that can be run anywhere and works well for both interactive use and automation. Today, Cloud Discovery supports asset identification on AWS, Azure, and Google Cloud Platform but it’s designed to be easily pluggable with support for more cloud platforms coming soon.

A quick walkthrough with sample outputs

The tool runs as as a container and requires the following environment variables:

  • BASIC_AUTH_USERNAME: This variable determines the username to use for basic authentication.
  • BASIC_AUTH_PASSWORD: This variable determines the password to use for basic authentication. (You can, of course, use Twistlock to secure inject this secret at runtime from secrets managers like Vault, CyberArk, or AWS Secrets Manager!)
  • TLS_CERT_PATH: This variable determines the path to the TLS certificate inside the container. By default the service generates self-signed certificates for localhost usage.
  • TLS_CERT_KEY: This variable determines the path to the TLS certificate key inside the container.

Starting the Cloud Discovery container

First, users would run the Cloud Discovery container using the following command:

docker run -d --name cloud-discovery --restart=always \ -e BASIC_AUTH_USERNAME=admin -e BASIC_AUTH_PASSWORD=pass -e PORT=9083 -p 9083:9083 twistlock/cloud-discovery

We built Cloud Discovery as service, rather than a locally run tool, to make automation easy. You can deploy Cloud Discovery as just another app on Kubernetes (or whatever platform you use for managing your containers) then have multiple tools and workflows call into it to generate up to date results on demand. For example, if you want to generate a weekly report on new services, you’d simply call the endpoints (as described below) and diff the current results from the previous one.

Scanning to list all AWS assets

To gather data about registries, container instances, and Kubernetes clusters, users can run the following curl command:

curl -k -v -u admin:pass --raw --data \
'{"credentials": [{"id":"<AWS_ACCESS_KEY>","secret":"<AWS_ACCESS_PASSWORD>"}]}' \
 https://localhost:9083/discover

The above curl command produces an output with data similar to the following:

Type Region ID
EKS us-east-1 k8s-cluster-1
ECS us-east-1 cluster-1
ECS us-east-1 cluster-2
ECS us-east-1 cluster-3
ECR us-east-2 cluster-1

Scanning all AWS assets to show full metadata for each of them

If you are interested in gathering a list of all AWS assets with a fuller set of metadata, the following curl command can be used:

curl -k -v -u admin:pass --raw --data \
'{"credentials": [{"id":"<AWS_ACCESS_KEY>","secret":"<AWS_ACCESS_PASSWORD>"}]}' https://localhost:9083/discover?format=json

Port scanning a subnet to discover cloud native infrastructure and cloud native apps

This capability allows users to scan all open ports and automatically detect insecure applications (native cloud apps configured without proper authorization).

curl -k -v -u admin:pass --raw   --data '{"subnet":"172.17.0.1", "debug": true}'   https://localhost:9083/nmap

The above curl command produces an output with data similar to the following:

Host Port App Insecure
172.17.0.1 5000 docker registry true
172.17.0.1 5003 docker registry false
172.17.0.1 27017 mongod true

Since inception, our research team Twistlock Labs has repeatedly uncovered threat scenarios that impact containers and cloud native applications. For example, members of the team spoke on stage at a previous DockerCon with Docker about the threats to insecure registries and cryptomining. With Cloud Discovery’s ability to identify insecure applications, we hope to provide vital visibility to help prevent risk from insecure configurations.

How the community can get involved

In the coming weeks and months, we look forward to working with the community to expand Cloud Discovery together. We plan to expand support for additional cloud platforms and provide further checks to identify insecure apps on Docker Hub.

We look forward to your feedback and contributions.

← Back to All Posts Next Post →