Protect Against Crypto Mining Threat
As part of our commitment to protecting cloud native environments, Twistlock Labs is engaged in industry research into concrete threats against containerized deployments. Late last year, we discovered a set of successful and sustained attacks against the ecosystem. One of the end goals of these attacks was to exploit the targeted infrastructure by surreptitiously running cryptocurrency miners on it, stealing computing resources to generate Bitcoin & other crypto currencies. We presented our research jointly with the Docker team during DockerconEU conference in late 2017 1.
Intro to cryptocurrency and crypto mining
Cryptocurrency is a form of digital money designed to be decentralized and pseudo-anonymous in most cases. The first cryptocurrency that had a significant worldwide impact appeared in October 2008 and called Bitcoin. Its principles were outlined in the paper called “Bitcoin: A Peer-to-Peer Electronic Cash System”2, published by Satoshi Nakamoto. Then, in early 2009 an open-source software was released that implemented those principles, sparking the interest of others to play with bitcoins. After some period of time, Satoshi Nakamoto mysteriously left the bitcoin community; in fact, his whereabouts are still unknown. Before he disappeared, he mined more than a 1M of bitcoins. To this day, Bitcoin continues to thrive, and in December 2017, it reached a price point of $19K per coin.
Crypto mining is the process of creating new coins. Conceptually it is very similar to gold mining in that: 1) it is difficult to mine new coins and 2) the more people that are mining coins, the more difficult it is to mine new coins. With cryptocurrencies the main and only resource that is used during the mining process is the computational resource. It can be a CPU, a GPU, or any other proprietary device. The more resources you have, the more coins you can acquire. Consequently, miners use any resource they can find to mine more coins.
Over time, hundreds of new additional cryptocurrencies appeared, and collectively they came to be called altcoins3. Some of them gained traction and some didn’t. Two of the most notorious examples of altcoins that attained high market capital are Ethereum, which reached a market capital of ~$133B during Dec 2017, and Ripple with a market capital of $128B during the same time period. Bitcoin market capital at that time was around $327B. Before I dig into this piece, I want to point out that crypto mining is a concept that is relevant to all versions of altcoins.
Leveraging a victim’s infrastructure to mine cryptocurrency
Initiating the mining process is very easy. One simply has to download a mining software appropriate for the currency he wants to mine, and run it on as many computing resources as possible. For some currencies, mining can only be done effectively using a GPU or some other dedicated hardware, for other currencies it can be completed using typical CPUs. Miners are eager to reach as many computing resources as they can, and there is a large number of them who partake in illegal activities to increase gain.
An approach we see quite often is a miner penetrating external environments solely for getting access to computing resources. This, as opposed to more common goals, like data exfiltration or denial of service. Also, as opposed to common goals above, mining is a much less intrusive action and typically does not involve sending large data chunks outside of an organization or accessing suspicious API. Its main impact is high CPU / GPU usage, and if done well enough, even this can be masked as a benign, temporary, system load. On one hand this fact makes crypto mining much harder to detect, on another it can create significant negative impact on the environment. Specifically, if done right, mining processes will be scheduled to run in a higher priority than the rest of the software within the environment, all while the rest of the software is exactly the type you want to be running within the environment – this is what you created this environment for! Consequently, having hidden mining processes can result in severe denial of service and can create an appearance that your environment is overloaded and requires to be extended – which is exactly what the attacker wants.
So, how does the crypto mining threat apply to cloud native environments, specifically? To answer that, we will need to define cloud native. Cloud Native Computing Foundation (CNCF) defines cloud native as4:
- Containerized. Each part (applications, processes, etc) is packaged in its own container. This facilitates reproducibility, transparency, and resource isolation.
- Dynamically orchestrated. Containers are actively scheduled and managed to optimize resource utilization.
- Microservices oriented. Applications are segmented into microservices. This significantly increases the overall agility and maintainability of applications.
Using this definition as the foundation, I’ll emphasize that: 1) There is a large number of entities, which are containerized microservices, and 2) There is a platform, an orchestration platform, that automatically manages the above entities. Now consider that an attacker got access to one of the microservices through exploiting a vulnerability in a proprietary software running within the microservice. This is not a rare scenario. The attacker can easily use it to gain revenue by mining crypto coins within this microservice. He can also try and do lateral expansion to exploit more microservices. I’ll elaborate on this in the next section.
Mining coins in an exploited cloud native environment
In a cloud native environment, software is packaged into images. Each image is an archive containing the initial file system the microservice will see once it’s instantiated from the given image. Basically, any piece of software can be packaged into an image. Orchestration platforms take images and instantiate microservices (containers or pods) from these images. They constantly make sure all microservices are alive, healthy, and appropriately distributed amongst different hosts.
Crypto mining software can be packaged into images as well. In fact, you can find quite a few examples in Docker’s public image hub5. Some specific examples are below:
It is very common to exploit publicly accessible cloud native environments in order to run crypto miners there. Some common attack vectors that can be used to infiltrate an environment for this goal are:
- Find a publicly accessible orchestration platform (such as Kubernetes, Shipyard, or similar). Filter those that have weak settings6 (e.g., default admin credentials), and use these to access the platform and run crypto-miners there. As part of our research we tried to find real instances of publicly facing orchestration platforms with default credentials and have found many hundreds of these around the Internet.
- Find a publicly accessible image registry (such as Docker registry) with default settings and modify images there to include miners. For example, modify a MongoDB image in such way that when instantiated it will run a crypto-miner alongside with actual mongo daemon.
- Exploit a vulnerability in a containerized, proprietary service to get into the microservice and run miner there. Then move laterally to deploy more miners in other microservices.
As mentioned in the beginning of the article, our security team, TwistlockLabs, is tasked to constantly discover new threats and monitor their level of spread in the public network. During the last year we saw a constant increase specifically in the area of abusing non-adequately secured cloud native environments in order to mine crypto coins. This is definitely related to the phenomenal growth in the popularity of crypto coins during the last year. Specifically, we discovered thousands of orchestration systems, some belonging to large corporations, that had miners running within them because of lax security measures.
Protecting cloud native environment from cryptocoin miners
As is true across the board in cybersecurity, a good protection mechanism consists of several layers. The first layer of defense for detecting and blocking crypto miners is a combination of static analysis and compliance checks. The static analysis portion is focused on detecting the mining software with file signatures by checking each executable’s signature against known crypto miner executables’ signatures. However, this by itself will definitely not stop a more advanced attacker. Today, there are free open source tools that enable the attacker to easily take a piece of software and obfuscate it. The result is a new executable with different overall and partial signatures that cannot be detected using conventional methods. One of the tools that can be used to achieve the above is Shiva7. The second portion of the first layer of defense is focused on checking for general system compliance, as described in several CIS guides89. While important, this again will not a stop a more sophisticated attacker.
The next, second, layer of defense is the ability to monitor and enforce how the entities behave in runtime. A current de-facto approach to monitoring and enforcing entity behavior in runtime is based on blacklisting. Blacklisting means creating a set of forbidden (blacklisted) behaviors for your environment. This is a manual and rigorous process. These behaviors usually consist of network access (e.g., what addresses the application should not access and what ports it should not use), file signatures (e.g., such and such signatures corresponds to malware and should be quarantined), processes (e.g., processes with some specific names are not allowed to run), etc.
The key thread running through all these areas is the fact that what is defined is a blacklisted behavior and that it is defined manually. As I described above, in case of crypto mining it is very easy to tune the mining software to escape blacklisting in all the bucklets above. Code obfuscation and CPU/GPU usage adjustments are just a couple of ways to get around blacklisting. One can package the miner code in an executable with unknown signature and name it with a random name that is not mentioned in the process blacklist. As a result the miner code will not be blocked.
In the previous section I have described three attack vectors to inject a crypto miner into an organization. Now I’ll dig deeper into how they can be mitigated. The first two vectors are based on weak platform configuration, either of orchestration platform or image registry. Such weak configurations can easily be detected by a cloud native aware security solution. A good solution, like Twistlock, would also block the ability to configure the platform in such a way. The last attack vector is more tricky. It does not use platform vulnerability, but rather a vulnerability in a proprietary service the organization exposes publicly. The next few paragraphs talk about how to mitigate this threat.
I believe that the only practical way of detecting crypto miners and other advanced malware within your cloud native environment is by whitelisting expected entity behavior. As opposed to blacklisting, whitelisting identifies what a benign behavior for a specific entity is. Think of it as a profile that contains the information about what specific application is allowed to do. It may contain a list of executables signatures it is allowed to run, a list of IP ranges it is allowed to access, a list of storage paths it is allowed to write to, etc. As you can imagine these profiles significantly differ between applications. The more tightly the profile describes the application, the better the protection is.
In typical cloud native environments, the number of entities (or microservices) is considerable. Therefore, it is completely impractical to assume a manual creation of profiles containing whitelisted behavior for each entity. My expectation from a modern, cloud native aware, security solution is to 1) detect all entities; 2) create whitelisted profiles for every entity; and 3) continuously enforce that every entity behaves within its profile. All of this should be done completely automatically. This is what we do at Twistlock and this is the core capability of our main product.
Going back to the crypto mining example, consider that one of the entities is exploited and an attacker runs a crypto miner in a container. Given a profile containing whitelisted behavior, the crypto miner will be immediately and automatically detected – its signature will not be in the profile, it will communicate with IP ranges that are not in the profile, it will access system calls that are not in the profile, etc. These are just few examples, and a typical profile is much richer than this.
The goal of this post is to raise awareness to protect against crypto mining threats. As stated, a vulnerability in an application is very easy to exploit by running a miner alongside the original application. With time, a lateral movement is possible to extend the miner deployment to other applications gradually imposing a significant negative impact on overall performance of the production environment. In case of a cloud native environment, the impact is much more significant, since the number of entities is higher, and a common, blacklist-oriented security solution iis much less effective.
Cloud native aware, whitelist oriented security solutions can easily detect any behavior that differs from the one outlined in a tight profile wrapping the application. This effectively applies to crypto miners as well.
Follow us on Twitter
Follow us on Twitter for real time updates on the cloud native ecosystem, Twistlock product, and cloud native security threats.
Five Best Practices for API SecurityRead the Blog
When On-Premise Serverless Beats the CloudRead the Blog
Kubernetes AuditSink: Real-time K8s Audits and ForensicsRead the Blog
Native Helm Charts for Frictionless Kubernetes DeploymentsRead the Blog
How Knative Can Unite Kubernetes and ServerlessRead the Blog