As you may recall, a couple of months ago we announced a new blog post series around Machine Learning (ML) and how it’s used in Twistlock. This blog post is the second one in this series. If you didn’t read the first one you are very encouraged to do so.
Today we are going to talk about CNAF – Twistlock’s Cloud-Native Application Firewall. Specifically, we will cover how CNAF enhances the standard application firewall prevention and detection capabilities with deep application knowledge and ML.
Similar to a standard Web Application Firewall (WAF), Twistlock CNAF protects against all common OWASP threats such as content injection, cross-site scripting (XSS), click-jacking, sensitive data exposure and more. A typical WAF is usually deployed as a frontier gateway that handles all traffic for an organization. As opposed to the standard WAF protection model, each CNAF instance is tightly coupled with a running app. As a result, it’s able to take advantage of our deep application understanding and provide more accurate and robust security models.
For example, deploying a firewall per app enables easy detection of application code and configuration updates, as well as the ability to correlate specific traffic patterns with restarts and shutdowns, which might be an indicator to DDOS attacks. It is important to emphasize that this new approach does not replace the existing tools and mechanisms for attack prevention, rather it adds another layer of security-thereby providing better deep defense.
The problem of detecting malicious traffic on production data without pre-defined information while adjusting the machine learning models based on app behavior is complex, and requires advanced solutions to ensure high precision and low false positives rates. There is some existing work on solving this problem using supervised learning (which uses predefined information), and as you’ll soon see, we’ll take advantage of this type of approach as a preliminary step in our algorithms.
Existing methods use historical network data to train the machine learning models. This historical traffic data is labeled by a human-typically by classifying every request as malicious or benign. Then it’s used to build a classifier that separates between malicious and benign requests. Technically, this model can be further extended to detect the exact attack category (e.g., XSS, SQL injection, etc.) using multi-class classification algorithms.
Unfortunately, using predefined models as the only data source has three core issues:
- Historical data does not contain patterns of new types of attacks
- Simple classifiers lacks the notation of a session or a user and thus cannot correlate between traffic flows and attacks
- The specific app under attack is not taken under consideration
At a high level, Twistlock’s CNAF applies a 3-step detection and prevention mechanism:
- We use standard WAF techniques such as continuously updated blacklist of regular expressions and signatures to prevent all top level attack types such as SQL injection, XSS, directory traversal and more
- Per API call, we build a simple machine learning model that classifies the request and response content. This model is based on URL parameters, body identification, headers and cookies. This blocks more advanced attack (e.g., when attacker is uploading malicious content to a valid image upload endpoint)
- Per transaction (combination of API calls and state), we detect whether the functional application paths that a consumer is undergoing makes sense in the context of the app. To do so, we build multiple statistical models, where each model describes a specific interaction pattern with the application. For example, one model is used when a user is accessing the system from a mobile device while another model is used when the application is used from the browser.
With this in mind, let’s discuss the underlying ideas and challenges of the advance detection mechanism.
First, we automatically divide the app to multiple logical states. A logical state of the app (such as login) may be composed from multiple API requests. For aggregating and classifying states, we use standard text classification methods on the URL, time and request parameters. Once all logical steps are determined, the second step of the algorithm builds a statistical model of the behavior of a typical consumer of the app.
In this step, we build our statistical model using a mathematical tools called Markov chains, which represents a matrix of transition probabilities from states in the system. That is, the system states are modeled as a directed graph, where each edge between two states s1 and s2 in the graph is associated with transition probability.
For example, given we are in s1 (login) what would the probability be to move to s2 (dashboard) in the next step of the app?
Note that a state does not necessarily map to a specific page or API request; rather to a combination of specific conditions, which define the current location of a user in the application. In practice, we will further enhance this simplified model to take other criteria in considerations when calculating probabilities. Also, when calculating probabilities, Markov chains do not consider the historical state of the user which might be relevant to determine the threat score, so for some conditions, Markov chains of higher order are used.
Let’s take a very simple example to illustrate the idea:
In this example we have three states. The conditional probability given a user is in a specific state is illustrated on the directed edge. Clearly, just by looking at the picture, we get immediate insights about the app:
- From login state, a user can only move to the home state
- From cart page, a user can only logout
Even this simple example raises some interesting challenges. Let’s discuss some of the advance techniques Twistlock uses to improve detection accuracy.
Determining whether a transition is valid or not: In practice, the transition probabilities are not absolute (0 or 1).
It’s essential to ensure that there is sufficient confidence before blocking or alerting on malicious traffic. One of the methods we use to tackle this is monitoring a user’s behavior within the application and constantly comparing it to the model described by a rich Markov chain. As a result we have a level of likelihood of the validity of each user session. Once the level of likelihood becomes too low,( i.e. there is a significant chance the session is malicious,) we alert and block the session. To make sure our models are accurate we include various session parameters, such as existence or lack of specific HTTP headers or query parameters.
Handling varying request parameters: Consider an example of a login page and a usage of authentication token. Is it the same to transition to the cart page with and without a token?
We solve this by having multiple edges with different probabilities for the same nodes. That is, there are multiple ways to transition from login to home states. For some scenarios, it also makes sense to normalize the probability by some risk factors and historical data of the user.
Handling stale browsers or refresh: In some implementations this means that a state returns to itself. To avoid false positives, and to easily detect continues attacks, we can model the probability that a state returns to itself, given that it already been in a given state N times.
To summarize, Twistlock’s CNAF significantly increases the defense in depth of the standard WAF security using advanced, application specific, machine learning models. It also applies a multi-step solution to ensure our classification accuracy is of the highest quality.
As we previously mentioned, this approach is one example of a set of different applications Twistlock takes to ML. Stay tuned for future posts on this topic.
Follow us on Twitter
Keep up to date with the latest news from TwistlockLabs and TwistlockTeam.
Cryptomining Malware Emerges
I have been watching for the spread of malware that, primarily, uses c...
Calling the Twistlock API from PowerShell
The Problem This morning, a colleague was looking for situations where...
What Makes Distributed Security ‘Cloud Native’: Podcast Overview
I caught up with Scott Fulton III on this edition of The New Stack Mak...
Reflections on the 20th Anniversary of Open Source Technology
Exactly twenty years ago in February 1998, the term “open source” ...
Enhanced Syslog Data Streams: 2.3 Deep Dive
In each of our Twistlock releases, we publish some truly remarkable fe...