Implementing & Architecting Secure Docker Environments

In this video you’ll learn:

  • 120
    How Role and Responsibilities of dev, devops, and security teams change in devops environment.
  • 346
    The conceptual design principles for scalable and secure devops environment that runs containers.
  • 841
    The three common pitfalls when designing security for containerized environments for the first time.


This is my baby daughter and her name is Roni. I’m going to talk to you today about implementing Secure Docker Environments At Scale. I’ll also try to be brief so you’ll get a little bit of your time back.

Actually that was a lie and I’m not going to talk to you about implementing Secure Docker Environments At Scale but about designing them. What I’m going to present is a point of view for the security architect on how to address these new challenges.

Anyone here a security architect? Cool a lot of people so that’s good. I’ll try and be brief and just show the challenges and what we think should be changed around that. It’s based on a company called Twistlock that I’m the CEO of and it’s also based on more of…I want to say hundreds but it would be a lie to but more than a hundred customers we’ve engaged with in the past 1 ½ years. You can actually see a lot of commonality between people who have been running containers in production for a while.

I’m going to talk about 3 areas. First is the roles and responsibilities and so an important point to notice is the roles and responsibilities are actually changing when you move to this new models of producing applications. The other point I’m going to go through is the conceptual design, the high level design of the system and how you think you see it and what it actually does. The 3rd point is just common pitfalls we actually see customers run into.

Let’s start with the roles and responsibilities. So you’re a security architect and part of the security team and what you need to do is design a secure continuum. So there are basically 3 important things you need to take care of.

First is compliance. This is a continuum and you have to understand any thought that goes through a developer’s head can find its way very quickly into production. So you must really understand that continuum and try and make it as compliant as possible… and push back thanks to developers and I’ll get to that.

The other important point is you need to have micro service active threat protection and so whatever you have today probably isn’t working very well for micro services and at the end of the day a lot of the security teams a lot of the important things they need to do is take care of cyber security. I mean it’s not just about compliance it’s also about whether someone breaks into your application.

One of the things we were taught in the past few years is that it’s not a matter of whether you’ll be hacked but more of when you’ll be hacked so you have to make sure there is something there.

The 3rd piece which is pretty important is the synergy with the developers. As we saw yesterday in the keynote the best tool is get out of the way, adapt to you and make the powerful simple. So if you’re going to try and stand in the way of dev ops it’s not going to work very well. You must make sure the developers actually help you tackle some of those issues at the end of the day that they were involved in creating but you can’t just tell them they can’t do certain things and we’ll get to that in a second.

So that’s your role in this process. Let’s go to the dev team and the dev team used to just write code but now developers actually introduce vulnerabilities that have to do with the actual image, the reusable component they use. They actually sort of declare how they’re going to use the infrastructure and might do something that’s not compliant with you what you’re organization wants.

The 3rd piece is identities and access and so you have to think about who is creating the identities in the organization and whether they’re compliant. It’s no longer the IT guys that do that. So you can expect the developers to do a lot of these things but they won’t proactively think about security and so this is a very fine line.

You can’t expect them to fix issues you point out to them you need to do it in an automatic manner. But you can’t proactively ask most of the developers to sit at the end of the day and write a manifest of how things should work or what they want…or what security should look like. So you have to automatically check whether they followed things and you want to try and push back to them things as soon as possible in the CICD, etc.

The last sort of role is the dev ops team right? I mean in the past if you’re a security architect your best friends would be the folks at the networking team. You’d just put a box somewhere and it would supposedly take care of security but that’s no longer the case. Security needs to be weaved inside the application and I’ll show you a little later why it’s important. But you’ll realize very fast that you need to work with the dev ops team.

And the dev ops team they care about security but it’s not their main priority all the time. So they will probably help you implement it and weave that security into the process but it doesn’t necessarily mean that they’ll have time, I don’t know, every day to look inside of the registry and get a list of issues and then follow up with each developer. So you need to be smart about how you’re going to use them. If you don’t do that they will very easily or quickly get tired of you.

Let’s talk about the conceptual design. This is typically how you do application security today. So you have the development and staging which is being taken care of by the development team. You typically give them some offline guidance on what’s secure and what’s not secure and expect them to produce things that are as secure as possible.

Then you typically do or have someone do an offline review of the products they created. Then when you move to production you typically have the IT operations team that installs the software and can basically take care of different policies you can set. They also take care of maintenance so if there is a vulnerability or things that are outdated they will typically do the patching. Obviously they take care of compliance of the platform, the networking and identity as well.

This works find today and the only issues where the situation gets a little hairy is when you get these little cute monsters that show up and these are the micro services. Suddenly you wake up one day and realize there are a bunch of VM’s that you got a security exception for certain people in the dev that said they were going to run Docker and is actually being used to run things in production. Then you realize this large hole is getting larger and larger and there are other teams doing this and then you get into some kind of issue.

The issue is a lot of the stuff that used to work for you doesn’t work very well anymore. You don’t get to do a milestone review because there is no actual milestone and if there are some of them are…I mean the whole point of dev ops is not to have these bottlenecks.

Then in production a lot of your fancy next gen firewall, application firewall, east/west firewall they’re not aware of the application and don’t know where it should run and even if they knew most of it would be encrypted anyway. So if you’re thinking that you actually control the applications behavior and they’re aware of it you’re totally wrong.

Another huge challenge is updates. The IT folks have no idea what’s running on these black holes you helped create and they don’t know if everything is updated or patched or not. Your IPS /IDS and deception are really good but you don’t really know if it doesn’t have any traces you just know there’s a problem. So you have to think back…how can you really trace that? So it’s not that helpful.

And the hosting is also kind of an issue because you used to work with certain kinds of hosts and suddenly you have project “Atomic”, CoreOS, Rancher and all these things and maybe you don’t have a lot of expertise in that and you need to learn about it. Also if you use encryption what type of encryption? What is the PKI story of these applications? So these are all challenges you have today.

You are left with some things and they’re not too much right? You’re left with doing code reviews and so you really hope the other guy sitting besides the developer really knows how to review what this guy is doing. You have Layer 3 and Layer 4 isolation which is kind of good and sometimes you also use hyper visors to further isolate containers and the identity which is also kind of helpful.

Overall, this doesn’t look good. You’re out of control. Then you look into it a little bit and realize what’s actually happening under the hood is you have this continuum between development and production and typically the staging is in the middle and it’s sort of automatic. And they launch these cute little animals from development all the way to production. Actually there are devs or dev op teams in charge of that and they’re also in charge of the maintenance so you probably want to try and see how you can work with them around that.

Most likely they’re also creating identities and not waiting for the IT guys to give them accounts and identities they might be doing a lot on their own. They are also pretty knowledgeable about the hosts who is probably running on these VMs. So all these things are actually happening under the hood and you need to take care of them.

What can you do and how can you get better? What you typically want to do first off is have some kind of automatic delivery review. So anything being pushed into production you want it to be automatically reviewed. You need some kind of a system that looks at everything the developer produced including the infrastructure as code, including the different layers and make sure it complies with what the enterprise thinks is important. It includes vulnerabilities and anything you can think that you wouldn’t want to get to staging or production.

Then in the production part you typically need to compensate for the fact that there is no application or anything. So if there’s any type of anomaly at run time you really need some kind of mechanism to support you against cyber threats because they are going to happen. The attackers don’t care that it’s a container it’s an application and they’re going to hack into it. There is no complete solution for it because nothing can actually stop human stupidity its infinite. So at the end of the day someone will write something that will get hacked.

So you must have something and I call it delivery aware because it’s no longer just an application. What we do at Twistlock is try and write these components. But there’s actually a lot of opportunity in understanding the delivery because it’s minimalistic, it contains more information and it’s immutable. There is actually opportunity in protecting these things automatically much better than you’d get in supposedly an application or next gen firewall who are completely unaware of that and not protecting you.

Obviously updates and security alerts and patches is something you need to think about as well. And you need to coordinate with the dev ops team when you update, how do you update? Do you try to automatically update which is probably not a good option because you have to get the devs involved because you might not know what’s going on there and the IT folks definitely don’t know what’s going on inside the micro service.

Obviously you need something that really understands the host configuration and makes it compliant. It’s not a big deal but still you need to learn about these new hosts and what are the capabilities.

The best thing you can do and this is taking it a notch further is also use the staging environment in a smart way. Again, not all these tools are written today and not all these capabilities exist today specifically these don’t exist but this is something that definitely going to evolve. But you can actually do automatic fuzzing, sandboxing, delivery aware…these actually are a really cool opportunity because imagine that a developer created some kind of engine “x” based application and then you can start doing fuzzing and penetration tests based on that and send it back to him if there are any issues.

So a best practice would be to use the staging environment in order to do a bit of fuzzing and penetration testing from a security point of view. And we’ve actually seen a lot of companies running dev ops for years and use these practices.

So that’s the gist of it really. The 3 common pitfalls that have to do with what I just mentioned…compliance policies don’t try and nothing will…if you try to create a policy delivery and you’ll try and talk to each person or each small crew and ask them what they’re going to deliver today and adjust the policy it’s just not going to work. You need to think in an R & D level, an org level, an application groups level and create the compliance policies in these levels otherwise again it’s not going to scale.

The other point is delivery hygiene. You want to try and push back as early as possible things to the development team because if you’re just going to monitor things in production it will be too late. Actually what we’ve seen with most customers when they install us out of the box and we look at their production and we see horrible things. Day 1 looks really bad because there are a lot of things that you couldn’t even imagine that you’re running in production. So the best thing would be as you put those controls in place to try and put them as early as possible so the developers can actually synergize and help you.

The last part which is really important is the active threat protection. So don’t trust your application next gen firewall, they’re blind and completely flying blind. You need something that really understands these micro services and you want to use something that I just called delivery aware. But you want to actually use the power of containers. Think micro segmentation and you get micro segmentation by default when you actually run containers because everything is already micro and so you just need to do the segmentation.

And so you actually want to unleash the power and we actually tell our customers that with containers you can actually be more secure and this is the reason. So you need to think hard about active threat protection. I know obviously compliance is the first thing that you think of but that’s probably the next thing you need to take care of.

With that I want to thank you for your time. I promised to relieve you early from this session so thank you very much.