So, my name Ian. I'm here to give a talk about Kubernetes security. I do DevSecOps at Jamf software and I focus on container security and I'm an ethical hacker.
I travel around the company playing capture the flag games, ethical hacking. I think coming from an offensive security background has made me a better defender.
Because being an attacker, teaches you a different way to think. Often developers will assume good intent. And make features for an ideal user to be used in exactly
the way that they want them to be used. And that's great. Assuming good intent is a great practice and often a good personal quality. But in fact not all users have good intentions.
And many of them might not use things in the way that they're intended. And so, a thing that's important for us to consider
is how to be able to think like an attacker in order to be able to secure things from those users.
Attackers have user stories too. As an attacker, I'm here to tell you some. As an attacker, I'm to PWN you.
So, because of the time constraints, this talk is long on high level abstractions and low on deep dives. We're barely able to scratch the surface of Kubernetes security, it's a big topic.
But if you are interested in more, I have a resources slide at the end. But before the attacker stories, let's set the stage.
In the beginning there were virtual machines. And virtual machines were very useful for people who wanted to be able to build things that require different
operating systems or different dependencies from the things that they were working on on the host.
But they came with a lot of resources and a lot of overhead. So, containers happened. They don't always contain kittens, but perhaps they should.
And containers are not tiny virtual machines. What they are is they share a kernel and have namespaces to separate them. And containers are great and very useful.
You can package all of your application and all of its dependencies, everything you need in together, but they have some problems.
They don't necessarily communicate very well on their own. They're difficult to scale in an automated way. And, you know, if you're trying to share resources like a database or anything like that,
Okay. If you're trying to share resources like a shared database in a microservice architecture, it can be a problem because they
they're stateless and they don't necessarily do that very well on their own. So, in order to solve those problems people created container orchestration frameworks like Kubernetes.
Which solved a lot of problems with being able to really scale microservice architecture and its applications in, you know, in a way that could be easily automated and done by people.
It also, as well as solving problems, added new complexity. And added like a lot of complexity. Who here thinks that Kubernetes is kind of hard? Anybody? Yeah.
I think most people who have played with Kubernetes think it's probably kind of hard. There's a really big learning curve.
There's a lot going on there, there's a lot of moving parts. And because of that, one thing that's happened is that defaults for Kubernetes make it so that security is an afterthought.
It's made to get people up and running really fast. It's made to, you know, be able to, like, smooth over the learning curve. That's cool.
But also that means that historically Kubernetes has been wide open for hackers to walk right into.
The cloud native computing foundation did a survey in 2017 that said Kubernetes was the biggest challenge security was the biggest challenge for Kubernetes users. And it that makes sense.
Because it is a challenge. There's a lot going on there. As well as Kubernetes itself being hard, all configurations have
they differ according to cloud provider, they differ according to user installation, and they differ according to whatever plugins and add ons that you're using.
And pretty much every Kubernetes cluster is going to be different. And because of that, making it like hardening
making it more secure, is going to vary really widely. And this has caused some problems. Well, Kubernetes gets hacked.
Tesla famously gets hacked. they're going to be the cautionary tale forever in every Kubernetes workshop for ever more.
Weight Watchers got hacked a few weeks ago on a server that didn't have a password at all. On open port 10250.
Which historically has been called the insecure API port. And spoiler alert, it's called the insecure API port because it was insecure.
Kubernetes 110 deprecated it. Kubernetes 110 was not that long ago. And a lot of these things still exist in the wild.
As any, you know, as anybody who has ever worked on any kind of production might know, what happens in production can sometimes stay in production.
And, you know, they're still out there for hackers to be able to get. So, security's hard. It's hard to secure it, it's hard to keep it secure, and it's easy to make mistakes.
Not only is it easy for us as developers and operations people to make mistakes, even hackers can make mistakes here. Because shit's difficult.
Besides which is kind of like DevOpsDays for the security community did a capture the flag competition a year ago, 2017.
And they left their backend wide open. So somebody who presumably had more experience with Kubernetes infrastructure than the people putting on the CTF walked right into the back of it.
Took all the flag then handed them to them and said you need to fix your API. It's not just you. We all make these mistakes.
Kubernetes has a very active community of contributors who are pushing changes very fast. And in recent versions, especially after like 1.6,
and especially in the last couple, security features have improved a lot. If you are still running 1.6, you really shouldn't be.
You should really update that. And, you know, if you can patch it, patch it. If you can update it, update it. It's really improved a lot.
That said, there's still a lot of older Kubernetes releases in prod. And they're wide open for people to discover.
How to attack those and how to defend different Kubernetes clusters varies the way you configure it. I have good news and bad news for you.
It is actually possible to secure Kubernetes clusters. The bad news is, it's not going to be secure by default.
The other bad news is, security doesn't actually end there. You have to make sure that your host, the platform it's on, your public cloud, are also secure.
So, you need to make sure that, for example, your kernel is safe from meltdown specter type attacks.
And you need to make sure that your public cloud has appropriate IAM roles applied to it. You can do this. But it'll take some work.
And you have to do this because there are bad people out there who actually do want to attack your stuff and take it. They have different motivations. But they're out there.
And they do have some things in common. Hackers don't give a shit. They don't give a shit that it's an internal system, they don't give shit about your business case.
They don't give a shit about you. That's your problem. They don't care about that. So, we water up here?
Leave that up for everybody to appreciate for a second. And different hackers do have different motivations.
Some of them might be motivated by money, some of them might be motivated by ideas, some might be motivated because they don't like you.
And your threat model is going to differ depending on what it is that you're afraid of. What is your threat model? It's important to consider it.
Are you concerned about external attackers from outside? Are you concerned about attackers specifically targeting you? Thanks.
How skilled are your adversaries? What are your capabilities in being able to protect against them? Targeted attackers are going to be more determined, possibly harder to stop.
Because they really want to get in to your system and attack you. And depending on their skill level, that can be complicated.
If you have nation states after you, you might be in deeper shit than your average fancy bear.
But if it's external attackers just looking for open ports on the Internet, a lot of the time they just want to do their thing and go.
And they're going to pick the easiest target with the lowest hanging fruit. If you aren't the easiest target with the lowest hanging fruit, they're not necessarily going to be interested.
There's a thousand other systems or million or whatever that they can come find. So, if you can be like, um, I'm not an attractive target.
That's good for you. That may be bad for me as an attacker. So, consider your threat model. What are you afraid of? What are you trying to protect?
In general Kubernetes threat model is external attackers, application or container compromise, or a compromised user or credentials.
And there's different ways to protect against all of these things. Generally speaking, regardless of her motivations,
every hacker is going to have roughly the same kind of workflow. They discover it if they're external. Or maybe they already know it's there if they're targeting you.
They are enumerating what is on that serve other what are whatever that they're looking at. What's running? What services are in there?
Are they patched? What versions are they running? Is there some vulnerable or CVE there that you can find?
And, you know, okay. So, is there anything exposed? What's running is privileged? What's connected to what?
Is there wider Internet access anywhere that I can go download some malware from and put it in there?
Are there credentials or secrets anywhere? You know, those kinds of things. Trying to figure it out. Once you figure out what's vulnerable, there is usually vulnerability.
You get the foothold, you do the hacker voice, I'm in thing. And you look around some more, if there's anything interesting in there.
If there's anything interesting you can steal. If there's anything interesting you can go. What can I do here? Anything I can loot or access?
And then you go from here and do the same thing again. Can I escalate my privileges? Can I get root? Can I get root on the host? Can I throw a crypto miner on it if?
Maybe. And see how far you can go and keep doing it as you get farther and farther. And eventually, you can't get farther anymore. But a lot of the time by then, it's already game over.
You want to make sure you're stopping hackers from being able to get from being able to get anywhere, really. But being able to get very far.
Because the farther that we are able to get, the more we're able to compromise and the less you're able to protect your own resources.
So, you know, if you have an if you're an external attacker and you're looking for things to attack, there are different ports that Kubernetes leaves open often by default that I might look for.
So, some of them could be things like the insecure API server. Or the Kubelet, which older versions of Kubelet were pretty much direct route into the container.
And to this day, depending on what cloud provider installation you're using can still be that way.
It can get around pod security policies and various other things if it is configured incorrectly.
So, I'm going to be looking for what's open. Older versions had everything in clear text in etcd. Including all passwords, database stuff, metadata for the cloud providers. I'm looking for that.
I want to go in, get what I can get. And you might think, I'm not I'm not a target. I might be too small. I might not be making, you know, like nothing in here matters.
You can't find me. You're wrong. Who here has ever heard of Shodan? Anybody? A few of you? Shodan is basically like a search engine for the Internet.
And you can go on there and search for things like open ports anywhere in the world. I did this search on Shodan last night. Because I wanted the most updated number.
As of last night this were 1975 open etcd ports on the wider Internet. A few of them I managed not to screenshot, had things like SSH keys and various other credentials.
I'm not personally interested in a CFAA case at this point in my life. I didn't go follow them. But you should know it's actually
just as easy as putting this thing in the search bar for people to find your SSH keys and other kinds of things if you're not securing them properly.
So, secure them properly. Don't leave your ports open if you don't have to. Shodan re-indexes every 24 hours. A lot of the SSH keys were several hours old.
Probably by the time I got around to seeing them other people had also seen them and gone about attacking them. It doesn't take long. So, just don't do it.
Just don't leave your stuff exposed. Make sure that if you do have anything exposed, visible, externally, that it requires authentication.
Because that's not necessarily the default. Especially with older versions of Kubernetes. Put TLS on absolutely everything. This is very important.
Because if it's cleartext, I have it. And limit SSH access to Kubernetes nodes. Because otherwise I can just wander right in there,
especially if you're doing things like leaving your SSH keys open on Shodan for all the world to see. And if you need SSH access to some of your nodes, you can create an SSH bastion
for the ones you do need. It's been said that defenders think in lists while attackers think in graphs.
And what's important to consider when thinking about securing your Kubernetes cluster is what's in your graph? This is a deceptively simple question.
Do you actually know everything that's in that cluster? Do you actually know everything that's on your public cloud act? You might not.
Especially in a big enough organization with silos or communication or process issues that can make it hard for people to know what other teams are doing.
It can be difficult or impossible to find that out. But, as an attacker, I'm determined to find that out. So, compliance might be satisfied with checking all the required boxes in the checklist.
But that isn't what I'm looking for when I'm in a network. I want to know what's in there. What's connected to what? Where I can go from here?
And for you as a defender, what you need to do is make sure that you know what microservices are communicating with each other, even within the cluster.
You need to be able to define and map out what exactly is talking to what and with which types of requests because that can allow you to define and lock down network communications.
Understand how your app operates. Understand how all of the services work together. And always remember the principle of least privilege.
The more that you limit information disclosure and things talking to one another and talking to the rest of the Internet
that don't need to be, the better shape you're going to be in and the worse shape I'm going to be in as an attacker.
So, another thing that I can do as an attacker is trying to compromise credentials or users. Computers are dumb pipes. if I have your credentials as far as the computer is concerned,
I'm probably you. So, there's a few different ways to do this. If I am trying to social engineer a developer, which developers are actually great targets
because they don't think that they're targets. They have a lot of access to things like production servers and keys and, you know, cloud metadata.
And because they don't think that they're targets, they're actually more likely to click on phishing links than marketing or sales. We should all be embarrassed about this.
So, social engineering can totally happen. If I, you know, manage to convince Joe Developer that, you know, this link is very important to click on, fantastic. I've got it.
If I can publish a helpful blog post saying, like, oh, here's this really hard thing to do with service meshes. Nobody understands the service mesh.
Like, let's be real. So, here's this really difficult thing that very few people actually know how to do. And the way that you do this is you just install my Docker image.
Kubectl create-f evilscript.yml Get my image down from Docker pull. It will walk you through the tutorial. It will be fantastic. No. Because actually it will be fantastic for me. It won't be fantastic
for you. And it's really important to not fall for social engineering. And it's really important to, like, know what your image is and what your code is doing.
Don't just throw like a YAML file into your cluster without actually knowing what it does. Actually read the code. Understand it all the way and look closely at it
because we can do things like kern our letters such that it's actually different letters or obfuscate it so that it looks fine but it isn't.
Limit access to your credentials and don't fall for any of my crap. Containers are easy to compromise. And container compromise can lead to bigger attack surfaces.
And those bigger attack surfaces can expand beyond containers and into hosts and into public clouds.
The Kubernetes control plane contains various kinds of vulnerabilities that are really important to know about.
The API server, as you can see, connects to Kube CTL, the master node and the worker node. And it if you have the API server, you have it all. And the API server historically is wide open.
The defaults are such that you can, you know, you can just wander in there. That's not necessarily the case anymore. But you need to be updating and make sure that you're
configuring your stuff correctly. etcd has all of the everything for Kubernetes in it. It has all of the things that control all of the things. And it has database information,
it has credentials, it has everything else. Historically it's plain text. If you have etcd, you have the rest of it. They are able to be controlled by Kube control.
They are able to be controlled in the cluster if you have access to open Kube control. The way that this gets prevented is by limiting communication and
movement between different parts of the cluster. So, lock your network down. Lock your cluster down. And make sure that you are only
allowing things to communicate that that should be. Really just keep it on a need to know basis. Because if you don't keep it on a need to know basis, shit can happen.
A cool thing about shared kernels is that they're also surfaces for shared kernel attacks. So, if I have meltdown or specter on one part of your container or, you know, on a container, anywhere,
then I also have meltdown or specter on the rest of the, you know, containers and the host that it's on. And if I have root in a cluster or container that's running as privileged,
ya'll, don't run your containers or your clusters as privileged. Just don't. It's a bad idea. Because the way that the entirety of the Linux file
system works is everything is based on namespaces. Everything is a file. And so, if I have access to the namespace, root in that cluster can also be root on the host.
Especially if you're mounting something on the host into the cluster which is a terrible idea. Especially if something is running privileged.
Because I can just wander into the rest of the cluster with my write access and write stuff up in there and also read it and also take it.
You can prevent container compromise in various ways. Write your own applications as securely as possible. I know it's hard. We have a lot going on. But try.
Look into resources like the OS top 10 and other kinds of secure code resources. Try to figure out how to do that well. Treat other people's code with caution.
As I said, like, you know, watch out for evil.yml files. Make sure that your dependencies and the images that you're dealing with are, you know, are actually good and not bad.
Run static code analysis on your applications and containers to check for vulnerabilities and do that periodically. Because only doing it once is just a timestamp for that moment in time.
So, if six months ago that was fine, but 24CVs have come out since that affect things in your cluster, it's probably not fine anymore. You have to keep doing that.
There are tools that are open source Clair by CoreOS is a good one. And patch or mitigate them. Immutability is a feature, not a bug, most of the time.
But if it means that every time malware knocks the container down, it comes back up with the same malware, that's not necessarily a positive thing.
Basically you want to practice defense in depth. You want to reduce your attack surface to make it so that there are as few ways for me to get in as possible.
Lockdown your networking, lockdown your secrets. You know, restrict communication. Put quotas on your resources. And just generally make it so that I'm going to have a hard time
getting in there. And if I am in there, you want to limit your blast radius. So, make sure that I can't get as far as I can so that I'm not getting into the rest of your cluster.
I'm not getting into your host, I'm not getting into your cloud account. If you can, you know, make it so that it's hard for me to get in and make it so that I can't go anywhere.
I'm going to have a lot less fun as a hacker, and you're going to have a better day as a defender. I pushed the wrong button. And that was the laser pointer.
Sorry if I pointed that at you. And follow the principle of least privilege. I said this more than once. But it's really important.
Because it makes it so that I can't escalate if I don't have it. And I want to be able to escalate, but you probably don't want me to.
There are different kinds of controls that you can use to do these things, which I have basically no time to talk about.
So, I'm putting a list of stuff here and you should take a picture of it or whatever and then Google all of them later. And look at the things on the resource slide about how to do that.
Because every single one of these is an hour long talk, at least, in and of itself. You want to, as I said, put TLS on your APIs.
imit access to those APIs, especially if they're insecure to the Kubelet, and limit control. Control what pods nodes are allowed to access. I pushed the wrong button again.
And you want to make sure that your keys are being rotated, that your secrets are being encrypted that that you are not having things that I have persistent access to.
Because if I can put a back door in your container, in your cluster, and it's just chilling there, I can keep going in there but you probably don't want that.
So, some general recommendations are, if you can upgrade to a newer Kubernetes version. I know backwards compatibility is often hard.
And in enterprise can often be complicated for other reasons, but really if you can, please do that. Especially if you're running anything 1.6 and below.
Everything by default is wide on open in 1.6 and below. Even after 1.6, even in recent versions, everything is not necessarily secure by default.
Especially with things added on to it like Helm Tiller, COPS, things that are fairly commonly used. Are wide open by default with no authorization required.
You have to put that there. If you're using those things, make sure to put them there.
Secure defaults, if you walk out with one thing here, are really, really important. Because once that default is there, once it's running in production, once people are used to it being there,
then it becomes a problem to do anything else with it. And add secure features. And then you just stay insecure.
And if you can make your defaults secure out of the box, then you don't have to worry about that kind of compatibility later.
Be careful with your secrets and your credentials where you put them. And make sure that you're logging and monitoring and paying attention to what's going on.
Observability, as we know as DevOps people, is really important. But your logging and monitoring should not actually
go inside the same cluster that I have access to because I'm just going to delete it and then you won't know that I've been there.
So, you want to make sure that those audit logs and monitoring are happening, or at least being doubled somewhere else so that you can have a more accurate record that I, as a hacker,
don't have access to. The TL;DR is basically a lot of this is really basic security advice.
It's like old school IP tables firewalls. It's what we have heard before. Defense and depth. Principle of least privilege.
The ideas of limiting your attack surface. This advice keeps being given over and over because this advice is important.
And we should all be doing it. Unfortunately we're not always doing it. Which makes things great for hackers but might make things harder for you.
So, do those things. Practice good cyber hygiene. Get your basics right. If you can make it so that it's harder for me to get in there, you are making it so that you're that much more secure
where you are. And you got this. I believe in you. You can do it. Here's my resource slide. There's a bunch of stuff on there.
And there's a lot there. The CIS benchmarks are a 276 page document. But it's worth looking into. And it's worth getting into these things in more depth
because it does take work to configure this stuff. And you can figure it out. I believe in you. And that's it.
[ Applause ]
For more infomation >> Father who lost daughter in boating accident isn't giving up on passing 'Kali's Law' - Duration: 2:10. 
For more infomation >> Cleaning Up Storm Damage - Duration: 2:00. 
For more infomation >> In The Bender Kitchen: Raquel Rivera-Pablo cooking up interesting dish with eggs and tomatillo salsa - Duration: 4:19. 
For more infomation >> HSN | Fashion & Accessories Clearance Frenzy Up To 60% Off 08.01.2018 - 03 AM - Duration: 1:00:00. 
For more infomation >> Fireworks Investigated After Apartment Building Goes Up In Flames - Duration: 2:44. 
For more infomation >> Should investors give up on the tech sector? - Duration: 3:02. 
Không có nhận xét nào:
Đăng nhận xét