Secure the Home! Making sense of K8s Security.

For those already in Salt Lake City ready for this years’ KubeCon North America, I’m jealous!

I’ll be sitting this one out this year, but it does give me time to sit at a real desk and talk Kubernetes security from a distance! (plus, my soda fridge doesn’t dissapear at specific times thoughout the day.)

While I’m sure this years conf will be dominated by AI and how Kubernetes supports those workloads supporting AI workloads. Security should still be top of mind for teams responsible for Kubernetes deployments. With SO MANY WAYS to deploy kubernetes even within a single cloud provider, running a comprehensive set of Kubernetes security checks can be a good way to normalise all those environments to ensure safe configuration and compliance, enhancing security.

Let’s break down why these checks are important and how tools like Prowler can get you up and running!

I live in a house, a very big house, on some servers!

Think of your Kubernetes cluster as a large house (sometimes even a Castle) with many, many rooms, each representing different components and services.

Each room has doors (service ports, ingresses), windows (metrics, logs, telemetry), valuable items (app data, secrets) and probably some wiring and plumbing (network connections, sidecars).

If you leave a door unlocked or a window open, it could allow intruders to enter the house and steal or damage your belongings. Pushing this analogy to it’s absolute limit, even leaving the blinds open in some neighborhoods may allow a potential theif to formulate a better plan of attack!

Comprehensive checks are like having a security team that regularly inspects each door and window to ensure they are locked, that there are no issues with the walls and wiring, and that only authorized people can access certain rooms. This way, you protect your house and all the wonderful little pods that reside there!

Layered Security, who doesn’t love a moat?

In cloud environments, security should be implemented in layers, much like how you would protect a physical building.

• Foundational Layers (Wider Cloud Environment): Kubernetes clusters don’t exist in a vaccum, the cloud environment you reside in is a whole neighborhood of IAM policies, inbound IP addresses and logging buckets. These items can quickly undermine everything you build on top if not also carefully considered.
• Outer Layers (Endpoint and Network Security): Ingress policies that control who can access your Kubernetes cluster from outside and network policies to ensure no unecessary outbound or service-to-service access.
• Middle Layers (Service Security): Ensuring that each service within your cluster is securely configured, not only end-user services, but also the Kubernetes services themselves (i’m guessing in this analogy the house is big enough to have a ‘plant room’ where etcd and the Kubernetes API server reside!)
• Middle Layers (Runtime Security): Ensuring our workloads aren’t putting the wider cluster at risk, not allowing pods to run as root, not allowing capabilities that pods don’t need and may allow lateral movement for an attacker. Same goes with overly permissive RBAC policies for pod Kubernetes API access.
• Inner Layers (Data Security): Protecting sensitive data with encryption and strict access controls, and relevant backup strategies.

Check all of the things!

The checks shown below help you secure each of these layers, with Prowler having hundreds of built-in policies for each area of focus.

For each area, we’re only going to look at high-severity items, we’re filtering these in prowler with --severity=high as you’ll see in the command outputs below!

Foundational Layer Checks (Wider Cloud Environment) :

Lets take Amazon Web Services (AWS) as our example cloud here. Listing all AWS-related checks with high severity in prowler gives us a number of sane items to check for in our underlying cloud configuration:

prowler aws --list-checks --severity=high

Firstly, lets address IAM, the absolutely most critical part of your AWS infrastructure to get right, as it provides access to your cloud environment, potentially from within VM’s, containers, or even other accounts if not configured properly.

We see a number of high severity checks in order to highlight any deficiencies in our IAM setup, these are most definitley recommended!

Another good policy is to ensure SSM, often used to store secrets for cloud configuration does not expose any public documents, and to ensure S3 buckets, often used as logging and data backup destinations within an AWS hosted kubernetes environment, are not public.

Finally, for those using the built-in EKS type of Kubernetes cluster, there are even checks to ensure this is configured with security settings enabled.

Outer Layer Checks (Endpoint and Network Security) :

prowler kubernetes --list-checks --severity=high

Into our Kubernetes cluster, knowing we have not left our foundations exposed, you will see different categories within the Kubernetes checks in Prowler, from checks protecting the APIServer and other internal components (such as the Kubelet), to more ‘runtime’ checks looking for misconfigured RBAC and admission policies of insecure workloads.

Below we see checks ensuring we have hardened our API Server, preventing anonymous auth, enabling logging and disabling ‘wide-open’ configuration options.

Middle Layer Checks (Services and Runtime Security):

As mentioned above, our Kubernetes checks continue into the RBAC category, ensuring we are not giving workloads within our cluster more permissions than they need, especially in these high-severity items, we are looking to prevent access to features and roles commonly used in lateral movement following initial compromise, such as cluster-admin roles, the ability to sign certificates, run attacker workloads as new pods, and access secrets.

The “Core” category looks at workloads *on top* of our Kubernetes cluster, ensuring we have an admission policy that prevents insecure containers being started, such as containers trying to run as UID root or with access to the host network (NET_RAW).

Inner Layer Checks (Data Security):

Last but definitley not least, we see throughout the categories items designed to protect data, such as secrets, for example in the RBAC check rbac_minimize_secret_access and forcing use of APIServer config encryption with check apiserver_encryption_provider_config_set.

Why Prowler?

Prowler is a powerful, 100% open, cloud security tool that further enhances your cloud security by automating checks across all of these boundaries, and for multiple cloud providers.

Prowler performs a comprehensive set of more than 80 checks specifically designed for Kubernetes, and over 500 in total; Covering various aspects of security, including configuration, access control, network policies, and more.

Think of Prowler as an advanced security consultant who not only patrols your house but also uses sophisticated tools to scan for hidden vulnerabilities that might not be immediately obvious. Prowler checks whether your doors are reinforced, whether the locks are strong enough, and whether your security cameras are functioning correctly. It’s like having an extra layer of vigilance that ensures your cloud environment and Kubernetes clusters are not just secure but also compliant with industry standards.

Preventing Drift

One of the key challenges in managing Kubernetes environments is configuration drift—where settings gradually become less secure over time as updates are made. Comprehensive checks, like those performed above, help detect and alert to these downgrades in security posture. Ensuring that your cluster remains secure over time, just like how regular maintenance of a house prevents deterioration.

Compliance and Best Practices

Finally, many of these checks ensure compliance with industry best practices and regulations, prowler can also filter by these standards such as SOC2 or CIS Kubernetes 1.7. Running these check subsets helps you not only protect your infrastructure but also meet legal and contractual obligations, reducing the risk of penalties or breaches.

prowler kubernetes --list-compliances
prowler aws --list-compliances

Summary

Comprehensive checks through tools like Prowler, are like having a vigilant, security team that can be instructed to constantly patrol your cloud infrastructure.

They ensure that every component is locked down, secure, and functioning as it should. Without these checks and the automation, your leave your cloud environment vulnerable to a potential array of attacks, broader access, or data leakage: much like an unguarded house!

Have fun in SLC, and stay Compliant!