Posted July 14, 2019 by Alexander Ullah1 Comment Kubernetes

PKS 1.4 Pod Security Policies

Pod Security Policies are available with VMware Enterprise PKS since version 1.4 and are considered a beta state feature of Kubernetes. This is true for Kubernetes version 1.13.5, which is the version shipped with Enterprise PKS 1.4, as well as version 1.15, which is the latest release of Kubernetes as of the time writing this blog post. But what are Pod Security Policies really and how do they work? I will try to answer some of the questions around Pod Security Policies in this article with simple examples.

Overview

With Pod Security Policies or short PSP, an Administrator or Platform Reliability Engineer can control certain security-related aspects of pod specifications. Such as execution of privileged containers, usage of volume types or host filesystem, usage of host networking and ports, and many more. Visit the following link of the official Pod Security Policies documentation page.

https://kubernetes.io/docs/concepts/policy/pod-security-policy/.

Pod Security Policies are enabled via an Admission Controller. Admission Controllers are intercepting Kubernetes API calls and are validating or mutating the requests. If a Controller rejects a request, it fails and the end-user gets an error. If you don’t have any policy created and authorized, the admission controller will prevent pods from being created. You can learn more about Admission Controllers on the Kubernetes documentation page.

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/.

Enable PSP in PKS

In the context of VMware Enterprise PKS, the Admission Plugin/Controller can be enabled per Plan via the configuration of the PKS tile within the Pivotal OpsManager. But be careful, you should not easily enable the plugin for existing Kubernetes Clusters as this would cause pods to fail if you don’t have the right policies created and authorized upfront.

Don’t forget to save and apply your changes! Otherwise, the Admission Controller will not be enabled on the PKS Kubernetes clusters. PKS will pre-create two policies (pks-privileged and pks-restricted) on a PSP enabled Kubernetes cluster. These policies can be used right away or you can create your own policies.

PSP	Description
PKS Privileged	Allows privileged access to pod containers, which allows the container to do almost everything a host can do. See Privileged in the Kubernetes PSP documentation for more information.
PKS Restricted	Restricts privileged access to pod containers.

We are not going to use these policies in the following example as I want to describe the policy creation process as well. Please read through the following PKS documentation to get more details on the pre-created policies.

https://docs.pivotal.io/runtimes/pks/1-4/pod-security-policy.html

I have already created a PKS Kubernetes cluster with Pod Security Policy Admission Controller enabled. So let’s see how we can make use of Pod Security Policies.

Example

In this example, we are going to create a very simple policy that prevents the execution of privileged pods. Here is how the yaml definition of such a policy looks like.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: yelbpsp
spec:
  privileged: false  # Don't allow privileged pods!
  # The rest fills in some required fields.
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'

1. Create a Policy

As a first step, we will create the Pod Security Policy via kubectl.

➜  ~ kubectl apply -f pspyelb.yaml
podsecuritypolicy.policy/yelbpsp created
➜  ~ kubectl get psp
NAME                    PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
a-wavefront-psp         false   *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            secret,hostPath
cert-generator          false          RunAsAny   MustRunAsNonRoot   RunAsAny    RunAsAny    false            secret
event-controller        false          RunAsAny   MustRunAsNonRoot   RunAsAny    RunAsAny    false            emptyDir,secret
fluent-bit              false          RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            hostPath,configMap,emptyDir,secret
kube-system-psp         false   *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            configMap,emptyDir,projected,secret,downwardAPI
metric-controller       false          RunAsAny   MustRunAsNonRoot   RunAsAny    RunAsAny    false            secret
observability-manager   false          RunAsAny   RunAsAny           RunAsAny    RunAsAny    false
pks-privileged          true    *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume
pks-restricted          false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
sink-controller         false          RunAsAny   MustRunAsNonRoot   RunAsAny    RunAsAny    false            secret
telegraf                false          RunAsAny   MustRunAsNonRoot   RunAsAny    RunAsAny    false            configMap,secret
telemetry-agent         false          RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            configMap,secret
validator               false          RunAsAny   MustRunAsNonRoot   RunAsAny    RunAsAny    false            secret
yelbpsp                 false          RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *

But creating the policy is not enough. If you now try to create a Kubernetes Deployment, you will get an error message such as this “Error creating: pods “yelb-ui-69c7745b49-” is forbidden: unable to validate against any pod security policy”.

Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  19s (x13 over 40s)  replicaset-controller  Error creating: pods "yelb-ui-69c7745b49-" is forbidden: unable to validate against any pod security policy: []

2. Create a Role

No pod can be created because we still don’t have any Kubernetes Role assigned to the policy. Let’s create a role which we can use for our demo.

➜  ~ kubectl create role psp:denyprivileged --verb=use --resource=podsecuritypolicy --resource-name=yelbpsp --namespace=yelb
role.rbac.authorization.k8s.io/psp:denyprivileged created
➜  ~ kubectl get role -n yelb
NAME                 AGE
psp:denyprivileged   8s

3. Create a Rolebinding

As a next step, we need to create a Kubernetes RoleBinding to bind the role to a service account. I have already pre-created a Kubernetes Service Account named “yelbsa” in the namespace “yelb” with the ClusterRoleBinding “editor”.

➜  ~ kubectl create rolebinding yelbsa:psp:denyprivileged --role=psp:denyprivileged --serviceaccount=yelb:yelbsa --namespace=yelb
rolebinding.rbac.authorization.k8s.io/yelbsa:psp:denyprivileged created
➜  ~ kubectl get rolebindings -n yelb
NAME                        AGE
yelbsa:psp:denyprivileged   15s

Now we have everything created and we should validate that the service account “yelbsa” can use the Pod Security Policy “yelbpsp” with the following command.

➜  ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb auth can-i use podsecuritypolicy/yelbpsp
yes

So everything “should” be fine, let’s try to create our yelb test application as “yelbsa” again.

➜  ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb apply -f yelb.yaml
service/redis-server created
service/yelb-db created
service/yelb-appserver created
service/yelb-ui created
deployment.extensions/yelb-ui created
deployment.extensions/redis-server created
deployment.extensions/yelb-db created
deployment.extensions/yelb-appserver created
➜  ~ kubectl get rs -n yelb
NAME                        DESIRED   CURRENT   READY   AGE
redis-server-664f84ccd8     1         0         0       15s
yelb-appserver-594975d7b4   1         0         0       15s
yelb-db-5c5f7bd8bf          1         0         0       15s
yelb-ui-69c7745b49          1         0         0       15s
➜  ~ kubectl describe rs yelb-ui-69c7745b49 -n yelb
Name:           yelb-ui-69c7745b49
Namespace:      yelb
...
Events:
  Type     Reason        Age                From                   Message
  ----     ------        ----               ----                   -------
  Warning  FailedCreate  2s (x14 over 43s)  replicaset-controller  Error creating: pods "yelb-ui-69c7745b49-" is forbidden: unable to validate against any pod security policy: []
➜  ~

It is still not working! The reason is simple, I haven’t specified the service account in the yaml definition and therefore Kubernetes would try to use the “default” account to create the ReplicaSet, which has no Pod Security Policy assigned.

4. Adjust YAML Manifest

In order to fix it, we will add the service account to every deployment of the yaml definition, see the following snippet:

...
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: yelb-appserver
  namespace: yelb
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: yelb-appserver
        tier: middletier
    spec:
      containers:
      - name: yelb-appserver
        image: mreferre/yelb-appserver:0.3
        ports:
        - containerPort: 4567
      serviceAccountName: yelbsa
...

Fingers crossed, we should be ready now! Let’s try it one more time.

➜  ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb apply -f yelbsa.yaml
service/redis-server created
service/yelb-db created
service/yelb-appserver created
service/yelb-ui created
deployment.extensions/yelb-ui created
deployment.extensions/redis-server created
deployment.extensions/yelb-db created
deployment.extensions/yelb-appserver created
➜  ~ kubectl get rs -n yelb
NAME                        DESIRED   CURRENT   READY   AGE
redis-server-67b84dbcc6     1         1         1       9s
yelb-appserver-7fb9f7dbb4   1         1         1       9s
yelb-db-6f6b64f89           1         1         1       9s
yelb-ui-dfb45f485           1         1         1       9s
➜  ~ kubectl get pods -n yelb
NAME                              READY   STATUS    RESTARTS   AGE
redis-server-67b84dbcc6-qzs7k     1/1     Running   0          14s
yelb-appserver-7fb9f7dbb4-xgxkf   1/1     Running   0          14s
yelb-db-6f6b64f89-8rk8s           1/1     Running   0          14s
yelb-ui-dfb45f485-ftgd7           1/1     Running   0          14s

Success! All the pods are up and running. We can create pods as the service account has a valid Pod Security Policy assigned and there is no privileged pod defined in the yaml manifest. As the last step, I would like to validate that the Pod Security Policy Admission Controller will truly prevent privileged pods from running as specified in our policy.

5. Test the Policy

For our last test, I will simply add the privileged security context to one of the deployments in the yaml manifest. See the following snippet:

...
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: yelb-ui
  namespace: yelb
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: yelb-ui
        tier: frontend
    spec:
      containers:
      - name: yelb-ui
        image: mreferre/yelb-ui:0.3
        ports:
        - containerPort: 80
        securityContext:
          privileged: true
      serviceAccountName: yelbsa
...

I have added the security context to the yelb-ui deployment. Let’s see what happens if we deploy the manifest again.

➜  ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb apply -f yelbsapriv.yaml
service/redis-server created
service/yelb-db created
service/yelb-appserver created
service/yelb-ui created
deployment.extensions/redis-server created
deployment.extensions/yelb-db created
deployment.extensions/yelb-appserver created
The Deployment "yelb-ui" is invalid: spec.template.spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy

As you can see, the Pod Security Policy Admission Controller is preventing the yelb-ui deployment from being created and everything is working as expected.

Conclusion

Pod Security Policies are a great way to control security-related aspects of Kubernetes Pods and to ensure only certain privileges are provided. Even though this was just a very simple example, I hope it demonstrated how useful PSPs can be for Cluster Administrators or Platform Reliability Engineers who are tasked with security-related duties. VMware Enterprise PKS customers can easily enable the PSP Admission Controller and make use of the pre-created policies (pks-privileged and pks-restricted) or create their own policies. I can only encourage everyone to follow the upstream development of Pod Security Policies as well as to try it out yourself.

Additional Sources

Alexander Ullah View All

One thought on “PKS 1.4 Pod Security Policies” Leave a comment ›

Pingback: Verify and Troubleshoot vSphere 7 with Kubernetes – Beyond Elastic