PKS 1.4 Pod Security Policies
Pod Security Policies are available with VMware Enterprise PKS since version 1.4 and are considered a beta state feature of Kubernetes. This is true for Kubernetes version 1.13.5, which is the version shipped with Enterprise PKS 1.4, as well as version 1.15, which is the latest release of Kubernetes as of the time writing this blog post. But what are Pod Security Policies really and how do they work? I will try to answer some of the questions around Pod Security Policies in this article with simple examples.
With Pod Security Policies or short PSP, an Administrator or Platform Reliability Engineer can control certain security-related aspects of pod specifications. Such as execution of privileged containers, usage of volume types or host filesystem, usage of host networking and ports, and many more. Visit the following link of the official Pod Security Policies documentation page.
Pod Security Policies are enabled via an Admission Controller. Admission Controllers are intercepting Kubernetes API calls and are validating or mutating the requests. If a Controller rejects a request, it fails and the end-user gets an error. If you don’t have any policy created and authorized, the admission controller will prevent pods from being created. You can learn more about Admission Controllers on the Kubernetes documentation page.
Enable PSP in PKS
In the context of VMware Enterprise PKS, the Admission Plugin/Controller can be enabled per Plan via the configuration of the PKS tile within the Pivotal OpsManager. But be careful, you should not easily enable the plugin for existing Kubernetes Clusters as this would cause pods to fail if you don’t have the right policies created and authorized upfront.
Don’t forget to save and apply your changes! Otherwise, the Admission Controller will not be enabled on the PKS Kubernetes clusters. PKS will pre-create two policies (pks-privileged and pks-restricted) on a PSP enabled Kubernetes cluster. These policies can be used right away or you can create your own policies.
|PKS Privileged||Allows privileged access to pod containers, which allows the container to do almost everything a host can do. See Privileged in the Kubernetes PSP documentation for more information.|
|PKS Restricted||Restricts privileged access to pod containers.|
We are not going to use these policies in the following example as I want to describe the policy creation process as well. Please read through the following PKS documentation to get more details on the pre-created policies.
I have already created a PKS Kubernetes cluster with Pod Security Policy Admission Controller enabled. So let’s see how we can make use of Pod Security Policies.
In this example, we are going to create a very simple policy that prevents the execution of privileged pods. Here is how the yaml definition of such a policy looks like.
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: yelbpsp spec: privileged: false # Don't allow privileged pods! # The rest fills in some required fields. seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny volumes: - '*'
1. Create a Policy
As a first step, we will create the Pod Security Policy via kubectl.
➜ ~ kubectl apply -f pspyelb.yaml podsecuritypolicy.policy/yelbpsp created ➜ ~ kubectl get psp NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES a-wavefront-psp false * RunAsAny RunAsAny RunAsAny RunAsAny false secret,hostPath cert-generator false RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false secret event-controller false RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false emptyDir,secret fluent-bit false RunAsAny RunAsAny RunAsAny RunAsAny false hostPath,configMap,emptyDir,secret kube-system-psp false * RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI metric-controller false RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false secret observability-manager false RunAsAny RunAsAny RunAsAny RunAsAny false pks-privileged true * RunAsAny RunAsAny RunAsAny RunAsAny false awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume pks-restricted false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim sink-controller false RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false secret telegraf false RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false configMap,secret telemetry-agent false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,secret validator false RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false secret yelbpsp false RunAsAny RunAsAny RunAsAny RunAsAny false *
But creating the policy is not enough. If you now try to create a Kubernetes Deployment, you will get an error message such as this “Error creating: pods “yelb-ui-69c7745b49-” is forbidden: unable to validate against any pod security policy”.
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 19s (x13 over 40s) replicaset-controller Error creating: pods "yelb-ui-69c7745b49-" is forbidden: unable to validate against any pod security policy: 
2. Create a Role
No pod can be created because we still don’t have any Kubernetes Role assigned to the policy. Let’s create a role which we can use for our demo.
➜ ~ kubectl create role psp:denyprivileged --verb=use --resource=podsecuritypolicy --resource-name=yelbpsp --namespace=yelb role.rbac.authorization.k8s.io/psp:denyprivileged created ➜ ~ kubectl get role -n yelb NAME AGE psp:denyprivileged 8s
3. Create a Rolebinding
As a next step, we need to create a Kubernetes RoleBinding to bind the role to a service account. I have already pre-created a Kubernetes Service Account named “yelbsa” in the namespace “yelb” with the ClusterRoleBinding “editor”.
➜ ~ kubectl create rolebinding yelbsa:psp:denyprivileged --role=psp:denyprivileged --serviceaccount=yelb:yelbsa --namespace=yelb rolebinding.rbac.authorization.k8s.io/yelbsa:psp:denyprivileged created ➜ ~ kubectl get rolebindings -n yelb NAME AGE yelbsa:psp:denyprivileged 15s
Now we have everything created and we should validate that the service account “yelbsa” can use the Pod Security Policy “yelbpsp” with the following command.
➜ ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb auth can-i use podsecuritypolicy/yelbpsp yes
So everything “should” be fine, let’s try to create our yelb test application as “yelbsa” again.
➜ ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb apply -f yelb.yaml service/redis-server created service/yelb-db created service/yelb-appserver created service/yelb-ui created deployment.extensions/yelb-ui created deployment.extensions/redis-server created deployment.extensions/yelb-db created deployment.extensions/yelb-appserver created ➜ ~ kubectl get rs -n yelb NAME DESIRED CURRENT READY AGE redis-server-664f84ccd8 1 0 0 15s yelb-appserver-594975d7b4 1 0 0 15s yelb-db-5c5f7bd8bf 1 0 0 15s yelb-ui-69c7745b49 1 0 0 15s ➜ ~ kubectl describe rs yelb-ui-69c7745b49 -n yelb Name: yelb-ui-69c7745b49 Namespace: yelb ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 2s (x14 over 43s) replicaset-controller Error creating: pods "yelb-ui-69c7745b49-" is forbidden: unable to validate against any pod security policy:  ➜ ~
It is still not working! The reason is simple, I haven’t specified the service account in the yaml definition and therefore Kubernetes would try to use the “default” account to create the ReplicaSet, which has no Pod Security Policy assigned.
4. Adjust YAML Manifest
In order to fix it, we will add the service account to every deployment of the yaml definition, see the following snippet:
... apiVersion: extensions/v1beta1 kind: Deployment metadata: name: yelb-appserver namespace: yelb spec: replicas: 1 template: metadata: labels: app: yelb-appserver tier: middletier spec: containers: - name: yelb-appserver image: mreferre/yelb-appserver:0.3 ports: - containerPort: 4567 serviceAccountName: yelbsa ...
Fingers crossed, we should be ready now! Let’s try it one more time.
➜ ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb apply -f yelbsa.yaml service/redis-server created service/yelb-db created service/yelb-appserver created service/yelb-ui created deployment.extensions/yelb-ui created deployment.extensions/redis-server created deployment.extensions/yelb-db created deployment.extensions/yelb-appserver created ➜ ~ kubectl get rs -n yelb NAME DESIRED CURRENT READY AGE redis-server-67b84dbcc6 1 1 1 9s yelb-appserver-7fb9f7dbb4 1 1 1 9s yelb-db-6f6b64f89 1 1 1 9s yelb-ui-dfb45f485 1 1 1 9s ➜ ~ kubectl get pods -n yelb NAME READY STATUS RESTARTS AGE redis-server-67b84dbcc6-qzs7k 1/1 Running 0 14s yelb-appserver-7fb9f7dbb4-xgxkf 1/1 Running 0 14s yelb-db-6f6b64f89-8rk8s 1/1 Running 0 14s yelb-ui-dfb45f485-ftgd7 1/1 Running 0 14s
Success! All the pods are up and running. We can create pods as the service account has a valid Pod Security Policy assigned and there is no privileged pod defined in the yaml manifest. As the last step, I would like to validate that the Pod Security Policy Admission Controller will truly prevent privileged pods from running as specified in our policy.
5. Test the Policy
For our last test, I will simply add the privileged security context to one of the deployments in the yaml manifest. See the following snippet:
... apiVersion: extensions/v1beta1 kind: Deployment metadata: name: yelb-ui namespace: yelb spec: replicas: 1 template: metadata: labels: app: yelb-ui tier: frontend spec: containers: - name: yelb-ui image: mreferre/yelb-ui:0.3 ports: - containerPort: 80 securityContext: privileged: true serviceAccountName: yelbsa ...
I have added the security context to the yelb-ui deployment. Let’s see what happens if we deploy the manifest again.
➜ ~ kubectl --as=system:serviceaccount:yelb:yelbsa -n yelb apply -f yelbsapriv.yaml service/redis-server created service/yelb-db created service/yelb-appserver created service/yelb-ui created deployment.extensions/redis-server created deployment.extensions/yelb-db created deployment.extensions/yelb-appserver created The Deployment "yelb-ui" is invalid: spec.template.spec.containers.securityContext.privileged: Forbidden: disallowed by cluster policy
As you can see, the Pod Security Policy Admission Controller is preventing the yelb-ui deployment from being created and everything is working as expected.
Pod Security Policies are a great way to control security-related aspects of Kubernetes Pods and to ensure only certain privileges are provided. Even though this was just a very simple example, I hope it demonstrated how useful PSPs can be for Cluster Administrators or Platform Reliability Engineers who are tasked with security-related duties. VMware Enterprise PKS customers can easily enable the PSP Admission Controller and make use of the pre-created policies (pks-privileged and pks-restricted) or create their own policies. I can only encourage everyone to follow the upstream development of Pod Security Policies as well as to try it out yourself.
One thought on “PKS 1.4 Pod Security Policies” Leave a comment ›