GitOps with FluxCD and Tanzu Packages
I often get asked how to install our Tanzu Packages in a GitOps fashion instead of using the Tanzu CLI or TMC Catalog. I want to quickly demonstrate how this can be done in this blog post. If you are not familiar with Tanzu Packages, I recommend reading my blog post about Tanzu Packages to understand the basic concepts. For this example, I will use FluxCD as the Continuous Deployment (CD) tool. FluxCD is an incubating project at the Cloud Native Computing Foundation (CNCF). However, to use a GitOps approach to deploy Tanzu Packages, other CD tools such as ArgoCD can be used too.
Install FluxCD
First of all, let’s install the FluxCD CLI via brew. You can also use different installation methods as described here. As a side note, FluxCD is also available as Tanzu Community Edition (TCE) Package, which can be found here. However, I will not use the TCE Package in this scenario.
brew install fluxcd/tap/fluxWe can run the pre-check to verify if our Tanzu Kubernetes Grid (TKG) cluster is suitable for FluxCD. Just run the following flux command while using the kubectl context of the cluster you want to use.
➜ ~ flux check --pre
► checking prerequisites
✔ Kubernetes 1.21.6+vmware.1 >=1.20.6-0
✔ prerequisites checks passedTo bootstrap FluxCD, we are going to specify our GITHUB config as follows:
export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>You can install FluxCD on the TKG cluster and configure it to manage itself from a Git repository using the flux bootstrap command.
➜ ~ flux bootstrap github \
--owner=$GITHUB_USER \
--repository=fleet-infra \
--branch=main \
--path=./clusters/my-cluster \
--personalWe will see the FluxCD resources being created on our Kubernetes cluster.
➜ ~ k get pods,svc -n flux-system
NAME READY STATUS RESTARTS AGE
pod/helm-controller-88f6889c6-sp4mp 1/1 Running 0 86s
pod/kustomize-controller-784bd54978-hg2j8 1/1 Running 0 86s
pod/notification-controller-648bbb9db7-h7bxr 1/1 Running 0 85s
pod/source-controller-79f7866bc7-wkxf5 1/1 Running 0 85s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notification-controller ClusterIP 195.48.118.105 <none> 80/TCP 88s
service/source-controller ClusterIP 195.59.108.18 <none> 80/TCP 88s
service/webhook-receiver ClusterIP 195.48.100.248 <none> 80/TCP 87sGreat FluxCD is running, and you will have a new private Git repository, “fleet-infra”. This repository will be used to manage your cluster. Now we need to clone the repository locally to use it:
git clone https://github.com/$GITHUB_USER/fleet-infra
cd fleet-infraUse FluxCD for Tanzu Packages
Easy, right? Next, we want to configure FluxCD to connect to a Git repo that includes the necessary manifests for our Tanzu Packages. I have created a repository with the required content here:
https://github.com/beyondelastic/gitops-tanzu-packages
From a high-level perspective, the process that we are going to implement looks like the following. FluxCD will reconcile the content of our Git repository with our TKG cluster and create Carvel resources such as the PackageInstall CR and others. Based on the created PackeInstall, the kapp-controller will reconcile the Tanzu Package with our cluster. The actual container images and configs will be pulled from the online Tanzu Package Repository! If you have the requirement to pull the Tanzu Package images from a private container registry, I recommend reading the blog post from my buddy Robert Guske “Deploy VMware Tanzu Packages from a private Container Registry“.
Please note: Not all available Tanzu Packages can be found in the repository yet. For the moment, the repository includes pre-requisites to make use of Tanzu Packages 1.5, Cert-Manager, Fluent-Bit, Contour, Prometheus, Grafana, and Harbor.
FluxCD uses gitrepositories.source.toolkit.fluxcd.io Custom Resource (CR) to connect additional Git sources. Just execute the following flux command to create the yaml file under your fleet-infra git folder. The yaml file will be stored under /clusters/my-cluster/.
➜ fleet-infra git:(main) ✗ flux create source git tanzu-packages \
--url=https://github.com/beyondelastic/gitops-tanzu-packages \
--branch=main \
--interval=30s \
--export > ./clusters/my-cluster/tanzu-packages-source.yamlNow we need to commit our change, and after 30 seconds, we should see that the “GitRepository” CR has been created on your cluster.
git add -A && git commit -m "Add Tanzu Packages GitRepository"
git push➜ k get gitrepositories tanzu-packages -n flux-system
NAME URL AGE READY STATUS
tanzu-packages https://github.com/beyondelastic/gitops-tanzu-packages 80m True stored artifact for revision 'main/154d08db27bfcca5733a6aca52019f3b87d09811'Prepare TKG Cluster via FluxCD
We now want to prepare our TKG clusters for Tanzu Packages via FluxCD. FluxCD is using the kustomizations.kustomize.toolkit.fluxcd.io CR to deploy manifest onto Kubernetes.
We can use the flux cli to create the yaml file similar to what we have done in the previous step for the source, or we just use the manifest that I have prepared here:
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: tanzu-packages-pre-reqs
namespace: flux-system
spec:
interval: 5m0s
path: ./pre-reqs
prune: true
sourceRef:
kind: GitRepository
name: tanzu-packages
This spec points to the “pre-reqs“ folder on the gitops-tanzu-packages repository. It deploys kapp-controller onto your cluster, adds the PackageRepository CR for Tanzu Packages 1.5, and for the sake of demonstration, configures all “system:serviceaccounts“ to use the “vmware-system-privileged” PodSecurityPolicy. Additionally, it creates a “packages” namespace in which all the package-related resources (carvel CR, service accounts, config secrets, etc…) are getting stored.
Please note: This repository has been tested with TKGs (vSphere with Tanzu – 7U3). If you are using a TKGm cluster or your TKGs cluster has been attached to TMC, it will have the kapp-controller installed already and the package repository configured. In that case, modify or remove some of the pre-reqs content on a fork of the repository.
Save the yaml file in your “fleet-infra” git repository under ./clusters/my-cluster/tanzu-packages-pre-reqs.yaml and push it to your repo.
git add -A && git commit -m "Add Tanzu Packages Pre-Reqs"
git pushAfter a few seconds, you should see the Kustomization resource and the kapp-controller pod being created on your Kubernetes cluster.
➜ k get kustomizations.kustomize.toolkit.fluxcd.io tanzu-packages-pre-reqs -n flux-system
NAME AGE READY STATUS
tanzu-packages-pre-reqs 154m True Applied revision: main/154d08db27bfcca5733a6aca52019f3b87d09811
➜ k get pods,svc -n tkg-system
NAME READY STATUS RESTARTS AGE
pod/kapp-controller-75c69cdcdc-zcvll 1/1 Running 0 139m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/packaging-api ClusterIP 195.56.65.203 <none> 443/TCP 139mDeploy Tanzu Packages via FluxCD
The different Tanzu Packages PackageInstall manifests are stored in separate folders on the gitops-tanzu-packages repository:
You can select which one you want to deploy by creating another Kustomization manifest pointing to a different path in the repo. For Cert-Manager, see the following example:
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: tanzu-packages-cert-manager
namespace: flux-system
spec:
dependsOn:
- name: tanzu-packages-pre-reqs
interval: 5m0s
path: ./cert-manager
prune: true
sourceRef:
kind: GitRepository
name: tanzu-packages
All Kustomization yaml files can be found under the “flux-config” folder of the repository for reference.
Please be aware that some of the packages depend on each other, and therefore, I am using Kustomization-dependencies. If you want to remove the dependencies, remove the “dependsOn” section of the Kustomization manifest. Here is an overview of the dependencies within the repository.
Just as before, save the manifest under your clusters folder of your fleet-infra repo and add, commit, and push it. FluxCD will connect to the source and deploy the content (e.g., cert-manager) onto your Kubernetes cluster.
Check the result on your cluster:
➜ k get packageinstalls -n packages
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
cert-manager cert-manager.tanzu.vmware.com 1.5.3+vmware.2-tkg.1 Reconcile succeeded 3h57m
➜ k get app -n packages
NAME DESCRIPTION SINCE-DEPLOY AGE
cert-manager Reconcile succeeded 53s 3h59m
➜ k get pods,svc -n cert-manager
NAME READY STATUS RESTARTS AGE
pod/cert-manager-56bdc799bd-dhpdx 1/1 Running 0 45h
pod/cert-manager-cainjector-678d7d49c8-vfmjn 1/1 Running 0 45h
pod/cert-manager-webhook-7f8689555c-rswz8 1/1 Running 0 45h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 195.50.34.144 <none> 9402/TCP 45h
service/cert-manager-webhook ClusterIP 195.50.247.222 <none> 443/TCP 45hWe can also use the Tanzu CLI to see if the package is listed and successfully reconciled.
➜ tanzu package installed list -n packages
/ Retrieving installed packages…
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
cert-manager cert-manager.tanzu.vmware.com 1.5.3+vmware.2-tkg.1 Reconcile succeeded defaultConfigure Tanzu Packages
We now know how to deploy Tanzu Packages via FluxCD Kustomization manifests and our gitops-tanzu-packages repository. However, some of the packages require custom configurations.
Note: Packages in this repository are pre-configured with custom virtual hosts (ingress), storage class, passwords, syslog target, etc… you have to change the configuration according to your environment first!
If you haven’t done it already, now is the time to fork the repository to make changes only valid for your environment.
To adjust the configuration, check the manifests in the data-values folder and modify them according to your needs. Refer to the official TKG documentation for available parameters.
If you have the imgpkg command installed on your client, you can also use the following commands to receive the configuration template of a specific Tanzu Package:
image_url=$(kubectl -n tanzu-package-repo-global get packages grafana.tanzu.vmware.com.7.5.7+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')
imgpkg pull -b $image_url -o /tmp/grafana-package-7.5.7+vmware.1-tkg.1
cp /tmp/grafana-package-7.5.7+vmware.1-tkg.1/config/values.yaml grafana-data-values.yaml
Once you have changed the configuration file according to your needs, you need to base64 encode it:
base64 -i grafana-data-values.yamlYou can also use https://www.base64encode.org/ to base64 encode your configuration file.
Use the outcome to replace the data value within the package secret (e.g., grafana-data-values-secret.yaml).
apiVersion: v1
data:
grafana-data-values.yaml: LS0tCmdyYWZhbmE6CiAgc2VjcmV0OgogICAgdHlwZTogIk9wYXF1ZSIKICAgIGFkbWluX3Bhc3N3b3JkOiAiVmsxM1lYSmxNU0U9IgogIHB2YzoKICAgIHN0b3JhZ2VDbGFzc05hbWU6CmluZ3Jlc3M6CiAgZW5hYmxlZDogdHJ1ZQogIHZpcnR1YWxfaG9zdF9mcWRuOiAiZ3JhZmFuYS5zeXN0ZW0udGFuenUiCiAgcHJlZml4OiAiLyIKICBzZXJ2aWNlUG9ydDogODAKCg==
kind: Secret
metadata:
annotations:
tkg.tanzu.vmware.com/tanzu-package: grafana-packages
name: grafana-packages-values
namespace: packages
type: Opaque
Once you have committed the change to your own repository, it will use your configuration whenever you deploy the corresponding Tanzu Package.
Conclusion
This blog post illustrates how to deploy Tanzu Packages via GitOps practices and FluxCD. The possibility to deploy and sync multiple Tanzu Packages via a GitOps approach is quite handy and makes the life of Platform Operators much easier. The Tanzu Packages Repo is not officially tested or supported by VMware! However, the Tanzu Packages are built on the Carvel toolset, and this repo uses constructs and APIs described and documented on the Carvel documentation. Feel free to use it at your own risk.
Sources
Categories
Beyond Elastic 