Skip to content

GitOps with FluxCD and Tanzu Packages

I often get asked how to install our Tanzu Packages in a GitOps fashion instead of using the Tanzu CLI or TMC Catalog. I want to quickly demonstrate how this can be done in this blog post. If you are not familiar with Tanzu Packages, I recommend reading my blog post about Tanzu Packages to understand the basic concepts. For this example, I will use FluxCD as the Continuous Deployment (CD) tool. FluxCD is an incubating project at the Cloud Native Computing Foundation (CNCF). However, to use a GitOps approach to deploy Tanzu Packages, other CD tools such as ArgoCD can be used too.

Install FluxCD

First of all, let’s install the FluxCD CLI via brew. You can also use different installation methods as described here. As a side note, FluxCD is also available as Tanzu Community Edition (TCE) Package, which can be found here. However, I will not use the TCE Package in this scenario.

brew install fluxcd/tap/flux

We can run the pre-check to verify if our Tanzu Kubernetes Grid (TKG) cluster is suitable for FluxCD. Just run the following flux command while using the kubectl context of the cluster you want to use.

➜  ~ flux check --pre
► checking prerequisites
✔ Kubernetes 1.21.6+vmware.1 >=1.20.6-0
✔ prerequisites checks passed

To bootstrap FluxCD, we are going to specify our GITHUB config as follows:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>

You can install FluxCD on the TKG cluster and configure it to manage itself from a Git repository using the flux bootstrap command.

➜ ~ flux bootstrap github \
--owner=$GITHUB_USER \
--repository=fleet-infra \
--branch=main \
--path=./clusters/my-cluster \
--personal

We will see the FluxCD resources being created on our Kubernetes cluster.

➜  ~ k get pods,svc -n flux-system
NAME                                           READY   STATUS    RESTARTS   AGE
pod/helm-controller-88f6889c6-sp4mp            1/1     Running   0          86s
pod/kustomize-controller-784bd54978-hg2j8      1/1     Running   0          86s
pod/notification-controller-648bbb9db7-h7bxr   1/1     Running   0          85s
pod/source-controller-79f7866bc7-wkxf5         1/1     Running   0          85s

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/notification-controller   ClusterIP   195.48.118.105   <none>        80/TCP    88s
service/source-controller         ClusterIP   195.59.108.18    <none>        80/TCP    88s
service/webhook-receiver          ClusterIP   195.48.100.248   <none>        80/TCP    87s

Great FluxCD is running, and you will have a new private Git repository, “fleet-infra”. This repository will be used to manage your cluster. Now we need to clone the repository locally to use it:

git clone https://github.com/$GITHUB_USER/fleet-infra
cd fleet-infra

Use FluxCD for Tanzu Packages

Easy, right? Next, we want to configure FluxCD to connect to a Git repo that includes the necessary manifests for our Tanzu Packages. I have created a repository with the required content here:

https://github.com/beyondelastic/gitops-tanzu-packages

From a high-level perspective, the process that we are going to implement looks like the following. FluxCD will reconcile the content of our Git repository with our TKG cluster and create Carvel resources such as the PackageInstall CR and others. Based on the created PackeInstall, the kapp-controller will reconcile the Tanzu Package with our cluster. The actual container images and configs will be pulled from the online Tanzu Package Repository! If you have the requirement to pull the Tanzu Package images from a private container registry, I recommend reading the blog post from my buddy Robert GuskeDeploy VMware Tanzu Packages from a private Container Registry“.

Please note: Not all available Tanzu Packages can be found in the repository yet. For the moment, the repository includes pre-requisites to make use of Tanzu Packages 1.5, Cert-Manager, Fluent-Bit, Contour, Prometheus, Grafana, and Harbor.

FluxCD uses gitrepositories.source.toolkit.fluxcd.io Custom Resource (CR) to connect additional Git sources. Just execute the following flux command to create the yaml file under your fleet-infra git folder. The yaml file will be stored under /clusters/my-cluster/.

➜ fleet-infra git:(main) ✗ flux create source git tanzu-packages \
--url=https://github.com/beyondelastic/gitops-tanzu-packages \
--branch=main \
--interval=30s \
--export > ./clusters/my-cluster/tanzu-packages-source.yaml

Now we need to commit our change, and after 30 seconds, we should see that the GitRepository” CR has been created on your cluster.

git add -A && git commit -m "Add Tanzu Packages GitRepository"
git push
➜ k get gitrepositories tanzu-packages -n flux-system
NAME             URL                                                      AGE   READY   STATUS
tanzu-packages   https://github.com/beyondelastic/gitops-tanzu-packages   80m   True    stored artifact for revision 'main/154d08db27bfcca5733a6aca52019f3b87d09811'

Prepare TKG Cluster via FluxCD

We now want to prepare our TKG clusters for Tanzu Packages via FluxCD. FluxCD is using the kustomizations.kustomize.toolkit.fluxcd.io CR to deploy manifest onto Kubernetes.

We can use the flux cli to create the yaml file similar to what we have done in the previous step for the source, or we just use the manifest that I have prepared here:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: tanzu-packages-pre-reqs
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./pre-reqs
  prune: true
  sourceRef:
    kind: GitRepository
    name: tanzu-packages

This spec points to the pre-reqs folder on the gitops-tanzu-packages repository. It deploys kapp-controller onto your cluster, adds the PackageRepository CR for Tanzu Packages 1.5, and for the sake of demonstration, configures all system:serviceaccounts to use the “vmware-system-privileged” PodSecurityPolicy. Additionally, it creates a “packages” namespace in which all the package-related resources (carvel CR, service accounts, config secrets, etc…) are getting stored.

Please note: This repository has been tested with TKGs (vSphere with Tanzu – 7U3). If you are using a TKGm cluster or your TKGs cluster has been attached to TMC, it will have the kapp-controller installed already and the package repository configured. In that case, modify or remove some of the pre-reqs content on a fork of the repository.

Save the yaml file in your “fleet-infra” git repository under ./clusters/my-cluster/tanzu-packages-pre-reqs.yaml and push it to your repo.

git add -A && git commit -m "Add Tanzu Packages Pre-Reqs"
git push

After a few seconds, you should see the Kustomization resource and the kapp-controller pod being created on your Kubernetes cluster.

➜ k get kustomizations.kustomize.toolkit.fluxcd.io tanzu-packages-pre-reqs -n flux-system
NAME AGE READY STATUS
tanzu-packages-pre-reqs 154m True Applied revision: main/154d08db27bfcca5733a6aca52019f3b87d09811

➜ k get pods,svc -n tkg-system
NAME                                   READY   STATUS    RESTARTS   AGE
pod/kapp-controller-75c69cdcdc-zcvll   1/1     Running   0          139m

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/packaging-api   ClusterIP   195.56.65.203   <none>        443/TCP   139m

Deploy Tanzu Packages via FluxCD

The different Tanzu Packages PackageInstall manifests are stored in separate folders on the gitops-tanzu-packages repository:

You can select which one you want to deploy by creating another Kustomization manifest pointing to a different path in the repo. For Cert-Manager, see the following example:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: tanzu-packages-cert-manager
  namespace: flux-system
spec:
  dependsOn:
    - name: tanzu-packages-pre-reqs
  interval: 5m0s
  path: ./cert-manager
  prune: true
  sourceRef:
    kind: GitRepository
    name: tanzu-packages

All Kustomization yaml files can be found under the “flux-config” folder of the repository for reference.

Please be aware that some of the packages depend on each other, and therefore, I am using Kustomization-dependencies. If you want to remove the dependencies, remove the “dependsOn” section of the Kustomization manifest. Here is an overview of the dependencies within the repository.

Just as before, save the manifest under your clusters folder of your fleet-infra repo and add, commit, and push it. FluxCD will connect to the source and deploy the content (e.g., cert-manager) onto your Kubernetes cluster.

Check the result on your cluster:

➜ k get packageinstalls -n packages
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
cert-manager cert-manager.tanzu.vmware.com 1.5.3+vmware.2-tkg.1 Reconcile succeeded 3h57m

➜ k get app -n packages
NAME           DESCRIPTION           SINCE-DEPLOY   AGE
cert-manager   Reconcile succeeded   53s            3h59m

➜ k get pods,svc -n cert-manager
NAME                                           READY   STATUS    RESTARTS   AGE
pod/cert-manager-56bdc799bd-dhpdx              1/1     Running   0          45h
pod/cert-manager-cainjector-678d7d49c8-vfmjn   1/1     Running   0          45h
pod/cert-manager-webhook-7f8689555c-rswz8      1/1     Running   0          45h

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   195.50.34.144    <none>        9402/TCP   45h
service/cert-manager-webhook   ClusterIP   195.50.247.222   <none>        443/TCP    45h

We can also use the Tanzu CLI to see if the package is listed and successfully reconciled.

➜ tanzu package installed list -n packages 
/ Retrieving installed packages…
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
cert-manager cert-manager.tanzu.vmware.com 1.5.3+vmware.2-tkg.1 Reconcile succeeded default

Configure Tanzu Packages

We now know how to deploy Tanzu Packages via FluxCD Kustomization manifests and our gitops-tanzu-packages repository. However, some of the packages require custom configurations.

Note: Packages in this repository are pre-configured with custom virtual hosts (ingress), storage class, passwords, syslog target, etc… you have to change the configuration according to your environment first!

If you haven’t done it already, now is the time to fork the repository to make changes only valid for your environment.

To adjust the configuration, check the manifests in the data-values folder and modify them according to your needs. Refer to the official TKG documentation for available parameters.

If you have the imgpkg command installed on your client, you can also use the following commands to receive the configuration template of a specific Tanzu Package:

image_url=$(kubectl -n tanzu-package-repo-global get packages grafana.tanzu.vmware.com.7.5.7+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')

imgpkg pull -b $image_url -o /tmp/grafana-package-7.5.7+vmware.1-tkg.1

cp /tmp/grafana-package-7.5.7+vmware.1-tkg.1/config/values.yaml grafana-data-values.yaml

Once you have changed the configuration file according to your needs, you need to base64 encode it:

base64 -i grafana-data-values.yaml

You can also use https://www.base64encode.org/ to base64 encode your configuration file.

Use the outcome to replace the data value within the package secret (e.g., grafana-data-values-secret.yaml).

apiVersion: v1
data:
  grafana-data-values.yaml: LS0tCmdyYWZhbmE6CiAgc2VjcmV0OgogICAgdHlwZTogIk9wYXF1ZSIKICAgIGFkbWluX3Bhc3N3b3JkOiAiVmsxM1lYSmxNU0U9IgogIHB2YzoKICAgIHN0b3JhZ2VDbGFzc05hbWU6CmluZ3Jlc3M6CiAgZW5hYmxlZDogdHJ1ZQogIHZpcnR1YWxfaG9zdF9mcWRuOiAiZ3JhZmFuYS5zeXN0ZW0udGFuenUiCiAgcHJlZml4OiAiLyIKICBzZXJ2aWNlUG9ydDogODAKCg==
kind: Secret
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: grafana-packages
  name: grafana-packages-values
  namespace: packages
type: Opaque

Once you have committed the change to your own repository, it will use your configuration whenever you deploy the corresponding Tanzu Package.

Conclusion

This blog post illustrates how to deploy Tanzu Packages via GitOps practices and FluxCD. The possibility to deploy and sync multiple Tanzu Packages via a GitOps approach is quite handy and makes the life of Platform Operators much easier. The Tanzu Packages Repo is not officially tested or supported by VMware! However, the Tanzu Packages are built on the Carvel toolset, and this repo uses constructs and APIs described and documented on the Carvel documentation. Feel free to use it at your own risk.

Sources

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: