Build a complete GitOps ecosystem

This article is written for DevOps and friends who want to understand.

Preface#

Before we start, let's talk about the traditional GitOps process. Use a git repository to configure the CI/CD pipeline. After git push, a mirror image is built based on the current git sha and then applied to Kubernetes. Deployment will be automatically updated.

This is simple and useful. Kubernetes will automatically roll out your image. But we will have more problems and requirements.

How to only change resources
Suppose I only made some changes to the arrangement, such as increasing the number of replicas or adding resources. It will create a new git commit. In terms of the process, this is the same as submitting new code, and you still have to go through the [ci]->build->deploy process. Of course, you can configure the pipeline to not trigger the build process when the deploy file changes. Only when files in directories like src/ or app/ change, do you need to build the image once. This will bring new problems. As mentioned earlier, we apply it to deployment based on the git commit sha as the image tag. Suppose I submitted a commit a1b2c3, but skipped the build. There will be an error in the deploy stage because it cannot find the image with the tag a1b2c3.
How to store sensitive information
Usually, applications are accompanied by sensitive information such as DATABASE_URL or ACCESS_TOKEN, which should not be stored in the git repository, even if it is a private or private domain repository. These are usually stored in the form of Kubernetes secrets, and secrets are simply base64 encrypted, which is equivalent to plaintext storage. Of course, we can store secrets separately or apply them to the cluster manually. But doing so will cause, on the one hand, the release of the entire application and the secret to be separated, and on the other hand, ineffective management.

ArgoCD#

ArgoCD is a CD tool. The official introduction is as follows:

What Is Argo CD?
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

ArgoCD manages applications at the application (hereinafter referred to as app) level. Generally, one app corresponds to a single application in a single cluster.

ArgoCD manages apps by listening to resources in a git repository. It supports various ways of managing Kubernetes resources, such as Helm/Kustomize/Directory, etc. In simple terms, as long as you put the files that need to be deployed in the git repository and configure the app correctly, ArgoCD will listen for changes in the deploy.

For example,

OK, so far, we have solved a problem — "How to change resources without rebuilding the image."

Secret management#

https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/

ArgoCD does not have a mandatory secret management solution. The link above mentions some recommended and supported secret management methods. This article will use Hashicorp Vault and argocd-vault-plugin as tools for management.

Vault#

Vault is a Hashicorp product that supports secret-based engines with KV key-value pairs. It can be understood as a server version of 1password. Due to limited space, the installation and configuration of Vault will not be elaborated here.

Suppose we store secrets in the kv/<namepsace>/<app> path in Vault, such as DATABASE_URL, etc.

The built-in pod inject of Vault does not meet the requirements, so it will not be elaborated here either. We hope to have a solution that can synchronize Vault secrets to Kubernetes secrets.

argocd-vault-plugin#

argocd-vault-plugin is a plugin used by ArgoCD to access Vault, hereinafter referred to as avp.

The main function of avp is to convert the corresponding template into Vault secrets.

For example, I have a yaml:

kind: Secret
apiVersion: v1
metadata:
  name: example-secret
  annotations:
    avp.kubernetes.io/path: "path/to/secret"
type: Opaque
data:
  password: <password-vault-key>

After processing with avp and applying it to the cluster, it becomes real data instead of a template. For details, please refer to: https://argocd-vault-plugin.readthedocs.io/en/stable/howitworks/

However, although it was mentioned above that argocd supports multiple resource orchestration tools, they will not enable avp by default. To use avp, you need to customize the plugin, that is, use the plugin method.

The plugin itself is just a preprocessing script. In the github yaml file example in AVP Install, there is a discover to find a kustomization.yaml and generate it using the avp plugin after building.

Special note: Since helm and kustomize use different plugins, when using them, you need to specify the plugin name in the env of the plugin, such as plugin = avp-helm and plugin = avp-kustomize, otherwise an error will occur.

Since kustomize supports helm-charts syntax in the kustomization.yaml in version 4.x, but you need to manually use the --enable-helm flags, we have customized the plugin. Some code is shown below.

      avp-kustomize:
        allowConcurrency: true
        discover:
          find:
            command:
              - sh
              - "-c"
              - "find . -name kustomization.yaml"
        generate:
          command:
            - sh
            - "-c"
            - "kustomize edit set annotation \"github.com/url:${ARGOCD_ENV_APP_REPO}\"; kustomize edit set image ${ARGOCD_ENV_IMAGE_NAME}:${ARGOCD_ENV_IMAGE_TAG}; kustomize build --enable-helm . | argocd-vault-plugin -s ${ARGOCD_ENV_AVP_SECRET} generate -"
        lockRepo: false

https://argo-cd.readthedocs.io/en/latest/operator-manual/upgrading/2.3-2.4/#update-plugins-to-use-newly-prefixed-environment-variables
As shown in the article, we need to add the ARGO_ENV prefix to the env passed to the plugin.

Here, we have added four environment variables, namely:

IMAGE_NAME: image name
IMAGE_TAG: image tag
AVP_SECRET: credentials for configuring Vault
APP_REPO: git repository address

The git repository address will be used for a global annotation to see which repository a certain resource belongs to in the cluster.

So far, we have completed most of it.

Git CI#

Regardless of whether you use GitHub Actions, GitLab CI, Drone, or any other pipeline, you can configure it in the corresponding way.

Two stages should be configured in CI: build and deploy.

The build stage is used to build the Docker image and push it to the image repository.
The deploy stage is a process of updating/creating an argocd app.

The deploy stage is the main focus. It will use the argocd CLI to update or create the tag of the image.
Example:

argocd app create  {your_app}  \
  --repo https://github.com/{git_url} \
  --path {your_repo_pass} \
  --project {your_project} \
  --dest-name {deploy_cluster} \
  --dest-namespace {cluster_ns} \
  --config-management-plugin=avp-kustomize \
  --revision {revision} \
  --plugin-env AVP_SECRET={secret_name} \
  --plugin-env APP_REPO={git_url} \
  --plugin-env IMAGE_NAME={image.name} \
  --plugin-env IMAGE_TAG={image.tag} \
  --sync-policy automated \
  --sync-option ApplyOutOfSyncOnly=true,ServerSideApply=true \
  --upsert

This will create an argocd app. In this way, ArgoCD will listen to the git repo itself. When you change the deploy file, because the image name and tag applied to the app have not changed, a new deployment can be generated based on the previous image.

Therefore, you can configure some trigger conditions in the git pipeline to exclude the deploy file or only include code files.

Summary#

This article mainly explains how to use argocd, vault, argocd-vault-plugin, and CI together to build a relatively complete GitOps ecosystem. It may not be as good as mature commercial solutions, but freely combining open-source components can find the best practice form.

For example, CI and Git repositories have no restrictions. Using vault and avp is just my solution. Any way of managing secrets mentioned in argocd can be used in the form of plugins.