An intro to Kustomize

What is Kustomize

#

Kubernetes native configuration management

With Kustomize you can define your Kubernetes deployments¹ for various environments with different parameters without learning a new templating language nor a DSL. Kustomize works with standard Kubernetes resources² and applies different parameters for different environments via a so-called overlay. More on this when we have a closer look at Kustomize.

Kustomize is included in kubectl and is also available as a standalone application for all major platforms.

What does Kustomize solve?

#

Generally spoken, Kustomize is a tool that supports you in declaratively writing Kubernetes manifests³. You might ask what is the difference to a simple kubectl apply -f /path/to/yaml/dir?

Think about the common scenario of teams deploying to Kubernetes in a multi-cluster setup for different environments⁴. For instance, you could have develop, test, and production environments. For each of them, a dedicated Kubernetes cluster is running. You need to deploy your applications to each Kubernetes cluster in order to perform the required tasks like testing or QA and, of course, running production load. The deployments are not equal on each cluster because the requirements can be different. On the production cluster, you probably require more resources in order to sustain availability under heavy load, in contrast, to develop where only a few developers work with.

A naive approach might look like this:

You define your Kubernetes manifests for each environment with their specific parameters and apply them to the cluster. This leads to a lot of redundant code and decreases the maintainability because you have to touch each environment in case of a common change that affects all environments.

In order to reduce code redundancy and increase maintainability we want to achieve an approach like this:

We want to define common Kubernetes manifests for all environments that are kustomized for each environment.

Kustomize in action

#

Kustomize introduces the base and overlay concept. The base includes all Kubernetes manifests that are common for all environments and tend to be very static as they do not change often, if at all. The overlay includes manifests for a specific environment and alters or adds specific values of the base.

A common practice for writing Kustomize deployments is a clear folder structure that separates base and overlay manifests, as shown in the following image:

Basic example

#

This example illustrates the simple overlay mechanism of Kustomize to write Kubernetes manifests for different environments. Kustomize generates/builds the manifests to be applied before they are transmitted to the Kubernetes controller.

The generated manifests are printed out and can be redirected to a file.

kustomize build /path/to/folder [> manifests.yaml]

And if you want to use the built-in Kustomize flag of kubectl:

kubectl apply -k /path/to/folder

Now, let’s think about the following scenario:

• three environments (dev, test, prod)
• two applications named foo and bar

We should write deployment manifests that satisfy the following requirements:

• both applications should be deployed to each environment
• resource labeling must match the environment
• scale the applications per environment
• allocate resources to the applications per environment
• inject ENV variables into pods per environment

In the base folder, we define our usual Kubernetes manifests. For this demo, we write resources for a deployment, a service account and a service for both bar and foo services.

Additionally, Kustomize requires a kustomization.yaml in each folder that contains at least a list of resource files to be processed by Kustomize. Additionally, we define common labels for this application.

base/bar/kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  app.kubernetes.io/name: bar
resources:
  - deployment.yaml
  - sa.yaml
  - service.yaml

The overlay folder contains the modifications we want for each environment. Again, each folder requires a kustomization.ymal.

The first kustomization.yaml defines the list of resources to be applied to the according environment. Note that you can define folders as well. Furthermore we define an additional common label specific to the environment (env) that will be merged to the labels from the already defined ones in the base resources.

overlay/dev/kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: app
commonLabels:
  env: dev
resources:
  - bar/
  - foo/

The second kustomization.yaml finally defines the resources to modify the application’s values.

overlay/dev/bar/kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../base/bar/
patchesStrategicMerge:
  - deployment.yaml

In this example, we use a patchesStrategicMerge which will use the provided deployment.yaml and patch the provided values to the deployment resource defined in base.

overlay/dev/foo/deployment.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: foo
          env:
            - name: ENVIRONMENT
              value: dev
          resources:
            requests:
              memory: "100Mi"
              cpu: "100m"
            limits:
              memory: "100Mi"
              cpu: "100m"

This patch will add spec.replicas, spec.template.spec.containers[0].resources, and spec.template.spec.containers[0].env to the deployment resource from the base. As we provide such a patch for each environment we can adjust our deployment for each of them.

Here you can see a diff in the deployment resources generated by Kustomize for each environment. As intended, each environment is kustomized with its own values.

Selectors

#

If you have a closer look at the code of the basic example, you can see that the deployment manifests lack the spec.selectors part and how this should work for a real deployment in a cluster.

The answer is a convenient feature of Kustomize for the handling of spec.selectors. Kustomize automatically handles them for you!

Let’s have a look at the generated deployment manifest for the foo service.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: foo
    env: prod
  name: foo
  namespace: app
spec:
  replicas: 3
  selector: # automatically generated by Kustomize
    matchLabels:
      app.kubernetes.io/name: foo
      env: prod
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: foo
        env: prod

Since Kustomize 4.1.0, there is a new option to distinguish between labels that should be included in selectors as the following example shows.

basic/base/bar/kustomization.yaml:

labels:
  - pairs:
      # both labels are not included in selectors
      version: 0.9.0
      msg: not in selector
      # this label is included as selector
  - pairs:
      criticality: high
    includeSelectors: true

By providing a set of base resources, we can reduce the number of redundant manifests. By applying patches via Kustomize, we can modify only the resources’ values we need. By following a clean folder structure (see resource layout for more details), it is intuitive and straight-forward to understand and maintain your Kubernetes manifests.

Resource layout

#

Kustomize works on different levels allowing you to organize your deployments and create a logical structure. Let us think about a more complex deployment scenario.

Basic conditions

• two dedicated clusters for a staging(stg) and a production(prod) environment
• a frontend consisting of a UI component and a BFF (backend for frontend) component
• a backend consisting of an engine component, an administration component and several scraper components
• a debugging service that is only required to deploy on stg

Implementation

With Kustomize, you can organize your deployments in folders and sub-folders in a nested structure.

You can see that we organized our applications in backend and frontend folders. Of course, each folder requires a kustomization.yaml. Furthermore you can see that the debug application only exists in stg and not in the base because of the requirements. If you think that there might be a chance that the debug component needs to be deployed to prod as well, you can define it in the base as well but only refer to it from staging.

Labels

#

Labels are merged through the path that Kustomize traverses and you can include new labels in each kustomization.yaml (if necessary).

The labels after Kustomize has built the manifest are as follows:

labels:
  app.kubernetes.io/name: editor # from /base/backend/editor/kustomization.yaml
  app.kubernetes.io/part-of: backend # from base/backend/kustomization.yaml
  env: prod # from prod/kustomization.yaml

Multiple deployments from one base

#

In Production, we want to deploy multiple scraper applications that might run just in a slightly different configuration (for instance, a different environment variable). With Kustomize, we can still define the common resources for the scraper in our base folder but create multiple deployments in the overlay.

How does this work and how does this not lead to naming collisions of resources?

Kustomize has a feature nameSuffix (and namePrefix) that modifies all resources’ names accordingly:

layering/overlay/prod/backend/scraper-bar/kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
nameSuffix: -bar
commonLabels:
  app.kubernetes.io/name: scraper-bar
resources:
  - ../../../../base/backend/scraper

Although in base, we defined metadata.name: scraper Kustomize generates manifests with a new name allowing us to deploy multiple scraper applications while only maintaining one common base.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: scraper-foo
    app.kubernetes.io/part-of: backend
    env: prod
  name: scraper-foo # from nameSuffix in kustomization.yaml
  namespace: team1

More kustomization

#

Kustomize supports more advanced mechanisms besides patchesStrategicMerge.

patches

#

This example demonstrates how to patch (replace) a single value, in this case, the host of an ingress resource.

patchesJson6902

#

This example changes the username of a Grafana notification channel per environment:

You can patch multiple values and use a JSON format. Different values for op, like delete, are available.

[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--override"
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "default.replication.factor=3"
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--override"
  }
]

CD integration

#

CD integration is available for common GitOps tools like Flux and ArgoCD. Due to its simplicity, it is likely that you can integrate Kustomize into your workflow as well. This is the most basic command to apply your manifests:

kustomize build overlay/prod | kubectl apply -f -

Keep in mind that this approach does not do a garbage collection of resources as Flux does.

Kustomize best practices

#

1. Naming conventions of files and resources
2. One kind of resources per yaml file
3. Think about your labels ( recommended labels)
4. Think about your folder structure and keep it simple and uniform across projects/teams
5. Prefix your commit messages to distinguish commits for specific environments
6. Think about your base. In doubt, move code to the overlay to reduce the impact on your production environment
7. Implement a pipeline that verifies your manifests

Kustomize vs. Helm

#

What is Helm

#

The package manager for Kubernetes.

Helm is the best way to find, share, and use software built for Kubernetes.

Helm is one of the old players in the Kubernetes ecosystem. Meanwhile, the current version of Helm is v3 which replaced version 2 with a new architecture. See the official migration guide for the changes between v3 and v2. Helm is available for all major operating systems.

Basically, Helm is used to define and distribute deployments for Kubernetes.

The main difference to Kustomize is the templating language of Helm.

Comparison

#

When to use Kustomize over Helm

#

• You want to deploy to different environments
• You want to deploy self-developed microservices that are loosely coupled to each other with respect to their deployment dependencies
• You don’t want to distribute your application to external parties
• You don’t need to bundle your application in releases
• Your deployment does not require complex configuration with loops and Conditional Clauses
• You want to keep things simple 😉

Downsides of Kustomize

#

There are some aspects to consider when working with Kustomize:

• Boiler-plate: For more complex deployments, you will write a lot of kustomization.yaml and version, kind, name, etc. fields
• Redundancy: There is still some redundancy. For instance, in the basic example, you have to write the deployment files to be patched three times with only minimal differences.
• Loose coupling: components defined with Kustomize are only loosely coupled and dependencies between components are not easy to declare.
• Limited “language”: Kustomize, unlike Helm, does not include a templating language and therefore does not offer templating features like loops or control flows.
• Distribution: Distributing your applications is not as straightforward as with Helm.