Post

GitOps with Flux on Kubernetes Part 2: Advanced Kustomization and the Kustomize Controller

Deep dive into Flux's Kustomize Controller, advanced kustomization techniques, multi-environment management, and configuration overlays for GitOps workflows

Posted
Preview Image
By Talha Juikar
12 min read
GitOps with Flux on Kubernetes Part 2: Advanced Kustomization and the Kustomize Controller

Introduction

In part 1 of this series, we covered the basics of installing Flux and setting up your first GitOps workflow. Now, we’ll dive deeper into one of Flux’s most powerful components: the Kustomize Controller and advanced kustomization techniques.

Kustomization is a powerful tool for managing Kubernetes configurations across multiple environments without duplicating YAML files. When combined with Flux’s GitOps capabilities, it provides a robust solution for managing complex, multi-environment deployments declaratively.

Understanding the Flux Kustomize Controller

The Kustomize Controller is one of the core components in Flux v2. It watches for Kustomization resources and applies the Kubernetes manifests from Git repositories, Helm repositories, or Bucket sources.

Key Features of the Kustomize Controller

  • Declarative Configuration: Define your desired state in Git using Kustomization resources
  • Dependency Management: Control the order of resource deployment and updates
  • Health Assessment: Monitor the health of deployed resources
  • Pruning: Automatically remove resources that are no longer defined in Git
  • Variable Substitution: Inject values dynamically into your manifests
  • Multi-tenancy: Isolate deployments across different namespaces and teams

Kustomization Resource Structure

A Flux Kustomization resource looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 10m0s
  path: "./apps/my-app"
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-repo
  timeout: 5m0s
  retryInterval: 2m0s
  validation: client
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: my-app
      namespace: default

Setting Up a Multi-Environment Structure

Let’s create a practical example that demonstrates advanced kustomization techniques. We’ll set up a repository structure that supports multiple environments (dev, staging, production) for a sample application. The sample repository is available at TalhaJuikar/flux-demo-website

Here’s a quick demonstration of the deployed website across different environments:

Repository Structure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

flux-demo-website/
├── apps
│   └── demo-website
│       ├── base
│       │   ├── deployment.yaml
│       │   ├── ingress.yaml
│       │   ├── kustomization.yaml
│       │   └── service.yaml
│       └── overlays
│           ├── dev
│           │   ├── config-patch.yaml
│           │   ├── ingress-patch.yaml
│           │   ├── kustomization.yaml
│           │   ├── replica-patch.yaml
│           │   └── resource-patch.yaml
│           ├── prod
│           │   ├── deployment-patch.yaml
│           │   ├── hpa.yaml
│           │   ├── ingress-patch.yaml
│           │   └── kustomization.yaml
│           └── staging
│               ├── deployment-patch.yaml
│               ├── ingress-patch.yaml
│               └── kustomization.yaml
└── clusters
    └── flux-demo
        ├── apps.yaml
        └── flux-system
            ├── gotk-components.yaml
            ├── gotk-sync.yaml
            └── kustomization.yaml

Bootstrapping the Repository

If you haven’t already, bootstrap your Flux setup with the following command:

1
2
3
4
5
6

flux bootstrap git \
  --url=ssh://[email protected]/user/repo \
  --branch=main \
  --private-key-file=<path-to-your-key> \
  --path=<path-to-cluster> \
  --components-extra image-reflector-controller,image-automation-controller

You can also bootstrap using token, check previous post for more details.

Creating the Base Application

Let’s start by creating our base application configuration:

Base Kustomization

1
2
3
4
5
6
7

# apps/demo-website/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml

Base Deployment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# apps/demo-website/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-website
  labels:
    app: demo-website
spec:
  selector:
    matchLabels:
      app: demo-website
  template:
    metadata:
      labels:
        app: demo-website
    spec:
      containers:
      - name: demo-website
        image: ghcr.io/talhajuikar/flux-demo-website:main
        imagePullPolicy: Always
        ports:
        - containerPort: 80

Base Service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# apps/demo-website/base/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: demo-website
  labels:
    app: demo-website
spec:
  selector:
    app: demo-website
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  type: ClusterIP

Base Ingress

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# apps/demo-website/base/ingress.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik-external
  name: demo-website
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`demo.plutolab.live`)
    services:
    - name: demo-website
      port: 80

Creating Environment-Specific Overlays

Now let’s create overlays for different environments, each with their specific configurations. You can customize replicas, resource limits, environment variables, and more. In dev I have split the replicas and resource limits into separate patches for clarity. In staging and production, I have combined them into a single patch for simplicity. This demonstrates how you can structure your overlays based on your team’s preferences and the complexity of your configurations.

Development Environment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# apps/demo-website/overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: dev
resources:
  - ../../base

patchesStrategicMerge:
  - replica-patch.yaml
  - resource-patch.yaml
  - config-patch.yaml
  - service-patch.yaml
  - ingress-patch.yaml

commonLabels:
  environment: dev

nameSuffix: -dev

1
2
3
4
5
6
7

# apps/demo-website/overlays/dev/replica-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-website
spec:
  replicas: 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# apps/demo-website/overlays/dev/resource-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-website
spec:
  template:
    spec:
      containers:
      - name: demo-website
        resources:
          limits:
            cpu: "0.5"
            memory: "512Mi"
          requests:
            cpu: "0.2"
            memory: "256Mi"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# apps/demo-website/overlays/dev/config-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-website
spec:
  template:
    spec:
      containers:
      - name: demo-website
        env:
        - name: BACKGROUND_COLOR
          value: "#f5f5f5"
        - name: ENVIRONMENT
          value: "dev"

1
2
3
4
5
6
7
8
9
10
11
12

# apps/demo-website/overlays/dev/ingress-patch.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: demo-website
spec:
  routes:
  - kind: Rule
    match: Host(`dev.plutolab.live`)
    services:
    - name: demo-website-dev
      port: 80

Staging Environment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# apps/demo-website/overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: staging
resources:
  - ../../base

patchesStrategicMerge:
  - deployment-patch.yaml
  - ingress-patch.yaml

commonLabels:
  environment: staging

nameSuffix: -staging

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# apps/demo-website/overlays/staging/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-website
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: demo-website
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
          requests:
            cpu: "0.5"
            memory: "512Mi"
        env:
        - name: BACKGROUND_COLOR
          value: "#d4edda"
        - name: ENVIRONMENT
          value: "staging"

1
2
3
4
5
6
7
8
9
10
11
12

# apps/demo-website/overlays/staging/ingress-patch.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: demo-website
spec:
  routes:
  - kind: Rule
    match: Host(`staging.plutolab.live`)
    services:
    - name: demo-website-staging
      port: 80

Production Environment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# apps/demo-website/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod
resources:
  - ../../base
  - hpa.yaml
patchesStrategicMerge:
  - deployment-patch.yaml
  - ingress-patch.yaml

commonLabels:
  environment: prod

nameSuffix: -prod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# apps/demo-website/overlays/prod/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-website
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: demo-website
        resources:
          limits:
            cpu: "2"
            memory: "2Gi"
          requests:
            cpu: "1"
            memory: "1Gi"
        env:
        - name: BACKGROUND_COLOR
          value: "#fc6f03"
        - name: ENVIRONMENT
          value: "prod"

1
2
3
4
5
6
7
8
9
10
11
12

# apps/demo-website/overlays/prod/ingress-patch.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: demo-website
spec:
  routes:
  - kind: Rule
    match: Host(`prod.plutolab.live`)
    services:
    - name: demo-website-prod
      port: 80

In production, we will also set up a Horizontal Pod Autoscaler (HPA) to manage scaling based on CPU and memory usage.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# apps/demo-website/overlays/prod/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: demo-website-prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: demo-website-prod
  minReplicas: 5
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Creating Flux Kustomization Resources

Since the repository where the kustomization files are stored is the same as the one where the Flux configuration is stored, you can skip creating a separate GitRepository resource. However, if you want to keep your Flux configuration and application manifests in separate repositories, you would need to create a GitRepository resource as shown below. Then push the GitRepository resource to the git repository that Flux is monitoring.

1
2
3
4
5
6
7
8
9
10

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-demo-website
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/TalhaJuikar/flux-demo-website.git
  ref:
    branch: main

Now let’s create the Flux Kustomization resources that will deploy our applications to different environments:

Development Environment Kustomization

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# clusters/flux-demo/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-demo-website-dev
  namespace: flux-system
spec:
  interval: 5m
  path: "./apps/demo-website/overlays/dev"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  timeout: 2m0s
  retryInterval: 1m0s

Staging Environment Kustomization

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# clusters/flux-demo/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-demo-website-staging
  namespace: flux-system
spec:
  interval: 5m
  path: "./apps/demo-website/overlays/staging"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  timeout: 5m0s
  retryInterval: 2m0s

Production Environment Kustomization

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# clusters/flux-demo/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-demo-website-prod
  namespace: flux-system
spec:
  interval: 5m
  path: "./apps/demo-website/overlays/prod"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  timeout: 10m0s
  retryInterval: 5m0s

In my case, I have a single cluster with different namespaces for each environment, so there will be only one apps.yaml file in the clusters/flux-demo directory.

Verifying Deployment

Wait for Flux to reconcile the kustomizations. Once flux has reconciled the kustomizations, you can verify that the resources have been created in the respective namespaces:

1
2
3
4
5

# Check the status of git repositories
flux get source git -A

# Check the status of kustomizations
flux get kustomizations -n flux-system

You should see the kustomizations for each environment listed with their respective statuses.

1
2
3
4
5

NAME                            REVISION                SUSPENDED       READY   MESSAGE
flux-demo-website-dev           main@sha1:7d3938ce      False           True    Applied revision: main@sha1:7d3938ce
flux-demo-website-prod          main@sha1:7d3938ce      False           True    Applied revision: main@sha1:7d3938ce
flux-demo-website-staging       main@sha1:7d3938ce      False           True    Applied revision: main@sha1:7d3938ce
flux-system                     main@sha1:7d3938ce      False           True    Applied revision: main@sha1:7d3938ce

If everything is set up correctly, you should see the resources created in the respective namespaces.

1
2
3
4
5
6

# Check resources in the dev namespace
kubectl get all -n dev
# Check resources in the staging namespace
kubectl get all -n staging
# Check resources in the prod namespace
kubectl get all -n prod

You can now access your application in each environment using the respective hostnames configured in the Ingress resources.

Advanced Kustomization Features

Variable Substitution

Flux supports variable substitution in your manifests using the postBuild.substitute field:

1
2
3
4
5
6
7
8
9
10
11
12

# In your Kustomization
spec:
  postBuild:
    substitute:
      cluster_name: "my-cluster"
      app_version: "v1.2.3"
      replica_count: "3"
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
      - kind: Secret
        name: cluster-secrets

Then in your manifests, use ${var_name} syntax:

1
2
3
4
5
6
7
8
9
10
11
12
13

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    cluster: ${cluster_name}
spec:
  replicas: ${replica_count}
  template:
    spec:
      containers:
      - name: app
        image: my-app:${app_version}

Dependency Management

In complex deployments, you may need to control the order in which resources are applied. The Kustomization controller allows you to specify dependencies using the dependsOn field. You can define dependencies between kustomizations to ensure that certain resources are applied before others. This is particularly useful for managing infrastructure components like databases, message queues, or ingress controllers that must be ready before your application is deployed.

1
2
3
4
5
6
7
8
9
10

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
spec:
  dependsOn:
    - name: infrastructure
    - name: cert-manager
    - name: ingress-controller
  # ... rest of spec

Health Checks and Wait Conditions

Configure health checks to ensure resources are ready before proceeding:

1
2
3
4
5
6
7
8
9
10
11
12

spec:
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: my-app
      namespace: default
    - apiVersion: v1
      kind: Service
      name: my-app-service
      namespace: default
  wait: true
  timeout: 10m0s

Monitoring and Troubleshooting

Checking Kustomization Status

1
2
3
4
5
6
7
8
9
10
11

# List all kustomizations
flux get kustomizations

# Get detailed status
kubectl describe kustomization flux-demo-website-prod -n flux-system

# Check events
flux events

# View logs
flux logs --kind=Kustomization --name=flux-demo-website-prod

Common Issues and Solutions

  1. Reconciliation Failures: Check source repository access and path correctness
  2. Dependency Issues: Ensure dependent resources are healthy before deploying
  3. Resource Conflicts: Use proper namespacing and unique naming
  4. Timeout Issues: Adjust timeout values based on resource complexity

Conclusion

The Flux Kustomize Controller provides a powerful foundation for managing complex, multi-environment Kubernetes deployments. By combining Kustomize’s configuration management capabilities with Flux’s GitOps automation, you can create robust, scalable deployment pipelines that maintain consistency across environments while allowing for necessary customizations.

In the next part of this series, we’ll explore Image Automation with Flux, which will enable you to automate the deployment of container images and manage image updates seamlessly.

Have you implemented multi-environment deployments with Flux and Kustomize? Share your experiences and challenges in the comments below!

Resources

Kubernetes, DevOps, GitOps
kubernetes flux gitops kustomize configuration-management multi-environment overlays devops continuous-delivery infrastructure-as-code
This post is licensed under CC BY 4.0 by the author.