A Shift-Left Philosophy for Kubernetes Manifest Validation

Introduction

The Current Paradigm

The generation and validation of manifests are typically performed during the deployment process itself, often interacting directly with the Kubernetes API. This conventional method varies, with some teams employing a straightforward helm release, while others utilize a GitOps continuous delivery tool like Argo CD, coupled with configuration management solutions like Helm charts or Kustomize to create the manifests.

The Shift-Left Concept: Early Validation

The concept of ‘Shift-Left’ in Kubernetes development advocates for the advance validation of Kubernetes manifests earlier in the development lifecycle. The optimal scenario is to perform these checks locally, as early as possible. This approach was highlighted by Alexander Zielenski and Stefan Schimanski in their discussion Shift-Left: Past, Present, and Future of Validation in CI for GitOps Workflows at KubeCon + CloudNativeCon, where they introduced ‘kubectl-validate’. This tool, developed by the Kubernetes Special Interest Group for Command Line Interfaces (SIG-CLI), enables local validation of manifests, a critical step given that these files must be accessible locally or within a Git repository for this process to function effectively.

Moreover, the discussion will extend to another significant concept introduced at the same conference by Nicholas Morey: The Rendered Manifests Pattern: Reveal Your True Desired State. The strategy of taking the plain manifest as the source of truth is the basis for further local validation.

The exploration of these concepts begins with the Rendered Manifests Pattern, as it sets the stage for effective validation by ensuring the necessary manifests are already in place.

Exploring the Rendered Manifests Pattern

The Rendered Manifests Pattern provides the primary source of truth using plain manifests, contrary to the dynamic rendering approaches like Helm Charts or Kustomize employed during continuous delivery. As there are usually multiple environments with different versions, we focus on environment directories rather than the described branching approach from Nicholas Morey in his blog post The Rendered Manifests Pattern with different environment branches. This approach simplifies the later stages of the validation process by reducing complications with Git management. Also there is an interesting blog post by Codefresh about the branching approach – Stop Using Branches for Deploying to Different GitOps Environments.

In this blog post, we will not dive into the details of the release process, nor whether the manifests are generated manually or automatically, or if they are in the same or in a separate GitOps repository. For one thing, that would be too much for this blog post, and for another, Codefresh has already published a good post addressing this topic – How to Model Your GitOps Environments and Promote Releases between Them.

Here is how the directory structure will appear:

├── environments
│   ├── dev
│   │   └── manifests
│   ├── int
│   │   └── manifests
│   └── prd
│       └── manifests

There are some resources which can not simply be render as plain manifests in your GitOps repository, e.g. secrets. However, this is not only a known issue with the Rendered Manifests pattern that needs to be solved, but also if the manifests were not rendered directly in the repository. There are good toolings to solve this kind of limitations e.g. the External Secrets Operator

Shift-left with kubectl-validate

Developed by the SIG-CLI, kubectl-validate is a tool designed to locally validate Kubernetes resources including native types and Custom Resource Definitions (CRDs) using schemas and rules such as the Common Expression Language (CEL), List-type or Embedded Resource.

Setting Up and Using kubectl-validate

To use kubectl-validate, one must first install the necessary software, such as Go, and then follow these steps to install the tool.

$ go install sigs.k8s.io/kubectl-validate@latest

The validation can be performed for native Kubernetes types using built-in schemas that are regularly updated to reflect the latest standards:

$ kubectl-validate ./environments/$(ENVIRONMENT)/manifests

For CRDs, the validation might require additional parameters depending on the local setup, such as the Kubernetes context or specific CRD schemas:

$ kubectl-validate ./environments/$(ENVIRONMENT)/manifests --local-crds ./environments/$(ENVIRONMENT)/manifests

Validating Native Kubernetes Types

Native Kubernetes resources such as services, deployments, and pods are fundamental units in Kubernetes architecture. These resources come with built-in schemas that are regularly updated to ensure compatibility and functionality. For developers and operations teams looking to validate these resources, the kubectl-validate command is a straightforward solution. This tool allows for local validation of Kubernetes manifests, providing a quick way to check if your configurations meet the required standards before proceeding to deployment.

Here’s a simple command to perform validation:

$ kubectl-validate ./environments/$(ENVIRONMENT)/manifests

Extending Validation to Custom Resource Definitions (CRD)

While native types cover a lot of resources, many Kubernetes applications rely on Custom Resource Definitions (CRDs) to define new types specific to their needs. CRDs extend the Kubernetes API and are validated differently from native Kubernetes types. Validation for CRDs can be achieved through OpenAPI v3 schemas, and Kubernetes’ own extension mechanisms such as x-kubernetes-validations when the Validation Rules feature is enabled.

For instance, validating a CRD with a locally running minkube cluster might look like this using kubectl-validate:

$ kubectl-validate ./environments/$(ENVIRONMENT)/manifests --kube-context minikube-shift-left

Alternatively, if the CRDs are locally available and not deployed to any cluster:

$ kubectl-validate ./environments/$(ENVIRONMENT)/manifests --local-crds ./environments/$(ENVIRONMENT)/manifests
$ kubectl-validate ./environments/$(ENVIRONMENT)/manifests --local-schemas ./environments/$(ENVIRONMENT)/manifests

These commands facilitate local validation of CRDs, enabling developers to verify changes independently of a live Kubernetes environment.

Practical Demonstration: Validating a CronTab CRD

The source code from the practical demonstration can be found on github.

To illustrate how CRD validation works in practice with kubectl-validate, consider a defined CRD for a CronTab:

      ...
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-validations:
                - rule: "self.minReplicas <= self.replicas"
                  message: "replicas should be greater than or equal to minReplicas."
                - rule: "self.replicas <= self.maxReplicas"
                  message: "replicas should be smaller than or equal to maxReplicas."
      ...

The CronTab has rules set within its schema to ensure that the number of replicas does not exceed the maximum allowed. If an attempt is made to deploy this CRD with invalid values (e.g., replicas set to 20 when the maximum is 10), the validation will fail, both in-cluster and locally.

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
  minReplicas: 0
  replicas: 20
  maxReplicas: 10

Validation performed by Kubernetes API

The traditional approach involves deploying the manifest to the cluster and then checking the feedback from the API to see if it meets the requirements.

$ kubectl apply -f environments/dev/manifests/crontab.yml

The API would typically respond with a message like the following:

The CronTab "my-new-cron-object" is invalid: spec: Invalid value: "object": replicas should be smaller than or equal to maxRepas.

Validation performed by kubectl-validate

Local Validation with kubectl-validate without a running Kubernetes environment.

$ kubectl-validate "./environments/dev/manifests/crontab.yml" --local-schemas "./environments/dev/manifests"

kubectl-validate responds with a very similar response to the kubernetes API, the schema definitions are simply loaded locally.

./environments/dev/manifests/crontab.yml...ERROR
CronTab.stable.example.com "my-new-cron-object" is invalid: spec: Invalid value: "object": replicas should be smaller than or equal to maxReplicas.
Error: validation failed

Conclusion

The Benefits of Early Validation

By employing tools like kubectl-validate, developers can significantly reduce the time and resources spent on late-stage error handling in the deployment pipeline. This proactive approach to configuration validation not only enhances security but also boosts the overall efficiency of your CI/CD pipeline.

Furthermore, integrating these validation checks into early stages, such as through pre-commit hooks, can prevent errors from ever making it into the repository, thereby maintaining a higher standard of code quality from the outset.

Unsupported Validation Capabilities

As Kubernetes continues to evolve, as well does the tooling around it, promising even more robust solutions for managing the complexities of modern software deployment in a cloud-native landscape. Kubectl-validate has planned to support Validation ratcheting or ObjectReference resolution in the near future.