Go Kubernetes Operator SDK Tutorial: Build a Controller from Scratch

This is the first article in a three-article Go-based Kubernetes Operator tutorial. The goal is not to show a toy code fragment; the goal is to build a real DemoApp operator in small checkpoints so a new reader can understand what every generated file and every line of controller code is doing.

The series uses the standard Go operator stack:

• Operator SDK to scaffold the project.
• Kubebuilder markers to generate CRDs and RBAC.
• controller-runtime to run the manager, cache, client, and reconcile loop.
• kind to test the operator in a real Kubernetes cluster.

This article builds the foundation: a DemoApp custom resource that creates and maintains a Kubernetes Deployment.

If you search for "Kubernetes operator tutorial in Go", "Operator SDK tutorial", "Kubebuilder controller-runtime example", or "how to write a custom controller in Kubernetes", most examples show the same first step: create a custom resource and write a reconciler. The difficult part for beginners is not the command itself. The difficult part is understanding which generated files are source code, which files are generated output, why the CRD alone does nothing, how Reconcile() is called, and why the controller must be idempotent.

That is the focus here. This article explains the moving pieces slowly:

• the CRD is the API contract users submit
• the Go API type is the source of truth for the CRD schema
• Kubebuilder markers generate OpenAPI validation and RBAC
• controller-runtime watches objects and calls Reconcile
• the reconciler compares desired state with actual state
• owner references connect child objects back to the custom resource
• CreateOrUpdate keeps repeated reconciles safe

The official Operator SDK Go tutorial and the Kubebuilder book are excellent references, but they move quickly. This tutorial uses the same standard tooling and fills in the explanations new platform engineers usually need when they build their first Go-based Kubernetes operator.

Build a Production-Style Reconciler with controller-runtime adds the core operator capabilities: multiple child resources, status conditions, finalizers, drift handling, watches, RBAC, and webhooks.

Testing Kubernetes Operators with envtest and kind proves and ships the operator: fake-client tests, envtest, kind smoke tests, image build, Kustomize deploy, safe upgrades, and troubleshooting.

Go operator series (3 parts): Part 1 — foundation (this page) ·

• Part 2 — status, finalizers, webhooks, drift
• Part 3 — envtest, fake client, kind, upgrades
• Full Kubernetes operator tutorial hub

What you will build

At the end of this article, a user can apply this custom resource:

apiVersion: demo.example.com/v1alpha1
kind: DemoApp
metadata:
  name: hello
  namespace: default
spec:
  image: nginx:1.27
  replicas: 2
  port: 80
  message: "hello from the Go operator"

The operator will create this child workload:

• a Deployment named hello
• two replicas
• container image nginx:1.27
• container port 80
• an environment variable named DEMO_MESSAGE
• an owner reference pointing back to the DemoApp

That is enough to teach the basic loop:

1. Watch a custom resource.
2. Read the desired state from .spec.
3. Build the desired child object.
4. Create or update the child object.
5. Return cleanly so the controller can reconcile again when something changes.

This is the same level-triggered model explained in Desired State vs Actual State in Kubernetes: the controller does not care which event happened; it only cares what the world should look like now.

Prerequisites

Use current tools. The exact patch version is less important than staying on modern Operator SDK, Kubebuilder layout, and controller-runtime APIs.

go version
operator-sdk version
kubectl version --client
kind version
docker version

Recommended baseline:

• Go 1.24 or newer
• Operator SDK 1.40 or newer
• kubectl 1.30 or newer
• kind 0.23 or newer
• Docker or a Docker-compatible container runtime

If you have not installed Operator SDK yet, follow Install Operator-SDK on Linux first.

Create a kind cluster for the tutorial:

kind create cluster --name go-operator
kubectl config use-context kind-go-operator

Step 1 - Create the Go operator project

Create a clean working directory:

mkdir -p ~/operators/demoapp-operator
cd ~/operators/demoapp-operator

Initialize the project:

operator-sdk init --plugins=go/v4 --domain example.com --repo github.com/example/demoapp-operator --owner "Demo Team"

The important flags are:

The project now contains:

.
├── Dockerfile
├── Makefile
├── PROJECT
├── cmd/main.go
├── config/
└── go.mod

For beginners, the most important files are:

• cmd/main.go: starts the manager and registers controllers.
• config/: generated YAML for CRDs, RBAC, manager deployment, and samples.
• Makefile: wraps code generation, manifests, build, docker build, and deploy commands.
• go.mod: Go module dependencies.

You usually do not hand-edit the generated YAML under config/crd and config/rbac. You edit Go structs and markers, then regenerate YAML.

Step 2 - Create the DemoApp API and controller

Create one API type and one controller:

operator-sdk create api --group demo --version v1alpha1 --kind DemoApp --resource --controller

Answer y if the command asks whether to create the resource and controller. In automation or CI, you can pipe confirmations (yes | operator-sdk create api ...) so the scaffold is non-interactive.

Use --plugins=go/v4 (stable layout) with a current Operator SDK. Older docs sometimes referenced go/v4-alpha; if you see no plugin could be resolved with key "go/v4-alpha", upgrade Operator SDK or switch the init flag to go/v4 per the plugin migration notes.

This creates:

api/v1alpha1/demoapp_types.go
internal/controller/demoapp_controller.go
config/samples/demo_v1alpha1_demoapp.yaml

The split matters:

• api/v1alpha1/demoapp_types.go defines the Kubernetes API.
• internal/controller/demoapp_controller.go defines behavior.

This is the clean mental model:

CRD/API type = what users are allowed to ask for
Controller   = what the operator does about it

If that distinction is still fuzzy, read Operator vs Controller vs CRD.

Step 3 - Design the first DemoApp API

Open api/v1alpha1/demoapp_types.go and replace the generated DemoAppSpec and DemoAppStatus with this:

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// DemoAppSpec defines the desired state of DemoApp.
type DemoAppSpec struct {
	// Image is the container image used by the DemoApp Deployment.
	//
	// +kubebuilder:validation:Required
	// +kubebuilder:validation:MinLength=1
	Image string `json:"image"`

	// Replicas is the desired number of application Pods.
	//
	// +kubebuilder:default=1
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=10
	// +optional
	Replicas *int32 `json:"replicas,omitempty"`

	// Port is the container port exposed by the application.
	//
	// +kubebuilder:default=8080
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=65535
	// +optional
	Port int32 `json:"port,omitempty"`

	// Message is injected into the application as DEMO_MESSAGE.
	//
	// +kubebuilder:default="hello from DemoApp"
	// +optional
	Message string `json:"message,omitempty"`
}

// DemoAppStatus defines the observed state of DemoApp.
type DemoAppStatus struct {
	// Conditions will be filled in the controller-runtime tutorial.
	// +optional
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// ObservedGeneration will be filled in the controller-runtime tutorial.
	// +optional
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`
}

Keep the generated DemoApp, DemoAppList, and init() sections below those structs. The final file should still contain markers like these:

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
type DemoApp struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   DemoAppSpec   `json:"spec,omitempty"`
	Status DemoAppStatus `json:"status,omitempty"`
}

The markers are not comments for humans only. controller-gen reads them and generates CRD schema.

Important markers in this API:

The status fields are intentionally not used yet. This foundation tutorial keeps the controller focused. The controller-runtime tutorial turns status into a real user-facing signal.

There are a few API design choices here that are easy to miss.

Image is required because the operator cannot build a meaningful Deployment without a container image. This is a good example of validation that belongs in the CRD schema: it is simple, structural, and does not require looking at any other object.

Replicas is a pointer because Kubernetes API conventions often use pointers for optional scalar fields. A missing replicas value and a user-provided 0 are different things. In this API, 0 is invalid and a missing value defaults to 1.

Port is not a pointer because 0 is not a valid application port for this tutorial, and the controller can safely treat 0 as "default to 8080" when it sees an older or manually constructed object. The CRD default should handle normal API-server-created CRs; the controller-side fallback is defensive programming.

Message looks simple, but it introduces an important operator pattern: the custom resource presents a small user-facing API, while the controller translates that API into lower-level Kubernetes objects. Later, this field moves into a ConfigMap so the tutorial can show multi-resource reconciliation.

Good CRD design is covered more deeply in Custom Resource Definitions Explained. The short rule for this tutorial is: keep the user API small, validate what you can in schema, and avoid exposing every Deployment field directly unless your operator is only a thin wrapper around Deployment.

Step 4 - Generate Go code and CRD manifests

Run:

make generate
make manifests

What these commands do:

Inspect the generated CRD:

grep -n "image\\|replicas\\|port\\|message" config/crd/bases/demo.example.com_demoapps.yaml

You should see your schema fields under openAPIV3Schema.

Apply the CRD:

make install
kubectl get crd demoapps.demo.example.com

Now Kubernetes understands the DemoApp kind.

Step 5 - Update the sample custom resource

Edit config/samples/demo_v1alpha1_demoapp.yaml:

apiVersion: demo.example.com/v1alpha1
kind: DemoApp
metadata:
  labels:
    app.kubernetes.io/name: demoapp-operator
    app.kubernetes.io/managed-by: kustomize
  name: hello
spec:
  image: nginx:1.27
  replicas: 2
  port: 80
  message: "hello from the Go operator"

Try applying it before the controller is implemented:

kubectl apply -f config/samples/demo_v1alpha1_demoapp.yaml
kubectl get demoapp hello -o yaml

The API server accepts the object because the CRD exists. Nothing creates a Deployment yet because the controller logic is still empty.

That distinction is critical: a CRD stores data; a controller acts on data.

Delete the sample for now:

kubectl delete -f config/samples/demo_v1alpha1_demoapp.yaml

Step 6 - Add RBAC markers for Deployment management

Open internal/controller/demoapp_controller.go. The generated file already contains RBAC markers for DemoApp.

Add permissions for Deployment objects:

// +kubebuilder:rbac:groups=demo.example.com,resources=demoapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=demo.example.com,resources=demoapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=demo.example.com,resources=demoapps/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete

Regenerate manifests:

make manifests

Check the generated ClusterRole:

grep -n "deployments" config/rbac/role.yaml

This is the normal Operator SDK workflow:

1. Add RBAC markers near the reconciler.
2. Run make manifests.
3. Review the generated role.

Do not manually patch config/rbac/role.yaml unless you have a specific reason. Generated YAML should come from source markers.

Step 7 - Implement the Reconcile function

Replace the generated reconciler body with this version.

package controller

import (
	"context"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/utils/ptr"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"

	demov1alpha1 "github.com/example/demoapp-operator/api/v1alpha1"
)

// DemoAppReconciler reconciles a DemoApp object.
type DemoAppReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=demo.example.com,resources=demoapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=demo.example.com,resources=demoapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=demo.example.com,resources=demoapps/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete

func (r *DemoAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	var demoapp demov1alpha1.DemoApp
	if err := r.Get(ctx, req.NamespacedName, &demoapp); err != nil {
		if apierrors.IsNotFound(err) {
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	deployment := buildDeployment(&demoapp)
	if err := controllerutil.SetControllerReference(&demoapp, deployment, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}

	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, deployment, func() error {
		desired := buildDeployment(&demoapp)
		deployment.Labels = desired.Labels
		deployment.Spec = desired.Spec
		return controllerutil.SetControllerReference(&demoapp, deployment, r.Scheme)
	})
	if err != nil {
		return ctrl.Result{}, err
	}

	logger.Info("reconciled DemoApp deployment", "deployment", deployment.Name)
	return ctrl.Result{}, nil
}

func buildDeployment(demoapp *demov1alpha1.DemoApp) *appsv1.Deployment {
	replicas := int32(1)
	if demoapp.Spec.Replicas != nil {
		replicas = *demoapp.Spec.Replicas
	}

	labels := map[string]string{
		"app.kubernetes.io/name":       "demoapp",
		"app.kubernetes.io/instance":   demoapp.Name,
		"app.kubernetes.io/managed-by": "demoapp-operator",
	}

	port := demoapp.Spec.Port
	if port == 0 {
		port = 8080
	}

	message := demoapp.Spec.Message
	if message == "" {
		message = "hello from DemoApp"
	}

	return &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      demoapp.Name,
			Namespace: demoapp.Namespace,
			Labels:    labels,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: ptr.To(replicas),
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{
					"app.kubernetes.io/instance": demoapp.Name,
				},
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: labels,
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name:  "app",
							Image: demoapp.Spec.Image,
							Ports: []corev1.ContainerPort{
								{
									Name:          "http",
									ContainerPort: port,
								},
							},
							Env: []corev1.EnvVar{
								{
									Name:  "DEMO_MESSAGE",
									Value: message,
								},
							},
						},
					},
				},
			},
		},
	}
}

func (r *DemoAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&demov1alpha1.DemoApp{}).
		Owns(&appsv1.Deployment{}).
		Complete(r)
}

There are four important ideas in this code.

First, Get fetches the current DemoApp from the API server cache-backed client. If the object was deleted, IsNotFound is not an error. The desired object is gone, so the reconcile ends.

Second, buildDeployment is a pure desired-state builder. Given the CR, it returns the Deployment the operator wants to exist. Keeping this logic separate makes the later envtest and kind testing tutorial much easier.

Third, controllerutil.CreateOrUpdate makes the reconcile idempotent. The same reconcile can run once or one hundred times and it still converges the Deployment to the same desired state.

Fourth, Owns(&appsv1.Deployment{}) tells controller-runtime that owned Deployment changes should enqueue the owning DemoApp. If someone edits the Deployment manually, the operator gets another chance to restore the desired state.

That last point is the beginning of drift correction. The controller-runtime tutorial expands it into a more complete pattern.

Read the reconcile body as a loop, not as an event handler.

The controller is not saying:

When a DemoApp is created, create a Deployment.
When a DemoApp is updated, update a Deployment.
When a Deployment is edited, undo the edit.

It is saying:

On each reconciliation for this DemoApp,
ensure the Deployment matches the DemoApp spec.

That distinction is the heart of Kubernetes controller design. The workqueue may contain the same key several times. The cache may resync. A previous error may be retried. A child Deployment update may enqueue the parent again. The reconciler should still converge to the same result.

This is why buildDeployment is separated from the API read and why CreateOrUpdate owns the write. The builder describes desired state. CreateOrUpdate compares desired state with current state and performs the required API operation. The controller does not need a separate create path and update path for the first version of this tutorial.

The dedicated Reconcile Loop Explained article goes deeper into workqueues, retries, requeues, and idempotency.

Step 8 - Run the controller locally

Make sure the CRD is installed:

make install

Run the controller on your workstation:

make run

In another terminal, apply the sample CR:

kubectl apply -f config/samples/demo_v1alpha1_demoapp.yaml

Check the objects:

kubectl get demoapp hello
kubectl get deployment hello
kubectl get pods -l app.kubernetes.io/instance=hello

Validated output:

kubectl get demoapp hello
# NAME    AGE
# hello   18s

kubectl get deployment hello
# NAME    READY   UP-TO-DATE   AVAILABLE
# hello   2/2     2            2

kubectl get pods -l app.kubernetes.io/instance=hello
# NAME                    READY   STATUS    RESTARTS
# hello-7866b4dc4-2j6nh   1/1     Running   0
# hello-7866b4dc4-kmhtn   1/1     Running   0

Describe the Deployment and look for the owner reference:

kubectl get deployment hello -o yaml | grep -A8 ownerReferences

If the owner reference exists, Kubernetes understands that the Deployment belongs to the DemoApp. For how GC uses that link, see Owner references and garbage collection.

Now edit the CR:

kubectl patch demoapp hello --type=merge -p '{"spec":{"replicas":3}}'
kubectl get deployment hello -o jsonpath='{.spec.replicas}{"\n"}'

The Deployment should move to three replicas.

Validated output:

demoapp.demo.example.com/hello patched
3

Stop make run with Ctrl+C.

Step 9 - Deploy the operator into kind

Running locally is useful during development, but a real operator runs as a Pod.

Set an image name:

IMG=demoapp-operator:part1

Build the manager image:

make docker-build IMG=$IMG

Load it into kind:

kind load docker-image $IMG --name go-operator

Deploy the operator:

make deploy IMG=$IMG

Wait for the manager:

kubectl -n demoapp-operator-system rollout status deploy/demoapp-operator-controller-manager
kubectl -n demoapp-operator-system get pods

Validated output (Pod name suffix varies):

deployment "demoapp-operator-controller-manager" successfully rolled out

NAME                                                   READY   STATUS    RESTARTS   AGE
demoapp-operator-controller-manager-xxxxxxxxxx-xxxxx   1/1     Running   0          45s

If the Pod is not Running, inspect logs:

kubectl -n demoapp-operator-system logs deploy/demoapp-operator-controller-manager -c manager

Apply the sample again:

kubectl apply -f config/samples/demo_v1alpha1_demoapp.yaml
kubectl get deployment hello

Validated output (names and IPs vary):

demoapp.demo.example.com/hello created

NAME    READY   UP-TO-DATE   AVAILABLE
hello   2/2     2            2

At this point the operator is running inside the cluster, using the generated RBAC and manager Deployment.

Step 10 - Verify Kubernetes validation

Try an invalid CR:

kubectl apply -f - <<'EOF'
apiVersion: demo.example.com/v1alpha1
kind: DemoApp
metadata:
  name: invalid
spec:
  image: nginx:1.27
  replicas: 99
EOF

The API server should reject it because replicas has Maximum=10.

Validated output:

The DemoApp "invalid" is invalid: spec.replicas: Invalid value: 99: spec.replicas in body should be less than or equal to 10

Try a CR without replicas:

kubectl apply -f - <<'EOF'
apiVersion: demo.example.com/v1alpha1
kind: DemoApp
metadata:
  name: defaulted
spec:
  image: nginx:1.27
EOF

kubectl get demoapp defaulted -o jsonpath='{.spec.replicas}{"\n"}'

You should see:

1

This defaulting happened at the API server because of the CRD schema, not because of your controller code.

That difference matters:

• CRD defaulting shapes the stored object.
• Controller defaults protect the reconciler from older CRs or unstructured clients.
• Admission webhooks can perform more advanced defaulting and validation. The controller-runtime tutorial adds those.

Tutorial checkpoint

You now have a working Go operator that:

• defines a DemoApp API
• generates a strict CRD from Go types
• validates and defaults basic fields
• reconciles a Deployment from the custom resource
• uses owner references
• watches the owned Deployment
• runs locally with make run
• runs inside kind as a manager Pod

This is the foundation. It is useful, but it is not production-style yet.

Missing pieces that the controller-runtime tutorial adds:

• Service and ConfigMap child resources
• status conditions
• finalizer cleanup
• drift handling across multiple resources
• watched Secret/ConfigMap inputs
• defaulting and validating webhooks
• tighter RBAC markers
• Events for user-visible state changes

The envtest and kind testing tutorial then tests and ships the operator.

Common beginner mistakes

Editing generated CRD YAML by hand

Edit Go types and Kubebuilder markers, then run make manifests. Hand-edited generated YAML is usually lost on the next regeneration.

Forgetting make install

The API server cannot store DemoApp objects until the CRD is installed.

Expecting the CRD to create Pods

A CRD only registers an API type. The controller creates Pods indirectly by creating a Deployment.

Missing RBAC for child resources

If the manager logs say deployments.apps is forbidden, your RBAC markers are missing or make deploy used stale manifests.

Updating immutable Deployment selector labels

Keep spec.selector.matchLabels stable. Changing a Deployment selector after creation is rejected by Kubernetes.

Using event-specific logic

Reconcile should not ask, "Was this a create event or update event?" It should ask, "Given the current desired state, what should exist now?"

make generate / post-scaffold failures (unable to run post-scaffold tasks, Error 127)

These almost always mean the Makefile could not build or run controller-gen during operator-sdk create api. Checklist:

• Go install is clean — a broken upgrade (unpacking a new Go tarball over an old tree without removing it first) can break the assembler or standard library; reinstall from a fresh directory. Match a supported Go release for your Operator SDK version.
• Run make generate manually after scaffold and read the first failing line; command not found or missing bin/controller-gen usually means the Makefile never finished downloading tools — ensure GOBIN/PATH are not blocking go install into bin/.
• Version skew — very old Operator SDK with very new Go (or the reverse) is a common upstream issue; align operator-sdk version with the docs you follow.

If the controller never reacts to changes, confirm kubectl config current-context points at your dev cluster and that SetupWithManager watches the primary For type plus any children you expect to drive drift (here: Owns(Deployment)).

What's next?

Continue to Part 2: Kubernetes Operator with controller-runtime — status, finalizers, webhooks, and drift. For testing and shipping, follow Part 3: envtest, fake client, and kind. The Kubernetes operator tutorial hub lists every chapter in recommended order.