Helm Hybrid Operator Tutorial Part 1 of 2 - Build the Foundation (Go + Helm v4 SDK)

A Helm hybrid operator is a Go-based Kubernetes operator whose Reconcile function calls the Helm SDK directly - it is the pre-built helm-operator binary, written by you, with full control over every line. The Helm hybrid operator pattern gives you everything the pre-built Helm operator gives you (Helm install/upgrade/uninstall driven by CR events) plus everything the pre-built operator cannot do: custom finalizers against external systems, custom status fields, custom decision logic, cross-CR coordination.

This is a two-part tutorial. Part 1 of 2 (this article) lays the foundation: scaffold the Go operator, define the CRD via Go API types, embed the chart, wire the Helm SDK, and implement install/upgrade. Part 2 of 2 adds the features that justified going from scratch in the first place - custom status, a real finalizer, drift via Owns(), and a worked cross-CR coordination demo.

A note on terminology. Searches for "helm hybrid operator" today often land on the operator-framework helm-operator-plugins library (scaffolded by operator-sdk init --plugins=hybrid.helm.sdk.operatorframework.io), which runs a Helm reconciler and a Go reconciler side-by-side in the same manager and still bundles the Helm v3 Go SDK as of mid-2026. This article uses "Helm hybrid operator" in a different (and arguably more useful) sense: a single Go reconciler that calls the Helm v4 SDK directly, with no helm-operator-plugins dependency. Either pattern lifts the operator above Level I of the capability model; the path here is the one you want when you would rather own every line than configure a generic reconciler.

Here is the Helm hybrid operator pattern in one table, framed against the pre-built path so you can decide which to read:

Prerequisites: Go 1.26+ (Helm v4 SDK requires go 1.26.0 in go.mod; go get helm.sh/helm/v4 will auto-bump your module directive), operator-sdk v1.40+, helm v4.x, kubectl, kind, Docker. Basic Go reading ability (you are not writing a Go application from scratch - mostly you are filling in three or four functions in the scaffolded controller). If you have built the pre-built Helm operator (Part 1), this article picks up the same demo-app chart so the comparison is apples-to-apples - same Deployment, Service, ConfigMap, Secret, just rendered by a Reconcile loop you own.

What "hybrid" means here (and how it differs from the pre-built helm plugin)

The pre-built helm-operator binary (the helm.sdk.operatorframework.io/v1 plugin) ships from Operator SDK as a pre-compiled, generic reconciler. You provide a chart and a watches.yaml; the binary does the rest. Nothing you write is Go. This is one valid operator design pattern, and it sits at Level I of the capability model.

A Helm hybrid operator is the same idea, written by you in Go. The chart still renders manifests; the Helm SDK still calls helm install / upgrade / uninstall. The difference is that your Reconcile function decides when and how those calls happen, and you can put anything else around them - validation, external API calls, custom status writes, finalizers with retries. By owning the reconcile body the hybrid pattern unlocks Level II and above of the operator capability / maturity model, which the pre-built plugin's three fixed conditions cannot reach.

A second "hybrid" to disambiguate. The Operator SDK also ships a separate --plugins=hybrid.helm.sdk.operatorframework.io scaffold that wires the operator-framework helm-operator-plugins library into main.go and mounts a Helm reconciler (built with reconciler.New(reconciler.WithChart(*chart), reconciler.WithGroupVersionKind(gvk))) alongside a Go reconciler in the same manager. That approach lets you customise the Helm reconciler's event recorder / logger / pre-/post-hooks (covered in Part 2) but inherits the library's still-Helm-v3 implementation. This article deliberately rejects the builder API: a single hand-written Reconcile that calls helm.sh/helm/v4 directly is simpler, depends on one less library, and lets you adopt Helm v4 today.

The cost is volume: about 350-450 lines of Go across both parts, vs zero in the pre-built. The payoff is the entire right-hand column. Conceptually this is the same Kubernetes operator vs controller vs CRD split you have always had - the difference is what you do inside the controller body.

What you will build across the two-part series

The same demo-app chart as the pre-built tutorial - four templates (Deployment, Service, ConfigMap, Secret) driven by a DemoApp CR. The user-visible behaviour at the end of Part 1 is identical to the pre-built operator. The difference comes in Part 2.

Prerequisites

go version                    # go1.26.0 or newer (required by Helm v4 SDK)
operator-sdk version          # v1.40.0+ (this article tested with v1.42.2)
helm version --short          # v4.x (this article tested with v4.2.0)
kubectl version --client      # any modern version
kind version                  # any modern version
docker version --format '{{.Server.Version}}'

Why Go 1.26? Helm 4.2.0's go.mod declares go 1.26.0. The first go get helm.sh/helm/v4@latest you run inside the scaffolded project will automatically bump your go.mod's go directive from the operator-sdk default (1.24.0) to 1.26.0 to satisfy this. If your local Go has GOTOOLCHAIN=auto, it can download and use the newer toolchain automatically; otherwise install Go 1.26+ on the host. Either way, bump the FROM golang:1.24 line in the scaffolded Dockerfile to golang:1.26 before make docker-build.

Spin up a kind cluster if you do not have one:

kind create cluster --name hybrid --image kindest/node:v1.31.0
kubectl config use-context kind-hybrid

Image distribution: this article builds a local Go operator image and needs to ship it into the kind cluster. We use ttl.sh — a public, ephemeral, zero-setup container registry — exactly as the pre-built Helm operator Part 1 and the prereq article do. Pattern: make docker-build IMG="$IMG" → docker push "$IMG" → make deploy IMG="$IMG" with IMG=ttl.sh/demoapp-hybrid-$(uuidgen):24h. Anything pushed to ttl.sh is public — don't use it for proprietary code.

Part A - Scaffold the Go operator

Step 1 - operator-sdk init --plugins=go/v4

mkdir -p ~/hybrid-operator/demoapp
cd ~/hybrid-operator/demoapp

operator-sdk init --plugins=go/v4 --domain example.com --repo github.com/example/demoapp --owner "Demo Team"

Three things just happened:

1. A Go module was initialised (go.mod with controller-runtime, client-go, ginkgo).
2. The Kubebuilder layout was scaffolded (cmd/, api/ placeholder, config/, Dockerfile, Makefile).
3. A PROJECT file was created recording the layout version (v4 is the current Kubebuilder layout).

Step 2 - operator-sdk create api

Add the DemoApp Kind:

operator-sdk create api --group demo --version v1alpha1 --kind DemoApp --resource --controller

This creates two new files:

• api/v1alpha1/demoapp_types.go - skeleton Spec and Status structs.
• internal/controller/demoapp_controller.go - skeleton Reconcile function and SetupWithManager.

Plus api/v1alpha1/groupversion_info.go and api/v1alpha1/zz_generated.deepcopy.go (the latter regenerated by make generate).

With current Operator SDK releases, create api also runs make generate automatically after creating the files. On a first run this may download controller-gen; the explicit make generate step below is still important after you edit *_types.go.

Step 3 - Project folder structure tour

Stripped to just the directories you will touch in this Part 1:

demoapp/
├── cmd/
│   └── main.go                          # operator entry point; wires manager + controllers
├── api/                                  # ← appears only AFTER `operator-sdk create api`
│   └── v1alpha1/
│       ├── demoapp_types.go             # YOU EDIT: Spec, Status, Kubebuilder markers
│       ├── groupversion_info.go         # generated
│       └── zz_generated.deepcopy.go     # regenerated by 'make generate'
├── internal/
│   └── controller/                       # ← also appears only after `create api`
│       ├── demoapp_controller.go        # YOU EDIT: Reconcile() function
│       ├── demoapp_controller_test.go   # generated test scaffold
│       └── suite_test.go                # generated test suite
├── config/
│   ├── crd/
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── bases/                       # appears after `make manifests`
│   ├── default/                         # kustomize base for 'make deploy'
│   ├── manager/                         # the operator Deployment
│   ├── rbac/                            # ClusterRole, ServiceAccount, RoleBinding
│   ├── samples/                         # generated sample CR YAML
│   ├── manifests/                       # OLM bundle manifests (unused here)
│   ├── network-policy/                  # default network-policy kustomization (unused here)
│   ├── prometheus/                      # ServiceMonitor for the metrics endpoint (unused here)
│   └── scorecard/                       # operator-sdk scorecard tests (unused here)
├── Dockerfile                            # multi-stage Go build (defaults to golang:1.24 — bump to 1.26 for Helm v4)
├── Makefile                              # generate / manifests / docker-build / deploy
├── PROJECT                               # operator-sdk metadata
├── hack/                                 # boilerplate header, etc.
└── go.mod / go.sum

The two files you edit during this Part 1 are: api/v1alpha1/demoapp_types.go (the API) and internal/controller/demoapp_controller.go (the reconcile logic). Everything else stays scaffolded.

Note: operator-sdk v1.40+ also scaffolds test/, .devcontainer/, .github/, .golangci.yml, and README.md. They are unrelated to this tutorial; leave them alone.

Step 4 - How this differs from the pre-built helm plugin

The single biggest mental shift is: in the pre-built path, the operator binary is the same for every Helm operator, configured by watches.yaml. In the hybrid path, the operator binary is your code, compiled with your reconciler and your embedded chart.

Part B - Define the CRD via Go types

Step 5 - Edit api/v1alpha1/demoapp_types.go

The scaffolded file has empty Spec and Status structs. Replace them with the demo-app chart's value shape:

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// DemoAppSpec defines the desired state of DemoApp.
type DemoAppSpec struct {
	// Replicas is the number of nginx Pods to run.
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=10
	// +kubebuilder:default=1
	Replicas int32 `json:"replicaCount,omitempty"`

	// Image is the container image for the nginx Deployment.
	// +kubebuilder:validation:Pattern=`^[a-z0-9./:-]+$`
	// +kubebuilder:default="nginx:1.27-alpine"
	Image string `json:"image,omitempty"`

	// Message is the HTML body served by the Pod.
	// +kubebuilder:validation:MinLength=1
	// +kubebuilder:validation:MaxLength=200
	Message string `json:"message"`

	// ApiKey is mounted into the Pod as an env var (demonstration only).
	// +kubebuilder:validation:MinLength=8
	ApiKey string `json:"apiKey,omitempty"`

	// Service controls the in-cluster Service in front of the Pods.
	// +kubebuilder:default={type: ClusterIP, port: 80}
	Service ServiceSpec `json:"service,omitempty"`
}

// ServiceSpec is the Service section of DemoApp.spec.
type ServiceSpec struct {
	// +kubebuilder:validation:Enum=ClusterIP;NodePort;LoadBalancer
	// +kubebuilder:default="ClusterIP"
	Type string `json:"type,omitempty"`

	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=65535
	// +kubebuilder:default=80
	Port int32 `json:"port,omitempty"`
}

// DemoAppStatus defines the observed state of DemoApp.
// Part 1 keeps this minimal; Part 2 adds custom status fields.
type DemoAppStatus struct {
	// Conditions reflect the observed state of the DemoApp.
	// +listType=map
	// +listMapKey=type
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// ObservedGeneration tracks the last spec generation reconciled.
	// +optional
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=".spec.replicaCount"
// +kubebuilder:printcolumn:name="Message",type=string,JSONPath=".spec.message"
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=".metadata.creationTimestamp"
type DemoApp struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   DemoAppSpec   `json:"spec,omitempty"`
	Status DemoAppStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true
type DemoAppList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []DemoApp `json:"items"`
}

func init() {
	SchemeBuilder.Register(&DemoApp{}, &DemoAppList{})
}

The json tag names (replicaCount, image, message, apiKey, service) match the chart's values.yaml keys exactly - this is what makes the implicit "CR spec → Helm values" mapping work without any conversion code (we will rely on this in Part E).

Step 6 - Kubebuilder markers explained

The +listType=map markers on Conditions are important - without them, status updates use list-replace semantics instead of map-merge, which causes condition flapping under concurrent reconciles. For the canonical status subresource and Conditions contract, see the language-neutral reference.

Step 7 - make generate (DeepCopy)

make generate
# bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."

Open api/v1alpha1/zz_generated.deepcopy.go - controller-gen produced DeepCopyInto, DeepCopy, and DeepCopyObject for DemoApp, DemoAppList, DemoAppSpec, ServiceSpec, and DemoAppStatus. Controller-runtime needs these to safely cache and reconcile your resources without aliasing.

You re-run make generate whenever you edit *_types.go.

Step 8 - make manifests (CRD YAML)

make manifests
# bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

Open config/crd/bases/demo.example.com_demoapps.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: demoapps.demo.example.com
spec:
  group: demo.example.com
  names:
    kind: DemoApp
    listKind: DemoAppList
    plural: demoapps
    singular: demoapp
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required:
                - message
              properties:
                replicaCount:
                  type: integer
                  minimum: 1
                  maximum: 10
                  default: 1
                image:
                  type: string
                  pattern: '^[a-z0-9./:-]+$'
                  default: 'nginx:1.27-alpine'
                message:
                  type: string
                  minLength: 1
                  maxLength: 200
                apiKey:
                  type: string
                  minLength: 8
                service:
                  type: object
                  default: {type: ClusterIP, port: 80}
                  properties:
                    type:
                      type: string
                      enum: [ClusterIP, NodePort, LoadBalancer]
                      default: ClusterIP
                    port:
                      type: integer
                      minimum: 1
                      maximum: 65535
                      default: 80
            status:
              type: object
              # ... conditions schema generated from Go struct ...
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Replicas
          type: integer
          jsonPath: .spec.replicaCount
        - name: Message
          type: string
          jsonPath: .spec.message
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp

This is strict by default - the opposite of the pre-built operator's permissive x-kubernetes-preserve-unknown-fields: true. Bad CRs are rejected at admission, never reach your Reconcile. (For a language-neutral primer on how OpenAPI v3 validation maps from Go types, see the CRD explainer.)

You re-run make manifests whenever you edit *_types.go markers.

How this differs from the pre-built plugin's CRD generation

The Go types are the canonical source. The CRD YAML is derived. Never hand-edit the generated CRD YAML in a hybrid operator - your edits get overwritten next make manifests.

Part C - Embed the chart in the operator binary

Step 9 - Add the chart under internal/controller/charts/

Critical: //go:embed patterns are resolved relative to the package directory containing the directive, not the module root. Because the //go:embed directive lives in internal/controller/chart.go (next step), the chart must live next to it — putting it at the repo-root charts/ like the Kubebuilder docs suggest will fail at build time with pattern all:charts/demo-app: no matching files found.

Copy the chart into the package directory:

mkdir -p internal/controller/charts
cp -r ~/hybrid-operator/demo-app internal/controller/charts/demo-app

ls internal/controller/charts/demo-app/
# Chart.yaml  templates  values.yaml

If you do not have the chart from the pre-built tutorial, recreate the four templates per Part 1 of the pre-built tutorial.

Step 10 - //go:embed all:charts/demo-app

Create a small helper that embeds the chart and loads it from embed.FS. Add internal/controller/chart.go:

package controller

import (
	"embed"
	"fmt"
	"io/fs"
	"strings"

	"helm.sh/helm/v4/pkg/chart/loader/archive"
	chartv2 "helm.sh/helm/v4/pkg/chart/v2"
	"helm.sh/helm/v4/pkg/chart/v2/loader"
)

//go:embed all:charts/demo-app
var chartFS embed.FS

const chartRoot = "charts/demo-app"

// loadChart loads the embedded demo-app chart at startup. The returned
// *chartv2.Chart is reusable across reconciles - it's cached in the controller.
func loadChart() (*chartv2.Chart, error) {
	var files []*archive.BufferedFile

	err := fs.WalkDir(chartFS, chartRoot, func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		if d.IsDir() {
			return nil
		}
		data, err := fs.ReadFile(chartFS, path)
		if err != nil {
			return fmt.Errorf("read embedded chart file %s: %w", path, err)
		}
		// loader.LoadFiles expects chart-relative paths (e.g. "templates/deployment.yaml"),
		// not embed-relative ("charts/demo-app/templates/deployment.yaml").
		rel := strings.TrimPrefix(path, chartRoot+"/")
		files = append(files, &archive.BufferedFile{Name: rel, Data: data})
		return nil
	})
	if err != nil {
		return nil, err
	}

	chrt, err := loader.LoadFiles(files)
	if err != nil {
		return nil, fmt.Errorf("load chart from embed: %w", err)
	}
	return chrt, nil
}

// LoadChart is exported for use from cmd/main.go.
func LoadChart() (*chartv2.Chart, error) { return loadChart() }

Four notes on this:

1. //go:embed all:charts/demo-app is required (the all: prefix includes files that would normally be skipped, like _helpers.tpl whose name starts with _).
2. In Helm 4 the chart loader is versioned per chart-format. The types (Chart, Metadata) live in helm.sh/helm/v4/pkg/chart/v2. The chart-v2 loader (LoadFiles) lives in helm.sh/helm/v4/pkg/chart/v2/loader. The BufferedFile type that LoadFiles accepts lives in helm.sh/helm/v4/pkg/chart/loader/archive — note the path: chart/loader/archive, not chart/v2/archive (which doesn't exist as of Helm v4.2.0).
3. The chart we are loading uses the chart v2 format (Chart.yaml: apiVersion: v2) — the format every existing Helm chart uses today. If/when chart v3 ships, swap chart/v2/loader for chart/v3/loader; loader/archive stays the same.
4. The returned *chartv2.Chart is loaded once at controller startup and reused across reconciles. Loading on every reconcile is wasteful.

Alternatives to go:embed

For 95% of teams, //go:embed is the right default - it makes the chart immutable per operator image, which makes rollouts auditable and reproducible.

Part D - Wire the Helm v4 SDK

Step 11 - Required imports

The Helm 4 SDK is split into more sub-packages than v3 was. Here is the verified-against-v4.2.0 layout for everything the Helm hybrid operator needs in this article:

Add them to your go.mod:

go get helm.sh/helm/v4@latest
go mod tidy

Heads-up: this auto-bumps Go. As of Helm v4.2.0, go get helm.sh/helm/v4@latest rewrites your go.mod's go directive from go 1.24.0 (operator-sdk default) to go 1.26.0 (Helm v4 requirement). In a tested run with Go 1.24.4 and GOTOOLCHAIN=auto, Go downloaded and used go1.26.4 automatically (see Prerequisites for context on GOTOOLCHAIN). The same go get also upgraded controller-runtime from v0.21.0 to v0.24.0 and Kubernetes libraries to v0.36.0, which is expected with current Helm v4.

Do not skip go mod tidy. Without it, make generate can fail with missing go.sum entries for Helm/Kubernetes transitive imports loaded by controller-gen.

Why v4 and not v3? Helm 4 (stable since 2026) is the version you should target for new work. The Go SDK import path is helm.sh/helm/v4/... (not v3/...). If you have an existing v3-based operator you want to migrate, the main mechanical changes are:

• Rewrite imports v3 → v4.
• Chart loader: chart/loader → chart/v2/loader.
• BufferedFile: chart/loader → chart/loader/archive (the v3 type moved into a sub-package).
• action.Configuration.Init: drop the trailing func(format string, v ...any) logger argument — the v4 signature takes only (getter, namespace, helmDriver).
• Install.Run / Upgrade.Run now return release.Releaser (which is any); type-assert to *releasev1.Release to read fields like Name, Version, Manifest (see Step 17).
• Install.Wait / Upgrade.Wait bool fields are gone — replaced by WaitStrategy kube.WaitStrategy (and Upgrade.Run errors at runtime if WaitStrategy is unset; see Step 18).
• Upgrade.Atomic is gone — replaced by Upgrade.RollbackOnFailure and Upgrade.CleanupOnFail.

Step 12 - Build an action.Configuration per reconcile

Add internal/controller/helm.go:

package controller

import (
	"context"
	"fmt"

	"helm.sh/helm/v4/pkg/action"
	"helm.sh/helm/v4/pkg/cli"
)

// newActionConfig constructs a Helm action.Configuration scoped to namespace.
// The returned config can drive Install / Upgrade / Uninstall / History for
// any release in that namespace.
func newActionConfig(_ context.Context, namespace string) (*action.Configuration, error) {
	settings := cli.New()
	settings.SetNamespace(namespace)

	cfg := new(action.Configuration)
	err := cfg.Init(settings.RESTClientGetter(), namespace, "secret", // release storage driver - "secret" remains the default in Helm 4
	)
	if err != nil {
		return nil, fmt.Errorf("helm action.Configuration.Init: %w", err)
	}
	return cfg, nil
}

Three things to notice:

• settings.RESTClientGetter() is the v4 idiom - the Helm 4 SDK example uses it directly and it picks up the in-cluster KUBECONFIG and namespace defaults without you having to build a genericclioptions.ConfigFlags by hand (which was the older v3-era pattern).
• "secret" as the storage driver. Helm stores releases as Kubernetes Secrets named sh.helm.release.v1.<release>.v<revision> - this storage format is unchanged between Helm 3 and Helm 4, so the releases your operator creates are interchangeable with helm history, helm rollback, and the pre-built helm-operator (which still bundles the v3 SDK as of mid-2026).
• No logger argument. Helm v3's Configuration.Init accepted a func(format string, v ...any) logger as the fourth argument. Helm v4 dropped it; the SDK now logs via the global log/slog handler. If you want Helm logs in your controller-runtime logger, set the slog handler at startup in cmd/main.go.

Helm 4 default: Server-Side Apply. Helm 4 uses Server-Side Apply (SSA) by default for new installs (action.Install.ServerSideApply defaults to true; for upgrades, action.Upgrade.ServerSideApply is a tri-state string "true"/"false"/"auto"). That means the resources your action.NewInstall.Run creates carry managedFields entries owned by helm, and conflict semantics match SSA rules. For drift correction to work (Part 2), you must also set Install.ForceConflicts = true and Upgrade.ForceConflicts = true, otherwise a kubectl patch against a chart-rendered field becomes "owned" by kubectl-patch and the next operator upgrade fails with conflict occurred while applying object … Apply failed with 1 conflict.

Step 13 - Load the chart at controller startup

In internal/controller/demoapp_controller.go, hold the chart on the reconciler struct so it loads once:

import (
	chartv2 "helm.sh/helm/v4/pkg/chart/v2"
)

type DemoAppReconciler struct {
	client.Client
	Scheme *runtime.Scheme

	// Chart is loaded once at startup; reused across reconciles.
	Chart *chartv2.Chart
}

Then in cmd/main.go, load it before starting the manager:

import (
	democtrl "github.com/example/demoapp/internal/controller"
)

func main() {
	// ... existing scaffold ...

	chrt, err := democtrl.LoadChart()      // export the helper from chart.go
	if err != nil {
		setupLog.Error(err, "load embedded chart")
		os.Exit(1)
	}

	if err = (&democtrl.DemoAppReconciler{Client: mgr.GetClient(), Scheme: mgr.GetScheme(),	Chart:  chrt}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "DemoApp")
		os.Exit(1)
	}

	// ... rest of scaffold ...
}

Export LoadChart from chart.go:

// LoadChart is exported for use from cmd/main.go.
func LoadChart() (*chartv2.Chart, error) { return loadChart() }

The chart is now in memory at startup and every reconcile reads from RAM, not disk.

Current Operator SDK scaffolds include additional metrics, webhook, and certificate-watcher setup in cmd/main.go; keep that code and only replace the DemoAppReconciler construction block with the chart-aware version above.

Part E - Implement Reconcile: install / upgrade

This is the core of Part 1. The full Reconcile is ~80 lines of focused Go. We will build it step by step, then show the assembled version.

Step 14 - The Reconcile skeleton

Open internal/controller/demoapp_controller.go and replace the scaffolded Reconcile:

func (r *DemoAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the CR.
	var demoapp demov1alpha1.DemoApp
	if err := r.Get(ctx, req.NamespacedName, &demoapp); err != nil {
		if apierrors.IsNotFound(err) {
			// CR deleted - Helm uninstall path is covered in Part 2 (finalizer).
			// For Part 1, rely on owner references for garbage collection.
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, fmt.Errorf("get DemoApp: %w", err)
	}

	// 2. Skip if already reconciled at this generation (idempotency).
	if demoapp.Status.ObservedGeneration == demoapp.Generation {
		logger.V(1).Info("generation matches, skip reconcile", "generation", demoapp.Generation)
		return ctrl.Result{}, nil
	}

	// 3. Build a Helm action.Configuration for this CR's namespace.
	cfg, err := newActionConfig(ctx, demoapp.Namespace)
	if err != nil {
		return ctrl.Result{}, fmt.Errorf("helm config: %w", err)
	}

	// 4. Decide install vs upgrade and run it.
	if err := r.installOrUpgrade(ctx, cfg, &demoapp); err != nil {
		// status writes go here in Part 2; Part 1 just requeues on error.
		return ctrl.Result{RequeueAfter: 30 * time.Second}, err
	}

	// 5. Mark this generation as observed.
	demoapp.Status.ObservedGeneration = demoapp.Generation
	if err := r.Status().Update(ctx, &demoapp); err != nil {
		return ctrl.Result{}, fmt.Errorf("update status: %w", err)
	}

	return ctrl.Result{}, nil
}

Five clear steps: fetch, idempotency check, build Helm config, install-or-upgrade, mark observed. Without a finalizer in Part 1, the orphan-cleanup story for chart resources falls back to owner references and garbage collection - Helm wires those up automatically when it applies the release. Part 2 will add finalizer detection, custom status writes, and richer error handling between these steps.

Step 15 - Check for an existing release (action.NewHistory)

func (r *DemoAppReconciler) installOrUpgrade(ctx context.Context, cfg *action.Configuration, demoapp *demov1alpha1.DemoApp) error {
	releaseName := demoapp.Name

	// Has Helm seen a release with this name in this namespace?
	hist := action.NewHistory(cfg)
	hist.Max = 1
	_, err := hist.Run(releaseName)

	if err == driver.ErrReleaseNotFound {
		return r.install(ctx, cfg, demoapp, releaseName)
	}
	if err != nil {
		return fmt.Errorf("helm history: %w", err)
	}
	return r.upgrade(ctx, cfg, demoapp, releaseName)
}

driver.ErrReleaseNotFound is the canonical sentinel - any other error is genuinely unexpected and should propagate.

Step 16 - Convert CR.Spec to Helm values (JSON marshal trick)

The cleanest way to turn a Go struct into the map[string]interface{} Helm wants is round-trip through JSON. The chart's values.yaml keys match the json tags on DemoAppSpec exactly (we ensured this in Step 5), so the mapping is automatic:

func specToValues(spec demov1alpha1.DemoAppSpec) (map[string]interface{}, error) {
	raw, err := json.Marshal(spec)
	if err != nil {
		return nil, fmt.Errorf("marshal spec: %w", err)
	}
	var values map[string]interface{}
	if err := json.Unmarshal(raw, &values); err != nil {
		return nil, fmt.Errorf("unmarshal to values: %w", err)
	}
	return values, nil
}

For complex specs you might curate a subset (skipping operator-only fields), but for the demo-app spec, the whole shape goes through.

Step 17 - Install path

import (
	"helm.sh/helm/v4/pkg/kube"
	release "helm.sh/helm/v4/pkg/release/v1"
)

func (r *DemoAppReconciler) install(ctx context.Context, cfg *action.Configuration,	demoapp *demov1alpha1.DemoApp, releaseName string) error {
	logger := log.FromContext(ctx)

	values, err := specToValues(demoapp.Spec)
	if err != nil {
		return err
	}

	inst := action.NewInstall(cfg)
	inst.ReleaseName = releaseName
	inst.Namespace = demoapp.Namespace
	inst.CreateNamespace = false              // CR cannot create its own namespace
	inst.WaitStrategy = kube.HookOnlyStrategy // operator-friendly: only block on hooks
	inst.ForceConflicts = true                // SSA: take ownership over manual patches

	resi, err := inst.RunWithContext(ctx, r.Chart, values)
	if err != nil {
		return fmt.Errorf("helm install: %w", err)
	}
	rel, ok := resi.(*release.Release)
	if !ok {
		return fmt.Errorf("helm install returned unexpected release type: %T", resi)
	}
	logger.Info("installed release",
		"release", rel.Name, "namespace", rel.Namespace, "revision", rel.Version)
	return nil
}

Four v4-specific knobs to call out:

• CreateNamespace: false - the CR cannot create its own namespace; namespace lifecycle is a cluster admin concern.
• WaitStrategy = kube.HookOnlyStrategy - the v3 Wait bool field is gone in v4. The replacement is WaitStrategy kube.WaitStrategy with three values: kube.StatusWatcherStrategy ("watcher", kstatus-based), kube.LegacyStrategy ("legacy", Helm 3-style polling), and kube.HookOnlyStrategy ("hookOnly", wait only for hook Jobs). For an operator, HookOnlyStrategy is the right pick: Reconcile should return fast and let the next reconcile observe rollout state, but you still want hook completion to block install (otherwise pre-install Jobs can race with the next reconcile).
• ForceConflicts = true - Helm 4 uses Server-Side Apply by default. Without this, the operator cannot overwrite fields a user has touched via kubectl patch, and the broader drift-detection patterns in operators demo in Part 2 depends on it.
• RunWithContext returns (release.Releaser, error) where release.Releaser is literally type Releaser any in v4. To read Name, Namespace, Version, Manifest, etc., you must type-assert to the concrete *releasev1.Release (imported as shown above as release "helm.sh/helm/v4/pkg/release/v1"). Helm's own SDK code does the same assertion — there is no releaserToV1Release public helper as of v4.2.0.

Step 18 - Upgrade path

func (r *DemoAppReconciler) upgrade(
	ctx context.Context, cfg *action.Configuration,
	demoapp *demov1alpha1.DemoApp, releaseName string,
) error {
	logger := log.FromContext(ctx)

	values, err := specToValues(demoapp.Spec)
	if err != nil {
		return err
	}

	up := action.NewUpgrade(cfg)
	up.Namespace = demoapp.Namespace
	up.MaxHistory = 10                       // keep last 10 revisions; default 0 = unbounded
	up.WaitStrategy = kube.HookOnlyStrategy  // REQUIRED: Upgrade errors at runtime if unset (see below)
	up.ServerSideApply = "true"              // Upgrade.ServerSideApply is a tri-state STRING in v4
	up.ForceConflicts = true                 // overwrite kubectl-patch'd fields (drift correction)

	resi, err := up.RunWithContext(ctx, releaseName, r.Chart, values)
	if err != nil {
		return fmt.Errorf("helm upgrade: %w", err)
	}
	rel, ok := resi.(*release.Release)
	if !ok {
		return fmt.Errorf("helm upgrade returned unexpected release type: %T", resi)
	}
	logger.Info("upgraded release",
		"release", rel.Name, "namespace", rel.Namespace, "revision", rel.Version)
	return nil
}

Four v4 gotchas worth memorising:

• MaxHistory is a real-world must - by default Helm keeps every revision Secret forever; with 1000 CRs and frequent updates you end up with thousands of stale Secrets and a slow API server.
• WaitStrategy is required for Upgrade. Unlike Install (which tolerates an unset WaitStrategy), Upgrade.RunWithContext errors with wait strategy not set. Choose one of: watcher, hookOnly, legacy if you forget. The error surfaces only at runtime — go build will not catch it.
• Upgrade.ServerSideApply is a string, not a bool. Valid values: "true", "false", "auto". (Install's equivalent Install.ServerSideApply IS a bool — yes, the inconsistency is real.)
• Upgrade.Atomic is gone. The replacement in v4 is Upgrade.RollbackOnFailure bool (auto-rollback on failure) and Upgrade.CleanupOnFail bool (delete newly created resources on failure). Both default to false — i.e., the v3 Atomic = false behaviour is now the default and requires no field.

Step 19 - Minimal error handling

For Part 1 we keep error handling simple: any Helm error bubbles up, Reconcile returns with RequeueAfter: 30s, and controller-runtime's exponential-backoff schedules the retry through the standard watches, events, and predicates pipeline. Part 2 will classify errors (retriable vs permanent) and write them into status.conditions.

if err := r.installOrUpgrade(ctx, cfg, &demoapp); err != nil {
	logger.Error(err, "install/upgrade failed; will retry")
	return ctrl.Result{RequeueAfter: 30 * time.Second}, err
}

SetupWithManager for Part 1

Watch only the CR itself for now. Drift via Owns() on chart-rendered resources (the broader multi-resource reconciliation pattern) comes in Part 2:

func (r *DemoAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&demov1alpha1.DemoApp{}).
		Complete(r)
}

Imports for the assembled controller file

package controller

import (
	"context"
	"encoding/json"
	"fmt"
	"time"

	"helm.sh/helm/v4/pkg/action"
	chartv2 "helm.sh/helm/v4/pkg/chart/v2"
	"helm.sh/helm/v4/pkg/kube"
	release "helm.sh/helm/v4/pkg/release/v1"
	"helm.sh/helm/v4/pkg/storage/driver"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	demov1alpha1 "github.com/example/demoapp/api/v1alpha1"
)

Reconciler struct (already shown in Step 13):

type DemoAppReconciler struct {
	client.Client
	Scheme *runtime.Scheme
	Chart  *chartv2.Chart
}

That is the full Part 1 reconciler. Total ~150 lines of Go across demoapp_controller.go, chart.go, and helm.go.

Part F - Build, deploy, demo install/upgrade

Step 20 - Dockerfile and make docker-build

The scaffolded Dockerfile is a multi-stage Go build. Before you can ship the Helm hybrid operator image, bump the builder image to match the Go version Helm v4 forced on your go.mod:

- FROM golang:1.24 AS builder
+ FROM golang:1.26 AS builder

The full file (after the bump and after our chart relocation in Step 9) looks like:

# Build the manager binary
FROM golang:1.26 AS builder
ARG TARGETOS
ARG TARGETARCH

WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download

COPY cmd/main.go cmd/main.go
COPY api/ api/
COPY internal/ internal/         # picks up internal/controller/charts/demo-app for //go:embed

RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go

# Use distroless as minimal base image
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532

ENTRYPOINT ["/manager"]

Note: because we placed the chart at internal/controller/charts/demo-app/ (Step 9), the existing COPY internal/ internal/ already brings it into the build context — no extra COPY charts/ line is needed. The //go:embed directive resolves at compile time, so the chart must be present when go build runs.

If you build behind a corporate proxy / TLS-intercepting proxy, two extra steps are usually needed: configure the Docker daemon's proxy via /etc/systemd/system/docker.service.d/http-proxy.conf (so docker pull golang:1.26 reaches Docker Hub), and either trust the proxy's CA system-wide (update-ca-certificates) or go mod vendor on the host and update the Dockerfile to COPY vendor/ vendor/ + go build -mod=vendor ... so the build container does not need to reach proxy.golang.org.

Build:

export IMG=ttl.sh/demoapp-hybrid-$(uuidgen):24h
make docker-build IMG="$IMG"
# docker build -t ttl.sh/demoapp-hybrid-<uuid>:24h .

If Docker Hub returns 429 Too Many Requests while pulling golang:1.26 or the distroless base image, authenticate with docker login, use an internal mirror, or pre-pull the base images from a network that is not rate-limited. That failure happens before your operator code is compiled.

Step 21 - push to ttl.sh + make deploy

docker push "$IMG"                        # ship the image to ttl.sh

make install                              # apply just the CRD
make deploy IMG="$IMG"                    # apply CRD + RBAC + Deployment

kubectl -n demoapp-system rollout status deploy/demoapp-controller-manager       # wait for the new pod before applying CRs

RBAC note. make deploy ships the scaffolded ClusterRole from config/rbac/. For a production Helm hybrid operator you will want to lock that down to the minimum permissions an operator actually needs - the scaffold is intentionally generous.

See the prereq article's ttl.sh section for why we use ttl.sh (zero setup, works from any cluster) instead of kind load docker-image (brittle on Docker 24+ with the containerd snapshotter) or a local registry:2 container (works but ~30 lines of cluster wiring).

Verify:

kubectl get crd demoapps.demo.example.com
# demoapps.demo.example.com   ...

kubectl -n demoapp-system get pods
# NAME                                          READY   STATUS    RESTARTS   AGE
# demoapp-controller-manager-5b9d5d757d-xdgnz   1/1     Running   0          30s

kubectl -n demoapp-system logs deploy/demoapp-controller-manager -c manager | head -10
# "Starting workers" "controller":"demoapp"

The pod shows 1/1 Running, not 2/2. Operator-sdk v1.40+ removed the kube-rbac-proxy sidecar from the default scaffold (the manager now serves authenticated operator metrics on :8443 itself).

Step 22 - Apply CR, see your install

cat <<EOF | kubectl apply -f -
apiVersion: demo.example.com/v1alpha1
kind: DemoApp
metadata:
  name: demoapp-sample
  namespace: default
spec:
  replicaCount: 1
  message: "Hello from the hybrid operator"
  apiKey: "verysecret"
EOF

Tail the operator log:

kubectl -n demoapp-system logs deploy/demoapp-controller-manager -c manager -f
# "installed release"  "release":"demoapp-sample"  "namespace":"default"  "revision":1

Check the cluster - same shape as the pre-built operator:

kubectl get deploy,svc,cm,secret -l app.kubernetes.io/name=demoapp-sample
# deployment.apps/demoapp-sample      1/1     1            1           8s
# service/demoapp-sample              ClusterIP   10.96.12.177   <none>   80/TCP   8s
# configmap/demoapp-sample            1      8s
# secret/demoapp-sample               Opaque   1      8s

kubectl get secret -l owner=helm
# NAME                                   TYPE                 DATA   AGE
# sh.helm.release.v1.demoapp-sample.v1   helm.sh/release.v1   1      8s

The Helm release Secret is created by your operator's action.NewInstall.Run - bit-for-bit identical to what the pre-built operator produces. From the user's perspective (or helm history's perspective), there is no way to tell which operator deployed it.

Curl the workload:

kubectl port-forward svc/demoapp-sample 8080:80 &
curl -s localhost:8080
# <html><body><h1>Hello from the hybrid operator</h1></body></html>
kill %1

Step 23 - Patch CR, see your upgrade

kubectl patch demoapp demoapp-sample --type=merge \
  -p '{"spec":{"replicaCount":3,"message":"Updated by hybrid operator"}}'

kubectl -n demoapp-system logs deploy/demoapp-controller-manager -c manager --tail=5
# "upgraded release"  "release":"demoapp-sample"  "revision":2

kubectl get secret -l name=demoapp-sample,owner=helm
# NAME                                   TYPE                 DATA   AGE
# sh.helm.release.v1.demoapp-sample.v1   helm.sh/release.v1   1      29s
# sh.helm.release.v1.demoapp-sample.v2   helm.sh/release.v1   1      7s

kubectl get deploy demoapp-sample -o jsonpath='{.spec.replicas}'
# 3

Two release secrets (v1 install, v2 upgrade), Deployment replicas at 3, ConfigMap updated. The Reconcile path you wrote handled this end-to-end.

Common Helm v4 SDK gotchas (you will hit at least one)

These are the issues that produced opaque errors in our test runs while building this Helm hybrid operator. The fixes are already baked into the Step 14–18 code above; the table is for grep-finding when you read someone else's hybrid operator.

What you have now (Helm hybrid operator equivalent of the pre-built)

A from-scratch Go operator that:

• Installs the demo-app chart on CR create.
• Upgrades on CR update.
• Stores releases as Helm Secrets (compatible with helm history, helm rollback, helm get).
• Validates CR input via a strict CRD generated from Go types.
• Idempotent: re-reconciling the same generation is a no-op.

What you do not have yet:

• A finalizer (deleting the CR right now leaves the Helm release orphaned).
• Custom status fields (the framework's Initialized/Deployed/ReleaseFailed conditions are not even being written).
• Drift detection (edit the rendered ConfigMap and your operator will not notice).
• Anything beyond the pre-built operator's capability.

That last bullet is the punch line. At the end of Part 1 of the Helm hybrid operator tutorial, you have rebuilt the pre-built operator. The Helm release looks the same, the Kubernetes resources look the same, the chart contents are the same. The work was non-trivial but the payoff so far is zero - you wrote ~200 lines of Go to do what operator-sdk init --plugins=helm.sdk.operatorframework.io/v1 would have done in one command.

The whole point of Part 2 is to use those 200 lines as a platform to do things the pre-built cannot.

What's next - Part 2 of 2 (the actual goldmine)

Helm Hybrid Operator Tutorial Part 2 of 2 - Beyond what pre-built can do picks up exactly here and adds:

• Custom status fields: a LastUpgradeReason field that captures why the last upgrade happened (the CR changed? a periodic resync? a drift correction?). Impossible in the pre-built operator because the status struct is fixed.
• A real finalizer: writes an audit ConfigMap to a demoapp-audit namespace before helm uninstall runs. Demonstrates the hybrid's killer feature - custom pre-uninstall logic with retries against another namespace (or in real life, an external billing API or service-mesh deregistration call).
• Drift detection via Owns(): add .Owns(&corev1.ConfigMap{}) etc. to SetupWithManager, then edit a ConfigMap and watch your operator revert it. More flexible than the pre-built's watchDependentResources - you can choose which resource types to watch and even add custom predicates to skip resourceVersion-only diffs.
• Cross-CR coordination: a worked demo where DemoApp B waits for DemoApp A to be Deployed before installing. The kind of orchestration the pre-built cannot express because its reconciler treats CRs in isolation.
• Pointer patterns: reading external state in Reconcile (HTTP calls), conditional template rendering (enable Ingress only if nginx-ingress CRD exists), and multi-CR lifecycle (one CR owns N child CRs).

If you stop after Part 1, you have a working Helm hybrid operator with no real-world differentiator. If you read Part 2, you have one with several.

Further reading

• Helm-based operator tutorial Part 1 (pre-built path) — the no-Go equivalent of this Helm hybrid operator tutorial.
• Helm-based operator tutorial Part 2 (lifecycle, drift, hooks, ceiling) — the exact list of pre-built ceiling items Part 2 of this series addresses.
• Helm operator vs Flux vs Argo CD — when each tool is the right pick.
• Finalizers explained — language-neutral finalizer reference; required reading before Part 2.
• Status subresource and Conditions — standard conditions contract; what you will implement in Part 2.
• controller-runtime architecture — the Go library powering SetupWithManager, Owns(), and the cache.
• Server-Side Apply in operators — the SSA semantics behind Helm 4's ForceConflicts knob used in Steps 17 / 18.
• Operator capability / maturity model — where the hybrid pattern sits on the Level I → V ladder.
• External: Helm 4 SDK godoc — upstream reference for action, chart/v2, chart/v2/loader, chart/loader/archive, release/v1, kube.
• External: Operator SDK — Go operator docs — the official Go operator tutorial (not Helm-specific).
• External: helm-operator-plugins library — the operator-framework "hybrid" path this article deliberately bypasses.

Summary

A Helm hybrid operator is the pre-built helm-operator binary, written by you in Go - a single Reconcile that calls the Helm v4 SDK directly, with no helm-operator-plugins dependency. Part 1 of 2 walks the foundation end-to-end: scaffold with operator-sdk init --plugins=go/v4, define the CRD via Go API types and Kubebuilder markers (make generate for DeepCopy, make manifests for the CRD YAML), embed the chart with //go:embed all:charts/demo-app, wire the Helm v4 SDK (action.Configuration, loader.LoadFiles from embed.FS), and implement a Reconcile that does install on first CR and upgrade on subsequent changes.

At the end of this Helm hybrid operator Part 1 you have a from-scratch operator that is functionally indistinguishable from the pre-built helm-operator - same chart, same Helm release Secret, same Kubernetes resources, same helm history output. The justification for writing those ~200 lines of Go is everything in Part 2 of 2: custom status fields, a real finalizer with pre-uninstall work, drift via Owns(), and cross-CR coordination - the features the pre-built operator cannot provide at any setting, and the rungs on the operator capability ladder the hybrid pattern was invented to climb.