Helm Hybrid Operator Tutorial Part 2 of 2 - Custom Status, Finalizer, Drift, Cross-CR

In devops
Last reviewed: by
Helm Hybrid Operator Tutorial Part 2 of 2 - Custom Status, Finalizer, Drift, Cross-CR

This is Part 2 of the Helm hybrid operator tutorial. Part 1 ended with a from-scratch Go operator that installs and upgrades the demo-app chart on DemoApp CR events - functionally equivalent to the pre-built helm-operator binary. At the end of Part 1 you had rebuilt the pre-built; you had nothing new. Part 2 is what makes the ~200 lines of Go from Part 1 worth writing.

Each Part below is one ceiling item from the pre-built operator's hard ceiling list. Each adds 30 to 80 lines of Go. The result is a hybrid operator with several features the pre-built cannot express at any setting.

Recap from Part 1

You have:

  • demoapp/ project scaffolded with operator-sdk init --plugins=go/v4.
  • api/v1alpha1/demoapp_types.go with DemoAppSpec (replicaCount, image, message, apiKey, service) and a minimal DemoAppStatus (Conditions + ObservedGeneration).
  • internal/controller/chart.go embedding internal/controller/charts/demo-app via //go:embed all:charts/demo-app (path is package-relative — see Part 1, Step 9).
  • internal/controller/helm.go building action.Configuration per reconcile via the 3-arg cfg.Init(getter, namespace, "secret") (Helm v4 dropped the logger arg).
  • internal/controller/demoapp_controller.go with Reconcile doing fetch → idempotency check → install-or-upgrade → mark observed generation. Install/upgrade explicitly set WaitStrategy = kube.HookOnlyStrategy and ForceConflicts = true (Helm v4 SSA defaults; see Part 1 Steps 17 & 18).
  • Image pushed to ttl.sh/demoapp-hybrid-<uuid>:24h and deployed to a kind cluster, one DemoApp CR running.

Quick sanity check:

bash 
kubectl get demoapps -A
kubectl get secret -l owner=helm -A

Part G - Custom status fields (incl. fields the pre-built CAN'T have)

The pre-built operator writes three condition types (Initialized, Deployed, ReleaseFailed) and a deployedRelease block. You cannot add status.lastUpgradeReason or status.lastSyncedFromExternalSourceAt. The hybrid operator has no such limit - whatever you put on DemoAppStatus becomes part of the API.

Step 24 - Set the standard conditions (Initialized, Deployed, Failed)

We'll mirror the three conditions the pre-built operator writes, then add a fourth one the pre-built cannot. Add helpers in internal/controller/status.go:

go 
package controller

import (
	"context"
	"fmt"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/api/meta"

	demov1alpha1 "github.com/example/demoapp/api/v1alpha1"
)

const (
	ConditionInitialized = "Initialized"
	ConditionDeployed    = "Deployed"
	ConditionFailed      = "Failed"
)

func setInitialized(demoapp *demov1alpha1.DemoApp) {
	meta.SetStatusCondition(&demoapp.Status.Conditions, metav1.Condition{
		Type:               ConditionInitialized,
		Status:             metav1.ConditionTrue,
		Reason:             "Reconciling",
		Message:            "Operator has bound the CR to the embedded chart",
		ObservedGeneration: demoapp.Generation,
	})
}

func setDeployed(demoapp *demov1alpha1.DemoApp, releaseName string, revision int) {
	meta.SetStatusCondition(&demoapp.Status.Conditions, metav1.Condition{
		Type:               ConditionDeployed,
		Status:             metav1.ConditionTrue,
		Reason:             "ReleaseDeployed",
		Message:            fmt.Sprintf("Release %q revision %d", releaseName, revision),
		ObservedGeneration: demoapp.Generation,
	})
	// Clear any previous failure
	meta.RemoveStatusCondition(&demoapp.Status.Conditions, ConditionFailed)
}

func setFailed(demoapp *demov1alpha1.DemoApp, reason, message string) {
	meta.SetStatusCondition(&demoapp.Status.Conditions, metav1.Condition{
		Type:               ConditionFailed,
		Status:             metav1.ConditionTrue,
		Reason:             reason,
		Message:            message,
		ObservedGeneration: demoapp.Generation,
	})
	// Mark Deployed as False so consumers can quickly read "is this app healthy?"
	meta.SetStatusCondition(&demoapp.Status.Conditions, metav1.Condition{
		Type:               ConditionDeployed,
		Status:             metav1.ConditionFalse,
		Reason:             reason,
		Message:            message,
		ObservedGeneration: demoapp.Generation,
	})
}

// updateStatus writes the status subresource. Use only at end of Reconcile.
func (r *DemoAppReconciler) updateStatus(ctx context.Context, demoapp *demov1alpha1.DemoApp) error {
	return r.Status().Update(ctx, demoapp)
}

meta.SetStatusCondition is the canonical helper - it preserves the existing LastTransitionTime if the Status field hasn't actually changed, which is what users expect.

Step 25 - Populate DeployedRelease info

Add a struct on DemoAppStatus that mirrors what the pre-built operator writes. Edit api/v1alpha1/demoapp_types.go:

go 
type DemoAppStatus struct {
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// +optional
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`

	// DeployedRelease captures the current Helm release for this CR.
	// +optional
	DeployedRelease *DeployedReleaseInfo `json:"deployedRelease,omitempty"`

	// LastUpgradeReason records WHY the last upgrade happened.
	// This is impossible in the pre-built helm operator - the framework
	// status struct is fixed.
	// +optional
	LastUpgradeReason string `json:"lastUpgradeReason,omitempty"`

	// LastUpgradeAt is the time of the last successful upgrade.
	// +optional
	LastUpgradeAt *metav1.Time `json:"lastUpgradeAt,omitempty"`
}

type DeployedReleaseInfo struct {
	Name     string `json:"name"`
	Revision int    `json:"revision"`
}

Re-generate and re-apply the CRD:

bash 
make generate manifests
kubectl apply -f config/crd/bases/demo.example.com_demoapps.yaml

Step 26 - Set ObservedGeneration correctly

You already had ObservedGeneration from Part 1. The convention to follow strictly: set it on the CR only after the reconcile has succeeded end-to-end. Setting it earlier means a partial reconcile is marked as "fully observed" and the next reconcile will skip work that should re-run.

The Reconcile flow becomes:

go 
// Inside Reconcile, after install/upgrade succeeded:
setDeployed(&demoapp, release.Name, release.Version)
demoapp.Status.DeployedRelease = &demov1alpha1.DeployedReleaseInfo{
	Name:     release.Name,
	Revision: release.Version,
}
demoapp.Status.LastUpgradeAt = &metav1.Time{Time: time.Now()}
demoapp.Status.LastUpgradeReason = upgradeReason
demoapp.Status.ObservedGeneration = demoapp.Generation
if err := r.updateStatus(ctx, &demoapp); err != nil {
	return ctrl.Result{}, err
}

Step 27 - The custom field: LastUpgradeReason

LastUpgradeReason records why an upgrade ran. Possible values:

Value When set
SpecGenerationChanged CR's .spec was modified (.Generation advanced)
DriftDetected An owned resource changed (drift via Owns(), Part I)
PeriodicResync Reconcile fired from the periodic resync loop
CrossCRDependencyBecameReady A CR this one was waiting on (Part J) flipped to Deployed
RetryAfterFailure Previous reconcile failed; this is the retry

To populate it accurately, the Reconcile needs to know why it was invoked. Controller-runtime doesn't tell you directly, but we can infer:

go 
// Inside Reconcile, before install/upgrade:
upgradeReason := "SpecGenerationChanged"
if meta.IsStatusConditionTrue(demoapp.Status.Conditions, ConditionFailed) {
	upgradeReason = "RetryAfterFailure"
} else if demoapp.Status.ObservedGeneration == demoapp.Generation {
	// Same generation - this is either a periodic resync or drift.
	// (Distinguishing drift vs periodic requires inspecting the event source;
	// for simplicity we lump them together as "DriftOrResync".)
	upgradeReason = "DriftOrResync"
}

Demo - patch the CR and observe LastUpgradeReason populated:

bash 
kubectl patch demoapp demoapp-sample --type=merge \
  -p '{"spec":{"message":"trigger an upgrade"}}'

kubectl get demoapp demoapp-sample -o jsonpath='{.status.lastUpgradeReason}'
# SpecGenerationChanged

kubectl get demoapp demoapp-sample -o jsonpath='{.status.deployedRelease}'
# {"name":"demoapp-sample","revision":2}

kubectl get demoapp demoapp-sample -o jsonpath='{.status.lastUpgradeAt}'
# 2026-06-02T11:34:21Z

This is the moment Part 2's investment starts paying off. The pre-built operator's status has none of these fields. Users (and dashboards, and alerting rules) can now distinguish "the CR was edited" from "the periodic resync ran." That is operationally invaluable and impossible without Go.

Part H - Custom finalizer (the killer feature)

The pre-built operator has no place to insert custom logic before helm uninstall runs. A Helm pre-delete Job runs during uninstall, lacks retry semantics, and runs as in-cluster code with limited cluster access. A Go finalizer runs before anything else and has the full controller-runtime client.

Step 28 - Add a finalizer on first reconcile

Define the finalizer constant and add it on first reconcile:

go 
const demoAppFinalizer = "demo.example.com/audit-and-cleanup"

// Inside Reconcile, right after fetching the CR and BEFORE the install/upgrade branch:
if demoapp.DeletionTimestamp.IsZero() {
	// CR is not being deleted - ensure finalizer is present.
	if !controllerutil.ContainsFinalizer(&demoapp, demoAppFinalizer) {
		controllerutil.AddFinalizer(&demoapp, demoAppFinalizer)
		if err := r.Update(ctx, &demoapp); err != nil {
			return ctrl.Result{}, fmt.Errorf("add finalizer: %w", err)
		}
		// The Update triggers a new reconcile event; return cleanly.
		return ctrl.Result{}, nil
	}
}

After applying any CR, verify the finalizer is set:

bash 
kubectl get demoapp demoapp-sample -o jsonpath='{.metadata.finalizers}'
# ["demo.example.com/audit-and-cleanup"]

Step 29 - Detect deletion (DeletionTimestamp != nil)

Branch on deletion right after the finalizer-add check:

go 
if !demoapp.DeletionTimestamp.IsZero() {
	// CR is being deleted - run finalizer logic, then helm uninstall, then remove finalizer.
	if controllerutil.ContainsFinalizer(&demoapp, demoAppFinalizer) {
		if err := r.handleDeletion(ctx, &demoapp); err != nil {
			return ctrl.Result{RequeueAfter: 15 * time.Second}, err
		}

		controllerutil.RemoveFinalizer(&demoapp, demoAppFinalizer)
		if err := r.Update(ctx, &demoapp); err != nil {
			return ctrl.Result{}, fmt.Errorf("remove finalizer: %w", err)
		}
	}
	return ctrl.Result{}, nil
}

While the finalizer is present, the CR shows DeletionTimestamp but stays around. Only after RemoveFinalizer + Update does Kubernetes complete the deletion.

Step 30 - Custom pre-uninstall: write an audit ConfigMap in another namespace

This is the concrete demo. Before helm uninstall runs, we write a ConfigMap to demoapp-audit recording the CR's last spec, the timestamp of deletion, and the deployed release info. Add internal/controller/audit.go:

go 
package controller

import (
	"context"
	"encoding/json"
	"fmt"
	"time"

	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/types"

	demov1alpha1 "github.com/example/demoapp/api/v1alpha1"
)

const auditNamespace = "demoapp-audit"

// writeAuditRecord writes a ConfigMap to the audit namespace BEFORE helm uninstall runs.
// This is the canonical "custom finalizer work" demo - a Helm pre-delete hook cannot
// safely write to another namespace, retry on partial failure, or include controller-side
// state (DeployedRelease info) the chart doesn't know about.
func (r *DemoAppReconciler) writeAuditRecord(ctx context.Context, demoapp *demov1alpha1.DemoApp) error {
	if err := r.ensureAuditNamespace(ctx); err != nil {
		return fmt.Errorf("ensure audit namespace: %w", err)
	}

	specJSON, _ := json.MarshalIndent(demoapp.Spec, "", "  ")
	statusJSON, _ := json.MarshalIndent(demoapp.Status, "", "  ")

	cmName := fmt.Sprintf("%s-%s-%d",
		demoapp.Namespace, demoapp.Name, time.Now().Unix())

	cm := &corev1.ConfigMap{
		ObjectMeta: metav1.ObjectMeta{
			Name:      cmName,
			Namespace: auditNamespace,
			Labels: map[string]string{
				"demo.example.com/source-namespace": demoapp.Namespace,
				"demo.example.com/source-name":      demoapp.Name,
			},
		},
		Data: map[string]string{
			"deleted-at": time.Now().UTC().Format(time.RFC3339),
			"spec":       string(specJSON),
			"status":     string(statusJSON),
		},
	}

	// Idempotent create: if the audit CM already exists (a previous reconcile attempt
	// succeeded before we removed the finalizer), treat it as success.
	if err := r.Create(ctx, cm); err != nil && !apierrors.IsAlreadyExists(err) {
		return fmt.Errorf("create audit configmap: %w", err)
	}
	return nil
}

func (r *DemoAppReconciler) ensureAuditNamespace(ctx context.Context) error {
	ns := &corev1.Namespace{}
	err := r.Get(ctx, types.NamespacedName{Name: auditNamespace}, ns)
	if err == nil {
		return nil
	}
	if !apierrors.IsNotFound(err) {
		return err
	}
	ns = &corev1.Namespace{
		ObjectMeta: metav1.ObjectMeta{Name: auditNamespace},
	}
	if err := r.Create(ctx, ns); err != nil && !apierrors.IsAlreadyExists(err) {
		return err
	}
	return nil
}

The operator also needs RBAC to write Namespaces and cross-namespace ConfigMaps. Drift correction later in this article also watches and patches Deployments, so include the apps RBAC marker now:

go 
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=secrets;configmaps;services;namespaces,verbs=get;list;watch;create;update;patch;delete

Re-run make manifests && make deploy IMG=$IMG to refresh the ClusterRole.

Three things this does that a Helm pre-delete hook cannot easily do:

  1. Writes to another namespace. The Helm release is in default; the audit lives in demoapp-audit. Helm hooks can technically do this with cluster-wide RBAC, but the configuration is finicky and the hook Job would need its own ServiceAccount + ClusterRoleBinding.
  2. Includes controller-side state. The audit captures status.DeployedRelease and status.LastUpgradeReason - state the chart doesn't even know about.
  3. Has retry semantics for free. If r.Create fails (transient API error), handleDeletion returns an error, the Reconcile returns RequeueAfter: 15s, and the finalizer is not removed. The next reconcile retries. A Helm pre-delete Job that fails has to be manually cleaned up; the chart is in a broken state.

Step 31 - Run action.NewUninstall.Run()

After the audit succeeds, uninstall the Helm release:

go 
import "helm.sh/helm/v4/pkg/kube"

func (r *DemoAppReconciler) handleDeletion(ctx context.Context, demoapp *demov1alpha1.DemoApp) error {
	logger := log.FromContext(ctx)

	// Step 1: custom pre-uninstall work.
	if err := r.writeAuditRecord(ctx, demoapp); err != nil {
		return err
	}
	logger.Info("audit record written", "name", demoapp.Name)

	// Step 2: helm uninstall the chart release.
	cfg, err := newActionConfig(ctx, demoapp.Namespace)
	if err != nil {
		return fmt.Errorf("helm config for uninstall: %w", err)
	}

	uninst := action.NewUninstall(cfg)
	uninst.KeepHistory = false               // delete the release Secrets too
	uninst.WaitStrategy = kube.HookOnlyStrategy // v4: same WaitStrategy requirement as install/upgrade

	_, err = uninst.Run(demoapp.Name)
	if err != nil && err != driver.ErrReleaseNotFound {
		return fmt.Errorf("helm uninstall: %w", err)
	}
	logger.Info("helm release uninstalled", "name", demoapp.Name)
	return nil
}

v4 gotcha: like Install and Upgrade, action.Uninstall also has the WaitStrategy kube.WaitStrategy field in v4. Forgetting it doesn't produce a hard error on Uninstall the way it does on Upgrade (Uninstall tolerates the unset case), but setting it explicitly keeps the four call sites symmetric.

Step 32 - Remove the finalizer

Already shown in Step 29 - after handleDeletion returns nil, RemoveFinalizer + Update complete the deletion.

Why this is the killer feature (vs Helm pre-delete hook)

Property Helm pre-delete hook Hybrid Go finalizer (this Part)
Runs before Helm uninstall Yes Yes
Has retry semantics No (Job runs once) Yes (RequeueAfter in Reconcile)
Can block deletion indefinitely No Yes (finalizer stays until removed)
Can call external APIs with credentials Hard (Job RBAC + Secrets) Trivial (HTTP client in reconciler)
Access to controller-side state None Full (status, conditions, derived data)
Cross-namespace writes Awkward (ClusterRole) Trivial (controller-runtime client)
Cleanup if the hook itself fails Manual Automatic (idempotent retry)

Demo - delete the CR and watch the audit appear before the Helm uninstall:

bash 
kubectl delete demoapp demoapp-sample &
sleep 5

# While the CR is still terminating, the audit is already written:
kubectl -n demoapp-audit get cm
# NAME                                DATA   AGE
# default-demoapp-sample-1780489214   3      4s
# kube-root-ca.crt                    1      4s

kubectl -n demoapp-audit get cm default-demoapp-sample-1780489214 -o yaml
# data:
#   deleted-at: "2026-06-03T12:20:14Z"
#   spec: |
#     {
#       "replicaCount": 3,
#       "image": "nginx:1.27-alpine",
#       "message": "Updated by hybrid operator",
#       ...
#     }
#   status: |
#     {
#       "conditions": [...],
#       "deployedRelease": {"name":"demoapp-sample","revision":2},
#       "lastUpgradeReason": "SpecGenerationChanged",
#       ...
#     }

wait
# demoapp "demoapp-sample" deleted from default namespace

kubectl get all,cm,secret -l app.kubernetes.io/name=demoapp-sample
# No resources found in default namespace.
kubectl get secret -l owner=helm
# No resources found in default namespace.

The audit ConfigMap remains in demoapp-audit after everything else is gone — proof your finalizer ran. The kube-root-ca.crt ConfigMap in the listing is auto-created by Kubernetes when the audit namespace was first provisioned; it is unrelated to our finalizer.

Part I - Drift detection via Owns()

The pre-built operator uses watchDependentResources: true to subscribe to events on all chart-rendered resource types. The hybrid uses Owns() in SetupWithManager, which gives per-type control and predicate filtering.

Two Helm v4 traps before you start this Part. Both bit us in end-to-end testing:

  1. ForceConflicts = true is required on Install and Upgrade (in helm.go/demoapp_controller.go). Without it, the first kubectl patch against a chart-rendered field assigns ownership of that field to kubectl-patch (Server-Side Apply); the operator's subsequent helm upgrade then fails with conflict occurred while applying object … Apply failed with 1 conflict: conflict with "kubectl-patch" using v1: .data.index.html. The drift demo below depends on this. Add it back in Step 17/18 of Part 1 if you haven't already.
  2. Gate the upgrade path on a real "did anything change?" check — otherwise enabling Owns() creates a reconcile loop. The flow is: Reconcile installs/upgrades → operator patches owner refs on chart resources → those Updates fire Owns() events → Reconcile fires again → upgrade re-renders manifests → SSA marks managedFields as helm → those Updates fire Owns() events again → loop. In our test runs the loop saturated at ~10 helm revisions per second until something else interrupted it. A minimal gate is shown below.

Add this guard near the top of your Reconcile, before calling installOrUpgrade:

go 
// Compute upgrade reason BEFORE running install/upgrade so we can decide whether to skip.
upgradeReason := "SpecGenerationChanged"
if meta.IsStatusConditionTrue(demoapp.Status.Conditions, ConditionFailed) {
	upgradeReason = "RetryAfterFailure"
} else if demoapp.Status.ObservedGeneration == demoapp.Generation &&
	demoapp.Status.DeployedRelease != nil {
	upgradeReason = "DriftOrResync"
}

// Skip the upgrade roundtrip when generation has already been observed AND
// the trigger is not a chart-resource change (Owns event).
// In production, replace this with a checksum-based comparison of the rendered manifest
// vs. the live cluster state to make drift detection precise.
if demoapp.Status.ObservedGeneration == demoapp.Generation &&
	demoapp.Status.DeployedRelease != nil && upgradeReason != "DriftOrResync" {
	return ctrl.Result{}, nil
}

A more robust (and more code-heavy) variant is to compute sha256(rel.Manifest) of the rendered chart after each successful upgrade, store it in status.appliedManifestHash, and on subsequent reconciles only call installOrUpgrade when either (a) generation changed, or (b) the live cluster manifest differs from the stored hash. The pre-built operator does the second; for most hybrid operators the simple gate above is enough as long as you accept that drift correction adds 2-3 helm revisions per detected drift event.

Step 33 - Add Owns() for chart-rendered resource types

Edit SetupWithManager in demoapp_controller.go:

go 
import (
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
)

func (r *DemoAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&demov1alpha1.DemoApp{}).
		Owns(&appsv1.Deployment{}).
		Owns(&corev1.Service{}).
		Owns(&corev1.ConfigMap{}).
		Owns(&corev1.Secret{}).
		Complete(r)
}

For Owns() to work, the chart-rendered resources must have an owner reference pointing back to the DemoApp CR. Helm does not automatically set this - the chart renders flat manifests. You have two options:

Option A (recommended) - Add helm.sh/resource-policy: keep to nothing; instead use Helm's own labels and add owner refs after install:

At this point, change installOrUpgrade, install, and upgrade from Part 1 to return the Helm release object as well as the error:

go 
func (r *DemoAppReconciler) installOrUpgrade(
	ctx context.Context, cfg *action.Configuration, demoapp *demov1alpha1.DemoApp,
) (*release.Release, error)

Use the concrete Helm v4 release type imported as release "helm.sh/helm/v4/pkg/release/v1". Status updates and the owner-reference helper both need release.Name, release.Version, and release.Manifest; returning only error is no longer enough.

After each install/upgrade, iterate the rendered manifests and set the owner reference:

go 
// After helm install/upgrade succeeded:
if err := r.setOwnerRefsOnChartResources(ctx, &demoapp, release); err != nil {
	logger.Error(err, "set owner refs; drift detection may be partial")
	// Non-fatal - the chart still works without drift detection.
}

A helper to walk the rendered manifest and patch owner refs:

go 
import (
	release "helm.sh/helm/v4/pkg/release/v1"
	yaml "sigs.k8s.io/yaml"
)

func (r *DemoAppReconciler) setOwnerRefsOnChartResources(
	ctx context.Context, demoapp *demov1alpha1.DemoApp, rel *release.Release,
) error {
	ownerRef := metav1.OwnerReference{
		APIVersion: demov1alpha1.GroupVersion.String(),
		Kind:       "DemoApp",
		Name:       demoapp.Name,
		UID:        demoapp.UID,
		Controller: ptr.To(true),
	}

	// rel.Manifest is the rendered YAML of all chart resources joined by "---".
	docs := strings.Split(rel.Manifest, "\n---\n")
	for _, doc := range docs {
		doc = strings.TrimSpace(doc)
		if doc == "" {
			continue
		}
		obj := &unstructured.Unstructured{}
		if err := yaml.Unmarshal([]byte(doc), obj); err != nil {
			continue
		}
		if obj.GetName() == "" {
			continue
		}
		if obj.GetNamespace() == "" {
			obj.SetNamespace(demoapp.Namespace)
		}

		// Fetch the live object, patch owner ref, update.
		live := obj.DeepCopy()
		key := client.ObjectKey{Namespace: obj.GetNamespace(), Name: obj.GetName()}
		if err := r.Get(ctx, key, live); err != nil {
			if apierrors.IsNotFound(err) {
				continue
			}
			return err
		}
		existing := live.GetOwnerReferences()
		if hasOwnerRef(existing, ownerRef) {
			continue
		}
		live.SetOwnerReferences(append(existing, ownerRef))
		if err := r.Update(ctx, live); err != nil {
			return fmt.Errorf("patch owner ref on %s/%s: %w",
				obj.GetKind(), obj.GetName(), err)
		}
	}
	return nil
}

func hasOwnerRef(refs []metav1.OwnerReference, want metav1.OwnerReference) bool {
	for _, r := range refs {
		if r.UID == want.UID {
			return true
		}
	}
	return false
}

Option B - Use Helm post-render to inject owner refs at chart-render time:

Helm supports a --post-renderer flag (and the SDK equivalent via action.Install.PostRenderer). A post-renderer is a binary or Go function that receives the rendered manifests and rewrites them before apply. You can inject owner refs there and Helm's normal helm upgrade will preserve them. More elegant for production; more setup overhead.

For this article we use Option A.

Step 34 - Demo: edit a ConfigMap, watch your operator revert

After deploying the operator with Owns() in place:

bash 
kubectl apply -f config/samples/demo_v1alpha1_demoapp.yaml
sleep 5

# Verify owner ref is set on the ConfigMap:
kubectl get cm demoapp-sample -o jsonpath='{.metadata.ownerReferences[0]}'
# {"apiVersion":"demo.example.com/v1alpha1","kind":"DemoApp","name":"demoapp-sample","uid":"...","controller":true}

# Now manually edit the ConfigMap (simulating drift):
kubectl patch cm demoapp-sample --type=merge \
  -p '{"data":{"index.html":"<html><body><h1>HACKED</h1></body></html>"}}'

sleep 3

# The operator notices via Owns() and reverts:
kubectl -n demoapp-system logs deploy/demoapp-controller-manager -c manager --tail=5
# "upgraded release"  "release":"demoapp-sample"  "revision":3

kubectl get cm demoapp-sample -o jsonpath='{.data.index\.html}'
# <html><body><h1>Hello from the hybrid operator</h1></body></html>

# And LastUpgradeReason captures the cause:
kubectl get demoapp demoapp-sample -o jsonpath='{.status.lastUpgradeReason}'
# DriftOrResync

The operator log does not include a literal "drift detected" line because the reconcile path doesn't distinguish drift events from CR-change events at log time — the distinction is captured only in status.lastUpgradeReason, computed from ObservedGeneration vs Generation plus DeployedRelease-set. If you want a "drift detected" log entry, add it in your reconcile right before the installOrUpgrade call when upgradeReason == "DriftOrResync".

Step 35 - How this compares to the pre-built's watchDependentResources

Capability Pre-built watchDependentResources Hybrid Owns()
Watch chart-rendered resources Yes (all types) Yes (one Owns() per type)
Skip certain resource types No Yes (omit the Owns() line)
Skip events with predicate (e.g., resourceVersion-only) No Yes (.WithPredicates(...))
Watch non-chart resources (e.g., Ingress added externally) No Yes (add Owns() or Watches())
Configure per-resource-type filter No Yes (per Owns() predicate)

Bonus: smarter drift (skip resourceVersion-only diffs)

Trivial change-only events (the API server bumps resourceVersion even when nothing meaningful changed) waste reconcile cycles. Filter them with a predicate:

go 
import "sigs.k8s.io/controller-runtime/pkg/predicate"

func (r *DemoAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	noisePred := predicate.Funcs{
		UpdateFunc: func(e event.UpdateEvent) bool {
			return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration() ||
				!reflect.DeepEqual(e.ObjectOld.GetAnnotations(), e.ObjectNew.GetAnnotations())
		},
	}

	return ctrl.NewControllerManagedBy(mgr).
		For(&demov1alpha1.DemoApp{}).
		Owns(&appsv1.Deployment{}, builder.WithPredicates(noisePred)).
		Owns(&corev1.Service{}, builder.WithPredicates(noisePred)).
		Owns(&corev1.ConfigMap{}).
		Owns(&corev1.Secret{}).
		Complete(r)
}

The pre-built operator has no equivalent - you get all events or none.

Part J - Custom features beyond Helm (one rich demo + pointers)

Rich demo: cross-CR coordination

DemoApp B should wait for DemoApp A to be Deployed before installing. This pattern is required whenever you have ordered dependencies (database before app, API before frontend, etc.).

Add WaitFor to the spec:

go 
// In api/v1alpha1/demoapp_types.go, on DemoAppSpec:

// WaitFor is the name of another DemoApp in the same namespace that
// must be in condition Deployed=True before this CR is installed.
// +optional
WaitFor string `json:"waitFor,omitempty"`

Regenerate:

bash 
make generate manifests
kubectl apply -f config/crd/bases/demo.example.com_demoapps.yaml

In Reconcile, gate the install on the dependency:

go 
// Inside Reconcile, BEFORE installOrUpgrade:
if demoapp.Spec.WaitFor != "" {
	ready, err := r.dependencyReady(ctx, demoapp.Namespace, demoapp.Spec.WaitFor)
	if err != nil {
		return ctrl.Result{}, fmt.Errorf("check dependency: %w", err)
	}
	if !ready {
		logger.Info("waiting for dependency", "waitFor", demoapp.Spec.WaitFor)
		meta.SetStatusCondition(&demoapp.Status.Conditions, metav1.Condition{
			Type:    "DependencyReady",
			Status:  metav1.ConditionFalse,
			Reason:  "Waiting",
			Message: fmt.Sprintf("Waiting for DemoApp %q to be Deployed", demoapp.Spec.WaitFor),
			ObservedGeneration: demoapp.Generation,
		})
		if err := r.updateStatus(ctx, &demoapp); err != nil {
			return ctrl.Result{}, err
		}
		// Requeue with backoff. In production, also Watch() the dependent CR
		// so reconcile fires immediately when it flips Deployed=True.
		return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
	}
	meta.SetStatusCondition(&demoapp.Status.Conditions, metav1.Condition{
		Type:    "DependencyReady",
		Status:  metav1.ConditionTrue,
		Reason:  "DependencyDeployed",
		Message: fmt.Sprintf("DemoApp %q is Deployed", demoapp.Spec.WaitFor),
		ObservedGeneration: demoapp.Generation,
	})
}

The dependency-ready check is straightforward - fetch the other CR and inspect its conditions:

go 
func (r *DemoAppReconciler) dependencyReady(ctx context.Context, namespace, name string) (bool, error) {
	var dep demov1alpha1.DemoApp
	err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, &dep)
	if err != nil {
		if apierrors.IsNotFound(err) {
			// Dependency doesn't exist yet - treat as not ready, no error.
			return false, nil
		}
		return false, err
	}
	return meta.IsStatusConditionTrue(dep.Status.Conditions, ConditionDeployed), nil
}

Demo (apply both CRs in reverse order so app-b is reconciled first and can demonstrate the wait):

bash 
cat <<EOF | kubectl apply -f -
apiVersion: demo.example.com/v1alpha1
kind: DemoApp
metadata:
  name: app-b
spec:
  message: "I am app B - waits for A"
  apiKey: "verysecret"
  waitFor: app-a
---
apiVersion: demo.example.com/v1alpha1
kind: DemoApp
metadata:
  name: app-a
spec:
  message: "I am app A - the dependency"
  apiKey: "verysecret"
EOF

# Within ~8 seconds, app-a is Deployed and the Watches() handler enqueues app-b:
sleep 8

kubectl get demoapp app-a -o jsonpath='{.status.deployedRelease}'
# {"name":"app-a","revision":3}

kubectl get demoapp app-b -o jsonpath='{.status.conditions[?(@.type=="DependencyReady")]}'
# {"type":"DependencyReady","status":"True","reason":"DependencyDeployed",
#  "message":"DemoApp \"app-a\" is Deployed", ...}

kubectl get demoapp app-b -o jsonpath='{.status.deployedRelease}'
# {"name":"app-b","revision":4}

kubectl get demoapp
# NAME    REPLICAS   MESSAGE                       AGE
# app-a   1          I am app A - the dependency   15s
# app-b   1          I am app B - waits for A      15s

If you apply app-a first then app-b, app-b will skip the Waiting state entirely because by the time its first reconcile runs, app-a is already Deployed=True. To force-observe the wait, you can either (a) apply only app-b first and watch its DependencyReady=False condition before creating app-a, or (b) set app-a's waitFor to a non-existent CR temporarily.

To make this snappier, add a Watches() on DemoApp itself with a handler that enqueues all CRs depending on the one that just changed:

go 
.Watches(
	&demov1alpha1.DemoApp{},
	handler.EnqueueRequestsFromMapFunc(r.findCRsWaitingOn),
)

func (r *DemoAppReconciler) findCRsWaitingOn(ctx context.Context, obj client.Object) []reconcile.Request {
	changed, ok := obj.(*demov1alpha1.DemoApp)
	if !ok {
		return nil
	}
	var list demov1alpha1.DemoAppList
	if err := r.List(ctx, &list, client.InNamespace(changed.Namespace)); err != nil {
		return nil
	}
	var reqs []reconcile.Request
	for _, item := range list.Items {
		if item.Spec.WaitFor == changed.Name {
			reqs = append(reqs, reconcile.Request{
				NamespacedName: types.NamespacedName{
					Namespace: item.Namespace,
					Name:      item.Name,
				},
			})
		}
	}
	return reqs
}

With this, app-b reconciles within milliseconds of app-a flipping Deployed, not on the 30s RequeueAfter cadence.

Pointer patterns (one paragraph each)

Reading external state in Reconcile. Inject an HTTP client (or any other external SDK) on the reconciler struct:

go 
type DemoAppReconciler struct {
	client.Client
	Scheme     *runtime.Scheme
	Chart      *chart.Chart
	HTTPClient *http.Client
}

Use it in Reconcile to call an external service before installing:

go 
resp, err := r.HTTPClient.Get(fmt.Sprintf("https://license.internal/check?app=%s", demoapp.Name))
if err != nil || resp.StatusCode != 200 {
	// External check failed; surface as a condition and back off.
	return ctrl.Result{RequeueAfter: 2 * time.Minute}, nil
}

The pre-built operator has no place to put this call.

Conditional template rendering (enable Ingress only if cluster supports it). Check at startup whether the cluster has a given CRD (e.g., IngressClass for nginx-ingress) by querying the discovery client, then inject the result as an overrideValues equivalent before passing to action.Install:

go 
hasIngress := r.discoverIngressSupport()  // call once at startup, cache
values["ingressEnabled"] = hasIngress

The chart's template uses {{- if .Values.ingressEnabled }} to conditionally render the Ingress. The pre-built operator cannot inject cluster-discovered values into the chart.

Custom validation beyond OpenAPI. OpenAPI handles syntactic validation (field types, regex, enum). Semantic validation - "this CR's image must exist in our registry," "this CR's port must not conflict with another CR's port" - requires Go. Add a check at the top of Reconcile, set a Validated condition, and gate installOrUpgrade on it:

go 
if err := r.validateBusinessRules(ctx, &demoapp); err != nil {
	setFailed(&demoapp, "ValidationFailed", err.Error())
	return ctrl.Result{RequeueAfter: 5 * time.Minute}, r.updateStatus(ctx, &demoapp)
}

The pre-built operator has no equivalent extension point.

Multi-CR lifecycle (one CR owns N child CRs). A DemoAppCluster CR could own N DemoApp CRs - the cluster reconciler creates one DemoApp per node and aggregates their status into a cluster-level status. The cluster reconciler is just a second controller in the same operator binary; both use the same Owns()/Watches() patterns. No additional infrastructure required.

End-to-end re-deploy and verify

Rebuild and redeploy with all of Part 2's changes. Generate a fresh ttl.sh URL for this iteration:

bash 
export IMG=ttl.sh/demoapp-hybrid-$(uuidgen):24h
make generate manifests          # regenerate DeepCopy + CRD + RBAC after every status/spec/marker change
go mod vendor                    # if your Dockerfile uses vendored deps (recommended behind corp proxies)
make docker-build IMG="$IMG"
docker push "$IMG"

kubectl apply -f config/crd/bases/demo.example.com_demoapps.yaml   # apply new CRD before pod uses new types
make deploy IMG="$IMG"

# Force a pod restart so the new image is picked up even when the kustomize image tag didn't change for k8s:
kubectl -n demoapp-system rollout restart deploy/demoapp-controller-manager
kubectl -n demoapp-system rollout status  deploy/demoapp-controller-manager --timeout=180s

Apply, patch, delete, observe everything end-to-end:

bash 
# Install:
kubectl apply -f config/samples/demo_v1alpha1_demoapp.yaml
sleep 10
kubectl get demoapp demoapp-sample -o jsonpath='{.status.deployedRelease}'
# {"name":"demoapp-sample","revision":1}

# Upgrade:
kubectl patch demoapp demoapp-sample --type=merge -p '{"spec":{"replicaCount":2}}'
sleep 5
kubectl get demoapp demoapp-sample -o jsonpath='{.status.lastUpgradeReason}'
# SpecGenerationChanged

# Drift correction:
kubectl patch cm demoapp-sample --type=merge -p '{"data":{"index.html":"<h1>HACKED</h1>"}}'
sleep 3
kubectl get cm demoapp-sample -o jsonpath='{.data.index\.html}'
# (reverted by operator)
kubectl get demoapp demoapp-sample -o jsonpath='{.status.lastUpgradeReason}'
# DriftOrResync

# Delete with finalizer + audit:
kubectl delete demoapp demoapp-sample
sleep 5
kubectl -n demoapp-audit get cm
# default-demoapp-sample-... (the audit record)
kubectl get all,cm,secret -l app.kubernetes.io/name=demoapp-sample
# No resources found (Helm uninstall completed)

Every one of the four behaviours above (rich status, drift correction with reason, pre-uninstall audit, dependency gating) is a capability the pre-built operator cannot provide.

Comparison: what you built vs the pre-built helm-operator

Property Pre-built helm-operator Hybrid (Parts 1+2)
Lines of Go you wrote 0 ~400
CRD source of truth YAML you edit Go types + Kubebuilder markers
Schema strictness Permissive default Strict default
Install / upgrade / uninstall Yes (free) Yes (you wrote it)
Custom status fields Impossible Anything you put on DemoAppStatus
LastUpgradeReason-style insight Impossible Yes (Part G)
Real finalizer (with retries, cross-NS writes) Impossible Yes (Part H)
Pre-uninstall audit in another namespace Awkward via hook + RBAC Trivial (Part H)
Drift detection Yes (watchDependentResources) Yes, with per-type predicates (Part I)
Cross-CR coordination Impossible Yes (Part J)
Reading external state in reconcile Impossible Yes (Part J pointer)
Conditional template render on cluster state Impossible Yes (Part J pointer)
Custom decision logic ("only upgrade in window") Impossible Yes (a few lines of Go in Reconcile)
Image maintenance burden Operator SDK does it for you You maintain

The right column dominates anywhere the operator's job is non-trivial. The left column dominates when "install this chart" is the whole job.

When to use which

Situation Pick
You have a chart and only need "deploy this on every CR" Pre-built helm-operator
You need drift detection on a chart and nothing else Pre-built helm-operator
You need any custom status field Hybrid
You need a real finalizer (external system, audit, license release) Hybrid
You need to gate install/upgrade on external state Hybrid
You need cross-CR coordination Hybrid
Two or more of the above Hybrid - the ~400 LoC pays off immediately
Brownfield migration from plain helm install to a Helm operator Either; see the brownfield article (upcoming)

Further reading

Summary

Part 2 of the Helm hybrid operator tutorial is the payoff for Part 1's investment. With about 200 additional lines of Go on top of the foundation, you gain features the pre-built helm-operator binary cannot provide at any setting: custom status fields like LastUpgradeReason that capture why an upgrade happened (Part G); a real finalizer that writes an audit ConfigMap to another namespace before helm uninstall runs, with retries and cross-namespace access (Part H); drift detection via Owns() that is per-type filterable and supports predicates to skip noisy updates (Part I); and a worked cross-CR coordination demo where one DemoApp waits for another to be Deployed before installing, plus pointer patterns for reading external state, conditional template rendering, and multi-CR lifecycle (Part J). The hybrid operator now has the right column of the pre-built operator's hard ceiling checked off - every one of those capability gaps closed by code you can read, modify, and own.

Deepak Prasad

R&D Engineer

Founder of GoLinuxCloud with over a decade of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels across development, DevOps, …

Certifications
  • Red Hat Certified System Administrator in Red Hat OpenStack
  • Certified Kubernetes Application Developer (CKAD)
  • Red Hat Certified Specialist in Ansible Automation
Expertise
  • Go (programming language)
  • Python (programming language)
  • DevOps
  • Computer Security

Related Articles

Kubernetes Operator Tutorial - Build Production-Ready Operators

Kubernetes Operator Tutorial - Build Production-Ready Operators
Install Operator-SDK on Linux: Complete Setup Guide (Go, kubectl, Helm 4, kind, ttl.sh registry)

Install Operator-SDK on Linux: Complete Setup Guide (Go, kubectl, Helm 4, kind, ttl.sh registry)
Kubernetes Operator Capability Levels I–V: The Operator Maturity Model

Kubernetes Operator Capability Levels I–V: The Operator Maturity Model
Kubernetes Operator Design Patterns: Singleton, Capability, Lifecycle, Auto-Pilot

Kubernetes Operator Design Patterns: Singleton, Capability, Lifecycle, Auto-Pilot
Kubernetes Status Subresource and Conditions Explained (KEP-1623)

Kubernetes Status Subresource and Conditions Explained (KEP-1623)
Kubernetes Finalizers: Two-Phase Deletion, Cleanup Patterns, and Stuck Objects

Kubernetes Finalizers: Two-Phase Deletion, Cleanup Patterns, and Stuck Objects
Kubernetes Operator Watches, Events, and Predicates Explained

Kubernetes Operator Watches, Events, and Predicates Explained
Controller-Runtime Architecture: Manager, Cache, Informer, Workqueue Explained

Controller-Runtime Architecture: Manager, Cache, Informer, Workqueue Explained