Owner References and Garbage Collection in Kubernetes Operators

The whole reason kubectl delete myapp myapp-sample cleanly removes the Deployment, the ReplicaSet, and three Pods is Kubernetes owner references. They are tiny pieces of metadata, easy to overlook, but they encode the entire owner-dependent contract that controller-runtime's Owns(), Kubernetes garbage collection, and finalizer-driven cleanup all rely on.

If you have already worked through the reconcile loop and Custom Resource Definitions, this is the next conceptual step in the Kubernetes operator pattern: the contract that decides what happens to a CR's dependents when the owner is deleted, the cluster restarts, or the operator crashes mid-reconcile.

We unpack that contract end-to-end: the difference between an owner and a controller, the three deletion propagation policies, the cross-namespace and cluster-scope rules, the /finalizers RBAC that makes blockOwnerDeletion actually work, and how to find orphans when the contract goes wrong.

What Are Owner References in Kubernetes?

Owner references are metadata links stored in a resource's metadata.ownerReferences field that establish a parent-child relationship between Kubernetes objects.

They tell Kubernetes:

• Which resource owns another resource.
• Which resources should be automatically deleted when the owner is removed.
• Which dependent resource changes should trigger reconciliation of the owner.
• Which resources belong to the same ownership chain.

For example, a standard Kubernetes Deployment creates a ReplicaSet, and that ReplicaSet creates Pods:

Deployment
└── ReplicaSet
    └── Pod

Each child resource stores a reference to its parent in metadata.ownerReferences. Because of these ownership links, deleting the Deployment automatically removes the ReplicaSet and Pods as well.

The same mechanism is heavily used by Kubernetes Operators. A Custom Resource (CR) typically owns the Deployments, Services, ConfigMaps, Secrets, Jobs, and StatefulSets that the operator creates:

MyApp (Custom Resource)
├── Deployment
├── Service
├── ConfigMap
└── Secret

When the CR is deleted, Kubernetes Garbage Collection (GC) follows those owner references and automatically removes the dependent resources.

Owner references are also what make controller-runtime's Owns() watches work. When a dependent resource changes, controller-runtime can trace the ownership chain back to the owning Custom Resource and enqueue it for reconciliation.

In short, owner references form the ownership contract between Kubernetes resources. They power cascade deletion, garbage collection, controller-runtime watches, and much of the automation that Operators rely on.

Owner References Explained in Plain Terms

You already use owner references every day in Kubernetes, even if you never look at the YAML.

Consider a Deployment:

Deployment
└── ReplicaSet
    └── Pod

The Deployment owns the ReplicaSet.

The ReplicaSet owns the Pod.

When you delete the Deployment, Kubernetes automatically removes the ReplicaSet and Pods.

That automatic cleanup happens because each child resource stores a pointer back to its owner in metadata.ownerReferences.

The Kubernetes garbage collector follows those ownership links and performs cascade deletion.

Anatomy of an owner reference

When controllerutil.SetControllerReference finishes, this is what it has written onto the dependent's metadata:

metadata:
  ownerReferences:
  - apiVersion: apps.example.com/v1alpha1
    kind: MyApp
    name: myapp-sample
    uid: a3c5e8f0-1234-5678-abcd-ef0123456789
    controller: true
    blockOwnerDeletion: true

Five fields, each with a job:

The uid is the field that makes orphan detection possible. If MyApp/myapp-sample is deleted and a fresh one is created with the same name, the new CR has a different uid — dependents that still reference the old uid are now orphans, and the new CR's Owns() watches will not enqueue for them.

SetControllerReference vs SetOwnerReference

In every reconciler that creates dependent resources, the line you will type ninety-nine times out of a hundred is controllerutil.SetControllerReference. The canonical pattern inside a multi-resource reconciler is one CreateOrUpdate per dependent, each one mutating its desired spec and re-asserting the owner reference:

deploy := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{
    Name: app.Name, Namespace: app.Namespace,
}}
op, err := controllerutil.CreateOrUpdate(ctx, r.Client, deploy, func() error {
    deploy.Spec = r.deploymentSpec(&app)
    return controllerutil.SetControllerReference(&app, deploy, r.Scheme)
})

That single SetControllerReference call:

1. Adds app to deploy.metadata.ownerReferences.
2. Sets controller: true and blockOwnerDeletion: true on that entry.
3. Enables Owns(&appsv1.Deployment{}) watches to enqueue Reconcile(app) when the Deployment changes.
4. Enables Kubernetes garbage collection to cascade-delete the Deployment when the CR is deleted.

If you forget that one line, three things break silently — and most operator authors discover it only when a production cluster ends up with hundreds of orphaned Deployments.

Controller-runtime exposes two helpers; they are not interchangeable:

controllerutil.SetControllerReference(owner, dependent, scheme)
controllerutil.SetOwnerReference(owner, dependent, scheme)

When to use which:

• SetControllerReference: the canonical owner-dependent link between your CR and a workload it creates. Use this for the primary dependent (the Deployment, StatefulSet, Job). It powers Owns() watches and cascade GC.
• SetOwnerReference: when an object has multiple owners but only one of them is the controller. Example: a ConfigMap shared by several CRs — set one CR as controller and the others as plain owners, so the ConfigMap gets cascade-deleted only when all owners are gone (with Foreground policy) or by the controller owner specifically.

A dependent can have many owners but only one controller owner. The API server enforces this — adding a second controller: true entry returns an error.

How deletion propagates

When kubectl delete <owner> runs, Kubernetes has to decide what happens to the dependents. The propagation policy makes that choice, and blockOwnerDeletion plus the cross-namespace rule together decide which dependents the policy applies to.

Background — the default

kubectl delete myapp myapp-sample
# explicit form:
kubectl delete myapp myapp-sample --cascade=background

Behaviour:

1. The owner is deleted from etcd immediately.
2. The garbage collector finds every dependent (ownerReferences[*].uid equals the deleted owner's uid) and asynchronously deletes them.
3. Dependents may still exist briefly after the owner is gone.

Pros: fast user response. Cons: a brief window where dependents exist without an owner — usually fine because nothing is watching them.

Foreground — synchronous cleanup

kubectl delete myapp myapp-sample --cascade=foreground

Behaviour:

1. The owner gets a deletionTimestamp and the special foregroundDeletion finalizer.
2. The garbage collector deletes every dependent whose owner reference has blockOwnerDeletion: true.
3. Once all blocking dependents are gone, the foregroundDeletion finalizer is removed, and the owner is finally deleted from etcd.

Pros: when the kubectl delete call returns, everything is gone. Useful in tests and in user-facing scripts that need a synchronous "all clear" signal.

Cons: slower; if a dependent's deletion hangs (its own finalizer is stuck), the owner gets stuck too.

Orphan — keep the dependents

kubectl delete myapp myapp-sample --cascade=orphan

Behaviour:

1. The owner is deleted.
2. The garbage collector removes the owner reference from each dependent but does not delete the dependents.
3. Each former dependent now has no owner — it survives until deleted by name.

Use case: emergency migration. You want to move dependents to a new owner without their workloads restarting. Delete with --cascade=orphan, recreate the CR with the same name, and manually re-attach owner references with kubectl patch.

The term orphan is part of the Kubernetes API (--cascade=orphan, DeletePropagationOrphan), so it shows up wherever the API surface does — including the next two sections, where we go hunting for them.

blockOwnerDeletion — what makes Foreground wait

blockOwnerDeletion: true is the field that turns Foreground deletion into a real precondition:

• The garbage collector treats each blocking dependent as a precondition on the owner's foregroundDeletion finalizer.
• The owner's foregroundDeletion finalizer stays in place until every blocking dependent has been deleted.

If blockOwnerDeletion: false on a dependent, that dependent does not block the owner's foreground deletion. The owner can be deleted even while that dependent still exists.

SetControllerReference sets blockOwnerDeletion: true by default because for the controller-owner relationship that is what you want: the owner's deletion shouldn't complete until I, the primary dependent, am gone. Plain SetOwnerReference does not set it because secondary owners don't typically block.

RBAC implication

To set blockOwnerDeletion: true on a dependent, the operator's ServiceAccount needs the update verb on the owner kind's finalizers subresource. This is a non-obvious permission, and it lives in the operator RBAC checklist:

- apiGroups: ["apps.example.com"]
  resources: ["myapps/finalizers"]
  verbs: ["update"]

The +kubebuilder:rbac:groups=apps.example.com,resources=myapps/finalizers,verbs=update marker generates it. Without that permission, SetControllerReference silently writes the reference without blockOwnerDeletion, which means Foreground deletion stops blocking on this dependent. Easy to miss; even easier to ship to production.

Setting the policy from Go

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

policy := metav1.DeletePropagationForeground
if err := r.Delete(ctx, &app, &client.DeleteOptions{
    PropagationPolicy: &policy,
}); err != nil {
    return err
}

Most operator code never explicitly sets a propagation policy — it deletes its own CRs and relies on background cleanup. The policy matters when you're writing test harnesses, custom tooling, or operator-on-operator chains.

The cross-namespace and cluster-scope rules

Kubernetes enforces a strict scoping rule on owner references:

A namespaced dependent must be in the same namespace as its namespaced owner.

If you try to create a Pod in namespace-a with an ownerReference pointing at a Deployment in namespace-b, the API server silently drops the reference. Not an error, not a warning — the reference is just gone, the dependent has no owner, and GC will never cascade-delete it.

What you can do:

Operators that need to manage resources across namespaces have two options:

1. Use a cluster-scoped CR. Then it can own resources in any namespace.
2. Use labels + finalizers instead of owner references. Tag each dependent with <group>/<cr-uid>: ..., list-and-cleanup on owner delete via finalizer. More work, but no scope restriction.

Finding orphans

A production cluster always accumulates some orphans:

• An operator was uninstalled before its CRs were deleted.
• A dependent resource was created with a stale uid pointing at a long-deleted CR.
• A cross-namespace owner reference was silently dropped.

To enumerate them:

# Every resource managed by this operator
kubectl get deployment,service,configmap,secret -A \
  -l app.kubernetes.io/managed-by=myapp-operator

# Group by owner uid to find orphans
kubectl get deploy -A -o json | jq '
  .items[] | select(.metadata.labels."app.kubernetes.io/managed-by" == "myapp-operator") |
  {name: .metadata.name, ns: .metadata.namespace,
   owners: [.metadata.ownerReferences[]? | {kind, name, uid}]}'

Cross-check each owners[].uid against the current CRs:

kubectl get myapp -A -o jsonpath='{range .items[*]}{.metadata.uid}{"\n"}{end}' > current-uids.txt

Any Deployment whose owner uid is not in current-uids.txt is an orphan. The cleanup tool is your label query:

kubectl delete deployment -A -l app.kubernetes.io/managed-by=myapp-operator \
  --field-selector=status.replicas=0   # be careful, narrow scope first

The same idea — label every dependent the operator creates with app.kubernetes.io/managed-by=<operator-name> plus the parent CR's name and uid — is also what lets a multi-resource reconciler detect drift on a per-dependent basis without trawling owner references in every reconcile.

When owner references are not enough

Owner references are great for in-Kubernetes dependents. They are useless for:

• External resources: cloud disks, DNS records, IAM roles, items in another cluster. GC cannot delete what it cannot see.
• Cross-namespace dependents: silently dropped owner reference, no GC.
• Things the controller wants to clean up itself: ordered shutdown sequences, drain operations, billing reconciliation.

For all of those, use a finalizer. Owner references and finalizers are complementary:

The general rule: owner references handle Kubernetes deletion; finalizers handle everything else. Most production operators have both, and they slot into the reconcile loop through the same deletionTimestamp check.

Verifying the ownership chain end-to-end

Once the operator is deployed and a sample CR is applied, trace the chain top-down to confirm every link is intact:

# Pod -> ReplicaSet
kubectl get pod -l app=myapp-sample -o jsonpath='{range .items[0].metadata.ownerReferences[*]}{.kind}/{.name}{"\n"}{end}'
# ReplicaSet/myapp-sample-7c9b-xyz9

# ReplicaSet -> Deployment
kubectl get rs myapp-sample-7c9b-xyz9 -o jsonpath='{range .metadata.ownerReferences[*]}{.kind}/{.name}{"\n"}{end}'
# Deployment/myapp-sample

# Deployment -> MyApp CR
kubectl get deploy myapp-sample -o jsonpath='{range .metadata.ownerReferences[*]}{.kind}/{.name}{"\n"}{end}'
# MyApp/myapp-sample

That chain is exactly what makes kubectl delete myapp myapp-sample cascade through three levels. If any link is missing, cascade stops there — leaving everything below as orphans.

Diagnosing a missing controller: true

Run:

kubectl get deploy myapp-sample -o yaml | grep -A6 ownerReferences

If controller: true is missing:

ownerReferences:
- apiVersion: apps.example.com/v1alpha1
  kind: MyApp
  name: myapp-sample
  uid: ...
  # controller: true        ← MISSING
  # blockOwnerDeletion: true ← MISSING

Two consequences:

• Owns(&appsv1.Deployment{}) watches in your operator do not enqueue reconciles on Deployment changes — they filter by controller: true.
• Foreground deletion does not wait for this Deployment.

Cause: you used SetOwnerReference instead of SetControllerReference. Fix: switch to SetControllerReference, or call SetOwnerReference first as a plain owner and then SetControllerReference to upgrade the entry to a controller reference.

Common pitfalls

1. Using SetOwnerReference instead of SetControllerReference

The Deployment is owned but not controlled. Owns(&Deployment{}) watches never fire. Easy to miss. Fix: use SetControllerReference for the primary dependent.

2. Missing the /finalizers RBAC

blockOwnerDeletion: true silently disappears, Foreground deletion doesn't block, and you only notice when test cleanup returns before child workloads have actually finished. Fix: add the +kubebuilder:rbac marker for the CR kind's finalizers subresource.

3. Cross-namespace owner reference

You wired the operator to create a Pod in another namespace; the reference was dropped; Pods become orphans on CR delete. Fix: keep dependents in the CR's namespace, use a cluster-scoped CRD, or manage cleanup via finalizer.

4. Recreating a CR with the same name

The dependents from the previous CR still carry the old uid. The new CR has a new uid. Owns() does not enqueue, and the new reconciler thinks no dependents exist. Fix: clean up orphans first, or design the operator to detect and adopt previously-owned resources by label.

5. Trusting cascade delete for external resources

Delete the CR, the Deployment is cascade-deleted — but the cloud disk the Deployment was using is orphaned. Fix: add a finalizer that unmounts and deletes the disk before letting the cascade run.

Summary

Owner references encode the contract every other part of the operator pattern relies on: Owns() watches enqueue the owner when a dependent changes, garbage collection cascade-deletes dependents when the owner is deleted, and blockOwnerDeletion makes Foreground deletion behave synchronously. The single most important line is controllerutil.SetControllerReference inside every CreateOrUpdate block — get that right and 90% of the machinery is automatic.

The other 10% — external resources, cross-namespace cleanup, ordered shutdown — needs finalizers. Used together, the two mechanisms cover every cleanup scenario a production operator will see, and they slot cleanly into the same level-triggered reconcile loop the rest of the operator already runs on.

Further reading

• Kubernetes finalizers, two-phase deletion, and stuck objects — the complement to GC for everything Kubernetes cannot delete on its own.
• Operator RBAC minimum permissions — the /finalizers subresource RBAC that makes blockOwnerDeletion work.
• Multi-resource reconciliation — the per-child builder pattern that wires SetControllerReference into every dependent the operator creates.
• Drift detection patterns in operators — what to do when a dependent has drifted from desired state but is still owned correctly.
• The reconcile loop explained — the level-triggered control loop that processes deletionTimestamp and runs the cascade in the first place.
• Custom Resource Definitions explained — the schema for the CR sitting at the root of the ownership tree.
• Controller-runtime architecture — where the Owns() watch, the cache, and the workqueue fit together.
• External: Kubernetes owner references docs, garbage collection reference, controllerutil package reference.