Multi-Resource Reconciliation: Managing N Child Resources per CR

A real operator rarely manages one Deployment. The Memcached tutorial starts with one Deployment and one Service, but production operators routinely manage a ServiceAccount, a Role and RoleBinding, two ConfigMaps, a Secret, a HorizontalPodAutoscaler, a NetworkPolicy, a PodDisruptionBudget, and sometimes more than one Deployment for sidecars or workers. By the time you write the fifth child resource, the reconciler has either become a sprawling mess or you have settled on a pattern.

The pattern is straightforward and has been refined across the controller-runtime ecosystem since 2018. This guide walks through it: one desired-state builder per child, a fixed reconcile order, controllerutil.CreateOrUpdate for idempotent writes, label-based enumeration for orphan cleanup, and a per-child status aggregator.

This article assumes you have already read the reconcile loop explained, owner references and garbage collection, Status subresource and Conditions explained, and ideally Server-Side Apply (SSA) in operators — each one sets up a primitive this guide composes.

TL;DR — multi-resource reconciliation in 60 seconds

For an operator that manages N child resource types per CR:

1. Write one builder function per child type. Each builder returns a desired-state object with name, namespace, labels, and the owner reference set. No API calls inside builders — pure functions.
2. In Reconcile, iterate the builders in dependency order. ServiceAccount, then ConfigMap, then Secret, then Deployment, then Service, then HPA — whatever your topology requires.
3. Apply each child with controllerutil.CreateOrUpdate (or Server-Side Apply for modern operators). The helper reads the current state, calls your mutate function, writes back if anything changed, and returns whether it created or updated.
4. Garbage-collect orphans by label selector. After applying the desired set, list every child matching managed-by=<operator>,instance=<cr-name> and delete anything not in the desired set.
5. Aggregate per-child status into the CR's conditions. Walk the children, summarise their readiness, and surface a Ready Condition reflecting the worst child.
6. Subscribe to child events via Owns() in SetupWithManager. Without it, drift correction does not work — the reconciler never wakes up when a Pod crashes or a child resource is edited out of band.

Six steps, ten to twenty lines per child, and the operator scales linearly with the number of child types.

A quick analogy: a general contractor on a building site

A general contractor (GC) does not lay bricks. The GC coordinates eight subcontractors — electrician, plumber, framer, roofer, drywaller, painter, HVAC, finisher — and the building gets built. The shape of the GC's job has nothing to do with the building itself; it is about coordination.

Three things every good GC does:

1. Has a written scope per subcontractor. The electrician knows exactly what they are responsible for. The painter knows the exact set of rooms. Nobody is guessing. (One desired-state builder per child.)
2. Calls subcontractors in the right order. Framing before drywall, drywall before paint, paint before finisher. Out-of-order work means re-work. (Dependency-aware reconcile order.)
3. Walks the site at the end of each day. Anything not in the plans gets fixed or removed. Forgotten cables, leftover pallets, old fixtures from the previous tenant — all gone. (Orphan garbage collection by label.)

If a subcontractor does not show up, the GC reschedules and keeps working on the others — the day is not lost. If two subcontractors fight over the same wall, the GC arbitrates. The GC's clipboard tracks the state of every trade in real time.

The analogy holds at every level. When the CR is deleted, the building is demolished and every subcontractor's work is removed — exactly how owner references cascade. When the CR is updated to drop a sidecar, the GC tells the electrician "this room no longer needs wiring; please rip it out" — exactly what label-based orphan cleanup does.

Why multi-resource reconciliation matters

Three reasons it is worth getting this pattern right rather than ad-hoc-ing each child write:

1. Real operators almost always own multiple children

The simplest Memcached tutorial — one Deployment, one Service — is a teaching example, not a production shape. Production operators routinely manage a ServiceAccount, two ConfigMaps, a Secret, a Deployment, a Service, an HPA, a NetworkPolicy, and a PodMonitor per CR. Once you cross the third child type, an ad-hoc reconcile loop becomes nine if err := blocks deep and impossible to extend without breaking the others.

2. Drift correction depends on the whole set being reconciled

The Kubernetes operator pattern is level-triggered — every reconcile must converge the entire desired set, not just the field that changed. That means each child write must be idempotent, the iteration must be dependency-ordered but not depend on order for correctness, and the reconciler must subscribe to events on every child via Owns(). The pattern below is just the disciplined way to make all three properties hold at once.

3. The pattern is uniform across every operator you will ever ship

controllerutil.CreateOrUpdate (or SSA), SetControllerReference, label-based orphan cleanup, status aggregation — the same six bullet points from the TL;DR apply whether your CR owns two children or twenty. The shape is so predictable that Operator design patterns like the Capability and Lifecycle patterns are largely re-expressions of "which children does the CR own, and how many of them".

Step 1 — One builder per child type

A builder is a pure function from CR + context → desired-state object. No API calls. No I/O. No reading from the cache. Just compute.

func deploymentFor(mc *cachev1alpha1.Memcached) *appsv1.Deployment {
    labels := commonLabels(mc)
    return &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      mc.Name,
            Namespace: mc.Namespace,
            Labels:    labels,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &mc.Spec.Size,
            Selector: &metav1.LabelSelector{MatchLabels: labels},
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{Labels: labels},
                Spec: corev1.PodSpec{
                    ServiceAccountName: mc.Name,
                    Containers: []corev1.Container{{
                        Name:  "memcached",
                        Image: mc.Spec.Image,
                        Ports: []corev1.ContainerPort{{ContainerPort: 11211}},
                    }},
                },
            },
        },
    }
}

func serviceFor(mc *cachev1alpha1.Memcached) *corev1.Service {
    labels := commonLabels(mc)
    return &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      mc.Name,
            Namespace: mc.Namespace,
            Labels:    labels,
        },
        Spec: corev1.ServiceSpec{
            Selector: labels,
            Ports:    []corev1.ServicePort{{Port: 11211}},
        },
    }
}

func serviceAccountFor(mc *cachev1alpha1.Memcached) *corev1.ServiceAccount {
    return &corev1.ServiceAccount{
        ObjectMeta: metav1.ObjectMeta{
            Name:      mc.Name,
            Namespace: mc.Namespace,
            Labels:    commonLabels(mc),
        },
    }
}

func commonLabels(mc *cachev1alpha1.Memcached) map[string]string {
    return map[string]string{
        "app.kubernetes.io/name":       "memcached",
        "app.kubernetes.io/instance":   mc.Name,
        "app.kubernetes.io/managed-by": "memcached-operator",
    }
}

Notice: every child carries the same commonLabels. This is what makes orphan cleanup possible later — the operator can list everything it owns by selector.

For children like the ServiceAccount above, the operator's own RBAC must include create / get / update / delete on that kind in the right scope — see least-privilege Operator RBAC for how to declare the minimum verbs your reconciler needs to manage each child type.

Step 2 — A reusable apply helper (controllerutil.CreateOrUpdate)

This is the workhorse — the shape kubebuilder and Operator SDK scaffold for every reconciler. It wraps controllerutil.CreateOrUpdate, sets the controller reference, and reports the operation result for logs and metrics.

func (r *MemcachedReconciler) apply(
    ctx context.Context,
    mc *cachev1alpha1.Memcached,
    obj client.Object,
) (controllerutil.OperationResult, error) {
    // Snapshot the desired spec before CreateOrUpdate mutates obj.
    desired := obj.DeepCopyObject().(client.Object)

    op, err := controllerutil.CreateOrUpdate(ctx, r.Client, obj, func() error {
        // Copy desired spec fields into the live object, but preserve
        // server-set fields (resourceVersion, status).
        copySpec(desired, obj)
        return controllerutil.SetControllerReference(mc, obj, r.Scheme)
    })
    return op, err
}

// copySpec copies the spec from desired to live, preserving live's
// server-managed metadata.
func copySpec(desired, live client.Object) {
    switch d := desired.(type) {
    case *appsv1.Deployment:
        l := live.(*appsv1.Deployment)
        l.ObjectMeta.Labels = d.ObjectMeta.Labels
        l.Spec = d.Spec
    case *corev1.Service:
        l := live.(*corev1.Service)
        l.ObjectMeta.Labels = d.ObjectMeta.Labels
        // Preserve cluster-assigned ClusterIP; only copy the parts you control.
        l.Spec.Selector = d.Spec.Selector
        l.Spec.Ports = d.Spec.Ports
    // ... one case per child kind ...
    }
}

Two non-obvious bits:

• SetControllerReference is called inside the mutate function. This guarantees the owner reference is set before the write goes out, even on first creation.
• Type-specific copySpec. You cannot blindly copy obj.Spec because for some types (Service's clusterIP, Deployment's replicas if managed by HPA), the server-side value must be preserved. Be deliberate about which fields you own.

Server-Side Apply as the alternative

Modern operators replace CreateOrUpdate + copySpec with Server-Side Apply, which expresses ownership at the field level and removes the entire copySpec switch statement:

func (r *MemcachedReconciler) applySSA(
    ctx context.Context,
    mc *cachev1alpha1.Memcached,
    obj client.Object,
) error {
    if err := controllerutil.SetControllerReference(mc, obj, r.Scheme); err != nil {
        return err
    }
    return r.Patch(ctx, obj, client.Apply,
        client.FieldOwner("memcached-operator"),
        client.ForceOwnership)
}

No read-modify-write loop, no per-kind case statement preserving clusterIP / resourceVersion / replicas. SSA tracks which controller owns which field, so when HPA writes Deployment.spec.replicas and your operator writes Deployment.spec.template, both writes coexist cleanly. See Server-Side Apply (SSA) in operators for the full conflict-resolution rules and migration playbook.

Step 3 — Reconcile in dependency order

A Kubernetes operator must reconcile in dependency order — a child that references another (Deployment → ServiceAccount, Deployment → ConfigMap) must be created after its dependency exists, otherwise the first apply fails and you rely on the next reconcile to catch up. The reconciler itself is now a flat sequence:

func (r *MemcachedReconciler) reconcileChildren(
    ctx context.Context, mc *cachev1alpha1.Memcached,
) error {
    log := log.FromContext(ctx)

    // 1. ServiceAccount (referenced by Deployment.spec.template.spec.serviceAccountName)
    sa := serviceAccountFor(mc)
    if op, err := r.apply(ctx, mc, sa); err != nil {
        return fmt.Errorf("serviceaccount: %w", err)
    } else if op != controllerutil.OperationResultNone {
        log.Info("serviceaccount reconciled", "op", op)
    }

    // 2. ConfigMap (referenced by Deployment.spec.template.spec.containers[*].envFrom)
    cm := configMapFor(mc)
    if op, err := r.apply(ctx, mc, cm); err != nil {
        return fmt.Errorf("configmap: %w", err)
    } else if op != controllerutil.OperationResultNone {
        log.Info("configmap reconciled", "op", op)
    }

    // 3. Deployment (depends on SA + ConfigMap)
    dep := deploymentFor(mc)
    if op, err := r.apply(ctx, mc, dep); err != nil {
        return fmt.Errorf("deployment: %w", err)
    } else if op != controllerutil.OperationResultNone {
        log.Info("deployment reconciled", "op", op)
    }

    // 4. Service (independent of others, but conventionally created after the workload)
    svc := serviceFor(mc)
    if op, err := r.apply(ctx, mc, svc); err != nil {
        return fmt.Errorf("service: %w", err)
    } else if op != controllerutil.OperationResultNone {
        log.Info("service reconciled", "op", op)
    }

    return nil
}

Three properties of this code:

1. Each step returns its error wrapped with the child type. When something fails, the log says exactly which child broke.
2. The order is dependency-aware but not strictly enforced — if the ConfigMap creation fails on the first reconcile, the Deployment creation will probably also fail (or succeed and pick up the ConfigMap on the next pass). Idempotency means out-of-order success is fine.
3. No if op == None { skip } short-circuits. Every child is touched on every reconcile, which is what makes drift correction work — the moment a child resource is manually edited, the next reconcile compares live vs desired and rewrites the diverged fields.

How the reconcile loop wakes up when a child changes — Owns()

Steps 1-3 cover what to do when reconcile runs. The remaining question is who fires reconcile when a Pod inside a managed Deployment crashes, or when an admin runs kubectl edit on one of your ConfigMaps. The answer is Owns() in SetupWithManager:

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1alpha1.Memcached{}).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Owns(&corev1.ConfigMap{}).
        Owns(&corev1.ServiceAccount{}).
        Complete(r)
}

For every child kind your operator creates with controllerutil.SetControllerReference, register a corresponding Owns(). controller-runtime walks the owner reference on each child event and enqueues the owning CR's key into the workqueue — your Reconcile() then runs against the full desired set and rewrites whatever drifted. Without Owns(), the reconciler only fires on CR .spec changes and the 10-hour resync; drift correction effectively does not work.

Owns is just sugar for Watches with the EnqueueRequestForOwner handler pre-wired. For the predicate menu (GenerationChangedPredicate, ResourceVersionChangedPredicate, etc.) and the full decision matrix for Owns vs Watches, see Watches, events, and predicates.

Step 4 — Orphan garbage collection

Owner references handle the case when the CR is deleted — Kubernetes' garbage collector cascades. They do not handle the case when the CR is updated to remove a child. If a user removes a sidecar from spec.sidecars, the desired set shrinks; the orphaned child Deployment must be deleted.

The pattern is "list, diff, delete":

func (r *MemcachedReconciler) cleanupOrphans(
    ctx context.Context, mc *cachev1alpha1.Memcached,
    desired []client.Object,
) error {
    // Build the set of (kind, name) tuples we currently want.
    wanted := make(map[string]struct{})
    for _, d := range desired {
        wanted[kindName(d)] = struct{}{}
    }

    // List every child of this CR by label selector.
    sel := client.MatchingLabels(commonLabels(mc))
    ns := client.InNamespace(mc.Namespace)

    var dList appsv1.DeploymentList
    if err := r.List(ctx, &dList, ns, sel); err != nil {
        return err
    }
    for _, d := range dList.Items {
        if _, ok := wanted[kindName(&d)]; !ok {
            if err := r.Delete(ctx, &d); err != nil && !apierrors.IsNotFound(err) {
                return err
            }
        }
    }

    // Repeat for Services, ConfigMaps, etc.
    return nil
}

func kindName(obj client.Object) string {
    return fmt.Sprintf("%T/%s", obj, obj.GetName())
}

For a small fixed set of child types (one Deployment, one Service, one ConfigMap per CR), orphan GC is unnecessary — the names are deterministic from the CR's name and won't drift. For variable child sets (a list of sidecars, dynamic ConfigMaps), it is essential.

A simpler alternative if you have ≤ 20 children per CR: tag every child with controller-revision: <CR generation> and delete anything where the revision is stale. This avoids the explicit desired-set bookkeeping at the cost of one label per child.

Step 5 — Aggregating per-child status

Each child has its own readiness signal. The CR's Ready condition reflects the worst of them.

func (r *MemcachedReconciler) aggregateReady(
    ctx context.Context, mc *cachev1alpha1.Memcached,
) (metav1.ConditionStatus, string, string) {

    var dep appsv1.Deployment
    if err := r.Get(ctx, client.ObjectKey{Name: mc.Name, Namespace: mc.Namespace}, &dep); err != nil {
        return metav1.ConditionUnknown, "DeploymentMissing", err.Error()
    }
    if dep.Status.ReadyReplicas < *dep.Spec.Replicas {
        return metav1.ConditionFalse, "DeploymentNotReady",
            fmt.Sprintf("ready replicas %d/%d",
                dep.Status.ReadyReplicas, *dep.Spec.Replicas)
    }

    var svc corev1.Service
    if err := r.Get(ctx, client.ObjectKey{Name: mc.Name, Namespace: mc.Namespace}, &svc); err != nil {
        return metav1.ConditionUnknown, "ServiceMissing", err.Error()
    }
    if svc.Spec.ClusterIP == "" {
        return metav1.ConditionFalse, "ServiceNotAllocated",
            "cluster IP not yet assigned"
    }

    return metav1.ConditionTrue, "AllChildrenReady",
        "deployment ready and service available"
}

Then in the reconciler:

status, reason, msg := r.aggregateReady(ctx, mc)
meta.SetStatusCondition(&mc.Status.Conditions, metav1.Condition{
    Type:               "Ready",
    Status:             status,
    Reason:             reason,
    Message:            msg,
    ObservedGeneration: mc.Generation,
})

This is "Ready by worst child" — the failure mode is unambiguous. Two details that catch first-time authors:

• Do not set LastTransitionTime yourself. meta.SetStatusCondition bumps it only when Status actually flips; manually setting metav1.Now() defeats that and triggers a status hot loop on every reconcile.
• Always set ObservedGeneration. Without it, kubectl wait --for=condition=Ready cannot tell whether the condition reflects the latest spec or a stale spec from before the user's last edit.

See Status subresource and Conditions explained for the full Ready / Progressing / Available / Degraded contract and the hot-loop pitfalls.

A worked example: a CR that manages eight children

Putting it all together for a more realistic operator that creates ServiceAccount, Role, RoleBinding, ConfigMap, Secret, Deployment, Service, and PodMonitor per CR:

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var mc cachev1alpha1.Memcached
    if err := r.Get(ctx, req.NamespacedName, &mc); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Build the full desired set.
    desired := []client.Object{
        serviceAccountFor(&mc),
        roleFor(&mc),
        roleBindingFor(&mc),
        configMapFor(&mc),
        secretFor(&mc),
        deploymentFor(&mc),
        serviceFor(&mc),
        podMonitorFor(&mc),
    }

    // Apply in declaration order.
    for _, child := range desired {
        if _, err := r.apply(ctx, &mc, child); err != nil {
            return ctrl.Result{}, fmt.Errorf("%T: %w", child, err)
        }
    }

    // Clean up anything that should not be here.
    if err := r.cleanupOrphans(ctx, &mc, desired); err != nil {
        return ctrl.Result{}, err
    }

    // Roll up status.
    original := mc.DeepCopy()
    status, reason, msg := r.aggregateReady(ctx, &mc)
    meta.SetStatusCondition(&mc.Status.Conditions, metav1.Condition{
        Type:               "Ready",
        Status:             status,
        Reason:             reason,
        Message:            msg,
        ObservedGeneration: mc.Generation,
    })
    if !equality.Semantic.DeepEqual(original.Status, mc.Status) {
        if err := r.Status().Update(ctx, &mc); err != nil {
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

This shape scales linearly. To add a ninth child type, add a builder function, add one line to the desired slice, optionally add a Get to aggregateReady. Nothing else changes.

Common pitfalls

Pitfall 1 — Building children inside CreateOrUpdate's mutate function.

// WRONG
controllerutil.CreateOrUpdate(ctx, r, obj, func() error {
    desired := deploymentFor(mc)  // built fresh every retry
    obj = desired                  // pointer assignment does not propagate
    return nil
})

The mutate function is called on every retry (conflict, etc.). Rebuilding desired state inside it is wasteful and the obj = desired reassignment does not actually mutate the caller's pointer. Build once before CreateOrUpdate, then in mutate fn copy field-by-field.

Pitfall 2 — Forgetting SetControllerReference on first creation.

If you skip the owner reference, deleting the CR will leave orphan children behind forever. SetControllerReference must be inside the mutate function so it runs on every Create (and is a no-op on Update because the reference is already set).

Pitfall 3 — Setting controller reference inconsistently.

SetControllerReference sets controller: true, blockOwnerDeletion: true. Only one owner can be marked controller: true per object. If two CRs try to control the same child, the second SetControllerReference call returns an error. Always own children whose names are unique to the parent CR.

Pitfall 4 — Listing all children with no namespace filter.

// WRONG: lists every Deployment in the cluster
r.List(ctx, &dList, sel)

// RIGHT: scoped to the CR's namespace
r.List(ctx, &dList, sel, client.InNamespace(mc.Namespace))

Without the namespace filter, the operator pays for an O(cluster) list on every reconcile and can mistakenly delete children of other CRs.

Pitfall 5 — Treating op == None as a signal to skip status update.

If no child changed, you might be tempted to skip the status write. But the cluster may have changed (a child crashed, a Pod went unready) even if your spec did not. Always re-aggregate status; only skip the Status().Update call if the new status equals the old one.

Pitfall 6 — Catching errors per-child and continuing.

// WRONG
for _, c := range children {
    if _, err := r.apply(ctx, mc, c); err != nil {
        log.Error(err, "child failed, continuing")  // bad
    }
}

A partial apply leaves the cluster half-configured and the operator's status is unable to express the reality. Return on the first error; let the workqueue requeue with backoff; the next reconcile will retry every child idempotently.

Pitfall 7 — Hard-coding the namespace in builder functions.

Builders should take the namespace from the CR, not from a global. This is what makes the same operator usable in both cluster-scoped and namespace-scoped deployments. See Operator multi-tenancy patterns for the bigger picture on cluster-scoped vs namespace-scoped operators.

Pitfall cheat sheet

Summary

Multi-resource reconciliation looks intimidating until you see the pattern. It is the same primitive — one CreateOrUpdate (or SSA Patch) per child, in dependency order, with owner references and labels for cleanup — repeated for as many child types as your CR manages.

The general-contractor analogy is the whole mental model. The reconciler is the GC; the builders are the subcontractors; the dependency order is the schedule; the label-based cleanup is the end-of-day walk-through. Whether you have two trades or twelve, the shape of the GC's job is identical.

If you internalise five things from this guide:

1. One pure builder per child type, no I/O inside builders.
2. controllerutil.CreateOrUpdate (or Server-Side Apply) for every write, with SetControllerReference inside the mutate fn.
3. Reconcile in dependency order but design for idempotent retry — order is for the happy path.
4. Owns() every child kind in SetupWithManager so child events wake the reconciler — without it, drift correction is broken.
5. Label-based orphan cleanup after the apply pass; status aggregation as the last step, gated by equality.Semantic.DeepEqual to avoid the hot loop.

That is enough to scale from one child to twenty without the reconciler turning into a sprawling mess.

What's next?

Once multi-resource reconciliation clicks, the rest of the operator-runtime toolbox falls into place around it:

• Modern child writes — Server-Side Apply (SSA) in operators shows how SSA collapses the copySpec boilerplate and resolves the multi-writer fights that controllerutil.CreateOrUpdate cannot.
• Owner references in depth — Owner references and garbage collection covers controller: true vs blockOwnerDeletion, foreground vs background propagation, and the cases where SetControllerReference is not enough.
• Subscribing to child events — Watches, events, and predicates expands on Owns vs Watches and the predicate menu (GenerationChangedPredicate, LabelSelectorPredicate, etc.).
• Status & Conditions — Status subresource and Conditions explained covers the Ready / Progressing / Available / Degraded contract your aggregated status should publish, plus kubectl wait --for=condition=... integration.
• Drift correction & resync — Drift detection patterns in operators explains how Owns() events and periodic resync cooperate to keep the desired-vs-actual gap closed.
• Operator-level concerns — least-privilege RBAC for the verbs your reconciler needs on each child kind, and Operator multi-tenancy patterns for cluster-scoped vs namespace-scoped deployments.

External references:

• controller-runtime: controllerutil.CreateOrUpdate · SetControllerReference
• Kubernetes: Owners and Dependents

Looking for the bigger picture? The Kubernetes Operator tutorial sequences every article in this series in pedagogical order — this article opens the Reconciliation Patterns chapter, between the controller-runtime internals and the production-operations chapters.