Kubernetes Operator Watches, Events, and Predicates Explained

Watches, events, and predicates are the three pieces that decide which changes wake your Operator's Reconcile() function. Get them right and your operator reacts in milliseconds to every meaningful change while ignoring noise. Get them wrong and you ship one of the two classic Operator failure modes — a hot reconcile loop that pegs CPU, or a silent operator that misses cleanups and drift.

This guide covers every piece end-to-end: the four event types, the difference between Owns and Watches, the catalogue of built-in predicates, how to write a custom predicate.Funcs, and the pitfalls that cost real production incidents.

If you have not yet read the controller-runtime architecture, that article puts these three pieces in their wider context (Manager, Cache, Workqueue). This one zooms in on the filtering part.

TL;DR — watches, events & predicates in 60 seconds

Every controller in your Operator is wired to its workqueue through three collaborators:

1. Source - a source.Kind(mgr.GetCache(), &SomeKind{}) that reads watch deltas from the shared cache.
2. EventHandler - turns one Create / Update / Delete / Generic event into zero or more Request{namespace/name} items.
3. Predicate - a four-function filter applied before the handler runs; return false and the event is dropped on the floor.

You build the wiring with three Builder verbs and one optional filter:

ctrl.NewControllerManagedBy(mgr).
    For(&v1alpha1.MyKind{}).                          // primary watch
    Owns(&corev1.Pod{}).                              // child via ownerRef
    Watches(                                          // arbitrary input
        &corev1.ConfigMap{},
        handler.EnqueueRequestsFromMapFunc(r.mapCMtoMyKind),
    ).
    WithEventFilter(predicate.GenerationChangedPredicate{}).
    Complete(r)

This is exactly the block you see inside every controller's SetupWithManager() — the function kubebuilder and Operator SDK scaffold to register watches against the Manager.

Every change in the cluster follows the same path:

Watch → Event → Predicate → Handler → Workqueue → Reconcile()

Predicates decide whether an event is interesting. Handlers decide which reconcile keys should be queued. The workqueue then schedules the actual reconciliation.

A quick analogy: your email inbox with rules

Picture your work email inbox before and after you set up filters.

• The mailbox itself is your watch. Every email lands in it. You cannot stop the flow.
• Each email arriving is an event — there are four flavours: a new email, a reply on an existing thread, a deleted email, and a "calendar ping" your phone synthesised because someone declined your meeting.
• The filter rules are your predicates: "if the subject contains 'PR review' AND it is from the team, mark it important. Otherwise leave it alone."
• The action is your handler: archive, star, forward, ping you on Slack.

A bad filter (predicate) wakes you for every newsletter (hot loop). A missing filter rule for "missed delete" lets a deleted invitation linger in your calendar (silent operator). The skill in writing a controller is the same skill as writing email filters: react to the changes that matter, ignore the noise, and make sure the delete and unsubscribe events get just as much attention as the create events.

The rest of this article maps that picture onto the controller-runtime API.

Why watches, events, and predicates matter

Three reasons it is worth getting this layer right rather than copy-pasting the default kubebuilder scaffold and hoping for the best:

1. They decide every reconcile your operator ever runs

A controller never reconciles on its own — something has to put a key in the workqueue. That "something" is always a watch + event + predicate + handler chain. Get the wiring wrong and one of two failure modes ships: either you miss legitimate changes (silent operator, broken drift correction) or you fire on every status write you make (hot reconcile loop, etcd write rate climbs, CPU pegs). Both are common production incidents and both live in the 30 lines of SetupWithManager.

2. They are the only place to enforce O(1) cluster-wide cost

A naive operator on a 5,000-Pod cluster wakes up for every Pod modification — millions of reconciles per day. The two levers that make that cost manageable both live in this layer: cache-layer filtering (cache.Options.ByObject with namespace allow-lists or label selectors) keeps the informer from ever loading those Pods, and field indexers (mgr.GetFieldIndexer().IndexField(...)) make cross-reference lookups O(1) instead of O(N). Both are invisible to your business logic and both are what differentiate a production operator from a toy one.

3. They power the multi-resource reconcile pattern

Owns() is what wakes the parent CR when a child Pod crashes or a child ConfigMap is hand-edited. Without it, the operator only reconciles on CR .spec changes and the 10-hour resync — and the entire premise of level-triggered drift correction collapses. The patterns in this article are the contract every multi-child operator depends on.

The four event types

controller-runtime delivers exactly four event types to your predicates and handlers, defined in sigs.k8s.io/controller-runtime/pkg/event. They map one-to-one onto the incremental change-notification feed the Kubernetes API server itself emits — see the API concepts documentation for the lower-level WatchEvent shape.

The shape of predicate.Funcs mirrors this exactly:

type Funcs struct {
    CreateFunc  func(event.CreateEvent) bool
    UpdateFunc  func(event.UpdateEvent) bool
    DeleteFunc  func(event.DeleteEvent) bool
    GenericFunc func(event.GenericEvent) bool
}

Trap to remember. When a Funcs field is left nil, the predicate framework returns true by default in current versions, but historically that was not always the case and the rule has bitten people across upgrades. Make the behaviour explicit - always set all four fields. Future you will thank present you.

For, Owns, Watches - three verbs, three queues

The Builder DSL has exactly three watch verbs, each backed by a different event-handler default:

For(&v1.MyKind{}) - the primary watch

There is always exactly one For per controller. It registers a watch on your custom resource, attaches the default handler.EnqueueRequestForObject, and the reconcile key becomes Request{ObjectMeta.namespace, ObjectMeta.name}.

Owns(&corev1.Pod{}) - child resource via owner-reference

Owns is sugar for "watch Pods, and for every Pod event, walk metadata.ownerReferences looking for an owner of MyKind; if you find one, enqueue its key". The default handler is handler.EnqueueRequestForOwner:

handler.EnqueueRequestForOwner(
    mgr.GetScheme(),
    mgr.GetRESTMapper(),
    &v1.MyKind{},
    handler.OnlyControllerOwner(),
)

For Owns to actually work, you must set the owner reference on the child at creation time:

if err := controllerutil.SetControllerReference(&parent, &pod, r.Scheme); err != nil {
    return ctrl.Result{}, err
}

Forgetting this is one of the most common operator bugs. The reconciler creates a child, the child is never linked back to the parent, and when the child changes the parent never wakes up. Symptom: drift correction is broken; root cause: SetControllerReference was missed. See owner references and garbage collection for controller: true vs blockOwnerDeletion and the cascade-delete semantics that follow from Owns. For the full multi-child reconcile pattern that Owns enables, see multi-resource reconciliation.

Watches(obj, handler) - the general escape hatch

Watches lets you map any resource's events to any set of reconcile keys. The classic use case is a ConfigMap referenced by name in your custom resource's .spec:

.Watches(
    &corev1.ConfigMap{},
    handler.EnqueueRequestsFromMapFunc(r.findMyKindsByConfigMap),
)

func (r *MyReconciler) findMyKindsByConfigMap(ctx context.Context, obj client.Object) []reconcile.Request {
    cm, ok := obj.(*corev1.ConfigMap)
    if !ok {
        return nil
    }

    var list v1alpha1.MyKindList
    if err := r.List(ctx, &list, client.InNamespace(cm.Namespace),
        client.MatchingFields{"spec.configMapRef": cm.Name},
    ); err != nil {
        return nil
    }

    reqs := make([]reconcile.Request, 0, len(list.Items))
    for _, mk := range list.Items {
        reqs = append(reqs, reconcile.Request{NamespacedName: client.ObjectKey{
            Namespace: mk.Namespace, Name: mk.Name,
        }})
    }
    return reqs
}

Field indexers — making the MapFunc lookup O(1)

The client.MatchingFields{"spec.configMapRef": cm.Name} lookup above needs the controller-runtime cache to index spec.configMapRef — without that, the List becomes a full scan of every cached CR on every ConfigMap event. The fix is a one-shot field indexer registration in SetupWithManager:

mgr.GetFieldIndexer().IndexField(ctx, &v1alpha1.MyKind{}, "spec.configMapRef",
    func(rawObj client.Object) []string {
        mk := rawObj.(*v1alpha1.MyKind)
        return []string{mk.Spec.ConfigMapRef}
    })

Two rules of thumb for controller-runtime field indexers:

• Register an indexer for every field you MatchingFields-query. Without it, the Watches MapFunc above does an O(N) walk on every ConfigMap update — and your operator quickly becomes the slowest thing in the cluster.
• Index the raw field path, not a derived value. Index spec.configMapRef directly, not a string you compute from it; this keeps the index updates cheap and avoids subtle staleness when the derivation changes.

Built-in predicates you will actually use

The predicate package ships a small catalogue of ready-made filters. The three you will reach for in 90% of Operators:

GenerationChangedPredicate{} - the hot-loop killer

Fires on Update events only when metadata.generation differs between old and new objects. The API server bumps generation only on .spec (or sometimes annotation) changes - never on .status. Apply this anywhere your reconciler writes status:

ctrl.NewControllerManagedBy(mgr).
    For(&v1alpha1.MyKind{}).
    WithEventFilter(predicate.GenerationChangedPredicate{}).
    Complete(r)

Without it, every r.Status().Update(...) you make triggers a fresh watch event, which triggers another reconcile, which writes status, which... you see where this is going. See the reconcile loop explained for the full hot-loop autopsy, and Status subresource and Conditions for the complementary equality.Semantic.DeepEqual guard you should pair with this predicate when writing .status.

ResourceVersionChangedPredicate{} - fires on every server-side update

The opposite of GenerationChangedPredicate. Use it when you genuinely need to react to every change - including .status updates from peer controllers and managedFields updates from Server-Side Apply. Most Operators do not want this.

LabelChangedPredicate{} and AnnotationChangedPredicate{}

Fire on Update only when labels or annotations differ. Useful when you react to user-managed tags - for instance, a backup operator that pauses backups when a Pod has acme.io/backup: paused.

Combining predicates

The predicate package ships And and Or combinators:

predicate.And(
    predicate.GenerationChangedPredicate{},
    predicate.LabelChangedPredicate{},
)

You can also scope a predicate to a specific Builder verb instead of applying it controller-wide:

labelPred, err := predicate.LabelSelectorPredicate(metav1.LabelSelector{
    MatchLabels: map[string]string{"app": "myapp"},
})
if err != nil {
    return err
}

.Watches(
    &corev1.ConfigMap{},
    handler.EnqueueRequestsFromMapFunc(r.mapCM),
    builder.WithPredicates(labelPred),
)

This is the right tool when the primary watch wants GenerationChangedPredicate but the secondary watch needs a different filter - otherwise the controller-wide WithEventFilter would over-filter ConfigMap events.

Writing a custom predicate.Funcs

The most maintainable custom predicates set all four fields explicitly, even when three of them just return true:

import "sigs.k8s.io/controller-runtime/pkg/event"

productionOnly := predicate.Funcs{
    CreateFunc: func(e event.CreateEvent) bool {
        return e.Object.GetLabels()["tier"] == "production"
    },
    UpdateFunc: func(e event.UpdateEvent) bool {
        oldTier := e.ObjectOld.GetLabels()["tier"]
        newTier := e.ObjectNew.GetLabels()["tier"]
        return oldTier == "production" || newTier == "production"
    },
    DeleteFunc: func(e event.DeleteEvent) bool {
        return e.Object.GetLabels()["tier"] == "production"
    },
    GenericFunc: func(e event.GenericEvent) bool {
        return e.Object.GetLabels()["tier"] == "production"
    },
}

Two patterns worth noting:

• The UpdateFunc checks both old and new labels - so removing the tier=production label still fires one final reconcile, which is usually what you want for cleanup.
• The DeleteFunc matches on the cached object - which may be slightly stale, but the only object the framework has when the watch reports a deletion.

Filtering at the cache layer

The cheapest filter is the one the informer never receives. controller-runtime lets you scope the cache itself via cache.Options.ByObject:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme: scheme,
    Cache: cache.Options{
        ByObject: map[client.Object]cache.ByObject{
            &corev1.Pod{}: {
                Label: labels.SelectorFromSet(labels.Set{"app": "myapp"}),
            },
        },
    },
})

This is dramatically more efficient than predicate-level filtering on a busy cluster - your operator's cache never holds the millions of Pods it does not care about. The catch: the SharedInformer treats "object stopped matching the selector" the same as "object was deleted" and emits a DeleteEvent. If your reconciler runs cleanup on DeleteEvent, that cleanup may run when the object is alive and well, just relabeled. Use a finalizer if you must run cleanup only on real deletion.

The controller-runtime cache documentation covers the full ByObject surface, including namespace allow-lists (a common knob for multi-tenant operators) and field selectors.

Predicate vs cache-layer filtering — which one when?

The article so far has shown two places you can filter: cache.Options.ByObject (informer-level, before the event ever exists) and WithEventFilter / builder.WithPredicates (workqueue-level, after the event but before reconcile). They are not interchangeable. A quick decision matrix:

Rule of thumb: filter as early as possible, but never trade away the ability to r.Get() an object you genuinely need to read. Cache filtering is the cheapest, predicate filtering is the most flexible, and they compose — use both in the same SetupWithManager when the situation calls for it.

The five pitfalls that ship to production

In rough order of how often they cause incidents:

1. Custom predicate.Funcs with an over-strict DeleteFunc. The predicate returns false for a deleted object that still needs cleanup, so the handler never sees it. Fix: make delete predicates deliberately permissive, and always test label-removal and deletion paths.

2. No GenerationChangedPredicate on the primary watch. Every Status().Update() triggers another reconcile. Fix: apply the predicate, or skip status writes when the new status equals the cached one.

3. Owns without SetControllerReference. The child is created but never linked back to the parent. Fix: call controllerutil.SetControllerReference(&parent, &child, r.Scheme) before every r.Create(...) of a child.

4. Watches with a MapFunc that does an un-indexed List. The secondary watch fires once per ConfigMap change, your MapFunc does a full-cluster List, your operator gets quoted from the apiserver. Fix: register a field indexer so the lookup is O(1).

5. Cache Label selector used for cleanup logic. "Object left the cache" is delivered as DeleteEvent whether the object was deleted or just relabeled. Fix: rely on finalizers for cleanup-on-real-deletion, not on DeleteEvent alone.

Pitfall cheat sheet

What's next?

With watches, events, and predicates handled, you have the full picture of how a controller observes the cluster. Natural next reads:

• The controller-runtime architecture — the wider picture (Manager, Cache, Workqueue) that this article zooms into.
• The Kubernetes reconcile loop explained — what your Reconcile() function is contractually expected to do once the workqueue hands it a key.
• Status subresource and Conditions — the companion to GenerationChangedPredicate: how to write status without triggering the hot loop the predicate exists to absorb.
• Multi-resource reconciliation — the multi-child reconcile pattern that depends end-to-end on Owns() and the cache filtering covered here.
• Owner references and garbage collection — the contract Owns() requires you to honour at child-creation time, plus the cascade-delete semantics that follow.
• Server-Side Apply (SSA) in operators — the writer-side counterpart to the managedFields updates you saw in ResourceVersionChangedPredicate.
• Custom Resource Definitions explained — the schema that the watches in this article are watching.
• Finalizers in Kubernetes — the AddFinalizer / RemoveFinalizer dance that pairs with DeleteEvent to make deletion safe even under cache-filter relabel churn.

Looking for the bigger picture? The Kubernetes Operator tutorial sequences every article in this series in pedagogical order — this article sits in the controller-runtime internals chapter alongside the architecture, status, finalizers, and owner-references guides.