Skip to content

PersistentErrorTracker prematurely marks errors "resolved" when an unrelated event triggers reconciliation #2102

Description

@Priyanshu-u07

Describe the bug
The non-blocking error handling introduced in #2043 can mark an error as resolved even though the underlying operation is still failing. PersistentErrorTracker.UpdateFromRegistry treats an error as resolved whenever it's absent from a reconciliation cycle's ErrorRegistry. But a workflow run only exercises the reconcilers whose subscription matches the triggering event, so an error recorded by reconciler A gets cleared by a workflow triggered by an unrelated event B, where A never ran and never re-recorded its still-failing error. The retry loop can therefore give up before the underlying condition clears, and the error is dropped from the tracker without having actually been resolved.

Surfaced while validating #2088 (fix for #1578).

To Reproduce
Steps to reproduce the behavior:

  1. Deploy the operator to a cluster (make local-setup).
  2. Remove the create verb from the authorinos rule in the kuadrant-operator-manager-role ClusterRole, so the operator can no longer create Authorino resources.
# strip "create" from the authorinos rule in the operator's ClusterRole
kubectl get clusterrole kuadrant-operator-manager-role -o json | \
jq --arg idx "$(kubectl get clusterrole kuadrant-operator-manager-role -o json | jq '.rules | map(.resources // [] | contains(["authorinos"])) | index(true)')" \
  '.rules[$idx|tonumber].verbs |= map(select(. != "create"))' | kubectl apply -f -
  1. Create a Kuadrant CR (this also generates unrelated Limitador reconcile events):
kubectl apply -f - <<'EOF'
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
  namespace: kuadrant-system
spec: {}
EOF
  1. Watch the operator logs, do not restore the RBAC. The Authorino create keeps failing, yet the error is marked resolved in between failures:
{"level":"error","ts":"2026-07-10T09:00:05.232Z","logger":"kuadrant-operator.AuthorinoReconciler","msg":"failed to create authorino resource","error":"...forbidden..."}
{"level":"info","ts":"2026-07-10T09:00:05.310Z","logger":"kuadrant-operator.PersistentErrorTracker","msg":"non-blocking error resolved","key":"Authorino.operator.authorino.kuadrant.io/kuadrant-system/authorino/create","attempts":2}
{"level":"error","ts":"2026-07-10T09:00:05.377Z","logger":"kuadrant-operator.AuthorinoReconciler","msg":"failed to create authorino resource","error":"...forbidden..."}
{"level":"error","ts":"2026-07-10T09:00:07.387Z","logger":"kuadrant-operator.AuthorinoReconciler","msg":"failed to create authorino resource","error":"...forbidden..."}
{"level":"info","ts":"2026-07-10T09:00:07.465Z","logger":"kuadrant-operator.PersistentErrorTracker","msg":"non-blocking error resolved","key":"Authorino.operator.authorino.kuadrant.io/kuadrant-system/authorino/create","attempts":2}

Expected behavior
An error should only be marked resolved when the operation that recorded it actually succeeds (or is genuinely no longer needed) not merely because a reconcile cycle triggered by an unrelated event didn't re-record it. Errors whose recording reconciler didn't run in a given cycle should be left untouched, so retries continue until the underlying condition actually clears.

Additional context

The non-blocking error resolved line in the reproduction is emitted by PersistentErrorTracker.UpdateFromRegistry in internal/controller/reconciliation_errors.go, which removes any tracked error that is not present in the current reconciliation cycle's ErrorRegistry.

This is shared error-handling code introduced in #2043; it is not specific to the Authorino or Limitador reconcilers. Because the resolution logic operates on the shared tracker, it applies to any reconciler that records errors into it (e.g. AuthConfigsReconciler and LimitadorLimitsReconciler, also wired in #2043).

The reproduction uses the Authorino create path (from #2088) only because it is the simplest way to inject a persistent, easily-triggered failure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions