[ST] MigrationST - some `KafkaTopics` are not deleted correctly after the migration tests

[im-konge]

When we have finalizers enabled in UTO and we are running the MigrationST, from time to time the KafkaTopics cannot be deleted.
The status of such topics is following:

- apiVersion: kafka.strimzi.io/v1beta2
  kind: KafkaTopic
  metadata:
    creationTimestamp: "2024-04-11T14:10:36Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2024-04-11T14:22:45Z"
    finalizers:
    - strimzi.io/topic-operator
    generation: 2
    labels:
      strimzi.io/cluster: cluster-820b5b0a
      test.case: testMigrationWithDeletionOfCOAndZkDeleteClaimFalse
    name: my-topic-368936459-2026653918-continuous
    namespace: co-namespace
    resourceVersion: "19253"
    uid: 73f22f1f-0b6a-42ed-abf1-9dd67a4e1719
  spec:
    config:
      min.insync.replicas: 2
      retention.ms: 7200000
      segment.bytes: 1073741824
    partitions: 3
    replicas: 3
  status:
    conditions:
    - lastTransitionTime: "2024-04-11T14:27:46.255798715Z"
      message: 'org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteTopics,
        deadlineMs=1712845666154, tries=1, nextAllowedTryMs=1712845666326) timed out
        at 1712845666215 after 1 attempt(s)'
      reason: KafkaError
      status: "False"
      type: Ready
    observedGeneration: 2
    topicId: 38fhq1a8RTWnnnAFvgd1UA
    topicName: my-topic-368936459-2026653918-continuous

For now, the finalizers in UTO were disabled - which fixes the issue now. But we should investigate it more and fix it in the tests.

[im-konge]

The MigrationST was removed as part of the ZK removal. Closing this as it's not relevant anymore.

[LaithS]

Shouldn't a topic operator remove finalizers it had injected upon shutdown?

[im-konge]

I don't think that's the way how the topic operator should work.
The finalizers ensure that the resources will not be deleted by something else - so there has to be operator that will delete it correctly.
That's useful in case that the operator is down or deleted and you have resources there still -> then when you try to delete them, it will not be possible.

So no I don't think that finalizers should be removed by the topic operator when it's being shut down.

[scholzj]

Correct, as Lukas said. It should not remove them. One of the reasons the finalizers are there is to not miss any deletions when the operator is not running. So removing them at shutdown would be against one of the main ideas.

[LaithS]

The finalizers ensure that the resources will not be deleted by something else - so there has to be operator that will delete it correctly.
Finalizers ensure that when an object is deleted from the cluster, an appropriate cleanup action is taken, not to leave any dependent resources lingering - at least that's how I understand its objective - .
That coupled with - an assumption - that the operator on this resource type is the entity operator which has a lifecycle bound to the cluster itself ( which contains the dependent resource, the kafka topics themselves ), I don't see the benefit of leaving the finalizers there after the entity operator has shut down. It will simply prevent the deletion of the KafkaTopic custom resource, and no entity will remove that finalizer automatically, unless with manual intervention.

[im-konge]

I think that in case that you implement the logic that "once the operator managing the resource is deleted, you don't need the finalizers", you don't have to implement the finalizers at all.
The finalizers - as you also mentioned correctly - are there for correct clean up and management of the resources. But if you don't have operator managing these resources - so there is nothing that can do the proper clean up - it shouldn't be simply removed. That's the purpose.

that the operator on this resource type is the entity operator which has a lifecycle bound to the cluster itself

The Topic Operator can be run also standalone - and it will not be bound to deletion to any of the Kafka cluster, you can manage topics on completely different Kafka cluster. So there are two ways how to run the operator actually.

Anyway, as mentioned before, removing the finalizers automatically when the TO is deleted wouldn't make any sense, as you want to have a possibility to continue from where you stopped before when you for example deleted the TO, or when you do recovery because it was crashlooping for longer time etc.

[scholzj]

@LaithS You are right that the finalizer makes little sense outside of the operator's lifecycle. But the operators do not control their own lifecycle. They do not know when they are being shutdown, whether they are being restarted, whether they are stopped for a maintenance procedure, for an upgrade, etc. Or whether you decided to uninstall it. So it cannot really cleanup after itself. You would not want the operator to delete all your Kafka topics just because your Kubernetes cluster has to undergo some maintenance, and the operator was stopped.

It is you who controls the life-cycle of the operator and you might need to take action. E.g. delete all topics before deleting the cluster / uninstalling the operator.