Timeout for SUBMITTED status

[aalopatin]

What question do you want to ask?

• ✋ I have searched the open/closed issues and my issue is not listed.

Is there a way to finish CRD SparkApplication by timeout if it is in a status SUBMITTED too long?

I have this problem if we make a mistake in CRD manifest, for example, mount not existing PVC, driver pod will be in state PENDING infinity long. AS a result SparkApplication is in a state SUBMITTED infinity long. I'd like to finish/delete/mark as failed SparkApplication after a given timeout.

Is it possible to set?

Additional context

No response

Have the same question?

Give it a 👍 We prioritize the question with most 👍

[tariq-hasan]

Hello @aalopatin! Thanks for raising the question.

Is there a way to finish CRD SparkApplication by timeout if it is in a status SUBMITTED too long?

I have this problem if we make a mistake in CRD manifest, for example, mount not existing PVC, driver pod will be in state PENDING infinity long. AS a result SparkApplication is in a state SUBMITTED infinity long. I'd like to finish/delete/mark as failed SparkApplication after a given timeout.

Is it possible to set?

I believe we don't.

The SparkApplication state transitions as a function of the Driver state as defined here currently does not contain any logic for timeouts.

The state transitions as defined here simply map a PENDING driver state to a SUBMITTED SparkApplication state.

There is, however, a grace period that can be defined for driver pod creation but it does not apply to a pending driver state.