Add cleanup job support for pre-deletion hooks via Kubernetes Jobs

[Copilot]

Enables custom cleanup scripts to run before deleting expired lease objects. Users can configure Jobs via annotations to backup data, notify webhooks, or clean up related resources.

Changes

Core Implementation

• New annotations (7 total): on-delete-job (ConfigMap/script reference), job-service-account, job-image, job-wait, job-timeout, job-ttl, job-backoff-limit
• pkg/util/cleanup_job.go: Job creation, config parsing, and completion polling with timeout
• pkg/controllers/lease_controller.go: Modified handleExpired() to check for cleanup config and execute jobs before deletion
  • Best-effort semantics: cleanup failures never block deletion
  • Supports sync (wait) and async (fire-and-forget) modes
• RBAC: Added batch/jobs permissions to ClusterRole

Observability

• Metrics: cleanup_jobs_created_total, cleanup_jobs_failed_total, cleanup_jobs_completed_total, cleanup_job_duration_seconds
• Events: CleanupJobCreated, CleanupJobCompleted, CleanupJobFailed, CleanupJobTimeout

Environment Variables

Jobs receive 11 env vars: OBJECT_NAME, OBJECT_NAMESPACE, OBJECT_KIND, OBJECT_GROUP, OBJECT_VERSION, OBJECT_UID, OBJECT_RESOURCE_VERSION, LEASE_STARTED_AT, LEASE_EXPIRED_AT, OBJECT_LABELS (JSON), OBJECT_ANNOTATIONS (JSON)

Example Usage

apiVersion: startpunkt.ullberg.us/v1alpha2
kind: Application
metadata:
  name: demo-app
  annotations:
    object-lease-controller.ullberg.io/ttl: "2h"
    object-lease-controller.ullberg.io/on-delete-job: "cleanup-scripts/backup.sh"
    object-lease-controller.ullberg.io/job-service-account: "backup-sa"
    object-lease-controller.ullberg.io/job-image: "amazon/aws-cli:latest"
    object-lease-controller.ullberg.io/job-wait: "true"
    object-lease-controller.ullberg.io/job-timeout: "10m"
spec:
  name: Demo Application

ConfigMap script has access to all object metadata via env vars. Jobs auto-cleanup via ttlSecondsAfterFinished. See examples/cleanup/ for S3 backup, webhook, and multi-resource cleanup examples.

Testing

• 10 unit tests covering config parsing, job creation, and completion polling
• Zero regressions, CodeQL clean

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Feature: Support custom cleanup scripts via Kubernetes Jobs before object deletion</issue_title>
<issue_description>## Summary

Add support for executing custom cleanup scripts before deleting expired lease objects. Scripts would run as Kubernetes Jobs with proper RBAC and secret access via ServiceAccount bindings.

Motivation

Users often need to perform cleanup actions before an object is deleted, such as:

• Backing up data to external storage (S3, GCS, etc.)
• Notifying external systems or webhooks
• Cleaning up dependent resources not covered by owner references
• Archiving logs or metrics
• Deregistering from external registries or service meshes
• Graceful shutdown procedures

Currently, when a lease expires, the object is immediately deleted without any opportunity for custom cleanup logic.

Proposed Solution

User Experience

1. Create a cleanup script in a ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: cleanup-scripts
  namespace: my-namespace
data:
  backup-to-s3.sh: |
    #!/bin/bash
    set -e
    echo "Backing up $OBJECT_NAME from namespace $OBJECT_NAMESPACE"
    
    # Fetch the object
    kubectl get $OBJECT_KIND $OBJECT_NAME -n $OBJECT_NAMESPACE -o yaml > /tmp/backup.yaml
    
    # Upload to S3 (credentials from ServiceAccount)
    aws s3 cp /tmp/backup.yaml s3://backups/$OBJECT_NAMESPACE/$OBJECT_KIND/$OBJECT_NAME-$(date +%s).yaml
    
    echo "Backup complete"

2. Create a ServiceAccount with necessary permissions and secrets

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cleanup-sa
  namespace: my-namespace
---
apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: my-namespace
  annotations:
    kubernetes.io/service-account.name: cleanup-sa
type: Opaque
data:
  aws-access-key-id: <base64-encoded>
  aws-secret-access-key: <base64-encoded>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cleanup-role
  namespace: my-namespace
rules:
- apiGroups: ["startpunkt.ullberg.us"]
  resources: ["applications"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["configmaps", "secrets", "pods"]
  verbs: ["get", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cleanup-binding
  namespace: my-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cleanup-role
subjects:
- kind: ServiceAccount
  name: cleanup-sa
  namespace: my-namespace

3. Annotate the resource with cleanup configuration

apiVersion: startpunkt.ullberg.us/v1alpha2
kind: Application
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    # Existing lease configuration
    object-lease-controller.ullberg.io/ttl: "2h"
    
    # New cleanup job configuration
    object-lease-controller.ullberg.io/on-delete-job: "cleanup-scripts/backup-to-s3.sh"
    object-lease-controller.ullberg.io/job-image: "amazon/aws-cli:latest"
    object-lease-controller.ullberg.io/job-service-account: "cleanup-sa"
    object-lease-controller.ullberg.io/job-wait: "true"
    object-lease-controller.ullberg.io/job-timeout: "5m"
    object-lease-controller.ullberg.io/job-ttl: "300"
spec:
  name: My Application
  url: https://example.com

Behavior

When a lease expires:

1. Controller checks for on-delete-job annotation
2. If present, creates a Kubernetes Job:
   • Mounts the script from the specified ConfigMap
   • Runs with the specified ServiceAccount (secrets bound via SA)
   • Passes object metadata as environment variables
   • Optionally waits for completion (configurable)
3. After Job completes (or if no hook configured), deletes the object
4. Job auto-cleans up after job-ttl seconds (via ttlSecondsAfterFinished)

New Annotations

Annotation	Required	Default	Description
object-lease-controller.ullberg.io/on-delete-job	Yes*	-	ConfigMap reference in format configmap-name/script-key
object-lease-controller.ullberg.io/job-service-account	No	default	ServiceAccount to run the Job as (with bound secrets)
object-lease-controller.ullberg.io/job-image	No	bitnami/kubectl:latest	Container image for running the script
object-lease-controller.ullberg.io/job-wait	No	false	Wait for Job completion before deleting object
object-lease-controller.ullberg.io/job-timeout	No	5m	Maximum time to wait for Job completion
object-lease-controller.ullberg.io/job-ttl	No	300	TTL in seconds for Job cleanup (ttlSecondsAfterFinished)
object-lease-controller.ullberg.io/job-backoff-limit	No	3	Number of retries for failed Jobs

* Only if cleanup hook is desired

Environment Variables Available in Script

The cleanup script receives these environment variables:

• OBJECT_NAME - Name of the object being deleted
• `OBJEC...

• Fixes Feature: Support custom cleanup scripts via Kubernetes Jobs before object deletion #47

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 98.44358% with 4 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
cmd/main.go	80.95%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull Request Overview

This PR adds cleanup job functionality to the object-lease-controller, enabling custom scripts to run via Kubernetes Jobs before expired objects are deleted. This feature allows users to perform tasks like backing up data, notifying external systems, or cleaning up related resources.

• Introduces a new CleanupJobConfig struct and utility functions for parsing cleanup job annotations and creating Kubernetes Jobs
• Integrates cleanup job execution into the lease expiration workflow with optional synchronous/asynchronous modes
• Adds new Prometheus metrics for tracking cleanup job lifecycle (created, completed, failed, duration)

Reviewed Changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/util/cleanup_job.go	Core implementation for parsing cleanup job config, creating Jobs, and waiting for completion
pkg/util/cleanup_job_test.go	Comprehensive test coverage for cleanup job utilities
pkg/controllers/lease_controller.go	Integrates cleanup job execution into lease expiration handling
pkg/metrics/metrics.go	Adds four new Prometheus metrics for cleanup job monitoring
cmd/main.go	Registers cleanup job annotations and adds batch/v1 scheme
object-lease-operator/helm-charts/leasecontroller/templates/role.yaml	Adds RBAC permissions for Job operations
examples/cleanup/*.yaml	Three complete example scenarios with documentation
README.md	Documents cleanup job feature, annotations, and environment variables
go.mod, go.sum	Dependency updates

[github-actions]

Pull Request closed and locked due to lack of activity.
If you'd like to build on this closed PR, you can clone it using this method: https://stackoverflow.com/a/14969986
Then open a new PR, referencing this closed PR in your message.