feat: background workers (config + ensure)

[nicolas-grekas]

feat: background workers (config + ensure)

First half of the split suggested in #2287. Lands a minimum-viable background-worker subsystem: config surface, lifecycle, lazy-start via ensure(), per-php_server scoping, named pools, multi-entrypoint, plus a $_SERVER flag for bg-aware scripts. The worker-to-HTTP shared-state APIs (frankenphp_set_vars / frankenphp_get_vars) and the docs are deferred to a follow-up PR — they're independent and easier to review separately.

What lands

PHP API

• frankenphp_ensure_background_worker(string|array $name, ?float $timeout = null): void — declares a dependency on one or more bg workers. Lazy-starts the named worker (or pulls from a catch-all) if not already running, then blocks until the worker calls frankenphp_get_worker_handle() (the readiness signal) or the timeout fires. null (the default) falls back to FrankenPHP's internal default deadline; a value <= 0 raises ValueError. The actual default is intentionally not exposed in the signature so it can become tunable later (e.g. via Caddyfile or env) without an API break. Input is validated upfront (ValueError for empty array / empty string / duplicate names / non-positive timeout; TypeError for non-string elements) so a bad batch never leaves a half-spawned set behind. On timeout the error names what didn't happen — a worker that never reached frankenphp_get_worker_handle() gets a self-teaching diagnostic instead of a silent hang.
• frankenphp_get_worker_handle(): resource — readable stream that signals graceful shutdown. PHP scripts park on stream_select; FrankenPHP closes the write end during drain so select wakes with EOF. Calling this also signals to ensure() that the worker has reached its main loop, so a well-formed bg worker satisfies both contracts with one call.

The follow-up PR will tighten ensure() to also require a first call to frankenphp_set_vars() — same ensure() signature, progressively stronger guarantee, no caller-visible API change.

In CLI mode these functions aren't exposed.

Caddyfile

php_server {
    # HTTP worker (unchanged)
    worker public/index.php { num 4 }

    # Named bg worker, eagerly started
    worker bin/jobs.php {
        background
        name job-runner
        num 1
    }

    # Catch-all bg worker, instantiated lazily by ensure(name)
    worker bin/jobs.php {
        background
        max_threads 16
    }
}

• background marks a worker as non-HTTP.
• name pins an exact worker name; declarations without name are catch-alls for lazy-started instances.
• num on a named bg worker eagerly starts that many instances; num 0 (or omitted) defers start until ensure().
• max_threads on a catch-all caps how many distinct lazy-started instances it can host.
• max_consecutive_failures defaults to 6 (same as HTTP workers).
• max_execution_time is automatically disabled for bg workers.

Go API

• WithWorkerBackground() marks a worker declaration as background.
• WithWorkerScope(scope) tags a declaration with an isolation scope.
• WithRequestScope(scope) tags a request so ensure() from a regular HTTP request resolves to the right block's lookup.
• NextScope() hands out a fresh Scope value (opaque uint64 under the hood; zero is the global/embed scope). The type is intentionally generic so it can be reused for other per-server contexts (e.g. Mercure hubs, Prometheus labels).

Per-php_server scoping

Each php_server block gets its own scope. The same user-facing worker name can live in multiple blocks without collision; ensure() resolves through the calling thread's scope (worker handler → request context → global).

Metrics

Background workers report under a prefixed Prometheus worker label so two php_server blocks declaring same-named bg workers stay on distinct series:

frankenphp_total_workers{worker="m#api.example.com:job-runner"}
frankenphp_total_workers{worker="m#admin.example.com:job-runner"}

Format: m#<scope-label>:<worker-name>. <scope-label> resolves per-php_server block via a cascade (first host of the route's host matcher → first listener address). Catch-all instances substitute the <worker-name> half with the name passed to ensure(). Embed mode (no Caddy module) leaves the label empty: m#:job-runner — uniform regex m#([^:]*):(.+) parses both. The m# prefix matches the existing module-worker convention.

HTTP-worker metric labels are unchanged.

Pools and multi-entrypoint

• num > 1 on a named bg worker spawns N threads sharing the same name. Each thread has its own stop pipe so drain can wake them independently.
• Two named bg workers in the same scope can share an entrypoint file. They keep independent options (env, watch mode, failure policy).

Server variables

• $_SERVER['FRANKENPHP_WORKER'] carries the resolved worker name (was previously "1"; pre-existing user code that only tests isset(...) keeps working). Catch-all instances see the name they were started under.
• $_SERVER['FRANKENPHP_WORKER_BACKGROUND'] = true for bg workers — single-key branch for "am I a bg worker?".

Readiness

One readiness channel per worker instance, closed exactly once on the first frankenphp_get_worker_handle() call. ensure() selects on that channel against an abort channel (populated when the worker exhausts max_consecutive_failures during boot) and the per-call deadline. Crash-restarts don't re-arm the signal — a worker that announced "ready" once stays ready for any future ensure() caller, which is the right semantics for a long-lived dependency.

Pre-readiness crashes capture metadata per attempt (entrypoint, exit status, attempt count) on the same state slot, so a timing-out ensure() surfaces a self-teaching diagnostic ("x did not become ready within Xs; last attempt N failed (exit status M, entrypoint …)") instead of just "did not call frankenphp_get_worker_handle()". The follow-up PR adds PG(last_error_message) capture to that record once the C-side helper lands.

For catch-all workers each lazy-spawned name has its own readiness slot, so a stuck foo doesn't keep ensure('bar') waiting; for named pools (num > 1) the threads share one slot and the first to reach frankenphp_get_worker_handle() wins.

Lifecycle

• Crash recovery: quadratic backoff capped at 1s; max_consecutive_failures aborts startup if hit during the boot phase.
• Graceful shutdown: stop pipe closes (EOF), workers exit cleanly. After a 30s grace period, a best-effort force-kill drill (per feat: cross-platform force-kill primitive for stuck PHP threads #2365) interrupts threads stuck in blocking syscalls.

What's deferred

A follow-up PR adds:

• frankenphp_set_vars(array $vars): void — publish persistent vars from a bg worker.
• frankenphp_get_vars(string $name): array — pure read, with generational cache so repeated calls within a request return the same array instance (=== is O(1)).
• A frankenphp_set_vars-driven readiness signal that lets ensure() block until a worker has bootstrapped (turning fire-and-forget into a stronger contract without an API change).
• The full docs/background-workers.md reference.

That split keeps the surfaces independent: this PR is the lifecycle/wiring; the follow-up is the data plane.

Tests

End-to-end tests use file sentinels (workers touch a path provided via env) instead of cross-thread observation, since this PR has no shared-state API yet:

• TestBackgroundWorkerLifecycle / TestBackgroundWorkerCrashRestarts / TestBackgroundWorkerWithoutHTTP
• TestBackgroundWorkerRestartForceKillsStuckThread (force-kill drill on a bg worker stuck in sleep(60))
• TestEnsureBackgroundWorkerNamedLazy / TestEnsureBackgroundWorkerCatchAll / TestEnsureBackgroundWorkerCatchAllCap / TestEnsureBackgroundWorkerUndeclared
• TestNextBackgroundWorkerScopeIsDistinct / TestBackgroundWorkerSameNameDifferentScope / TestBackgroundWorkerCatchAllPerScope
• TestBackgroundWorkerPool / TestBackgroundWorkerMultiEntrypoint
• TestEnsureBackgroundWorkerBatch (and the three validation-error variants)
• TestBackgroundWorkerBgFlag

Test plan

• go test ./... clean
• Sanitizers (asan, msan) clean
• Manual: declare a named bg worker, observe its sentinel; restart it via RestartWorkers(), observe re-spawn
• Manual: declare two php_server blocks with same-named bg workers, verify they don't collide

[Copilot]

Pull request overview

Adds a first-cut “background worker” subsystem to FrankenPHP, including Go/C/PHP APIs and Caddyfile configuration, to run long-lived non-HTTP PHP scripts with scoped name resolution and lazy start via ensure().

Changes:

• Introduces background-worker declarations (Caddy + Go options) with per-php_server scoping and lazy-start support (frankenphp_ensure_background_worker).
• Adds a background worker thread handler with drain signaling via a stop-pipe exposed to PHP (frankenphp_get_worker_handle) and crash-restart backoff.
• Adds end-to-end tests and PHP fixtures validating lifecycle, crash restart, scoping, pools, batch ensure validation, and $_SERVER bg flag injection.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
worker.go	Tracks background-worker metadata on workers; builds per-scope lookups; starts bg threads with a dedicated handler.
threadbackgroundworker.go	New thread handler implementing background worker lifecycle, drain behavior, and restart/backoff.
background_worker.go	New background-worker registry/lookup system, scoping, and ensure() lazy-start implementation.
frankenphp.c	Adds TLS state for bg workers, stop-pipe primitives, PHP functions frankenphp_ensure_background_worker and frankenphp_get_worker_handle, and injects $_SERVER flags.
frankenphp.h	Exposes C primitives for bg worker name + stop-pipe operations.
frankenphp.go	Reserves thread budget for background workers and resets bg lookup globals on shutdown.
options.go	Adds Go WorkerOptions WithWorkerBackground and WithWorkerBackgroundScope.
requestoptions.go	Adds RequestOption WithRequestBackgroundScope to scope ensure() resolution per request.
context.go	Stores the request’s background scope in frankenPHPContext.
phpthread.go	Invokes handler drain() during shutdown to wake bg workers blocked in C calls.
frankenphp.stub.php	Adds PHP stubs/docs for frankenphp_ensure_background_worker and frankenphp_get_worker_handle.
frankenphp_arginfo.h	Adds arginfo for new PHP functions (but currently with a placeholder stub hash).
caddy/workerconfig.go	Adds background worker subdirective; rejects unsupported directives (e.g., match) for bg workers.
caddy/module.go	Assigns a unique background-worker scope per php_server and tags requests/workers with it.
caddy/app.go	Wires Caddy config to Go via WithWorkerBackground().
background_worker_test.go	Integration tests for bg worker lifecycle, crash restart, and ensuring bg workers don’t intercept HTTP.
background_worker_scope_test.go	Tests scope isolation and catch-all behavior per scope.
background_worker_pool_test.go	Tests named pool workers and multi-entrypoint support.
background_worker_internal_test.go	Adds force-kill integration test, but it is currently unconditionally skipped.
background_worker_ensure_test.go	Tests ensure() for named lazy workers, catch-all, caps, and undeclared errors.
background_worker_batch_test.go	Tests array-form ensure() and its validation error cases, plus bg flag injection.
testdata/background-worker.php	Fixture: long-lived bg worker that touches a sentinel and blocks on stop pipe.
testdata/background-worker-stuck.php	Fixture: intentionally stuck worker (sleep) for force-kill testing.
testdata/background-worker-pool.php	Fixture: pool worker writes unique sentinels per thread then blocks.
testdata/background-worker-named.php	Fixture: writes per-name sentinel based on FRANKENPHP_WORKER_NAME.
testdata/background-worker-crash.php	Fixture: crash-once-then-succeed to validate crash-restart behavior.
testdata/background-worker-bg-flag.php	Fixture: writes the exact PHP value of FRANKENPHP_WORKER_BACKGROUND.
testdata/background-worker-batch-errors.php	HTTP fixture exercising batch ensure() validation paths.
testdata/background-worker-batch-ensure.php	HTTP fixture ensuring multiple workers in a single batch call.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dunglas]

First batch. I didn't finish the review yet.

[nicolas-grekas]

I merged the metrics concern into this PR since it's only for bgworkers.

Here is the description I added above:

Background workers report under a prefixed Prometheus worker label so two php_server blocks declaring same-named bg workers stay on distinct series:

frankenphp_total_workers{worker="m#api.example.com:job-runner"}
frankenphp_total_workers{worker="m#admin.example.com:job-runner"}

Format: m#<scope-label>:<worker-name>. <scope-label> resolves per-php_server block via a cascade (first host of the route's host matcher → first listener address). Catch-all instances substitute the <worker-name> half with the name passed to ensure(). Embed mode (no Caddy module) leaves the label empty: m#:job-runner — uniform regex m#([^:]*):(.+) parses both. The m# prefix matches the existing module-worker convention.

HTTP-worker metric labels are unchanged.

[henderkes]

Yes, I like that design! Let me know when this is ready for review.

[AlliBalliBaba]

Not sure I like this flow. If a background worker enters a 'failed state' and is eager started, we should just fail on the startup channel.

The issue with lazily started threads is that there's not really a good way to handle a failure state, the server is just broken in this case. You should either panic or retry endlessly, shutting down the thread doesn't really solve anything.

[AlliBalliBaba]

Not having a clear point to catch a failures is by design, which is also why I still dislike the lazy starts in general.

Best thing to do here is just repeatedly notifying the user in the logs that the background worker failed.

[AlliBalliBaba]

Maybe better to just get the scope directly:

Suggested change

	scopedWorker() *worker
	scope() scope

[AlliBalliBaba]

Don't you just need the second part of that if-statement? fc should always be assigned to the worker.

Suggested change

	var workerName string
	var isBackgroundWorker bool
	if bgHandler, ok := thread.handler.(*backgroundWorkerThread); ok {
	workerName = strings.TrimPrefix(bgHandler.runtimeName, "m#")
	isBackgroundWorker = true
	} else if fc.worker != nil {
	workerName = strings.TrimPrefix(fc.worker.name, "m#")
	isBackgroundWorker = fc.worker.bg != nil
	}
	var workerName string
	var isBackgroundWorker bool
	fc.worker != nil {
	workerName = strings.TrimPrefix(fc.worker.name, "m#")
	isBackgroundWorker = fc.worker.bg != nil
	}

[AlliBalliBaba]

Couldn't you just put all of this into the backgroundWorkerState?

[AlliBalliBaba]

Still feels a bit too complex, will have to do more reviewing later.

[AlliBalliBaba]

Just to clarify: something like this is still how you envision the end result with background workers?

# http-worker.php
frankenphp_ensure_background_worker(['something', 'something else']);

frankenphp_handle_request(function(){
    $vars = frankenphp_get_vars('something')
    ....
})

# bg-worker.php
$whatToDo = $_SERVER['FRANKENPHP_WORKER_NAME');

switch($whatToDo){
    case 'somehting';
        # ...
        frankenphp_set_vars([...]);
    case 'something else':
        # ...
        frankenphp_set_vars([...]);
}

$handle = frankenphp_get_handle();

# put the handle into an async event loop react/amp

[nicolas-grekas]

@AlliBalliBaba correct!
frankenphp_ensure_background_worker() would also work in non-worker HTTP threads, which means the function could be called anywhere.

That being said: I agree the PR is complex. I'm going to submit one with lazy-start removed and declared-only workers. This means no ensure() function needed in this step. It might be easier to review and to agree on.

I'll still want to discuss the other steps to eventually reach the state in sidekicks/#2287 - feature/DX-wise. But we'll also have more steps to discuss as progress is made.