[Debian 13] Slow Quadlet startup time, but the equivalent podman run command is instant

[regunakyle]

From a fresh Debian 13 install:

sudo apt install podman crun -y
podman pull docker.io/codercom/code-server:latest

Then:

podman run --name code-server -p 8000:8080 \
-v /home/eleung/coder/volume:/home/coder \
--userns "keep-id" --env DOCKER_USER=root \
--pull never docker.io/codercom/code-server:latest

This starts up in an instant.

But the equivalent user Quadlet file takes about a minute to start/print logs to journalctl output:

[Unit]
Description=Code-Server Development Environment

[Container]
Image=docker.io/codercom/code-server:latest
ContainerName=code-server
PublishPort=8000:8080
Volume=/home/eleung/coder/volume:/home/coder
UserNS=keep-id
Environment=DOCKER_USER=root
Pull=never

[Service]
Restart=always

[Install]
WantedBy=default.target

eleung@vfio-linux:~/coder$ time systemctl --user start code-server.service

real    0m57.748s
user    0m0.003s
sys     0m0.000s

Is this normal? If not, does anyone know how do I fix this? Thanks!

[eriksjolund]

Maybe related to

Creating a container takes significantly longer with _native overlayfs_ than _fuse-overlayfs_
when these conditions are all met:

* rootless Podman is used
* a modified UID/GID mapping is used
* _native overlayfs_ is used
* no container has yet been created with the specified container image and UID/GID mapping

copy-paste from

podman/docs/tutorials/performance.md

Lines 63 to 84 in 00012e3

	1. native overlayfs
	2. fuse-overlayfs
	3. vfs
	There is one notable exception to this speed ranking.
	Creating a container takes significantly longer with _native overlayfs_ than _fuse-overlayfs_
	when these conditions are all met:
	* rootless Podman is used
	* a modified UID/GID mapping is used
	* _native overlayfs_ is used
	* no container has yet been created with the specified container image and UID/GID mapping
	Runtime speed is not affected. Only __podman create__ and the container creation phases of
	__podman run__ and __podman build__ are affected.
	For more details, see [GitHub comment](https://github.com/containers/podman/issues/16541#issuecomment-1352790422).
	Command-line options that modify the UID/GID mapping are for example __--userns__, __--uidmap__ and __--gidmap__.
	The command-line option `--userns auto` is particularly affected by this performance penalty,
	because different UID/GID mappings could potentially be used on each invocation. For other uses of
	__--userns__, __--uidmap__ and __--gidmap__ the performance penalty is a one-time cost
	that only occurs the first time the command is run.

Did you try the quadlet before trying "podman run" from the command-line?

[regunakyle]

If I understand you correctly, creating the container manually before starting up the quadlet should speed up the latter.
But it doesn't seem to be the case:

eleung@vfio-linux:~/coder$ podman container ls -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

eleung@vfio-linux:~/coder$ podman run --name code-server -p 8000:8080 -v /home/eleung/coder/volume:/home/coder --userns "keep-id" --env DOCKER_USER=root --pull never docker.io/codercom/code-server:latest
<omitted output>

eleung@vfio-linux:~/coder$ $ podman container ls -a
CONTAINER ID  IMAGE                                  COMMAND     CREATED         STATUS                   PORTS                   NAMES
69630790e1ae  docker.io/codercom/code-server:latest              17 seconds ago  Exited (0) 1 second ago  0.0.0.0:8000->8080/tcp  code-server

eleung@vfio-linux:~/coder$ time systemctl --user start code-server

real    1m30.262s
user    0m0.003s
sys     0m0.000s

eleung@vfio-linux:~/coder$ podman container ls -a
CONTAINER ID  IMAGE                                  COMMAND     CREATED         STATUS         PORTS                   NAMES
ec236fb6ab1b  docker.io/codercom/code-server:latest              12 seconds ago  Up 12 seconds  0.0.0.0:8000->8080/tcp  code-server

[eriksjolund]

But it doesn't seem to be the case:

I agee. It doesn't seem to be the case here.

Another thing, it would be interesting to see which Podman is run in the quadlet.

What is the output of this command?

systemctl --user cat code-server.service | grep ExecStart=

[regunakyle]

But it doesn't seem to be the case:

I agee. It doesn't seem to be the case here.

Another thing, it would be interesting to see which Podman is run in the quadlet.

What is the output of this command?

systemctl --user cat code-server.service | grep ExecStart=

eleung@vfio-linux:~/coder/volume$ systemctl --user cat code-server.service | grep ExecStart=
ExecStart=/usr/bin/podman run --name code-server --cidfile=%t/%N.cid --replace --rm --cgroups=split --pull never --sdnotify=conmon -d --userns keep-id -v /home/eleung/coder/volume:/home/coder --publish 8000:8080 --env DOCKER_USER=root docker.io/codercom/code-server:latest

[eriksjolund]

Comparing

/usr/bin/podman run --name code-server --cidfile=%t/%N.cid --replace --rm --cgroups=split --pull never --sdnotify=conmon -d --userns keep-id -v /home/eleung/coder/volume:/home/coder --publish 8000:8080 --env DOCKER_USER=root docker.io/codercom/code-server:latest

with

podman run --name code-server -p 8000:8080 -v /home/eleung/coder/volume:/home/coder --userns "keep-id" --env DOCKER_USER=root --pull never docker.io/codercom/code-server:latest

I see there are some differences

• --rm
• --cgroups=split
• --cidfile=%t/%N.cid
• --sdnotify=conmon
• --replace
• -d

I just wanted to highlight that the commands are not identical.

I don't know if any of the differences could cause any difference in startup time. (Probably not is my gut feeling)

[regunakyle]

Comparing

/usr/bin/podman run --name code-server --cidfile=%t/%N.cid --replace --rm --cgroups=split --pull never --sdnotify=conmon -d --userns keep-id -v /home/eleung/coder/volume:/home/coder --publish 8000:8080 --env DOCKER_USER=root docker.io/codercom/code-server:latest

with

podman run --name code-server -p 8000:8080 -v /home/eleung/coder/volume:/home/coder --userns "keep-id" --env DOCKER_USER=root --pull never docker.io/codercom/code-server:latest

I see there are some differences

• --rm
• --cgroups=split
• --cidfile=%t/%N.cid
• --sdnotify=conmon
• --replace
• -d

I just wanted to highlight that the commands are not identical.

I don't know if any of the differences could cause any difference in startup time. (Probably not is my gut feeling)

FWIW I just confirmed the content of my Quadlet file:

The extra params in the ExecStart were generated by podman, they are not specified in my quadlet file.

[bgdnlp]

I'm seeing the same on an Ubuntu 24.04. I don't really have much to add to the above. Mine is doing GPU inference, starts in about a minute if I run it on the command line, but 10 minutes is not enough as a Quadlet setup.

[bgdnlp]

In my case it's not Podman, or Systemd. It's something else, Docker behaves the same way. The container starts, but then at some point it just takes very long to continue. On second run that point is much quicker. But, it's not Podman.