Running fizzy on docker uses a lot of memory

[locilocisu]

Is anyone else seeing the same behavior where my fizzy docker container would just hold on to the memory it was allocated? It is only when I restart the container then the memory usage is reset.

[c00llin]

Same problem here. I've set a max on the container.

[monorkin]

Hey @locilocisu can you share how much memory your container is using?

[c00llin]

Extra data point from my side ;)
I've got my container capped at 768MB, but before I did that it was eating all memory of my NAS until it died. At that point it was around 1.5GB. I only have 60 cards in fizzy.

[locilocisu]

It was about what @c00llin here reports. I saw it was taking about 1GB plus of ram, and my fizzy was a brand new install, so not even a single card yet. It does not start taking 1GB right away, after several days running it goes up to that and that's when like @c00llin, I applied a max RAM limit on the docker compose config. But it looks like whenever it grabs the RAM, it never lets go, hence just the ever increasing usage.

[estevao]

That's because of Solid Queue. If you are running it for personal use and is ok with disabling it, set 'SOLID_QUEUE_IN_PUMA: "false"' in your docker compose file, or tweak Solid Queue accordingly for your needs... #2343

[locilocisu]

Odd, indeed doesn't do anything for me and I tried fully recreating the container. With or without the setting, I never had issues with the CPU utilization. I'd have spikes of CPU usage, but never a sustained use.

[c00llin]

I tried the variable already as well. It did help, but still leaks, mem goes up more slowly. I also reduced the number of workers from 4 to 2 which also helped but only to postpone the issue. Wrt to CPU, it was taking about 20% of my CPU once the mem filled up.
I've shut it down until it's fixed, since it's draining my swap mem well. Unusable like this.

[estevao]

I was able to make is stable at ~650MB mem and ~4% cpu utilization (mini M1):

Configs:

 RAILS_ENV: "production"
 SOLID_QUEUE_IN_PUMA: "false"
 WEB_CONCURRENCY: 1
 RAILS_MAX_THREADS: 2
 MALLOC_ARENA_MAX: 2

What are you using?

[c00llin]

Now using:
- RAILS_ENV=production
- SOLID_QUEUE_IN_PUMA=false
- WEB_CONCURRENCY=2 #lower value here gives a warning in the log file
- RAILS_MAX_THREADS=2
- MALLOC_ARENA_MAX=2
With these settings mem usage gradually climbs without even using the app, only running the container. When then starting to use the app, mem goes up very high. I then didn't touch it for a while and it still went up until reaching max again. Pretty concerning how much leakage this app has even with a minimal number of workers and the solid queue disabled.

[jirpok]

SOLID_QUEUE_IN_PUMA: "false" means Fizzy won't send the magic link. I couldn't find much in logs as to why that is (could've totally missed it, not a Rails person).

[landure]

After some investigation, it seems that SolidQueue 1.3 with async mode has a reduced memory usage. See rails/solid_queue#330 (comment).

@jzimdars , could you please update Fizzy SolidQueue version ? over 1 Gb RAM use for a self-hosted single user kanban is over the top. Thank you for sharing your tool though.

[steamwings]

I saw as high as 4GiB in use after running for a long time. Two users and <100 cards. CPU was nontrivial as well.

Fizzy does seem to handle capping the memory just fine, which is good.

I was able to bypass this issue more generally by dynamically spinning down fizzy when not in use and starting up the container when a request comes in (using traefik + sablier). The startup time is pretty quick for Rails--I get a responsive page in 6 or 7 seconds after a cold start.

[kevinmcconnell]

For folks seeing high memory consumption, can I ask how many web and job workers you have running in your container?

The default configuration will scale these according to the physical core count of the host. So if you are running on a machine with quite a few cores you might find it's running more workers than you actually need, and as a result, the memory load is high. Each process will end up settling on some baseline memory consumption, and across a lot of processes it can quickly add up. I don't think this is leakage people are seeing; but more likely that the steady state requires a lot of memory.

For comparison, I have a Fizzy container that's been running on a small Digital Ocean droplet for 2 weeks, and it's steady at about 900MB. But that's a single core VPS. If I launch Fizzy on my 32-core desktop machine it quickly reaches around 3GB -- and is also running far more workers than I really need.

If you find this to be the issue you can dial down the workers by setting WEB_CONCURRENCY and JOB_CONCURRENCY to lower values. I'll be looking at ways to make this an easier (and more obvious) lever to reach for, because it's not very obvious right now.

It's hard to have a default configuration that suits every use case. What we have now is more suited to the case where you have plenty of available memory and want to max out Fizzy's capacity on the host. But in the (common) case that you're running Fizzy alongside other workloads it can be too memory-hungry.

It's possible something else is afoot here too! But I'd start with trying to reduce the number of web and job processes and see if that helps.

[locilocisu]

Not a ruby on rails guy, so i presume the solidqueue and puma workers are the job and web workers you're referring to.

My RAM is at 779MB, although I've seen it gone over 1GB if I remove the docker limit. I'm running on a n100, so it has 4 cores. Currently seeing 4 solid queue worker, with one using 140MB and the other using about 70MB each.

Seeing 4 puma workers. The largest RAM usage looks to be from one of the puma worker, taking about 400MB. The other 3 puma worker only takes about 20MB though.

[david-uhlig]

On my AMD R5 3600X (6 cores / 12 threads) system with 32 GB RAM, fizzy spawns 6 workers using the default configuration, resulting in about 4–5 % CPU utilization and around 1.5–1.8 GB of RAM shortly after startup.

When I set WEB_CONCURRENCY=0 and JOB_CONCURRENCY=1, CPU usage stays around 1–2 %, and memory usage drops to roughly 500–600 MB after a day of runtime. I haven’t noticed any negative effects on performance for my single‑user use case.

Thanks @kevinmcconnell for the clear explanation! It would be indeed nice to include this in the documentation.

[flavorjones]

@david-uhlig I would be very thankful if you opened a PR that updates the documentation to give some guidance here.

[david-uhlig]

Fair. Noted in my fizzy. I'll give it a shot, might take a bit, though.