Everything on my home server got slow at once. Web pages from it took ages, the house's ad blocker stopped answering, and a command to stop one program sat there for ten minutes doing nothing. The obvious suspect was a big music import I had running that evening. It was innocent.
The real cause was a small app I had barely thought about all day. On a machine with 16 GB of memory, it had grown to 33 GB.
The fix took one line per app, and it is worth doing before you ever see this happen, because during the meltdown your tools are too slow to help you.
What does "running out of memory" actually look like?
Not like a crash. That is what makes it nasty.
A computer has two places to keep the things programs are working on. Memory (RAM) is the desk: small, instant. Swap is the storage room in the basement: big, on the disk, and slow. When the desk is full, the computer carries the least-used papers down to the basement, and carries them back up whenever someone needs them.
That works fine for a few boxes. It falls apart when one program keeps piling paper on the desk. The computer starts spending its time running up and down the stairs instead of doing work. People call this thrashing, and from the outside it looks like this:
- Everything still "works", but slowly, and it gets slower.
- Nothing shows an error. No program has crashed; they are all waiting for the stairs.
- The busiest part of the machine is the disk, not the processor.
My numbers at the worst point, on a four-core machine:
| Measure | Normal | That evening |
|---|---|---|
| Free memory | about 12 GB | 0.4 GB |
| Free swap | 30 GB | 0 |
| Load (programs waiting to run) | under 4 | 110 to 120 |
| Processor time spent waiting for the disk | near 0% | 69% |
A load of 110 on four cores means more than a hundred things queued for a turn. They were all stuck behind the same staircase.
How do you find which program is doing it?
Do not look at what is busy. Look at what is big, and count both places.
My music import was busy and looked guilty. It used almost nothing. The program that had eaten the machine was idle most of the time, which is exactly why I had not looked at it.
On a Linux server, this ranks every process by memory plus swap; the biggest one is your suspect:
for p in /proc/[0-9]*; do
awk '/^VmRSS/{r=$2} /^VmSwap/{s=$2} END{print r+s, r, s}' "$p/status" 2>/dev/null |
sed "s|\$| $(cat $p/comm 2>/dev/null)|"
done | sort -rn | head -5
Mine printed this at the top (numbers in KB: total, in memory, in swap, name):
32773508 6059304 26714204 dotnet
2599520 2583924 15596 Lidarr
1097424 16436 1080988 mariadbd
Six gigabytes on the desk and almost twenty-seven in the basement, for one app. The music manager in second place, at 2.6 GB, was behaving normally.
The swap column is the important one. Most "memory" views only show what is on the desk, so a program that has pushed most of itself into swap looks small. On a laptop the same question is answered by Activity Monitor (Mac) or Task Manager (Windows): sort by memory, and look at "memory pressure" or "committed", not just "in use".
Why couldn't I just stop it?
Because stopping a program needs the program to wake up, and waking it means carrying its 27 GB back up the stairs.
The polite stop timed out. The forced stop never returned either: the container manager that would deliver it was itself stuck in the queue. Then the manager restarted itself and stopped answering altogether. In the end the only way out was a clean shutdown of the whole machine, which took about fifteen minutes because even shutting down has to get through the same staircase.
That is the part I want you to remember. Every repair you might attempt during a memory meltdown runs on the machine that is melting. Whatever limit you want, you have to set it while things are calm.
What is the one setting that stops this?
Give every app a memory limit, with no swap on top of it.
If you run your apps in Docker Compose, that is two lines per service:
services:
book-manager:
image: example/book-manager:1.2.3
mem_limit: 2g # the most memory this app may hold
memswap_limit: 2g # memory + swap together: equal to mem_limit = no swap at all
restart: unless-stopped
The second line is the one people skip, and it matters. With only mem_limit, a
leaking app hits its limit and then starts filling the shared swap instead: the
same staircase, just a slower route to it. With memswap_limit equal to
mem_limit, the app has nowhere else to go. When it reaches 2 GB, the system
stops that one app, and restart: unless-stopped starts it again a few seconds
later, fresh and small.
That changes the failure completely:
| Without limits | With limits |
|---|---|
| The whole machine slows to a crawl | One app restarts |
| No error anywhere | A clear "killed: out of memory" on that one app |
| You cannot stop it | It stops itself |
| Ends with a forced shutdown | Nobody notices |
A crash you can see in a log beats a slow death you can't diagnose.
For apps not in Compose, docker update --memory 1g --memory-swap 1g <name>
does the same on a running container, no restart needed.
How big should each limit be?
About twice what the app normally uses. Generous enough that a busy moment never trips it, tight enough that a leak is caught long before the machine is in trouble.
Measure first: docker stats --no-stream shows every container's current use
next to its limit. Then round up. On my machine that came out as:
| Kind of app | Normal use | Limit |
|---|---|---|
| Music library manager (large catalogue) | 0.5 to 2.6 GB | 4 GB |
| Download tools | under 0.5 GB | 2 GB |
| Most web apps (TV, films, subtitles, ebooks) | 0.1 to 0.4 GB | 1 to 1.5 GB |
| Small helpers (DNS, proxies, scripts) | 10 to 80 MB | 64 to 512 MB |
| The app that leaked | 1.5 GB after a fresh start | 2 GB |
Add up all the limits and the total can be well above your real memory. That is fine: they are ceilings, not reservations. The point is that no single app can take everything.
Then watch for a week. An app that keeps restarting at its limit is either sized too small or leaking. Either way, you now find out from a log line instead of from a frozen house.
Is there anything else to check after the reboot?
Yes, and it caught me. After the restart, the machine came back with every app stopped. The setting that should have started them again ("restart unless stopped") was there; the container manager simply did not act on it within the first several minutes after boot.
The first thing that broke was not media, it was name lookups: my ad blocker also answers "what is the address of this website?" for the whole house, and it was one of the stopped apps. So after any unplanned restart:
- Check that everything is running (
docker ps -a, look for "Exited"). - Start the things the rest of the house depends on first: DNS, then everything else.
- Only then look at whatever caused the restart.
Does this apply if you don't run a server?
The principle does. Every device with many programs sharing memory has this problem, and most of the fixes are already built in:
- Phones do it for you. Android and iOS quietly close background apps when memory runs low. That is exactly a per-app limit, decided by the system.
- Browsers have it as an option. Chrome's "Memory Saver" and Edge's "sleeping tabs" put idle tabs to sleep so one heavy page cannot starve the rest.
- Laptops mostly do not. A desktop app that leaks will happily fill memory and swap. If your laptop gets slower over a day and is fast again after a restart, open Activity Monitor or Task Manager, sort by memory, and look for the one that keeps growing.
The general rule
A limit is a decision you make while things are calm, so you don't have to make it during the crisis. During the crisis you are slow, your tools are slow, and every option costs more than it would have an hour earlier. That holds for memory on a server, for spending on a credit card, and for how long a meeting may run.
What I still don't know
Why the app leaked. My best guess is a feature that kept a live watch on 25,000 files, which I have turned off; it is beta software, and its notes for the next versions mention no memory fix. With the limit in place I can afford to find out slowly: if it leaks again, it restarts, and I get a log line instead of an evening.
I also don't know yet whether every limit is right. They are educated guesses, twice the normal use. A week of measurements will tell me which ones to move.