Keeping a DGX Spark alive when unified memory runs out

The GB10 in my ASUS Ascent GX10 (the same platform as the NVIDIA DGX Spark) has one pool of memory that the CPU and the GPU share. When vLLM or llama.cpp asks the NVIDIA driver for GPU memory, it comes out of the same pages the kernel needs for itself and for everything else, sshd included. On a machine with a discrete GPU, running out of GPU memory fails one process. On this box it can take the whole machine down.

That happened three times in July. Each time the box stopped answering for 35 to 65 minutes and only came back after a power cycle. This post explains what was happening, why the usual Linux tools did not catch it, and what I run now to prevent it. The scripts are in gb10-hardening on GitHub.

What happened in July

On 2026-07-01 I started Qwen3.5-122B-A10B (NVFP4) in vLLM with --gpu-memory-utilization 0.85. About ten minutes into loading, the kernel log filled with about 920 of these lines in one minute:

NVRM: GPU0 nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] ... _memdescAllocInternal(pMemDesc) @ mem_desc.c:1336

After that the journal went silent for 39 minutes and SSH stopped responding. The same model failed the same way when I relaunched it that afternoon, and Nemotron-3-Super-120B did it again on 07-07 at --gpu-memory-utilization 0.70.

The machine had not crashed. The kernel kept printing hung-task reports on its own clock, so it was running, but every process that needed memory was stuck waiting for it. That state is a reclaim livelock. Nothing on the box was set up to turn it into a reboot (panic=0, hung_task_panic=0, no watchdog), so it stayed that way until I cut the power.

One practical note for anyone debugging the same thing: rsyslog's /var/log/kern.log kept lines that the journal lost, because journald stalled first. Read kern.log before journalctl.

Why nothing stepped in

Two things should have stopped this, and neither did. And the one tool I needed to intervene by hand, SSH, was unusable.

The kernel OOM killer ran, but it killed wireplumber, pipewire, dbus-daemon and a few other small processes. It never picked vLLM, because the task dump listed VLLM::EngineCor at about 32 MB of RSS. The roughly 100 GB that process held through the driver was not counted as its memory.

earlyoom was installed by 07-07 and logged this:

17:20:21 earlyoom: mem avail: 11054 of 124546 MiB ( 8.88%), swap free: 13294 of 16383 MiB (81.14%)
17:21:21 earlyoom: mem avail:     0 of 124546 MiB ( 0.00%), swap free: 13233 of 16383 MiB (80.77%)

It saw zero available memory and did nothing, because my settings were -m 15 -s 5, and earlyoom only acts when memory and swap are both low. GPU memory never goes to swap, so swap stayed 80% free the whole time.

sshd was not OOM-killed either (it already runs at oom_score_adj -1000). It was starved: new logins hung for more than two minutes in uninterruptible sleep, waiting for memory.

Where the memory went

The kernel's OOM report from 07-01 shows the problem in one place:

active_anon:28827 inactive_anon:1132 active_file:803 inactive_file:909 shmem:30076
slab_unreclaimable:402640 free:149983      Free swap = 11559468kB / 16777212kB
Node 0 Normal free:41640kB min:44924kB ... all_unreclaimable

These are counts of 4 KiB pages. Anonymous memory, file cache, shared memory and slab add up to about 2 GB of the 127 GB. The other roughly 118 GB was allocated by the NVIDIA driver for the GPU, outside every list the kernel can reclaim from. Free memory fell below the min watermark, every allocation in the system went into direct reclaim, and there was nothing to reclaim.

Why vLLM got there: --gpu-memory-utilization is a fraction of the full memory the driver reports, and the host needs part of that same pool. The spike right after the weights load (KV cache allocation, the profiling pass, repacking the FP4 weights) took the driver's share to about 118 GB.

How much memory a model actually gets

The box is sold as 128 GB, and it does have 128 GiB of physical memory. Linux reports 121.6 GiB as MemTotal. Adding up /proc/iomem and the kernel's boot log shows where the rest goes:

• 0.37 GiB is never handed to Linux by the firmware.
• 3.55 GiB is reserved by firmware before Linux starts. One block of 2.56 GiB is the largest; /proc/iomem only labels it reserved.
• 2.10 GiB is the struct page array, 64 bytes of bookkeeping for every 4 KiB page.
• About 0.45 GiB is the kernel image and its early boot allocations.

So every fit calculation should start from 121.6 GiB, not 128. The first time this mattered for me was the NVIDIA NVFP4 build of Qwen3.8-Flash-Next: its weights alone are 132.6 GB (123.5 GiB), more than MemTotal, so it cannot load on one box at all.

MemAvailable is wrong on this box

The obvious fix is a watchdog that kills the model when available memory runs low. The problem is that the number most tools read, MemAvailable in /proc/meminfo, is wrong here while vLLM is running. In a controlled over-commit test in October, with vLLM holding about 101 GB after loading:

MemAvailable:  111.5 GiB
MemFree:         7.0 GiB
KReclaimable:  101.6 GB

The driver books vLLM's GPU memory as KReclaimable (kernel memory that can supposedly be reclaimed), and the kernel counts that as available. So free -h, earlyoom and my first version of the guard all treated 100 GB of memory in active use as free. MemAvailable does drop briefly during a fast allocation, which is why earlyoom fired once in that test at 10.46%, but once the allocation settles the number goes back up.

The number I use instead:

TrueAvailable = MemAvailable - KReclaimable + SReclaimable

SReclaimable is the slab part of KReclaimable, which really can be reclaimed, so it is added back. In that test, TrueAvailable was 14.8 GiB, and a fresh SSH login took about 50 seconds.

You can check this on your own machine while a model is loaded:

grep -E 'MemFree|MemAvailable|KReclaimable|SReclaimable' /proc/meminfo

If KReclaimable is tens of GB, your driver is doing the same accounting as mine.

Two more details matter for anything that measures memory on a GB10:

• After vLLM exits, its pages stay in KReclaimable until something writes to /proc/sys/vm/drop_caches. Dropping caches brings KReclaimable from about 100 GB back to 0.1 GB at once. In July the kernel's own reclaim did not free those pages fast enough.
• llama.cpp behaves differently. During a run on 10-19, KReclaimable stayed at 0.3 GiB and the server's memory showed up as ordinary used memory, so MemAvailable was accurate. Its memory only moved to KReclaimable when the process exited. I have not worked out why the two engines are booked differently. TrueAvailable gives the right answer for both.

The guard

The main protection is gb10-memguard, a Python daemon that uses only the standard library. Every 100 ms it reads /proc/meminfo, computes TrueAvailable and its rate of change, and applies three rules:

• If memory is falling fast enough to reach 3 GiB within 0.5 s, SIGKILL immediately.
• If TrueAvailable is below 3 GiB, SIGKILL immediately.
• Once TrueAvailable has spent 2 s below 8 GiB, SIGTERM, then SIGKILL if the process has not exited 10 s later.

The target is the newest Docker container running vLLM, llama-server or Ollama.

Why a rate rule

During a vLLM load the KV cache allocation drains memory at 17 to 41 GiB/s in my tests, and a killed process takes seconds to give memory back. A rule based only on the level has to sit around 20 GiB to survive that, which is what an early version did, and it kept a sixth of the machine unusable. Projecting the current rate 0.5 s ahead catches the ramp early while still letting a steady workload run closer to the floor.

The 0.5 s figure comes from measurement. After a kill, memory kept falling by 0.6 GiB in one test and 14.6 GiB in another, because the driver finishes the allocation already in progress even after the process is dead. That overshoot depends on how much was in flight, so a longer horizon would mostly kill loads that would have been fine.

This rule has a cost I should be clear about: it fires when TrueAvailable - rate * 0.5 s drops under 3 GiB. A vLLM load still allocating at 17 GiB/s is killed once headroom falls below about 11.5 GiB. A config whose KV allocation would finish with 6 GiB free does not survive the load.

Why the slow rule has hysteresis

Replaying 2.5 days of the guard's own recordings through the rules showed a problem with the 8 GiB line: memory crossed it 116 times. A workload sitting at 8.0 to 8.1 GiB crosses it constantly, and every crossing reset the timers, so the 2 s wait and the escalation never ran as designed. Now an episode starts below 8 GiB and only ends above 9 GiB, and the 2 s counts only time actually spent below 8 GiB. A one-second dip during a load adds one second and does nothing. A workload stuck on the line accumulates time and gets stopped. In the replay, the 116 crossings became 9 episodes.

The 10 s between SIGTERM and SIGKILL also comes from logs. Processes that exited cleanly after SIGTERM took between 0.5 and 7.6 s. The ones still running at 10 s only went away after a second SIGTERM, so 10 s is where waiting stops paying off.

How it kills, and why not through Docker

Killing through the Docker API worked in the early tests and failed in the hardest one. Under that much memory pressure dockerd stopped responding, the API call timed out after 3 s, the exception crashed the guard, and systemd restarted it twice while memory went negative. The box recovered only because the kill had already landed.

So the guard never talks to dockerd. It finds targets by reading /sys/fs/cgroup/system.slice/docker-*.scope and each process's command line, and it kills by writing to the container's cgroup (simplified from the repo):

def kill_scope(scope, sig):
    if sig == "KILL":
        with open(os.path.join(scope, "cgroup.kill"), "w") as f:
            f.write("1")   # the kernel kills every process in the cgroup at once

Two other rules came out of failures:

• On 10-19 a llama.cpp server sat below 8 GiB for minutes while the guard logged "no vLLM container found", because the target match was only vllm. earlyoom killed it in the end. The match now includes llama-server and ollama, and if nothing matches when a SIGKILL is due, the guard kills the container with the largest resident memory (at least 8 GiB).
• If a killed container has not disappeared 20 s later and memory is still below 3 GiB, the guard moves on to the next target instead of waiting.

Cleaning up after a kill

An early version dropped caches 3 s after a kill. The killed vLLM took about 10 s to exit, so the freed memory landed in KReclaimable after the drop and about 114 GB stayed stranded. The guard now waits until the killed container's cgroup is empty before dropping caches. Every 5 s it also checks for stranded driver memory when no GPU workload is running and frees it.

Keeping the guard itself alive

The guard locks all of its memory with mlockall() (about 30 MiB), runs at OOMScoreAdjust=-1000 and Nice=-10, and is restarted by systemd if it dies. Every decision goes to an fsync'd log file, the journal and /dev/kmsg, so the record survives even if journald stalls. It also writes a 1 Hz CSV of memory and pressure stall information (PSI), about 80 MB for two weeks with rotation.

Results

With the guard running, an 82 GB over-commit was killed during the ramp at 19.4 GiB, memory bottomed at 18.8 GiB, and the box recovered in 4 s. A fresh SSH login during the ramp was instant. A 95 GB overshoot bottomed at 3.6 GiB and the box recovered on its own, with 2 NVRM error lines in total. A 75 GB model then served for 4.1 hours at 12 to 21 GiB of headroom with no guard actions. Those tests ran with an older 4 GiB floor and a 1 GiB kernel reserve, which I explain below.

The other layers

The guard is the main defense. The other layers are there for when it is late: they keep SSH usable and the kernel able to make progress, and they make sure the logs reach disk. apply.sh in the repo applies them in numbered sections.

sshd

Ubuntu starts sshd through socket activation, so if sshd is not already running, the first connection has to start it, and a memory stall is the worst moment to need a new process. apply.sh keeps ssh.service running all the time and gives it and journald this drop-in:

[Service]
OOMScoreAdjust=-1000
MemoryMin=256M
CPUWeight=1000
IOWeight=1000

MemoryMin keeps their pages in memory, and the weights let them run while everything else is waiting on reclaim. user.slice gets MemoryLow=1G for interactive shells. These protections only cover memory a unit is already using, so they cost nothing when it is idle.

Kernel settings

vm.min_free_kbytes = 262144
vm.watermark_boost_factor = 0
kernel.sysrq = 1
kernel.hung_task_timeout_secs = 120
kernel.hung_task_warnings = -1

min_free_kbytes sets the reserve the kernel keeps for its own allocations, about 45 MB by default. I first set it to 1 GiB, then measured what that cost: the kernel keeps free memory near the high watermark (min plus about 0.25 GiB here), and MemAvailable subtracts it, so 1 GiB of reserve took about 1.2 GiB away from models. The reserve was never touched in testing. Even when TrueAvailable went negative, MemFree stayed at 3.6 GiB. 256 MiB is still six times the default, and it gives back about 1 GiB.

watermark_boost_factor = 0 stops kswapd from temporarily evicting extra page cache (up to 1.5 times the high watermark) when memory is fragmented. That cache includes memory-mapped model weights, which then have to be read from disk again. sysrq = 1 makes memory dumps and a manual OOM kill (echo f > /proc/sysrq-trigger) available from a shell that still works.

earlyoom as a backstop

-r 60 -m 5,3 -s 100,100 -g --prefer '^(vllm|VLLM::.*|llama-server|ollama)$'

-s 100,100 removes the swap condition that kept it idle in July. The 5% threshold sits below the guard's, because at 15% it killed a healthy model whose normal steady state was 14.86% free. It reads MemAvailable, so it only catches fast ramps and llama.cpp-style memory, which makes it a last line rather than a first one.

Automatic recovery

apply.sh --auto-recover adds:

kernel.hung_task_panic = 1
kernel.softlockup_panic = 1
kernel.panic_on_oops = 1
kernel.panic = 20

and RuntimeWatchdogSec=60s for the hardware watchdog. If a livelock gets past everything else, a task stuck for 120 s now causes a panic and a reboot 20 s later, instead of waiting for someone to cut the power. I keep it on, at least while the box runs unattended benchmarks.

The smaller pieces

journald syncs every 5 s instead of every 5 minutes, so the minutes before a hang reach disk, and the journal is capped at 2 GB. apply.sh --trim disables NVIDIA's DGX Dashboard, which held about 380 MiB; fwupd and snapd stay, one for firmware updates and the other because Tailscale is a snap. Measured against a stock install, the whole kit costs about 0.2 GiB of kernel reserve plus 36 MiB for the daemons, and the trim gives back more than that. The real cost is the 8 GiB rule: a model has to leave that much free once it is running.

The SSH client side

The server settings only help if the client does not give up first. With ServerAliveInterval 15, the default ServerAliveCountMax 3 drops the connection after 45 s without a reply, and with SSH multiplexing that takes the shared connection with it. A stall can last longer than that. I added:

Host gx10
  ServerAliveInterval 15
  ServerAliveCountMax 12
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 24h

Logged-in SSH sessions run in user.slice, not under ssh.service. A benchmark started from a shell shares that slice, and MemoryLow=1G mostly ends up protecting the benchmark. Starting long jobs with systemd-run --unit=<name> puts them in system.slice instead.

Caveats

The 3 GiB floor and 256 MiB reserve have not been through the 95 GB over-commit test yet. That test is next.

The rate rule means a fast vLLM load has to finish with more than about 11.5 GiB free. Fixing that properly would mean telling the guard how much the load will allocate (vLLM's --kv-cache-memory-bytes is known in advance) instead of projecting from the rate.

The 8 GiB rule stops a workload that would have been stable at 6 GiB. The guard can make that rule require a measured memory stall first (MEMGUARD_SOFT_NEEDS_PSI=1), but I have left it off until I have a week of PSI data showing what stalling looks like on real workloads.

Only Docker containers are guarded. A native llama-server or anything inside a VM is not seen by the target scan, and with several GPU containers running, the newest one is killed first.

Some behavior is measured but not explained. In July the OOM killer saw vLLM at 32 MB of RSS; in October, on a newer driver (595.99.02 instead of 595.71.05) and a newer vLLM, earlyoom saw the same process at about 100 GB. I have not tested what changed. The llama.cpp difference above is in the same category.

The one rule I would pass on: on a GB10, do not trust MemAvailable or the "available" column of free. If you have one of these machines on a different driver and see KReclaimable behave differently, I would like to hear about it.