Node.js consuming and reporting heap_size_limit larger than memory available via cgroups v2

[laurisvan]

Version

v26.3.1 (reproduced with latest Docker images for 20.x-26.x)

Platform

Linux dfcddcdef896 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 Linux
Linux 4013555a9d85 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

I noticed strange discrepancy between the memory limits set for node.js containers and the actual values I get for heap_size_limit and total_available_size. I suspected these are the values used for V8 memory management, but I might be wrong.

I would like to get clarity on which values to observe, what to expect, and how to set safe limits so that garbage collection surely works. Most likely the problem is still on our side - just wanted to clear my doubts.

Steps to reproduce:

1. Start a Node.js container with docker run -it -m 256m node:26-bookworm /bin/sh
2. Within the container, check the memory given for the container cat /sys/fs/cgroup/memory.max -> 268435456 (=256 * 1024 * 1024)
3. Start node.js process, examine the heap size node -e "console.log(v8.getHeapStatistics().heap_size_limit)" -> 274726912 (=262 * 1024 * 1024)

How often does it reproduce? Is there a required condition?

Always (the OOM kills and out of memory errors vary, but we get them on daily basis).

What is the expected behavior? Why is that the expected behavior?

The heap size limit should always be smaller than the limit set for the container and reflect the total memory available for V8.

What do you see instead?

The heap size limit is reported higher than the memory limit set for the container.

Additional information

It seems heap_size_limit and total_available_size are computed by some kind of step function, with 274726912 maybe being some kind of lower bound (see below). I am unsure what are the exact memory boundaries V8 would get; and if this stepwise behavior can explain our OOM kills and/or memory allocation failures (V8 killed in garbage collecting phase). We generally would like to keep our Node.js container nimble and well below gigabyte, as we do not need much (e.g. we have containers where we never expect >128Mb being allocated and can still get OOM kills with 256 or larger limits).

The following steps show the behavior with altering memory limits:

for mem in 96 128 256 384 449 512 768 1024 1286 1536; do
    docker run --rm -it -m "${mem}m" node:26-bookworm \
        /usr/local/bin/node -e 'console.log(v8.getHeapStatistics().heap_size_limit)'
done

274726912
274726912
274726912
281018368
281018368
562036736
855638016

for mem in 96 128 256 384 449 512 768 1024 1286 1536; do
    docker run --rm -it -m "${mem}m" node:26-bookworm \
        /usr/local/bin/node -e 'console.log(v8.getHeapStatistics().total_available_size)'
done

274726912
274726912
274726912
281018368
281018368
562036736
724566016
855638016

I repeated the same with the latest Docker images for v26, v24, v22 and v20 branch, and similar results (not the exact same heap size limit but always higher than available) repeats for all of them. The problem suspectedly has always existed? For reference, cat /sys/fs/cgroup/memory.high reports as max. The same problem repeats in Kubernetes environment as well, but I used Docker setup here for the sake of easy repetition.

As for background context, these occasional out of memory errors have haunted us for almost two years already. Previously we suspected it was due to old libuv cgroups handling and later due pointer compression problems. Until recently the problem manifested itself as V8 killing itself due to heap allocation failures when doing garbage collection; and now more recently as OOM kills by Kubernetes. Now that no known issues with libuv cgroups or pointer compression exist, we wanted to see what the metrics reveal. We have tried the usual tricks of limiting old and new space sizes, but to our understanding, now that sizing by cgroups limits should work, this is no longer best practice.

[Archkon]

v8 would have default vaule of allocated young and old generation if you don't use any command line flag to limit

[Archkon]

And the flag be used by one v8 isolate so I don't know whether use worker thread? As one worker thread would be one v8 isolate

[laurisvan]

v8 would have default vaule of allocated young and old generation if you don't use any command line flag to limit

@Archkon do you suggest the heap_size_limit would be set based on defaults of allocated young and old generation, not vice versa? I quickly checked that in this sample case both used_heap_size and total_allocated_bytes were well below the heap_size_limit, so I assume the heap had not inflated by the initial young and old generation...

[Archkon]

v8 would have default vaule of allocated young and old generation if you don't use any command line flag to limit

@Archkon do you suggest the heap_size_limit would be set based on defaults of allocated young and old generation, not vice versa? I quickly checked that in this sample case both used_heap_size and total_allocated_bytes were well below the heap_size_limit, so I assume the heap had not inflated by the initial young and old generation...

That would be expected bahavior as v8 would use default value of young and old generation as hard cap when initialize but not definitely use that completely at all.So you could try using command line flag like --max-heap-size and could refer to other using
node --v8-options | grep "heap" to limit