build: size container build parallelism by container memory, not cores - #10086
Open
natbro wants to merge 1 commit into
Open
build: size container build parallelism by container memory, not cores#10086natbro wants to merge 1 commit into
natbro wants to merge 1 commit into
Conversation
… cores
J defaulted to nproc, which assumes the host's core count is the binding
constraint. In a container it is usually memory: kaldi's nnet2/decoder and
dxvk's d3d11 peak near 1GB of RAM per compiland, so a host with less than
roughly 1GiB per core lets the OOM killer take out compilers at random once
enough heavy translation units overlap.
That failure is hard to read. The kill shows up as
x86_64-linux-gnu-g++: fatal error: Killed signal terminated program cc1plus
and make then reports a plain "Error 1", while "Waiting for unfinished jobs..."
lets the surviving jobs drain -- so the last line before the failure is whatever
recipe happened to finish last, thousands of lines from the real cause.
Ask the container engine how much memory it actually has and use one job per
GiB, capped by the core count, printing ":: limiting to -jN (M GiB container
memory)" when that reduces the count. Hosts with at least 1GiB per core are
unaffected, and an unreachable engine falls back to nproc, so this is a no-op
for a normally provisioned machine. J=N still overrides.
Also warn when -j is passed on a host whose make cannot forward the count:
GNU make 4.x keeps it in MAKEFLAGS, but 3.81 records a bare "-j" and keeps the
count in the jobserver, so -jN reaches the container unbounded.
Signed-off-by: Nat Brown <natbro@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Despite the "macOS" prefix, this change is inspired by building Proton on any host where docker/podman and container tuning defaults might yield lower-memory and lower-cpu-count guests. Constrained guests fail building this large project in unpredictable ways.
J (the number-of-jobs setting) defaulted to nproc, which assumes the host's core count is the binding constraint. In a container it is more often memory: kaldi's nnet2/decoder and dxvk's d3d11 peak near 1GB of RAM per compiland, so a host with less than roughly 1GiB per core lets the OOM killer take out compilers at random once enough heavy translation units randomly overlap.
That failure is hard to read. The kill shows up as
and make then reports a plain "Error 1", while "Waiting for unfinished jobs..." lets the surviving jobs drain -- so the last line before the failure is whatever recipe happened to finish last, thousands of lines from the real cause.
This heuristic instead asks the container engine how much memory it actually has and uses one job per GiB, capped by the core count, printing ":: limiting to -jN (M GiB container memory)" when that reduces the count. Hosts with at least 1GiB per core are unaffected, and an unreachable engine falls back to nproc, so this is a no-op for a normally provisioned machine. J=N still overrides so experienced users who grok OOM fails can continue to parallelize when they know what they're doing.
It also warns when -j is passed on a host whose make cannot forward the count: GNU make 4.x keeps it in MAKEFLAGS, but 3.81 (the ancient macOS default version, future PR about this) records a bare "-j" and keeps the count in the jobserver, so -jN reaches the container unbounded.
If this project's intention is to have builders specify -J themselves, or to be as concise as possible, then feel free to ignore this PR. For novice developers looking to build Proton, whether on Windows, macOS, or a small Linux machine, this heuristic saves quite a bit of spin-up pain at the expense of slowing things down a bit to prevent random OOM failures.