Skip to content

tweak how memory of worker effect number of parallel jobs #142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
KristofferC wants to merge 1 commit into
base: main
Choose a base branch
from
+71 −21
Open

tweak how memory of worker effect number of parallel jobs #142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ Usage: runtests.jl [--help] [--list] [--jobs=N] [TESTS...]
--list List all available tests.
--verbose Print more information during testing.
--quickfail Fail the entire run as soon as a single test errored.
--jobs=N Launch `N` processes to perform tests.
--jobs=N Launch `N` processes to perform tests. Can also be set
with the PARALLELTESTRUNNER_NUM_JOBS environment
variable, with `--jobs=N` taking precedence.

Remaining arguments filter the tests that will be executed.
```
Expand Down
51 changes: 37 additions & 14 deletions src/ParallelTestRunner.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -496,17 +496,26 @@ available_memory() = Sys.free_memory()

end

# This is an internal function, not to be used by end users. The keyword
# arguments are only for testing purposes.
# Assumed memory footprint of a single test worker, used to clamp the default
# number of jobs on memory-constrained machines (e.g. many cores but little
# total memory). Packages whose tests are heavier can pass a larger
# `memory_per_worker` to `runtests`.
const DEFAULT_MEMORY_PER_WORKER = Int64(2)^30

# This is an internal function, not to be used by end users. The
# `cpu_threads` and `total_memory` keyword arguments are only for testing
# purposes.
"""
default_njobs()
default_njobs(; memory_per_worker = 2^30)

Determine default number of parallel jobs.
Determine default number of parallel jobs: the number of CPU threads, clamped
such that each worker can be assumed to use `memory_per_worker` bytes of the
total system memory.
"""
function default_njobs(; cpu_threads = Sys.CPU_THREADS, free_memory = available_memory())
jobs = cpu_threads
memory_jobs = Int64(free_memory) ÷ (2 * Int64(2)^30)
return max(1, min(jobs, memory_jobs))
function default_njobs(; cpu_threads = Sys.CPU_THREADS, total_memory = Sys.total_memory(),
memory_per_worker = DEFAULT_MEMORY_PER_WORKER)
memory_jobs = Int64(total_memory) ÷ Int64(memory_per_worker)
return max(1, min(cpu_threads, memory_jobs))
Comment on lines -506 to +518

@giordano giordano Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why using total memory instead of the actually available one? This would oversubscribe (read: crash very badly) busy machines.

@KristofferC KristofferC Jul 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why using total memory instead of the actually available one?

Many many reasons:

  • Free memory has nothing really to do with how much memory you can actually use. Most computers will cache up most of the RAM by just using the computer for a while (why not, the memory is there to be used), but if an application needs more memory, it drops some of that cache.
  • You only do the free memory sample at the start, if the machine is actually busy, this snapshot is unlikely to mean much during the course of the testing.
  • Using free memory is not deterministic, you hit some memory spike at the point where you do the free memory sample and all of a sudden your CI takes 30 times longer
  • It is arguably better to crash loudly than to have a CI run silently extremely slowly. At that point, you know you have to set some explicit options (which you already have to do if your workers are very memory heavy).
  • If you are on a very large shared machine and want to run heavily parallel tests, you probably should use some queuing system (slurm) instead of just blasting it on the login node.
  • For a package whose purpose is to make your tests go faster by using the power of your machine, optimizing for the uncommon case of running tests on a very strong machine that you apparently can not run heavy stuff on is the wrong default in my opinion. You probably have to set explicit options there anyway.

@giordano giordano Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is arguably better to crash loudly than to have a CI run silently extremely slowly.

Hard disagree on this, because the premise is false. An OOM on Linux isn't a loud crash, it's a job hanging without explanation until it's cancelled because it hits a timeout. Doesn't seem like a great outcome to me.

@KristofferC KristofferC Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So run with 1 worker always by default then if the goal is to maximize the probability of not running out of memory. Anything else can always be argued in the way you are doing now. It's like saying julia -tauto should somehow guess how much memory each thread will use (based on some anecdotal example), look at free memory and do this type of calculation and give you 1 thread today and 5 threads tomorrow etc.

end

# Historical test duration database
Expand Down Expand Up @@ -723,7 +732,9 @@ function parse_args(args; custom::Array{String} = String[])
--list List all available tests.
--verbose Print more information during testing.
--quickfail Fail the entire run as soon as a single test errored.
--jobs=N Launch `N` processes to perform tests."""
--jobs=N Launch `N` processes to perform tests. Can also be set
with the PARALLELTESTRUNNER_NUM_JOBS environment
variable, with `--jobs=N` taking precedence."""

if !isempty(custom)
usage *= "\n\nCustom arguments:"
Expand Down Expand Up @@ -799,7 +810,8 @@ end
env = Vector{Pair{String, String}}(),
stdout = Base.stdout,
stderr = Base.stderr,
max_worker_rss = get_max_worker_rss())
max_worker_rss = get_max_worker_rss(),
memory_per_worker = 2^30)
runtests(mod::Module, ARGS; ...)

Run Julia tests in parallel across multiple worker processes.
Expand Down Expand Up @@ -842,14 +854,21 @@ Several keyword arguments are also supported:
`test_worker` hook are the caller's responsibility.
- `stdout` and `stderr`: I/O streams to write to (default: `Base.stdout` and `Base.stderr`)
- `max_worker_rss`: RSS threshold where a worker will be restarted once it is reached.
- `memory_per_worker`: Assumed memory footprint (in bytes) of a single worker, used to
clamp the default number of jobs on memory-constrained machines (default: 1 GiB).
Packages whose tests use a lot of memory can pass a larger value to reduce the default
parallelism. Ignored when the number of jobs is set explicitly via `--jobs=N` or the
`PARALLELTESTRUNNER_NUM_JOBS` environment variable.

## Command Line Options

- `--help`: Show usage information and exit
- `--list`: List all available test files and exit
- `--verbose`: Print more detailed information during test execution
- `--quickfail`: Stop the entire test run as soon as any test fails
- `--jobs=N`: Use N worker processes (default: based on CPU threads and available memory)
- `--jobs=N`: Use N worker processes (default: based on CPU threads and total memory;
can also be set with the `PARALLELTESTRUNNER_NUM_JOBS` environment variable, with
`--jobs=N` taking precedence)
- `TESTS...`: Filter test files by name, matched using `startswith`

## Behavior
Expand Down Expand Up @@ -922,7 +941,8 @@ function runtests(mod::Module, args::ParsedArgs;
exename = nothing,
exeflags = nothing,
env = Vector{Pair{String, String}}(),
stdout = Base.stdout, stderr = Base.stderr, max_worker_rss = get_max_worker_rss())
stdout = Base.stdout, stderr = Base.stderr, max_worker_rss = get_max_worker_rss(),
memory_per_worker = DEFAULT_MEMORY_PER_WORKER)

#
# set-up
Expand Down Expand Up @@ -962,6 +982,7 @@ function runtests(mod::Module, args::ParsedArgs;
stdout,
stderr,
max_worker_rss,
memory_per_worker,
)
end

Expand All @@ -981,12 +1002,14 @@ function _runtests(mod::Module, args::ParsedArgs;
stdout = Base.stdout,
stderr = Base.stderr,
max_worker_rss = get_max_worker_rss(),
memory_per_worker = DEFAULT_MEMORY_PER_WORKER,
)

# determine parallelism
jobs = something(args.jobs, default_njobs())
env_jobs = tryparse(Int, get(ENV, "PARALLELTESTRUNNER_NUM_JOBS", ""))
jobs = @something args.jobs env_jobs default_njobs(; memory_per_worker)
jobs = clamp(jobs, 1, length(tests))
println(stdout, "Running $(length(tests)) tests using $jobs parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.")
println(stdout, "Running $(length(tests)) tests using $jobs parallel jobs. To change the number of jobs, specify the `--jobs=N` argument to the tests, or set the `PARALLELTESTRUNNER_NUM_JOBS` environment variable.")
!isnothing(args.verbose) && println(stdout, "Available memory: $(Base.format_bytes(available_memory()))")
sem = Base.Semaphore(max(1, jobs))
worker_pool = Channel{Union{Nothing, PTRWorker}}(jobs)
Expand Down
37 changes: 31 additions & 6 deletions test/runtests.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,37 @@ include(joinpath(@__DIR__, "utils.jl"))
end

@testset "default njobs" begin
@test ParallelTestRunner.default_njobs(; cpu_threads=4, free_memory=UInt64(2) ^ 28) == 1
@test ParallelTestRunner.default_njobs(; cpu_threads=4, free_memory=UInt64(2) ^ 30) == 1
@test ParallelTestRunner.default_njobs(; cpu_threads=4, free_memory=UInt64(2) ^ 31) == 1
@test ParallelTestRunner.default_njobs(; cpu_threads=4, free_memory=UInt64(2) ^ 32) == 2
@test ParallelTestRunner.default_njobs(; cpu_threads=4, free_memory=UInt64(2) ^ 33) == 4
@test ParallelTestRunner.default_njobs(; cpu_threads=4, free_memory=UInt64(2) ^ 34) == 4
@test ParallelTestRunner.default_njobs(; cpu_threads=4, total_memory=UInt64(2) ^ 28) == 1
@test ParallelTestRunner.default_njobs(; cpu_threads=4, total_memory=UInt64(2) ^ 30) == 1
@test ParallelTestRunner.default_njobs(; cpu_threads=4, total_memory=UInt64(2) ^ 31) == 2
@test ParallelTestRunner.default_njobs(; cpu_threads=4, total_memory=UInt64(2) ^ 32) == 4
@test ParallelTestRunner.default_njobs(; cpu_threads=4, total_memory=UInt64(2) ^ 33) == 4

# heavier per-worker memory estimate lowers the default
@test ParallelTestRunner.default_njobs(; cpu_threads=4, total_memory=UInt64(2) ^ 32,
memory_per_worker=2 * Int64(2) ^ 30) == 2
end

@testset "number of jobs" begin
testsuite = Dict(
"t1" => :(@test true),
"t2" => :(@test true),
"t3" => :(@test true),
)

# environment variable overrides the default
io = IOBuffer()
withenv("PARALLELTESTRUNNER_NUM_JOBS" => "2") do
runtests(ParallelTestRunner, String[]; testsuite, stdout=io, stderr=io)
end
@test contains(String(take!(io)), "using 2 parallel jobs")

# --jobs takes precedence over the environment variable
io = IOBuffer()
withenv("PARALLELTESTRUNNER_NUM_JOBS" => "2") do
runtests(ParallelTestRunner, ["--jobs=1"]; testsuite, stdout=io, stderr=io)
end
@test contains(String(take!(io)), "using 1 parallel jobs")
end

@testset "subdir use" begin
Expand Down
Loading