Skip to content

[cmake] Option to create Ninja job pools depending on available resources #65274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 20, 2023
Merged

[cmake] Option to create Ninja job pools depending on available resources #65274

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions llvm/cmake/modules/HandleLLVMOptions.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,21 @@ string(TOUPPER "${LLVM_ENABLE_LTO}" uppercase_LLVM_ENABLE_LTO)
# The following only works with the Ninja generator in CMake >= 3.0.
set(LLVM_PARALLEL_COMPILE_JOBS "" CACHE STRING
"Define the maximum number of concurrent compilation jobs (Ninja only).")
if(LLVM_RAM_PER_COMPILE_JOB OR LLVM_RAM_PER_LINK_JOB)
cmake_host_system_information(RESULT available_physical_memory QUERY AVAILABLE_PHYSICAL_MEMORY)
cmake_host_system_information(RESULT number_of_logical_cores QUERY NUMBER_OF_LOGICAL_CORES)
endif()
if(LLVM_RAM_PER_COMPILE_JOB)
math(EXPR jobs_with_sufficient_memory "${available_physical_memory} / ${LLVM_RAM_PER_COMPILE_JOB}" OUTPUT_FORMAT DECIMAL)
if (jobs_with_sufficient_memory LESS 1)
set(jobs_with_sufficient_memory 1)
endif()
if (jobs_with_sufficient_memory LESS number_of_logical_cores)
set(LLVM_PARALLEL_COMPILE_JOBS "${jobs_with_sufficient_memory}")
else()
set(LLVM_PARALLEL_COMPILE_JOBS "${number_of_logical_cores}")
endif()
endif()
if(LLVM_PARALLEL_COMPILE_JOBS)
if(NOT CMAKE_GENERATOR MATCHES "Ninja")
message(WARNING "Job pooling is only available with Ninja generators.")
Expand All @@ -47,6 +62,17 @@ endif()

set(LLVM_PARALLEL_LINK_JOBS "" CACHE STRING
"Define the maximum number of concurrent link jobs (Ninja only).")
if(LLVM_RAM_PER_LINK_JOB)
math(EXPR jobs_with_sufficient_memory "${available_physical_memory} / ${LLVM_RAM_PER_LINK_JOB}" OUTPUT_FORMAT DECIMAL)
if (jobs_with_sufficient_memory LESS 1)
set(jobs_with_sufficient_memory 1)
endif()
if (jobs_with_sufficient_memory LESS number_of_logical_cores)
set(LLVM_PARALLEL_LINK_JOBS "${jobs_with_sufficient_memory}")
else()
set(LLVM_PARALLEL_LINK_JOBS "${number_of_logical_cores}")
endif()
endif()
if(CMAKE_GENERATOR MATCHES "Ninja")
if(NOT LLVM_PARALLEL_LINK_JOBS AND uppercase_LLVM_ENABLE_LTO STREQUAL "THIN")
message(STATUS "ThinLTO provides its own parallel linking - limiting parallel link jobs to 2.")
Expand Down
13 changes: 13 additions & 0 deletions llvm/docs/CMake.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,19 @@ enabled sub-projects. Nearly all of these variable names begin with
**LLVM_PARALLEL_LINK_JOBS**:STRING
Define the maximum number of concurrent link jobs.

**LLVM_RAM_PER_COMPILE_JOB**:STRING
Calculates the amount of Ninja compile jobs according to available resources.
Value has to be in MB, overwrites LLVM_PARALLEL_COMPILE_JOBS. Compile jobs
will be between one and amount of logical cores.

**LLVM_RAM_PER_LINK_JOB**:STRING
Calculates the amount of Ninja link jobs according to available resources.
Value has to be in MB, overwrites LLVM_PARALLEL_LINK_JOBS. Link jobs will
be between one and amount of logical cores. Link jobs will not run
exclusively therefore you should add an offset of one or two compile jobs
to be sure its not terminated in your memory restricted environment. On ELF
platforms also consider ``LLVM_USE_SPLIT_DWARF`` in Debug build.

**LLVM_PROFDATA_FILE**:PATH
Path to a profdata file to pass into clang's -fprofile-instr-use flag. This
can only be specified if you're building with clang.
Expand Down