Skip to content

Commit ebec54f

Browse files
aaronmondalaaronmondal
committed
[Bazel] Add support for Bzlmod
The workspace build is now gated behind `--config=deprecated_workspace`. While not necessarily the most intuitive implementation, I'm fairly confident that this is able to accomodate all usecases I can think of. The main challenge is that we need to support custom checkouts, a practice that bzlmod doesn't really support yet. The new setup supports the following usecases: - The llvm-project in a directory next to "myrepo" (e.g. for local development and testing. Also very interesting for Nix users as it allows referencing nix store paths). - The llvm-project as a submodule in "myrepo" (e.g. IREE needs this) - The llvm-project fetched from github (e.g. Carbon and rules_ll currently implement elaborate custom patch logic for this). See `utils/bazel/examples/bzlmod` for the new setup instructions. The configuration options for zlib-ng and zstd have been moved to accomodate both the workspace and the module build: ```bash --@llvm-project-overlay//:llvm_enable_zstd --@llvm-project-overlay//:llvm_enable_zlib --@llvm-project//llvm:enable_zstd --@llvm-project//llvm:enable_zlib ``` This fixes a subtle issue wehere the way that we used `zlib` and `zstd` was technically incorrect. Our existing implementation relied on custom defines exposed by build files. This bled llvm-specific configuration into what were supposed to be independent third-party dependencies. In practice this means that the `//llvm:Support` and `//lld:ELF` targets now explicitly set `LLVM_ZLIB_ENABLE` and `LLVM_ZLIB_ENABLE`. Future changes may now integrate these with the `configure.bzl` logic. The module build pulls these dependencies from the BCR. The workspace build will continue to use the vendored variants and we'll remove them once we remove workspace support. For the module build, users that need to link against nonhermetic system variants of these libraries should use custom extensions in conjunction with [overrides](https://bazel.build/external/module#overrides) in their `MODULE.bazel`, for instance like so: ```python load( "@bazel_tools//tools/build_defs/repo:local.bzl", "new_local_repository" ) def _system_zlib_impl(ctx): new_local_repository( name = "zlib-ng" build_file = "@myrepo//third_party//zlib/system:BUILD.bazel", path = "/path/to/local/zlib", ) system_zlib = module_extension(implementation = _system_zlib_impl) ``` ```python zlib = use_extension( "//third_party/zlib/system:extension.bzl", "system_zlib" ) use_repo( zlib, "zlib-ng", ) ```
1 parent a3799f2 commit ebec54f

File tree

15 files changed

+1109
-151
lines changed

15 files changed

+1109
-151
lines changed

‎utils/bazel/.bazelrc

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,9 @@
99

1010
# Flip off to disable MODULE.bazel until we're ready.
1111
# https://github.com/llvm/llvm-project/issues/55924
12-
common --enable_bzlmod=false --enable_workspace
12+
#
13+
# WARNING: This option will be removed soon. Please migrate to bzlmod.
14+
common:deprecated_workspace --enable_bzlmod=false --enable_workspace
1315

1416
# TODO: Remove lit test reliance on this
1517
common --legacy_external_runfiles
@@ -58,14 +60,6 @@ build --experimental_cc_shared_library
5860
# https://github.com/bazelbuild/bazel/commit/03246077f948f2790a83520e7dccc2625650e6df
5961
build --build_runfile_links=false
6062

61-
###############################################################################
62-
# Options to select different strategies for linking potential dependent
63-
# libraries. The default leaves it disabled.
64-
###############################################################################
65-
66-
build:zlib_external --repo_env=BAZEL_LLVM_ZLIB_STRATEGY=external
67-
build:zlib_system --repo_env=BAZEL_LLVM_ZLIB_STRATEGY=system
68-
6963
###############################################################################
7064
# Options for "generic_clang" builds: these options should generally apply to
7165
# builds using a Clang-based compiler, and default to the `clang` executable on

‎utils/bazel/.bazelversion

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
8.0.0
1+
8.0.1

‎utils/bazel/MODULE.bazel

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# This file is licensed under the Apache License v2.0 with LLVM Exceptions.
2+
# See https://llvm.org/LICENSE.txt for license information.
3+
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
4+
5+
module(
6+
name = "llvm-project-overlay",
7+
version = "main",
8+
compatibility_level = 0,
9+
)
10+
11+
bazel_dep(name = "bazel_skylib", version = "1.7.1")
12+
bazel_dep(name = "platforms", version = "0.0.11")
13+
bazel_dep(name = "rules_cc", version = "0.0.17")
14+
bazel_dep(name = "rules_python", version = "1.1.0")
15+
bazel_dep(name = "apple_support", version = "1.17.1")
16+
bazel_dep(name = "rules_foreign_cc", version = "0.13.0")
17+
bazel_dep(name = "robin-map", version = "1.3.0")
18+
bazel_dep(name = "zlib-ng", version = "2.0.7")
19+
bazel_dep(name = "zstd", version = "1.5.6")
20+
21+
llvm_project_overlay = use_extension(
22+
"//:extensions.bzl",
23+
"llvm_project_overlay",
24+
)
25+
26+
# Don't add targets here. An empty list resolves to "all targets" and allows
27+
# downstream repos to configure subsets.
28+
llvm_project_overlay.configure()
29+
30+
use_repo(
31+
llvm_project_overlay,
32+
"llvm-project",
33+
"llvm-raw",
34+
35+
# Vendored.
36+
"vulkan_headers",
37+
"vulkan_sdk",
38+
"gmp",
39+
"mpfr",
40+
"pfm",
41+
"pybind11",
42+
"nanobind",
43+
)

‎utils/bazel/MODULE.bazel.lock

Lines changed: 766 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎utils/bazel/WORKSPACE

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ maybe(
4343

4444
maybe(
4545
http_archive,
46-
name = "llvm_zlib",
46+
name = "zlib-ng",
4747
build_file = "@llvm-raw//utils/bazel/third_party_build:zlib-ng.BUILD",
4848
sha256 = "e36bb346c00472a1f9ff2a0a4643e590a254be6379da7cddd9daeb9a7f296731",
4949
strip_prefix = "zlib-ng-2.0.7",
@@ -71,13 +71,13 @@ maybe(
7171
)
7272

7373
http_archive(
74-
name = "build_bazel_apple_support",
74+
name = "apple_support",
7575
sha256 = "c4bb2b7367c484382300aee75be598b92f847896fb31bbd22f3a2346adf66a80",
7676
url = "https://github.com/bazelbuild/apple_support/releases/download/1.15.1/apple_support.1.15.1.tar.gz",
7777
)
7878

7979
load(
80-
"@build_bazel_apple_support//lib:repositories.bzl",
80+
"@apple_support//lib:repositories.bzl",
8181
"apple_support_dependencies",
8282
)
8383

@@ -137,7 +137,7 @@ maybe(
137137

138138
maybe(
139139
http_archive,
140-
name = "llvm_zstd",
140+
name = "zstd",
141141
build_file = "@llvm-raw//utils/bazel/third_party_build:zstd.BUILD",
142142
sha256 = "7c42d56fac126929a6a85dbc73ff1db2411d04f104fae9bdea51305663a83fd0",
143143
strip_prefix = "zstd-1.5.2",
@@ -157,7 +157,7 @@ maybe(
157157

158158
maybe(
159159
http_archive,
160-
name = "robin_map",
160+
name = "robin-map",
161161
build_file = "@llvm-raw//utils/bazel/third_party_build:robin_map.BUILD",
162162
sha256 = "a8424ad3b0affd4c57ed26f0f3d8a29604f0e1f2ef2089f497f614b1c94c7236",
163163
strip_prefix = "robin-map-1.3.0",

‎utils/bazel/configure.bzl

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,10 +31,10 @@ DEFAULT_TARGETS = [
3131
]
3232

3333
def _overlay_directories(repository_ctx):
34-
src_path = repository_ctx.path(Label("@llvm-raw//:WORKSPACE")).dirname
35-
bazel_path = src_path.get_child("utils").get_child("bazel")
36-
overlay_path = bazel_path.get_child("llvm-project-overlay")
37-
script_path = bazel_path.get_child("overlay_directories.py")
34+
workspace_path = repository_ctx.path(Label("@llvm-raw//utils/bazel:WORKSPACE")).dirname
35+
src_path = workspace_path.get_child("../..")
36+
overlay_path = workspace_path.get_child("llvm-project-overlay")
37+
script_path = workspace_path.get_child("overlay_directories.py")
3838

3939
python_bin = repository_ctx.which("python3")
4040
if not python_bin:
@@ -81,7 +81,7 @@ def _extract_cmake_settings(repository_ctx, llvm_cmake):
8181

8282
# It would be easier to use external commands like sed(1) and python.
8383
# For portability, the parser should run on Starlark.
84-
llvm_cmake_path = repository_ctx.path(Label("//:" + llvm_cmake))
84+
llvm_cmake_path = repository_ctx.path(Label("@llvm-raw//:" + llvm_cmake))
8585
for line in repository_ctx.read(llvm_cmake_path).splitlines():
8686
# Extract "set ( FOO bar ... "
8787
setfoo = line.partition("(")

‎utils/bazel/extensions.bzl

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# This file is licensed under the Apache License v2.0 with LLVM Exceptions.
2+
# See https://llvm.org/LICENSE.txt for license information.
3+
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
4+
5+
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
6+
load("@bazel_tools//tools/build_defs/repo:local.bzl", "new_local_repository")
7+
load(":vulkan_sdk.bzl", "vulkan_sdk_setup")
8+
load(":configure.bzl", "llvm_configure", "DEFAULT_TARGETS")
9+
10+
def _llvm_configure_extension_impl(ctx):
11+
targets = []
12+
13+
# Aggregate targets across imports.
14+
targets.extend([
15+
target
16+
for module in ctx.modules
17+
for config in module.tags.configure
18+
for target in config.targets
19+
if target not in targets
20+
])
21+
22+
# Fall back to the default targets if all configurations of this extension
23+
# omit the `target` attribute.
24+
if targets == []:
25+
targets = DEFAULT_TARGETS
26+
27+
llvm_configure(name = "llvm-project", targets = targets)
28+
29+
# Deliberately omit the "llvm-raw" directory if we're not in the utils/bazel
30+
# directory.
31+
#
32+
# In downstream repos this intentionally causes the extension to immediately
33+
# error out if "llvm-raw" wasn't injected explicitly.
34+
#
35+
# We can't add this repo to the utils/bazel/MODULE.bazel as it would cause
36+
# submodule imports to resolve the new_local_repository at wrong paths.
37+
[
38+
new_local_repository(
39+
name = "llvm-raw",
40+
path = "../..",
41+
build_file_content = "# Empty."
42+
)
43+
for module in ctx.modules
44+
if module.name == "llvm-project-overlay" and module.is_root
45+
]
46+
47+
print(ctx.path(Label("@llvm-raw//:BUILD.bazel")))
48+
49+
http_archive(
50+
name = "vulkan_headers",
51+
build_file = "@llvm-raw//utils/bazel/third_party_build:vulkan_headers.BUILD",
52+
sha256 = "19f491784ef0bc73caff877d11c96a48b946b5a1c805079d9006e3fbaa5c1895",
53+
strip_prefix = "Vulkan-Headers-9bd3f561bcee3f01d22912de10bb07ce4e23d378",
54+
urls = [
55+
"https://github.com/KhronosGroup/Vulkan-Headers/archive/9bd3f561bcee3f01d22912de10bb07ce4e23d378.tar.gz",
56+
],
57+
)
58+
59+
vulkan_sdk_setup(name = "vulkan_sdk")
60+
61+
http_archive(
62+
name = "gmp",
63+
build_file = "@llvm-raw//utils/bazel/third_party_build:gmp.BUILD",
64+
sha256 = "fd4829912cddd12f84181c3451cc752be224643e87fac497b69edddadc49b4f2",
65+
strip_prefix = "gmp-6.2.1",
66+
urls = [
67+
"https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz",
68+
"https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz",
69+
],
70+
)
71+
72+
# https://www.mpfr.org/mpfr-current/
73+
#
74+
# When updating to a newer version, don't use URLs with "mpfr-current" in them.
75+
# Instead, find a stable URL like the one used currently.
76+
http_archive(
77+
name = "mpfr",
78+
build_file = "@llvm-raw//utils/bazel/third_party_build:mpfr.BUILD",
79+
sha256 = "9cbed5d0af0d9ed5e9f8dd013e17838eb15e1db9a6ae0d371d55d35f93a782a7",
80+
strip_prefix = "mpfr-4.1.1",
81+
urls = ["https://www.mpfr.org/mpfr-4.1.1/mpfr-4.1.1.tar.gz"],
82+
)
83+
84+
http_archive(
85+
name = "pfm",
86+
build_file = "@llvm-raw//utils/bazel/third_party_build:pfm.BUILD",
87+
sha256 = "d18b97764c755528c1051d376e33545d0eb60c6ebf85680436813fa5b04cc3d1",
88+
strip_prefix = "libpfm-4.13.0",
89+
urls = ["https://versaweb.dl.sourceforge.net/project/perfmon2/libpfm4/libpfm-4.13.0.tar.gz"],
90+
)
91+
92+
http_archive(
93+
name = "pybind11",
94+
build_file = "@llvm-raw//utils/bazel/third_party_build:pybind.BUILD",
95+
sha256 = "201966a61dc826f1b1879a24a3317a1ec9214a918c8eb035be2f30c3e9cfbdcb",
96+
strip_prefix = "pybind11-2.10.3",
97+
url = "https://github.com/pybind/pybind11/archive/v2.10.3.zip",
98+
)
99+
100+
# TODO: Make `NB_BUILD=1` and `NB_SHARED=1` configurable in the BCR variant
101+
# and import this via `MODULE.bazel`.
102+
http_archive(
103+
name = "nanobind",
104+
build_file = "@llvm-raw//utils/bazel/third_party_build:nanobind.BUILD",
105+
sha256 = "bb35deaed7efac5029ed1e33880a415638352f757d49207a8e6013fefb6c49a7",
106+
strip_prefix = "nanobind-2.4.0",
107+
url = "https://github.com/wjakob/nanobind/archive/refs/tags/v2.4.0.tar.gz",
108+
)
109+
110+
111+
llvm_project_overlay = module_extension(
112+
doc = """Configure the llvm-project.
113+
114+
Tags:
115+
targets: List of targets which Clang should support.
116+
""",
117+
implementation = _llvm_configure_extension_impl,
118+
tag_classes = {
119+
"configure": tag_class(
120+
attrs = {
121+
"targets": attr.string_list(),
122+
},
123+
),
124+
},
125+
)

‎utils/bazel/llvm-project-overlay/lld/BUILD.bazel

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,17 @@ cc_library(
8585
textual_hdrs = [
8686
"ELF/Arch/PPCInsns.def",
8787
],
88+
defines = select({
89+
"//llvm:zlib_enabled": [
90+
"LLVM_ENABLE_ZLIB=1",
91+
],
92+
"//conditions:default": [],
93+
}) + select({
94+
"//llvm:zstd_enabled": [
95+
"LLVM_ENABLE_ZSTD=1",
96+
],
97+
"//conditions:default": [],
98+
}),
8899
deps = [
89100
":Common",
90101
":elf_options_inc_gen",
@@ -111,9 +122,17 @@ cc_library(
111122
"//llvm:TargetParser",
112123
"//llvm:TransformUtils",
113124
"//llvm:config",
114-
"@llvm_zlib//:zlib",
115-
"@llvm_zstd//:zstd",
116-
],
125+
] + select({
126+
"//llvm:zlib_enabled": [
127+
"@zlib-ng",
128+
],
129+
"//conditions:default": [],
130+
}) + select({
131+
"//llvm:zstd_enabled": [
132+
"@zstd",
133+
],
134+
"//conditions:default": [],
135+
}),
117136
)
118137

119138
gentbl(

‎utils/bazel/llvm-project-overlay/lldb/BUILD.bazel

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
load("@bazel_skylib//lib:selects.bzl", "selects")
66
load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
77
load("@bazel_skylib//rules:expand_template.bzl", "expand_template")
8-
load("@build_bazel_apple_support//rules:apple_genrule.bzl", "apple_genrule")
8+
load("@apple_support//rules:apple_genrule.bzl", "apple_genrule")
99
load("@rules_python//python:defs.bzl", "py_binary")
1010
load("//:vars.bzl", "LLVM_VERSION_MAJOR", "LLVM_VERSION_MINOR", "LLVM_VERSION_PATCH", "LLVM_VERSION_SUFFIX", "PACKAGE_VERSION")
1111
load("//lldb/source/Plugins:plugin_config.bzl", "DEFAULT_PLUGINS", "DEFAULT_SCRIPT_PLUGINS", "OBJCPP_COPTS")

‎utils/bazel/llvm-project-overlay/lldb/source/Plugins/BUILD.bazel

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -476,6 +476,12 @@ cc_library(
476476
"Process/gdb-remote/GDBRemoteErrno.def",
477477
],
478478
includes = [".."],
479+
defines = select({
480+
"//llvm:zlib_enabled": [
481+
"LLVM_ENABLE_ZLIB=1",
482+
],
483+
"//conditions:default": [],
484+
}),
479485
deps = [
480486
":PluginProcessUtility",
481487
":ProcessGDBRemoteProperties",
@@ -493,8 +499,12 @@ cc_library(
493499
"//llvm:Support",
494500
"//llvm:TargetParser",
495501
"//llvm:config",
496-
"@llvm_zlib//:zlib",
497-
],
502+
] + select({
503+
"//llvm:zlib_enabled": [
504+
"@zlib-ng",
505+
],
506+
"//conditions:default": [],
507+
})
498508
)
499509

500510
cc_library(
@@ -1405,6 +1415,12 @@ cc_library(
14051415
srcs = glob(["SymbolFile/CTF/*.cpp"]),
14061416
hdrs = glob(["SymbolFile/CTF/*.h"]),
14071417
includes = [".."],
1418+
defines = select({
1419+
"//llvm:zlib_enabled": [
1420+
"LLVM_ENABLE_ZLIB=1",
1421+
],
1422+
"//conditions:default": [],
1423+
}),
14081424
deps = [
14091425
":PluginExpressionParserClangHeaders",
14101426
":PluginTypeSystemClangHeaders",
@@ -1416,8 +1432,13 @@ cc_library(
14161432
"//lldb:Utility",
14171433
"//llvm:Support",
14181434
"//llvm:config",
1419-
"@llvm_zlib//:zlib",
1420-
],
1435+
] + select({
1436+
"//llvm:zlib_enabled": [
1437+
"@zlib-ng",
1438+
],
1439+
"//conditions:default": [],
1440+
})
1441+
14211442
)
14221443

14231444
cc_library(

0 commit comments

Comments
 (0)