|
| 1 | +--- |
| 2 | +name: bazel |
| 3 | +description: Conventions for editing Bazel files in the github/codeql repository: the shared `misc/bazel` helpers, `codeql_platform_select` and the `{CODEQL_PLATFORM}` packaging placeholder, adding `MODULE.bazel` dependencies, and the `semmle_code` stub that keeps the standalone build working. Use when editing any `BUILD.bazel`, `*.bzl`, `MODULE.bazel` or `.bazelrc` here, and before validating such an edit. |
| 4 | +--- |
| 5 | + |
| 6 | +# Bazel in the codeql repository |
| 7 | + |
| 8 | +A bzlmod module named `ql`, repo name `@codeql` ([`MODULE.bazel`](../../../MODULE.bazel)). It builds standalone, and is |
| 9 | +also consumed by an internal module; standalone builds replace that module with a stub. |
| 10 | + |
| 11 | +## Traps |
| 12 | + |
| 13 | +Things that will waste your time or produce a wrong edit here. Read this section even if you skip the rest. |
| 14 | + |
| 15 | +* **`//...` does not work.** `bazel build //...`, and even `bazel query //...`, fail at the repo root: the patched |
| 16 | + modules under [`misc/bazel/registry`](../../../misc/bazel/registry) are real packages referencing repos that are not |
| 17 | + visible from the main repo, and [`.bazelrc`](../../../.bazelrc) notes separately that transitions break `...` builds. |
| 18 | + The error names a registry directory unrelated to your edit, so it is easy to misdiagnose. **Validate the specific |
| 19 | + target or package you changed**, not a recursive pattern. |
| 20 | +* **There is no `MODULE.bazel.lock`, deliberately.** [`.bazelrc`](../../../.bazelrc) sets `--lockfile_mode=off` because |
| 21 | + the workspace-relative module override makes a lockfile unstable. Do not add one, and do not "fix" its absence. |
| 22 | +* **`linux_arm64` vs `linux-arm64`.** The keyword argument and config setting use an underscore; the platform *string* |
| 23 | + substituted into paths and zip names uses a hyphen. They are not interchangeable. |
| 24 | +* **Do not stub your way out of a missing internal dependency.** See [Building standalone](#building-standalone) for the |
| 25 | + one narrow case where extending the stub is correct. |
| 26 | + |
| 27 | +## Conventions |
| 28 | + |
| 29 | +* **Copy a neighboring target rather than inventing a shape.** Packaging is not uniform: some packages use the |
| 30 | + `codeql_*` wrappers, others still use `pkg_files` directly. Copy the closest *working* neighbor, and prefer the |
| 31 | + wrapper for new code. |
| 32 | +* **Pin anything fetched over the network** with `sha256` or `integrity`. Bazel only *warns* on an unpinned download, so |
| 33 | + nothing fails loudly, but the build stops being reproducible and a retagged upstream release silently changes what you |
| 34 | + build. `lfs_archive` is the exception: content is pinned by git object. |
| 35 | +* **Format with `bazel run //misc/bazel/buildifier`.** It rewrites in place, so do not hand-tune formatting. Also wired |
| 36 | + as a `pre-commit` hook ([`.pre-commit-config.yaml`](../../../.pre-commit-config.yaml)). |
| 37 | + |
| 38 | +## Where new code goes |
| 39 | + |
| 40 | +Bazel's own macro / rule / repository-rule distinction applies as usual. What is repo-specific: |
| 41 | + |
| 42 | +| Adding | Goes in | |
| 43 | +| --- | --- | |
| 44 | +| a new packaging shape | extend [`misc/bazel/pkg.bzl`](../../../misc/bazel/pkg.bzl), do not fork `pkg_files` | |
| 45 | +| a new OS or arch split | [`misc/bazel/os.bzl`](../../../misc/bazel/os.bzl), do not hand-roll a `select()` over `@platforms//` | |
| 46 | +| a fetch of something external | a repository rule ([`lfs.bzl`](../../../misc/bazel/lfs.bzl), [`ripunzip.bzl`](../../../misc/ripunzip/ripunzip.bzl)), not a `genrule` | |
| 47 | +| a wrapper used by one language | next to that language ([`swift/rules.bzl`](../../../swift/rules.bzl)) | |
| 48 | +| a wrapper used across languages | `misc/bazel/` | |
| 49 | + |
| 50 | +Prefer inline rules in `BUILD.bazel`. A `.bzl` file earns its `load()` only when the shape repeats across packages or a |
| 51 | +value must be computed: [`rust.bzl`](../../../misc/bazel/rust.bzl) is worth it because every Rust binary that ships in a |
| 52 | +pack must get the same universal-binary wrapper and symbols test, and forgetting either is a release bug. A local |
| 53 | +debugging aid opts out and declares a plain `rust_binary`; see `swift-syntax-parse` in |
| 54 | +[`unified/swift-syntax-rs/BUILD.bazel`](../../../unified/swift-syntax-rs/BUILD.bazel). The `_gen_binaries` list in |
| 55 | +[`go/BUILD.bazel`](../../../go/BUILD.bazel) does not earn a `.bzl`, because it is shared within a single file, where a |
| 56 | +local variable does the job. |
| 57 | + |
| 58 | +Macros here are typically a thin public wrapper around a private rule (`codeql_csharp_binary`, `swift_cc_binary`). Keep |
| 59 | +the rule narrow and the ergonomics in the macro. Each macro decorates the caller's `name` to mint its helper targets, |
| 60 | +for example `internal/<name>` or `single_arch/<name>`. Their visibility is that macro's choice (private, package |
| 61 | +default, or the caller's own), so read the macro instead of assuming. When an error names a target you cannot find in |
| 62 | +any source file, a macro minted it: grep the suffix under `misc/bazel/`. |
| 63 | + |
| 64 | +## Shared helpers |
| 65 | + |
| 66 | +Frequently-used pieces, so you load the existing one instead of rewriting it. Read the file for its actual exports. |
| 67 | + |
| 68 | +| `load()` path | Covers | |
| 69 | +| --- | --- | |
| 70 | +| `//misc/bazel:pkg.bzl` | CodeQL packs and packaging | |
| 71 | +| `//misc/bazel:os.bzl` | platform and architecture selection | |
| 72 | +| `//misc/bazel:lfs.bzl` | on-demand git-LFS repositories | |
| 73 | +| `//misc/bazel:rust.bzl` | Rust binary wrapper | |
| 74 | +| `//misc/bazel:csharp.bzl` | C# binary/library/test wrappers | |
| 75 | +| `//misc/bazel:utils.bzl` | `select_os`; prefer `os.bzl`'s `os_select` in new code | |
| 76 | + |
| 77 | +[`defs.bzl`](../../../defs.bzl) at the root exports `codeql_platform` for *dependent* modules. It is not the way to get |
| 78 | +the platform string here; use `os.bzl`. |
| 79 | + |
| 80 | +## Platform selection |
| 81 | + |
| 82 | +[`codeql_platform_select`](../../../misc/bazel/os.bzl) takes one keyword argument per CodeQL platform: `linux64`, |
| 83 | +`linux_arm64`, `osx64` and `win64`. `otherwise` supplies the value for whichever of those you leave unset; it is **not** |
| 84 | +a `//conditions:default`. **There is deliberately no fallback from `linux_arm64` to `linux64`.** If you only care about |
| 85 | +the OS, use `os_select`, which gives Linux the same value on both architectures and has a `posix` shorthand for the |
| 86 | +shared Linux/macOS value. |
| 87 | + |
| 88 | +In a macro (no `ctx`) it returns a `select()`: |
| 89 | + |
| 90 | +```python |
| 91 | +load("//misc/bazel:os.bzl", "codeql_platform_select") |
| 92 | +load("//misc/bazel:pkg.bzl", "codeql_pkg_files") |
| 93 | + |
| 94 | +codeql_pkg_files( |
| 95 | + name = "extractor-arch", |
| 96 | + exes = codeql_platform_select( |
| 97 | + otherwise = ["//unified/extractor"], |
| 98 | + win64 = ["//unified/extractor-unsupported-os:extractor"], |
| 99 | + ), |
| 100 | + prefix = "tools/{CODEQL_PLATFORM}", |
| 101 | +) |
| 102 | +``` |
| 103 | + |
| 104 | +If implementation code needs to *branch* on the value rather than pass it through, pass `ctx` and add |
| 105 | +`OS_DETECTION_ATTRS` to the rule's attributes. The value is then resolved eagerly instead of being an opaque `select()`: |
| 106 | + |
| 107 | +```python |
| 108 | +load("//misc/bazel:os.bzl", "OS_DETECTION_ATTRS", "os_select") |
| 109 | + |
| 110 | +def _impl(ctx): |
| 111 | + ext = os_select(ctx, windows = ".exe", posix = "") |
| 112 | + ... |
| 113 | + |
| 114 | +my_rule = rule( |
| 115 | + implementation = _impl, |
| 116 | + attrs = {"src": attr.label()} | OS_DETECTION_ATTRS, |
| 117 | +) |
| 118 | +``` |
| 119 | + |
| 120 | +## Packs |
| 121 | + |
| 122 | +`codeql_pack` assembles the files that become an extractor pack. See [`unified/BUILD.bazel`](../../../unified/BUILD.bazel) |
| 123 | +for a minimal complete example and [`pkg.bzl`](../../../misc/bazel/pkg.bzl) for the arguments. The non-obvious parts: |
| 124 | + |
| 125 | +* **`{CODEQL_PLATFORM}` in a destination path is the routing mechanism**, not just a substitution. A path containing it |
| 126 | + is *arch-specific* and lands in the per-architecture zip; every other path is *common*. So `prefix = |
| 127 | + "tools/{CODEQL_PLATFORM}"` both places the file and marks it arch-specific. `arch_overrides` forces named |
| 128 | + destinations into the arch-specific part without a placeholder. |
| 129 | +* **`codeql_pkg_files` splits `srcs` (plain) from `exes` (mode 755)** and **rejects `attributes =`** with an explicit |
| 130 | + error. Use `exes` rather than hand-rolling `pkg_attributes(mode = "755")`. |
| 131 | +* **`pkg_dirs` and `pkg_symlinks` are unsupported** and fail at analysis time. |
| 132 | +* `codeql_pack` also generates an installer and an `install` alias, hence `bazel run //unified:install`. Pass |
| 133 | + `installer_alias = None` if one package defines several packs. |
| 134 | +* `codeql_pack_group` exists for bundling packs into distribution zips, but nothing in this repo instantiates it. |
| 135 | + |
| 136 | +## Adding a dependency |
| 137 | + |
| 138 | +In order of preference: |
| 139 | + |
| 140 | +1. **A [Bazel Central Registry](https://registry.bazel.build/) module.** Add a `bazel_dep` in |
| 141 | + [`MODULE.bazel`](../../../MODULE.bazel). |
| 142 | +2. **A patched upstream module.** Add it under [`misc/bazel/registry`](../../../misc/bazel/registry), which `.bazelrc` |
| 143 | + puts ahead of the BCR. Put patches in `modules/<repo>/<version>/patches`, rename the version with a `-codeql.N` |
| 144 | + suffix, and run [`fix.py`](../../../misc/bazel/registry/fix.py) to realign the metadata. |
| 145 | +3. **A raw archive.** Use `http_archive` via `use_repo_rule`, or a repository rule. Copy an adjacent declaration and |
| 146 | + keep its checksum field populated. |
| 147 | + |
| 148 | +Vendored Rust crates under [`misc/bazel/3rdparty`](../../../misc/bazel/3rdparty) are generated. Regenerate with |
| 149 | +[`update_cargo_deps.sh`](../../../misc/bazel/3rdparty/update_cargo_deps.sh) rather than editing, and keep the |
| 150 | +`use_repo` lists in sync, which `bazel mod tidy` does for module extensions. |
| 151 | + |
| 152 | +## Building standalone |
| 153 | + |
| 154 | +[`MODULE.bazel`](../../../MODULE.bazel) declares `semmle_code` with a `local_path_override` pointing at `..`, which |
| 155 | +resolves when this repo is checked out inside the internal module. [`.bazelrc`](../../../.bazelrc), which Bazel reads |
| 156 | +when invoked in *this* workspace, overrides that with a stub. This line is the whole reason a standalone build resolves: |
| 157 | + |
| 158 | +``` |
| 159 | +common --override_module=semmle_code=%workspace%/misc/bazel/semmle_code_stub |
| 160 | +``` |
| 161 | + |
| 162 | +Do not change either the override path or the `local_path_override`; they work as a pair. |
| 163 | + |
| 164 | +[`misc/bazel/semmle_code_stub`](../../../misc/bazel/semmle_code_stub) is an otherwise empty module supplying no-op |
| 165 | +versions of the internal helpers that shared `.bzl` files load *unconditionally*. That is its only job, and it is small |
| 166 | +enough to read. |
| 167 | + |
| 168 | +**Extend the stub only when a `.bzl` file every standalone target loads gains a new internal `load()`**, which breaks |
| 169 | +package loading outright, for everyone. Prefer not needing one. **Never add a stub so that an internal-only target |
| 170 | +appears to build**: some targets depend on internal libraries (`grep -rl @semmle_code --include=*.bazel` finds them) and |
| 171 | +are correctly unbuildable here. Unlike a `load()`, such a dependency only fails when that target is actually requested. |
| 172 | + |
| 173 | +[`.bazelrc.internal`](../../../.bazelrc.internal) is **not** read here; it carries settings for the internal build. A |
| 174 | +setting needed by both has to be written in both files, with paths differing because this repo sits at a different depth |
| 175 | +there. |
0 commit comments