machine files: add [compilers] section for hermetic toolchain configuration

docs/markdown/Machine-files.md

@@ -195,6 +195,120 @@ here is:
 - sdl2-config
 - wx-config (or wx-3.0-config or wx-config-gtk)
 
+### compilers
+
+*New in 1.12.0*
+
+The `[compilers]` section allows declaring compiler configuration explicitly,
+rather than relying on Meson to auto-detect it from a binary. This is primarily
+useful for hermetic toolchains where the compiler and its subprograms require a
+custom execution environment that is not present on the host system.
+
+When a language is declared in this section:
+
+- If `<lang>.type` is set, Meson skips family detection and uses that type directly.
+- If `<lang>.version` is also set, Meson skips version detection entirely and uses
+  the declared version. When `<lang>.version` is omitted, Meson still runs the
+  binary with `--version` to determine the version.
+
+The compiler binary itself is always specified via `<lang>` in the `[binaries]`
+section.
+
+Each key is prefixed with the language identifier (`c.`, `cpp.`, etc.).
+
+#### Keys
+
+| Key | Type | Description |
+|-----|------|-------------|
+| `<lang>.type` | string | Compiler family. Supported values: `'gcc'`, `'clang'`, `'clang-cl'`, `'msvc'`, `'intel'` (Classic icc/icpc), `'intel-llvm'` (oneAPI icx/icpx), `'arm'` (Arm Compiler 5 armcc), `'armclang'` (Arm Compiler 6), `'pgi'` (NVIDIA HPC/PGI nvc/nvc++), `'emscripten'`. |
+| `<lang>.version` | string | Version string (e.g. `'10.3.0'`). When specified, used in place of `--version` output. When omitted, Meson runs the binary with `--version` and parses the result as usual. |
+| `<lang>.ccache` | bool | Whether to use ccache when invoking this compiler. Default: `true`. When `true`, ccache is used if found on `PATH`; the build proceeds without caching if ccache is not present. Set to `false` to disable caching unconditionally. |
+| `<lang>.sysroot` | string | Override the root directory for the target system's headers and libraries. Has no effect on compilers that do not support a sysroot concept. |
+| `<lang>.no-default-includes` | bool | Suppress the compiler's built-in system include search. Default: `false`. When set, the compiler's default system include directories are not searched; use `system-include-dirs` to specify them explicitly. Has no effect on compilers that have no equivalent flag (a warning is emitted). |
+| `<lang>.system-include-dirs` | array | System include directories. When set alongside `no-default-includes`, these directories are the only system include directories searched. When set without `no-default-includes`, these directories are searched in addition to the compiler's defaults. |
+| `<lang>.tool-search-paths` | array | Directories in which to search for compiler subtools (assembler, linker helpers, etc.). When omitted, the compiler uses its default search. |
+| `<lang>.subprocess-interpreter` | array | Command used to run compiler subprograms (e.g. cc1, cc1plus, lto1 on GCC). Compiler subprograms are found automatically. The resulting compile commands are fully cacheable by ccache and do not require `-wrapper` or `LD_LIBRARY_PATH`. Accepted but ignored (with an informational note) for monolithic compilers. |
+
+#### Flag translation by compiler family
+
+`—` means the key is not applicable for this compiler family.
+`no-op` means the key is accepted but has no effect.
+
+| Key | GCC, Clang | Intel Classic, Intel oneAPI, ARM 6 (armclang), Emscripten | MSVC, Clang-CL |
+|-----|------------|-----------------------------------------------------------|----------------|
+| `sysroot` | `--sysroot=<path>` | `--sysroot=<path>` (NVIDIA HPC/PGI: —) | — |
+| `no-default-includes` | `-nostdinc` | `-nostdinc` | `/X` |
+| `system-include-dirs` | `-isystem <dir>` | `-isystem <dir>` | `/imsvc <dir>` |
+| `tool-search-paths` | `-B <dir>` | warning, ignored | warning, ignored |
+| `subprocess-interpreter` | generates cc1/cc1plus/lto1 wrappers (GCC only); no-op (Clang) | no-op | no-op |
+| `ccache` | wraps invocation with ccache if available | wraps invocation with ccache if available | wraps invocation with ccache if available |
+
+Notes:
+- **NVIDIA HPC/PGI** (`nvc`/`nvc++`): otherwise identical to the Intel/ARM 6/Emscripten column but does not expose a sysroot concept; `sysroot` is silently ignored.
+- **Clang-CL**: the MSVC-compatible Clang frontend. Uses MSVC-style flags (`/X`, `/imsvc`) matching its `cl.exe` compatibility mode; `--sysroot` and `-B` are not meaningful in this mode.
+- **ARM Compiler 5** (`armcc`): a legacy proprietary compiler with its own flag dialect. None of the structured keys translate to equivalent armcc flags. A warning is emitted at setup for each key that is set; all are ignored. Use `[binaries]` and `[built-in options]` to pass armcc-specific flags directly.
+
+#### Example: hermetic GCC toolchain
+
+A self-contained GCC 12.2.0 toolchain bundled with the project under `sdk/`.
+The `cc1` and `cc1plus` subprograms require shared libraries from `sdk/runtime/lib64`
+that are not installed on the build host.
+
+```ini
+[constants]
+_sdk     = '@GLOBAL_SOURCE_ROOT@' / 'sdk'
+_gcc     = _sdk / 'gcc-12.2.0'
+_runtime = _sdk / 'runtime'    # ELF loader and companion shared libraries
+
+[binaries]
+c   = _gcc / 'bin' / 'x86_64-linux-gnu-gcc'
+cpp = _gcc / 'bin' / 'x86_64-linux-gnu-g++'
+
+[compilers]
+c.type    = 'gcc'
+c.version = '12.2.0'
+c.subprocess-interpreter = [_runtime / 'lib64' / 'ld-linux-x86-64.so.2',
+                              '--library-path', _runtime / 'lib64']
+
+cpp.type    = 'gcc'
+cpp.version = '12.2.0'
+cpp.subprocess-interpreter = c.subprocess-interpreter
+```
+
+With this configuration:
+
+- Compiler subprograms are located and invoked automatically through the bundled ELF
+  loader; no manual wrapper scripts or `LD_LIBRARY_PATH` setup is needed.
+- Include directories and subtool locations are discovered automatically from the
+  compiler.
+- Compilation is fully cacheable by ccache.
+
+When cross-compiling or using a non-standard sysroot, the optional override keys
+(`sysroot`, `no-default-includes`, `system-include-dirs`, `tool-search-paths`)
+can be used to override the discovered values.
+
+#### Example: Clang with custom sysroot
+
+```ini
+[binaries]
+c   = '/path/to/clang-17/bin/clang'
+cpp = '/path/to/clang-17/bin/clang++'
+
+[compilers]
+c.type    = 'clang'
+c.version = '17.0.6'
+c.sysroot             = '/path/to/sdk'
+c.no-default-includes = true
+c.system-include-dirs = ['/path/to/clang-17/lib/clang/17/include',
+                          '/path/to/sdk/usr/include']
+
+cpp.type    = 'clang'
+cpp.version = '17.0.6'
+cpp.sysroot             = c.sysroot
+cpp.no-default-includes = true
+cpp.system-include-dirs = c.system-include-dirs
+```
+
 ### Paths and Directories
 
 *Deprecated in 0.56.0* use the built-in section instead.

mesonbuild/compilers/detect.py

@@ -8,15 +8,17 @@
     search_version, is_windows, Popen_safe, Popen_safe_logged, version_compare, windows_proof_rm,
 )
 from ..programs import ExternalProgram
-from ..envconfig import BinaryTable, detect_cpu_family
+from ..envconfig import BinaryTable, CompilerDescriptor, detect_cpu_family
 from .. import mlog
 
 from ..linkers import guess_win_linker, guess_nix_linker
 
 import subprocess
 import platform
 import re
+import shlex
 import shutil
+import stat
 import tempfile
 import os
 import typing as T
@@ -117,6 +119,50 @@ def detect_compiler_for(env: 'Environment', lang: Language, for_machine: Machine
     return comp
 
 
+# Subprocess wrapper generation
+# ==============================
+
+def _generate_subprocess_wrappers(
+        env: 'Environment',
+        compiler: T.List[str],
+        desc: 'CompilerDescriptor',
+) -> T.Optional[str]:
+    """Generate cc1/cc1plus/lto1 wrapper scripts for hermetic GCC.
+
+    Returns the wrapper directory path (to be passed as -B<dir>), or None if
+    no subprograms could be located.
+    """
+    wrapper_dir = os.path.join(env.scratch_dir, 'compiler-wrappers', desc.lang)
+    os.makedirs(wrapper_dir, exist_ok=True)
+
+    interpreter_parts = desc.subprocess_interpreter
+    interpreter_str = ' '.join(shlex.quote(x) for x in interpreter_parts)
+
+    generated_any = False
+    for prog in ('cc1', 'cc1plus', 'lto1'):
+        try:
+            p, out, _ = Popen_safe(compiler + ['--print-prog-name', prog])
+        except OSError:
+            continue
+        real_path = out.strip()
+        if not real_path or not os.path.isabs(real_path) or not os.path.isfile(real_path):
+            continue
+
+        wrapper_path = os.path.join(wrapper_dir, prog)
+        wrapper_content = (
+            '#!/bin/sh\n'
+            f'exec {interpreter_str} {shlex.quote(real_path)} "$@"\n'
+        )
+        with open(wrapper_path, 'w', encoding='utf-8') as f:
+            f.write(wrapper_content)
+        current_mode = os.stat(wrapper_path).st_mode
+        os.chmod(wrapper_path, current_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
+        mlog.debug(f'Generated {prog} wrapper: {wrapper_path}')
+        generated_any = True
+
+    return wrapper_dir if generated_any else None
+
+
 # Helpers
 # =======
 
@@ -126,9 +172,19 @@ def _get_compilers(env: 'Environment', lang: str, for_machine: MachineChoice,
     The list of compilers is detected in the exact same way for
     C, C++, ObjC, ObjC++, Fortran, CS so consolidate it here.
     '''
+    desc = env.lookup_compiler_desc(for_machine, lang)
+
     value = env.lookup_binary_entry(for_machine, lang)
     if value is not None:
         comp, ccache = BinaryTable.parse_entry(value)
+        # [compilers] descriptor may override ccache wrapping.
+        if desc is not None:
+            if not desc.ccache:
+                ccache = None
+            elif ccache is None:
+                # [binaries] has no ccache prefix; [compilers] ccache=true → auto-detect.
+                detected = BinaryTable.detect_ccache()
+                ccache = detected if detected.found() else None
         # Return value has to be a list of compiler 'choices'
         compilers = [comp]
     else:
@@ -292,6 +348,63 @@ def _detect_c_or_cpp_compiler(env: 'Environment', lang: str, for_machine: Machin
     cls: T.Union[T.Type[CCompiler], T.Type[CPPCompiler]]
     lnk: T.Union[T.Type[StaticLinker], T.Type[DynamicLinker]]
 
+    # Fast path: [compilers] section declared the family (type) explicitly.
+    # Skip pattern-matching on --version output for GCC; generate subprocess
+    # wrappers before running any preprocessing step that invokes cc1.
+    if override_compilers is None:
+        desc = env.lookup_compiler_desc(for_machine, lang)
+        if desc is not None and desc.type == 'gcc' and compilers:
+            compiler = list(compilers[0])
+
+            # Apply structured flags from [compilers] descriptor.
+            if desc.sysroot:
+                compiler.append(f'--sysroot={desc.sysroot}')
+            if desc.no_default_includes:
+                compiler.append('-nostdinc')
+            for d in desc.tool_search_paths:
+                compiler.append(f'-B{d}')
+            for d in desc.system_include_dirs:
+                compiler.append(f'-isystem{d}')
+
+            # Generate subprocess wrappers after structured flags are applied so
+            # --print-prog-name can use tool-search-paths to locate cc1.
+            # Prepend -B<wrapper_dir> right after the binary so it wins over
+            # any -B<libexec> from tool-search-paths.
+            if desc.subprocess_interpreter:
+                wrapper_dir = _generate_subprocess_wrappers(env, compiler, desc)
+                if wrapper_dir:
+                    compiler = compiler[:1] + [f'-B{wrapper_dir}'] + compiler[1:]
+
+            # Determine version: use declared version or run --version.
+            if desc.version is not None:
+                version = desc.version
+                full_version = version
+            else:
+                try:
+                    p, out, _ = Popen_safe_logged(compiler + ['--version'],
+                                                  msg='Detecting compiler via')
+                except OSError as e:
+                    raise EnvironmentException(
+                        f'Failed to run GCC compiler {compiler[0]!r}: {e}')
+                full_version = out.split('\n', 1)[0]
+                version = search_version(out)
+
+            # Still run preprocessor defines detection (requires cc1; wrappers
+            # are now in place so this succeeds even in hermetic builds).
+            defines = _get_gnu_compiler_defines(compiler, lang)
+            if not defines:
+                raise EnvironmentException(
+                    f'GCC compiler at {compiler[0]!r} returned no preprocessor defines; '
+                    'check that the compiler is accessible')
+            if desc.version is None:
+                version = _get_gnu_version_from_defines(defines)
+
+            cls_gcc = c.GnuCCompiler if lang == 'c' else cpp.GnuCPPCompiler
+            linker = guess_nix_linker(env, compiler, cls_gcc, version, for_machine)
+            return cls_gcc(
+                ccache, compiler, version, for_machine,
+                env, defines=defines, full_version=full_version, linker=linker)
+
     for compiler in compilers:
         compiler_name = os.path.basename(compiler[0])
 

mesonbuild/envconfig.py

@@ -3,7 +3,7 @@
 
 from __future__ import annotations
 
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 import typing as T
 from enum import Enum
 import os
@@ -489,6 +489,102 @@ def lookup_entry(self, name: str) -> T.Optional[T.List[str]]:
             return None
         return command
 
+COMPILER_TYPES: T.FrozenSet[str] = frozenset({
+    'gcc', 'clang', 'clang-cl', 'msvc',
+    'intel', 'intel-llvm',
+    'arm', 'armclang',
+    'pgi', 'emscripten',
+})
+
+
+@dataclass
+class CompilerDescriptor:
+    """Per-language compiler configuration from a [compilers] machine-file section."""
+    lang: str
+    type: T.Optional[str] = None
+    version: T.Optional[str] = None
+    ccache: bool = True
+    sysroot: T.Optional[str] = None
+    no_default_includes: bool = False
+    system_include_dirs: T.List[str] = field(default_factory=list)
+    tool_search_paths: T.List[str] = field(default_factory=list)
+    subprocess_interpreter: T.List[str] = field(default_factory=list)
+
+
+class CompilerTable:
+    """Parsed [compilers] section from a native or cross file."""
+
+    KNOWN_KEYS: T.FrozenSet[str] = frozenset({
+        'type', 'version', 'ccache',
+        'sysroot', 'no-default-includes', 'system-include-dirs',
+        'tool-search-paths', 'subprocess-interpreter',
+    })
+
+    def __init__(
+            self,
+            entries: T.Optional[T.Mapping[str, 'ElementaryOptionValues']] = None,
+    ) -> None:
+        self.compilers: T.Dict[str, CompilerDescriptor] = {}
+        if not entries:
+            return
+
+        by_lang: T.Dict[str, T.Dict[str, 'ElementaryOptionValues']] = {}
+        for dotkey, value in entries.items():
+            if '.' not in dotkey:
+                raise mesonlib.MesonException(
+                    f'Invalid [compilers] key {dotkey!r}: expected <lang>.<property>')
+            lang, _, key = dotkey.partition('.')
+            if key not in self.KNOWN_KEYS:
+                mlog.warning(f'Unknown [compilers] key {dotkey!r}, ignoring', once=True)
+                continue
+            by_lang.setdefault(lang, {})[key] = value
+
+        for lang, props in by_lang.items():
+            desc = CompilerDescriptor(lang=lang)
+
+            if 'type' in props:
+                t = props['type']
+                if t not in COMPILER_TYPES:
+                    raise mesonlib.MesonException(
+                        f'Unknown compiler type {t!r} for language {lang!r} in [compilers]; '
+                        f'valid values: {sorted(COMPILER_TYPES)}')
+                desc.type = str(t)
+
+            if 'version' in props:
+                desc.version = str(props['version'])
+
+            if 'ccache' in props:
+                v = props['ccache']
+                if not isinstance(v, bool):
+                    raise mesonlib.MesonException(
+                        f'[compilers] {lang}.ccache must be a bool, got {v!r}')
+                desc.ccache = v
+
+            if 'sysroot' in props:
+                desc.sysroot = str(props['sysroot'])
+
+            if 'no-default-includes' in props:
+                v = props['no-default-includes']
+                if not isinstance(v, bool):
+                    raise mesonlib.MesonException(
+                        f'[compilers] {lang}.no-default-includes must be a bool, got {v!r}')
+                desc.no_default_includes = v
+
+            if 'system-include-dirs' in props:
+                desc.system_include_dirs = mesonlib.stringlistify(props['system-include-dirs'])
+
+            if 'tool-search-paths' in props:
+                desc.tool_search_paths = mesonlib.stringlistify(props['tool-search-paths'])
+
+            if 'subprocess-interpreter' in props:
+                desc.subprocess_interpreter = mesonlib.stringlistify(props['subprocess-interpreter'])
+
+            self.compilers[lang] = desc
+
+    def lookup(self, lang: str) -> T.Optional[CompilerDescriptor]:
+        return self.compilers.get(lang)
+
+
 class CMakeVariables:
     def __init__(self, variables: T.Optional[T.Dict[str, T.Any]] = None) -> None:
         variables = variables or {}