How to best work around needing external dependencies for source installs?

[matthewfeickert]

With pixi add we have the ability to add Python packages to the workspace from source by

pixi add --pypi --git <repository url> --rev <commit sha> <package name>

which is very useful. However, we are not able to use software defined in our environment to build the installed packages and depend on external software. For example, if pixi install is run on the following Pixi manifest

[workspace]
channels = ["conda-forge"]
name = "demo"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = ">=3.14.3,<3.15"
git = ">=2.53.0,<3"

[pypi-dependencies]
astpretty = { git = "https://github.com/asottile/astpretty", rev = "3d04b049b49b5202212c4ca348abf5fb41d94770" }

in an environment without an externally defined Git, it will fail.

$ docker run --rm -ti ghcr.io/prefix-dev/pixi:latest 
root@3cf95b6cfb50:/# pixi init example && cd example
Created /example/pixi.toml
root@3cf95b6cfb50:/example# command -v git
root@3cf95b6cfb50:/example# pixi add python git
Added python >=3.14.3,<3.15
Added git >=2.53.0,<3
root@3cf95b6cfb50:/example# pixi add --pypi --git https://github.com/asottile/astpretty --rev 3d04b049b49b5202212c4ca348abf5fb41d94770 astpretty
Error:   x failed to solve the pypi requirements of environment 'default' for platform 'linux-64'
  |-> Failed to download and build `astpretty @ git+https://github.com/asottile/astpretty@3d04b049b49b5202212c4ca348abf5fb41d94770`
  |-> Git operation failed
  `-> Git executable not found. Ensure that Git is installed and available.

root@3cf95b6cfb50:/example#

We instead need to install Git first from an external process for the build to use

$ docker run --rm -ti ghcr.io/prefix-dev/pixi:latest
root@3cf95b6cfb50:/# pixi global install git
└── git: 2.53.0 (installed)
    └─ exposes: git, git-cvsserver, git-receive-pack, git-shell, git-upload-archive, git-upload-pack, gitk, scalar
root@3cf95b6cfb50:/# pixi init example && cd example
Created /example/pixi.toml
root@3cf95b6cfb50:/example# pixi add python
Added python >=3.14.3,<3.15
root@3cf95b6cfb50:/example# pixi add --pypi --git https://github.com/asottile/astpretty --rev 3d04b049b49b5202212c4ca348abf5fb41d94770 astpretty
Added astpretty
Added these as pypi-dependencies.
root@3cf95b6cfb50:/example# cat pixi.toml 
[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = ">=3.14.3,<3.15"

[pypi-dependencies]
astpretty = { git = "https://github.com/asottile/astpretty", rev = "3d04b049b49b5202212c4ca348abf5fb41d94770" }

While the obvious answer here is "use built software distributions", it is still interesting to note that a source distribution of software even if pinned to the rev is not actually fully reproducible by the Pixi workspace itself.

What are the recommended approaches to working around this in a reproducible way? I suppose one of them would be "ensure that your source dependencies have pixi-build support"?

[lucascolley]

It should be possible to add support for using git as a conda dependency similarly to Python at https://github.com/prefix-dev/pixi/blob/main/crates/pixi_core/src/lock_file/resolve/pypi.rs#L397-L407

[matthewfeickert]

@lucascolley That sounds good. Would you be willing to transfer this Discussion to an Issue? Or would you prefer to close this out and I just make a new one?

[lucascolley]

done gh-5816

[matthewfeickert]

Thanks!

[maurosilber]

Have you tried with no-build-isolation? https://pixi.prefix.dev/latest/reference/pixi_manifest/#the-pypi-options-table

[matthewfeickert]

@maurosilber I understand why you would suggest that given

One can choose to not isolate the build for a certain package name, this allows the build to access the pixi environment.

but that doesn't have any effect here, and this would also require me to have to define setuptools as a dependency, which isn't great.

$ docker run --rm -ti ghcr.io/prefix-dev/pixi:latest
root@90e72202abc8:/# pixi init example && cd example
Created /example/pixi.toml
root@90e72202abc8:/example# pixi add python git setuptools
Added python >=3.14.3,<3.15
Added git >=2.53.0,<3
Added setuptools >=82.0.1,<83
root@90e72202abc8:/example# echo -e '\n[pypi-options]\nno-build-isolation = ["astpretty"]\n' >> pixi.toml
root@90e72202abc8:/example# cat pixi.toml 
[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = ">=3.14.3,<3.15"
git = ">=2.53.0,<3"
setuptools = ">=82.0.1,<83"

[pypi-options]
no-build-isolation = ["astpretty"]

root@90e72202abc8:/example# pixi add --pypi --git https://github.com/asottile/astpretty --rev 3d04b049b49b5202212c4ca348abf5fb41d94770 astpretty
Error:   x failed to solve the pypi requirements of environment 'default' for platform 'linux-64'
  |-> Failed to download and build `astpretty @ git+https://github.com/asottile/astpretty@3d04b049b49b5202212c4ca348abf5fb41d94770`
  |-> Git operation failed
  `-> Git executable not found. Ensure that Git is installed and available.

root@90e72202abc8:/example#