Improved monorepo support

[GuillaumeQuenneville]

Hi all,

I’ve really enjoyed working with Pixi so far! After using it in a monorepo-style workflow, I’ve put together a few ideas that I believe could significantly improve the UX. Half the time, the features I "come up" with already exist since you guys work so fast. Sorry in advance if they do.

• Local "Working Packages" (.pixi-dev): Some of our packages have long compilation times. It would be useful to have something like a .pixi-dev file (added to .gitignore) where developers can specify which monorepo packages they are currently modifying. This would allow a dev working on package_a to build it locally while pulling packages b->z directly from the channel. This would drastically reduce compile times, run times, and the overall size of Pixi environments. A non-existent .pixi-dev file would simply default to building everything as it does now.

• Shell Access on Build Failure: Currently, if a C++ package fails to build and is listed as a dependency in the [dev] section, pixi shell fails to load if it's listed as a dependency in a package in the monorepo. This prevents me from entering the shell to debug the build environment itself. To get around this, I have to manually remove the dependency from all downstream packages before pixi shell resolves. A solution to this catch-22 would help since the output of pixi run/shell is very verbose and hard to parse for fast iterations on build failures (i'd rather just have the cmake log being shown).

• CLI Support for Workspace Toggling: CLI commands to quickly swap between remote and local dependencies suggested in point 1. For example, pixi dev package_a could move a package from a channel dependency to a workspace dependency (e.g., by updating the suggested .pixi-dev file). pixi dev --build package_b could add it in [dev] also.

• Inherited Dependency Versions: It would be great to use { workspace = true } for package dependencies, not just package metadata. Updating a common dependency like Python across a large monorepo requires running weird sed commands across all pixi.toml files to ensure MatchSpec consistency. It would also be nice if pixi add was aware of the package depdencies.

• Pixi Port of "Changesets": The JavaScript Changesets tool is widely used in TypeScript monorepos to manage versioning of monorepo packages and changelogs via the CLI during PRs. This isn't strictly a Pixi feature and could exist as a standalone package, but a similar workflow within the Pixi ecosystem would be very powerful.

Thanks again to the Pixi team and all the contributors for the great work. Let me know if the team approves of any of these features and I might have the time to contribute them myself.

[baszalmstra]

Local "Working Packages" (.pixi-dev): Some of our packages have long compilation times. It would be useful to have a .pixi-dev file (added to .gitignore) where developers can specify which monorepo packages they are currently modifying. This would allow a dev working on package_a to build it locally while pulling packages b->z directly from the channel. This would drastically reduce compile times, run times, and the overall size of Pixi environments. A non-existent .pixi-dev file would simply default to building everything as it does now.

I like this idea a lot. We already have --only (and we want to add more of these like --up-to). However, having to specify these on the command line everytime is a pain in the but. A file like .pixi-dev with some of those default values makes a lot of sense to me!

Similar to https://colcon.readthedocs.io/en/released/user/configuration.html#defaults-yaml for colcon.

[baszalmstra]

Shell Access on Build Failure: Currently, if a C++ package fails to build and is listed as a dependency in the [dev] section, pixi shell fails to load if it's listed as a dependency in a package in the monorepo. This prevents me from entering the shell to debug the build environment itself. To get around this, I have to manually remove the dependency from all downstream packages before pixi shell resolves. A solution to this catch-22 would help since the output of pixi run/shell is very verbose and hard to parse for fast iterations on build failures (i'd rather just have the cmake log being shown).

I think you can use pixi shell --as-is to get into the environment without trying to build it.

[baszalmstra]

Inherited Dependency Versions: It would be great to use { workspace = true } for package dependencies, not just package metadata. Updating a common dependency like Python across a large monorepo requires running weird sed commands across all pixi.toml files to ensure MatchSpec consistency. It would also be nice if pixi add was aware of the package depdencies.

We have been talking about adding:

[workspace.dependencies]
foo = "1"

And allowing you to do:

[package.host-dependencies]
foo = { workspace = true }

Similar to how cargo workspace dependencies work. Would that help?

[GuillaumeQuenneville]

That's exactly what I had in mind. Thanks!

[baszalmstra]

I added #5945

[baszalmstra]

Pixi Port of "Changesets": The JavaScript Changesets tool is widely used in TypeScript monorepos to manage versioning of monorepo packages and changelogs via the CLI during PRs. This isn't strictly a Pixi feature and could exist as a standalone package, but a similar workflow within the Pixi ecosystem would be very powerful.

I can see how thats useful! I think this could be achieve with just a pixi extension.

[baszalmstra]

CLI Support for Workspace Toggling: CLI commands to quickly swap between remote and local dependencies suggested in point 1. For example, pixi dev package_a could move a package from a channel dependency to a workspace dependency (e.g., by updating the suggested .pixi-dev file). pixi dev --build package_b could add it in [dev] also.

I think if we have workspace dependencies, this would be extremely simple! It will then become a one-line change. Having a CLI around workspace.dependencies also makes sense to me!