-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
PEP 825: Various clarifications #4969
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 13 commits
510c525
1fd013b
58778b0
db4fe20
f5a98eb
7a4f9a9
68f49fc
c4ccf6c
19acda7
bea8e82
93f883f
9d3bcf7
51187f6
d9f0f44
2dbbf45
69bf1b2
51e47c9
fce4d46
f209235
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -308,11 +308,17 @@ there MUST exist a corresponding ``{name}-{version}-variants.json`` | |
| file. The ``{name}`` and ``{version}`` placeholders correspond to the | ||
| package name and version, normalized according to the same rules as | ||
| wheel files, as found in the :ref:`packaging:wheel-file-name-spec` of | ||
| the Binary Distribution Format specification. The exact URL where the | ||
| file is hosted is insignificant, but a link to it MUST be present on all | ||
| index pages where the variant wheels are linked. It is presented in the | ||
| same simple repository format as source distribution and wheel links in | ||
| the index, including an (OPTIONAL) hash. | ||
| the Binary Distribution Format specification. | ||
|
|
||
| The exact URL where the file is hosted is insignificant, but it MUST | ||
| be provided in all the responses where the variant wheels are included. | ||
| It should follow the rules for files in the | ||
| :ref:`packaging:simple-repository-api`, except that the various metadata | ||
| served by the index (such as ``core-metadata``, ``dist-info-metadata``, | ||
| ``requires-python`` or ``yanked``) are not meaningful for that file. | ||
| Indexes MAY publish or skip these attributes, as long as the values do | ||
| not prevent correct operation. Tools MAY either use or ignore these | ||
| values. | ||
|
|
||
| This file uses the same structure as `variant metadata`_, except that | ||
| the ``variants`` object MUST list all variants available on the package | ||
|
|
@@ -379,17 +385,20 @@ like: | |
| Variant ordering | ||
| ---------------- | ||
|
|
||
| To determine which variant wheel to install when multiple wheels are | ||
| compatible, variants MUST be totally ordered by their variant | ||
| properties. | ||
| This specification defines an ordering between different wheels based on | ||
| the presence of variant metadata. | ||
|
|
||
| For the purpose of ordering, variant properties are grouped into | ||
| features, and features into namespaces. For every namespace, the tool | ||
| MUST obtain an ordered list of compatible features, and for every | ||
| feature, an ordered list of compatible values. The method of obtaining | ||
| these lists will be defined in a subsequent PEP. | ||
|
|
||
| The default ordering MUST be performed equivalent to the following | ||
| MUST obtain a list of compatible features, and for every feature, a list | ||
| of compatible values. The method of obtaining these lists will be | ||
| defined in a subsequent PEP. The items in these lists will be provided | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "The method of obtaining these lists is not defined (and hence is tool-specific)." We should avoid making this PEP explicitly dependent on "future PEPs", as if we do so it's impossible to approve this PEP on its own merits.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't really see that contradiction, I think we have to live with some of that tension if we want to avoid having one massive PEP 817. There's other PEPs too which specifically leave gaps for future specifications, and I don't see that as fundamentally different. If we say it's tool-specific, that would be rather misleading, as we don't want to open up the namespace to everyone beyond experimentation, we have the expectation that in the end we'll have a compatibility standard where everyone speak the same language.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Agreed (although we have to accept that if the tension is too great, a single PEP 817 may be a better approach). The key here is that we have a way to implement PEP 825 in the absence of those other PEPs. If we can't start implementation work once PEP 825 is (provisionally) accepted, there's no point in accepting it independently. So regardless of what you might prefer, anything not specified in PEP 825 will be tool-defined. When the follow-up PEP XXX comes along, people who correctly guessed what it would say will be ready, and everyone else will have to change their implementation, but we can't know what the answer will be in advance. I'd rather we made it clear that's the situation, rather than having people think they can't start implementation work because parts of the behaviour need a further PEP to specify them.
Maybe I'm getting too concerned about this. Can you give me some examples and I'll check how they handle the situation?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The whole point is that you can start the implementation (provided that you're ready to take the risk that the details will change). We say "subsequent PEP" in very specific places, telling you to leave specific gaps. Like "you can implement most of the sort algorithm, just leave the gap for functions that will provide these lists"; even the example below literally provides an implementation with gaps for these functions. I don't think "guessing" really works here. I suppose you can infer what the implementation will be from PEP 817, or you can do your own thing. Whether this makes you PEP 825 compliant is unclear to me, since the PEP by design is part of the larger series, and therefore full compliance implies implementing all of them. A tool that implements PEP 825 but then diverges from subsequent PEPs does not really implement "variant wheels"; it implements a custom solution that is partially based on "variant wheels" but it is definitely not compliant with the design as a whole, and therefore it is not guaranteed to be interoperable.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK. Let me think about this some more. The idea was that we split PEP 817 into independent parts, but if it's simply not possible to make those parts independent, then maybe we're better accepting that and going back to the single unified approach with PEP 817. Or maybe this approach of having a series of PEPs that don't work independently, but which need to be accepted or rejected as a whole is the right approach - but I don't know of any prior occasions when this was done, so I'd need to do some research to decide how viable it is.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I personally think split PEPs are better, but rather than independent, building one on top of another. If anything, they are easier to read, and I found it easier to describe the concepts that way. It also keeps the discussion more focused.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I just looked back at the PEP 817 discussion, where splitting the PEP was first discussed. At the time, I said:
With that as context, I'll note that:
That would leave PEP 825 as basically just a set of terminology definitions, a JSON data schema, a wheel naming convention and a definition of where the variant data is stored in the wheel. That, IMO, would be a good foundation, and should2 be uncontroversial. It doesn't do anything to address the issues that are blocking us here, all it does is put them into their own PEPs. So I'd completely understand if you don't want to do that (writing more PEPs is more work). But if what matters to you is getting something approved sooner rather than later, then it might help with that.
I agree 100% that smaller PEPs make understanding the proposal easier - in a sense, they are acting like chapters in a larger document. What I'm not sure about is whether it makes sense to approve individual PEPs from a series structured like that (given that you've said you can't make them completely independent). If you're OK with writing the various PEPs and having them all remain in Draft status until they are submitted as a group for approval, I'd have no problem with that. But I got the distinct impression that you want an early approval of at least part of the work, for motivational reasons if nothing else. Would it help if I said formally that I support the idea of wheel variants, but until the full set of PEPs is available and the proposal has been discussed and is documented in its full form (across multiple PEPs) I won't be approving individual parts? Footnotes
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Ordering" is a bit of unclear fit indeed. I've included it in PEP 825 since it can be defined independently of actual data (with the minimal gaps for input), and I've figured out that the next PEP will be big enough to justify keeping it out of it. On the other hand, I suppose moving it later would have the advantage of giving more context why things were designed this way (not really relevant to specification but to rationale). That said, I'm not convinced that moving stuff at this point is really worth the effort.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that the data structure for ordering data is reasonable to include here, but manipulation of it (merging orderings, overriding, using it to select wheels) could be deferred. But I do agree that deferring just puts off the problem, it doesn't make it go away. So I'm happy to keep it here and address the issues now. Also, as I noted on DPO, my thinking regarding the index-level files has changed somewhat. They are only an optimisation for the label->property mapping data. For ordering data, they aren't, because you don't define merging of ordering data from multiple sources. So a consumer can't go to the ordering data in the wheels, because they then can't merge that data, and so can't establish a single ordering for all of the wheels on the index. As an aside, one point I'm not clear on - is it right that a project could legitimately provide no ordering data at all, relying solely on the default ordering that the providers give?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This conversation seems resolved.
@mgorny answered this here: |
||
| in specific order that will impact variant wheel ordering. | ||
|
|
||
| The compatible wheels corresponding to a particular combination of | ||
| package name, version and build number MUST be grouped by their variant | ||
| label, and a separate group of non-variant wheels MUST be formed. The | ||
| groups of variant wheels MUST then be ordered according to the following | ||
| algorithm: | ||
|
|
||
| 1. Construct the ordered list of namespaces by copying the value of the | ||
|
|
@@ -402,9 +411,10 @@ algorithm: | |
| value of the respective ``default-priorities.feature.{namespace}`` | ||
| key. | ||
|
konstin marked this conversation as resolved.
|
||
|
|
||
| ii. Obtain the compatible feature names, in order. For every feature | ||
| name that is not present in the constructed list, append it to | ||
| the end. | ||
| ii. Take the ordered list of compatible feature names obtained | ||
| previously and iterate over it, in order. For every feature name | ||
| that is not present in the constructed list, append it to the | ||
| end. | ||
|
|
||
| After this step, a list of ordered feature names is available for | ||
| every namespace. This is ``feature_order`` in the example. | ||
|
|
@@ -415,48 +425,49 @@ algorithm: | |
| of the respective | ||
| ``default-priorities.property.{namespace}.{feature_name}`` key. | ||
|
konstin marked this conversation as resolved.
|
||
|
|
||
| ii. Obtain the compatible feature values, in order. For every value | ||
| that is not present in the constructed list, append it to the | ||
| end. | ||
| ii. Take the ordered list of compatible feature values obtained | ||
| previously and iterate over it, in order. For every value that is | ||
| not present in the constructed list, append it to the end. | ||
|
|
||
| After this step, a list of ordered property values is available for | ||
| every feature. This is ``value_order`` in the example. | ||
|
|
||
| 4. For every compatible variant, determine the most preferred value | ||
| corresponding to every feature in that variant. This is done by | ||
| finding among the values present in the variant properties the one | ||
| that has the lowest position in the ordered property value list. | ||
| After this step, a list of features along with their best values | ||
| is available for every variant. This is done in the | ||
| ``Variant.best_value_properties()`` method in the example. | ||
| 4. For every group, determine the most preferred value corresponding to | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't understand this step. A group, as I understand it, is a set of wheels with the same variant label. So each wheel in a group has the same value for every property. So how can there be a "most preferred value" when there's only one value? I suspect I'm misunderstanding here because the terminology still isn't clear to me. But I'm more concerned with making sure that the reader can follow what's being described than I am arguing about terms. Or is this actually about trying to order the different groups, so you're trying to identify the most preferred group based on the values the labels denote?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See "variant properties". A single wheel can have multiple values corresponding to a single feature. You select the most preferred one from them for ordering. If the wheel has: you would sort only on:
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This question seems answered; the specification text seems fine. If a small rephrasing would have helped, a comment with the exact suggested rephrase could be applied. I'd say it's fine as is though; specification text doesn't get more clear by pre-emptively explaining questions like these (assuming the spec is indeed unambiguous, which it is here). |
||
| every variant feature present in the variant properties corresponding | ||
| to the group. This is done by finding among the values the one that | ||
| has the lowest position in the ordered property value list. After | ||
| this step, a list of features along with their best values is | ||
| available for every variant. This is done in the | ||
| ``VariantWheel.best_value_properties()`` method in the example. | ||
|
|
||
| 5. For every item in the list constructed in the previous step, | ||
| construct a sort key that is a 3-tuple consisting of | ||
| its namespace, feature name and best feature value indices in the | ||
| respective ordered lists. This is done by the ``property_key()`` | ||
| function in the example. | ||
|
|
||
| 6. For every compatible variant, sort the list constructed in step 4 | ||
| using the sort keys constructed in step 5, in ascending order. This | ||
| is done by the ``Variant.sorted_properties()`` method in the example. | ||
| 6. For every group, sort the list constructed in step 4 using the sort | ||
| keys constructed in step 5, in ascending order. This is done by the | ||
| ``VariantWheel.sorted_properties()`` method in the example. | ||
|
|
||
| 7. To order variants, compare their sorted lists from step 6. If the | ||
| sort keys at the first position are different, the variant with the | ||
| 7. To order groups, compare their sorted lists from step 6. If the | ||
| sort keys at the first position are different, the group with the | ||
| lower key is sorted earlier. If they are the same, compare the keys | ||
| at the second position, and so on, until either a tie-breaker is | ||
| found or the list in one of the variants is exhausted. In the latter | ||
| case, the variant with more keys is sorted earlier. As a fallback, | ||
| if both variants have the same number of keys, they are ordered | ||
| lexically by their variant label, ascending. This is done by the | ||
| found or the list in one of the groups is exhausted. In the latter | ||
| case, the group with more keys is sorted earlier. As a fallback, | ||
| if both groups have the same number of keys, they are ordered | ||
| lexically by the variant label, ascending. This is done by the | ||
| ultimate step of the example algorithm, with the comparison function | ||
| being implemented as ``Variant.__lt__()``. | ||
| being implemented as ``VariantWheel.__lt__()``. | ||
|
|
||
| After this process, the variant wheels are sorted from the most | ||
| preferred to the least preferred. The algorithm sorts the null variant | ||
| after all the other variants. The non-variant wheel MUST be ordered | ||
| after the null variant. Multiple wheels with the same variant property | ||
| set (and multiple non-variant wheels) MUST then be ordered according to | ||
| their platform compatibility tags. | ||
| The algorithm sorts the group of null variant wheels last, as they | ||
| feature no variant properties. The group of non-variant wheels MUST be | ||
| placed after all the other groups. | ||
|
|
||
| Within every group, the wheels MUST then be ordered according to their | ||
| platform compatibility tags. After this process, the variant wheels are | ||
| sorted from the most preferred to the least preferred. | ||
|
|
||
| The tools MAY provide options to override the default ordering, for | ||
| example by specifying a preference for specific namespaces, features | ||
|
|
@@ -465,7 +476,8 @@ variants, or to select a particular variant. | |
|
|
||
| Alternatively, the sort algorithm for variant wheels could be described | ||
| using the following pseudocode. For simplicity, this code does not | ||
| account for non-variant wheels or tags. | ||
| account for non-variant wheels or the subsequent ordering by platform | ||
| compatibility tags. | ||
|
|
||
| .. code:: python | ||
|
|
||
|
|
@@ -530,7 +542,7 @@ account for non-variant wheels or tags. | |
| ) | ||
|
|
||
|
|
||
| class Variant: | ||
| class VariantWheel: | ||
| """Example class exposing properties of a variant wheel""" | ||
|
|
||
| label: str | ||
|
|
@@ -571,13 +583,13 @@ account for non-variant wheels or tags. | |
| return self.label < other.label | ||
|
|
||
|
|
||
| # A list of variants to sort. | ||
| variants: list[Variant] = [...] | ||
| # A list of variant wheels to sort. | ||
| variant_wheels: list[VariantWheel] = [...] | ||
|
|
||
|
|
||
| # 7. Order variants by comparing their sorted properties | ||
| # (see Variant.__lt__()) | ||
| variants.sort() | ||
| # 7. Order variant wheels by comparing their sorted properties | ||
| # (see VariantWheel.__lt__()) | ||
| variant_wheels.sort() | ||
|
|
||
|
|
||
| Environment markers | ||
|
|
@@ -737,6 +749,10 @@ Note that steps 4. through 8. are introduced specifically for variant | |
| wheels. The remaining steps correspond to the current installer | ||
| behavior. | ||
|
|
||
| When installing from a source that does not provide an `index-level | ||
| metadata`_, the same algorithm can be used, except that the variant | ||
| metadata needs to be read directly from the wheels. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This doesn't feel like it's true. If you have 2 wheels served from a local directory, with different
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's a single source, so the same rules of consistency as in "index-level metadata" apply, and the same merging algorithm described there.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, consistency rules apply. |
||
|
|
||
|
|
||
| Installing a local wheel | ||
| '''''''''''''''''''''''' | ||
|
|
@@ -786,28 +802,68 @@ To generate the ``{name}-{version}-variants.json`` file: | |
| Rationale | ||
| ========= | ||
|
|
||
| This PEP is part of a larger variant wheel design that was originally | ||
| proposed as :pep:`817`. However, due to its complexity, we decided to | ||
| split it into smaller parts that build one upon another. This PEP is the | ||
| first in the series, providing foundations including the file format | ||
| along with necessary metadata, index support and basic tool algorithms. | ||
| Aspects such as providing actual variant properties or building wheels | ||
| are deferred into subsequent PEPs. | ||
|
|
||
| Variant wheels use structured `variant properties`_ to express | ||
| multidimensional wheel compatibility matrices. For example, it permits | ||
| expressing that a single variant requires certain CPU and GPU features | ||
| independently. It can express both AND-style dependencies (such as | ||
| different CPU instruction sets) and OR-style dependencies (such as | ||
| different GPUs supported by a single package). | ||
| multidimensional wheel compatibility matrices. Properties are organized | ||
| in namespaces that can be defined and governed independently. The | ||
| key-value structure makes the properties more flexible: adding a new | ||
| compatibility axis can be done by adding a new key. It can support both | ||
| AND-style dependencies (for example, a CPU plugin could define multiple | ||
| keys corresponding to different instructions sets, all of which are used | ||
| in the package and therefore must be supported) and OR-style | ||
| dependencies (for example, a GPU plugin can define a single key listing | ||
| multiple GPU types, indicating that all of them are supported by the | ||
| package, and therefore the users needs to own only one of them). | ||
|
|
||
| The specification does not impose any formal limits on the number of | ||
| properties expressed, and specifically accounts for the possibility of | ||
| property sets being very long (for example, a long list of GPUs or CPU | ||
| extension sets). To avoid wheel filenames becoming very long, the | ||
| property lists are stored inside the wheel and mapped to a short label | ||
| that is intended to be human-readable. | ||
|
|
||
| To facilitate variant selection while installing from remote index, | ||
| the variant metadata is mirrored in a JSON file published on the index. | ||
| This enables installers to obtain variant property mapping without | ||
| having to fetch individual wheels. | ||
|
|
||
| Since JSON format does not feature a set type, sets are represented as | ||
| sorted lists. Sorting ensures that tools can safely use equality | ||
| comparison over dictionaries. | ||
| extension sets). To avoid wheel filenames becoming hard to comprehend | ||
| because of excess of information and potentially causing technical | ||
| issues because of their length, the property lists are stored inside | ||
| the wheel and mapped to a short label that is chosen by the package | ||
| maintainer and intended to be human-readable. | ||
|
|
||
| Wheel filenames alone do not provide sufficient metadata to drive | ||
| variant wheel selection. To avoid tools having to fetch the variant | ||
| metadata straight from multiple wheel files, the metadata from wheels | ||
| for every package version is combined and republished. This metadata is | ||
| scoped to a single package version to permit variants changing in the | ||
| future version. | ||
|
|
||
| The index support aims to account for three scenarios: | ||
|
|
||
| 1. An index implementation that cannot embed additional metadata as part | ||
| of file list responses. For example, this covers installing straight | ||
| from a directory listing created by a webserver. To account for this | ||
| scenario, index-level metadata is published as a plain JSON file that | ||
| can be generated by the package maintainer and placed alongside | ||
| wheels. | ||
|
|
||
| 2. An index implementation that has more complete wheel support but does | ||
| not wish to implement full variant wheel support immediately. The | ||
| index needs only to permit the user to upload said JSON file. To | ||
| account for minimalistic implementation, the specification permits | ||
| the index to treat said file similarly to a wheel, including | ||
| publishing attributes such as ``yanked``, as long as their values do | ||
| not prevent clients from working. | ||
|
|
||
| 3. An index implementation that implements complete wheel variant | ||
| support. Such an index will parse uploaded variant wheels, and | ||
| dynamically create the index-level metadata. The JSON file path would | ||
| then be treated as an API endpoint rather than an actual file. | ||
|
|
||
| Since JSON format does not feature a set type, sets in the metadata are | ||
| represented as sorted lists. Sorting ensures reproducibility and makes | ||
| it possible to use equality comparison over whole dictionaries without | ||
| having to convert specific fields back to sets after deserialization. | ||
|
|
||
| The variant ordering algorithm has been proposed with the assumption | ||
| that variant properties take precedence over Platform compatibility | ||
|
|
@@ -817,15 +873,25 @@ variant may require a different minimal libc version, in which case the | |
| selection should be driven by the desired CUDA preference rather than | ||
| incidental platform tag difference. | ||
|
|
||
| While the provision of variant properties is deferred to a future PEP to | ||
| keep the specification easier to comprehend, a baseline assumption is | ||
| made that the compatible properties will be provided in specific order | ||
| corresponding to their preference, much like Platform compatibility tags | ||
| conventionally are. The variant metadata provides the ability to | ||
| override this order at package level. However, namespaces are unordered | ||
| by design (e.g. we will not decide upfront which GPU vendors take | ||
| precedence) and therefore they always need to be ordered by the package | ||
| maintainer. | ||
| While a future PEP will define how variant properties are provided, a | ||
| baseline assumption is made that the compatible properties will be | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This shouldn't be an assumption, it should be explicitly required in this PEP. Then this sentence can be reworded something along these lines: This PEP simply requires that compatible properties are provided in a specific order corresponding to their preference. We do not state how tools will provide this ordered list, but a future PEP is planned which will standardise the mechanism.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Shouldn't the rationale section be describing why were the specific decisions made in the specification part, rather than stating what is required (I.e. effectively repeating specification)? I dare say the "context" is the whole point of having a rationale in the first place.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, if you want to move this "baseline assumption" into the specification section as a requirement, that works for me.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But it literally does that, in the fragment you've commented on above:
We're merely trying to provide a bit more context of why this requirement is baked in there.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems resolved. |
||
| provided in specific order corresponding to their preference. This makes | ||
| it possible to use a generic sorting algorithm that, and later define | ||
|
konstin marked this conversation as resolved.
Outdated
|
||
| properties as data without having to change the algorithm. | ||
|
|
||
| A future PEP will define how the ordering for features and values is | ||
| provided. However, namespaces are governed independently and considered | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here. You need to say what this PEP requires, so that tools can (in principle) implement their own mechanism for providing an ordering. The fact that we'll standardise that mechanism in a future PEP isn't important here, except as context.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same comment as higher up; this PEP requires |
||
| on equal footing, and therefore there will be no standard ordering for | ||
| them. Instead, the package maintainer will decide which namespaces have | ||
|
konstin marked this conversation as resolved.
Outdated
|
||
| higher priority, and therefore which variants will be preferred. For | ||
| completeness, the specification also permits the maintainers to override | ||
| the ordering for features and values as well. Tools can also further | ||
| override the variant choice, much like they can do with regular wheels. | ||
|
|
||
| In the vast majority of real use cases, ordering based on properties | ||
| will suffice. However, in a pathological case two different variant | ||
| wheels may end up with equal sort keys. To provide reproducible results | ||
| in this case, fallback sorting on variant label is performed. | ||
|
|
||
| A concept of null variant is introduced that is distinct from | ||
| non-variant wheels to facilitate a transition period. This variant is | ||
|
|
@@ -938,7 +1004,7 @@ Reference Implementation | |
| The `variantlib <https://github.com/wheelnext/variantlib>`__ project | ||
| contains a reference implementation of a complete variant wheel | ||
| solution. It is compliant with this PEP, but also goes beyond it, | ||
| providing example solutions to `open issues`_. | ||
| providing example solutions to some of deferred items. | ||
|
konstin marked this conversation as resolved.
Outdated
|
||
|
|
||
| A client for installing variant wheels is implemented in a | ||
| `uv branch <https://github.com/astral-sh/uv/pull/12203>`__. | ||
|
|
@@ -982,15 +1048,34 @@ would be incorrectly deemed compatible because of the | |
| ``manylinux_2_27_x86_64`` part. | ||
|
|
||
|
|
||
| Open Issues | ||
| =========== | ||
| Replacing Platform compatibility tags entirely | ||
| ---------------------------------------------- | ||
|
|
||
| Technically, it would be entirely possible to convey the information | ||
| currently passed via the Platform compatibility tags via variant | ||
| properties, and remove these explicit tags from the filename. However, | ||
| we decided not to pursue this and instead preserve the existing | ||
| filenames for wheels that do not need additional variants, as we do not | ||
| believe that the effort required to update all the existing workflows | ||
| justifies the benefit of more compact, and slightly more consistent | ||
| naming. | ||
|
|
||
|
|
||
| Out of scope | ||
| ------------ | ||
|
|
||
| The following problems are deferred to subsequent PEPs: | ||
| The following problems are deferred to subsequent PEPs in the series: | ||
|
|
||
| - governance of variant namespaces | ||
| - determining which variant properties are compatible with the system | ||
| - building variant wheels | ||
|
|
||
| In addition to that, the following questions are left undefined: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The question is well-defined, what you are saying is that you aren't going to answer the question. Which means that this PEP doesn't support installation from multiple sources of wheels. I don't think that's an acceptable position to take, as installing from multiple sources is an extremely common scenario, and if this PEP doesn't support it, that's a significant usability problem. IMO, what you need to do is identify all of the places in the PEP where having wheels from multiple sources would cause an issue (I've noted a few of them above) and explicitly state in those places that the PEP doesn't define behaviour in that situation1. That changes the position from "multiple indexes are out of scope" to "behaviour for multiple indexes is tool-defined". Footnotes
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
We're specifically say "undefined", not "not supported". Tools can support multiple index in any shape they want, by virtue of the spec saying nothing about it.
Can you explain what the difference between those two is? Currently, there is no spec for multiple index support: Multiple indexes are out of scope in any other PEP, we're just writing it out explicitly because otherwise people complained about the absence of that, and it's still tool-defined, as tool make choices about multiple indexes right now, in the absence of a spec.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Multiple indexes are out of scope" means that an implementation that doesn't allow multiple sources for variant wheels is PEP compliant. That seems to me like it would be a severe limitation on the usability of variants, especially as it would effectively disallow installing from a local directory of wheels (because in the absence of an index, each wheel is its own independent source of variant data). On the other hand, "behaviour for multiple indexes is tool-defined" means that PEP-compliant implementations need to support multiple indexes/sources, but they get to choose the behaviour in those specific places where the PEP says that behaviour is undefined. I appreciate that it's a subtle distinction, but IMO it's important. The precise wording doesn't matter to me, but the difference in semantics does.
I don't know of any other PEP/standard where it matters whether multiple indexes are involved1. This PEP is different precisely because the ordering algorithm is described in terms that assume a single Footnotes
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you point us to a document explaining this difference? I don't really understand how "tool-defined" implies that something has to be implemented. Isn't "reject as not supported" also a valid tool-defined behavior? Also, does this mean that a tool that currently does not support multiple sources at all won't be able to be compliant with variant wheels without actually implementing support for multiple sources, i.e. something it currently didn't need to do when installing regular wheels?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, I can't point you to a document. This isn't a technical term with a precise definition, it's a matter of how the PEP reads to me. And by extension, how I believe it could be read in the future by others - I'm basing this on my experience with other standards where we've had problems after the fact because people have interpreted imprecise wording in a way that we hadn't originally intended. I don't think this side discussion is helping. It's a minor point in isolation, and it's threatening to overwhelm the main issue I'm trying to draw attention to, which is that I don't see how we can simply ignore (or "leave to tool implementers" if you prefer) the question of how to calculate ordering when faced with multiple sources of information to combine. At the end of the day, the PEP will need to address this issue if it's going to be accepted. I've tried to read the ordering algorithm as described in the PEP multiple times now, and every time I've got stuck, usually because I can't work out how I'd generalise the given algorithm to handle multiple sources (something I expect to have to do for pip, where "pick the first index" isn't a valid option unless we can assume all indexes have the same data). To be very explicit here (please don't take this as any sort of threat, it's not intended that way, but I do want to be sure there's no misunderstanding), I'm currently not seeing a way I can accept the PEP if it doesn't somehow clarify how tools are supposed to handle multiple sources. I'm trying to help fix that problem, but if my comments are having the opposite effect, I should probably just state the issue and leave it to you to decide how to take things forward. The PEP doesn't say who the sponsor is (that's something that should be fixed, by the way), but I'd recommend you speak to them as well - they should be able to help navigate the process around incorporating feedback like this, if it's a problem.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
A sponsor isn't needed here because Barry and Donald are co-authors and on the core team, so you can speak to them.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I'll post about this on Discourse, because it's a content/process discussion either way. This PR just contains clarifications which all seem ready to go, and address some questions and confusion. |
||
|
|
||
| - Selecting variant wheels from multiple sources. Currently, there is no | ||
| standard defined behavior for regular wheels, nor consensus across | ||
| different packaging tools on how to handle that. | ||
|
|
||
|
|
||
| Acknowledgements | ||
| ================ | ||
|
|
@@ -1036,6 +1121,12 @@ Change History | |
| - Changed ``pylock.toml`` integration to inline variant metadata | ||
| rather than storing a URL and a hash. | ||
|
|
||
| - 11-May-2026 | ||
|
|
||
| - Added replacing platform compatibility tags entirely to rejected | ||
| ideas. | ||
| - Clarified interpretation of sorting algorithm and index support. | ||
|
|
||
|
|
||
| Appendices | ||
| ========== | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you're putting
variant.jsonfiles in thefilessection of the JSON response, then you need to follow the existing rules, which say that for eachfilesentry,filename,url,hashesandsizeare mandatory, and all other keys are optional. There's nothing in the existing spec that suggests that tools can ignore attributes if they are present, and I don't think we should be special-casingvariant.jsonfiles.I actually don't understand why you're allowing these fields to be present at all. What's wrong with simply saying that
variant.jsonfiles must not include any of the optional fields defined in the index spec (with the exception of theupload-timefield, which may be present)1?Honestly, this feels like a problem caused by trying to put
variant.jsonfiles into a field that's intended to hold details of distribution files. I'd argue that a cleaner design would be to have a separatevariant-datafield in the project details response2.Footnotes
I'm only making an exception for upload time because I can see cases where it might be useful, in analysing index data. ↩
The HTML version of the index would still need to have all the links together, but that should be viewed as secondary, as per PEP 833 (which is not yet approved, but is likely to be). ↩
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, "optional fields" is what we meant. We'll correct this.
For the rest, see rationale. The goal is to support indexes during the transitional period with minimal modifications necessary. Should the tools be rejecting projects if the index gives unnecessary-yet-harmless fields?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Specifically note the provision:
This is intended as "you don't have to go out of your way to special case that one file", as long as you don't do something really wrong there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was done in commit 2dbbf45. Other parts of the review comment seem answered.