How to disable Python 2.6/7 testing for unit tests (using Github actions template)

[SHxKM]

We've adapted the following workflow template. While sanity tests seem to look at tests/config.yml which in our case includes:

modules:
    python_requires: '>=3.6'

Unit tests do not, so they're always failing with something along these lines:

needs_pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
                                ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-0YTCSc/corp/
ERROR: Command "/usr/bin/python2.6 /root/ansible/test/lib/ansible_test/_data/quiet_pip.py install --disable-pip-version-check -r 

What do I need to do so unit tests also refrain from testing against Python 2.6/7? (or rather, always run against Python>=3.6)

[SHxKM]

I'm starting to suspect that this is because the underlying library (dependency) does not properly declare it's Python>=3.6 only, but even adding boto3==1.20.23 (which does) doesn't prevent the unit tests from trying to run against Python 2.6/7. I know this is not a solution, I'm just trying to understand that the problem is first.

[SHxKM]

Hmm, now it seems like sanity is wanting to run 2.6/2.7 as well. Confused..

[SHxKM]

I'm starting to understand what's happening: the tests/config.yml feature was introduced only in Ansible 2.12. For all other versions (2.9, 2.10, 2.11) - I will need to add a X-Y.txt under tests/sanity for sanity-tests as explained by @tadeboro, but how can I do the same for unit tests?

[SHxKM]

Have you tried using pytest.importorskip instead of a regular import?

Did not try that. So I should use that instead of from foo.bar import * for example? and together with pytest.mark.skipif?

I'm wondering how much of the effort it's worth. Ansible (when building its "core" collections) is not going to run unit tests, is it? because in that case I have no choice but to try what you suggested. Obviously Ansible is not going to use our Github Workflow for its build..

[tadeboro]

Right now, no one will run your tests. Ansible Galaxy import process runs some linting on the roles and collection metadata, but that is it. Automation Hub runs sanity tests, but you should be able to ignore enough of them to make things pass (this is only relevant if you would like to certify your content).

So, in the end, it is up to you to decide how much effort you want to put into Python compatibility. If I were in your place, I would:

1. make sure ansible-test sanity --docker passes on all supported Ansible versions by adding enough stuff to ignore files to make ansible-test happy,
2. run integration tests using a decent mix of Ansible and Python versions, and
3. only run unit tests on ansible-core>=2.12 that know how to skip specific Python versions.

Note that this would also be more than enough to pass the review process for inclusion into the Ansible package.

[SHxKM]

Thanks @tadeboro. This is both comforting and surprising. We’ve got into some kind of race against time to put everything in place so that Ansible’s sanity tests pass and we can apply for inclusion…

[tadeboro]

We are talking about the community stuff here, and I tend to trust maintainers who ask questions to do the right thing and take care of their collection. We will check that your CI process runs enough tests at the right time when reviewing your collection for inclusion in Ansible, but no one will continuously check if you are running your tests.

Community members started throwing around some ideas in the ansible-community/community-topics#56 about the future direction of the Ansible package. That issue also talks about regularly assessing the state of included collections, but nothing is fixed yet.

[SHxKM]

Of course. it’s also, or rather mainly, in our interest that all of the tests that can and should run given our constraints are indeed run. And I’m not going to throw them all out after all we’ve gone through in these threads :)

[SHxKM]

It seems like I'm making progress, I guess unit-test ignores also go under tests/sanity/X-Y.txt. How can I now prevent the installation of a dependency in these 2.6/2.7 environments?

[SHxKM]

I think I'm very close, our own unit tests are failing because they're running under Python 2.6. I guess this goes in the CI workflow itself under unit.matrix.python, right? Or should I still tell Ansible somehow not to run own unit tests for X python version?

EDIT: unfortunately defining:

        ansible:
          - stable-2.9 # Only if your collection supports Ansible 2.9
#          - stable-2.10
#          - stable-2.12
#          - stable-2.11
#          - stable-2.12
#          - devel
        python:
          - 3.6
          - 3.7
          - 3.8
          - 3.9

Does not really prevent the runner from trying to run tests with Python 2.6:

Unit test with Python 2.6
....
platform linux4 -- Python 2.6.9, pytest-3.2.5, py-1.4.34, pluggy-0.4.0 -- /tmp/python-wj0rezia-ansible/python
....
plugins: xdist-1.20.1, mock-1.6.3, forked-1.0.1
gw0 I / gw1 I

[gw0] linux4 Python 2.6.9 cwd: /root/ansible/ansible_collections/XXX/YYY

[gw1] linux4 Python 2.6.9 cwd: /root/ansible/ansible_collections/XXX/YYY