sanity tests without --python specified?

[hille721]

Hi,

I'm thinking about if it would make more sense to not set the Python version (--python) for the ansible-test sanity command as this is leading to the fact that the import and compile check for modules is only running for the given Python version, thus basically only for the supported controller Python version but not for the supported remote machine Python versions.

Here the out put with normal tox-ansible behavior for one of my collections:

$ tox -c tox-ansible.ini --ansible -e sanity-py3.9-2.12
...
sanity-py3.9-2.12: commands[0]> bash -c 'cd /foobar/.tox/sanity-py3.9-2.12/tmp/collections/ansible_collections/foo/bar && ansible-test sanity --local --requirements --python 3.9'
Running sanity test "action-plugin-docs"
Running sanity test "ansible-doc"
Running sanity test "changelog"
Running sanity test "compile" on Python 3.9
Running sanity test "empty-init"
Running sanity test "future-import-boilerplate"
Running sanity test "ignores"
Running sanity test "import" on Python 3.9
Running sanity test "line-endings"
Running sanity test "metaclass-boilerplate"
Running sanity test "no-assert"
Running sanity test "no-basestring"
Running sanity test "no-dict-iteritems"
Running sanity test "no-dict-iterkeys"
Running sanity test "no-dict-itervalues"
Running sanity test "no-get-exception"
Running sanity test "no-illegal-filenames"
Running sanity test "no-main-display"
Running sanity test "no-smart-quotes"
Running sanity test "no-unicode-literals"
Running sanity test "pep8"
Running sanity test "pslint"
Running sanity test "pylint"
Running sanity test "replace-urlopen"
Running sanity test "runtime-metadata"
Running sanity test "shebang"
Running sanity test "shellcheck"
Running sanity test "symlinks"
Running sanity test "use-argspec-type-path"
Running sanity test "use-compat-six"
Running sanity test "validate-modules"
Running sanity test "yamllint"
  sanity-py3.9-2.12: OK (23.93=setup[11.75]+cmd[0.00,0.01,1.22,0.01,10.93] seconds)
  congratulations :) (24.15 seconds)

Looking good, everything green.

But know the same simple by not setting the Python version in the ansible-test command (by removing the --python 3.9):

$ tox -c tox-ansible.ini --ansible -e sanity-py3.9-2.12
...
sanity-py3.9-2.12: commands[0]> bash -c 'cd /foobar/.tox/sanity-py3.9-2.12/tmp/collections/ansible_collections/foo/bar && ansible-test sanity --local --requirements '
Running sanity test "action-plugin-docs"
Running sanity test "ansible-doc"
Running sanity test "changelog"
Running sanity test "compile" on Python 2.6
Running sanity test "compile" on Python 2.7
Running sanity test "compile" on Python 3.5
Running sanity test "compile" on Python 3.6
Running sanity test "compile" on Python 3.7
Running sanity test "compile" on Python 3.8
Running sanity test "compile" on Python 3.9
Running sanity test "compile" on Python 3.10
Running sanity test "empty-init"
Running sanity test "future-import-boilerplate"
Running sanity test "ignores"
Running sanity test "import" on Python 2.6
ERROR: Found 1 import issue(s) on python 2.6 which need to be resolved:
ERROR: plugins/modules/maad.py:90:0: traceback: ImportError: cannot import name timezone
See documentation for help: https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/import.html
Running sanity test "import" on Python 2.7
ERROR: Found 1 import issue(s) on python 2.7 which need to be resolved:
ERROR: plugins/modules/maad.py:90:0: traceback: ImportError: cannot import name timezone
See documentation for help: https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/import.html
Running sanity test "import" on Python 3.5
Running sanity test "import" on Python 3.6
Running sanity test "import" on Python 3.7
Running sanity test "import" on Python 3.8
Running sanity test "import" on Python 3.9
Running sanity test "import" on Python 3.10
Running sanity test "line-endings"
Running sanity test "metaclass-boilerplate"
Running sanity test "no-assert"
Running sanity test "no-basestring"
Running sanity test "no-dict-iteritems"
Running sanity test "no-dict-iterkeys"
Running sanity test "no-dict-itervalues"
Running sanity test "no-get-exception"
Running sanity test "no-illegal-filenames"
Running sanity test "no-main-display"
Running sanity test "no-smart-quotes"
Running sanity test "no-unicode-literals"
Running sanity test "pep8"
Running sanity test "pslint"
Running sanity test "pylint"
Running sanity test "replace-urlopen"
Running sanity test "runtime-metadata"
Running sanity test "shebang"
Running sanity test "shellcheck"
Running sanity test "symlinks"
Running sanity test "use-argspec-type-path"
Running sanity test "use-compat-six"
Running sanity test "validate-modules"
Running sanity test "yamllint"
FATAL: The 2 sanity test(s) listed below (out of 46) failed. See error output above for details.
import --python 2.6
import --python 2.7
sanity-py3.9-2.12: exit 1 (15.65 seconds) /foobar> bash -c 'cd /foobar/.tox/sanity-py3.9-2.12/tmp/collections/ansible_collections/foo/bar && ansible-test sanity --local --requirements ' pid=930
  sanity-py3.9-2.12: FAIL code 1 (16.94=setup[0.05]+cmd[0.01,0.01,1.21,0.01,15.65] seconds)
  evaluation failed :( (17.18 seconds)

And now it shows me that I have a problem with one of my modules when they will executed on Python 2

[cidrblock]

I think you are probably right here for 100% coverage, but I think we would need a way to enable/disable it.

So much of the recent content has been written as action plugins (control node execution) or require delegation to localhost, managed node exectuion is really limited to linux/windows where I would "hope" not many people are still using python2 as it's pretty far behind us in the rear view mirror.

There is a little bit of a grey area here when comparing the guidelines between the guidelines for ansible package inclusion and pushing content to automation hub for RH customers as "certified" or "validated". Platform users will be using execution environments for control node execution and have more control over the controller python version.

Many collections have dropped python2 as well because the python packages they depend on have dropped it, and it's impossible to support a collection on py2 using a 3rd party library that doesn't.

If you need/want it a PR would be great. For the collections maintained by the teams I work with, I don't think we have a need for it.

[hille721]

For now I solved it by simple overwriting the command in my tox-ansible.ini:

[testenv:sanity-{py3.8,py3.9,py3.10,py3.11}-{2.12,2.13,2.14,2.15,devel}]
commands =
    bash -c 'cd {envdir}/tmp/collections/ansible_collections/ifx/community && ansible-test sanity --local --requirements {posargs}'

I can live with that as well :)