tox -e py 37 doesnt pickup glance tests

I can't reproduce this. When I do:

git checkout master
git fetch --all
git pull origin master
tox -r -e py37

I get:

======
Totals
======
Ran: 1853 tests in 9.0883 sec.
 - Passed: 1835
 - Skipped: 18
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 67.7211 sec.

(Note the '-r' option to tox ... it rebuilds the virtualenv, maybe you are running with an outdated env?)

I rebuilt the env, executed the above:

ource ~/virtualenvs/glance-stein/bin/activate
(glance-stein) C02T8EARGTFM:glance$ git checkout master
M tox.ini
Already on 'master'
Your branch is up to date with 'origin/master'.
(glance-stein) C02T8EARGTFM:glance$ git fetch --all
Fetching origin
(glance-stein) C02T8EARGTFM:glance$ git pull origin master
From https://github.com/sapcc/glance
 * branch master -> FETCH_HEAD
Already up to date.
(glance-stein) C02T8EARGTFM:glance$ tox -r -e py37
py37 recreate: /Users/sapcc/stein/glance/.tox/py37
py37 installdeps: -r/Users/sapcc/stein/glance/test-requirements.txt
py37 develop-inst: /Users/sapcc/stein/glance
py37 installed: /Users/sapcc/stein/glance/.tox/py37/lib/python3.7/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working, from collections import Mapping,alabaster==0.7.12,alembic==1.0.11,amqp==2.5.0,appdirs==1.4.3,asn1crypto==0.24.0,automaton==1.16.0,Babel==2.7.0,cachetools==3.1.1,castellan==1.3.0,certifi==2019.6.16,cffi==1.12.3,chardet==3.0.4,cliff==2.15.0,cmd2==0.8.9,coverage==4.5.3,cryptography==2.7,cursive==0.2.2,ddt==1.2.1,debtcollector==1.21.0,decorator==4.4.0,defusedxml==0.6.0,dnspython==1.15.0,doc8==0.8.0,docutils==0.14,dogpile.cache==0.7.1,eventlet==0.25.0,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.5.5,future==0.17.1,futurist==1.8.1,-e git+https://github.com/sapcc/glance.git@e33841879c959b60f97bf22d936741961b7a0bcf#egg=glance,glance-store==0.29.1,greenlet==0.4.15,hacking==0.12.0,httplib2==0.13.0,idna==2.8,imagesize==1.1.0,iso8601==0.1.12,Jinja2==2.10.1,jmespath==0.9.4,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,keystoneauth1==3.14.0,keystonemiddleware==6.1.0,kombu==4.6.3,linecache2==1.0.0,Mako==1.0.12,MarkupSafe==1.1.1,mccabe==0.2.1,mock==3.0.5,monotonic==1.5,mox3==0.27.0,msgpack==0.6.1,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.9,networkx==2.3,openstacksdk==0.31.1,os-client-config==1.32.0,os-service-types==1.7.0,os-testr==1.0.0,oslo.cache==1.36.0,oslo.concurrency==3.29.1,oslo.config==6.10.0,oslo.context==2.22.1,oslo.db==5.0.1,oslo.i18n==3.23.1,oslo.log==3.44.0,oslo.messaging==9.7.2,oslo.middleware==3.38.0,oslo.policy==2.2.0,oslo.serialization==2.29.1,oslo.service==1.40.0,oslo.upgradecheck==0.2.1,oslo.utils==3.41.0,oslotest==3.7.1,osprofiler==2.8.0,packaging==19.0,Paste==3.0.8,PasteDeploy==2.0.1,pbr==5.3.1,pep8==1.5.7,prettytable==0.7.2,psutil==5.6.3,psycopg2==2.8.3,pycadf==2.9.0,pycparser==2.19,pydot==1.4.1,pyflakes==0.8.1,Pygments==2.4.2,PyMySQL==0.9.3,pyOpenSSL==19.0.0,pyparsing==2.4.0,pyperclip==1.7.0,pysendfile==2.0.1,python-barbicanclient==4.8.1,python-dateutil==2.8.0,python-editor==1.0.4,python-keystoneclient==3.19.0,python-mimeparse==1.6.0,python-subunit==1.3.0,python-swiftclient==3.7.0,pytz==2019.1,PyYAML==5.1.1,repoze.lru==0.7,requests==2.22.0,requestsexceptions==1.4.0,restructuredtext-lint==1.3.0,retrying==1.3.3,rfc3986==1.3.2,Routes==2.4.1,simplegeneric==0.8.1,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.1.2,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,SQLAlchemy==1.3.5,sqlalchemy-migrate==0.12.0,sqlparse==0.3.0,statsd==3.3.0,stestr==2.3.1,stevedore==1.30.1,taskflow==3.7.0,Tempita==0.5.2,tenacity==5.0.4,testrepository==0.0.20,testresources==2.0.1,testscenarios==0.5.0,testtools==2.3.0,traceback2==1.4.0,unittest2==1.1.0,urllib3==1.25.3,vine==1.3.0,voluptuous==0.11.5,wcwidth==0.1.7,WebOb==1.8.5,wrapt==1.11.2,WSME==0.9.3,xattr==0.9.6,yappi==1.0
py37 run-test-pre: PYTHONHASHSEED='2345561484'
py37 run-test: commands[0] | stestr slowest
/Users/sapcc/stein/glance/.tox/py37/lib/python3.7/site-packages/unittest2/compatibility.py:143: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  class ChainMap(collections.MutableMapping):
/Users/sapcc/stein/glance/.tox/py37/lib/python3.7/site-packages/future/standard_library/__init__.py:65: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
ERROR: InvocationError for command /Users/sapcc/stein/glance/.tox/py37/bin/stestr slowest (exited with code 3)
__________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
ERROR: py37: commands failed

We run our infra on k8s containers, when i run the below, the tests are being picked up:

kubectl run -i glance-stein-unit-test --rm --image=ubuntu --restart=Never --command -- /bin/bash -c 'apt-get update && apt-get install -y build-essential python-pip python-dev python3-dev python-psycopg2 git libpq-dev libpcre++-dev gettext && pip install tox && pip install --upgrade pip && git clone -b stable/stein-m3 --single-branch https://github.com/sapcc/glance.git && cd glance && UPPER_CONSTRAINTS_FILE=https://raw.githubusercontent.com/sapcc/requirements/stable/stein-m3/upper-constraints.txt tox -r -e py36'

y36 develop-inst: /glance
py36 installed: alabaster==0.7.12,alembic==1.0.8,amqp==2.4.2,appdirs==1.4.3,asn1crypto==0.24.0,automaton==1.16.0,Babel==2.6.0,cachetools==3.1.0,castellan==1.2.2,certifi==2018.11.29,cffi==1.12.2,chardet==3.0.4,cliff==2.14.1,cmd2==0.8.9,coverage==4.5.2,cryptography==2.6.1,cursive==0.2.2,ddt==1.2.1,debtcollector==1.21.0,decorator==4.3.2,defusedxml==0.5.0,doc8==0.8.0,docutils==0.14,dogpile.cache==0.7.1,enum-compat==0.0.2,eventlet==0.20.0,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.5.5,future==0.17.1,futurist==1.8.1,-e git+https://github.com/sapcc/glance.git@bbd49c133ddb95dd207a363c0d95c6d351040dd0#egg=glance,glance-store==0.28.0,greenlet==0.4.15,hacking==0.12.0,httplib2==0.12.1,idna==2.8,imagesize==1.1.0,iso8601==0.1.12,Jinja2==2.10,jmespath==0.9.4,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,keystoneauth1==3.13.1,keystonemiddleware==5.3.1.dev24,kombu==4.4.0,linecache2==1.0.0,Mako==1.0.7,MarkupSafe==1.1.1,mccabe==0.2.1,mock==2.0.0,monotonic==1.5,mox3==0.27.0,msgpack==0.6.1,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.9,networkx==2.2,openstacksdk==0.27.0,os-client-config==1.32.0,os-service-types==1.6.0,os-testr==1.0.0,oslo.cache==1.33.3,oslo.concurrency==3.29.1,oslo.config==6.8.1,oslo.context==2.22.1,oslo.db==4.45.0,oslo.i18n==3.23.1,oslo.log==3.42.3,oslo.messaging==9.5.0,oslo.middleware==3.37.1,oslo.policy==2.1.1,oslo.serialization==2.28.2,oslo.service==1.38.0,oslo.upgradecheck==0.2.1,oslo.utils==3.40.3,oslotest==3.7.1,osprofiler==2.6.0,packaging==19.0,Paste==3.0.7,PasteDeploy==2.0.1,pbr==5.1.3,pep8==1.5.7,prettytable==0.7.2,psutil==5.6.0,psycopg2==2.7.7,pycadf==2.9.0,pycparser==2.19,pydot==1.4.1,pyflakes==0.8.1,Pygments==2.4.2,pyinotify==0.9.6,PyMySQL==0.9.3,pyOpenSSL==19.0.0,pyparsing==2.3.1,pyperclip==1.7.0,pysendfile==2.0.1,python-barbicanclient==4.8.1,python-dateutil==2.8.0,python-editor==1.0.4,python-keystoneclient==3.19.0,python-mimeparse==1.6.0,python-subunit==1.3.0,python-swiftclient==3.7.0,pytz==2018.9,PyYAML==3.13,repoze.lru==0.7,requests==2.21.0,requestsexceptions==1.4.0,restructuredtext-lint==1.2.2,retrying==1.3.3,rfc3986==1.2.0,Routes==2.4.1,simplegeneric==0.8.1,six==1.12.0,snowballstemmer==1.2.1,Sphinx==1.8.4,sphinxcontrib-websupport==1.1.0,SQLAlchemy==1.2.18,sqlalchemy-migrate==0.12.0,sqlparse==0.2.4,statsd==3.3.0,stestr==2.2.0,stevedore==1.30.1,taskflow==3.5.0,Tempita==0.5.2,tenacity==5.0.3,testrepository==0.0.20,testresources==2.0.1,testscenarios==0.5.0,testtools==2.3.0,traceback2==1.4.0,unittest2==1.1.0,urllib3==1.24.1,vine==1.2.0,voluptuous==0.11.5,wcwidth==0.1.7,WebOb==1.8.5,wrapt==1.11.1,WSME==0.9.3,xattr==0.9.6,yappi==1.0
py36 run-test-pre: PYTHONHASHSEED='2919339678'
py36 run-test: commands[0] | find . -type f -name '*.pyc' -delete
___________________________________ summary ____________________________________
  py36: commands succeeded
  congratulations :)

awaiting a reply

Not sure what's going on, here are a few guesses:

1. When you did the 'git checkout master', it said that your tox.ini was modified. Do a 'git diff' to see what the difference is.

2. My output has the line:
py37 runtests: commands[0] | stestr run --slowest

whereas yours has
py37 run-test: commands[0] | stestr slowest

the stuff before the '|' is coming from tox; maybe it's a tox version problem? (I'm using 3.4.0). I wonder if you're using an older tox; you do a 'pip install tox' when you run the py36 tests in the container, which would give you the newest version of tox, maybe that's why the tests run in the container.

the stuff after the '|' is defined in tox.ini (could be that's the diff in your tox.ini?)

1) Difference was i restricted `envlist to py37`

2) Good and bad news:

GOOD News:

Post implementing the above changes in tox.ini i.e.

a) downgrade to tox==3.4.0 it works fine
b) change ostestr to stestr run --slowest

BAD News:

a) I am using the latest tox version, Stein doesnt support tox v3.12.1 ?
b) why hasnt there been any changes to https://github.com/openstack/glance/blob/stable/stein/tox.ini#L39 ?

======
Totals
======
Ran: 1853 tests in 12.0599 sec.
 - Passed: 1852
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 1
Sum of execute time for each test: 105.8513 sec.

glance.tests.unit.async_.flows.test_import.TestImportTask.test_import_flow_revert 0.654
glance.tests.unit.test_misc.UtilsTestCase.test_encryption 0.651
glance.tests.unit.common.test_property_utils.TestPropertyRulesWithPolicies.test_check_property_rules_create_all_permitted 0.625
ERROR: InvocationError for command '/glance/.tox/py36/bin/stestr run --slowest' (exited with code 1)
___________________________________ summary ____________________________________
ERROR: py36: commands failed

Not sure, what i am missing now, any suggestions ?

i get the below when i run stestr run:

======
Totals
======
Ran: 1853 tests in 12.9945 sec.
 - Passed: 1853
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 118.6251 sec.

To sum-up the above:

For me all tests pass with tox==3.4.0 and stestr run stestr run {posargs}. How do you suggest i go forward ?

Regards,
Rajiv

Hi Brian,

Any update? is this a bug in latest tox release ?

Regards,
Rajiv

I don't think this a tox bug. I can get the tests to run fine with tox 3.4.0 and tox 3.9.0

Glance stable/stein still uses os-testr:
https://opendev.org/openstack/glance/src/branch/stable/stein/test-requirements.txt#L24
so these lines are correct in stable/stein tox.ini:
https://opendev.org/openstack/glance/src/branch/stable/stein/tox.ini#L37-L39

I can run the tests fine locally:
[master] wha'ppen? git fetch --all
Fetching origin
Fetching gerrit
[master] wha'ppen? git checkout stable/stein
Switched to branch 'stable/stein'
Your branch is up to date with 'origin/stable/stein'.
[stable/stein] wha'ppen? git lg -n 1
f7723218 3 months ago (HEAD -> stable/stein, origin/stable/stein, gerrit/stable/stein) OpenDev Migration Patch (OpenDev Sysadmins)
[stable/stein] wha'ppen? tox -r -e py37
py37 recreate: /home/brosmait/repos/openstack/glance/.tox/py37
py37 installdeps: -r/home/brosmait/repos/openstack/glance/test-requirements.txt
py37 develop-inst: /home/brosmait/repos/openstack/glance
py37 installed: /home/brosmait/repos/openstack/glance/.tox/py37/lib/python3.7/site.py:165: DeprecationWarning: 'U' mode is deprecated, f = open(fullname, "rU"),/home/brosmait/repos/openstack/glance/.tox/py37/lib/python3.7/distutils/__init__.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses, import imp,/home/brosmait/repos/openstack/glance/.tox/py37/lib/python3.7/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working, from collections import Mapping,alabaster==0.7.12,alembic==1.0.8,amqp==2.4.2,appdirs==1.4.3,asn1crypto==0.24.0,automaton==1.16.0,Babel==2.6.0,cachetools==3.1.0,castellan==1.2.2,certifi==2018.11.29,cffi==1.12.2,chardet==3.0.4,cliff==2.14.1,cmd2==0.8.9,coverage==4.5.2,cryptography==2.6.1,cursive==0.2.2,ddt==1.2.1,debtcollector==1.21.0,decorator==4.3.2,defusedxml==0.5.0,dnspython==1.16.0,doc8==0.8.0,docutils==0.14,dogpile.cache==0.7.1,eventlet==0.24.1,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.5.5,future==0.17.1,futurist==1.8.1,-e git+https://opendev.org/openstack/glance@f77232182c56cb4d4dcafadfbf086ce4aa09fa1e#egg=glance,glance-store==0.28.0,greenlet==0.4.15,hacking==0.12.0,httplib2==0.12.1,idna==2.8,imagesize==1.1.0,iso8601==0.1.12,Jinja2==2.10,jmespath==0.9.4,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,keystoneauth1==3.13.1,keystonemiddleware==6.0.0,kombu==4.4.0,linecache2==1.0.0,Mako==1.0.7,MarkupSafe==1.1.1,mccabe==0.2.1,mock==2.0.0,monotonic==1.5,mox3==0.27.0,msgpack==0.6.1,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.9,networkx==2.2,openstacksdk==0.27.0,os-client-config==1.32.0,os-service-types==1.6.0,os-testr==1.0.0,os-win==4.2.0,oslo.cache==1.33.3,oslo.concurrency==3.29.1,oslo.config==6.8.1,oslo.context==2.22.1,oslo.db==4.45.0,oslo.i18n==3.23.1,oslo.log==3.42.3,oslo.messaging==9.5.0,oslo.middleware==3.37.1,oslo.policy==2.1.1,oslo.serialization==2.28.2,oslo.service==1.38.0,oslo.upgradecheck==0.2.1,oslo.utils==3.40.3,oslotest==3.7.1,osprofiler==2.6.0,packaging==19.0,Paste==3.0.7,PasteDeploy==2.0.1,pbr==5.1.3,pep8==1.5.7,prettytable==0.7.2,psutil==5.6.0,psycopg2==2.7.7,pycadf==2.9.0,pycparser==2.19,pydot==1.4.1,pyflakes==0.8.1,Pygments==2.4.2,pyinotify==0.9.6,PyMySQL==0.9.3,pyOpenSSL==19.0.0,pyparsing==2.3.1,pyperclip==1.7.0,pysendfile==2.0.1,python-barbicanclient==4.8.1,python-dateutil==2.8.0,python-editor==1.0.4,python-keystoneclient==3.19.0,python-mimeparse==1.6.0,python-subunit==1.3.0,python-swiftclient==3.7.0,pytz==2018.9,PyYAML==3.13,repoze.lru==0.7,requests==2.21.0,requestsexceptions==1.4.0,restructuredtext-lint==1.2.2,retrying==1.3.3,rfc3986==1.2.0,Routes==2.4.1,simplegeneric==0.8.1,six==1.12.0,snowballstemmer==1.2.1,Sphinx==1.8.4,sphinxcontrib-websupport==1.1.0,SQLAlchemy==1.2.18,sqlalchemy-migrate==0.12.0,sqlparse==0.2.4,statsd==3.3.0,stestr==2.2.0,stevedore==1.30.1,taskflow==3.5.0,Tempita==0.5.2,tenacity==5.0.3,testrepository==0.0.20,testresources==2.0.1,testscenarios==0.5.0,testtools==2.3.0,traceback2==1.4.0,unittest2==1.1.0,urllib3==1.24.1,vine==1.2.0,voluptuous==0.11.5,wcwidth==0.1.7,WebOb==1.8.5,wrapt==1.11.1,WSME==0.9.3,xattr==0.9.6,yappi==1.0
py37 run-test-pre: PYTHONHASHSEED='3750445202'
py37 runtests: commands[0] | ostestr --slowest
[snip]
======
Totals
======
Ran: 1853 tests in 10.6932 sec.
 - Passed: 1835
 - Skipped: 18
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 73.7338 sec.

==============
Worker Balance
==============
 - Worker 0 (223 tests) => 0:00:10.237251
 - Worker 1 (232 tests) => 0:00:09.045271
 - Worker 2 (232 tests) => 0:00:09.263069
 - Worker 3 (233 tests) => 0:00:08.680933
 - Worker 4 (233 tests) => 0:00:10.261483
 - Worker 5 (233 tests) => 0:00:09.361644
 - Worker 6 (233 tests) => 0:00:09.446217
 - Worker 7 (234 tests) => 0:00:09.217904

Slowest Tests:
Test id Runtime (s)
-------------------------------------------------------------------------------------------------------------------- -----------
glance.tests.unit.test_db.RetryOnDeadlockTestCase.test_raise_deadlock 2.155
glance.tests.unit.test_misc.UtilsTestCase.test_encryption 1.050
glance.tests.unit.async_.flows.test_import.TestImportTask.test_import_flow_no_import_flows 0.446
glance.tests.unit.common.test_utils.TestUtils.test_cooperative_reader_preserves_size_chunk_more_then_read 0.397
glance.tests.unit.async_.flows.test_convert.TestImportTask.test_import_flow_with_convert_and_introspect 0.393
glance.tests.unit.common.test_wsgi.RequestTest.test_http_error_response_codes 0.342
glance.tests.unit.common.test_wsgi.GetSocketTestCase.test_get_socket_with_bind_problems 0.307
glance.tests.unit.v2.test_registry_api.TestRegistryRPC.test_get_index_filter_name 0.303
glance.tests.unit.async_.flows.test_import.TestImportTask.test_import_flow_revert_import_to_fs 0.292
glance.tests.unit.v2.test_images_resource.TestImagesController.test_prop_protection_with_update_and_permitted_policy 0.282
____________________________________________________________ summary _____________________________________________________________
  py37: commands succeeded
  congratulations :)

I suspect there's some kind of problem with your environment. Make sure that your stable/stein tox.ini is exactly like the one in the git repository for stable/stein and try again.

If you still have problems, you can file a Glance bug explaining the problem and propose a backport to stable/stein of the commit that removed os-testr in master:
https://review.opendev.org/#/c/578529/

(Note: you'll get a conflict when you backport that patch; there's not a py36 environment in stable/stein tox.ini, so don't include it in the backport.)

I don't know what the odds are of the stable-maint team accepting the backport; as far as I can tell, everything is working in stable/stein right now. One argument you could make is that stable/stein is using os-testr for unit tests and stestr for functional tests, it would be better to be consistent. I don't know if that's a good argument for a backport, though. So it will depend on what you can figure out about why the tests won't run in your containerized environment, and whether this problem will prevent us from backporting other changes to stable/stein (all the tests have to pass for backport to be accepted).

I did the cherry-pick I described above locally, and it doesn't seem to break anything; but on the other hand, nothing seems to be broken now, and the stable-maint team tends to be very conservative about allowing changes into the stable branches.