Step 10. Verifying cloud via Tempest

In this guide, we are going to show how to use Tempest and Rally together. We assume that you have a Rally installation and an already registered OpenStack deployment in Rally. So let's get started!

Tempest installation (rally verify install/uninstall/reinstall)

In order to install Tempest execute the following command:

$ rally verify install
2016-05-09 13:23:51.897 21850 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:23:51.897 21850 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:23:51.908 21850 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-ljZwwS'...
remote: Counting objects: 70000, done.
remote: Compressing objects: 100% (37320/37320), done.
remote: Total 70000 (delta 53645), reused 47857 (delta 32525)
Receiving objects: 100% (70000/70000), 9.92 MiB | 1.03 MiB/s, done.
Resolving deltas: 100% (53645/53645), done.
Checking connectivity... done.
2016-05-09 13:24:11.180 21850 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:24:25.596 21850 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!

By default, the command clones Tempest from the https://git.openstack.org/openstack/tempest repository and installs it for the current deployment. But it is possible to install Tempest for any registered deployment in Rally, using the --deployment argument.

$ rally verify install --deployment <UUID or name of a deployment>

Also, Rally allows users to specify a source to clone Tempest from, using the --source argument. The source can be both the path to a local Tempest repository and a URL, e.g. to some GitHub repository.

$ rally verify install --source /home/ubuntu/tempest/
2016-05-09 13:29:05.004 22382 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:29:05.004 22382 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:29:05.013 22382 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-pscTA7'...
done.
2016-05-09 13:29:05.902 22382 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:29:18.052 22382 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
$ rally verify install --source https://github.com/openstack/tempest.git
2016-05-09 13:30:15.804 22541 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:30:15.804 22541 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:30:15.814 22541 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-BLUv1E'...
remote: Counting objects: 70000, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 70000 (delta 0), reused 0 (delta 0), pack-reused 69993
Receiving objects: 100% (70000/70000), 20.66 MiB | 2.67 MiB/s, done.
Resolving deltas: 100% (52246/52246), done.
Checking connectivity... done.
2016-05-09 13:30:37.602 22541 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:30:49.432 22541 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!

Moreover, it is possible to specify a Tempest commit ID or tag, using the --version argument, to install a certain version of Tempest.

$ rally verify install --source /home/ubuntu/tempest/ --version 198e5b4b871c3d09c20afb56dca9637a8cf86ac8
2016-05-09 13:45:55.764 23259 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:45:55.764 23259 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:45:55.773 23259 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-TcKvAX'...
done.
T   doc/source/HACKING.rst
T   doc/source/REVIEWING.rst
T   doc/source/field_guide/api.rst
T   doc/source/field_guide/index.rst
T   doc/source/field_guide/scenario.rst
T   doc/source/field_guide/stress.rst
T   doc/source/field_guide/unit_tests.rst
T   doc/source/overview.rst
Note: checking out '198e5b4b871c3d09c20afb56dca9637a8cf86ac8'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 198e5b4... Merge "Pass server to RemoteClient in API tests"
2016-05-09 13:45:56.061 23259 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:46:15.278 23259 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
$ rally verify install --source /home/ubuntu/tempest/ --version 10.0.0
2016-05-09 13:50:42.559 23870 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:50:42.559 23870 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:50:42.568 23870 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-cUe5p8'...
done.
T   doc/source/HACKING.rst
T   doc/source/REVIEWING.rst
T   doc/source/field_guide/api.rst
T   doc/source/field_guide/index.rst
T   doc/source/field_guide/scenario.rst
T   doc/source/field_guide/stress.rst
T   doc/source/field_guide/unit_tests.rst
T   doc/source/overview.rst
Note: checking out '10.0.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 09a6015... Merge "Make data_processing/baremetal use rest_client"
2016-05-09 13:50:42.903 23870 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:50:55.827 23870 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!

Finally, users can specify the --system-wide argument that will tell Rally not to create a virtual environment for Tempest. In this case, it is assumed that all Tempest requirements are already installed in the local environment.

$ rally verify install --source /home/ubuntu/tempest/ --version 10.0.0 --system-wide
2016-05-09 13:52:34.085 24216 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:52:34.085 24216 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:52:34.094 24216 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-8jFGJU'...
done.
T   doc/source/HACKING.rst
T   doc/source/REVIEWING.rst
T   doc/source/field_guide/api.rst
T   doc/source/field_guide/index.rst
T   doc/source/field_guide/scenario.rst
T   doc/source/field_guide/stress.rst
T   doc/source/field_guide/unit_tests.rst
T   doc/source/overview.rst
Note: checking out '10.0.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 09a6015... Merge "Make data_processing/baremetal use rest_client"
2016-05-09 13:52:34.519 24216 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!

To remove a local Tempest installation for the current deployment execute the following command:

$ rally verify uninstall

Also, it is possible to remove the local Tempest installation for any registered deployment in Rally, using the --deployment argument.

$ rally verify uninstall --deployment <UUID or name of a deployment>

In addition, there is the

$ rally verify reinstall

command that combines two commands:

$ rally verify uninstall
$ rally verify install

Arguments for rally verify reinstall are the same like for rally verify install.

Tempest config generation (rally verify genconfig/showconfig)

In order to generate a Tempest config file execute the following command:

$ rally verify genconfig
2016-05-09 14:31:48.050 25906 INFO rally.verification.tempest.tempest [-] Tempest is not configured for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:31:48.050 25906 INFO rally.verification.tempest.tempest [-] Creating Tempest configuration file for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:31:56.738 25906 INFO rally.verification.tempest.tempest [-] Tempest configuration file has been successfully created!

By default, the command generates the config file for the current deployment, but it is possible to generate the config file for any registered deployment in Rally, using the --deployment argument.

$ rally verify genconfig --deployment <UUID or name of a deployment>

Also, Rally allows users to specify any local path to the future config file, e.g. /home/ubuntu/tempest.conf, and in this case, Rally will generate the tempest.conf file in the /home/ubuntu/ directory.

$ rally verify genconfig --tempest-config /home/ubuntu/tempest.conf
2016-05-09 14:34:07.619 26204 INFO rally.verification.tempest.tempest [-] Tempest is not configured for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:34:07.619 26204 INFO rally.verification.tempest.tempest [-] Creating Tempest configuration file for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:34:09.449 26204 INFO rally.verification.tempest.tempest [-] Tempest configuration file has been successfully created!

Moreover, it is possible to override the existing Tempest config file by providing the --override argument in the rally verify genconfig command:

$ rally verify genconfig --override
2016-05-09 14:35:11.608 26270 INFO rally.verification.tempest.tempest [-] Creating Tempest configuration file for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:35:13.395 26270 INFO rally.verification.tempest.tempest [-] Tempest configuration file has been successfully created!

In order to see the generated config file execute the following command:

$ rally verify showconfig
Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf

[DEFAULT]
debug = True
log_file = tempest.log
use_stderr = False

[auth]
use_dynamic_credentials = True
...

To see the generated config file for a certain deployment specify the --deployment argument.

$ rally verify showconfig --deployment <UUID or name of a deployment>

Start a verification (rally verify start)

In order to start a verification execute the following command:

$ rally verify start
2016-05-09 14:54:07.446 27377 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:54:07.529 27377 INFO rally.verification.tempest.tempest [-] Verification de083a94-8b42-46fe-9cdd-2b6066f9c13c | Starting:  Run verification.
2016-05-09 14:54:07.613 27377 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpcbg8BK
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpJEOWsG
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpD8Hsxu
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmp2UQC55
{1} setUpClass (tempest.api.baremetal.admin.test_ports_negative.TestPortsNegative) ... SKIPPED: TestPortsNegative skipped as Ironic is not available
{2} setUpClass (tempest.api.baremetal.admin.test_api_discovery.TestApiDiscovery) ... SKIPPED: TestApiDiscovery skipped as Ironic is not available
{2} setUpClass (tempest.api.baremetal.admin.test_chassis.TestChassis) ... SKIPPED: TestChassis skipped as Ironic is not available
{2} setUpClass (tempest.api.baremetal.admin.test_drivers.TestDrivers) ... SKIPPED: TestDrivers skipped as Ironic is not available
{3} setUpClass (tempest.api.baremetal.admin.test_nodes.TestNodes) ... SKIPPED: TestNodes skipped as Ironic is not available
{3} setUpClass (tempest.api.baremetal.admin.test_ports.TestPorts) ... SKIPPED: TestPorts skipped as Ironic is not available
{0} setUpClass (tempest.api.baremetal.admin.test_nodestates.TestNodeStates) ... SKIPPED: TestNodeStates skipped as Ironic is not available
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [0.712663s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.502782s] ... ok
{3} tempest.api.compute.admin.test_flavors_access_negative.FlavorsAccessNegativeTestJSON.test_add_flavor_access_duplicate [1.011901s] ... ok
...

By default, the command runs the full suite of Tempest tests for the current deployment, but it is possible to run the tests for any registered deployment in Rally, using the --deployment argument.

$ rally verify start --deployment <UUID or name of a deployment>

Also, Rally allows users to specify a certain Tempest config file location to use a certain Tempest config file for running the tests.

$ rally verify start --tempest-config /home/ubuntu/tempest.conf
2016-05-09 15:24:02.474 29197 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:24:02.558 29197 INFO rally.verification.tempest.tempest [-] Verification 85b90b77-ee32-4e56-83ed-aabf306cb509 | Starting:  Run verification.
2016-05-09 15:24:02.641 29197 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpqJcBEn
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmplKu5tZ
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpww2PLm
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmp6ip_UK
{0} setUpClass (tempest.api.baremetal.admin.test_api_discovery.TestApiDiscovery) ... SKIPPED: TestApiDiscovery skipped as Ironic is not available
{0} setUpClass (tempest.api.baremetal.admin.test_ports.TestPorts) ... SKIPPED: TestPorts skipped as Ironic is not available
{0} setUpClass (tempest.api.baremetal.admin.test_ports_negative.TestPortsNegative) ... SKIPPED: TestPortsNegative skipped as Ironic is not available
{3} setUpClass (tempest.api.baremetal.admin.test_nodestates.TestNodeStates) ... SKIPPED: TestNodeStates skipped as Ironic is not available
{3} setUpClass (tempest.api.compute.admin.test_fixed_ips.FixedIPsTestJson) ... SKIPPED: FixedIPsTestJson skipped as neutron is available
{1} setUpClass (tempest.api.baremetal.admin.test_chassis.TestChassis) ... SKIPPED: TestChassis skipped as Ironic is not available
{1} setUpClass (tempest.api.baremetal.admin.test_drivers.TestDrivers) ... SKIPPED: TestDrivers skipped as Ironic is not available
{1} setUpClass (tempest.api.baremetal.admin.test_nodes.TestNodes) ... SKIPPED: TestNodes skipped as Ironic is not available
{3} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram [0.642174s] ... ok
{0} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_existent_host [1.069448s] ... ok

Also, there is a possibility to run a certain suite of Tempest tests, using the --set argument.

$ rally verify start --set compute
2016-05-09 14:56:45.258 27685 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:56:45.342 27685 INFO rally.verification.tempest.tempest [-] Verification ab0acb96-f664-438a-8323-198fe68d8a96 | Starting:  Run verification.
2016-05-09 14:56:45.425 27685 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpm1QuaD
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpxmGWlN
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpsaG1BU
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpbZzU2y
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_existent_host [1.623109s] ... ok
{3} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [1.125569s] ... FAILED
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_host_as_user [2.267328s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [2.507743s] ... ok
{0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list [1.132218s] ... ok
{0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list_detail [0.518452s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.796207s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents [0.735133s] ... ok
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_non_exist_host [1.941015s] ... ok
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_create_aggregate_name_length_exceeds_255 [0.183736s] ... ok
...

For now, available sets are full, scenario, smoke, baremetal, compute, database, data_processing, identity, image, messaging, network, object_storage, orchestration, telemetry, volume.

Moreover, users can run a certain set of tests, using the --regex argument and specifying a regular expression.

$ rally verify start --regex tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON
2016-05-09 15:04:50.089 28117 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:04:50.173 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Starting:  Run verification.
2016-05-09 15:04:50.257 28117 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmp3QMRkn
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram [0.574063s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_verify_entry_in_list_details [0.539422s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_int_id [0.542389s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_none_id [0.525429s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_uuid_id [0.539657s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_list_flavor_without_extra_data [0.782256s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_server_with_non_public_flavor [0.536828s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_is_public_string_variations [1.931141s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_non_public_flavor [0.691936s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_public_flavor_with_other_user [0.569325s] ... ok

======
Totals
======
Ran: 10 tests in 18.0000 sec.
 - Passed: 10
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 7.2324 sec.

==============
Worker Balance
==============
 - Worker 0 (10 tests) => 0:00:07.236862
2016-05-09 15:05:10.473 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Completed: Run verification.
2016-05-09 15:05:10.474 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Starting:  Saving verification results.
2016-05-09 15:05:10.677 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Completed: Saving verification results.
Verification UUID: 32348bcc-edf1-4434-a10b-9449e2370a16

In such a way it is possible to run tests from a certain directory or class and even run a single test.

$ rally verify start --regex tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram
2016-05-09 15:06:18.088 28217 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:06:18.170 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Starting:  Run verification.
2016-05-09 15:06:18.254 28217 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpoEkv6Q
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram [0.547252s] ... ok

======
Totals
======
Ran: 1 tests in 10.0000 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 0.5473 sec.

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:00:00.547252
2016-05-09 15:06:31.207 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Completed: Run verification.
2016-05-09 15:06:31.207 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Starting:  Saving verification results.
2016-05-09 15:06:31.750 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Completed: Saving verification results.
Verification UUID: dbd4bc2d-2b76-42b7-b737-fce86a92fbfa

Also, there is a possibility to run Tempest tests from a file. Users can specify a list of tests in the file and run them, using the --tests-file argument:

$ cat some-file.txt
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent[id-1fc6bdc8-0b6d-4cc7-9f30-9b04fabe5b90]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent[id-470e0b89-386f-407b-91fd-819737d0b335]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents[id-6a326c69-654b-438a-80a3-34bcc454e138]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter[id-eabadde4-3cd7-4ec4-a4b5-5a936d2d4408]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent[id-dc9ffd51-1c50-4f0e-a820-ae6d2a568a9e]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details[id-eeef473c-7c52-494d-9f09-2ed7fc8fc036]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list[id-7f6a1cc5-2446-4cdb-9baa-b6ae0a919b72]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host[id-c8e85064-e79b-4906-9931-c11c24294d02]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete[id-0d148aa3-d54c-4317-aa8d-42040a475e20]
$ rally verify start --tests-file some-file.txt
2016-05-09 15:09:10.864 28456 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:09:10.948 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Starting:  Run verification.
2016-05-09 15:09:11.033 28456 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpjHUGip
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmp358n_n
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [0.601839s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.501781s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents [0.375056s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [1.036974s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter [0.640392s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.850647s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent [0.371227s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.803282s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.635170s] ... ok

======
Totals
======
Ran: 9 tests in 11.0000 sec.
 - Passed: 9
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 5.8164 sec.

==============
Worker Balance
==============
 - Worker 0 (4 tests) => 0:00:03.328229
 - Worker 1 (5 tests) => 0:00:02.492475
2016-05-09 15:09:24.668 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Completed: Run verification.
2016-05-09 15:09:24.669 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Starting:  Saving verification results.
2016-05-09 15:09:24.872 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Completed: Saving verification results.
Verification UUID: 526b0c54-3805-48eb-8a04-4fec0aad3fe5

Sometimes users may want to use the specific concurrency for running tests based on their deployments and available resources. In this case, they can use the --concurrency argument to specify how many processes to use to run Tempest tests. The default value (0) auto-detects CPU count.

$ rally verify start --tests-file some-file.txt --concurrency 1
2016-05-09 15:10:39.050 28744 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:10:39.132 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Starting:  Run verification.
2016-05-09 15:10:39.216 28744 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpl_FWjP
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [0.586906s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.499466s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents [0.370536s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter [0.620824s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent [0.365948s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [0.942561s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.897054s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.743319s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.629131s] ... ok

======
Totals
======
Ran: 9 tests in 16.0000 sec.
 - Passed: 9
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 5.6557 sec.

==============
Worker Balance
==============
 - Worker 0 (9 tests) => 0:00:09.701447
2016-05-09 15:10:57.861 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Completed: Run verification.
2016-05-09 15:10:57.861 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Starting:  Saving verification results.
2016-05-09 15:10:58.173 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Completed: Saving verification results.
Verification UUID: 95fef399-0cfa-4843-ad50-b5ed974928dc

Sometimes users may want to re-run only those tests that failed in the last verification. In order to re-run failed tests in the last verification execute the following command:

$ rally verify start --failing

For example, we have one failed test:

$ rally verify start --regex tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON
2016-05-09 15:32:39.666 29727 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:32:39.751 29727 INFO rally.verification.tempest.tempest [-] Verification 1a71f82c-e59d-4b8f-9abd-8a98f53c2531 | Starting:  Run verification.
2016-05-09 15:32:39.836 29727 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpFQO_SW
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [0.572658s] ... FAILED
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [0.877286s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.938150s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.902238s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.633860s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete_with_az [0.654307s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_metadata_get_details [0.792414s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_with_az [0.823757s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_verify_entry_in_list [0.505302s] ... ok

==============================
Failed 1 tests - output below:
==============================

tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
--------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/admin/test_aggregates.py", line 214, in test_aggregate_add_host_create_server_with_az
        self.client.add_host(aggregate['id'], host=self.host)
      File "tempest/lib/services/compute/aggregates_client.py", line 92, in add_host
        post_body)
      File "tempest/lib/common/rest_client.py", line 259, in post
        return self.request('POST', url, extra_headers, headers, body)
      File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
        method, url, extra_headers, headers, body)
      File "tempest/lib/common/rest_client.py", line 641, in request
        resp, resp_body)
      File "tempest/lib/common/rest_client.py", line 709, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: An object with that identifier already exists
    Details: {u'message': u'Cannot add host node-2.domain.tld in aggregate 422: host exists', u'code': 409}
    ...

Now let's re-run it.

$ rally verify start --failing
2016-05-09 15:36:17.389 30104 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:36:17.474 30104 INFO rally.verification.tempest.tempest [-] Verification f4e857a7-f032-452c-9ffb-dc42f0d2e124 | Starting:  Run verification.
2016-05-09 15:36:17.559 30104 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpiYREcb
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [0.665381s] ... FAILED

==============================
Failed 1 tests - output below:
==============================

tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
--------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/admin/test_aggregates.py", line 214, in test_aggregate_add_host_create_server_with_az
        self.client.add_host(aggregate['id'], host=self.host)
      File "tempest/lib/services/compute/aggregates_client.py", line 92, in add_host
        post_body)
      File "tempest/lib/common/rest_client.py", line 259, in post
        return self.request('POST', url, extra_headers, headers, body)
      File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
        method, url, extra_headers, headers, body)
      File "tempest/lib/common/rest_client.py", line 641, in request
        resp, resp_body)
      File "tempest/lib/common/rest_client.py", line 709, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: An object with that identifier already exists
    Details: {u'message': u'Cannot add host node-2.domain.tld in aggregate 431: host exists', u'code': 409}
    ...

Also, it is possible to specify the path to a YAML file with a list of Tempest tests that are expected to fail. In this case, the specified test will have the xfail status instead of fail in the verification report. How to build a verification report we tell you bellow.

$ cat xfails-file.yaml
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]: Some reason why the test fails
$ rally verify start --regex tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON --xfails-file xfails-file.yaml
2016-05-09 16:31:36.236 772 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 16:31:36.320 772 INFO rally.verification.tempest.tempest [-] Verification 76d41e5d-bf24-4e16-a9ae-5a722f8fad05 | Starting:  Run verification.
2016-05-09 16:31:36.402 772 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmp9sB5u5
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [0.625294s] ... FAILED
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [0.897577s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.865686s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.710349s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.620124s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete_with_az [0.642956s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_metadata_get_details [0.766061s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_with_az [0.795929s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_verify_entry_in_list [0.495695s] ... ok

==============================
Failed 1 tests - output below:
==============================

tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
--------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/admin/test_aggregates.py", line 214, in test_aggregate_add_host_create_server_with_az
        self.client.add_host(aggregate['id'], host=self.host)
      File "tempest/lib/services/compute/aggregates_client.py", line 92, in add_host
        post_body)
      File "tempest/lib/common/rest_client.py", line 259, in post
        return self.request('POST', url, extra_headers, headers, body)
      File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
        method, url, extra_headers, headers, body)
      File "tempest/lib/common/rest_client.py", line 641, in request
        resp, resp_body)
      File "tempest/lib/common/rest_client.py", line 709, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: An object with that identifier already exists
    Details: {u'message': u'Cannot add host node-2.domain.tld in aggregate 450: host exists', u'code': 409}
    ...
../_images/Report-verify-xfail.png

Finally, users can specify the --system-wide argument that will tell Rally not to use the Tempest virtual environment for tests. In this case, it is assumed that all Tempest requirements are already installed in the local environment. This argument is useful when users don't have an Internet connection to install requirements, but they have pre-installed ones in the local environment.

$ rally verify start --system-wide
...