Changelog

unreleased

Added

  • New documentation pages: the Environment Component, the Configuration and a guide for writing platform plugins.

  • The generated plugin reference describes arguments in more detail: the keys of dictionaries, the elements of lists and the options of values that accept several formats, groups of arguments one of which should be provided, and exclusive minimum and maximum values. Plugins that accept a single value instead of named arguments show it as an “Input value”.

  • The CLI reference lists the global options, separates positional arguments from options and shows the type, whether an argument is required or can be repeated, its default value and environment variable.

  • rally.env.platform provides typed dicts that describe the results of platform plugin methods (HealthInfo, PlatformInfo, CleanupInfo and SysEnvSpec).

Changed

  • max_avg_duration_per_atomic SLA now inspects atomic actions of all nesting levels, so a criterion can be set for a nested atomic action as well as for a top-level one.

  • rally plugin show uses full type names: integer, string, number, boolean and dictionary instead of int, str, float, bool and dict. Arguments that must not be set are shown as such instead of accepting any value.

  • Links to the source code of plugins in the plugin reference point to the release of the installed rally and rally-openstack packages instead of the master branch.

  • The changelog is now a part of the release notes in the documentation.

  • The list of core reviewers is cleaned up. Many thanks to everyone who has moved on for their contributions to Rally!

Fixed

  • max_avg_duration_per_atomic SLA no longer fails with KeyError while processing an atomic action that had not been finished (i.e the scenario died in the middle of it). Such actions have no measured duration at all, so they are ignored instead of being counted as zero-length ones.

  • max_avg_duration_per_atomic SLA no longer loses the data of atomic actions that were observed only by a part of the workers while merging results of a distributed run.

  • Validation of a task failed with AttributeError for an environment with a platform plugin that does not implement _get_validation_context(). Rally calls this method, while the base class of platform plugins declared get_validation_context() instead. Platform plugins that implemented get_validation_context() should rename it, since it was never called.

  • Deployment.list(name=...) failed with TypeError instead of returning the deployment with that name.

[5.1.1] - 2026-08-13

Fixed

  • Added backward compatibility for subclasses of rally.task.types.ResourceType that call parent’s __init__ method as super().__init__(context, cache). The proper deprecation warning is added.

[5.1.0] - 2026-08-13

Added

  • CI jobs for checking compatibility with python 3.13

  • rally task start and rally task validate can now read the task from stdin – pass - in place of the file name, for example cat task.yaml | rally task start -.

Changed

  • The command-line interface has been rebuilt on typer, replacing the custom argparse/oslo.config wrapper it grew up on. The change is backward compatible for documented usage: every command, option and primary identifier continues to work, and each primary identifier may now also be supplied positionally, for example rally task status <uuid> in addition to rally task status --uuid <uuid>. The only user-visible differences are cosmetic: help and error messages are now rendered by typer, and bash completion no longer lists the primary-id flags (--uuid, --id, --env, --deployment, --task), which still work when typed explicitly.

  • We no longer suppress PyMySQL’s connect-time warnings. That workaround only existed to hide the old @@tx_isolation deprecation warning (SQLAlchemy #4120 / PyMySQL #614), which modern SQLAlchemy and PyMySQL have since fixed, so the mysql extra now pins PyMySQL>=0.9.0.

  • The pep8 gate now runs ruff alongside flake8: ruff owns the pycodestyle, pyflakes and import-convention checks, while flake8 keeps the remaining hacking and Rally-specific rules.

  • Scenario run() arguments can be validated and documented from their Python type annotations. See the scenario plugin docs for the supported annotations.

  • A dict scenario argument that is forwarded to another callable can derive its schema from that callable’s signature with scenario.ArgsOf(func, ignore=(...)), instead of restating it as a TypedDict.

  • rally plugin show now lists the CONFIG_SCHEMA parameters of any plugin (contexts, runners, SLAs), not only scenarios.

  • The generated plugin reference now surfaces more of a scenario’s derived argument schema: default values (rendered as Defaults to ...), which arguments are required, and a note describing the extra **kwargs a scenario accepts.

  • New [DEFAULT]strict_type_annotations option to fail, rather than warn, on a scenario argument type Rally cannot map to a JSON Schema.

  • Resource types (the plugins that convert a task argument into the value a scenario receives, such as a file path into its contents) have been reworked. An argument can be bound to one inline in its annotation with types.Convert("<name>"), and the specification a resource type accepts is now validated and documented from the resource_spec annotation of its pre_process. A resource type may implement a richer keyword-only pre_process(self, *, resource_spec, config, output_type) (with a scenario_cls constructor argument) to resolve a value per iteration by returning a types.DeferredResource or to shape the result to the argument’s declared type. These signature changes are backward compatible: the legacy pre_process(resource_spec, config) and __init__(context, cache) forms are detected automatically and keep working, with a deprecation warning. See the resource type plugin docs.

  • The documentation now uses rally env commands instead of the deprecated rally deployment ones.

Deprecated

  • The --deployment / --deployment-id options of rally task and rally verify are renamed to --env. The old names still work but now print a deprecation warning – switch to --env.

Removed

  • Support for Python 3.9

  • A workaround for old rally-openstack <= 1.4.0 which is ~7 years old and supported up to python 3.7.

Fixed

  • Publishing to ReadTheDocs and docs.openstack.org/rally

  • Compatibility with paramiko 4+, which dropped the removed DSA key support.

  • The failure during Scenario plugin initialization will no longer be silently ignored. Such failures are now counted as interaction failures.

[5.0.1] - 2025-10-07

Fixed

  • Fix include section of pyproject.toml to include only rally file

[5.0.0] - 2025-09-03

Added

  • CI jobs for checking compatibility with python 3.12

  • Python type hints for task-related plugins and base classes

Changed

  • Replaced deprecated pkg_resources library with importlib & packaging for plugins discovery and loading.

  • Implements pep-517 (pyproject.toml)

  • Ported all samples to Task format v2

Removed

  • Support for Python3.8 is dropped

  • Support for SQLAlchemy <2

  • Methods get_method_class, first_index, merge and prettify_xml of rally/common/utils.py module were deprecated in Rally 3.0.0 and they are gone now.

[4.1.0] - 2024-04-29

Changed

  • Add support for SQLAlchemy 2.0

Fixed

  • Correct python_requires version to indicate correct minimum supported python version

[4.0.0] - 2024-01-10

Fixed

  • Parsing subunit v2 stream (rally-verify component) when the result is wrapped by memoryview python object.

Changed

  • xrally/xrally docker image switched

    to use python3.11-slim as a base image instead of python3.9-slim.

  • Use ‘draft-7’ as a default jsonschema validator when there is no meta-schema field specified at $schema.

Added

  • CI checks for Python 3.11 compatibility

  • Support for specifying platform of context as a part of required_context validator like bellow:

    from rally.task import scenario
from rally.task import validation

@scenario.configure(name="Dummy.scenario")
@validation.add("required_context", contexts=["ctx_name@platform"])
class ElasticsearchLogInstanceName(scenario.Scenario):
    def run(self):
         pass

Removed

  • Support for Python 3.6 and Python 3.7

[3.4.0] - 2023-05-23

This is the last release with support of Python 3.6 and Python 3.7

Fixed

Changed

xrally/xrally docker image switched to use python3.9-slim as a base image instead of ubuntu 20.04.

Added

  • Pin SQLAlchemy to <2.0.0

  • CI for running unit and functional tests using python 3.10

  • CI jobs that check Rally installation compatibility with CentOS 9 Stream and Ubuntu Jammy

Removed

  • CI jobs with installation compatibility checks for CentOS 7, CentOS 8 (CentOS 8 Stream is checked instead), Ubuntu Bionic.

[3.3.0] - 2021-06-16

Changed

  • rally verify returns 0 on success, 1, 2 or 3 depending on detected issue.

  • Switch docker image from ubuntu 18.04 to ubuntu 20.04

  • Move from Freenode to OFTC irc network

Removed

  • dependency on decorator library.

[3.2.0] - 2020-10-30

Added

  • Extend CI for checking installation on Ubuntu Focal environment.

  • Extend rally task start command output with the progress of finishing workloads

  • Introduce rally task restart command.

Fixed

[3.1.0] - 2020-05-08

Changed

  • Validator for scenario arguments do not initialize scenario class anymore for discovering argument spec of the run method.

  • Docker image includes the latest setuptools

Fixed

  • [verification component] Failure while parsing subunit v2 stream in case of unwanted test_id

    Launchpad-bug #1871406

[3.0.0] - 2020-03-23

Added

  • CI for covering unit and functional tests against Python 3.8 environment. Everything works, so we have proved Python 3.8 support

  • Add CI job for testing installation of Rally at Centos 8.

  • Updating a latest tag of docker image on every merged commit.

Changed

  • rally plugin show command returns not-zero exit code in case of not found or multiple match errors

  • docker image is switched to use python3.6.

  • path_or_url plugin follows redirects while validating urls now.

  • rally task sla-check fails if there is no data.

Deprecated

  • Command rally task results is deprecated. Use rally task report –json instead.

  • Module rally.common.sshutils is deprecated. Use rally.utils.sshutils instead.

  • Module rally.common.yamlutils is deprecated. It was designed for CLI usage and moves to right place.

  • Module rally.common.fileutils is deprecated.

  • All modules from rally.plugins.common.contexts are deprecated. Use rally.plugins.task.contexts instead.

  • All modules from rally.plugins.common.exporters are deprecated. Use rally.plugins.task.exporters instead.

  • Module rally.plugins.common.hook.sys_call is deprecated. Use rally.plugins.task.hooks.sys_call instead.

  • All modules from rally.plugins.common.hook.triggers are deprecated. Use rally.plugins.task.hook_triggers instead.

  • All modules from rally.plugins.common.runners are deprecated. Use rally.plugins.task.runners instead.

  • All modules from rally.plugins.common.scenarios are deprecated. Use rally.plugins.task.scenarios instead.

  • All modules from rally.plugins.common.sla are deprecated. Use rally.plugins.task.sla instead.

  • All modules from rally.plugins.common.verification are deprecated. Use rally.plugins.verification instead.

Removed

  • Python 2.7, Python 3.4 and Python 3.5 support

  • Devstack plugin. It was deprecated long time ago. rally-openstack project should be used instead

  • rally.common.utils.distance method was deprecated since Rally 0.4.1

  • rally.common.utils.format_float_to_str method was deprecated since Rally 0.11.2. rally.utils.strutils.format_float_to_str should be used instead.

  • rally.task.atomic.optional_action_timer decorator was deprecated since Rally 0.10.0

  • rally.task.hook.Hook class was deprecated since Rally 0.10.0. rally.task.hook.HookAction should be used instead.

  • rally.task.trigger module was deprecated since Rally 0.10.0. rally.task.hook.HookTrigger should be used instead.

  • rally.common.i18n module was deprecated since Rally 0.10.0

  • namespace argument of configure decorator of Scenario, Context, Validators plugins. It was deprecated since Rally 0.10.0 in favor of platform.

  • install_rally.sh script is too complicated and installs only rally framework without plugins.

Fixed

  • inaccurate calculation of 90 and 95 percentiles in case of 10k+ iterations

[2.1.0] - 2019-11-19

Please note that Python 2.7 will reach the end of its life on January 1st, 2020. A future version of Rally will drop support for Python 2.7, it will happen soon. Also, the same will happen with support of Python 3.4 and Python 3.5

Removed

Library netaddr from direct project requirements. We never use it at Rally framework.

Fixed

Support of latest alembic

Launchpad-bug #1844884

[2.0.0] - 2019-09-13

Changed

python jsonschema dependency is not limited by <3.0.0 anymore and you can use draft-7 as for now.

Removed

  • rally task sla_check command was deprecated in Rally 0.8.0 in favor of rally task sla-check.

  • rally-manage db command (and the whole rally-manage entry-point) was deprecated in Rally 0.10.0 in favor of rally db command.

  • –namespace argument was deprecated in Rally 0.10.0 in favor of –platform which has better meaning. Affected commands: rally plugin show, rally plugin list, rally verify list-plugins, rally verify create-verifier.

  • –tasks argument of rally task report command and –task argument of rally task use command were deprecated in Rally 0.10.0 in favor of unified –uuid argument.

  • –junit argument of rally task report command is deprecated in Rally 0.10.0 in favor of rally task export –type junit-xml

[1.6.0] - 2019-06-19

Added

A list of tests to skip while running verification now supports regular expressions.

Fixed

  • incompatibility with SQLAlchemy 1.3

  • several py3 issues of verification component

[1.5.1] - 2019-05-15

Fixed

rally deployment create –fromenv creates wrong spec for rally-openstack<=1.4.0 which doesn’t pass rally deployment check.

Launchpad-bug #1829030

[1.5.0] - 2019-05-08

Added

New two charts EmbeddedChart and EmbeddedExternalChart for embedding custom html code or external pages as complete charts of scenarios.

[1.4.1] - 2019-02-28

Fixed

  • Python 3 issue of Verification component

  • Docker README file

[1.4.0] - 2019-02-04

Changed

  • Add the –html-static option to commands rally task trends, it could generate trends report with embedded js/css.

  • Removed dependency to morph library.

Fixed

  • rally command crashes while calling without any arguments

  • Fix the ssh error while passing an dss key in ssh utils.

    Launchpad-bug #1807870

[1.3.0] - 2018-12-01

Added

  • Add the –deployment option to commands rally task report and rally task export that allows to report/export all tasks from defined deployment.

  • Briefly: the new base image is published at Docker Hub

    Detailed story: Long time ago Rally team introduced first docker images which were hosted by rallyforge account at Docker Hub. Due to various circumstances we lost access to that account and Docker support restored access to it in a strange way (we lost all repositories and could not recreate them). That is why Rally team started publishing docker images from scratch. The new organization was created - xRally. Since we already had plans to move OpenStack plugins to the separate repository, we started publishing images with in-tree OpenStack plugins to xrally/xrally-openstack repository. As soon as, a separate package for OpenStack plugins was introduced, we switched the source of xrally/xrally-openstack Docker Hub repository to rally-openstack git repository. As for Rally 1.0.0 we finally have pure framework without heavy dependencies and can start publishing separate images for Rally framework itself which can be used as a base image for all plugins. New images will be located at xrally/xrally Docker Hub repository.

Changed

  • rally --version prints version of Rally framework with versions of installed plugins instead of printing just version of Rally framework.

  • Dockerfile moved from the root directory to ./etc/docker/

Fixed

A floating bug with constant_for_duration runner.

Launchpad-bug #1800447

[1.2.1] - 2018-09-27

Minor inner fixes

[1.2.0] - 2018-09-19

Added

  • New validator map_keys for checking keys of specific argument.

  • Support of ElasticSearch 6.x cluster elastic exporter.

Changed

  • Improved validation errors for task component.

  • [ElasticSearch exporter] Do not send ‘no-name-action’ index when the item fails after some atomic actions completed and there is a root atomic. For example, there is ‘wait-for-some-resource-ready’ action. It consists of a bunch of get requests to update the current status. After specified timeout this action can fail if the resource is not in the right state. In such case, there is no reason to use ‘no-name-action’ for saving the error, the parent index (i.e ‘wait-for-some-resource-ready’) will already store it.

[1.1.0] - 2018-08-07

Added

  • Introducing rally env cleanup command for performing disaster cleanup.

  • New CI jobs for checking compatibility with Python 3.4, 3.6, 3.7 .

Changed

  • The output of json task result exporter (rally task report --json) is extended with information about environment where task was executed (new env_name and env_uuid properties)

  • Add the –filter-by option to the command rally task detailed, which allows us to show only those workloads which we are interested in (see the examples below). Examples:

    1. show only failed workloads rally task detailed --filter-by sla-failures

    2. show only those workloads which include the next scenario plugin(s) rally task detailed --filter-by scenarios=scenario1[,scenarios2...]

  • requirements and constraints (suggested versions) files are updated.

Removed

  • Disturbing warning message about removing in-tree OpenStack plugins. This message became redundant after Rally 1.0.0 when such plugins were removed.

  • OpenStack related configuration options for sample file.

  • Deprecated in Rally 0.10 rally.task.exporter.Exporter class in favor of rally.task.exporter.TaskExporter.

Fixed

Deprecated

  • ‘async’ argument of API method task.abort in favor of ‘wait’ argument which doesn’t conflict with a reserved keyword in python 3.7

[1.0.0] - 2018-06-20

It finally happened. We are happy to inform you that OpenStack plugins has a single home - https://github.com/openstack/rally-openstack . All in-tree plugins are removed now and framework part become more lightweight.

What does it mean for you?!

If you are interested only in OpenStack plugins, just change the package you are installing from rally to rally-openstack. If you have custom OpenStack plugins which inherits from upstream, change python imports from rally.plugins.openstack to rally_openstack. That is all.

If you are interested not only in OpenStack, you can start using your favourite tool for various platforms and systems. Here you can find our first attempts to seize the world - https://github.com/xrally/xrally-docker and https://github.com/xrally/xrally-kubernetes.

Changed

Since OpenStack plugins were moved to the separate repository, the new release notes should become light as well, so there is no need in separate pages for each release. All release notes will be aggregated in a single file CHANGELOG.rst.

Also, it is sad to mention, but due to OpenStack policies we need to stop duplicating release notes at git tag message. At least for now.

Removed

  • All OpenStack related plugins.

Fixed

  • Validation of existing platforms in Python 3 environment.

  • Support of testr for verifiers.

[0.0.0] - [0.12.1]

Release notes for Rally 0.0.0-0.12.1 are available at https://github.com/openstack/rally/tree/master/doc/release_notes/archive