Python Roles

build-python-release

Build sdist and wheel for Python projects.

Role Variables

release_python Default: python: The python interpreter to use. Set it to “python3” to use python 3, for example.

build_wheel Default: true: Whether to build a wheel. Set to false to just build an sdist tarball.

bdist_wheel_xargs Default: '': Extra arguments to pass to the bdist_wheel command when building packages.

build-releasenotes

Build releasenotes for a project, optionally incorporating translations.

Role Variables

zuul_work_virtualenv Default: ~/.venv: Virtualenv location in which to install things.

zuul_work_dir Default: {{ zuul.project.src_dir }}: Directory to build releasenotes in.

ensure-babel

Ensure babel is installed

Role Variables

constraints_file: Optional path to a pip constraints file for installing python libraries.

ensure-if-python

Install the contents of a directory if they contain a python project.

Installs into a virtualenv.

Role Variables

install_package Default: true: Flag indicating whether or not the software in the zuul_work_dir should be installed.

error_on_failure: Flag that indicates installation errors should result in failure. Failures in installing the target directory are ignored by default.

constraints_file: Optional path to a pip constraints file to use when installing.

zuul_work_virtualenv Default: ~/.venv: Virtualenv location in which to install things.

zuul_work_dir Default: {{ zuul.project.src_dir }}: Directory to operate in.

ensure-pip

Ensure pip is available

This role is intended install the requirements for the pip module on hosts.

Jobs that also wish to call pip via shell commands directly can also use this to ensure pip is available. However, it should be noted that calling pip is ambiguous when supporting many platforms. On some platforms it may install the package under the Python 2 interpreter and in others Python 3. You should use a qualified name (pip2 or pip3) to avoid confusion.

This role will also install wheel components sufficient to run bdist_wheel builds or pip wheel on a source tree.

Role Variables

ensure_pip_from_packages Default: True: Install the distribution packages for Python 3 pip, setuptools and wheel on the system.

ensure_pip_from_packages_with_python2 Default: False

Install the distribution packages for Python 2 pip, setuptools and wheel packages. This defaults to True when Ansible is running under Python 2.

CentOS 7 requies the epel repository if this flag is enabled because python-pip packages come from there.

This may not be valid on distributions that have removed Python 2 support.

ensure_pip_from_upstream Default: False: Install pip from latest upstream sources locally. Note this is probably not what you want and should be used with extreme caution. The installed pip does not coordinate with the system packaged versions, and can lead to wide variety of problems if CI jobs re-install pip packages, for example.

ensure_pip_from_upstream_interpreters Default: [ ansible_python_interpreter ]: A list of interpreters to install pip from upstream with. Note that by default the last entry in the list will likely own the /usr/local/bin/pip command; this can create confusion for legacy jobs if they assume pip installs Python 2 libraries but it is actually installing into the Python 3 environment. This role does not install the Python 2 interpreter, which may not be available on the system, so caution should be used when specifying python2 in this list.

Output Variables

ensure_pip_virtualenv_cmd: This variable will be set to a command appropriate for general usage with the pip module virtualenv_command argument on the host. On Python 3 hosts this will be the inbuilt venv module, on Python 2 hosts the virtualenv package will be installed (this is avoided on Python 3 hosts as an unnecessary dependency).

ensure-python

Ensure specified python interpreter and development files are installed

There are three ways to install the python interpreter:

Using distribution packages: This is the default (python_use_pyenv and python_use_stow are both false).
Install using pyenv.
Install using stow.

Note

You cannot use both pyenv and stow method for the same job. That means that python_use_pyenv and python_use_stow cannot be set both to True at the same time.

Role Variables

python_version: Optional version of python interpreter to install, such as 3.7.

python_use_pyenv Default: False: Whether to optionally use pyenv to install python instead of distro packages.

python_use_stow Default: False: In case you have image with already prepared python versions, for example used the python-stow-versions element, you can activate them with stow utility by setting this variable to true.

python_stow_dir Default: /usr/local/stow: Sets the target directory for stow. This should be the path to the directory where prepared python packages are located.

ensure-sphinx

Ensure sphinx is installed

Installs sphinx. Also installs any dependencies needed in the first of doc/requirements.txt, releasenotes/requirements.txt and test-requirements.txt to be found.

All pip installs are done with a provided constraints file, if given.

Role Variables

constraints_file: Optional path to a pip constraints file for installing python libraries.

doc_building_packages Default: []: List of python packages to install for building docs. The default package list is based on the python version in use.

doc_building_extra_packages Default: []: List of python additional packages to install for building docs. By default this list is empty.

sphinx_python Default: python2: Version of python to use, either python2 or python3.

zuul_work_virtualenv Default: ~/.venv: Virtualenv location in which to install things.

zuul_work_dir Default: {{ zuul.project.src_dir }}: Directory to operate in.

ensure-tox

Ensure tox is installed

Look for tox, and if not found, install it via pip into a virtual environment for the current user.

Role Variables

tox_prefer_python2 Default: False

If tox is not detected, prefer to install tox inside Python 2 instead of Python 3.

If set, ensure-pip.ensure_pip_from_packages_with_python2 will be automatically set to True to enable a Python 2 installation of pip.

ensure_global_symlinks Default: False

Install a symlink to the tox executable into /usr/local/bin/tox. This can be useful when scripts need to be run that expect to find tox in a more standard location and plumbing through the value of tox_executable would be onerous.

Setting this requires root access, so should only be done in circumstances where root access is available.

Output Variables

tox_executable Default: tox

After running this role, tox_executable will be set as the path to a valid tox.

At role runtime, look for an existing tox at this specific path. Note the default (tox) effectively means to find tox in the current $PATH. For example, if your base image pre-installs tox in an out-of-path environment, set this so the role does not attempt to install the user version.

ensure-twine

Ensure twine is installed.

Role Variables

twine_python Default: python: The python interpreter to use to install twine if it is not already installed. Set it to “python3” to use python 3, for example.

Output Variables

twine_excutable Default: twine

After running this role, twine_executable will be set as the path to a valid twine.

At role runtime, look for an existing twine at this specific path. Note the default (twine) effectively means to find tox in the current $PATH. For example, if your base image pre-installs twine in an out-of-path environment, set this so the role does not attempt to install the user version.

ensure-virtualenv

Ensure virtualenv is available

This role installs the requirements for the virtualenv command on the current distribution.

Users should be aware of some portability issues when using virtualenv:

Distributions differ on the interpreter that virtualenv is provided by, so by calling virtualenv with no other arguments means that on some platforms you will get a Python 2 environment and others a Python 3 environment.
If you wish to call virtualenv as a module (e.g. python -m virtualenv) you will need to know which interpreter owns the virtualenv package for your distribution; e.g. on some, such as Bionic, virtualenv is provided by python3-virtualenv but python refers to Python 2, so python -m virtualenv is not a portable way to call virtualenv.
virtualenv -p python3 is likely the most portable way to consistently get a Python 3 environment. virtualenv -p python2 may not work on some platforms without Python 2.
If you use Python 3 and do not require the specific features of virtualenv, it is likely easier to use Python’s inbuilt python3 -m venv module to create an isolated environment. If you are using pip: in your Ansible roles and require an environment, see the documentation for ensure-pip.

fetch-coverage-output

Collect output from a coverage run

By default, this copies the output from a coverage run on the worker to the log root of the executor.

Role Variables

zuul_executor_dest Default: {{ zuul.executor.log_root }}: The destination directory on the executor. By default, the log root.

coverage_output_src Default: {{ zuul.project.src_dir }}/cover/: The location on the worker from which to fetch the coverage output detail. By default, the cover dir of the current project.

fetch-python-sdist-output: Collect output from a python sdist build

fetch-sphinx-output

Collect output from a sphinx build

By default, this copies the output from the sphinx build on the worker to the log root of the executor.

Role Variables

zuul_executor_dest Default: {{ zuul.executor.log_root }}: The destination directory on the executor. By default, the log root.

sphinx_build_dir Default: doc/build: Directory relative to zuul_work_dir where build output will be put.

sphinx_output_suffix Default: '': Suffix to use for constructing the path. This is normally an empty string. If set to ‘/’ then, rsync will remove the last part from the original path.

sphinx_output_src Default: {{ zuul_work_dir }}/{{ sphinx_build_dir }}/html{{ sphinx_output_suffix }}: The location on the worker from which to fetch the generated sphinx content. By default, the HTML doc build dir of the current project.

zuul_work_dir Default: {{ zuul.project.src_dir }}: The location of the main working directory of the job.

zuul_use_fetch_output Default: false

Whether to synchronize files to the executor work dir, or to only copy them on the test instance.

When set to False, the default, the role synchronizes the tarball archives and extracted documentation files to the executor log_root.

When set to True, the content is copied locally to {{ ansible_user_dir }}/zuul-output/logs/. The fetch-output role needs to be run to copy this output to the executor log_root.

fetch-sphinx-tarball

Collect output from a sphinx build as a tarball

By default, this copies the output from the sphinx build on the worker to the log root of the executor as a tarball, and then extracts the archive into the log root for viewing.

Role Variables

sphinx_build_dir Default: doc/build: Directory relative to zuul_work_dir where build output should be found.

zuul_work_dir Default: {{ zuul.project.src_dir }}: The location of the main working directory of the job.

sphinx_pdf_files Default: list: A list of file names of PDF files to collect. By default, the list contains as entry only doc-{{ zuul.project.short_name }}.pdf.

zuul_use_fetch_output Default: false

Whether to synchronize files to the executor work dir, or to only copy them on the test instance.

When set to False, the default, the role synchronizes the tarball archives and extracted documentation files to the executor log_root.

When set to True, the content is copied locally to {{ ansible_user_dir }}/zuul-output/logs/. The fetch-output role needs to be run to copy this output to the executor log_root.

fetch-subunit-output

Collect subunit outputs

Role Variables

zuul_work_dir Default: {{ ansible_user_dir }}/{{ zuul.project.src_dir }}: Directory to work in. It has to be a fully qualified path.

fetch_subunit_output_additional_dirs Default: []: List of additional directories which contains subunit files to collect. The content of zuul_work_dir is always checked, so it should not be added here.

zuul_use_fetch_output Default: false: Whether to synchronize files to the executor work dir, or to copy them on the test instance. When set to false, the role synchronizes the file to the executor. When set to true, the job needs to use the fetch-output role later.

fetch-tox-output

Collect log output from a tox build

Role Variables

tox_envlist: Comma separated string with test environmens to fetch log output from. ALL fetches all environments while an empty string fetches all test environments configured with envlist in tox.

tox_executable Default: tox: Location of the tox executable.

zuul_work_dir Default: {{ zuul.project.src_dir }}: Directory tox was run in.

zuul_use_fetch_output Default: false: Whether to synchronize files to the executor work dir, or to copy them on the test instance. When set to false, the role synchronizes the file to the executor. When set to true, the job needs to use the fetch-output role later.

find-constraints

Find a pip constraints file

Sets a variable upper_constraints which can be passed to a pip invocation.

Role Variables

constraints_file: Optional path to a pip constraints file for installing python libraries.

sphinx

Run sphinx to generate documentation

Role Variables

sphinx_source_dir Default: doc/source: Directory relative to zuul_work_dir that contains the Sphinx sources.

sphinx_build_dir Default: doc/build: Directory relative to zuul_work_dir where build output will be put.

sphinx_builders Default: ['html']: Which sphinx builders to run.

sphinx_warning_is_error: Whether to treat sphinx build warnings as errors. Defaults to the value of [build_sphinx] warning-is-error in setup.cfg if defined, False if the [build_sphinx] section is present but the warning-is-error option is undefined, or True if the entire section is undefined.

zuul_work_virtualenv Default: ~/.venv: Virtualenv that sphinx is installed in.

zuul_work_dir Default: {{ zuul.project.src_dir }}: Directory to operate in.

tox

Runs tox for a project

This role overrides Python packages installed into tox environments with corresponding Zuul sibling projects and runs tox tests as follows:

Create tox environments.
Get Python sibling package names for sibling projects created by Zuul (using required-projects job variable). Package names are searched in following sources:
- setup.cfg of pbr projects,
- setup.py,
- tox_package_name role variable.
Remove sibling packages from tox environments.
Create temporary constraints file, lines for sibling packages are removed.
Install sibling packages from Zuul projects into tox environments with temporary constraints file.
Run tox tests.

Role Variables

tox_environment Type: dict: Environment variables to pass in to the tox run.

tox_envlist: Comma separated string with test environments tox should run. ALL runs all test environments while an empty string runs all test environments configured with envlist in tox.

tox_executable Default: tox: Location of the tox executable.

tox_config_file: Path to a tox configuration file, or directory containing a tox.ini file. Will be provided to tox via its -c command-line option if set.

tox_extra_args Default: -vv: String of extra command line options to pass to tox.

tox_constraints_file: Path to a pip constraints file. Will be provided to tox via TOX_CONSTRAINTS_FILE (deprecated but currently still supported name is UPPER_CONSTRAINTS_FILE) environment variable if it exists.

tox_install_siblings Default: true: Flag controlling whether to attempt to install python packages from any other source code repos zuul has checked out. Defaults to True.

tox_package_name: Allows a user to setup the package name to be used by tox, over reading a setup.cfg file in the project.

zuul_work_dir Default: {{ zuul.project.src_dir }}: Directory to run tox in.

upload-pypi

Upload python packages to PyPI

Role Variables

pypi_info

Complex argument which contains the information about the PyPI server as well as the authentication information needed. It is expected that this argument comes from a Secret.

pypi_info.username: Username to use to log in to PyPI.

pypi_info.password: Password to use to log in to PyPI.

pypi_info.repository Default: pypi: Name of the repository to upload to.

pypi_info.repository_url Default: The built-in twine default for the production pypi.org service.: URL of the PyPI repostory.

pypi_path Default: src/{{ zuul.project.canonical_name }}/dist: Path containing artifacts to upload.

pypi_twine_executable Default: twine: Path to twine executable.

pypi_register_first Default: false: Whether the role should register the package before uploading it. This may be required when uploading for the first time to a devPI instance.