Ussuri Series Release Notes

21.2.10

Security Issues

21.2.5

Deprecation Notes

  • Remove octavia_amp_image_id option as the corresponding configuration option in Octavia amp_image_id is deprected and image tags should be used instead.

21.2.3

New Features

  • Added new variable haproxy_hatop_install, that allows to conditionally enable or disable hatop installation.

Upgrade Notes

  • Variable haproxy_hatop_downloader has been removed, Deployers supposed to use haproxy_hatop_download_url override if needed to install in deployments with limited internet connection.

21.2.1

New Features

  • The role now supports creating system scoped credentials alongside project scoped credentials. The default behavior of the role did not changed, until openrc_system_scope variable was set to true. If the openrc_system_scope is true the default cloud in clouds.yaml will set to system scoped credentials and another credentials named default_project_scope will get created with project scoped credentials. Due to usage of openrc file in other roles, the opposite logic applies to openrc files, which means if openrc_system_scope is set to true the credentials in openrc will set to project scoped credentials and another openrc filec named openrc.system_scope will get created with system scoped credentials and will be placed in destination of openrc_system_file_dest variable.

21.2.0

Prelude

This is minor bugfix release that brings up overall improvements and bugfixes to the roles.

New Features

  • Added variable nova_scheduler_extra_filters which allows to extend list of defaulted nova_scheduler_default_filters

  • Added deployment of the keystone_auth_default_policy.json file for Magnum.

Upgrade Notes

  • String value of nova_scheduler_default_filters is converted to the list At the moment there is compatability for overriden values, that are string, but this support will be removed in the Wallaby release. So deployers are recommended to replace their string overrides with list ones.

  • Ironic Inspector service was bumped to master version instead of stable/ussuri. Which means that during upgrade to this version Inspector version will be downgraded, which may result in problems with migrations. We recommend to ensure you have full DB backup before Inspector upgrade or overwrite ironic_inspector_git_install_branch with 10.4.0 tag

Bug Fixes

  • uWSGI service restart is now properly triggered upon service config change

  • Notifications are enabled now if either ceilometer or designate service is present in the inventory

  • Fixed manila support for CentOS 8

  • Fixed ceph_client role for distro installs

  • Fixed Ubuntu Focal ceph deployments

  • Since Ubuntu has dropped older base images, which resulted in all previous tags being broken, we’ve switched to downloading always latest base image available. This should guarantee that we retrieve relevant images only.

21.0.0

New Features

  • Support is added for deploying OpenStack on CentOS 8 with source and distro based installs. However, nspawn support can’t be offered, as machinectl relies on btrfs which has been dropped by CentOS.

  • Support is added for deploying OpenStack on Ubuntu Focal (20.04) with source based installs. Ubuntu Cloud Archive is not available for Focal at this point so it is not possible to offer distro package based installs for Focal.

  • Added new parameter tempest_services for setting tempest_service_available_{service_name} var automatically.

  • A new playbook os-zun-install.yml has been added which will deploy the zun service to hosts assigned to the host group zun-infra_hosts.

  • The ansible version used by OSA is updated from the 2.7 to the 2.8 series. This requires an upgrade of ceph-ansible to 4.0 and this in turn requires an upgrade of ceph from Mimic to Nautilus. This version dependancy applies where OSA uses ceph-ansible directly to deploy the ceph infrastructure, but not when OSA is integrated with an externally provisioned ceph cluster.

  • Each openstack service role has a new variable <role>_bind_address which defaults to 0.0.0.0. A global override openstack_service_bind_address may be used by a deployer either in group_vars or user_variables to define an alternative IP address for services to bind to. This feature allows a deployer to bind all of the services to a specific network, for example the openstack management network. In this release the default binding remains 0.0.0.0, and future releases may default the binding to the management network.

  • Added possibility to override ceph.conf partially by defining ceph_client_ceph_conf_overrides variable. It uses regular format of OpenStack-Ansible overrides. From now on, config_template plugin should be present for ansible.

  • Get ceph keyrings from files, if variable``ceph_keyrings_dir`` is defined the keyrings will be extracted from files. All files in the directory must have .keyring extention and be named with its corresponding ceph_client name. For example, if cinder_ceph_client is cinder the cinder keyring file must be named cinder.keyring. Each file must contain username and the key and nothing more, below an example for cinder.keyring content.

    [client.cinder] key = XXXXXXXXXXX

  • Multiple HAProxy nodes can now be deployed in an HA configuration with keepalived and LetsEncrypt certificates. Certbot can be treated as a backend service for haproxy and acme-challenge requests from LetsEncrypt can be directed to whichever HAProxy server is running a certificate renewal. New variables are defined for frontend ACLs and options to be passed to Certbot which enable this new feature, but the majority of the required configuration is done via the existing HAProxy service setup. An example is provided in the documentation.

  • Added variables cinder_active_active_cluster and cinder_active_active_cluster_name that allow to explicitly enable or disable active/active feature, and set cluster name.

  • The service setup in keystone for cloudkitty will now be executed through delegation to the cloudkitty_service_setup_host which, by default, is localhost (the deploy host). Deployers can opt to rather change this to the utility container by implementing the following override in user_variables.yml.

    cloudkitty_service_setup_host: "{{ groups['utility_all'][0] }}"

  • For the os_cloudkitty role, the systemd unit TimeoutSec value which controls the time between sending a SIGTERM signal and a SIGKILL signal when stopping or restarting the service has been reduced from 300 seconds to 120 seconds. This provides 2 minutes for long-lived sessions to drain while preventing new ones from starting before a restart or a stop. The RestartSec value which controls the time between the service stop and start when restarting has been reduced from 150 seconds to 2 seconds to make the restart happen faster. These values can be adjusted by using the cloudkitty_*_init_config_overrides variables which use the config_template task to change template defaults.

  • The galera_server role now includes the functionality from the galera_client role, and can optionally install the client and server components. This is controlled using two booleans, galera_install_server and galera_install_client, both of which default to false.

  • OpenStack-Ansible now support deployments on Debian 10 (buster)

  • Cinder is deployed with Active-Active enabled by default if you are using Ceph as a backend storage.

  • Passed –extra-vars flag to the openstack-ansible should have precedence over the user-variables*.yml now.

  • Add the possibility to disable openrc v2 download in the dashboard. new var horizon_show_keystone_v2_rc can be set to False to remove the entry for the openrc v2 download.

  • Added variables magnum_cluster_templates and magnum_flavors which allow deployers to define coe cluster template and nova flavors creation during role execution. These variables may contain list of resources to add. All keys supported by appropriate ansible modules may be passed as items in the list.

  • The ceph_client role will now look for and configure manila services to work with ceph and cephfs.

  • The os_masakari role now covers the monitors installation and configuration, completing the full service configuration.

  • Added support for using mod_auth_openidc instead of shibboleth as a service provider for supporting users who have a preference to use OIDC for federation. Mod_auth_openidc is the apache module that is recommended in the keystone documentation for implementing openidc. Added a variable to called apache_mod to keystone_sp, if left undefined shibboleth will continue to be installed by default provided keystone_sp is not empty. Mod_auth_openidc will not be installed unless it is spelled correctly, any misspellings will result in a shibboleth install. Note that installing shibboleth on Debian based metal distro deployments may break services that depend on libcurl4, as shib2 requires libcurl3, and they are unable to coexist. This can be resolved when there is a shib3 package available in a future release of Ubuntu/Debian. There is currently no support for simultaneous use of shibboleth2 and mod_auth_openidc.

  • The murano dashboard is available in Horizon. Deployers can enable the panel by setting the following Ansible variable:

    horizon_enable_murano_ui: True

  • Support for the networking-generic-switch mechanism driver has been implemented. This allows Ironic to interface with Neutron when the neutron network interface has been configured. This feature may be enabled by adding ml2.genericswitch to the neutron_plugin_types list in /etc/openstack_deploy/user_variables.yml.

  • The override rabbitmq_memory_high_watermark can be used to set the maximum size of the erlang Virtual Machine before the garbage collection is triggered. The default is lowered to 0.2, from 0.4 as the garbage collection can require 2x of allocated amount during its operation. This can result in a equivalent use of 0.4, resulting in 40% of memory usage, visible to the rabbitMQ container. The original default setting of 0.4 can lead to 80% memory allocation of rabbitMQ, potentially leading to a scenario where the underlying Linux kernel is killing the process due to shortage of virtual memory.

  • The role now supports using the distribution packages for the OpenStack services instead of the pip ones. This feature is disabled by default and can be enabled by simply setting the octavia_install_method variable to distro.

  • Support separate oslo.messaging services for RPC and Notifications to enable operation of separate and different messaging backend servers.

  • Added 2 new varibles for all groups - oslomsg_notify_policies and oslomsg_rpc_policies. These variables contain default rabbitmq policies, which will be applied for every rabbitmq vhost. As for now they will enable [HA mode](https://www.rabbitmq.com/ha.html) for all vhosts. If you would like to disable HA mode just set these variables to empty lists inside your user_config.yml

  • The os_cloudkitty role now supports the ability to configure whether apt/yum tasks install the latest available package, or just ensure that the package is present. The default action is to ensure that the latest package is present. The action taken may be changed to only ensure that the package is present by setting cloudkitty_package_state to present.

  • Deployments will now default to using python3 when a python3 interpreter is present in the operating system. Each openstack-ansible role has a new variable for the form <role>_venv_python_executable which defaults to python2 but a global variable openstack_venv_python_executable in the openstack-ansible group variables sets this to python3 on supporting operating systems. This enables a deployer is to selectively use python2 or python3 on a per service basis if required. The ansible-runtime venv is also created using python3 on the deploy host if possible.

  • Added variables horizon_session_engine and horizon_session_caches which can be used to configure horizon session engine and a backend for it.

  • Several environment variables have been added in the bootstrapping functions used by the gate-check-commit script. These variables can be used to skip various phases of the bootstrap during the gate-check-commit or bootstrap-ansible script execution.

    The environment variables added are:

    • SKIP_OSA_RUNTIME_VENV_BUILD: Skip bootstrapping of the OSA ansible venv in bootstrap-ansible.sh

    • SKIP_OSA_BOOTSTRAP_AIO: Skip execution of the bootstrap-aio playbook in gate-check-commit

    • SKIP_OSA_ROLE_CLONE: Skip execution of the get-role-requirements-playbook in the bootstrap-ansible.sh script

  • Adds new variable tempest_tempestconf_profile_extras which allows to extend tempest_tempestconf_profile dictionary without need to override all existing keys. tempest_tempestconf_profile_extras has presedence over tempest_tempestconf_profile. For example, it may be used to define extra image or it’s format.

  • The galera_server role now uses mariabackup in order to complete SST operations due to the fact that this is the recommended choice from MariaDB.

  • The galera_server role now ships with the latest MariaDB release of 10.3.13.

  • All roles are migrated from usage of regular log files to systemd-journald

  • Deployers may require custom CA certificates installing on their openstack hosts or service containers. A new variable openstack_host_ca_certificates is added which is a list of certificates that should be copied from the deploy host to the target hosts. Certificates may be selectively deployed by defining the variable either in user_variables.yml or via host/group vars.

  • A new optional file /etc/openstack_deploy/user-role-requirements.yml is now available for a deployer to override individual entries in the upstream ansible-role-requirements file. This can be used to point to alternative repos containing local fixes, or to add supplementary ansible roles that are not specified in the standard ansible-role-requirements.

Known Issues

  • Due to a change in how backend_host is defined when using Ceph, all the Cinder volumes will restart under the same backend name. This will mean that any volumes which previously were assigned to the host or container that hosted the volume will no longer be manageable. The workaround for this is to use the cinder-manage volume update_host command to move those volumes to the new backend host. This known issue will be resolved soon with an upgrade playbook.

  • The journald-remote is disabled from execution inside setup-infrastructure until https://github.com/systemd/systemd/issues/2376 has been incorporated in current systemd packages. The playbook can be enabled by setting journald_remote_enabled to True

  • The previous way of using a common backend_host across all deployments was not recommended by the Cinder team and it will cause duplicate messages that cause problems in the environment.

Upgrade Notes

  • Any ceph infrastructure components (OSDs, MONs etc) deployed using the OSA/ceph-ansible tooling will be upgraded to the Ceph Nautilus release. Deployers should verify that this upgrade is suitable for their environment before commencing a major upgrade to Train, and consult the ceph-ansible and ceph release notes for Nautilus. For integration with external ceph clusters where OSA does not deploy any of the ceph cluster infrastructure, overrides can be used to select the specific version of ceph repositories used by the OSA ceph_client ansible role.

  • Variable uca_repo has been removed. Deployers are appreciated to use user_external_repos_list instead if they want to define extra repository Variable uca_apt_repo_url was renamed to apt_repo_url in order to correspond it’s usage as it also affect Debian deployments now.

  • Generation of records for /etc/hosts is now made with blockinfile ansible module. During upgrade you will have records doubled in yours /etc/hosts as we don’t drop previously created records for safety reasons if openstack_host_manage_hosts_file is set to true.

  • For the os_cloudkitty role, the systemd unit TimeoutSec value which controls the time between sending a SIGTERM signal and a SIGKILL signal when stopping or restarting the service has been reduced from 300 seconds to 120 seconds. This provides 2 minutes for long-lived sessions to drain while preventing new ones from starting before a restart or a stop. The RestartSec value which controls the time between the service stop and start when restarting has been reduced from 150 seconds to 2 seconds to make the restart happen faster. These values can be adjusted by using the cloudkitty_*_init_config_overrides variables which use the config_template task to change template defaults.

  • The galera_server role now includes the functionality from the galera_client role, and as a result a number of the variables from the galera_client defaults are now available to override in the galera_server role defaults. In addition, a number of default variables have been generalised, removing the specific _client_ or _server_ parts of the names. Users of this role should check that any overrides they are using have the correct variables names for the new combined role.

  • On OpenStack-Ansible Train release you should upgrade your Debian from 9 (stretch) to 10 (buster). Debian 9 support will be deprecated during the next release of OpenStack-Ansible (Ussuri).

  • Python 2.7 support has been dropped. Last release of openstack ansible to support python 2.7 is OpenStack Train.

  • It is possible that you may need to use the cinder-manage command to migrate volumes to a specific host. In addition, you will have to remove the old rbd:volumes service which will be stale.

  • Variable libvirt_package in ceph_client role has been renamed to libvirt_packages and converted from string to a list.

  • The rabbitMQ high watermark is set to 0.2 rather than 0.4 to prevent possible OOM situations, which limits the maximum memory usage by rabbitMQ to 40% rather than 80% of the memory visible to the rabbitMQ container. The override rabbitmq_memory_high_watermark can be used to alter the limit.

  • The default nova console type has been changed to novnc. Spice is still supported however due to novnc being more actively maintained it is now a better default option.

  • Installation of cloudkitty and its dependent pip packages will now only occur within a Python virtual environment. The cloudkitty_venv_bin and cloudkitty_venv_enabled variables have been removed.

  • The os_cloudkitty role always checks whether the latest package is installed when executed. If a deployer wishes to change the check to only validate the presence of the package, the option cloudkitty_package_state should be set to present.

  • New virtual environments will be created using python3, giving a straightforward transition from python2 to python3 during the major upgrade. The ansible-runtime virtual environment on the deployment host will also be upgraded from python2 to python3 where the operating system allows.

  • The default Mnesia dump_log_write_threshold value has changed to 300 instead of 100 for efficiency. dump_log_write_threshold specifies the maximum number of writes allowed to the transaction log before a new dump of the log is performed. Increasing this value can increase the performances during the queues/exchanges/bindings creation/destroying. The values should be between 100 and 1000. More detail [1].

    [1] http://erlang.org/doc/man/mnesia.html#dump_log_write_threshold

  • The option rabbitmq_disable_non_tls_listeners has been removed in favor of setting the bind address and port configuration directly using a new option rabbitmq_port_bindings. This new option is a hash allowing for multiple bind addresses and port configurations.

  • The repo server no longer uses pypiserver, so it has been removed. Along with this, the following variables have also been removed.

    • repo_pypiserver_port

    • repo_pypiserver_pip_packages

    • repo_pypiserver_package_path

    • repo_pypiserver_bin

    • repo_pypiserver_working_dir

    • repo_pypiserver_start_options

    • repo_pypiserver_init_overrides

  • The variables cloudkitty_requirements_git_repo and cloudkitty_requirements_git_install_branch have been removed in favour of using the URL/path to the upper-constraints file using the variable pip_install_upper_constraints instead.

  • The following Nova tunables have been removed, users need to start using the nova_nova_conf_overrides dictionary to override them. If those values were not previously overridden, there should be no need to override them. - nova_quota_cores - nova_quota_injected_file_content_bytes - nova_quota_injected_file_path_length - nova_quota_injected_files - nova_quota_instances - nova_quota_key_pairs - nova_quota_metadata_items - nova_quota_ram - nova_quota_server_group_members - nova_quota_server_groups - nova_max_instances_per_host - nova_scheduler_available_filters - nova_scheduler_weight_classes - nova_scheduler_driver - nova_scheduler_driver_task_period - nova_rpc_conn_pool_size - nova_rpc_thread_pool_size - nova_rpc_response_timeout - nova_force_config_drive - nova_enable_instance_password - nova_default_schedule_zone - nova_fatal_deprecations - nova_resume_guests_state_on_host_boot - nova_cross_az_attach - nova_remove_unused_resized_minimum_age_seconds - nova_cpu_model - nova_cpu_model_extra_flags

  • The following Nova variables have been removed because they have no effect in the current release of Nova. - nova_max_age - nova_osapi_compute_workers - nova_metadata_workers

  • SESSION_ENGINE has been changed to memcached by default. So that horizon do not require it’s own database for storing session information anymore. horizon_galera_* variables has no effect now.

  • Tacker role now uses default systemd_service role. Due to this upstart is not supported anymore. Was added variable tacker_init_config_overrides, with wich deployer may override predifined options. Also variable program_override has no effect now, and tacker_service_names was removed in favor of tacker_service_name.

  • Gnocchi migrated from usage of Apache mod_wsgi or native daemon to uWSGI daemon. This means, that some variables are not available and has no effect anymore, specifically * gnocchi_use_mod_wsgi * gnocchi_apache_* * gnocchi_ssl* (except gnocchi_ssl_external - it’s still in place) * gnocchi_user_ssl_*

    During upgrade process role will drop gnocchi_service_port from apache listeners (ports.conf) and gnocchi virtualhost, which by default means misconfigured apache service (since it won’t have any listeners) unless it’s aio build and this apache server is in use by other role/service. Apache server won’t be dropped from gnocchi_api hosts, so deployers are encoureged to remove it manually.

  • Panko migrated from usage of Apache mod_wsgi or native daemon to uWSGI daemon. This means, that panko_apache_* variables are not available and has no effect anymore.

    During upgrade process role will drop panko_service_port from apache listeners (ports.conf) and panko virtualhost, which by default means misconfigured apache service (since it won’t have any listeners) unless it’s aio build and this apache server is in use by other role/service. Apache server won’t be dropped from panko_api hosts, so deployers are encoureged to remove it manually.

Deprecation Notes

  • In the ceph_client role, the only valid values for ceph_pkg_source are now ceph and distro. For Ubuntu, the Ubuntu Cloud Archive apt source is already setup by the openstack_hosts role, so there is no need for it to also be setup by the ceph_client role.

  • The variable cloudkitty_requires_pip_packages is no longer required and has therefore been removed.

  • The compression option in the galera_server role has been removed due to the fact that it is not recommended by MariaDB anymore. This means that all the dependencies from Percona such as QPress are no longer necessary.

  • The following variables have been removed because they are no longer used. * galera_percona_xtrabackup_repo * use_percona_upstream * galera_xtrabackup_compression * galera_server_percona_distro_packages

  • The variable galera_xtrabackup_threads has been renamed to galera_mariabackup_threads to reflect the change in the SST provider.

  • To provide compatibility with Centos-8 the LXC cache preparation has been greatly simplified to remove the requirement for machinectl and btrfs, which is a combination not available on Centos-8. This has the side effect of machinectl no longer being a supported backing store for LXC.

  • The PowerVM driver has been removed as it is not tested and it has been broken since late 2016 due to the driver name being renamed to powervm_ext instead of powervm.

  • Support of the legacy neutron L3 tool has been dropped. Deployers are appreciated to use built-in l3-agent options for configuring HA.

  • Fedora is no longer tested in CI for each commit.

  • The log path, /var/log/cloudkitty is no longer used to capture service logs. All logging for the cloudkitty service will now be sent directly to the systemd journal.

  • The deprecated Neutron LBaaS v2 plugin has been removed from the Neutron role.

  • The deprecated Neutron LBaaS v2 plugin support has been removed from openstack-ansible.

  • nova-placement-api has been removed from the os_nova role, along with all nova_placement_* variables. Please review the os_placement role for information about how to configure the new placement service.

  • The rabbitmq server parameters have been replaced by corresponding oslo.messaging RPC and Notify parameters in order to abstract the messaging service from the actual backend server deployment. - cloudkitty_oslomsg_rpc_servers replaces rabbitmq_servers - cloudkitty_oslomsg_rpc_port replaces rabbitmq_port - cloudkitty_oslomsg_rpc_userid replaces rabbitmq_userid - cloudkitty_oslomsg_rpc_vhost replaces rabbitmq_vhost - cloudkitty_oslomsg_rpc_use_ssl replaces rabbitmq_use_ssl - added cloudkitty_oslomsg_notify_servers - added cloudkitty_oslomsg_notify_port - added cloudkitty_oslomsg_notify_use_ssl - added cloudkitty_oslomsg_notify_userid - added cloudkitty_oslomsg_notify_vhost

  • rabbitmq_install_method: file is deprecated now and will be removed in the Wallaby release. As for now supported options are only external_repo and distro. Among with that the following variables are deprecated and prepared for the removal:

    • rabbitmq_package_url

    • rabbitmq_package_sha256

    • rabbitmq_package_path

    Variable rabbitmq_release_version has been removed as not used anymore.

  • The nova-lxd driver is no longer supported upstream, and the git repo for it’s source code has been retired on the master branch. All code for deploying or testing nova-lxd has been removed from the os_nova ansible role. The following variables have been removed:

    • nova_supported_virt_types ‚lxd‘ list entry

    • nova_compute_lxd_pip_packages

    • lxd_bind_address

    • lxd_bind_port

    • lxd_storage_backend

    • lxd_trust_password

    • lxd_storage_create_device

  • Removal of the netloc, netloc_no_port and netorigin filters. Please use the urlsplit filter instead. All usages of the deprecated filters in openstack repos have been updated.

  • Removed PKI token support and the cloudkitty_signing_dir parameter.

  • The py_pkgs and packages_file Ansible lookups are no longer used in OSA and have been removed from the plugins repository.

  • Remove cloudkitty_rpc_backend option due to deprecation of rpc_backend option in oslo.messaging.

  • Support for openSUSE in Ussuri is moved from fully supported to experimental due to insufficient resources to maintain reliable CI jobs. The zuul jobs are moved to be non-voting. Support will be removed entirely in the Victoria cycle.

  • Due to usage of systemd-journald mapping of /openstack/log/ to /var/log/$SERVICE is not present anymore. Also rsyslog_client role is not called for projects since logs are stored in journald. Also variables like service_log_dir are not supported anymore and have no effect.

Bug Fixes

  • ceilometer-polling services running on compute nodes did not have the polling namespace configured. Because of this they used the default value of running all pollsters from the central and compute namespaces. But the pollsters from the central namespace don’t have to run on every compute node. This is fixed by only running the compute pollsters on compute nodes.

  • The RyuBgpDriver is no longer available and replaced by the OsKenBgpDriver of the neutron_dynamic_routing project.

  • Usage of tempest_tempest_conf_overrides, where it could have 2 different formats. one for config_template and another one for tempestconf.profile Now for tempestconf.profile overrides tempest_tempestconf_profile_overrides variable should be used.

Other Notes

  • Ubuntu Trusty (14.04) support is now removed from the os_cloudkitty role.