Role - tripleo_container_manage¶
Role Documentation¶
Welcome to the “tripleo_container_manage” role documentation.
Role Defaults¶
This section highlights all of the defaults and variables set within the “tripleo_container_manage” role.
# All variables intended for modification should place placed in this file.
tripleo_container_manage_hide_sensitive_logs: '{{ hide_sensitive_logs | default(true)
}}'
tripleo_container_manage_debug: '{{ ((ansible_verbosity | int) >= 2) | bool }}'
tripleo_container_manage_clean_orphans: true
# All variables within this role should have a prefix of "tripleo_container_manage"
tripleo_container_manage_check_puppet_config: false
tripleo_container_manage_cli: podman
tripleo_container_manage_concurrency: 1
tripleo_container_manage_config: /var/lib/tripleo-config/
tripleo_container_manage_config_id: tripleo
tripleo_container_manage_config_overrides: {}
tripleo_container_manage_config_patterns: '*.json'
# Some containers where Puppet is run, can take up to 10 minutes to finish
# in slow environments.
tripleo_container_manage_create_retries: 120
# Default delay is 5s so 120 retries makes a timeout of 10 minutes which is
# what we have observed a necessary value for nova and neutron db-sync execs.
tripleo_container_manage_exec_retries: 120
tripleo_container_manage_healthcheck_disabled: false
tripleo_container_manage_log_path: /var/log/containers/stdouts
tripleo_container_manage_systemd_order: false
tripleo_container_manage_systemd_teardown: true
tripleo_container_manage_valid_exit_code: []
Molecule Scenarios¶
Molecule is being used to test the “tripleo_container_manage” role. The following section highlights the drivers in service and provides an example playbook showing how the role is leveraged.
- Driver: delegated
Scenario: default¶
Molecule Options¶
managed: false
login_cmd_template: >-
ssh
-o UserKnownHostsFile=/dev/null
-o StrictHostKeyChecking=no
-o Compression=no
-o TCPKeepAlive=yes
-o VerifyHostKeyDNS=no
-o ForwardX11=no
-o ForwardAgent=no
{instance}
ansible_connection_options:
ansible_connection: ssh
Molecule Inventory¶
hosts:
all:
hosts:
instance:
ansible_host: localhost
ansible_connection: local
ansible_distribution: centos8
Example default playbook¶
- name: Create fedora container from /tmp/container-configs with old healthcheck
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_healthcheck_disabled: true
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: fedora.json
tripleo_container_manage_systemd_order: true
tasks:
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Verify that Fedora container was created correctly and manually create old
healthcheck for migration testing
when:
- not ansible_check_mode|bool
block:
# Reproduce what was done before to create and enable healthchecks
- name: Enable and start systemd timers
systemd:
state: started
name: tripleo_fedora_healthcheck.timer
enabled: true
daemon_reload: false
- name: Add systemd requires for healthchecks
command: systemctl add-requires tripleo_fedora.service tripleo_fedora_healthcheck.timer
# Check that migration is ready to be tested
- name: Check for fedora container
command: podman container exists fedora
- name: Check if tripleo_fedora systemd healthcheck service is active
command: systemctl is-active --quiet tripleo_fedora_healthcheck.timer
register: tripleo_fedora_healthcheck_active_result
- name: Assert that tripleo_fedora systemd healthcheck service is active
assert:
that:
- tripleo_fedora_healthcheck_active_result.rc == 0
fail_msg: tripleo_fedora systemd healthcheck service is not active
success_msg: tripleo_fedora systemd healthcheck service is active
- name: Create all containers from /tmp/container-configs
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: '*.json'
tripleo_container_manage_systemd_order: true
tripleo_container_manage_valid_exit_code: [0]
tasks:
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Verify that Fedora container was created correctly
when:
- not ansible_check_mode|bool
block:
- name: Check for fedora container
command: podman container exists fedora
- name: Gather facts about fedora container
podman_container_info:
name: fedora
register: fedora_infos
- name: Assert that fedora container has the right image
assert:
that:
- "'fedora:latest' in fedora_infos.containers.0.ImageName"
fail_msg: fedora container has wrong image
success_msg: fedora container has the right image
- name: Check if tripleo_fedora systemd service is active
command: systemctl is-active --quiet tripleo_fedora
register: tripleo_fedora_active_result
- name: Assert that tripleo_fedora systemd service is active
assert:
that:
- tripleo_fedora_active_result.rc == 0
fail_msg: tripleo_fedora systemd service is not active
success_msg: tripleo_fedora systemd service is active
- name: Check if tripleo_fedora healthcheck is active and healthy
assert:
that:
- "'healthy' in fedora_infos.containers.0.State.Healthcheck.Status"
fail_msg: fedora container healthcheck is not healthy
success_msg: fedora container healthcheck is healthy
- name: Verify that Fedora systemd healthcheck container was removed correctly
command: systemctl is-active --quiet tripleo_fedora_healthcheck.timer
register: tripleo_fedora_healthcheck_active_result
failed_when:
- tripleo_fedora_healthcheck_active_result.rc == 0
- name: Verify that Fedora bis container was created correctly
block:
- name: Check for fedora_bis container
command: podman container exists fedora_bis
- name: Gather facts about fedora_bis container
podman_container_info:
name: fedora_bis
register: fedora_bis_infos
- name: Assert that fedora_bis container has the right image
assert:
that:
- "'fedora:latest' in fedora_bis_infos.containers.0.ImageName"
fail_msg: fedora_bis container has wrong image
success_msg: fedora_bis container has the right image
- name: Verify that Fedora three container was created correctly
block:
- name: Check for fedora_three container
command: podman container exists fedora_three
- name: Gather facts about fedora_three container
podman_container_info:
name: fedora_three
register: fedora_three_infos
- name: Assert that fedora_three container has the right image
assert:
that:
- "'fedora:latest' in fedora_three_infos.containers.0.ImageName"
fail_msg: fedora_three container has wrong image
success_msg: fedora_three container has the right image
- name: Test idempotency on fedora container
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: '*.json'
tripleo_container_manage_systemd_order: true
tasks:
- name: Gather facts about fedora container before new run
podman_container_info:
name: fedora
register: fedora_infos_old
when:
- not ansible_check_mode|bool
- include_role:
name: tripleo_container_manage
- name: Gather facts about fedora container after new run
podman_container_info:
name: fedora
register: fedora_infos_new
when:
- not ansible_check_mode|bool
post_tasks:
- name: Assert that fedora container has not been re-created
assert:
that:
- fedora_infos_new['containers'][0]['Id'] == fedora_infos_old['containers'][0]['Id']
fail_msg: fedora container was wrongly re-created
success_msg: fedora container was not re-created
when:
- not ansible_check_mode|bool
- name: Test systemd state on fedora container after a manual stop
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: '*.json'
tripleo_container_manage_systemd_order: true
tasks:
- name: Stop systemd service for tripleo_fedora in a manual stop
systemd:
name: tripleo_fedora.service
state: stopped
enabled: false
daemon_reload: true
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Check if tripleo_fedora systemd service is active after a manual stop
command: systemctl is-active --quiet tripleo_fedora
register: tripleo_fedora_active_result
- name: Assert that tripleo_fedora systemd service is active after a manual stop
when:
- not ansible_check_mode|bool
assert:
that:
- tripleo_fedora_active_result.rc == 0
fail_msg: tripleo_fedora systemd service is not active after a manual stop
success_msg: tripleo_fedora systemd service is active after a manual stop
- name: Manage only one container
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: fedora.json
tripleo_container_manage_clean_orphans: false
tripleo_container_manage_config_overrides:
fedora:
image: fedora:rawhide
tasks:
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Verify that all containers still exist
when:
- not ansible_check_mode|bool
block:
- name: Check for fedora container
command: podman container exists fedora
- name: Gather facts about fedora container
podman_container_info:
name: fedora
register: fedora_infos
- name: Assert that fedora container has the right image
assert:
that:
- "'fedora:rawhide' in fedora_infos.containers.0.ImageName"
fail_msg: fedora container has wrong image
success_msg: fedora container has the right image
- name: Check if tripleo_fedora systemd service is active
command: systemctl is-active --quiet tripleo_fedora
register: tripleo_fedora_active_result
- name: Assert that tripleo_fedora systemd service is active
assert:
that:
- tripleo_fedora_active_result.rc == 0
fail_msg: tripleo_fedora systemd service is not active
success_msg: tripleo_fedora systemd service is active
- name: Check if tripleo_fedora healthcheck is active and healthy
assert:
that:
- "'healthy' in fedora_infos.containers.0.State.Healthcheck.Status"
fail_msg: fedora container healthcheck is not healthy
success_msg: fedora container healthcheck is healthy
- name: Check for fedora_bis container
command: podman container exists fedora_bis
- name: Check for fedora_three container
command: podman container exists fedora_three
- name: Manage a wrong container (user error)
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: feduraaa.json
tripleo_container_manage_clean_orphans: false
tasks:
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Verify that all containers still exist
when:
- not ansible_check_mode|bool
block:
- name: Check for fedora container
command: podman container exists fedora
- name: Gather facts about fedora container
podman_container_info:
name: fedora
register: fedora_infos
- name: Check if tripleo_fedora systemd service is active
command: systemctl is-active --quiet tripleo_fedora
register: tripleo_fedora_active_result
- name: Assert that tripleo_fedora systemd service is active
assert:
that:
- tripleo_fedora_active_result.rc == 0
fail_msg: tripleo_fedora systemd service is not active
success_msg: tripleo_fedora systemd service is active
- name: Check if tripleo_fedora healthcheck is active and healthy
assert:
that:
- "'healthy' in fedora_infos.containers.0.State.Healthcheck.Status"
fail_msg: fedora container healthcheck is not healthy
success_msg: fedora container healthcheck is healthy
- name: Check for fedora_bis container
command: podman container exists fedora_bis
- name: Check for fedora_three container
command: podman container exists fedora_three
- name: Test a container removal
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: fedora_*.json
tripleo_container_manage_systemd_order: true
tasks:
- name: Remove fedora container config
file:
path: /tmp/container-configs/fedora.json
state: absent
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Verify that all containers still exist
when:
- not ansible_check_mode|bool
block:
- name: Check that fedora container was removed
command: podman container exists fedora
register: container_exist
failed_when: container_exist.rc == 0
- name: Check if tripleo_fedora systemd service is still active
command: systemctl is-active --quiet tripleo_fedora
register: tripleo_fedora_active_result
failed_when: tripleo_fedora_active_result.rc == 0
- name: Check for fedora_bis container
command: podman container exists fedora_bis
- name: Check for fedora_three container
command: podman container exists fedora_three
- name: Test a container update
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: fedora_*.json
tripleo_container_manage_systemd_order: true
tasks:
- name: Modify the fedora_bis container config
copy:
content: |
{
"image": "fedora:rawhide",
"net": "host",
"command": "sleep 10"
}
dest: /tmp/container-configs/fedora_bis.json
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Verify that Fedora bis container was re-created correctly
when:
- not ansible_check_mode|bool
block:
- name: Check for fedora_bis container
command: podman container exists fedora_bis
- name: Gather facts about fedora_bis container
podman_container_info:
name: fedora_bis
register: fedora_bis_infos
- name: Assert that fedora_bis container has the right image
assert:
that:
- "'fedora:rawhide' in fedora_bis_infos.containers.0.ImageName"
fail_msg: fedora_bis container has wrong image
success_msg: fedora_bis container has the right image
- name: Check for fedora_three container
command: podman container exists fedora_three
when:
- not ansible_check_mode|bool
- name: Test a container config override
become: true
hosts: all
gather_facts: false
vars:
tripleo_container_manage_config: /tmp/container-configs
tripleo_container_manage_debug: true
tripleo_container_manage_config_patterns: fedora_*.json
tripleo_container_manage_systemd_order: true
tripleo_container_manage_config_overrides:
fedora_bis:
image: fedora:latest
tasks:
- include_role:
name: tripleo_container_manage
post_tasks:
- name: Verify that Fedora bis container was re-created correctly
when:
- not ansible_check_mode|bool
block:
- name: Check for fedora_bis container
command: podman container exists fedora_bis
- name: Gather facts about fedora_bis container
podman_container_info:
name: fedora_bis
register: fedora_bis_infos
- name: Assert that fedora_bis container has the right image
assert:
that:
- "'fedora:latest' in fedora_bis_infos.containers.0.ImageName"
fail_msg: fedora_bis container has wrong image
success_msg: fedora_bis container has the right image
- name: Check for fedora_three container
command: podman container exists fedora_three
when:
- not ansible_check_mode|bool
Usage¶
Note that right now, only Podman is supported by this role. Docker support is in the roadmap though.
This Ansible role allows to do the following tasks:
Collect container configs data, generated by TripleO Heat Templates. This data is used as a source of truth on which configuration we expect to apply with this role. It means that if a container is already managed by this role, no matter its state now, the configs data will reconfigure the container if needed.
Manage systemd shutdown files. It takes care of cleaning up the Paunch services and files and create the TripleO Container systemd service, required for service ordering when it comes to shutdown or start a node. It also manages the netns-placeholder service.
Delete containers that aren’t needed anymore or that will need to be re-configured. It uses a custom filter, named needs_delete() which has a set of rules which allow to determine if whether or not the container needs to be deleted. These reasons will make the containers not deleted:
The container is not managed by tripleo_ansible.
The container config_id doesn’t match with the one in input.
Once the previous conditions checked, then these reasons will make the containers deleted:
The container has no config_data.
The container has a config_data which doesn’t match the one in input.
Note that when a container is removed, the role also disable and remove the systemd services and healtchecks if present.
Create containers in a specific order defined by start_order container config, where default is 0.
If the container is an exec, we’ll run a dedicated playbook for execs, using async so multiple execs can be run at the same time.
Otherwise, the podman_container is used, in async, to create the containers. If the container has a restart policy, we’ll configure the systemd service. If the container has a healthcheck script, we’ll configure the systemd healthcheck service.
Note: tripleo_container_manage_concurrency parameter is set to 1 by default, and putting higher value than 2 can be expose issue with Podman locks. If a container is meant to exit after running a script (defined in EntryPoint), we can check its return code and fail if the code isn’t expected. It can be done with tripleo_container_manage_valid_exit_code. If defined to a list of integers, the role will wait for the container to be exited and then checks the return code.
Here is an example of a playbook:
- name: Manage step_1 containers using tripleo-ansible
block:
- name: "Manage containers for step 1 with tripleo-ansible"
include_role:
name: tripleo_container_manage
vars:
tripleo_container_manage_systemd_order: true
tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1"
tripleo_container_manage_config_id: "tripleo_step1"
Roles variables¶
Name |
Default Value |
Description |
---|---|---|
tripleo_container_manage_cli |
podman |
Container CLI |
tripleo_container_manage_concurrency |
1 |
Number of containers managed at same time |
tripleo_container_manage_config |
/var/lib/tripleo-config/ |
Container config path |
tripleo_container_manage_config_id |
tripleo |
Config ID |
tripleo_container_manage_config_patterns |
*.json |
Bash REGEX to find configs |
tripleo_container_manage_debug |
false |
Debug toggle |
tripleo_container_manage_healthcheck_disable |
false |
Allow to disable Healthchecks |
tripleo_container_manage_log_path |
/var/log/containers/stdouts |
Containers stdouts path |
tripleo_container_manage_systemd_order |
false |
Manage systemd shutdown ordering |
tripleo_container_manage_config_overrides |
{} |
Allows to override any container configuration |
tripleo_container_manage_clean_orphans |
true |
Option to clean orphans |
tripleo_container_manage_valid_exit_code |
[] |
Allow to check if a container returned the exit code in parameter. Must be a list. e.g. [0,3] |
Healthchecks¶
Previously, the container healthcheck was implemented by a systemd timer which
would run podman exec
to determine if a given container was healthy..
Now, we are using the native healthcheck interface in Podman; which is easier
to integrate and consume.
We are now using the native healthcheck interface in Podman; which is easier to integrate with and consume.
To check if a container (e.g. keystone) is healthy, run the following command:
$ sudo podman healthcheck run keystone
The return code should be 0 and “healthy” should be printed as the output.
One can also use the podman inspect keystone
output to figure out that
the healthcheck is periodically running and healthy:
"Healthcheck": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2020-04-14T18:48:57.272180578Z",
"End": "2020-04-14T18:48:57.806659104Z",
"ExitCode": 0,
"Output": ""
},
(...)
]
}
Debug¶
The role allows you to perform specific actions on a given container. This can be used to:
Run a container with a specific one-off configuration.
Output the container commands that are run to to manage containers lifecycle.
Output the changes that would have been made on containers by Ansible.
Note
To manage a single container, you need to know 2 things:
At which step the container is deployed.
The name of the generated JSON file for container config.
Here is an example of a playbook to manage HAproxy container at step 1 which overrides the image setting in one-off.
- hosts: localhost
become: true
tasks:
- name: Manage step_1 containers using tripleo-ansible
block:
- name: "Manage HAproxy container at step 1 with tripleo-ansible"
include_role:
name: tripleo_container_manage
vars:
tripleo_container_manage_systemd_order: true
tripleo_container_manage_config_patterns: 'haproxy.json'
tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1"
tripleo_container_manage_config_id: "tripleo_step1"
tripleo_container_manage_clean_orphans: false
tripleo_container_manage_config_overrides:
haproxy:
image: docker.io/tripleomaster/centos-binary-haproxy:hotfix
If Ansible is run in check mode, no container will be removed nor created, however at the end of the playbook a list of commands will be displayed to show what would have been run. This is useful for debug purposes, as it was something that one could do with paunch debug command.
$ ansible-playbook haproxy.yaml --check
Adding the diff mode will output the changes what would have been made on containers by Ansible.
$ ansible-playbook haproxy.yaml --check --diff
The tripleo_container_manage_clean_orphans
parameter is optional
and can be set to false to not clean orphaned containers for a
config_id. It can be used to manage a single container without
impacting other running containers with same config_id.
The tripleo_container_manage_config_overrides
parameter is optional
and can be used to override a specific container attribute like the image
or the container user. The parameter takes a dictionary where each key is the
container name and its parameters that we want to override. These parameters
have to exist and are the ones that define the container configuration in
TripleO Heat Templates. Note that it doesn’t write down the overrides in the
JSON file so if an update / upgrade is executed, the container will be
re-configured with the configuration that is in the JSON file.