Replace control plane service under HA¶
Introduction¶
The subordinate hacluster charm is used to provide high availability for OpenStack control plane applications that lack native (built-in) HA functionality. This article demonstrates how to replace a node of such a cluster using Keystone as an example.
Note
Cloud operation Implement HA with a VIP shows how to perform an initial deploy of a service under HA.
Important
This procedure will not result in cloud downtime providing that there is at least one functional service node present at all times.
Procedure¶
A node can be first added, followed by the removal of the unwanted node, or the inverse - removed and then added. This article will use the latter method. One advantage to doing so is the preservation of the original topology.
As is common for Charmed OpenStack clouds, the Keystone node to be removed is a containerised unit residing alongside other containerised services on the same host (hyperconvergence). Removing the unit will therefore remove the container but not the underlying machine.
List the application units¶
Display the units, in this case for the keystone application:
juju status keystone
This article will be based on the following (partial) output:
App Version Status Scale Charm Channel Rev Exposed Message
keystone 21.0.0 active 3 keystone yoga/edge 566 no Application Ready
keystone-hacluster active 3 hacluster 2.0.3/edge 83 no Unit is ready and clustered
keystone-mysql-router 8.0.28 active 3 mysql-router 8.0.19/edge 15 no Unit is ready
Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0/lxd/1 10.246.114.63 5000/tcp Unit is ready
keystone-hacluster/0* active idle 10.246.114.63 Unit is ready and clustered
keystone-mysql-router/0* active idle 10.246.114.63 Unit is ready
keystone/1 active idle 2/lxd/6 10.246.114.80 5000/tcp Unit is ready
keystone-hacluster/1 active idle 10.246.114.80 Unit is ready and clustered
keystone-mysql-router/1 active idle 10.246.114.80 Unit is ready
keystone/2 active idle 1/lxd/5 10.246.114.79 5000/tcp Unit is ready
keystone-hacluster/2 active idle 10.246.114.79 Unit is ready and clustered
keystone-mysql-router/2 active idle 10.246.114.79 Unit is ready
In this example, the node to be removed, the unwanted node, will be represented
by unit keystone/2.
Pause the subordinate hacluster unit¶
Pause the hacluster unit that corresponds to the principal application unit
being removed. Here, unit keystone-hacluster/2 corresponds to unit
keystone/2:
juju run-action --wait keystone-hacluster/2 pause
Remove the unwanted node¶
Remove the unwanted node:
juju remove-unit keystone/2
This will also remove all subordinate units: keystone-hacluster/2 and
keystone-mysql-router/2.
The current state of the model is:
App Version Status Scale Charm Channel Rev Exposed Message
keystone 21.0.0 waiting 2 keystone yoga/edge 566 no Some units are not ready
keystone-hacluster blocked 2 hacluster 2.0.3/edge 83 no Insufficient peer units for ha cluster (require 3)
keystone-mysql-router 8.0.28 active 2 mysql-router 8.0.19/edge 15 no Unit is ready
Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0/lxd/1 10.246.114.63 5000/tcp Unit is ready
keystone-hacluster/0* blocked idle 10.246.114.63 Insufficient peer units for ha cluster (require 3)
keystone-mysql-router/0* active idle 10.246.114.63 Unit is ready
keystone/1 active idle 2/lxd/6 10.246.114.80 5000/tcp Unit is ready
keystone-hacluster/1 blocked idle 10.246.114.80 Insufficient peer units for ha cluster (require 3)
keystone-mysql-router/1 active idle 10.246.114.80 Unit is ready
At this time, Keystone will continue to service requests, and the cloud will remain operational.
Add a principal application unit¶
Scale out the existing keystone application and place the new (containerised) unit on the same host that the removed unit was on (machine 1):
juju add-unit --to lxd:1 keystone
Caution
If network spaces are in use the above command will not succeed. See Juju issue LP #1969523 for a workaround.
It will take a while for the model to settle. Please be patient.
Verify cloud services¶
The final juju status keystone (partial) output is:
App Version Status Scale Charm Channel Rev Exposed Message
keystone 21.0.0 active 3 keystone yoga/edge 566 no Application Ready
keystone-hacluster active 3 hacluster 2.0.3/edge 83 no Unit is ready and clustered
keystone-mysql-router 8.0.28 active 3 mysql-router 8.0.19/edge 15 no Unit is ready
Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0/lxd/1 10.246.114.63 5000/tcp Unit is ready
keystone-hacluster/0* active idle 10.246.114.63 Unit is ready and clustered
keystone-mysql-router/0* active idle 10.246.114.63 Unit is ready
keystone/1 active idle 2/lxd/6 10.246.114.80 5000/tcp Unit is ready
keystone-hacluster/1 active idle 10.246.114.80 Unit is ready and clustered
keystone-mysql-router/1 active idle 10.246.114.80 Unit is ready
keystone/3 active idle 1/lxd/6 10.246.114.79 5000/tcp Unit is ready
keystone-hacluster/9 active idle 10.246.114.79 Unit is ready and clustered
keystone-mysql-router/15 active idle 10.246.114.79 Unit is ready
Ensure that all cloud services are working as expected.
