Pre-caching images on Compute Nodes¶

Fetching an image from Glance is often the most time consuming step when booting an instance. This can be particularly significant in cases where the data must traverse a high latency or limited bandwidth network, for example with Distributed Compute Node deployment.

An ansible playbook is available to ensure images are already cached in cases where instance creation times must be minimized.

Ussuri

Since Ussuri Nova also provides an API to pre-cache images on Compute nodes. See the Nova Image pre-caching documentation.

Note

The Nova Image Cache is not used when using Ceph RBD for Glance images and Nova ephemeral disk. See Nova Image Caching documentation.

Image Cache Cleanup¶

The nova-compute service remains responsible for cleaning up old unused images on a compute node. A periodic job examines each of the images that are not currently used by an instance on the host. If an image is older than the configured maximum age it will be removed.

When an image is pre-cached the modification time is set to the current time. This effectively sets the image age back to 0. Therefore the pre-caching task should be repeated on an interval that is less than the maximum image age to ensure images remain cached.

Configuring the maximum image age¶

The default maximum image age is 86400 seconds (24 hours). This can be increased for all computes by setting NovaImageCacheTTL in the deployment parameters:

[stack@undercloud-0 ~]$ cat nova-cache-environment.yaml
parameter_defaults:
  # Set the max image age to 30 days
  NovaImageCacheTTL: 2592000

Alternatively NovaImageCacheTTL can be set for individual compute roles:

[stack@undercloud-0 ~]$ cat nova-cache-environment.yaml
parameter_defaults:
  # Set the max image age to 30 days for the ComputeSite1 role
  ComputeSite1Parameters:
     NovaImageCacheTTL: 2592000
  # Set the max image age to 7 days for the ComputeSite2 role
  ComputeSite2Parameters:
     NovaImageCacheTTL: 604800
  # Any other Compute roles default to 86400

Pre-caching a list of images on all Compute nodes¶

Get an ansible inventory for the stack name (default overcloud):

Wallaby

tripleo-ansible-inventory is deprecated as of Wallaby.

[stack@undercloud-0 ~]$ mkdir -p inventories
[stack@undercloud-0 ~]$ . stackrc
(undercloud) [stack@undercloud-0 ~]$ tripleo-ansible-inventory \
  --plan overcloud --static-yaml-inventory inventories/inventory.yaml

[stack@undercloud-0 ~]$ find ~/overcloud-deploy/*/config-download \
  -name tripleo-ansible-inventory.yaml |\
  while read f; do cp $f inventories/$(basename $(dirname $f)).yaml; done

Determine the list of image IDs to pre-cache:

[stack@undercloud-0 ~]$ . overcloudrc
(overcloud) [stack@undercloud-0 ~]$ openstack image list
+--------------------------------------+---------+--------+
| ID                                   | Name    | Status |
+--------------------------------------+---------+--------+
| 07bc2424-753b-4f65-9da5-5a99d8383fe6 | image_0 | active |
| d5187afa-c821-4f22-aa4b-4e76382bef86 | image_1 | active |
+--------------------------------------+---------+--------+

Add the image ids to an argument file for the ansible playbook:

(overcloud) [stack@undercloud-0 ~]$ cat <<EOF > nova_cache_args.yml
tripleo_nova_image_cache_images:
  - id: 07bc2424-753b-4f65-9da5-5a99d8383fe6
  - id: d5187afa-c821-4f22-aa4b-4e76382bef86
EOF

Source the overcloud rc file to provide the necessary credentials for image download:

[stack@undercloud-0 ~]$ . overcloudrc

Run the tripleo_nova_image_cache playbook:

(overcloud) [stack@undercloud-0 ~]$ ansible-playbook -i inventories --extra-vars "@nova_cache_args.yml" /usr/share/ansible/tripleo-playbooks/tripleo_nova_image_cache.yml

PLAY [TripleO Nova image cache management] ***************************************************************************************************************************************************************************************************

TASK [tripleo-nova-image-cache : Cache image 07bc2424-753b-4f65-9da5-5a99d8383fe6] ***********************************************************************************************************************************************************
changed: [compute-0]
changed: [compute-1]

TASK [tripleo-nova-image-cache : Cache image d5187afa-c821-4f22-aa4b-4e76382bef86] ***********************************************************************************************************************************************************
changed: [compute-0]
changed: [compute-1]

Note

If the image already exists in cache then no change is reported however the image modification time is updated

Warning

The ansible forks config option (default=5) will affect the number of concurrent image downloads. Consider the load on the image service if adjusting this.

Multi-stacks inventory¶

When a multi-stack deployment is used, such as in Distributed Compute Node deployment and Deploy an additional nova cell v2, a merged inventory allows images to be cached on all compute nodes with a single playbook run.

For each deployed stack, its ansible inventory is generated in overcloud-deploy/<stack>/config-download/tripleo-ansible-inventory.yaml. Collect all inventories under the inventories directory:

Wallaby

tripleo-ansible-inventory is deprecated as of Wallaby. A multi-stack inventory can be created by specifying a comma separated list of stacks:

[stack@undercloud-0 ~]$ mkdir -p inventories
[stack@undercloud-0 ~]$ . stackrc
(undercloud) [stack@undercloud-0 ~]$ tripleo-ansible-inventory \
  --plan overcloud,site1,site2 \
  --static-yaml-inventory inventories/multiinventory.yaml

[stack@undercloud-0 ~]$ mkdir -p inventories
[stack@undercloud-0 ~]$ find ~/overcloud-deploy/*/config-download \
  -name tripleo-ansible-inventory.yaml |\
  while read f; do cp $f inventories/$(basename $(dirname $f)).yaml; done

When all inventory files are stored in a single directory, ansible merges it. The playbook can then be run once as in Pre-caching a list of images on all Compute nodes to pre-cache on all compute nodes.

Pre-caching on one node and distributing to remaining nodes¶

In the case of a Distributed Compute Node deployment it may be desirable to transfer an image to a single compute node at a remote site and then redistribute it from that node to the remaining compute nodes. The SSH/SCP configuration that exists between the compute nodes to support cold migration/resize is reused for this purpose.

Warning

SSH/SCP is inefficient over high latency networks. The method should only be used when the compute nodes targeted by the playbook are all within the same site. To ensure this is the case set tripleo_nova_image_cache_plan to the stack name of the site. Multiple runs of ansible-playbook are then required, targeting a different site each time.

To enable this simply set tripleo_nova_image_cache_use_proxy: true in the arguments file. The image is distributed from the first compute node by default. To use a specific compute node also set tripleo_nova_image_cache_proxy_hostname.

For example:

(central) [stack@undercloud-0 ~]$ cat <<EOF > dcn1_nova_cache_args.yml
tripleo_nova_image_cache_use_proxy: true
tripleo_nova_image_cache_proxy_hostname: dcn1-compute-1
tripleo_nova_image_cache_images:
  - id: 07bc2424-753b-4f65-9da5-5a99d8383fe6
tripleo_nova_image_cache_plan: dcn1
EOF

(central) [stack@undercloud-0 ~]$ ansible-playbook -i inventories --extra-vars "@dcn1_nova_cache_args.yml" /usr/share/ansible/tripleo-playbooks/tripleo_nova_image_cache.yml

PLAY [TripleO Nova image cache management] ***************************************************************************************************************************************************************************************************

TASK [tripleo-nova-image-cache : Show proxy host] ********************************************************************************************************************************************************************************************
ok: [dcn-compute-0] => {
    "msg": "Proxy host is dcn-compute-1"
}

TASK [tripleo-nova-image-cache : Cache image 07bc2424-753b-4f65-9da5-5a99d8383fe6] ***********************************************************************************************************************************************************
skipping: [dcn1-compute-0]
changed: [dcn1-compute-1]

TASK [tripleo-nova-image-cache : Cache image (via proxy) 07bc2424-753b-4f65-9da5-5a99d8383fe6] ***********************************************************************************************************************************************
skipping: [dcn1-compute-1]
changed: [dcn1-compute-0]

(central) [stack@undercloud-0 ~]$ cat <<EOF > dcn2_nova_cache_args.yml
tripleo_nova_image_cache_use_proxy: true
tripleo_nova_image_cache_images:
  - id: 07bc2424-753b-4f65-9da5-5a99d8383fe6
tripleo_nova_image_cache_plan: dcn2
EOF

(central) [stack@undercloud-0 ~]$ ansible-playbook -i inventories --extra-vars "@dcn2_nova_cache_args.yml" /usr/share/ansible/tripleo-playbooks/tripleo_nova_image_cache.yml

PLAY [TripleO Nova image cache management] ***************************************************************************************************************************************************************************************************
...
...

Warning

The ansible forks config option (default=5) will affect the number of concurrent SCP transfers. Consider the load on the proxy compute node if adjusting this.

Pre-caching images on Compute Nodes

Pre-caching images on Compute Nodes¶

Image Cache Cleanup¶

Configuring the maximum image age¶

Pre-caching a list of images on all Compute nodes¶

Multi-stacks inventory¶

Pre-caching on one node and distributing to remaining nodes¶

TripleO 3.0.0

Page Contents