Integrating 3rd Party Containers in TripleO

One of the following methods can be used to extend or build from scratch custom 3rd party containers.

Extend TripleO Containers

Any extra RPMs required by 3rd party drivers may need to be post-installed into our stock TripleO containers. In this case the 3rd party vendor may opt to add a layer to an existing container in order to deploy their software.

Adding layers to existing containers using TripleO tooling

The example below demonstrates how to extend a container image, where the goal is to create a layer on top of the cinder-volume image that will be named “cinder-cooldriver”.

  • Make sure python-tripleoclient and the dependencies are installed:

    sudo dnf install -y python-tripleoclient

  • Create a vendor directory (which later can be pushed into a git repository):

    mkdir ~/vendor

  • Create a tcib directory under the vendor folder. All container build yaml needs to live in a tcib folder as a root directory.

    mkdir ~/vendor/tcib

  • Create the ~/vendor/containers.yaml which contains the list of images that we want to build:

    container_images:
  - image_source: tripleo
    imagename: localhost/tripleomaster/openstack-cinder-cooldriver:latest

  • Create ~/vendor/tcib/cinder-cooldriver to hold our container image configuration.

    mkdir ~/vendor/tcib/cinder-cooldriver

  • Optionally, add custom files into the build environment.

    mkdir ~/vendor/tcib/cinder-cooldriver/files
cp custom-package.rpm ~/vendor/tcib/cinder-cooldriver/files

  • Create ~/vendor/tcib/cinder-cooldriver/cinder-cooldriver.yaml file which contains the container image configuration:

    ---
# that's the parent layer, here cinder-volume
tcib_from: localhost/tripleomaster/openstack-cinder-volume:latest
tcib_actions:
  - user: root
  - run: mkdir /tmp/cooldriver/example.py
  - run: mkdir -p /rpms
  - run: dnf install -y cooldriver_package
tcib_copies:
  - '{{lookup(''env'',''HOME'')}}/vendor/tcib/cinder-cooldriver/files/custom-package.rpm /rpms'
tcib_gather_files: >
  {{ lookup('fileglob', '~/vendor/tcib/cinder-cooldriver/files/*', wantlist=True) }}
tcib_runs:
  - dnf install -y /rpms/*.rpm
tcib_user: cinder

Note

Here tcib_runs provides a shortcut to tcib_actions:run. See more tcib parameters documented in the tcib role.

  • The result file structure should look something like:

    $ tree vendor
vendor
├── containers.yaml
└── tcib
    └── cinder-cooldriver
            └── cinder-cooldriver.yaml
            └── files
                └── custom-package.rpm

  • Build the vendor container image:

    openstack tripleo container image build \
  --config-file ~/vendor/containers.yaml \
  --config-path ~/vendor

  • Use sudo buildah images command to check if the image was built:

    localhost/tripleomaster/openstack-cinder-cooldriver latest  257592a90133   1 minute ago    1.22 GB

Note

If you want to push the image into a Docker Registry, you can use –push with –registry. Use openstack tripleo container image build –help for more details.

  • Push the image into the TripleO Container registry:

    sudo openstack tripleo container image push \
    --local --registry-url 192.168.24.1:8787 \
    localhost/tripleomaster/openstack-cinder-cooldriver:latest

  • Use openstack tripleo container image list to check if the image was pushed:

    +--------------------------------------------------------------------------------------------------+
| Image Name                                                                                       |
+--------------------------------------------------------------------------------------------------+
| docker://undercloud.ctlplane.localdomain:8787/tripleomaster/openstack-cinder-vendor:latest       |
+--------------------------------------------------------------------------------------------------+

Adding layers to existing containers using Docker

Note

Note that this method has been simplified in Victoria and backported down to train, with the new openstack tripleo container image build command.

The example below demonstrates how to extend a container on the Undercloud host machine. It assumes you are running a local docker registry on the undercloud. We recommend that you create a Dockerfile to extend the existing container. Here is an example extending the cinder-volume container:

FROM 127.0.0.1:8787/tripleo/centos-binary-cinder-volume
MAINTAINER Vendor X
LABEL name="tripleo/centos-binary-cinder-volume-vendorx" vendor="Vendor X" version="2.1" release="1"

# switch to root and install a custom RPM, etc.
USER root
COPY vendor_x.rpm /tmp
RUN rpm -ivh /tmp/vendor_x.rpm

# switch the container back to the default user
USER cinder

Docker build the container above using docker build on the command line. This will output a container image <ID> (used below to tag it). Create a docker tag and push it into the local registry:

docker tag <ID> 127.0.0.1:8787/tripleo/centos-binary-cinder-volume-vendorx:rev1
docker push 127.0.0.1:8787/tripleo/centos-binary-cinder-volume-vendorx:rev1

Start an overcloud deployment as normal with the extra custom Heat environment above to obtain the new container.

Warning

Note that the new container will have the complete software stack built into it as is normal for containers. When other containers are updated and include security fixes in these lower layers, this container will NOT be updated as a result and will require rebuilding.

Building new containers with tripleo container image build

Usage

Use the following command to build all of the container images used in TripleO:

openstack tripleo container image build

Different options are provided for advanced usage. They can be discovered by using –help argument. Here are some of them:

  • –config-file to use a custom YAML config file specifying the images to build.

  • –config-path to use a custom base configuration path. This is the base path for all container-image files. If this option is set, the default path for <config-file> will be modified.

  • –extra-config to apply additional options from a given configuration YAML file. This will apply to all containers built.

  • –exclude to skip some containers during the build.

  • –registry to specify a Container Registry where the images will be pushed.

  • –authfile to specify an authentication file if the Container Registry requires authentication.

  • –skip-build if we don’t want to build and push images. It will only generate the configuration files.

  • –push to push the container images into the Container Registry.

  • –volume to overrides the default bind mounts needed when the container images are built. If you use this argument, don’t forget that you might need to include the default ones.

  • –work-dir to specify the place where the configuration files will be generated.

Tips and Tricks with tripleo_container_image_build

Here’s a non-exaustive list of tips and tricks that might make things faster, especially on a dev env where you need to build multiple times the containers.

Inject a caching proxy

Using a caching proxy can make things faster when it comes to package fetching.

One of the way is to either expose the dnf.conf/yum.conf using –volume. Since dnf.conf is edited during the container build, you want to expose a copy of your host config:

sudo cp -r /etc/dnf /srv/container-dnf
openstack tripleo container image build --volume /srv/container-dnf:/etc/dnf:z

Another way is to expose the http_proxy and https_proxy environment variable.

In order to do so, create a simple yaml file, for instance ~/proxy.yaml:

---
tcib_envs:
  LANG: en_US.UTF-8
  container: oci
  http_proxy: http://PROXY_HOST:PORT
  https_proxy: http://PROXY_HOST:PORT

Then, pass that file using the –extra-config parameter:

openstack tripleo container image build --extra-config proxy.yaml

And you’re set.

Note

Please ensure you also pass the default values, since ansible isn’t configured to merge dicts/lists by default.

Get a minimal environment to build containers

As a dev, you might want to get a daily build of your container images. While you can, of course, run this on an Undercloud, you actually don’t need an undercloud: you can use this playbook from tripleo-operator-ansible project

With this, you can set a nightly cron that will ensure you’re always getting latest build on your registry.

Building new containers with kolla-build

Note

Note that this method will be deprecated during the Victoria cycle and replaced by the new openstack tripleo container image build command.

To create new containers, or modify existing ones, you can use kolla-build from the Kolla project to build and push the images yourself. The command to build a new containers is below. Note that this assumes you are on an undercloud host where the registry IP address is 192.168.24.1.

Configure Kolla to build images for TripleO, in /etc/kolla/kolla-build.conf:

[DEFAULT]
base=centos
type=binary
namespace=master
registry=192.168.24.1:8787
tag=latest
template_override=/usr/share/tripleo-common/container-images/tripleo_kolla_template_overrides.j2
rpm_setup_config=http://trunk.rdoproject.org/centos7/current-tripleo/delorean.repo,http://trunk.rdoproject.org/centos7/delorean-deps.repo
push=True

Use the following command to build all of the container images used in TripleO:

openstack overcloud container image build \
      --config-file /usr/share/tripleo-common/container-images/overcloud_containers.yaml \
      --kolla-config-file /etc/kolla/kolla-build.conf

Note

Add –use-buildah argument to use Buildah instead of Docker. It’ll be the default once CentOS8 becomes the testing platform during the Train cycle and onward.

Or use kolla-build to build the images yourself, which provides more flexibility and allows you to rebuild selectively just the images matching a given name, for example to build only the heat images with the TripleO customization:

kolla-build heat

Notice that TripleO already uses the /usr/share/tripleo-common/container-images/tripleo_kolla_template_overrides.j2 to add or change specific aspects of the containers using the kolla template override mechanism. This file can be copied and modified to create custom containers. The original copy of this file can be found in the tripleo-common repository.

The following template is an example of the template used for building the base images that are consumed by TripleO. In this case we are adding the puppet RPM to the base image:

{% extends parent_template %}
{% set base_centos_binary_packages_append = ['puppet'] %}

Integrating 3rd party containers with tripleo-heat-templates

The TripleO Heat Templates repo is where most of the logic resides in the form of heat templates. These templates define each service, the containers’ configuration and the initialization or post-execution operations.

The docker templates can be found under the docker sub directory in the tripleo-heat-templates root. The services files are under the docker/service directory.

For more information on how to integrate containers into the TripleO Heat templates, see the Containerized TripleO architecture document.

If all you need to do is change out a container for a specific service, you can create a custom heat environment file that contains your override. To swap out the cinder container from our previous example we would add:

parameter_defaults:
    ContainerCinderVolumeImage: centos-binary-cinder-volume-vendorx:rev1

Note

Image parameters were named Docker*Image prior to the Train cycle.

3rd party kernel modules

Some applications (like Neutron or Cinder plugins) require specific kernel modules to be installed and loaded on the system.

We recommend two different methods to deploy and load these modules.

kernel module is deployed on the host

The kernel module is deployed on the base Operating System via RPM or DKMS. It is suggested to deploy the module via virt-customize. The libguestfs-tools package contains the virt-customize tool. Install the libguestfs-tools:

sudo yum install libguestfs-tools

Then you need to create a repository file where the module will be downloaded from, and uplaod the repo into the image:

virt-customize --selinux-relabel -a overcloud-full.qcow2 --upload my-repo.repo:/etc/yum.repos.d/

Once the repository is deployed, you can now install the rpm that contains the kernel module:

virt-customize --selinux-relabel -a overcloud-full.qcow2 --install my-rpm

Now that the rpm is deployed with the kernel module, we need to configure TripleO to load it. To configure an extra kernel module named “dpdk_module” for a specific role, we would add:

parameter_defaults:
  ControllerExtraKernelModules:
    dpdk_module: {}

Since our containers don’t get their own kernels, we load modules on the host. Therefore, ExtraKernelModules parameter is used to configure which modules we want to configure. This parameter will be applied to the Puppet manifest (in the kernel.yaml service). The container needs the modules mounted from the host, so make sure the plugin template has the following configuration (at minimum):

volumes:
  - /lib/modules:/lib/modules:ro

However, this method might be problematic if RPMs dependencies are too complex to deploy the kernel module on the host.

kernel module is containerized

Kernel modules can be loaded from the container. The module can be deployed in the same container as the application that will use it, or in a separated container.

Either way, if you need to run a privileged container, make sure to set this parameter:

privileged: true

If privilege mode isn’t required, it is suggested to set it to false for security reasons.

Kernel modules will need to be loaded when the container will be started by Docker. To do so, it is suggested to configure the composable service which deploys the module in the container this way:

kolla_config:
  /var/lib/kolla/config_files/neutron_ovs_agent.json:
  command: /dpdk_module_launcher.sh
docker_config_scripts:
  dpdk_module_launcher.sh:
    mode: "0755"
    content: |
      #!/bin/bash
      set -xe
      modprobe dpdk_module
docker_config:
  step_3:
    neutron_ovs_bridge:
      volumes:
        list_concat:
          - {get_attr: [ContainersCommon, volumes]}
          -
            - /var/lib/docker-config-scripts/dpdk_module_launcher.sh:/dpdk_module_launcher.sh:ro

That way, the container will be configured to load the module at start, so the operator can restart containers without caring about loading the module manually.