Scaling a Deployment¶

Assumptions¶

This guide, using a devstack installation of OpenStack, assumes that:

1. You have configured devstack from Single Machine Installation Guide;
2. You have set up heat on devstack, as defined at heat and DevStack;
3. You have installed HAProxy on the devstack server.

Architecture¶

This section shows the basic heat architecture, the load balancing mechanism used and the target scaled out architecture.

                  |- [REST API] -|
[CLI] -- <HTTP> --                -- <AMQP> -- [ENGINE]
                  |- [CFN API]  -|

                             |- [REST-API] -|
                             |-    ...     -|
                             |- [REST-API] -|             |- [ENGINE] -|
[CLI] -- <HTTP> -- [PROXY] --                -- <AMQP> -- |-    ...   -|
                             |- [API-CFN]  -|             |- [ENGINE] -|
                             |-    ...     -|
                             |- [API-CFN]  -|

Deploying Multiple APIs¶

In order to run a heat component separately, you have to execute one of the python scripts located at the bin directory of your heat repository.

These scripts take as argument a configuration file. When using devstack, the configuration file is located at /etc/heat/heat.conf. For instance, to start new REST and CFN API services, you must run:

python bin/heat-api --config-file=/etc/heat/heat.conf
python bin/heat-api-cfn --config-file=/etc/heat/heat.conf

Each API service must have a unique address to listen. This address have to be defined in the configuration file. For REST and CFN APIs, modify the [heat_api] and [heat_api_cfn] blocks, respectively.

[heat_api]
bind_port = {API_PORT}
bind_host = {API_HOST}

...

[heat_api_cfn]
bind_port = {API_CFN_PORT}
bind_host = {API_CFN_HOST}

If you wish to run multiple API processes on the same machine, you must create multiple copies of the heat.conf file, each containing a unique port number.

In addition, if you want to run some API services in different machines than the devstack server, you have to update the loopback addresses found at the sql_connection and rabbit_host properties to the devstack server’s IP, which must be reachable from the remote machine.

Deploying Multiple Engines¶

All engines must be configured to use the same AMQP service. Ensure that all of the rabbit_* and kombu_* configuration options in the [DEFAULT] section of /etc/heat/heat.conf match across each machine that will be running an engine. By using the same AMQP configuration, each engine will subscribe to the same AMQP engine queue and pick up work in round-robin fashion with the other engines.

One or more engines can be deployed per host. Depending on the host’s CPU architecture, it may be beneficial to deploy several engines on a single machine.

To start multiple engines on the same machine, simply start multiple heat-engine processes:

python bin/heat-engine --config-file=/etc/heat/heat.conf &
python bin/heat-engine --config-file=/etc/heat/heat.conf &

Deploying the Proxy¶

In order to simplify the deployment of the HAProxy server, we will replace the REST and CFN APIs deployed when installing devstack by the HAProxy server. This way, there is no need to update, on the CLI, the addresses where it should look for the APIs. In this case, when it makes a call to any API, it will find the proxy, acting on their behalf.

Note that the addresses that the HAProxy will be listening to are the pairs API_HOST:API-PORT and API_CFN_HOST:API_CFN_PORT, found at the [heat_api] and [heat_api_cfn] blocks on the devstack server’s configuration file. In addition, the original heat-api and heat-api-cfn processes running in these ports have to be killed, because these addresses must be free to be used by the proxy.

To deploy the HAProxy server on the devstack server, run haproxy -f apis-proxy.conf, where this configuration file looks like:

global
    daemon
    maxconn 4000

defaults
    log  global
    maxconn  8000
    option  redispatch
    retries  3
    timeout  http-request 10s
    timeout  queue 1m
    timeout  connect 10s
    timeout  client 1m
    timeout  server 1m
    timeout  check 10s

listen rest_api_proxy
    # The values required below are the original ones that were in
    # /etc/heat/heat.conf on the devstack server.
    bind {API_HOST}:{API_PORT}
    balance  source
    option  tcpka
    option  httpchk
    # The values required below are the different addresses supplied when
    # running the REST API instances.
    server SERVER_1 {HOST_1}:{PORT_1}
    ...
    server SERVER_N {HOST_N}:{PORT_N}

listen cfn_api_proxy
    # The values required below are the original ones that were in
    # /etc/heat/heat.conf on the devstack server.
    bind {API_CFN_HOST}:{API_CFN_PORT}
    balance  source
    option  tcpka
    option  httpchk
    # The values required below are the different addresses supplied when
    # running the CFN API instances.
    server SERVER_1 {HOST_1}:{PORT_1}
    ...
    server SERVER_N {HOST_N}:{PORT_N}

Sample¶

This section aims to clarify some aspects of the scaling out solution, as well as to show more details of the configuration by describing a concrete sample.

[DEFAULT]
...
sql_connection = mysql+pymysql://root:admin@127.0.0.1/heat?charset=utf8
rabbit_host = localhost
...
[heat_api]
bind_port = 8004
bind_host = 10.0.0.1
...
[heat_api_cfn]
bind_port = 8000
bind_host = 10.0.0.1

[DEFAULT]
...
sql_connection = mysql+pymysql://root:admin@10.0.0.1/heat?charset=utf8
rabbit_host = 10.0.0.1
...
[heat_api]
bind_port = 8004
bind_host = 10.0.0.2
...
[heat_api_cfn]
bind_port = 8000
bind_host = 10.0.0.2

 global
    daemon
    maxconn 4000

defaults
    log  global
    maxconn  8000
    option  redispatch
    retries  3
    timeout  http-request 10s
    timeout  queue 1m
    timeout  connect 10s
    timeout  client 1m
    timeout  server 1m
    timeout  check 10s

listen rest_api_proxy
    bind 10.0.0.1:8004
    balance  source
    option  tcpka
    option  httpchk
    server rest-server-1 10.0.0.2:8004
    server rest-server-2 10.0.0.3:8004

listen cfn_api_proxy
    bind 10.0.0.1:8000
    balance  source
    option  tcpka
    option  httpchk
    server cfn-server-1 10.0.0.2:8000
    server cfn-server-2 10.0.0.3:8000