Background¶

In the Tricircle, XJob provides OpenStack multi-region functionality. It receives and processes jobs from the Admin API or Tricircle Central Neutron Plugin and handles them in an asynchronous way. For example, when booting an instance in the first time for the project, router, security group rule, FIP and other resources may have not already been created in the local Neutron(s), these resources could be created asynchronously to accelerate response for the initial instance booting request, different from network, subnet and security group resources that must be created before an instance booting. Central Neutron could send such creation jobs to local Neutron(s) through XJob and then local Neutron(s) handle them with their own speed.

Implementation¶

XJob server may strike occasionally so tenants and cloud administrators need to know the job status and delete or redo the failed job if necessary. Asynchronous job management APIs provide such functionality and they are listed as following:

• Create a job

  Create a job to synchronize resource if necessary.

  Create Job Request:

  POST /v1.0/jobs
  {
      "job": {
          "type": "port_delete",
          "project_id": "d01246bc5792477d9062a76332b7514a",
          "resource": {
              "pod_id": "0eb59465-5132-4f57-af01-a9e306158b86",
              "port_id": "8498b903-9e18-4265-8d62-3c12e0ce4314"
          }
      }
  }
  
  Response:
  {
      "job": {
          "id": "3f4ecf30-0213-4f1f-9cb0-0233bcedb767",
          "project_id": "d01246bc5792477d9062a76332b7514a",
          "type": "port_delete",
          "timestamp": "2017-03-03 11:05:36",
          "status": "NEW",
          "resource": {
              "pod_id": "0eb59465-5132-4f57-af01-a9e306158b86",
              "port_id": "8498b903-9e18-4265-8d62-3c12e0ce4314"
          }
      }
  }
  
  Normal Response Code: 202
  

• Get a job

  Retrieve a job from the Tricircle database.

  The detailed information of the job will be shown. Otherwise it will return “Resource not found” exception.

  List Request:

  GET /v1.0/jobs/3f4ecf30-0213-4f1f-9cb0-0233bcedb767
  
  Response:
  {
      "job": {
          "id": "3f4ecf30-0213-4f1f-9cb0-0233bcedb767",
          "project_id": "d01246bc5792477d9062a76332b7514a",
          "type": "port_delete",
          "timestamp": "2017-03-03 11:05:36",
          "status": "NEW",
          "resource": {
              "pod_id": "0eb59465-5132-4f57-af01-a9e306158b86",
              "port_id": "8498b903-9e18-4265-8d62-3c12e0ce4314"
          }
      }
  }
  
  Normal Response Code: 200
  

• Get all jobs

  Retrieve all of the jobs from the Tricircle database.

  List Request:

  GET /v1.0/jobs/detail
  
  Response:
  {
     "jobs":
         [
              {
                  "id": "3f4ecf30-0213-4f1f-9cb0-0233bcedb767",
                  "project_id": "d01246bc5792477d9062a76332b7514a",
                  "type": "port_delete",
                  "timestamp": "2017-03-03 11:05:36",
                  "status": "NEW",
                  "resource": {
                      "pod_id": "0eb59465-5132-4f57-af01-a9e306158b86",
                      "port_id": "8498b903-9e18-4265-8d62-3c12e0ce4314"
                  }
              },
              {
                  "id": "b01fe514-5211-4758-bbd1-9f32141a7ac2",
                  "project_id": "d01246bc5792477d9062a76332b7514a",
                  "type": "seg_rule_setup",
                  "timestamp": "2017-03-01 17:14:44",
                  "status": "FAIL",
                  "resource": {
                      "project_id": "d01246bc5792477d9062a76332b7514a"
                  }
              }
         ]
  }
  
  Normal Response Code: 200
  

• Get all jobs with filter(s)

  Retrieve job(s) from the Tricircle database. We can filter them by project ID, job type and job status. If no filter is provided, GET /v1.0/jobs will return all jobs.

  The response contains a list of jobs. Using filters, a subset of jobs will be returned.

  List Request:

  GET /v1.0/jobs?project_id=d01246bc5792477d9062a76332b7514a
  
  Response:
  {
     "jobs":
         [
              {
                  "id": "3f4ecf30-0213-4f1f-9cb0-0233bcedb767",
                  "project_id": "d01246bc5792477d9062a76332b7514a",
                  "type": "port_delete",
                  "timestamp": "2017-03-03 11:05:36",
                  "status": "NEW",
                  "resource": {
                      "pod_id": "0eb59465-5132-4f57-af01-a9e306158b86",
                      "port_id": "8498b903-9e18-4265-8d62-3c12e0ce4314"
                  }
              },
              {
                  "id": "b01fe514-5211-4758-bbd1-9f32141a7ac2",
                  "project_id": "d01246bc5792477d9062a76332b7514a",
                  "type": "seg_rule_setup",
                  "timestamp": "2017-03-01 17:14:44",
                  "status": "FAIL",
                  "resource": {
                      "project_id": "d01246bc5792477d9062a76332b7514a"
                  }
              }
         ]
  }
  
  Normal Response Code: 200
  

• Get all jobs’ schemas

  Retrieve all jobs’ schemas. User may want to know what the resources are needed for a specific job.

  List Request:

  GET /v1.0/jobs/schemas
  
  return all jobs' schemas.
  Response:
  {
     "schemas":
         [
              {
                  "type": "configure_route",
                  "resource": ["router_id"]
              },
              {
                  "type": "router_setup",
                  "resource": ["pod_id", "router_id", "network_id"]
              },
              {
                  "type": "port_delete",
                  "resource": ["pod_id", "port_id"]
              },
              {
                  "type": "seg_rule_setup",
                  "resource": ["project_id"]
              },
              {
                  "type": "update_network",
                  "resource": ["pod_id", "network_id"]
              },
              {
                  "type": "subnet_update",
                  "resource": ["pod_id", "subnet_id"]
              },
              {
                  "type": "shadow_port_setup",
                  "resource": [pod_id", "network_id"]
              }
         ]
  }
  
  Normal Response Code: 200
  

• Delete a job

  Delete a failed or duplicated job from the Tricircle database. A pair of curly braces will be returned if succeeds, otherwise an exception will be thrown. What’s more, we can list all jobs to verify whether it is deleted successfully or not.

  Delete Job Request:

  DELETE /v1.0/jobs/{id}
  
  Response:
  This operation does not return a response body.
  
  Normal Response Code: 200
  

• Redo a job

  Redo a halted job brought by the XJob server corruption or network failures. The job handler will redo a failed job with time interval, but this Admin API will redo a job immediately. Nothing will be returned for this request, but we can monitor its status through the execution state.

  Redo Job Request:

  PUT /v1.0/jobs/{id}
  
  Response:
  This operation does not return a response body.
  
  Normal Response Code: 200
  

Data Model Impact¶

In order to manage the jobs for each tenant, we need to filter them by project ID. So project ID is going to be added to the AsyncJob model and AsyncJobLog model.

Dependencies¶

None

Documentation Impact¶

• Add documentation for asynchronous job management API

• Add release note for asynchronous job management API

References¶

None

Background¶

The Tricircle provides unified OpenStack API gateway and networking automation functionality. Those main functionalities allow cloud operators to manage multiple OpenStack instances which are running in one site or multiple sites as a single OpenStack cloud.

Each bottom OpenStack instance which is managed by the Tricircle is also called a pod.

The Tricircle has the following components:

• Nova API-GW

• Cinder API-GW

• Neutron API Server with Neutron Tricircle plugin

• Admin API

• XJob

• DB

Nova API-GW provides the functionality to trigger automatic networking creation when new VMs are being provisioned. Neutron Tricircle plug-in is the functionality to create cross Neutron L2/L3 networking for new VMs. After the binding of tenant-id and pod finished in the Tricircle, Cinder API-GW and Nova API-GW will pass the cinder api or nova api request to appropriate bottom OpenStack instance.

Please refer to the Tricircle design blueprint[1], especially from ‘7. Stateless Architecture Proposal’ for the detail description of each components.

Problem Description¶

When a user wants to create a network in Neutron API Server, the user can specify the ‘availability_zone_hints’(AZ or az will be used for short for availability zone) during network creation[5], in the Tricircle, the ‘az_hints’ means which AZ the network should be spread into. The ‘az_hints’ meaning in Tricircle is a little different from the ‘az_hints’ meaning in Neutron[5]. If no ‘az_hints’ was specified during network creation, this created network will be spread into any AZ. If there is a list of ‘az_hints’ during the network creation, that means the network should be able to be spread into these AZs which are suggested by a list of ‘az_hints’.

When a user creates VM or Volume, there is also one parameter called availability zone. The AZ parameter is used for Volume and VM co-location, so that the Volume and VM will be created into same bottom OpenStack instance.

When a VM is being attached to a network, the Tricircle will check whether a VM’s AZ is inside in the network’s AZs scope. If a VM is not in the network’s AZs scope, the VM creation will be rejected.

Currently, the Tricircle only supports one pod in one AZ. And only supports a network associated with one AZ. That means currently a tenant’s network will be presented only in one bottom OpenStack instance, that also means all VMs connected to the network will be located at one bottom OpenStack instance. If there are more than one pod in one AZ, refer to the dynamic pod binding[6].

There are lots of use cases where a tenant needs a network being able to be spread out into multiple bottom OpenStack instances in one AZ or multiple AZs.

• Capacity expansion: tenants add VMs more and more, the capacity of one OpenStack may not be enough, then a new OpenStack instance has to be added to the cloud. But the tenant still wants to add new VMs into same network.

• Cross Neutron network service chaining. Service chaining is based on the port-pairs. Leveraging the cross Neutron L2 networking capability which is provided by the Tricircle, the chaining could also be done by across sites. For example, vRouter1 in pod1, but vRouter2 in pod2, these two VMs could be chained.

• Applications are often required to run in different availability zones to achieve high availability. Application needs to be designed as Active-Standby/Active-Active/N-Way to achieve high availability, and some components inside one application are designed to work as distributed cluster, this design typically leads to state replication or heart beat among application components (directly or via replicated database services, or via private designed message format). When this kind of applications are distributedly deployed into multiple OpenStack instances, cross Neutron L2 networking is needed to support heart beat or state replication.

• When a tenant’s VMs are provisioned in different OpenStack instances, there is E-W (East-West) traffic for these VMs, the E-W traffic should be only visible to the tenant, and isolation is needed. If the traffic goes through N-S (North-South) via tenant level VPN, overhead is too much, and the orchestration for multiple site to site VPN connection is also complicated. Therefore cross Neutron L2 networking to bridge the tenant’s routers in different Neutron servers can provide more light weight isolation.

• In hybrid cloud, there is cross L2 networking requirement between the private OpenStack and the public OpenStack. Cross Neutron L2 networking will help the VMs migration in this case and it’s not necessary to change the IP/MAC/Security Group configuration during VM migration.

The spec[5] is to explain how one AZ can support more than one pod, and how to schedule a proper pod during VM or Volume creation.

And this spec is to deal with the cross Neutron L2 networking automation in the Tricircle.

The simplest way to spread out L2 networking to multiple OpenStack instances is to use same VLAN. But there is a lot of limitations: (1) A number of VLAN segment is limited, (2) the VLAN network itself is not good to spread out multiple sites, although you can use some gateways to do the same thing.

So flexible tenant level L2 networking across multiple Neutron servers in one site or in multiple sites is needed.

Proposed Change¶

Cross Neutron L2 networking can be divided into three categories, VLAN, Shared VxLAN and Mixed VLAN/VxLAN.

• VLAN

  Network in each bottom OpenStack is VLAN type and has the same VLAN ID. If we want VLAN L2 networking to work in multi-site scenario, i.e., Multiple OpenStack instances in multiple sites, physical gateway needs to be manually configured to make one VLAN networking be extended to other sites.

  Manual setup physical gateway is out of the scope of this spec

• Shared VxLAN

  Network in each bottom OpenStack instance is VxLAN type and has the same VxLAN ID.

  Leverage L2GW[2][3] to implement this type of L2 networking.

• Mixed VLAN/VxLAN

  Network in each bottom OpenStack instance may have different types and/or have different segment IDs.

  Leverage L2GW[2][3] to implement this type of L2 networking.

There is another network type called “Local Network”. For “Local Network”, the network will be only presented in one bottom OpenStack instance. And the network won’t be presented in different bottom OpenStack instances. If a VM in another pod tries to attach to the “Local Network”, it should be failed. This use case is quite useful for the scenario in which cross Neutron L2 networking is not required, and one AZ will not include more than bottom OpenStack instance.

Cross Neutron L2 networking will be able to be established dynamically during tenant’s VM is being provisioned.

There is assumption here that only one type of L2 networking will work in one cloud deployment.

Background¶

Currently we only support VLAN as the cross-Neutron network type. For VLAN network type, central plugin in Tricircle picks a physical network and allocates a VLAN tag(or uses what users specify), then before the creation of local network, local plugin queries this provider network information and creates the network based on this information. Tricircle only guarantees that instance packets sent out of hosts in different pods belonging to the same VLAN network will be tagged with the same VLAN ID. Deployers need to carefully configure physical networks and switch ports to make sure that packets can be transported correctly between physical devices.

For more flexible deployment, VxLAN network type is a better choice. Compared to 12-bit VLAN ID, 24-bit VxLAN ID can support more numbers of bridge networks and cross-Neutron L2 networks. With MAC-in-UDP encapsulation of VxLAN network, hosts in different pods only need to be IP routable to transport instance packets.

Proposal¶

There are some challenges to support cross-Neutron VxLAN network.

1. How to keep VxLAN ID identical for the same VxLAN network across Neutron servers

2. How to synchronize tunnel endpoint information between pods

3. How to trigger L2 agents to build tunnels based on this information

4. How to support different back-ends, like ODL, L2 gateway

The first challenge can be solved as VLAN network does, we allocate VxLAN ID in central plugin and local plugin will use the same VxLAN ID to create local network. For the second challenge, we introduce a new table called “shadow_agents” in Tricircle database, so central plugin can save the tunnel endpoint information collected from one local Neutron server in this table and use it to populate the information to other local Neutron servers when needed. Here is the schema of the table:

How to collect tunnel endpoint information

When the host where a port will be located is determined, local Neutron server will receive a port-update request containing host ID in the body. During the process of this request, local plugin can query agent information that contains tunnel endpoint information from local Neutron database with host ID and port VIF type; then send tunnel endpoint information to central Neutron server by issuing a port-update request with this information in the binding profile.

How to populate tunnel endpoint information

When the tunnel endpoint information in one pod is needed to be populated to other pods, XJob will issue port-create requests to corresponding local Neutron servers with tunnel endpoint information queried from Tricircle database in the bodies. After receiving such request, local Neutron server will save tunnel endpoint information by calling real core plugin’s “create_or_update_agent” method. This method comes from neutron.db.agent_db.AgentDbMixin class. Plugins that support “agent” extension will have this method. Actually there’s no such agent daemon running in the target local Neutron server, but we insert a record for it in the database so the local Neutron server will assume there exists an agent. That’s why we call it shadow agent.

The proposed solution for the third challenge is based on the shadow agent and L2 population mechanism. In the original Neutron process, if the port status is updated to active, L2 population mechanism driver does two things. First, driver checks if the updated port is the first port in the target agent. If so, driver collects tunnel endpoint information of other ports in the same network, then sends the information to the target agent via RPC. Second, driver sends the tunnel endpoint information of the updated port to other agents where ports in the same network are located, also via RPC. L2 agents will build the tunnels based on the information they received. To trigger the above processes to build tunnels across Neutron servers, we further introduce shadow port.

Let’s say we have two instance ports, port1 is located in host1 in pod1 and port2 is located in host2 in pod2. To make L2 agent running in host1 build a tunnel to host2, we create a port with the same properties of port2 in pod1. As discussed above, local Neutron server will create shadow agent during the process of port-create request, so local Neutron server in pod1 won’t complain that host2 doesn’t exist. To trigger L2 population process, we then update the port status to active, so L2 agent in host1 will receive tunnel endpoint information of port2 and build the tunnel. Port status is a read-only property so we can’t directly update it via ReSTful API. Instead, we issue a port-update request with a special key in the binding profile. After local Neutron server receives such request, it pops the special key from the binding profile and updates the port status to active. XJob daemon will take the job to create and update shadow ports.

Here is the flow of shadow agent and shadow port process:

+-------+       +---------+                                                          +---------+
|       |       |         |     +---------+                                          |         |
| Local |       | Local   |     |         |      +----------+       +------+         | Local   |
| Nova  |       | Neutron |     | Central |      |          |       |      |         | Neutron |
| Pod1  |       | Pod1    |     | Neutron |      | Database |       | XJob |         | Pod2    |
|       |       |         |     |         |      |          |       |      |         |         |
+---+---+       +---- ----+     +----+----+      +----+-----+       +--+---+         +----+----+
    |               |                |                |                |                  |
    | update port1  |                |                |                |                  |
    | [host id]     |                |                |                |                  |
    +--------------->                |                |                |                  |
    |               | update port1   |                |                |                  |
    |               | [agent info]   |                |                |                  |
    |               +---------------->                |                |                  |
    |               |                | save shadow    |                |                  |
    |               |                | agent info     |                |                  |
    |               |                +---------------->                |                  |
    |               |                |                |                |                  |
    |               |                | trigger shadow |                |                  |
    |               |                | port setup job |                |                  |
    |               |                | for pod1       |                |                  |
    |               |                +--------------------------------->                  |
    |               |                |                |                | query ports in   |
    |               |                |                |                | the same network |
    |               |                |                |                +------------------>
    |               |                |                |                |                  |
    |               |                |                |                |     return port2 |
    |               |                |                |                <------------------+
    |               |                |                |   query shadow |                  |
    |               |                |                |     agent info |                  |
    |               |                |                |      for port2 |                  |
    |               |                |                <----------------+                  |
    |               |                |                |                |                  |
    |               |                |                |  create shadow |                  |
    |               |                |                | port for port2 |                  |
    |               <--------------------------------------------------+                  |
    |               |                |                |                |                  |
    |               | create shadow  |                |                |                  |
    |               | agent and port |                |                |                  |
    |               +-----+          |                |                |                  |
    |               |     |          |                |                |                  |
    |               |     |          |                |                |                  |
    |               <-----+          |                |                |                  |
    |               |                |                |  update shadow |                  |
    |               |                |                | port to active |                  |
    |               <--------------------------------------------------+                  |
    |               |                |                |                |                  |
    |               | L2 population  |                |                | trigger shadow   |
    |               +-----+          |                |                | port setup job   |
    |               |     |          |                |                | for pod2         |
    |               |     |          |                |                +-----+            |
    |               <-----+          |                |                |     |            |
    |               |                |                |                |     |            |
    |               |                |                |                <-----+            |
    |               |                |                |                |                  |
    |               |                |                |                |                  |
    +               +                +                +                +                  +

Bridge network can support VxLAN network in the same way, we just create shadow ports for router interface and router gateway. In the above graph, local Nova server updates port with host ID to trigger the whole process. L3 agent will update interface port and gateway port with host ID, so similar process will be triggered to create shadow ports for router interface and router gateway.

Currently Neutron team is working on push notification 1, Neutron server will send resource data to agents; agents cache this data and use it to do the real job like configuring openvswitch, updating iptables, configuring dnsmasq, etc. Agents don’t need to retrieve resource data from Neutron server via RPC any more. Based on push notification, if tunnel endpoint information is stored in port object later, and this information supports updating via ReSTful API, we can simplify the solution for challenge 3 and 4. We just need to create shadow port containing tunnel endpoint information. This information will be pushed to agents and agents use it to create necessary tunnels and flows.

How to support different back-ends besides ML2+OVS implementation

We consider two typical back-ends that can support cross-Neutron VxLAN networking, L2 gateway and SDN controller like ODL. For L2 gateway, we consider only supporting static tunnel endpoint information for L2 gateway at the first step. Shadow agent and shadow port process is almost the same with the ML2+OVS implementation. The difference is that, for L2 gateway, the tunnel IP of the shadow agent is set to the tunnel endpoint of the L2 gateway. So after L2 population, L2 agents will create tunnels to the tunnel endpoint of the L2 gateway. For SDN controller, we assume that SDN controller has the ability to manage tunnel endpoint information across Neutron servers, so Tricircle only helps to allocate VxLAN ID and keep the VxLAN ID identical across Neutron servers for one network. Shadow agent and shadow port process will not be used in this case. However, if different SDN controllers are used in different pods, it will be hard for each SDN controller to connect hosts managed by other SDN controllers since each SDN controller has its own mechanism. This problem is discussed in this page 2. One possible solution under Tricircle is as what L2 gateway does. We create shadow ports that contain L2 gateway tunnel endpoint information so SDN controller can build tunnels in its own way. We then configure L2 gateway in each pod to forward the packets between L2 gateways. L2 gateways discussed here are mostly hardware based, and can be controlled by SDN controller. SDN controller will use ML2 mechanism driver to receive the L2 network context and further control L2 gateways for the network.

To distinguish different back-ends, we will add a new configuration option cross_pod_vxlan_mode whose valid values are “p2p”, “l2gw” and “noop”. Mode “p2p” works for the ML2+OVS scenario, in this mode, shadow ports and shadow agents containing host tunnel endpoint information are created; mode “l2gw” works for the L2 gateway scenario, in this mode, shadow ports and shadow agents containing L2 gateway tunnel endpoint information are created. For the SDN controller scenario, as discussed above, if SDN controller can manage tunnel endpoint information by itself, we only need to use “noop” mode, meaning that neither shadow ports nor shadow agents will be created; or if SDN controller can manage hardware L2 gateway, we can use “l2gw” mode.

Data Model Impact¶

New table “shadow_agents” is added.

Dependencies¶

None

Documentation Impact¶

• Update configuration guide to introduce options for VxLAN network

• Update networking guide to discuss new scenarios with VxLAN network

• Add release note about cross-Neutron VxLAN networking support

References¶

1

https://blueprints.launchpad.net/neutron/+spec/push-notifications

2

https://etherealmind.com/help-wanted-stitching-a-federated-sdn-on-openstack-with-evpn/

Background¶

Most public cloud infrastructure is built with Availability Zones (AZs). Each AZ is consisted of one or more discrete data centers, each with high bandwidth and low latency network connection, separate power and facilities. These AZs offer cloud tenants the ability to operate production applications and databases deployed into multiple AZs are more highly available, fault tolerant and scalable than a single data center.

In production clouds, each AZ is built by modularized OpenStack, and each OpenStack is one pod. Moreover, one AZ can include multiple pods. Among the pods, they are classified into different categories. For example, servers in one pod are only for general purposes, and the other pods may be built for heavy load CAD modeling with GPU. So pods in one AZ could be divided into different groups. Different pod groups for different purposes, and the VM’s cost and performance are also different.

The concept “pod” is created for the Tricircle to facilitate managing OpenStack instances among AZs, which therefore is transparent to cloud tenants. The Tricircle maintains and manages a pod binding table which records the mapping relationship between a cloud tenant and pods. When the cloud tenant creates a VM or a volume, the Tricircle tries to assign a pod based on the pod binding table.

Motivation¶

In resource allocation scenario, when a tenant creates a VM in one pod and a new volume in a another pod respectively. If the tenant attempt to attach the volume to the VM, the operation will fail. In other words, the volume should be in the same pod where the VM is, otherwise the volume and VM would not be able to finish the attachment. Hence, the Tricircle needs to ensure the pod binding so as to guarantee that VM and volume are created in one pod.

In capacity expansion scenario, when resources in one pod are exhausted, then a new pod with the same type should be added into the AZ. Therefore, new resources of this type should be provisioned in the new added pod, which requires dynamical change of pod binding. The pod binding could be done dynamically by the Tricircle, or by admin through admin api for maintenance purpose. For example, for maintenance(upgrade, repairement) window, all new provision requests should be forwarded to the running one, but not the one under maintenance.

Solution: dynamic pod binding¶

It’s quite headache for capacity expansion inside one pod, you have to estimate, calculate, monitor, simulate, test, and do online grey expansion for controller nodes and network nodes whenever you add new machines to the pod. It’s quite big challenge as more and more resources added to one pod, and at last you will reach limitation of one OpenStack. If this pod’s resources exhausted or reach the limit for new resources provisioning, the Tricircle needs to bind tenant to a new pod instead of expanding the current pod unlimitedly. The Tricircle needs to select a proper pod and stay binding for a duration, in this duration VM and volume will be created for one tenant in the same pod.

For example, suppose we have two groups of pods, and each group has 3 pods, i.e.,

GroupA(Pod1, Pod2, Pod3) for general purpose VM,

GroupB(Pod4, Pod5, Pod6) for CAD modeling.

Tenant1 is bound to Pod1, Pod4 during the first phase for several months. In the first phase, we can just add weight in Pod, for example, Pod1, weight 1, Pod2, weight2, this could be done by adding one new field in pod table, or no field at all, just link them by the order created in the Tricircle. In this case, we use the pod creation time as the weight.

If the tenant wants to allocate VM/volume for general VM, Pod1 should be selected. It can be implemented with flavor or volume type metadata. For general VM/Volume, there is no special tag in flavor or volume type metadata.

If the tenant wants to allocate VM/volume for CAD modeling VM, Pod4 should be selected. For CAD modeling VM/Volume, a special tag “resource: CAD Modeling” in flavor or volume type metadata determines the binding.

When it is detected that there is no more resources in Pod1, Pod4. Based on the resource_affinity_tag, the Tricircle queries the pod table for available pods which provision a specific type of resources. The field resource_affinity is a key-value pair. The pods will be selected when there are matched key-value in flavor extra-spec or volume extra-spec. A tenant will be bound to one pod in one group of pods with same resource_affinity_tag. In this case, the Tricircle obtains Pod2 and Pod3 for general purpose, as well as Pod5 an Pod6 for CAD purpose. The Tricircle needs to change the binding, for example, tenant1 needs to be bound to Pod2, Pod5.

Implementation¶

Background¶

Currently we are using cast method in our RPC client to trigger asynchronous job in XJob daemon. After one of the worker threads receives the RPC message from the message broker, it registers the job in the database and starts to run the handle function. The registration guarantees that asynchronous job will not be lost after the job fails and the failed job can be redone. The detailed discussion of the asynchronous job process in XJob daemon is covered in our design document [1].

Though asynchronous jobs are correctly saved after worker threads get the RPC message, we still have risk to lose jobs. By using cast method, it’s only guaranteed that the message is received by the message broker, but there’s no guarantee that the message can be received by the message consumer, i.e., the RPC server thread running in XJob daemon. According to the RabbitMQ document, undelivered messages will be lost if RabbitMQ server stops [2]. Message persistence or publisher confirm can be used to increase reliability, but they sacrifice performance. On the other hand, we can not assume that message brokers other than RabbitMQ will provide similar persistence or confirmation functionality. Therefore, Tricircle itself should handle the asynchronous job reliability problem as far as possible. Since we already have a framework to register, run and redo asynchronous jobs in XJob daemon, we propose a cheaper way to improve reliability.

Proposal¶

One straightforward way to make sure that the RPC server has received the RPC message is to use call method. RPC client will be blocked until the RPC server replies the message if it uses call method to send the RPC request. So if something wrong happens before the reply, RPC client can be aware of it. Of course we cannot make RPC client wait too long, thus RPC handlers in the RPC server side need to be simple and quick to run. Thanks to the asynchronous job framework we already have, migrating from cast method to call method is easy.

Here is the flow of the current process:

+--------+     +--------+         +---------+     +---------------+   +----------+
|        |     |        |         |         |     |               |   |          |
| API    |     | RPC    |         | Message |     | RPC Server    |   | Database |
| Server |     | client |         | Broker  |     | Handle Worker |   |          |
|        |     |        |         |         |     |               |   |          |
+---+----+     +---+----+         +----+----+     +-------+-------+   +----+-----+
    |              |                   |                  |                |
    | call RPC API |                   |                  |                |
    +-------------->                   |                  |                |
    |              | send cast message |                  |                |
    |              +------------------->                  |                |
    | call return  |                   | dispatch message |                |
    <--------------+                   +------------------>                |
    |              |                   |                  | register job   |
    |              |                   |                  +---------------->
    |              |                   |                  |                |
    |              |                   |                  | obtain lock    |
    |              |                   |                  +---------------->
    |              |                   |                  |                |
    |              |                   |                  | run job        |
    |              |                   |                  +----+           |
    |              |                   |                  |    |           |
    |              |                   |                  |    |           |
    |              |                   |                  <----+           |
    |              |                   |                  |                |
    |              |                   |                  |                |
    +              +                   +                  +                +

We can just leave register job phase in the RPC handle and put obtain lock and run job phase in a separate thread, so the RPC handle is simple enough to use call method to invoke it. Here is the proposed flow:

+--------+     +--------+          +---------+     +---------------+   +----------+   +-------------+   +-------+
|        |     |        |          |         |     |               |   |          |   |             |   |       |
| API    |     | RPC    |          | Message |     | RPC Server    |   | Database |   | RPC Server  |   | Job   |
| Server |     | client |          | Broker  |     | Handle Worker |   |          |   | Loop Worker |   | Queue |
|        |     |        |          |         |     |               |   |          |   |             |   |       |
+---+----+     +---+----+          +----+----+     +-------+-------+   +----+-----+   +------+------+   +---+---+
    |              |                    |                  |                |                |              |
    | call RPC API |                    |                  |                |                |              |
    +-------------->                    |                  |                |                |              |
    |              | send call message  |                  |                |                |              |
    |              +-------------------->                  |                |                |              |
    |              |                    | dispatch message |                |                |              |
    |              |                    +------------------>                |                |              |
    |              |                    |                  | register job   |                |              |
    |              |                    |                  +---------------->                |              |
    |              |                    |                  |                |                |              |
    |              |                    |                  | job enqueue    |                |              |
    |              |                    |                  +------------------------------------------------>
    |              |                    |                  |                |                |              |
    |              |                    | reply message    |                |                | job dequeue  |
    |              |                    <------------------+                |                |-------------->
    |              | send reply message |                  |                | obtain lock    |              |
    |              <--------------------+                  |                <----------------+              |
    | call return  |                    |                  |                |                |              |
    <--------------+                    |                  |                |        run job |              |
    |              |                    |                  |                |           +----+              |
    |              |                    |                  |                |           |    |              |
    |              |                    |                  |                |           |    |              |
    |              |                    |                  |                |           +---->              |
    |              |                    |                  |                |                |              |
    |              |                    |                  |                |                |              |
    +              +                    +                  +                +                +              +

In the above graph, Loop Worker is a new-introduced thread to do the actual work. Job Queue is an eventlet queue used to coordinate Handle Worker who produces job entries and Loop Worker who consumes job entries. While accessing an empty queue, Loop Worker will be blocked until some job entries are put into the queue. Loop Worker retrieves job entries from the job queue then starts to run it. Similar to the original flow, since multiple workers may get the same type of job for the same resource at the same time, workers need to obtain the lock before it can run the job. One problem occurs whenever XJob daemon stops before it finishes all the jobs in the job queue; all unfinished jobs are lost. To solve it, we make changes to the original periodical task that is used to redo failed job, and let it also handle the jobs which have been registered for a certain time but haven’t been started. So both failed jobs and “orphan” new jobs can be picked up and redone.

You can see that Handle Worker doesn’t do many works, it just consumes RPC messages, registers jobs then puts job items in the job queue. So one extreme solution here, will be to register new jobs in the API server side and start worker threads to retrieve jobs from the database and run them. In this way, we can remove all the RPC processes and use database to coordinate. The drawback of this solution is that we don’t dispatch jobs. All the workers query jobs from the database so there is high probability that some of the workers obtain the same job and thus race occurs. In the first solution, message broker helps us to dispatch messages, and so dispatch jobs.

Considering job dispatch is important, we can make some changes to the second solution and move to the third one, that is to also register new jobs in the API server side, but we still use cast method to trigger asynchronous job in XJob daemon. Since job registration is done in the API server side, we are not afraid that the jobs will be lost if cast messages are lost. If API server side fails to register the job, it will return response of failure; If registration of job succeeds, the job will be done by XJob daemon at last. By using RPC, we dispatch jobs with the help of message brokers. One thing which makes cast method better than call method is that retrieving RPC messages and running job handles are done in the same thread so if one XJob daemon is busy handling jobs, RPC messages will not be dispatched to it. However when using call method, RPC messages are retrieved by one thread(the Handle Worker) and job handles are run by another thread(the Loop Worker), so XJob daemon may accumulate many jobs in the queue and at the same time it’s busy handling jobs. This solution has the same problem with the call method solution. If cast messages are lost, the new jobs are registered in the database but no XJob daemon is aware of these new jobs. Same way to solve it, use periodical task to pick up these “orphan” jobs. Here is the flow:

+--------+     +--------+         +---------+     +---------------+   +----------+
|        |     |        |         |         |     |               |   |          |
| API    |     | RPC    |         | Message |     | RPC Server    |   | Database |
| Server |     | client |         | Broker  |     | Handle Worker |   |          |
|        |     |        |         |         |     |               |   |          |
+---+----+     +---+----+         +----+----+     +-------+-------+   +----+-----+
    |              |                   |                  |                |
    | call RPC API |                   |                  |                |
    +-------------->                   |                  |                |
    |              | register job      |                  |                |
    |              +------------------------------------------------------->
    |              |                   |                  |                |
    |              | [if succeed to    |                  |                |
    |              |  register job]    |                  |                |
    |              | send cast message |                  |                |
    |              +------------------->                  |                |
    | call return  |                   | dispatch message |                |
    <--------------+                   +------------------>                |
    |              |                   |                  | obtain lock    |
    |              |                   |                  +---------------->
    |              |                   |                  |                |
    |              |                   |                  | run job        |
    |              |                   |                  +----+           |
    |              |                   |                  |    |           |
    |              |                   |                  |    |           |
    |              |                   |                  <----+           |
    |              |                   |                  |                |
    |              |                   |                  |                |
    +              +                   +                  +                +

Discussion¶

In this section we discuss the pros and cons of the above three solutions.

Downtime means that after a job is dispatched to a worker, other workers need to wait for a certain time to determine that job is expired and take over it.

Conclusion¶

We decide to implement the third solution(API server register jobs + uses cast) since it improves the asynchronous job reliability and at the mean time has better work load dispatch.

Data Model Impact¶

None

Dependencies¶

None

Documentation Impact¶

None

References¶

..[1] https://docs.google.com/document/d/1zcxwl8xMEpxVCqLTce2-dUOtB-ObmzJTbV1uSQ6qTsY ..[2] https://www.rabbitmq.com/tutorials/tutorial-two-python.html ..[3] https://www.rabbitmq.com/confirms.html ..[4] http://eventlet.net/doc/modules/queue.html

Background¶

To achieve cross-Neutron layer-3 networking, we utilize a bridge network to connect networks in each Neutron server, as shown below:

East-West networking:

+-----------------------+                +-----------------------+
| OpenStack1            |                | OpenStack2            |
|                       |                |                       |
| +------+  +---------+ | +------------+ | +---------+  +------+ |
| | net1 |  |      ip1| | | bridge net | | |ip2      |  | net2 | |
| |      +--+    R    +---+            +---+    R    +--+      | |
| |      |  |         | | |            | | |         |  |      | |
| +------+  +---------+ | +------------+ | +---------+  +------+ |
+-----------------------+                +-----------------------+

Fig 1

North-South networking:

+---------------------+                  +-------------------------------+
| OpenStack1          |                  | OpenStack2                    |
|                     |                  |                               |
| +------+  +-------+ | +--------------+ | +-------+  +----------------+ |
| | net1 |  |    ip1| | |  bridge net  | | |    ip2|  |  external net  | |
| |      +--+  R1   +---+              +---+  R2   +--+                | |
| |      |  |       | | | 100.0.1.0/24 | | |       |  | 163.3.124.0/24 | |
| +------+  +-------+ | +--------------+ | +-------+  +----------------+ |
+---------------------+                  +-------------------------------+

Fig 2

To support east-west networking, we configure extra routes in routers in each OpenStack cloud:

In OpenStack1, destination: net2, nexthop: ip2
In OpenStack2, destination: net1, nexthop: ip1

To support north-south networking, we set bridge network as the external network in OpenStack1 and as the internal network in OpenStack2. For instance in net1 to access the external network, the packets are SNATed twice, first SNATed to ip1, then SNATed to ip2. For floating ip binding, ip in net1 is first bound to ip(like 100.0.1.5) in bridge network(bridge network is attached to R1 as external network), then the ip(100.0.1.5) in bridge network is bound to ip (like 163.3.124.8)in the real external network (bridge network is attached to R2 as internal network).

Problems¶

The idea of introducing a bridge network is good, but there are some problems in the current usage of the bridge network.

Problems¶

There are already several scenarios fulfilled in Tricircle for north- south networking.

Scenario “North South Networking via Multiple External Networks”[1] meets the demand for multiple external networks, but local network can not reach other local networks which are not in the same OpenStack cloud.

Scenario “North South Networking via Single External Network”[2] can meet local networks east-west networking requirement, but the north-south traffic needs to go to single gateway.

In multi-region cloud deployment, a requirement is that each OpenStack cloud provides external network, north-south traffic is expected to be handled locally for shortest path, and/or use multiple external networks to ensure application north-south traffic redundancy, at the same time east-west networking of tenant’s networks between OpenStack cloud is also needed.

Proposal¶

To address the above problems, the key limitation is the pattern for router gateway, one router in Neutron can only be attached to one external network. As what’s described in the spec of combined bridge network[3], only external network is suitable for working as bridge network due to DVR challenge.

North-south traffic via the external network in the same region is conflict with external network as bridge network.

The proposal is to introduce a new networking mode for this scenario:

+-----------------------+             +----------------------+
|       ext-net1        |             |           ext-net2   |
|      +---+---+        |             |            +--+---+  |
|RegionOne |            |             | RegionTwo     |      |
|      +---+---+        |             |          +----+--+   |
|      |  R1   |        |             |          |  R2   |   |
|      +--+----+        |             |          +--+----+   |
|         | net1        |             |        net2 |        |
|     +---+--+---+-+    |             |   ++-----+--+---+    |
|            |   |      |             |    |     |           |
|  +---------+-+ |      |             |    |  +--+--------+  |
|  | Instance1 | |      |             |    |  | Instance2 |  |
|  +-----------+ |      |             |    |  +-----------+  |
|           +----+--+   | bridge-net  |  +-+-----+           |
|           | R3(1) +--------------------+ R3(2) |           |
|           +-------+   |             |  +-------+           |
+-----------------------+             +----------------------+
 Figure.1 Multiple external networks with east-west networking

R1 is the router to connect the external network ext-net1 directly in RegionOne. Net1’s default gateway is R1, so all north-south traffic will be forwarded by R1 by default. In short, north-south traffic of net2 will be processed by R2 in RegionTwo. R1 and R2 are local routers which is supposed to be presented in only one region. Region name should be specified in availability-zone-hint during router creation in central Neutron, for example:

openstack --os-region-name=CentralRegion router create --availability-zone-hint=RegionOne R1
openstack --os-region-name=CentralRegion router create --availability-zone-hint=RegionTwo R2

openstack --os-region-name=CentralRegion router add subnet R1 <net1's subnet>
openstack --os-region-name=CentralRegion router add subnet R2 <net2's subnet>

In order to process the east-west traffic from net1 to net2, R3(1) and R3(2) will be introduced, R3(1) and R3(2) will be inter-connected by bridge-net. Bridge-net could be VLAN or VxLAN cross Neutron L2 network, and it’s the “external network” for both R3(1) and R3(2), please note here the bridge-net is not real external network, just the concept of Neutron network. R3(1) and R3(2) will only forward the east-west traffic across Neutron for local networks, so it’s not necessary to work as DVR, centralized router is good enough.

In central Neutron, we only need to create a virtual logical router R3, and R3 router is called as east-west gateway, to handle the east-west traffic for local networks in different region, and it’s non-local router. Tricircle central Neutron plugin will help to create R3(1) in RegionOne and R3(2) in RegionTwo, and use the bridge network to inter-connect R3(1) and R3(2). The logical topology in central Neutron looks like follows:

     ext-net1             ext-net2
    +-------+              +--+---+
        |                     |
    +---+---+            +----+--+
    |  R1   |            |  R2   |
    +--+----+            +--+----+
       | net1          net2 |
   +---+--+---++  ++-----+--+---+
          |   |    |     |
+---------+-+ |    |  +--+--------+
| Instance1 | |    |  | Instance2 |
+-----------+ |    |  +-----------+
            +-+----+--+
            |   R3    |
            +---------+

Figure.2 Logical topology in central Neutron

Tricircle central Neutron plugin will use logical router R3 to create R3(1) in RegionOne, and R3(2) in RegionTwo.

Please note that R3(1) is not the default gateway of net1, and R3(2) is not the default gateway of net2 too. So the user has to create a port and use this port as the router interface explicitly between router and local network.

In central Neutron, the topology could be created like this:

openstack --os-region-name=CentralRegion port create --network=net1 net1-R3-interface
openstack --os-region-name=CentralRegion router add port R3 <net1-R3-interface's id>

openstack --os-region-name=CentralRegion port create --network=net2 net2-R3-interface
openstack --os-region-name=CentralRegion router add port R3 <net2-R3-interface's id>

Tricircle central Neutron plugin will automatically configure R3(1), R3(2) and bridge-network as follows:

For net1, host route should be added:

destination=net2's cidr, nexthop=<net1-R3-interface's IP>

For net2, host route should be added:

destination=net1's cidr, nexthop=<net2-R3-interface's IP>

In R3(1), extra route will be configured:

destination=net2's cidr, nexthop=R3(2)'s interface in bridge-net

In R3(2), extra route will be configured:

destination=net1's cidr, nexthop=R3(1)'s interface in bridge-net

R3(1) and R3(2) will set the external gateway to bridge-net:

router-gateway-set R3(1) bridge-net
router-gateway-set R3(2) bridge-net

Now, north-south traffic of Instance1 and Instance2 work like follows:

Instance1 -> net1 -> R1 -> ext-net1
Instance2 -> net2 -> R2 -> ext-net2

Only one hop for north-south traffic.

East-west traffic between Instance1 and Instance2 work like follows:

Instance1 <-> net1 <-> R3(1) <-> bridge-net <-> R3(2) <-> net2 <-> Instance2

Two hops for cross Neutron east-west traffic.

The topology will be more complex if there are cross Neutron L2 networks except local networks:

+-----------------------+             +----------------------+
|       ext-net1        |             |           ext-net2   |
|      +-------+        |             |            +--+---+  |
|RegionOne |            |             | RegionTwo     |      |
|      +---+----------+ |             | +-------------+--+   |
|      |    R1        | |             | |      R2        |   |
|      +--+--+---+--+-+ |             | ++-+----+---+----+   |
|    net1 |  |   |  |   |             |  | |    |   | net2   |
|     ++--++ |   |  |   |             |  | |    | +-+---+    |
|      | net3|   |  |   |             |  | |    |net4|       |
|      |  ++---+ |  |   |             |  | |  ++---+ |       |
|      |   |     |  |   |  net5       |  | |   |     |       |
|      |   |   +++-------------------------+-++|     |       |
|      |   |    |   |   |  net6       |  |   | |     |       |
|      |   |    |++-+--------------------+++ | |     |       |
|      |   |    | |     |             |   |  | |     |       |
|      |   |    | |     |             |   |  | |     |       |
|      |   |    | |     |             |   |  | |     |       |
|      |   |    | |     |             |   |  | |     |       |
| +----+---+----+-+-+   | bridge-net  |  ++--+-+-----+-----+ |
| |      R3(1)      +--------------------+      R3(2)      | |
| +-----------------+   |             |  +-----------------+ |
+-----------------------+             +----------------------+

Figure.3 Multi-NS and cross Neutron L2 networks

The logical topology in central Neutron for Figure.3 looks like as follows:

     ext-net1                                  ext-net2
    +-------+                                   +--+---+
        |                                          |
    +---+----------+                 +-------------+--+
    |    R1        |                 |      R2        |
    +--+--+---+--+-+                 ++-+----+---+----+
  net1 |  |   |  |                    | |    |   | net2
   ++--++ |   |  |                    | |    | +-+---+
    | net3|   |  |                    | |    |net4|
    |  ++---+ |  |                    | |  ++---+ |
    |   |     |  |      net5          | |   |     |
    |   |     +------+------------------+   |     |
    |   |        |   |  net6          |     |     |
    |   |        +-------------+------+     |     |
    |   |            |         |            |     |
    |   |            |         |            |     |
    |   |            |         |            |     |
    |   |            |         |            |     |
  +-+---+------------+---------+------------+-----+-+
  |                    R3                           |
  +-------------------------------------------------+
Figure.4 Logical topology in central Neutron with cross Neutron L2 network

East-west traffic inside one region will be processed locally through default gateway. For example, in RegionOne, R1 has router interfaces in net1, net3, net5, net6, the east-west traffic between these networks will work as follows:

net1 <-> R1 <-> net3
net1 <-> R1 <-> net5
net1 <-> R1 <-> net6
net3 <-> R1 <-> net5
net3 <-> R1 <-> net6
net5 <-> R1 <-> net6

There is nothing special for east-west traffic between local networks in different OpenStack regions.

Net5 and net6 are cross Neutron L2 networks, instances could be attached to network from different regions, and instances are reachable in a remote region via the cross Neutron L2 network itself. There is no need to add host route for cross Neutron L2 network, for it’s routable in the same region for other local networks or cross Neutron L2 networks, default route is enough for east-west traffic.

It’s needed to address how one cross Neutron L2 network will be attached different local router: different gateway IP address will be used. For example, in central Neutron, net5’s default gateway IP is 192.168.0.1 in R1, the user needs to create a gateway port explicitly for local router R2 and net5, for example 192.168.0.2, then net5 will be attached to R2 using this gateway port 192.168.0.2. Tricircle central Neutron plugin will make this port’s IP 192.168.0.2 as the default gateway IP for net5 in RegionTwo.

Besides of gateway ports creation for local router R2, it’s also needed to create a gateway port for R3 and net5, which is used for east-west traffic. Because R3 will be spread into RegionOne and RegionTwo, so net5 will have different gateway ports in RegionOne and RegionTwo. Tricircle central Neutron plugin needs to reserve the gateway ports in central Neutron, and create these gateway ports in RegionOne and RegionTwo for net5 on R3. Because R3 is the east-west gateway router for net5, so these gateway ports are not the default gateway port. Then host route in net5 should be updated for local networks which are not in the same region:

For net5 in RegionOne, host route should be added:

destination=net2's cidr, nexthop=<net5-R3-RegionOne-interface's IP>
destination=net4's cidr, nexthop=<net5-R3-RegionOne-interface's IP>

For net5 in RegionTwo, host route should be added:

destination=net1's cidr, nexthop=<net5-R3-RegionTwo-interface's id>
destination=net3's cidr, nexthop=<net5-R3-RegionTwo-interface's IP>

Similar operation for net6 in RegionOne and RegionTwo.

If R1 and R2 are centralized routers, cross Neutron L2 network will work, but if R1 and R2 are DVRs, then DVR MAC issue mentioned in the spec “l3-networking-combined-bridge-net” should be fixed[2].

In order to make the topology not too complex, this use case will not be supported: a cross Neutron L2 network is not able to be stretched into the region where there are local networks. This use case is not useful and will make the east-west traffic even more complex:

+-----------------------+             +----------+     +-----------------+
|       ext-net1        |             | ext-net2 |     |      ext-net4   |
|      +-------+        |             | +------+ |     |       +--+---+  |
|RegionOne |            |             | RegionTwo|     |  Region4 |      |
|      +---+----------+ |             | +------+ |     |  +-------+--+   |
|      |    R1        | |             | | R2   | |     |  |  R4      |   |
|      +--+--+---+--+-+ |             | ++-+---+ |     |  +-+---+----+   |
|    net1 |  |   |  |   |             |  | |     |     |    |   | net2   |
|     ++--++ |   |  |   |             |  | |     |     |    | +-+---+    |
|      | net3|   |  |   |             |  | |     |     |    |net4|       |
|      |  ++---+ |  |   |             |  | |     |     |  ++---+ |       |
|      |   |     |  |   |  net5       |  | |     |     |   |     |       |
|      |   |   +-+-------------------------+-+   |     |   |     |       |
|      |   |        |   |  net6       |  |       |     |   |     |       |
|      |   |      +-+--------------------+       |     |   |     |       |
|      |   |            |             |          |     |   |     |       |
|      |   |            |             |          |     |   |     |       |
|      |   |            |             |          |     |   |     |       |
|      |   |            |             |          |     |   |     |       |
| +----+---+--------+   |             |  +-----+ |     | +-+-----+-----+ |
| |      R3(1)      |   |             |  |R3(2)| |     | |   R3(3)     | |
| +-----------+-----+   |             |  +-+---+ |     | +-----+-------+ |
|             |         |             |    |     |     |       |         |
+-----------------------+             +----------+     +-----------------+
              |          bridge-net        |                   |
              +----------------------------+-------------------+

Figure.5 Cross Neutron L2 network not able to be stretched into some region

Background¶

Currently, LBaaS (Load-Balancing-as-a-Service) is not supported in the Tricircle. This spec is to describe how LBaaS will be implemented in the Tricircle. LBaaS is an advanced service of Neutron, which allows for proprietary and open-source load balancing technologies to drive the actual load balancing of requests. Based on the networking guide of Ocata release, LBaaS can be configured with an agent or Octavia. Given that the OpenStack community try to take Octavia as the reference implementation of LBaaS, we only enable LBaaS based on Octavia in the Tricircle.

Different from existing LBaaS implementation, Octavia accomplishes its delivery of load balancing services by managing a fleet of virtual machines, containers, or bare metal servers, collectively known as amphorae, which it spins up on demand. This spec file is dedicated to how to implement LBaaS in multiple regions with the Tricircle.

Overall Implementation¶

The Tricircle is designed in a central-local fashion, where all the local neutrons are managed by the central neutron. As a result, in order to adapt the central-local design and the amphorae mechanism of Octavia, we plan to deploy LBaaS as follows.

              +---------------------------+
              |                           |
              |      Central Neutron      |
              |                           |
              +---------------------------+
                     Central Region

+----------------------------+    +-----------------------------+
|     +----------------+     |    |     +----------------+      |
|     |  LBaaS Octavia |     |    |     |  LBaaS Octavia |      |
|     +----------------+     |    |     +----------------+      |
| +------+ +---------------+ |    | +-------+ +---------------+ |
| | Nova | | Local Neutron | |    | | Nova  | | Local Neutron | |
| +------+ +---------------+ |    | +-------+ +---------------+ |
+----------------------------+    +-----------------------------+
          Region One                          Region Two

As demonstrated in the figure above, for each region where a local neutron is installed, admins can optionally choose to configure and install Octavia. Typically, Octavia leverages nova installed in its region to spin up amphorae. By employing load balancing softwares (e.g. haproxy) installed in the amphorae and Virtual Router Redundancy Protocol (VRRP), a load balancer which consists of a VIP and an amphora, can balance load across members with high availability. However, under the central-local scenario, we plan to let Octavia employ the central neutron in Central Region to manage networking resources, while still employ services in its region to manage amphora. Hence, the workflow of networking resource management in Tricircle can be described as follows.

Tenant–>local neutron API–>neutron-LBaaS—>local Octavia—>central neutron

Specifically, when a tenant attempts to create a load balancer, he/she needs to send a request to the local neutron-lbaas service. The service plugin of neutron-lbaas then prepares for creating the load balancer, including creating port via local plugin, inserting the info of the port into the database, and so on. Next the service plugin triggers the creating function of the corresponding driver of Octavia, i.e., Octavia.network.drivers.neutron.AllowedAddressPairsDriver to create the amphora. During the creation, Octavia employs the central neutron to complete a series of operations, for instance, allocating VIP, plugging in VIP, updating databases. Given that the main features of managing networking resource are implemented, we hence need to adapt the mechanism of Octavia and neutron-lbaas by improving the functionalities of the local and central plugins.

Considering the Tricircle is dedicated to enabling networking automation across Neutrons, the implementation can be divided as two parts, i.e., LBaaS members in one OpenStack instance, and LBaaS members in multiple OpenStack instances.

LBaaS members in single region¶

For LBaaS in one region, after installing octavia, cloud tenants should build a management network and two security groups for amphorae manually in the central neutron. Next, tenants need to create an interface for health management. Then, tenants need to configure the newly created networking resources for octavia and let octavia employ central neutron to create resources. Finally, tenants can create load balancers, listeners, pools, and members in the local neutron. In this case, all the members of a loadbalancer are in one region, regardless of whether the members reside in the same subnet or not.

LBaaS members in multiple regions¶

Background¶

Originally the Tricircle provided unified OpenStack API gateway and networking automation functionality. But now the Tricircle narrows its scope to networking automation across Neutron servers, the functionality of OpenStack API gateway is developed in another project called Trio2o[1].

Problem Description¶

After this splitting, many tables would no longer be used, including quota, volume, aggregate and pod binding, etc. The data models, tables and APIs of them should be removed. As for the rest of the tables that are still in use in the Tricircle, they should be renamed for better understanding.

Apart from the table cleaning work and table renaming work, a new feature will be developed to remove the dependency on old table. During the period of external network creation, it will take ‘availability_zone_hints’ (AZ or az will be used for short for availability zone) as a parameter. Previously az_hints was searched in the pod binding table by az_name and tenant_id, now the pod binding table is deprecated and new search strategy is needed to fix the problem[2]. A function named find_pod_by_az will be developed to find the az_hints by az_name in the pod table. Given the az_name, if it is not empty, we first match it with region_name in the pod table. When a pod with the same region_name is found, it will be returned back. The search procedure is complete. If no pod is found with the same region_name, then we try to match it with az_name in the pod table. If multiple pods are found, then we will raise an exception. If only one pod is found, this pod will be returned back. An exception will be raised if no pod is matched at the end of the previous search procedure. However, if the az_name is empty, we will return None, a new configuration item “default_region_for_external_network” will be used.

Proposed Change¶

All tables that need to be changed can be divided into two categories, Table to be removed, Table to be renamed.

Table to be removed:

• quality_of_service_specs

• quota_classes

• quota_usages

• quotas

• reservations

• volume_type_extra_specs

• volume_type_projects

• volume_types

• aggregates

• aggregate_metadata

• instance_types

• instance_type_projects

• instance_type_extra_specs

• key_pairs

• pod_binding

Table to be renamed:

• cascaded_pod_service_configuration(new name: cached_endpoints)

• cascaded_pods(new name: pods)

• cascaded_pods_resource_routing(new name: resource_routings)

• job(new name: async_jobs)

The deprecated tables will be removed from the repository directly, and other tables containing old meanings will be renamed for better understanding.

After the deletion of pod binding table, a new feature will be developed to lookup the az in the pod table rather than the pod binding table.

Data Model Impact¶

In database, many tables are removed, other tables are renamed for better understanding.

Documentation Impact¶

After the pod binding table is removed, the explanation of the pod binding API in the doc/source/api_v1.rst will be removed as well.

Dependencies¶

None

References¶

[1] https://github.com/openstack/trio2o

[2] https://review.openstack.org/#/c/412325/

Background¶

One of the key value we would like to achieve via the Tricircle project is to provide networking automation functionality across several Neutron servers. Each OpenStack instance runs its own Nova and Neutron services but shares the same Keystone service or uses federated Keystone, which is a multi-region deployment mode. With networking automation, virtual machines or bare metals booted in different OpenStack instances can inter-communicate via layer2 or layer3 network.

Considering the cross Neutron layer2 network case, if Neutron service in each OpenStack instance allocates ip address independently, the same ip address could be assigned to virtual machines in different OpenStack instances, thus ip address conflict could occur. One straightforward solution to this problem is to divide the ip allocation pool into several parts and each OpenStack instance has one. The drawback is that since virtual machines are not distributed evenly in each OpenStack instance, we may see some OpenStack instances uses up ip addresses while other OpenStack instances still have ip addresses not allocated. What’s worse, dividing the ip allocation pool makes it impossible for us to process virtual machine migration from one OpenStack instance to another.

Thanks to Neutron’s flexible plugin framework, by writing a new plugin and configuring Neutron server to use it, developers can define what Neutron server should do after receiving a network resources operation request. So for the ip address conflict issue discussed above, we decide to run one central Neutron server with the Tricircle central Neutron plugin(abbr: “central plugin”) to manage ip allocation pool centrally.

Besides central plugin, we need a bridge to connect central and local Neutron servers since each OpenStack instance has its own local Nova and Neutron server but these two services are not aware of the central Neutron server. This bridge should validate requested network data via the central Neutron server, then create necessary network resources in the target OpenStack instance with the data retrieved from the central Neutron server.

Local Plugin¶

For connecting central and local Neutron servers, Neutron plugin is again a good place for us to build the bridge. We can write our own plugin, the Tricircle local Neutron plugin(abbr: “local plugin”) to trigger the cross Neutron networking automation in local Neutron server. During virtual machine booting, local Nova server will interact with local Neutron server to query network or create port, which will trigger local plugin to retrieve data from central Neutron server and create necessary network resources according to the data. To support different core plugins, we will introduce a new option “real_core_plugin” in the “tricircle” configuration group. During initialization, local plugin will load the plugin specified by “real_core_plugin”. Local plugin only adds logic to interact with central Neutron server, but invokes the real core plugin to finish the CRUD operations of local network resources. The following graph shows the relation between user and Nova and Neutron servers:

               +------+
               | user |
               +-+--+-+
                 |  |
     +-----------+  +----------------------+
     |  boot vm         create and query   |
     |                  network resource   |
     v                                     |
+----+-------+                             |
| local Nova |     xxxxxxxxxxxxxxx         |
+----+-------+   xxx             xxx       |
     |          xx                 xx      |
     +---+    xxx      +--------+   xxx    |
         |    x        |        |     x    |
         |    x        |        |     x    |
         v    V        |        v     x    v
+--------+---------+   |   +----+----------+----+
| local Neutron    |   |   | central Neutron    |
| +--------------+ |   |   | +----------------+ |
| | local plugin | |   |   | | central plugin | |
| +--------------+ |   |   | +----------------+ |
+------------------+   |   +--------------------+
         |             |
         +-------------+

Next using virtual machine booting procedure to elaborate how local plugin works. To begin with, user creates network and subnet via central Neutron server. Then this user passes the network id as the requested network information to local Nova server to boot a virtual machine. During parameter validation, local Nova server queries local Neutron server to ensure the passed-in network id is valid, which is a “network-get” request. In the “network-get” handle function, local plugin first checks if local Neutron already has a network with that id. If not, local plugin retrieves network and also subnet information from central Neutron server then creates network and subnet based on this information. User may pass an invalid network id by mistake, in this case, local plugin will receive a 404 response from central Neutron server, it just returns a 404 response to local Nova server.

After the network id validation passes, local Nova server continues to schedule a host so compute manager running in that host will do the left works. Compute manager creates a port in the requested network via local Neutron server, which is a “port-create” request. In the “port-create” handle function, local plugin sends the same request to central Neutron server to create a port, and uses the returned port information to create a local port. With local plugin, we ensure all ip addresses are allocated by central Neutron server.

At the end of the network setup of the virtual machine, compute manager issues a “port-update” request to local Neutron server to associate the host with the port. In the “port-update” handle function, local plugin recognizes that this request is sent from local Nova server by the request body that the request body contains host information, so it sends a “port-update” request to central Neutron server with region name in the request body. In Keystone, we register services inside one OpenStack instance as one unique region, so we can use region name to identify one OpenStack instance. After receiving the request, central Neutron server is informed that one virtual machine port is correctly setup in one OpenStack instance, so it starts the cross Neutron networking automation process, like security group rule population, tunnel setup for layer2 communication and route setup for layer3 communication, which are done by making Neutron API call to each local Neutron server.

Implementation¶

Implementation details of the local plugin is discussed in this section.

Problems¶

Based on spec for l3 networking [1], a l3 networking which enables multiple NS traffic along with EW traffic is demonstrated. However, in the aforementioned l3 networking model, the host route will be only valid after DHCP lease time expired and renewed. It may take dhcp_lease_duration for VMs in the subnet to update the host route, after a new pod with external network is added to Tricircle. To solve the problem, this spec is written to introduce a new l3 networking model.

Proposal¶

For the networking model in [1], a tenant network is attached to two routers, one for NS traffic, the other for EW traffic. In the new networking model, inspired by combined bridge network [2], we propose to attach the tenant network to one router, and the router takes charge of routing NS and EW traffic. The new networking mode is plotted in Fig. 1.

+-----------------------+             +----------------------+
|            ext-net1   |             |        ext-net2      |
|      +---+---+        |             |            +--+---+  |
|RegionOne |            |             | RegionTwo     |      |
|      +---+---+        |             |          +----+--+   |
|      |  R1   +------+ |             | +--------+  R2   |   |
|      +-------+      | |             | |        +-------+   |
|           net1      | |             | |      net2          |
|     +------+---+-+  | |             | | +-+----+------+    |
|            |   |    | |             | |   |    |           |
|  +---------+-+ |    | |             | |   | +--+--------+  |
|  | Instance1 | |    | |             | |   | | Instance2 |  |
|  +-----------+ |    | |             | |   | +-----------+  |
|           +----+--+ | |             | |  ++------+         |
|           | R3(1) +-+-----------------+--+ R3(2) |         |
|           +-------+   |  bridge net |    +-------+         |
+-----------------------+             +----------------------+

Figure 1 Multiple external networks with east-west networking

As shown in Fig. 1, R1 connects to external network (i.e., ext-net1) and ext-net1 is the default gateway of R1. Meanwhile, net1 is attached to R3 and R3’s default gateway is the bridge net. Further, interfaces of bridge net are only attached to R1 and R2 which are regarded as local routers.

In such a scenario, all traffic (no matter NS or EW traffic) flows to R3. For EW traffic, from net1 to net2, R3(1) will forwards packets to the interface of net2 in R3(2) router namespace. For NS traffic, R3 forwards packets to the interface of an available local router (i.e., R1 or R2) which attached to the real external network. As a result, bridge net is an internal net where NS and EW traffic is steered, rather than the real external network of R3.

To create such a topology, we need to create a logical (non-local) router R3 in the central Neutron. Tricircle central Neutron plugin then creates R3(1) in RegionOne and R3(2) in RegionTwo, as well as the bridge network to inter-connect R3(1) and R3(2). As such, the networking for EW traffic is ready for tenants. To enable NS traffic, real external networks are required to be attached to R3. When explicitly adding the gateway port of each external network to R3, Tricircle automatically creates a local router (e.g. R1) for external network and set the gateway to the local router. Then to connect the local router (e.g. R1) and the non-local router (R3), two interfaces of bridge-net are also created and attached to respect router. The logical topology in central Neutron is plotted in Fig. 2.

  ext-net1             ext-net2
+---+---+            +---+---+
    |                    |
+---+---+            +---+---+
|  R1   |            |  R2   |
+---+---+            +---+---+
    |                    |
+---+--------------------+---+
|         bridge-net         |
+-------------+--------------+
              |
              |
+-------------+--------------+
|            R3              |
+---+--------------------+---+
    | net1          net2 |
+---+-----+-+      +---+-+---+
          |            |
+---------+-+       +--+--------+
| Instance1 |       | Instance2 |
+-----------+       +-----------+

Figure 2 Logical topology in central Neutron

To improve the logic of building l3 networking, we introduce routed network to manage external networks in central Neutron. In central Neutron, one routed network is created as a logical external network, and real external networks are stored as segments of the external network. As such, the local routers (e.g., R1 and R2 in Fig. 2) are transparent to users. As a result, when a real external network is created, a local router is created and the external network’s gateway is set to the router. Moreover, a port of bridge-net is created and added to the local router.

The routed network is created as follows:

openstack --os-region-name=CentralRegion network create --share --provider-physical-network extern --provider-network-type vlan --provider-segment 3005 ext-net
openstack --os-region-name=CentralRegion network segment create --physical-network extern  --network-type vlan --segment 3005 --network ext-net ext-sm-net1
openstack --os-region-name=CentralRegion network segment create --physical-network extern  --network-type vlan --segment 3005 --network ext-net ext-sm-net2
openstack --os-region-name=CentralRegion subnet create --network ext-net --network-segment ext-net1 --ip-version 4 --subnet-range 203.0.113.0/24 net1-subnet-v4
openstack --os-region-name=CentralRegion subnet create --network ext-net --network-segment ext-net1 --ip-version 4 --subnet-range 203.0.114.0/24 net2--subnet-v4

The logical topology exposed to users is plotted in Fig. 3.

             ext-net (routed network)
           +---+---+
               |
               |
+--------------+-------------+
|            R3              |
+---+--------------------+---+
    | net1          net2 |
+---+-----+-+      +---+-+---+
          |            |
+---------+-+       +--+--------+
| Instance1 |       | Instance2 |
+-----------+       +-----------+

Figure 3 Logical topology exposed to users in central Neutron

For R3, net1 and net2 should be attached to R3:

openstack --os-region-name=CentralRegion router add subnet R3 <net1's subnet>
openstack --os-region-name=CentralRegion router add subnet R3 <net2's subnet>

The gateway of the ext-net, i.e., the routed network, is set to R3:

openstack --os-region-name=CentralRegion router set <ext-net> R3

However, a routed network does not have a gateway. Consequently, the command above fails for trying adding the gateway of a routed network to the router, i.e., R3. To ensure the command works, we plan to create a gateway port for the routed network before setting the gateway to a router. Actually, the port is a blank port which does not have an IP, because a routed network is a software entity of multiple segments (i.e., subnets). To make sure the gateways of real external networks can be retrieved, we manage the IPs of gateways in “tags” field of the gateway port.

This command creates a port of bridget-net and add it to R3, which is plotted in Fig. 2.

Tricircle central Neutron plugin will automatically configure R3(1), R3(2) and bridge-network as follows:

For net1 and net2, no host route is needed, so in such an l3 networking model, users are no longer required to wait for DHCP renew to update host route. All traffic is forwarded to R3 by default.

In R3(1), extra route will be configured:

destination=net2's cidr, nexthop=R3(2)'s interface in bridge-net
destination=ext-net1's cidr, nexthop=R1's interface in bridge-net

In R3(2), extra route will be configured:

destination=net1's cidr, nexthop=R3(1)'s interface in bridge-net
destination=ext-net2's cidr, nexthop=R2's interface in bridge-net

R3(1) and R3(2) will set the external gateway to bridge-net:

router-gateway-set R3(1) bridge-net
router-gateway-set R3(2) bridge-net

Now, north-south traffic of Instance1 and Instance2 work as follows:

Instance1 -> net1 -> R3(1) -> R1 -> ext-net1
Instance2 -> net2 -> R3(2) -> R2 -> ext-net2

Two hops for north-south traffic.

East-west traffic between Instance1 and Instance2 work as follows:

Instance1 <-> net1 <-> R3(1) <-> bridge-net <-> R3(2) <-> net2 <-> Instance2

Two hops for cross Neutron east-west traffic.

The topology with cross Neutron L2 networks except local networks is illustrated in Fig. 4.

+-----------------------+            +-----------------------+
|    ext-net1           |            |          ext-net2     |
|      +---+---+        |            |             +--+---+  |
|RegionOne |            |            |  RegionTwo     |      |
|      +---+------+     |            |     +----------+--+   |
|      |    R1    +---+ |            | +---+   R2        |   |
|      +----------+   | |            | |   +-------------+   |
|    net1             | |            | |              net2   |
|     ++---+          | |            | |          +-----+    |
|      | net3         | |            | |         net4|       |
|      |  ++---+      | |            | |      +--+-+ |       |
|      |   |          | |  net5      | |         |   |       |
|      |   | +-+-----------------------------+-+ |   |       |
|      |   |   |      | |  net6      | |     |   |   |       |
|      |   |   | ++-----------------------++ |   |   |       |
|      |   |   |  |   | |            | |  |  |   |   |       |
|      |   |   |  |   | |            | |  |  |   |   |       |
|      |   |   |  |   | |            | |  |  |   |   |       |
|      |   |   |  |   | |            | |  |  |   |   |       |
| +----+---+---+--+-+ | | bridge-net | | ++--+---+---+-----+ |
| |      R3(1)      +-+----------------+-+      R3(2)      | |
| +-----------------+   |            |   +-----------------+ |
+-----------------------+            +-----------------------+

Figure 4 Multi-NS and cross Neutron L2 networks

The logical topology in central Neutron for Figure. 4 is plotted in Fig. 5.

 ext-net1                           ext-net2
+---+---+                          +--+---+
    |                                 |
 +--+-----------+                 +---+------------+
 |    R1        |                 |      R2        |
 +----------+---+                 +----+-----------+
            |                          |
 +----------+--------------------------+-----------+
 |                   bridge-net                    |
 +-----------------------+-------------------------+
                         |
 +-----------------------+-------------------------+
 |                    R3                           |
 +--+----+------+-----------------+---------+----+-+
    |    |      |                 |         |    |
    |    |      |                 |         |    |
    |    |      |                 |         |    |
    |    |    +-+--------------------+      |    |
    |    |     net5               |         |    |
    |    |         +--------------+------+  |    |
    |    |            net6                  |    |
    |  +-+---+                          +---+-+  |
    |   net3                             net2    |
  +-+---+                                    +---+-+
   net1                                       net4

Figure 5 Logical topology in central Neutron with cross Neutron L2 network