Networking with nova-network¶

Networking concepts¶

Compute assigns a private IP address to each VM instance. Compute makes a distinction between fixed IPs and floating IP. Fixed IPs are IP addresses that are assigned to an instance on creation and stay the same until the instance is explicitly terminated. Floating IPs are addresses that can be dynamically associated with an instance. A floating IP address can be disassociated and associated with another instance at any time. A user can reserve a floating IP for their project.

The network controller with nova-network provides virtual networks to enable compute servers to interact with each other and with the public network. Compute with nova-network supports the following network modes, which are implemented as Network Manager types:

Flat Network Manager

In this mode, a network administrator specifies a subnet. IP addresses for VM instances are assigned from the subnet, and then injected into the image on launch. Each instance receives a fixed IP address from the pool of available addresses. A system administrator must create the Linux networking bridge (typically named br100, although this is configurable) on the systems running the nova-network service. All instances of the system are attached to the same bridge, which is configured manually by the network administrator.

Flat DHCP Network Manager

In this mode, OpenStack starts a DHCP server (dnsmasq) to allocate IP addresses to VM instances from the specified subnet, in addition to manually configuring the networking bridge. IP addresses for VM instances are assigned from a subnet specified by the network administrator.

Like flat mode, all instances are attached to a single bridge on the compute node. Additionally, a DHCP server configures instances depending on single-/multi-host mode, alongside each nova-network. In this mode, Compute does a bit more configuration. It attempts to bridge into an Ethernet device (flat_interface, eth0 by default). For every instance, Compute allocates a fixed IP address and configures dnsmasq with the MAC ID and IP address for the VM. Dnsmasq does not take part in the IP address allocation process, it only hands out IPs according to the mapping done by Compute. Instances receive their fixed IPs with the dhcpdiscover command. These IPs are not assigned to any of the host's network interfaces, only to the guest-side interface for the VM.

In any setup with flat networking, the hosts providing the nova-network service are responsible for forwarding traffic from the private network. They also run and configure dnsmasq as a DHCP server listening on this bridge, usually on IP address 10.0.0.1 (see DHCP server: dnsmasq). Compute can determine the NAT entries for each network, although sometimes NAT is not used, such as when the network has been configured with all public IPs, or if a hardware router is used (which is a high availability option). In this case, hosts need to have br100 configured and physically connected to any other nodes that are hosting VMs. You must set the flat_network_bridge option or create networks with the bridge parameter in order to avoid raising an error. Compute nodes have iptables or ebtables entries created for each project and instance to protect against MAC ID or IP address spoofing and ARP poisoning.

VLAN Network Manager

This is the default mode for OpenStack Compute. In this mode, Compute creates a VLAN and bridge for each project. For multiple-machine installations, the VLAN Network Mode requires a switch that supports VLAN tagging (IEEE 802.1Q). The project gets a range of private IPs that are only accessible from inside the VLAN. In order for a user to access the instances in their project, a special VPN instance (code named cloudpipe) needs to be created. Compute generates a certificate and key for the user to access the VPN and starts the VPN automatically. It provides a private network segment for each project's instances that can be accessed through a dedicated VPN connection from the internet. In this mode, each project gets its own VLAN, Linux networking bridge, and subnet.

The subnets are specified by the network administrator, and are assigned dynamically to a project when required. A DHCP server is started for each VLAN to pass out IP addresses to VM instances from the subnet assigned to the project. All instances belonging to one project are bridged into the same VLAN for that project. OpenStack Compute creates the Linux networking bridges and VLANs when required.

These network managers can co-exist in a cloud system. However, because you cannot select the type of network for a given project, you cannot configure multiple network types in a single Compute installation.

All network managers configure the network using network drivers. For example, the Linux L3 driver (l3.py and linux_net.py), which makes use of iptables, route and other network management facilities, and the libvirt network filtering facilities. The driver is not tied to any particular network manager; all network managers use the same driver. The driver usually initializes only when the first VM lands on this host node.

All network managers operate in either single-host or multi-host mode. This choice greatly influences the network configuration. In single-host mode, a single nova-network service provides a default gateway for VMs and hosts a single DHCP server (dnsmasq). In multi-host mode, each compute node runs its own nova-network service. In both cases, all traffic between VMs and the internet flows through nova-network. Each mode has benefits and drawbacks.

All networking options require network connectivity to be already set up between OpenStack physical nodes. OpenStack does not configure any physical network interfaces. All network managers automatically create VM virtual interfaces. Some network managers can also create network bridges such as br100.

The internal network interface is used for communication with VMs. The interface should not have an IP address attached to it before OpenStack installation, it serves only as a fabric where the actual endpoints are VMs and dnsmasq. Additionally, the internal network interface must be in promiscuous mode, so that it can receive packets whose target MAC address is the guest VM, not the host.

All machines must have a public and internal network interface (controlled by these options: public_interface for the public interface, and flat_interface and vlan_interface for the internal interface with flat or VLAN managers). This guide refers to the public network as the external network and the private network as the internal or project network.

For flat and flat DHCP modes, use the nova network-create command to create a network:

$ nova network-create vmnet \
  --fixed-range-v4 10.0.0.0/16 --fixed-cidr 10.0.20.0/24 --bridge br100

This example uses the following parameters:

--fixed-range-v4

Specifies the network subnet.

--fixed-cidr

Specifies a range of fixed IP addresses to allocate, and can be a subset of the --fixed-range-v4 argument.

--bridge

Specifies the bridge device to which this network is connected on every compute node.

DHCP server: dnsmasq¶

The Compute service uses dnsmasq as the DHCP server when using either Flat DHCP Network Manager or VLAN Network Manager. For Compute to operate in IPv4/IPv6 dual-stack mode, use at least dnsmasq v2.63. The nova-network service is responsible for starting dnsmasq processes.

The behavior of dnsmasq can be customized by creating a dnsmasq configuration file. Specify the configuration file using the dnsmasq_config_file configuration option:

dnsmasq_config_file=/etc/dnsmasq-nova.conf

For more information about creating a dnsmasq configuration file, see the Configuration Options, and the dnsmasq documentation.

Dnsmasq also acts as a caching DNS server for instances. You can specify the DNS server that dnsmasq uses by setting the dns_server configuration option in /etc/nova/nova.conf. This example configures dnsmasq to use Google's public DNS server:

dns_server=8.8.8.8

Dnsmasq logs to syslog (typically /var/log/syslog or /var/log/messages, depending on Linux distribution). Logs can be useful for troubleshooting, especially in a situation where VM instances boot successfully but are not reachable over the network.

Administrators can specify the starting point IP address to reserve with the DHCP server (in the format n.n.n.n) with this command:

$ nova-manage fixed reserve --address IP_ADDRESS

This reservation only affects which IP address the VMs start at, not the fixed IP addresses that nova-network places on the bridges.

Configure Compute to use IPv6 addresses¶

If you are using OpenStack Compute with nova-network, you can put Compute into dual-stack mode, so that it uses both IPv4 and IPv6 addresses for communication. In dual-stack mode, instances can acquire their IPv6 global unicast addresses by using a stateless address auto-configuration mechanism [RFC 4862/2462]. IPv4/IPv6 dual-stack mode works with both VlanManager and FlatDHCPManager networking modes.

In VlanManager networking mode, each project uses a different 64-bit global routing prefix. In FlatDHCPManager mode, all instances use one 64-bit global routing prefix.

This configuration was tested with virtual machine images that have an IPv6 stateless address auto-configuration capability. This capability is required for any VM to run with an IPv6 address. You must use an EUI-64 address for stateless address auto-configuration. Each node that executes a nova-* service must have python-netaddr and radvd installed.

Switch into IPv4/IPv6 dual-stack mode

1. For every node running a nova-* service, install python-netaddr:

   # apt-get install python-netaddr
   

2. For every node running nova-network, install radvd and configure IPv6 networking:

   # apt-get install radvd
   # echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
   # echo 0 > /proc/sys/net/ipv6/conf/all/accept_ra
   

3. On all nodes, edit the nova.conf file and specify use_ipv6 = True.

4. Restart all nova-* services.

IPv6 configuration options

You can use the following options with the nova network-create command:

• Add a fixed range for IPv6 addresses to the nova network-create command. Specify public or private after the network-create parameter.

  $ nova network-create public --fixed-range-v4 FIXED_RANGE_V4 \
    --vlan VLAN_ID --vpn VPN_START --fixed-range-v6 FIXED_RANGE_V6
  

• Set the IPv6 global routing prefix by using the --fixed_range_v6 parameter. The default value for the parameter is fd00::/48.

  When you use FlatDHCPManager, the command uses the original --fixed_range_v6 value. For example:

  $ nova network-create public  --fixed-range-v4 10.0.2.0/24 \
    --fixed-range-v6 fd00:1::/48
  

• When you use VlanManager, the command increments the subnet ID to create subnet prefixes. Guest VMs use this prefix to generate their IPv6 global unicast addresses. For example:

  $ nova network-create public --fixed-range-v4 10.0.1.0/24 --vlan 100 \
    --vpn 1000 --fixed-range-v6 fd00:1::/48
  

Metadata service¶

Compute uses a metadata service for virtual machine instances to retrieve instance-specific data. Instances access the metadata service at http://169.254.169.254. The metadata service supports two sets of APIs: an OpenStack metadata API and an EC2-compatible API. Both APIs are versioned by date.

To retrieve a list of supported versions for the OpenStack metadata API, make a GET request to http://169.254.169.254/openstack:

$ curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
latest

To list supported versions for the EC2-compatible metadata API, make a GET request to http://169.254.169.254:

$ curl http://169.254.169.254
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
latest

If you write a consumer for one of these APIs, always attempt to access the most recent API version supported by your consumer first, then fall back to an earlier version if the most recent one is not available.

Metadata from the OpenStack API is distributed in JSON format. To retrieve the metadata, make a GET request to http://169.254.169.254/openstack/2012-08-10/meta_data.json:

$ curl http://169.254.169.254/openstack/2012-08-10/meta_data.json

{
   "uuid": "d8e02d56-2648-49a3-bf97-6be8f1204f38",
   "availability_zone": "nova",
   "hostname": "test.novalocal",
   "launch_index": 0,
   "meta": {
      "priority": "low",
      "role": "webserver"
   },
   "project_id": "f7ac731cc11f40efbc03a9f9e1d1d21f",
   "public_keys": {
       "mykey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDYVEprvtYJXVOBN0XNKV\
                 VRNCRX6BlnNbI+USLGais1sUWPwtSg7z9K9vhbYAPUZcq8c/s5S9dg5vTH\
                 bsiyPCIDOKyeHba4MUJq8Oh5b2i71/3BISpyxTBH/uZDHdslW2a+SrPDCe\
                 uMMoss9NFhBdKtDkdG9zyi0ibmCP6yMdEX8Q== Generated by Nova\n"
   },
   "name": "test"
}

Instances also retrieve user data (passed as the user_data parameter in the API call or by the --user_data flag in the openstack server create command) through the metadata service, by making a GET request to http://169.254.169.254/openstack/2012-08-10/user_data:

$ curl http://169.254.169.254/openstack/2012-08-10/user_data
#!/bin/bash
echo 'Extra user data here'

The metadata service has an API that is compatible with version 2009-04-04 of the Amazon EC2 metadata service. This means that virtual machine images designed for EC2 will work properly with OpenStack.

The EC2 API exposes a separate URL for each metadata element. Retrieve a listing of these elements by making a GET query to http://169.254.169.254/2009-04-04/meta-data/:

$ curl http://169.254.169.254/2009-04-04/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups

$ curl http://169.254.169.254/2009-04-04/meta-data/block-device-mapping/
ami

$ curl http://169.254.169.254/2009-04-04/meta-data/placement/
availability-zone

$ curl http://169.254.169.254/2009-04-04/meta-data/public-keys/
0=mykey

Instances can retrieve the public SSH key (identified by keypair name when a user requests a new instance) by making a GET request to http://169.254.169.254/2009-04-04/meta-data/public-keys/0/openssh-key:

$ curl http://169.254.169.254/2009-04-04/meta-data/public-keys/0/openssh-key
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDYVEprvtYJXVOBN0XNKVVRNCRX6BlnNbI+US\
LGais1sUWPwtSg7z9K9vhbYAPUZcq8c/s5S9dg5vTHbsiyPCIDOKyeHba4MUJq8Oh5b2i71/3B\
ISpyxTBH/uZDHdslW2a+SrPDCeuMMoss9NFhBdKtDkdG9zyi0ibmCP6yMdEX8Q== Generated\
by Nova

Instances can retrieve user data by making a GET request to http://169.254.169.254/2009-04-04/user-data:

$ curl http://169.254.169.254/2009-04-04/user-data
#!/bin/bash
echo 'Extra user data here'

The metadata service is implemented by either the nova-api service or the nova-api-metadata service. Note that the nova-api-metadata service is generally only used when running in multi-host mode, as it retrieves instance-specific metadata. If you are running the nova-api service, you must have metadata as one of the elements listed in the enabled_apis configuration option in /etc/nova/nova.conf. The default enabled_apis configuration setting includes the metadata service, so you do not need to modify it.

Hosts access the service at 169.254.169.254:80, and this is translated to metadata_host:metadata_port by an iptables rule established by the nova-network service. In multi-host mode, you can set metadata_host to 127.0.0.1.

For instances to reach the metadata service, the nova-network service must configure iptables to NAT port 80 of the 169.254.169.254 address to the IP address specified in metadata_host (this defaults to $my_ip, which is the IP address of the nova-network service) and port specified in metadata_port (which defaults to 8775) in /etc/nova/nova.conf.

The default Compute service settings assume that nova-network and nova-api are running on the same host. If this is not the case, in the /etc/nova/nova.conf file on the host running nova-network, set the metadata_host configuration option to the IP address of the host where nova-api is running.

Enable ping and SSH on VMs¶

You need to enable ping and ssh on your VMs for network access. This can be done with the openstack command.

Enable ping and SSH with openstack security group rule create commands:

$ openstack security group rule create --protocol icmp default
$ openstack security group rule create --protocol tcp --dst-port 22:22 default

If you have run these commands and still cannot ping or SSH your instances, check the number of running dnsmasq processes, there should be two. If not, kill the processes and restart the service with these commands:

# killall dnsmasq
# service nova-network restart

Configure public (floating) IP addresses¶

This section describes how to configure floating IP addresses with nova-network. For information about doing this with OpenStack Networking, see L3-routing-and-NAT.

public_interface=VLAN100

dmz_cidr=x.x.x.x/y

$ cat /proc/sys/net/ipv4/ip_forward
0

$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

# sysctl -w net.ipv4.ip_forward=1

# echo 1 > /proc/sys/net/ipv4/ip_forward

net.ipv4.ip_forward = 1

# sysctl -p

auto_assign_floating_ip=True

Remove a network from a project¶

You cannot delete a network that has been associated to a project. This section describes the procedure for dissociating it so that it can be deleted.

In order to disassociate the network, you will need the ID of the project it has been associated to. To get the project ID, you will need to be an administrator.

Disassociate the network from the project using the nova-manage project scrub command, with the project ID as the final parameter:

# nova-manage project scrub --project ID

Multiple interfaces for instances (multinic)¶

The multinic feature allows you to use more than one interface with your instances. This is useful in several scenarios:

• SSL Configurations (VIPs)
• Services failover/HA
• Bandwidth Allocation
• Administrative/Public access to your instances

Each VIP represents a separate network with its own IP block. Every network mode has its own set of changes regarding multinic usage:

$ nova network-create first-net --fixed-range-v4 20.20.0.0/24 --project-id $your-project
$ nova network-create second-net --fixed-range-v4 20.20.10.0/24 --project-id $your-project

$ openstack server list
+---------+----------+--------+-----------------------------------------+------------+
|ID       | Name     | Status | Networks                                | Image Name |
+---------+----------+--------+-----------------------------------------+------------+
| 1234... | MyServer | ACTIVE | network2=20.20.0.3; private=20.20.10.14 | cirros     |
+---------+----------+--------+-----------------------------------------+------------+

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet dhcp

$ openstack server create --image ed8b2a37-5535-4a5f-a615-443513036d71 \
  --flavor 1 --nic net-id=NETWORK1_ID --nic net-id=NETWORK2_ID test-vm1

Troubleshooting Networking¶

firewall_driver=nova.virt.firewall.NoopFirewallDriver

# modprobe vhost_net