SR-IOV¶

SR-IOV agent¶

The SR-IOV agent allows you to set the admin state of ports, configure port security (enable and disable spoof checking), and configure QoS rate limiting and minimum bandwidth. You must include the SR-IOV agent on each compute node using SR-IOV ports.

Supported Ethernet controllers¶

The following manufacturers are known to work:

• Intel
• Mellanox
• QLogic
• Broadcom

For information on Mellanox SR-IOV Ethernet ConnectX cards, see:

• Mellanox: How To Configure SR-IOV VFs on ConnectX-4 or newer.
• Mellanox: How To Configure SR-IOV VFs on ConnectX-3/ConnectX-3 Pro.

For information on QLogic SR-IOV Ethernet cards, see:

• User’s Guide OpenStack Deployment with SR-IOV Configuration.

For information on Broadcom NetXtreme-E Series Ethernet cards, see the Broadcom NetXtreme-C/NetXtreme-E User Guide.

For information on Broadcom NetXtreme-S Series Ethernet cards, see the Broadcom NetXtreme-S Product Page.

Create Virtual Functions (Compute)¶

Create the VFs for the network interface that will be used for SR-IOV. We use eth3 as PF, which is also used as the interface for the VLAN provider network and has access to the private networks of all machines.

1. Ensure SR-IOV and VT-d are enabled in BIOS.

2. Enable IOMMU in Linux by adding intel_iommu=on to the kernel parameters, for example, using GRUB.

3. On each compute node, create the VFs via the PCI SYS interface:

   # echo '8' > /sys/class/net/eth3/device/sriov_numvfs
   

   Note

   On some PCI devices, observe that when changing the amount of VFs you receive the error Device or resource busy. In this case, you must first set sriov_numvfs to 0, then set it to your new value.

   Note

   A network interface could be used both for PCI passthrough, using the PF, and SR-IOV, using the VFs. If the PF is used, the VF number stored in the sriov_numvfs file is lost. If the PF is attached again to the operating system, the number of VFs assigned to this interface will be zero. To keep the number of VFs always assigned to this interface, modify the interfaces configuration file adding an ifup script command.

   On Ubuntu, modify the /etc/network/interfaces file:

   auto eth3
   iface eth3 inet dhcp
   pre-up echo '4' > /sys/class/net/eth3/device/sriov_numvfs
   

   On RHEL and derivatives, modify the /sbin/ifup-local file:

   #!/bin/sh
   if [[ "$1" == "eth3" ]]
   then
       echo '4' > /sys/class/net/eth3/device/sriov_numvfs
   fi
   

   Warning

   Alternatively, you can create VFs by passing the max_vfs to the kernel module of your network interface. However, the max_vfs parameter has been deprecated, so the PCI SYS interface is the preferred method.

   You can determine the maximum number of VFs a PF can support:

   # cat /sys/class/net/eth3/device/sriov_totalvfs
   63
   

4. Verify that the VFs have been created and are in up state. For example:

   # lspci | grep Ethernet
   82:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
   82:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
   82:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   82:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   82:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   82:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   82:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   82:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   82:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   82:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
   

   # ip link show eth3
   8: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
      link/ether a0:36:9f:8f:3f:b8 brd ff:ff:ff:ff:ff:ff
      vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
   

   If the interfaces are down, set them to up before launching a guest, otherwise the instance will fail to spawn:

   # ip link set eth3 up
   

5. Persist created VFs on reboot:

   # echo "echo '7' > /sys/class/net/eth3/device/sriov_numvfs" >> /etc/rc.local
   

   Note

   The suggested way of making PCI SYS settings persistent is through the sysfsutils tool. However, this is not available by default on many major distributions.

Whitelist PCI devices nova-compute (Compute)¶

1. Configure which PCI devices the nova-compute service may use. Edit the nova.conf file:

   [pci]
   passthrough_whitelist = { "devname": "eth3", "physical_network": "physnet2"}
   

   This tells the Compute service that all VFs belonging to eth3 are allowed to be passed through to instances and belong to the provider network physnet2.

   Alternatively the [pci] passthrough_whitelist parameter also supports whitelisting by:

   • PCI address: The address uses the same syntax as in lspci and an asterisk (*) can be used to match anything.

     [pci]
     passthrough_whitelist = { "address": "[[[[<domain>]:]<bus>]:][<slot>][.[<function>]]", "physical_network": "physnet2" }
     

     For example, to match any domain, bus 0a, slot 00, and all functions:

     [pci]
     passthrough_whitelist = { "address": "*:0a:00.*", "physical_network": "physnet2" }
     

   • PCI vendor_id and product_id as displayed by the Linux utility lspci.

     [pci]
     passthrough_whitelist = { "vendor_id": "<id>", "product_id": "<id>", "physical_network": "physnet2" }
     

   If the device defined by the PCI address or devname corresponds to an SR-IOV PF, all VFs under the PF will match the entry. Multiple [pci] passthrough_whitelist entries per host are supported.

   In order to enable SR-IOV to request “trusted mode”, the [pci] passthrough_whitelist parameter also supports a trusted tag.

   Note

   This capability is only supported starting with version 18.0.0 (Rocky) release of the compute service configured to use the libvirt driver.

   Important

   There are security implications of enabling trusted ports. The trusted VFs can be set into VF promiscuous mode which will enable it to receive unmatched and multicast traffic sent to the physical function.

   For example, to allow users to request SR-IOV devices with trusted capabilities on device eth3:

   [pci]
   passthrough_whitelist = { "devname": "eth3", "physical_network": "physnet2", "trusted":"true" }
   

   The ports will have to be created with a binding profile to match the trusted tag, see Launching instances with SR-IOV ports.

2. Restart the nova-compute service for the changes to go into effect.

Configure neutron-server (Controller)¶

1. Add sriovnicswitch as mechanism driver. Edit the ml2_conf.ini file on each controller:

   [ml2]
   mechanism_drivers = openvswitch,sriovnicswitch
   

2. Ensure your physnet is configured for the chosen network type. Edit the ml2_conf.ini file on each controller:

   [ml2_type_vlan]
   network_vlan_ranges = physnet2
   

3. Add the plugin.ini file as a parameter to the neutron-server service. Edit the appropriate initialization script to configure the neutron-server service to load the plugin configuration file:

   --config-file /etc/neutron/neutron.conf
   --config-file /etc/neutron/plugin.ini
   

4. Restart the neutron-server service.

Configure nova-scheduler (Controller)¶

1. On every controller node running the nova-scheduler service, add PciPassthroughFilter to [filter_scheduler] enabled_filters to enable this filter. Ensure [filter_scheduler] available_filters is set to the default of nova.scheduler.filters.all_filters:

   [filter_scheduler]
   enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
   available_filters = nova.scheduler.filters.all_filters
   

2. Restart the nova-scheduler service.

Enable neutron-sriov-nic-agent (Compute)¶

1. Install the SR-IOV agent, if necessary.

2. Edit the sriov_agent.ini file on each compute node. For example:

   [securitygroup]
   firewall_driver = neutron.agent.firewall.NoopFirewallDriver
   
   [sriov_nic]
   physical_device_mappings = physnet2:eth3
   exclude_devices =
   

   Note

   The physical_device_mappings parameter is not limited to be a 1-1 mapping between physical networks and NICs. This enables you to map the same physical network to more than one NIC. For example, if physnet2 is connected to eth3 and eth4, then physnet2:eth3,physnet2:eth4 is a valid option.

   The exclude_devices parameter is empty, therefore, all the VFs associated with eth3 may be configured by the agent. To exclude specific VFs, add them to the exclude_devices parameter as follows:

   exclude_devices = eth1:0000:07:00.2;0000:07:00.3,eth2:0000:05:00.1;0000:05:00.2
   

3. Ensure the SR-IOV agent runs successfully:

   # neutron-sriov-nic-agent \
     --config-file /etc/neutron/neutron.conf \
     --config-file /etc/neutron/plugins/ml2/sriov_agent.ini
   

4. Enable the neutron SR-IOV agent service.

   If installing from source, you must configure a daemon file for the init system manually.

Launching instances with SR-IOV ports¶

Once configuration is complete, you can launch instances with SR-IOV ports.

1. If it does not already exist, create a network and subnet for the chosen physnet. This is the network to which SR-IOV ports will be attached. For example:

   $ openstack network create --provider-physical-network physnet2 \
       --provider-network-type vlan --provider-segment 1000 \
       sriov-net
   
   $ openstack subnet create --network sriov-net \
       --subnet-pool shared-default-subnetpool-v4 \
       sriov-subnet
   

2. Get the id of the network where you want the SR-IOV port to be created:

   $ net_id=$(openstack network show sriov-net -c id -f value)
   

3. Create the SR-IOV port. vnic-type=direct is used here, but other options include normal, direct-physical, and macvtap:

   $ openstack port create --network $net_id --vnic-type direct \
       sriov-port
   

   Alternatively, to request that the SR-IOV port accept trusted capabilities, the binding profile should be enhanced with the trusted tag.

   $ openstack port create --network $net_id --vnic-type direct \
       --binding-profile trusted=true \
       sriov-port
   

4. Get the id of the created port:

   $ port_id=$(openstack port show sriov-port -c id -f value)
   

5. Create the instance. Specify the SR-IOV port created in step two for the NIC:

   $ openstack server create --flavor m1.large --image ubuntu_18.04 \
       --nic port-id=$port_id \
       test-sriov
   

   Note

   There are two ways to attach VFs to an instance. You can create an SR-IOV port or use the pci_alias in the Compute service. For more information about using pci_alias, refer to nova-api configuration.