Download userguide_plugin_v5

Guide to the Mellanox Plugin
version 1.0.0 for Fuel
1
Contents
Guide to the Mellanox Plugin version 1.0.0 for Fuel ................................................................... 1
Revision History ......................................................................................................................... 3
Definitions, Acronyms and abbreviations .......................................................................................... 4
Mellanox Plugin.......................................................................................................................... 5
Requirements ......................................................................................................................... 5
Limitations .............................................................................................................................. 5
Installation Guide ....................................................................................................................... 6
Mellanox plugin backend configuration ................................................................................... 6
Mellanox plugin installation ..................................................................................................... 6
User Guide................................................................................................................................. 9
Known issues ............................................................................................................................10
Supported images .....................................................................................................................10
Appendix ...................................................................................................................................11
2
Revision History
Version
Revision date
Editor
Comment
1.0
15.04.2015
Nurit Vilosny
([email protected])
First revision.
1.1
22.04.2015
Nurit Vilosny
([email protected])
After first review.
1.2
10.06.2015
Lenny Verkhovsky
([email protected])
Second review.
1.3
20.07.2015
Aviram Bar Haim
([email protected])
Third review.
3
Definitions, Acronyms and abbreviations
SR-IOV
SR-IOV stands for "Single Root I/O Virtualization". It is a
specification that allows a PCI device to appear virtually in multiple
virtual machines (VMs), each of which has its own virtual function.
The specification defines virtual functions (VFs) for the VMs and a
physical function for the hypervisor. Using SR-IOV in a cloud
infrastructure helps reaching higher performance since traffic
bypasses the TCP/IP stack in the kernel
iSER
iSER stands for "iSCSI Extensions for RDMA". It is an extension of
the data transfer model of iSCSI, a storage networking standard
for TCP/IP. iSER enables the iSCSI protocol to take advantage of
the RDMA protocol suite to supply higher bandwidth for block
storage transfers (zero time copy behavior). To that fact, it
eliminates the TCP/IP processing overhead while preserving the
compatibility with iSCSI protocol.
RDMA
RDMA stands for "Remote Direct Memory Access". It is a
technology that enables to read and write data from remote server
without involving the CPU. It reduces latency and increases
throughput. In addition, the CPU is free to perform other tasks.
ConnectX-3 Pro
ConnectX-3 Pro adapter cards with Virtual Protocol Interconnect
(VPI) supporting InfiniBand and Ethernet connectivity provide the
highest performing and most flexible interconnect solution for PCI
Express Gen3 servers used in Enterprise Data Centers, HighPerformance Computing, and Embedded environments
Infiniband
A computer-networking communications standard used in highperformance computing, features very high throughput and very
low latency. It is used for data interconnect both among and within
computers. InfiniBand is also utilized as either a direct, or switched
interconnect between servers and storage systems, as well as an
interconnect between storage systems
4
Mellanox Plugin
Beginning with version 5.1, Fuel can configure Mellanox ConnectX-3 Pro network adapters to
accelerate the performance of compute and storage traffic.
This implements the following performance enhancements:
 Compute nodes use SR-IOV based networking.
 Cinder nodes use iSER block storage as the iSCSI transport rather than the default
iSCSI over TCP.
These features reduce CPU overhead, boost throughput, reduce latency, and enable network
traffic to bypass the software switch layer (e.g. Open vSwitch).
Starting with version 6.1, Mellanox plugin can deploy those features over Infiniband network as
well.
Requirements
The Mellanox ConnectX-3 Pro adapters family supports up to 40/56GbE. To reach 56 GbE
speed in your network with ConnectX-3 Pro adapters, you must use Mellanox Ethernet /
Infiniband switches (e.g. SX1036) with the additional 56GbE license. The switch ports should be
configured specifically to use 56GbE speed. No additional configuration is required on the
adapter side. For additional information about how to run in 56GbE speed, see HowTo
Configure 56GbE Link on Mellanox Adapters and Switches.
For detailed setup configuration and BOM (Bill of Material) requirements please see Fuel
Ethernet cloud details, Fuel Infiniband cloud details or Mellanox plugin for Fuel 6.1 configuration
guide.
Limitations

OVS bonding and Mellanox SR-IOV based networking over the Mellanox ConnectX-3 Pro
adapter family are not supported.

In order to use the SR-IOV feature, one should choose KVM hypervisor and “Neutron with
Vlan segmentation” in the Network settings tab
5
Installation Guide
Mellanox plugin backend configuration
To configure Mellanox backend, follow these steps:
Ethernet network:
1. Configure the required VLANs and enable flow control on the Ethernet switch ports.
All related VLANs should be enabled on the 40/56GbE switch (Private, Management,
Storage networks). On Mellanox switches, use the commands in Mirantis reference
configuration, flow to enable VLANs (e.g. VLAN 1-100 on all ports). An example of
configuring the switch for Ethernet deployment can be found in Mirantis planning guide.
Infiniband network:
1. Configure your Switch’s ports to be Infiniband.
2. Configure SM (subnet manager) in your network. Please see Openstack’s wiki page for
more details.
NOTE: without configuration of the SM, InfiniBand ports will not be available.
Mellanox plugin installation
To install Mellanox plugin, follow these steps:
1. Download the plugin rpm file into your Fuel Master node from here. For more information
on how to create a Fuel Master node, please see Mirantis Quick Start Guide.
2. Login to the Fuel Master node.
3. Run: fuel plugins --install mellanox-plugin-1.0-1.0.0-0.noarch.rpm

NOTE:
The Mellanox plugin replaces your bootstrap image, the original image is backed up
in /opt/old_bootstrap_image/
4. Verify the plugin was successfully installed by having it listed in fuel plugins
command.
6
5. You must boot your target nodes with the new bootstrap image (installed by the plugin)
after the plugin is installed.
Check your Fuel’s node status by running fuel node command.
a. If you already have nodes in “discover” status (with the original bootstrap image)
Use the reboot_bootstrap_nodes script to reboot your nodes with the new
image. For more info about using the script run reboot_bootstrap_nodes -help.
b. If fuel node command doesn’t show any nodes then you can boot your nodes
only once after the plugin is installed.
6. Create an environment – for more information please see how to create an environment.
a. Open the Settings tab of the Fuel web UI and scroll down the page. In Mellanox
OpenStack features section, select the required features.
7
b. The SR-IOV feature supports only KVM hypervisor
With Neutron with VLAN segmentation.
c. The iSER feature requires “Cinder LVM over iSCSI for volumes” enabled in the
“Storage” section.
When configuring Mellanox plugin, please mind the following:
1. You cannot install a plugin at the already existing environment. That means, the plugin
will appear in the certain environment only if the plugin was installed before creating the
environment.
2. Enabling the “Mellanox OpenStack features” section enables Mellanox hardware support
on your environment, regardless of the iSER & SR-IOV features.
3. In Ethernet cloud, when using SR-IOV & iSER, one of the virtual NICs for SR-IOV will be
reserved to the storage network.
8
4. When using SR-IOV you can set the number of virtual NICs (virtual functions) to up to 64,
if your HW and system capabilities (memory, BIOS etc) support it). In any case of SRIOV
HW limitation, the installation will try to fallback the VF number to the default of 16 VFs.
User Guide
1. Mellanox plugin supports both Ethernet and InfiniBand configuration. The plugin will
recognize the network type, and install the cloud according to it.
2. However, in order to support the Infiniband auto-detection, the default bootstrap should
be upgraded. The Mellanox plugin installation will automatically replace and upgrade the
bootstrap as part of its installation.
Please note – if any server was online before installing the plugin, please reboot it
manually, or use the reboot_bootstrap_nodes script.
3. The SR-IOV feature supports only KVM hypervisor + Neutron with VLAN segmentation.
4. The iSER feature requires “Cinder LVM over iSCSI for volumes” enabled in the “Storage”
section.
5. For more details, please follow Mellanox plugin for Fuel 6.1 configuration guide .
9
Known issues
Issue
#
Description
Workaround
1
This release supports Mellanox ConnectX®-3
family adapters only
NA
2
Ubuntu on InfiniBand is not supported on the
latest Ubuntu kernel (default Ubuntu kernel in
Fuel 6.1)
NA
3
For custom (OEM) adapter cards based on
For firmware installation instructions see
Mellanox ConnectX-3 / ConnectX-3 Pro ICs,
http://www.mellanox.com/page/oem_firmware_download
adapter firmware must be manually burnt prior
to the installation with SR-IOV support
4
The number of SR-IOV virtual functions
supported by Mellanox adapters is 16 on
ConnectX-3 adapters and 128 on ConnectX-3
Pro adapters
Deploying more than 10 nodes at a time over a
slow PXE network can cause timeouts during
the OFED installation
5
NA
Deploy chunks of up to 10 nodes or increase the delaybefore-timeout in the plugin’s tasks.yaml file on the Fuel
master node
If timeout occurs, click “deploy-changes” again
6
7
8
NA
Network verification is not supported after
deployment using this tool over and InfiniBand
network
For InfiniBand networks, it is recommended not Change recv_queue_size and send_queue_size ib_ipoib
kernel module parameters to 32 in your controllers.
to use more than 20 Neutron networks with
These parameters are configured in
DHCP in a live MOS environment
/etc/modprobe.d/ipoib.conf with a default value of 128.
Using an untagged storage network on the
same interface with a private network over
Ethernet is not supported when using iSER
Use a separate interface for untagged storage networks
for iSER over Ethernet or use a tagged storage network
instead
Supported images
Issue
#
Supported OS
Tested kernel
1
CentOS 6.5
2
Ubuntu 13.10
2.6.32-431.el6.x86_64
3.11.0-26-generic
3
Mellanox Cirros / Cirros IB images
3.2.0-60-virtual #91-Ubuntu
10
Appendix
Provide links to Mellanox site where users can read about possible configurations:
1. Mellanox ConnectX-3 pro
2. HowTo Install Mirantis Fuel OpenStack with Mellanox
3. Mellanox InfiniBand Switches
11