	     Open Fabrics Enterprise Distribution (OFED)
		    IPoIB in OFED 1.1 Release Notes
			  
			   October 2006


===============================================================================
Table of Contents
===============================================================================
1. Overview
2. New Features
3. Known Issues
4. DHCP Support of IPoIB
5. High Availability (HA) Service

===============================================================================
1. Overview
===============================================================================
IPoIB is a network driver implementation that enables transmission of IP and
ARP protocol packets over an InfiniBand UD channel. The implementation conforms
to the relevant IETF working group's RFCs (http://www.ietf.org).


===============================================================================
2. New Features
===============================================================================
IPoIB supports increasing the verbosity of debug messages through a module
parameter. The parameter value can be controlled at load time or runtime.
At load time this can be done by inserting the following line in
/etc/modprobe.conf:
  options ib_ipoib debug_level=<integer number>

At runtime the value can be controlled by writing the following to sysfs:
  echo <integer number> > /sys/module/ib_ipoib/parameters/debug_level

The value can also be inspected by running:
  cat /sys/module/ib_ipoib/parameters/debug_level

Note that on some older kernels (like the one supplied with Redhat AS 4.0),
the path for inspecting the debug level is different and is as follows:
  /sys/module/ib_ipoib/debug_level 

===============================================================================
3. Known Issues
===============================================================================
1. If a host has multiple interfaces, each belonging to a different IP subnet,
   yet they use the same InfiniBand switch, the host may build an incorrect ARP
   table. This may lead to problems which seem like violations of the IP rule
   requiring different broadcast domains -- a rule not observed in this
   implementation of IPoIB.

2. On Fedora Core 4, SuSE 10 and SLES 10:
   a.   There are IPoIB alias lines in modprobe.conf which prevent stopping/
        unloading the stack (i.e., '/etc/init.d/openibd stop' will fail). 
	These alias lines cause the drivers to be loaded again by udev scripts.

	Workaround: Change modprobe.conf to set
	OPENIB_PARAMS="--without-modprobe" before running install.sh, 
	or remove the alias lines from modprobe.conf.
   
   b.   The ib1 interface uses the configuration script of ib0.

        Workaround: Invoke ifup/ifdown using both the interface name and the
		configuration script name (example: ifup ib1 ib1).

3. On RedHat EL 4 up2 the driver may not load properly if SELINUX is enforced.

   Workaround: Change the value of the parameter SELINUX in
   /etc/sysconfig/selinux from "enforcing" to "permissive" or "disabled".

4. Since the IPoIB configuration files (ifcfg-ib<n>) are installed at the
   standard networking scripts location (RedHat:/etc/sysconfig/network-scripts/
   and SuSE: /etc/sysconfig/network/), the option IPOIB_LOAD=no in openib.conf
   does not prevent the loading of IPoIB on boot.

5. On RedHat EL 4 up4, ipoib multicast group membership does not work
   due to missing code in the kernel which was available in u3 and removed
   in u4.


===============================================================================
4. DHCP Support of IPoIB
===============================================================================
IPoIB is configured by default to use information obtained dynamically from a
DHCP server, at driver startup time, to configure its interfaces.

Note: To use DHCP the user must apply a special patch (see "DHCP Notes" below).

DHCP Supported Operating Systems
--------------------------------
1. SLES 10
2. SuSE 10
3. Any kernel from 2.6.14 (tested with kernel 2.6.16.18)

DHCP Unsupported Operating Systems
----------------------------------
No RedHat EL distributions are supported.


DHCP Notes
----------
1. It may be required to run over different UDP ports than the well known ports
   (67 and 68). Free port numbers greater than 0x8000 must be chosen. To
   specify a server or client port number, use the option -p <port number>.
   The client's port number must be the chosen server's port number plus one.

2. For IPoIB to use DHCP, it is required to patch ISC's DHCP. The patch file can
   be found under OFED-1.1/docs/dhcp after extracting the distribution file
   (after installation it can also be found under <prefix>/docs/dhcp). The patch
   should be applied for the server and for each client. Tests were run on
   version 3.0.4 of the DHCP package.


===============================================================================
5. High Availability (HA) Servicey
===============================================================================
High Availability (HA) service for IPoIB interfaces is provided via the
ipoibtools package. Ipoibtools currently includes a perl script, ipoib_ha.pl,
and two executables, arpingib and mcasthandle.

The HA service operates as follows: A user-level daemon runs in background to
detect failure of the primary IPoIB interface.  If such a failure is detected
(e.g., port down), the daemon configures the secondary IPoIB interface with the
configuration parameters of the primary IPoIB interface (so that the secondary
interface assumes the IP identity of the primary interface).

Enabling the HA Service
-----------------------
To enable HA service automatically (upon bootup of the driver),
perform the following steps:

1. Edit file '/etc/infiniband/openib.conf' as follows:

		IPOIBHA_ENABLE=yes
		PRIMARY_IPOIB_DEV=ib0
		SECONDARY_IPOIB_DEV=ib1

2. Run '/etc/init.d/openibd restart' to restart the driver.

The HA service may also be activated manually, via the following command:

   ipoib_ha.pl -p <primary IPoIB interface> -s <secondary IPoIB interface> \
               --with-arping --with-multicast [-v]

    -p                  primary IPoIB interface (default: ib0)
    -s                  secondary IPoIB interface (default: ib1)
    --with-arping       use modified arping utility to send an unsolicited
                        ARP REPLY
    --with-multicast    support applications that are using multicast
    -v                  verbose output

