	  Open Fabrics Enterprise Distribution (OFED)
                          Version 1.1
		       Installation Guide
			  
			 October 2006

==============================================================================
Table of contents
==============================================================================

 1.  Overview
 2.  Contents of the OFED Distribution
 3.  HW and SW Requirements
 4.  How to download and extract the OFED
 5.  Installing OFED Software
 6.  Building OFED RPMs
 7.  IPoIB Configuration
 8.  Uninstalling OFED
 9.  Configuration
 10. Related Documentation


==============================================================================
1. Overview
==============================================================================

This is the OpenFabrics Enterprise Distribution (OFED) version 1.1
software package supporting InfiniBand fabrics. It is composed of
several software modules intended for use on a computer cluster
constructed as an InfiniBand subnet.

This document describes how to install the various modules and test them in
a Linux environment.

General Notes:
 1) The install script removes all previously installed OFED packages
    and re-installs from scratch. (Note: Configuration files will not
    be removed).  You will be prompted to acknowledge the deletion of
    the old packages.

 2) When installing OFED on an entire [homogeneous] cluster, a common
    strategy is to build the software only once (perhaps on a shared
    file system such as NFS).  The resulting RPMs can then be installed
    on all nodes in the cluster using any cluster-aware tools (such as
    pdsh).

==============================================================================
2. OFED Package Contents
==============================================================================

The OFED Distribution package generates RPMs for installing the following:

  o   OpenFabrics core and ULPs:
        - HCA drivers (mthca, ipath, ehca)
        - core
        - Upper Layer Protocols: IPoIB, SDP, SRP Initiator, iSER Initiator 
	  and uDAPL
  o   OpenFabrics utilities:
        - OpenSM: InfiniBand Subnet Manager
        - Diagnostic tools
        - Performance tests
  o   MPI:
        - OSU MPI stack supporting the InfiniBand interface
        - Open MPI stack supporting the InfiniBand interface
        - MPI benchmark tests (OSU BW/LAT, Intel MPI Benchmark, Presta)
  o   Sources of all software modules (under conditions mentioned in the 
      modules' LICENSE files)
  o   Documentation

==============================================================================
3. HW and SW Requirements
==============================================================================

1) Server platform with InfiniBand HCA (see OFED Distribution
   Release Notes for details)

2) Linux OS (see OFED Distribution Release Notes for details)

3) Administrator privileges on your machine(s)

4) Disk Space:  - For Build & Installation: 300MB
                - For Installation only:    200MB

5) For the OFED Distribution to compile on your machine, some software
   packages of your OS distribution are required. These are listed here.

OS Distribution         Required Packages
---------------         ----------------------------------
General:
o  Common to all        gcc, glib, glib-devel, glibc, glibc-devel,
                        automake, autoconf, libtool.
o  RedHat               kernel-devel, sysfsutils, sysfsutils-devel, rpm-build
o  SLES 9.0             kernel-source, udev, rpm
o  SLES 10.0            kernel-source, sysfsutils, sysfsutils-devel, rpm

Specific Component Requirements:
o  OSU MPI              a Fortran Compiler (such as gcc-g77)
o  ibutils              tcl-8.4, tcl-devel-8.4
o  oiscsi-iser-support  open-iscsi, db-devel
o  tvflash              pciutils-devel

   The installer will warn you if you attempt to compile any of the
   above packages and do not have the prerequisites installed.

==============================================================================
4. How to download and extract the OFED	Distribution
==============================================================================

1) Download the OFED-X.X.X.tgz file to your target Linux host.

   If this package is to be installed on a cluster, it is recommended to
   download it to an NFS shared directory.

2) Extract the package using:

     tar xzvf OFED-X.X.X.tgz 

==============================================================================
5. Installing OFED Software
==============================================================================

1) Go to the directory into which the package was extracted:

     cd /..../OFED-X.X.X

2) Installing the OFED package must be done as root.  For a
   menu-driven, first build and installation, run the installer
   script:

     ./install.sh

   Interactive menus will direct you through the install process.
   
   Note: After the installer completes, information about the OFED
         installation such as the prefix, kernel version, and
         installation parameters can be found by running
         /etc/infiniband/info.

   During the interactive installation of OFED, two files are
   generated: ofed.conf and ofed_net.conf. ofed.conf holds the
   installed software modules and configuration settings chosen by the
   user. ofed_net.conf holds the IPoIB settings chosen by the user.

   If the package is installed on a cluster-shared directory, these
   files can then be used to perform an automatic, unattended
   installation of OFED on other machines in the cluster. The
   unattended installation will use the same choices as were selected
   in the interactive installation.

   For an automatic installation on any host, run the following:

     ./OFED-X.X.X/install.sh -c <path>/ofed.conf -net <path>/ofed_net.conf

   Note: It is possible to rename and/or edit the ofed.conf and ofed_net.conf
         files. Thus it is possible to change user choices (observing the
         original format). See examples of ofed.conf and ofed_net.conf under
         OFED-X.X.X/docs.

Install Process Results:
------------------------

o The OFED package is installed under <prefix> directory.
o Kernel modules are copied to:
  /lib/modules/`uname -r`/kernel/drivers/infiniband/
o The package kernel include files are placed under <prefix>/src/openib.
  These includes should be used when building kernel modules which use
  the Infiniband stack. (Note that these includes, if needed, have
  been "backported" to your kernel).
o The package raw (unbackported) source files are placed under
  <prefix>/src/openib-1.1.
o The script "openibd" is installed under /etc/init.d/. This script can 
  be used to load and unload the software stack.
o The directory /etc/infiniband is created with the files "info" and
  "openib.conf". The "info" script can be used to retrieve OFED
  installation information. The "openib.conf" file contains the list of
  modules that will be loaded when the "openibd" script is used.
o The file "90-ib.rules" is installed to /etc/udev/rules.d/
o If libibverbs-utils is installed, then ofed.sh and ofed.csh are
  installed under /etc/profile.d/. These automatically update the PATH
  environment variable with <prefix>/bin.  In addition, ofed.conf is
  installed under /etc/ld.so.conf.d/ to update dynamic linker's
  run-time search path to find the InfiniBand shared libraries.
o The file /etc/modprobe.conf is updated to include the following:
  - "alias ib<n> ib_ipoib" for each ib<n> interface. 
  - "alias net-pf-27 ib_sdp" for sdp.
o If opensm is installed, the daemon opensmd is installed under /etc/init.d/
  and opensm.conf is installed under /etc.
o If IPoIB configuration files were included, ifcfg-ib<n> files will be
  installed at:
  - RedHat: /etc/sysconfig/network-scripts/ 
  - SuSE: /etc/sysconfig/network/
  

==============================================================================
6. Building OFED RPMs
==============================================================================

1) Go to the directory into which the package was extracted:

     cd /..../OFED-X.X.X

   Building RPMs can be done as a non-root user.

2) For interactive build run the build.sh script:

     ./build.sh

   Interactive menus will direct you through the build process.

   During the manual building of OFED RPMs, ofed.conf is generated.
   ofed.conf holds the selected software modules and configuration
   settings chosen by the user.

3) For an automated build, run the following:

     ./OFED-X.X.X/build.sh -c <path>/ofed.conf

   Note: It is possible to rename and/or edit the ofed.conf file. Thus
         it is possible to change user choices (observing the original format).
	 See an example of ofed.conf under OFED-X.X.X/docs.

Build Process Results
---------------------

The OFED build.sh script builds OFED binary RPMs under
OFED-X.X.X/RPMS; the sources are placed in OFED-X.X.X/SRPMS/.
Running this script does not change any currently installed
components, and the script does not change the current kernel build.

Once the build process has completed, the user may run ./install.sh to
install the new RPMs. This time, however, any previously installed
OFED components will be uninstalled and the newly built package will
be installed.

Note: Depending on your hardware, the build procedure may take 30-45
      minutes.  Installation, however, is a relatively short process
      (~5 minutes).  A common strategy for OFED installation on large
      homogeneous clusters is to extract the tarball on a network
      file system (such as NFS), do the build on NFS, and then run
      installer on each node with the RPMs that were previously built.

*** Important Note for Open MPI users ONLY: The Open MPI software
    requires that the InfiniBand drivers be installed before it is
    built.  Hence, Open MPI will only be built if you select the
    "install" option.  Open MPI will *not* be built if you only select
    the "build" option.

==============================================================================
7. IP-over-IB (IPoIB) Configuration
==============================================================================

Configuring IPoIB is an optional step during the installation.  During
interactive installation, the user may choose to insert the ifcfg-ib<n>
files.  If this option is chosen, the ifcfg-ib<n> files will be
installed at:

- RedHat: /etc/sysconfig/network-scripts/
- SuSE:   /etc/sysconfig/network/

Setting IPoIB Configuration:
----------------------------

The default IPoIB interface configuration is based on DHCP.  Note that
a special patch for DHCP servers is required for supporting IPoIB
clients.  A patch for dhcp v3.0.4 is is available under
OFED-X.X.X/docs/dhcp.

If you are not using DHCP to obtain IP addresses for clients using
IPoIB, you must manually specify the full IP configuration during the
interactive installation: IP address, network address, netmask, and
broadcast address.

For unattended installations, a configuration file can be provided
with this information.  The configuration file must specify the
following information:
- Fixed values for each IPoIB interface.
- Base IPoIB configuration on Ethernet configuration (may be useful for
  cluster configuration).

Here are some examples of ofed_net.conf:

# Static settings; all values provided by this file
IPADDR_ib0=172.16.0.4
NETMASK_ib0=255.255.0.0
NETWORK_ib0=172.16.0.0
BROADCAST_ib0=172.16.255.255
ONBOOT_ib0=1

# Based on eth0; each '*' will be replaced with corresponding octet 
# from eth0.
LAN_INTERFACE_ib0=eth0
IPADDR_ib0=172.16.'*'.'*'
NETMASK_ib0=255.255.0.0
NETWORK_ib0=172.16.0.0
BROADCAST_ib0=172.16.255.255
ONBOOT_ib0=1

# Based on the first eth<n> interface that is found (for n=0,1,...);
# each '*' will be replaced with corresponding octet from eth<n>.
LAN_INTERFACE_ib0=
IPADDR_ib0=172.16.'*'.'*'
NETMASK_ib0=255.255.0.0
NETWORK_ib0=172.16.0.0
BROADCAST_ib0=172.16.255.255
ONBOOT_ib0=1

==============================================================================
8. Uninstalling OFED
==============================================================================

There are two ways to uninstall OFED:
1) Via the installation menu.
2) Using the script uninstall.sh. The script resides under OFED-X.X.X/
   and under the installation <prefix> directory.


==============================================================================
9. Configuration
==============================================================================

Most of the OFED components can be configured or reconfigured after
the installation by modifying the relevant configuration files.  The
list of the modules that will be loaded automatically upon boot can be
found in the /etc/infiniband/openib.conf file.  Other configuration
files include:
- SDP configuration file:    <prefix>/etc/libsdp.conf 
- OpenSM configuration file: /etc/opensm.conf

See packages Release Notes for more details.

Note: After the installer completes, information about the OFED
      installation such as the prefix, kernel version, and
      installation parameters can be found by running
      /etc/infiniband/info.

==============================================================================
10. Related Documentation
==============================================================================

OFED documentation is located in the ofed-docs RPM.  After
installation the documents are located under the directory:
<prefix>/docs (the default prefix is /usr/local/ofed).

Document list:

   o README.txt
   o OFED_Installation_Guide.txt
   o MPI_README.txt
   o Examples of configuration files
   o OFED_tips.txt
   o HOWTO.build_ofed
   o All release notes

For more information, please visit the OpenFabrics web site:

   http://www.openfabrics.org/
