************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2000, 2001 by Myricom, Inc.            *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************

README for GM-1.6.3

GM is a low-level communication layer for Myrinet.

************************************************************************
For updates to this software, visit `http://www.myri.com/scs'.

The FAQ file is located at `http://www.myri.com/scs/GM_FAQ.html'.

All Myrinet hardware and software technical questions should be directed
to help@myri.com.
************************************************************************

Table of Contents:

   I. Directory structure of GM distribution
  II. Installation
 III. GM Ports
  IV. Thread Safety
   V. GM Security 
  VI. GM with Different Topologies
 VII. GM Mapper and Mapper Tools
VIII. Recompiling GM MCP (optional)
  IX. Building User-Level GM programs
   X. GM Test Programs
  XI. GM Performance

=============================================
I. Directory Structure of the GM distribution
=============================================

The basic directory structure for the GM distribution is below:

gm-1.6.3
|-- doc                              GM API Documentation
|-- drivers                          GM Kernel Drivers
|   |-- aix
|   |-- freebsd
|   |-- hpux
|   |-- irix
|   |-- linux
|   |-- macosx
|   |-- nt4.0 
|   |-- nt5.0 
|   |-- solaris2.5
|   `-- vxworks
|-- examples                         GM Example Programs
|-- include                          Include directory
|-- libgm                            User-level GM API
|-- mcp                              GM Myrinet Control Program (MCP)
|   `-- libc
|-- mt                               GM Mapper
|   |-- ad_hoc_calculator
|   |-- barrel_simulation
|   |-- config
|   |-- deadlock
|   |-- dijkstra
|   |-- file_calculator
|   |-- force
|   |-- griddle
|   |-- libmt
|   |-- mapgen
|   |-- maps
|   |-- merlin
|   |-- mmapper
|   |-- nway
|   |-- ping
|   |-- simple
|   |-- smapper
|   |-- switch
|   |-- tools                        GM Mapper tools
|   `-- watch
|-- tests                            GM Test Suite
|-- util
`-- zlib

READMEs are located at various levels in this tree structure.

================
II. Installation
================

GM-1.6.3 is supported on the following operating systems and processors:

        FreeBSD v4.x for IA32 and Alpha.
        Linux 2.4 for IA32, IA64, Alpha, and PowerPC
        Linux 2.2 for IA32, Alpha, and PowerPC

An earlier release of GM is supported on the following operating
systems and processors:

        Linux 2.0 for IA32 and Alpha.
        IRIX 6.5 for Origin 200, Origin 2000, and Octane
        Mac OS X 10.1.x for PowerPC
        Solaris 6, 7, 8 for UltraSPARC (32-bit and 64-bit modes).
        Tru64 (OSF1 V4.0e, V5.0, V5.1, V5.1A) for Alpha.
        VxWorks for PowerPC Motorola MTX, Force, and CSPI boards.
        Windows NT 4.0 and Windows 2000 for IA32.
        Windows XP for IA64.

The port of gm-1.6.3 to these operating systems and processors
will be available soon.

Supported interfaces:  LANai7 and LANai9

        (If you have LANai4, you will need to upgrade your
        interface, or use a previous version of GM (gm-1.2.3 for 256K,
        and gm-1.5.2 for larger memory sizes).
        For installation instructions of an earlier GM version
        please refer to the respective README and README-<arch> files.
        Please also note that Linux 2.4 is not supported on gm-1.2.3).

Installation instructions for GM 1.6.3 (and operating-system-specific
caveats) can be found in the respective README files:

     * README-FreeBSD
     * README-linux

=============
III. GM Ports
=============

In the OS-bypass design, a port is a way for a process to communicate
directly with the Myrinet interface through GM.  Thus, the number of 
ports limits the number of processes concurrently using the Myrinet 
interface.  GM ports guarantee fairness and security between GM processes.

By default, this software supports 8 ports, and only 5 of these
(ports 2,4,5,6,7) are available to non-privileged user-level software.
Port 0 is for internal GM use and is never available.
Port 1 is used by the GM Mapper when running.
Port 3 is used by the IP-over-Myrinet driver, and the IP device
must be setup with
        ifconfig myri0 <ip_addr> up

To check the status of the GM ports:
   cd gm-1.5/binary/bin
   ./gm_board_info

If you need more ports than this, please contact help@myri.com.  
A 16-port configuration can be built for you (depending upon the type
of the interface and the amount of embedded memory).

Note that these software ports are independent from the hardware ports 
on the Myrinet switch.

=================
IV. Thread Safety
=================

GM is not thread-safe but can be used successfully in a multi-threaded
application with a few constraints.  Basically, each call to GM should
be protected by a mutex.  As the host overhead of GM is particularly
small, the impact on performance of protecting the GM calls is
usually negligible.  Special provisions need to be made for the blocking
receive case.  Refer to the FAQ for further details.
    http://www.myri.com/scs/faq/faq-gm.html#gmq4

==============
V. GM Security
==============

By default, the security features of GM are enabled.  In particular,
opening privileged ports (like the mapper port) is restricted 
to privileged users.

Since GM_ENABLE_SECURITY is set in the driver source code, GM performs checks
to ensure that the Unix security model is not violated.  For example, only
root would be able to open the mapper port and configure the network.  With
GM_ENABLE_SECURITY set, GM is believed to maintain the integrity of Unix 
system security.

=================================
VI.  GM with Different Topologies
=================================

     * Loopback Configuration
     * Point-to-Point Configuration
     * Switch Configuration

GM will provide a communication layer in a Myrinet interface in these
three topologies: (1) connected to a loopback cable, (2) connected via
a cable to another installed interface (point-to-point), or (3)
connected via a cable to a switch. When two machines are directly
connected by a cable it is NOT possible to send a message from a gm
port on one host to itself or to another gm port on the same host.  
In short, it is not possible for a message leaving the network interface
to return to this network interface without a switch in between.
Specifically,
	gm_allsize -v 
will FAIL to work in this configuration.  However, MPICH-over-GM will
work in this configuration if the shared memory support is enabled as 
described in the MPICH-GM documentation, as will IP-over-GM since both 
of these protocols loop back messages above the GM messaging layer.

The loopback and point-to-point configurations require that
gm_simpleroute be run before GM applications can be run.
Refer to the FAQ for further details.
    http://www.myri.com/scs/GM_FAQ.html
The switch configuration requires the mapper to be run before
GM applications can be run.

===============================
VII. GM Mapper and Mapper Tools
===============================

You will find the GM Mapper source in gm/mt.  Details about
running the GM Mapper can be found in the respective README-<arch>
files.  

For the advanced user, further details about the GM Mapper 
and its tools can be found in mt/README.

To build the mapper tools (simulator etc):
	[in a source distribution]
	cd gm/mt
	make all gm


========================================
VIII.  Recompiling the GM MCP (optional)
========================================

The GM distribution contains prebuilt GM Myrinet Control Programs 
(firmware) for all supported LANai versions.

The user doesn't need to compile the firmware.  The default prebuilts
cover the large majority of configurations.  However, you may need
to recompile the MCP if you change the number of ports (GM_NUM_PORTS),
or if you use unsupported configure options that would force a
rebuild.  Myricom does not provide support if the firmware has been
modified.

Building the firmware requires lanai-tools, lanai-gcc and gendat.
These tools are available at http://www.myri.com/scs/L3.  It is
also necessary to configure with --enable-maintainer-mode.


===================================
IX. Building User-Level GM Programs
===================================

Building GM programs is straightforward.  Using the "gm.h" header file
and the associated library, one can build GM applications just as with 
any other library.  The only difference is that before the program can
be run, the GM driver and any dynamically linked GM library must be 
installed, and the GM Mapper run.

Example programs which call GM can be found in {GM_HOME}/examples/.

Architecture-specific details follow.

Unix
----

Under Unix, the header file is called "gm.h" and the library is "libgm.a."
Unix dynamically linked libraries are not yet supported (with the exception 
of Solaris and Irix shared libs).

Windows 2000
------------

Under Windows 2000, the header file is "gm.h," the import library is
"gm.lib," and the dynamically linked library is "gm.dll."


===================
X. GM Test Programs
===================

If the GM driver has been installed and the cabling is correct,
there will be a green LED lit up on the switch for each connection 
that is active.  If not, check the power supply to the switch,
and check that the Myrinet cable is firmly attached at both ends.
On the host there should be a green LED and blinking yellow LED
on each Myrinet interface.

You can then run the five recommended tests, as described in the 
README.tests file in the binary/bin directory.

==================
XI. GM Performance
==================

As stated in the document "Guide to Myrinet/PCI Host Interfaces",
the Myrinet-2000 interfaces are capable of sustained PCI data rates 
approaching the limits of the PCI bus (528 MB/s for 64-bit, 66MHz;
264 MB/s for 64-bit, 33HMz or 32-bit, 66MHz; 132 MB/s for 32-bit,
33MHz).  However, the data rate to/from system memory will depend
upon the host's memory and PCI-bus implementation.  The achievable
performance can vary greatly from one PCI chipset to another.

A host-to-host communication is limited by the weakest component
in the data path.  If the performance of the PCI bus does not
exceed the data rate of the Myrinet link, it will be a bottleneck
and thus limit the global data rate of the communication.  The
maximum performance of GM should be very close to the maximum
performance of the PCI bus if the latter is a limitation.

From the GM Test Suite, you can run `gm_debug -L' to determine the
maximum sustained bandwidth from the PCI bus, and determine if the PCI
bus is the bottleneck to your performance.  You can also obtain
information from the documentation of your motherboard, or run an
OS-specific command to determine the type of PCI chipset on the
motherboard.

For performance comparison, we refer you to the following webpage,
which collects performance data of several motherboards and PCI
chipsets,
  http://www.conservativecomputers.com/myrinet/perf.html
and encourage you to submit the specifications of your machine
(motherboard) with the result of the previous benchmark.