Teaming: Broadcom NetXtreme® 57XX User Guide
Broadcom Advanced Server Program Overview
Load Balancing and Fault Tolerance
NOTE: See Broadcom Gigabit Ethernet Teaming Services for detailed information on the following topics:
- Glossary of Terms and Acronyms
- Teaming Concepts
- Software Components
- Hardware Requirements
- Supported Teaming by Processor
- Configuring Teaming by Operating System
- Supported Features by Team Type
- Selecting a Team Type
- Teaming Mechanisms
- Architecture
- Types of Teams
- Driver Support by Operating System
- Supported Teaming Speeds
- Teaming and Other Advanced Networking Features
- General Network Considerations
- Application Considerations
- Troubleshooting Teaming Problems
- Frequently-Asked Questions
- Event Log Messages
Broadcom Advanced Server Program Overview
Adapter teaming allows you to group any available network adapters together to function as a team, the benefit of which enables membership to VLANs on different subnets. You also can add virtual LANs (VLANs) to a team, which enables you to add multiple virtual adapters that are on different subnets. The benefit of adding VLANs to a team, is that each virtual adapter can belong to multiple subnets. With a VLAN, you can couple the functionality of load balancing for the load balance members, and with the capability of employing a failover adapter.
Broadcom Advanced Server Program (BASP) is the Broadcom teaming software for Windows 2000, Windows Server 2003, Windows XP, Linux, and NetWare operating systems. For Windows operating systems, BASP runs within the Broadcom Advanced Control Suite 2 (BACS2) utility. For Linux operating systems, a BASP script is used to configure a team (see Configuring Teaming); for other Linux distributions, a command line tool is used to configure a team (see Configuring Teaming for Other Linux Distributions). For NetWare operating systems, teams are configured by loading BASP with all the necessary frames for the team (see Configuring Teaming). For Linux operating systems, teaming is done with channel bonding (see Teaming with Channel Bonding).
BASP provides heterogeneous support for adapter teaming to include all Broadcom NetXtreme Gigabit Ethernet adapters.
BASP supports four types of load balancing teams:
Load Balancing and Fault Tolerance
Teaming provides traffic load balancing and fault tolerance (redundant adapter operation in the event that a network connection fails). When multiple adapters are installed in the same system, they can be grouped into teams, creating a virtual adapter.
A team can consist of one to eight network interfaces, and each interface can be designated as a primary interface or a standby interface (standby interfaces can be used only in a Smart Load Balancing™ and Failover (SLB) or SLB (Auto-Fallback Disabled) type of team, and only one standby interface can be designated per SLB team). If traffic is not identified on any of the adapter team member connections due to failure of the adapter, cable, switch port, or switch (where the teamed adapters are attached to separate switches), the load distribution is reevaluated and reassigned among the remaining team members. In the event that all of the primary adapters are down, the hot standby adapter becomes active. Existing sessions are maintained and there is no impact on the user.
Types of Teams
The available types of teams for the supported operating systems are shown in the following table:
Smart Load Balancing™ and Failover
Smart Load Balancing™ and Failover is the Broadcom implementation of load balancing based on IP flow. This feature supports balancing IP traffic across multiple adapters (team members) in a bidirectional manner. This type of team provides automatic fault detection and dynamic failover to other team member or to a hot standby member. This is done independently of layer 3 protocol (IP, IPX, NetBEUI); rather, it works with existing layer 2 and layer 3 switches. No switch configuration (such as trunk, link aggregation) is necessary for this type of team to work.
NOTES:
- If you do not enable LiveLink™ when configuring SLB teams, disabling Spanning Tree Protocol (STP) at the switch or port is recommended. This minimizes the downtime due to spanning tree loop determination when failing over. LiveLink mitigates such issues.
- IPX balances only on the transmit side of the team; other protocols are limited to the primary adapter.
- If a team member is linked at 1000 Mbit/s and another team member is linked at 100 Mbit/s, most of the traffic is handled by the 1000 Mbit/s team member.
Link Aggregation (802.3ad)
This mode supports link aggregation and conforms to the IEEE 802.3ad (LACP) specification. Configuration software allows you to dynamically configure which adapters you want to participate in a given team. If the link partner is not correctly configured for 802.3ad link configuration, errors are detected and noted. With this mode, all adapters in the team are configured to receive packets for the same MAC address. The outbound load-balancing scheme is determined by our BASP driver. The team link partner determines the load-balancing scheme for inbound packets. In this mode, at least one of the link partners must be in active mode.
Generic Trunking (FEC/GEC)/802.3ad-Draft Static
The Generic Trunking (FEC/GEC)/802.3ad-Draft Static type of team is very similar to the Link Aggregation (802.3ad) type of team in that all adapters in the team are configured to receive packets for the same MAC address. The Generic Trunking (FEC/GEC)/802.3ad-Draft Static) type of team, however, does not provide LACP or marker protocol support. This type of team supports a variety of environments in which the adapter link partners are statically configured to support a proprietary trunking mechanism. For instance, this type of team could be used to support Lucent's OpenTrunk or Cisco's Fast EtherChannel (FEC). Basically, this type of team is a light version of the Link Aggregation (802.3ad) type of team. This approach is much simpler, in that there is not a formalized link aggregation control protocol (LACP). As with the other types of teams, the creation of teams and the allocation of physical adapters to various teams is done statically through user configuration software.
The Generic Trunking (FEC/GEC/802.3ad-Draft Static) type of team supports load balancing and failover for both outbound and inbound traffic.
SLB (Auto-Fallback Disable)
The SLB (Auto-Fallback Disable) type of team is identical to the Smart Load Balance and Failover type of team, with the following exceptionwhen the standby member is active, if a primary member comes back on line, the team continues using the standby member, rather than switching back to the primary member.
If any primary adapter assigned to a team is disabled, the team functions as a Smart Load Balancing and Failover type of team in which auto-fallback occurs.
All primary interfaces in a team participate in load-balancing operations by sending and receiving a portion of the total traffic. Standby interfaces take over in the event that all primary interfaces have lost their links.
Failover teaming provides redundant adapter operation (fault tolerance) in the event that a network connection fails. If the primary adapter in a team is disconnected because of failure of the adapter, cable, or switch port, the secondary team member becomes active, redirecting both inbound and outbound traffic originally assigned to the primary adapter. Sessions will be maintained, causing no impact to the user.
Limitations of Smart Load Balance and Failover/SLB (Auto-Fallback Disable) Types of Teams
Smart Load Balancing™ (SLB) is a protocol-specific scheme. The level of support for IP, IPX, and NetBEUI protocols is listed below.
Drivers for third-party network adapters must be patched with Broadcom Network Interface Card Extension (NICE) patches for Linux (see Linux Driver Software), or be NetWare Event Service Layer (NESL) compliant for NetWare to be fault-tolerant and load-balanced in a multivendor team.
The Smart Load Balancing type of team works with all Ethernet switches without having to configure the switch ports to any special trunking mode. Only IP traffic is load-balanced in both inbound and outbound directions. IPX traffic is load-balanced in the outbound direction only. Other protocol packets are sent and received through one primary interface only. Failover for non-IP traffic is supported only for Broadcom network adapters. The Generic Trunking type of team requires the Ethernet switch to support some form of port trunking mode (for example, Cisco's Gigabit EtherChannel or other switch vendor's Link Aggregation mode). The Generic Trunking type of team is protocol-independent, and all traffic should be load-balanced and fault-tolerant.
NOTE: If you do not enable LiveLink™ when configuring teams, disabling Spanning Tree Protocol (STP) at the switch is recommended. This minimizes the downtime due to the spanning tree loop determination when failing over. LiveLink mitigates such issues.
LiveLink™ Functionality
LiveLink™ functionality is a feature of BASP that is available only for the Smart Load Balancing™ and Failover type of teaming. The purpose of LiveLink is to detect network connectivity beyond the switch and to route traffic only through team members that have a live link. This function is accomplished though the teaming software (see Configuring LiveLink for a Smart Load Balance and Failover Team). The teaming software periodically probes (issues a link packet from each team member) one or more specified target network adapter(s). The probe target(s) responds when it receives the link packet. If a team member does not detect a response within a specified amount of time after a specified number of retries, the teaming software discontinues passing traffic through that team member. Later, if that team member begins to detect a response from a probe target, this indicates that the link has been restored, and the teaming software automatically resumes passing traffic through that team member. LiveLink works only with TCP/IP.
LiveLink™ functionality is supported in both 32-bit and 64-bit Linux and Windows operating systems (except for Windows NT), but is not supported in NetWare systems. See the Channel Bonding documentation for similar functionality in Linux Channel Bonding (see http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/ref-guide/s1-modules-ethernet.html).
Teaming and Large Send Offload/Checksum Offload Support
Large Send Offload (LSO) and Checksum Offload are automatically enabled for a team only if LSO and Checksum Offload are supported and configured on all members of a team.