PreviousNextIndex

Chapter 3: Planning Cluster Network Connectivity


This chapter describes planning the network support for an HACMP cluster. This chapter contains the following sections:

  • Prerequisites
  • Overview
  • General Network Considerations for HACMP
  • Heartbeating in HACMP
  • Heartbeating over IP Aliases
  • Heartbeating over Disk
  • Designing the Network Topology
  • Planning for IP Address Takeover via IP Aliases
  • Planning for IP Address Takeover via IP Replacement
  • Planning for Other Network Conditions
  • Planning for SP Networks
  • Completing the Network Worksheets
  • Defining Hardware Addresses
  • Adding the Network Topology to the Cluster Diagram.
  • Prerequisites

    In Chapter 2: Initial Cluster Planning, you began planning your cluster, identifying the number of nodes and the key applications you want to make highly available. You started drawing the cluster diagram. This diagram is the starting point for the planning you will do in this chapter.

    Also, by now you should have decided whether or not you will use IP address takeover (IPAT) to maintain specific service IP addresses.

    Overview

    This chapter explains how to plan networking support for the cluster. Your primary goal is to use redundancy to design a cluster topology that eliminates network components as potential single points of failure. The following table lists these network components with solutions:

    Cluster Network Object
    Eliminated as a Single Point of Failure by...
    Network
    Using multiple networks to connect nodes
    TCP/IP subsystem
    Using a non-IP network to back up TCP/IP
    Network Interface Card (NIC)
    Using redundant NICs on each network

    In this chapter you complete the following planning tasks:

  • Designing the cluster network topology, that is, the combination of IP and non-IP (point-to-point) networks that connect your cluster nodes and the number of connections each node has to each network
  • Note: To avoid cluster partitioning, we highly recommend configuring redundant networks in the HACMP cluster and using both IP and non-IP networks. Out of these networks, some networks will be used for heartbeating purposes. For information on planning heartbeating networks, see Heartbeating in HACMP in this chapter.
  • Determining whether service IP labels will be made highly available with IP address takeover (IPAT) via IP Aliases or IPAT via IP Replacement with or without Alternate Hardware Address Takeover
  • Completing the network and network interface planning worksheets
  • Adding networking to your cluster diagram.
  • This chapter also includes detailed information about setting up IPAT via IP Replacement with Hardware Address Takeover (HWAT) via alternate hardware addresses.

    General Network Considerations for HACMP

    This section provides general guidelines for the type of networking required for successful HACMP operation.

    Note: If you are planning networks for a cluster with sites, see the documentation for the corresponding disaster recovery solution (HACMP/XD for Metro Mirror, HACMP/XD for GLVM or HACMP/XD for HAGEO).

    The following topics are described in this section:

  • Supported Network Types
  • IP Aliases
  • Network Connections
  • ARP Cache Updating
  • IP Labels
  • Cluster Partitioning
  • General Network Connection Example
  • HACMP Configuration in Switched Networks
  • HACMP and Virtual Ethernet (VLAN)
  • Topology Services and Heartbeat Rings.
  • Supported Network Types

    HACMP allows inter-node communication with the following TCP/IP-based networks:

  • Ethernet
  • Token Ring
  • Fiber Distributed Data Interchange (FDDI)
  • ATM and ATM LAN Emulation
  • SP Switch1 and SP Switch2
  • Etherchannel
  • The following networks are not supported:

  • Virtual IP Address (VIPA) facility of AIX 5L
  • Serial Optical Channel Converter (SOCC)
  • SLIP
  • FC Switch (FCS)
  • 802_ether
  • IP V6.
  • IP Aliases

    An IP alias is an IP label/address that is configured onto a NIC in addition to the normally-configured IP label/address on the NIC. The use of IP aliases is an AIX 5L function that HACMP supports. AIX 5L supports multiple IP aliases on a NIC. Each IP alias on a NIC can be on a separate subnet. AIX 5L also allows IP aliases with different subnet masks to be configured for an interface; HACMP does not yet support this function.

    IP aliases are used in HACMP both as service and non-service addresses for IP address takeover (IPAT), in addition to configuring networks for heartbeating.

    For information about subnet requirements for service and non-service IP labels/addresses managed by HACMP, see Planning Service IP Labels in Resource Groups in Chapter 6: Planning Resource Groups in this chapter.

    Network Connections

    HACMP requires that each node in the cluster has at least one direct, non-routed network connection with every other node. The software uses these network connections to pass heartbeat messages among the cluster nodes to determine the state of all cluster nodes, networks and network interfaces.

    HACMP requires all of the communication interfaces for a given cluster network be defined on the same physical network and route packets (including broadcast packets) to each other. They must also be able to receive responses from each other without interference by any network equipment.

    Between cluster nodes, do not place intelligent switches, routers, or other network equipment that do not transparently pass through UDP broadcasts and other packets to all cluster nodes. This prohibition includes equipment that optimizes protocols, such as:

  • Proxy ARP and MAC address caching
  • Transforming multicast and broadcast protocol requests into unicast requests
  • ICMP optimizations
  • Routing protocol manipulation
  • Manipulating or optimizing any other Internet protocol.
  • If such equipment is placed in the paths between cluster nodes and clients, use a $PING_CLIENT_LIST variable (in clinfo.rc) to help inform clients of IP address movements. The specific network topology may require other solutions.

    Bridges, hubs, and other passive devices that do not modify the packet flow may be safely placed between cluster nodes, and between nodes and clients.

    ARP Cache Updating

    During manufacturing, every NIC is given a unique hardware address, the Media Access Control (MAC) address. The MAC address is the address used by the network drivers to send packets between NICs on the local network. The MAC address is not an IP address.

    Most TCP/IP systems maintain a list of recently used IP addresses and the corresponding MAC addresses. This list is called an Address Resolution Protocol (ARP) cache.

    HACMP may be configured such that, if a NIC fails, the IP label on that NIC moves to another NIC. IP labels may also be moved to a NIC on another node if the original node fails. When an IP label moves, its ARP cache entry is then inaccurate. After a cluster event, HACMP nodes and network devices that support promiscuous listen automatically update their ARP caches. Clients and network appliances that do not support promiscuous listen continue to have incorrect entries. You can manage these addressing updates in one of two ways:

  • Use alternate hardware addresses. This function of HACMP assigns an alternate MAC address to NICs with IP labels that may move. When the IP label moves, HACMP moves the MAC address as well. In this case, the ARP cache remains correct.
  • Update the ARP cache. Through use of PING_CLIENT_LIST entries in clinfo.rc, clinfo can update the ARP caches for clients and network devices such as routers.
  • For more information about these strategies, see the chapter on installing HACMP on client nodes in the Installation Guide.

    IP Labels

    In a non-HACMP environment, a hostname typically identifies a system, with the hostname also being the IP label of one of the network interfaces in the system. Thus, a system can be reached by using its hostname as the IP label for a connection.

    Hostnames and Node Names

    Typically, the hostname can be the same as the node name. If you use the Standard configuration path in HACMP, HACMP retrieves the hostname from a node when you enter an IP address, an associated IP label, or a fully-qualified domain name (FQDN) as a communication path when adding a node to the cluster, and uses the hostname as the node name, unless you specify otherwise. In the Extended configuration path, you always specify the node name.

    In some cases, a node name should be different from the hostname (for example, for an application that requires the interface through which it connects to have an IP label that corresponds to the system hostname).

    In a case where an application requires that the AIX 5L “hostname attribute” move with an application to another node at fallover, ensure that HACMP changes the node hostname to correspond to the service IP label when the resource group that contains this application falls over to another node. Use pre- and post-event scripts to change the hostname. For information about event scripts, see Chapter 6: Configuring Cluster Events in the Administration Guide.

    IP Labels in TCP/IP Networks

    For TCP/IP networks, an IP label and its associated IP address must appear in the /etc/hosts file.

    The name of the service IP label/address must be unique within the cluster and distinct from the volume group and resource group names; it should relate to the application it serves, as well as to any corresponding device, such as websphere_service_address.

    When you assign a service IP label to an interface, use a naming convention that helps identify the interface’s role in the cluster. The related entries in the /etc/hosts file would be similar to the following:

    100.100.50.1		net1_en0 
    100.100.60.1		net2_en1 
    

    You configure the NIC by following the instructions in the relevant AIX 5L documentation. AIX 5L assigns an interface name to the NIC when it is configured. The interface name is made up of two or three characters that indicate the type of NIC, followed by a number that AIX 5L assigns in sequence for each adapter of a certain type. For example, AIX 5L assigns an interface name such as en0 for the first Ethernet NIC it configures, en1 for the second, and so on.

    Cluster Partitioning

    Partitioning, also called node isolation, occurs when a network or NIC failure isolates cluster nodes from each other. When an HACMP node stops receiving network traffic from another node, it assumes that the other node has failed. Depending on your HACMP configuration, it may begin acquiring disks from the “failed” node, and making that node’s applications and IP labels available. If the “failed” node is actually still up, data corruption may occur when the disks are taken from it. If the network becomes available again, HACMP stops one of the nodes to prevent further disk contention and duplicate IP addresses on the network.

    You can configure additional networks to help prevent cluster partitioning by:

  • Adding point-to-point networks to your network topology.
  • For information about using point-to-point networks in an HACMP cluster, see Planning Point-to-Point Networks in this chapter.
  • Tuning parameters for the HACMP network module on each node.
  • HACMP heartbeats that flow over IP links are sent as UDP datagrams. Therefore, if the network is congested, or a node is congested, the IP subsystem can silently discard the heartbeats.
    For information about setting tuning parameters, see the section Monitoring Clusters in this chapter.

    General Network Connection Example

    An example of correct HACMP networking consists of two separate Ethernet networks, each with two network interfaces on each node. Two routers connect the networks, and route packets between the cluster and clients, but not between the two networks. A clinfo.rc file is installed on each node in the cluster, containing the IP addresses of several client machines.

    HACMP Configuration in Switched Networks

    Unexpected network interface failure events can occur in HACMP configurations using switched networks, if the networks and the switches are incorrectly defined or configured.

    Follow these guidelines when configuring switched networks:

  • VLANs. If VLANs are used, all interfaces defined to HACMP on a given network must be configured on the same VLAN (one network per VLAN). These interfaces are really the only ones “known” to HACMP, the other interfaces are only known to RSCT Topology Services.
  • Topology Services uses other interfaces to determine interface failure, if the failing interface is the last one. So, if multiple interfaces are defined for HACMP on each node, Topology Services can determine interface failure reliably, without requiring interfaces not configured for HACMP.
  • Autonegotiation settings. Some Ethernet NICs are capable of automatically negotiating their speed and other characteristics, such as half or full duplex. Configure NIC not to use autonegotiate, but to run at the desired speed and duplex value. Set the switch port to which the NIC is connected to the same fixed speed and duplex value.
  • ARP settings. Some network switches have settings that can affect ARP responses, either by delaying them, or by providing proxy ARP responses that may not be correct. Either of these behaviors can cause unexpected network events. To remedy this situation, change the settings causing the problem, if possible, to ensure that the switch provides a timely response to ARP requests.
  • For many brands of switches, this means turning off the following: the spanning tree algorithm, portfast, uplinkfast, and backbonefast. If it is necessary to have spanning tree turned on, then portfast should also be turned on.

    HACMP and Virtual Ethernet (VLAN)

    HACMP 5.4 supports Virtual Ethernet (from now on referred to as VLAN, Virtual Local Area Network) with the applicable APARs installed. The following restrictions apply to using VLAN in a cluster configuration:

  • In general, we recommend using IPAT via IP Aliasing for all HACMP networks that can support VLANs. IPAT via Replacement and Hardware Address Takeover is not supported on a VLAN network.
  • If you are using a VLAN network, the PCI Hot Plug utility in HACMP is not applicable. This is because the PCI Hot Plug facility uses the VIO (Virtual I/O) Server and the I/O adapters used are virtual (and not physical network interface cards).
  • Note: If you are using VLANs, you may need to use distribution preferences for service IP labels aliases. For more information about the types of distribution preferences for service IP labels, see the section Types of Distribution for Service IP Label Aliases in this chapter.

    The following list contains additional configuration requirements for HACMP with Virtual Ethernet:

  • If the VIO server has multiple physical interfaces defined on the same network, or if there are two or more HACMP nodes using VIO servers in the same frame, HACMP will not be informed of (and therefore will not react to) single physical interface failures. This does not limit the availability of the entire cluster because VIO server itself routes traffic around the failure. The VIO server support is analogous to EtherChannel in this regard. Use methods that are not based on the VIO server to provide notification of individual physical interface failures.
  • If the VIO server has only a single physical interface on a network, then HACMP detects a failure of that physical interface. However, the failure will isolate the node from the network. Although some of these considerations may be viewed as configuration restrictions, many are direct consequences of I/O Virtualization.
  • Troubleshooting VLANs

    To troubleshoot Virtual LAN (from now on referred to as VLAN, Virtual Local Area Network) interfaces defined to HACMP and to detect an interface failure, consider these interfaces as interfaces defined on single adapter networks. For information on single adapter networks and the use of the netmon.cf file, see the section Identifying Service Adapter Failure for Two-Node Clusters in this chapter.

    In particular, list the network interfaces that belong to a VLAN in the etc/cluster/ping_client_list or in the PING_CLIENT_LIST variable in the /usr/es/sbin/cluster/etc/clinfo.rc script and run clinfo. This way, whenever a cluster event occurs, clinfo monitors and detects a failure of the listed network interfaces. Due to the nature of VLAN, other mechanisms to detect the failure of network interfaces are not effective.

    Heartbeating in HACMP

    The primary task of HACMP is to recognize and respond to failures. HACMP uses heartbeating to monitor the activity of its network interfaces, devices and IP labels.

    Heartbeating connections between cluster nodes are necessary because they enable HACMP to recognize the difference between a network failure and a node failure. For instance, if connectivity on the HACMP network (this network’s IP labels are used in a resource group) is lost, and you have another TCP/IP based network and a non-IP network configured between the nodes, HACMP recognizes the failure of its cluster network and takes recovery actions that prevent the cluster from becoming partitioned.

    To avoid cluster partitioning, we highly recommend configuring redundant networks in the HACMP cluster and using both IP and non-IP networks. Out of these networks, some networks will be used for heartbeating purposes.

    In general, heartbeats in HACMP can be sent over:

  • TCP/IP networks.
  • Serial (non-IP) networks (RS232, TMSCSI, TMSSA and disk heartbeating).
  • The Topology Services component of RSCT carries out the heartbeating function in HACMP.

    The following sections describe heartbeating mechanisms in HACMP:

  • Topology Services and Heartbeat Rings
  • Heartbeating over IP Aliases
  • Heartbeating over Disk.
  • Topology Services and Heartbeat Rings

    HACMP uses the Topology Services component of RSCT for monitoring networks and network interfaces. Topology Services organizes all the interfaces in the topology into different heartbeat rings. The current version of RSCT Topology services has a limit of 48 heartbeat rings, which is usually sufficient to monitor networks and network interfaces.

    Heartbeat rings are dynamically created and used internally by RSCT. They do not have a direct, one-to-one correlation to HACMP networks or number of network interfaces. The algorithm for allocating interfaces and networks to heartbeat rings is complex, but generally follows these rules:

  • In an HACMP network, there is one heartbeat ring to monitor the service interfaces, and one for each set of non-service interfaces that are on the same subnet. The number of non-service heartbeat rings is determined by the number of non-service interfaces in the node with the largest number of interfaces.
  • The number of heartbeat rings is approximately equal to the largest number of interfaces found on any one node in the cluster.
  • Note that during cluster verification, HACMP calls the RSCT verification API. This API performs a series of verifications, including a check for the heartbeat ring calculations, and issues an error if the limit is exceeded.

    Heartbeating over IP Aliases

    This section contains the following subsections:

  • Overview
  • Setting up Heartbeating over IP Aliases
  • How HACMP Assigns Heartbeat Rings
  • Examples of Heartbeat Rings
  • Viewing IP Addresses Assigned by HACMP for Heartbeating over IP Aliases.
  • Overview

    In general, HACMP subnetting requirements can be complicated to understand and may require that you reconfigure networks in AIX 5L to avoid features such as multiple subnet routes, which can lead to a single point of failure for network traffic.

    When planning your cluster networks, you may need to:

  • Reconfigure IP addresses of HACMP interfaces that will be used at boot time
  • or
  • Update /etc/hosts with the new boot time IP addresses.
  • Heartbeating over IP Aliases is useful because it:

  • Uses automatically generated IP aliases for heartbeating
  • Heartbeating over IP Aliasing provides an option where the addresses used for heartbeating can be automatically configured by HACMP in a subnet range that is outside of the range used for the base NIC or any service addresses.
    Note: Although Heartbeating over IP Aliasing automatically configures proper aliases for heartbeating, you must still be aware of the implications of subnet routing for all boot and service IP addresses. That is, failure to plan subnets properly can lead to application failures that are not detectable by HACMP. Reliable HACMP cluster communication still requires that the interfaces on a single network can communicate with the other nodes on that network.
  • Enables you to avoid reconfiguration of boot time addresses and /etc/hosts.
  • RSCT sets up the heartbeat rings to go over a separate range of IP aliases. This lets you use a specified subnet in a non-routable range for a heartbeat ring, preserving your other subnets for routable traffic. This also allows you to avoid reconfiguring boot time addresses and entries in /etc/hosts.
  • Makes HACMP topology configuration easier to understand.
  • Does not require that you obtain additional routable subnets from the network administrator.
  • For instance, you can use heartbeating over aliases in HACMP, if due to the network system administration restrictions, the IP addresses that your system can use at boot time must reside on the same subnet. (In general, if there are no system administration restrictions, the IP addresses that your system can use at boot time can reside on either the same or different subnets).

    Setting up Heartbeating over IP Aliases

    Although Heartbeating over IP Aliasing automatically configures proper aliases for heartbeating, you must still be aware of the implications of subnet routing for all boot and service IP addresses. That is, failure to properly plan subnets can lead to application failures that are not detectable by HACMP. Once you are confident that you understand the subnetting requirements and that Heartbeating over IP Aliasing is appropriate to use, use the instructions in this section and in the Administration Guide.

    To set up heartbeating over IP aliases, use SMIT to set an IP Address Offset for Heartbeating over IP Aliases as part of the network configuration.

    The IP Address Offset for Heartbeating over IP Aliases has the following characteristics:

  • HACMP uses it as a starting address to set up the range of IP aliases on multiple subnets on all the interfaces in the cluster network. The subnet mask is the same as the one used for the network.
  • The range of IP aliases should not overlap with any addresses that are already used on this network.
  • In other words, the starting address that you select must reside on a subnet that is not used by any other network in your physical network setup, and you must have enough available subnets above the one you type in for N networks, where N is the number of interfaces that each node has on the network. HACMP verifies this during the cluster verification process.
  • The IP alias addresses in the range are only used by HACMP for heartbeats. They do not need to be routed and should not be used for any other traffic.
  • HACMP Configuration Database HACMP adds the IP aliases to the network interfaces at cluster startup and during the cluster verification and synchronization.
  • HACMP also gives RSCT the addresses to use for heartbeating. At shutdown, HACMP removes the aliases from the interfaces.
  • To activate heartbeating over IP aliases in HACMP, you:

      1. Add an HACMP network using the Extended Configuration path in SMIT and specify for this network an IP Address Offset for Heartbeating over IP Aliases.
      2. Add an IP Address Offset to an existing network using the Extended HACMP Configuration path. The netmask used for the IP alias address will be the same as the netmask of the underlying interfaces.

    To deactivate heartbeating over aliases, remove the IP Address Offset from the network.

    How HACMP Assigns Heartbeat Rings

    This section explains how HACMP manages IP aliases and service labels used for heartbeating over IP aliases. Heartbeating over aliases can be used with either type of IPAT: IPAT via IP aliasing or IPAT via IP replacement:

  • With IPAT via replacement, during fallover, the service IP label is swapped with the boot time address, not with the heartbeating alias IP address.
  • With IPAT via aliasing, during fallover, the service IP label is aliased onto the boot interface along with the heartbeat alias.
  • HACMP assigns heartbeating aliases based on the interface address:

  • For each interface in the network on a node, HACMP increments the subnet address.
  • For each node in the network, HACMP increments the host address.
  • Examples of Heartbeat Rings

    For example, if you specify in SMIT the IP Address Offset 192.168.1.1, HACMP builds the following heartbeating rings:

    192.168.1.1 <---------> 192.168.1.2

    192.168.2.1 <---------> 192.168.2.2

    The following diagram displays these heartbeat rings:

    Heartbeating over IP Aliases: Heartbeating Rings 
    

    The following table displays an example of configured IP addresses:

    IP Address Example
    Role in HACMP
    Action by HACMP
    1.1.1.1
    service IP alias
    monitored by HACMP
    192.168.1.1.
    IP alias used for heartbeating
    monitored by HACMP
    2.2.2.1
    IP address used at boot time
    not monitored by HACMP

    IP Address Example
    Role in HACMP
    Action by HACMP
    1.1.1.2
    service IP alias
    monitored by HACMP
    192.168.2.1.
    IP alias used for heartbeating
    monitored by HACMP
    2.2.2.2
    IP address used at boot time
    not monitored by HACMP

    Note: In the previous table, the base addresses (such as 2.2.2.1) are not used for heartbeating traffic, but the health of the underlying NIC is still monitored using the heartbeating via IP alias address. Any failure of the NIC alerts HACMP to recover any service and persistent labels that are currently configured on that NIC.

    As another example, you can use 10.10.10.1 as the IP Address Offset (starting address). If you have a network with two NICs on each node, and a subnet mask of 255.255.255.0, HACMP assigns with the following heartbeat IP aliases:

    Node A:

    For network en0, boot_label = nodeA_boot
    hb_IP_alias_label1 = 10.10.10.1
    For network en1, boot_label = nodeA_boot2
    hb_IP_alias_label2 = 10.10.11.1

    Similarly, for Node B:

    hb_IP_alias_label1 = 10.10.10.2
    hb_IP_alias_label2 = 10.10.11.2

    The HACMP cluster verification utility checks that you have a valid configuration for the address range. HACMP verifies that:

  • All interfaces have the same netmask and the same type.
  • The IP Offset Address allows for enough addresses and subnets.
  • Note: Verification does not check the interfaces or subnet requirements for a heartbeating over IP aliases configuration because they use a separate address range.

    Viewing IP Addresses Assigned by HACMP for Heartbeating over IP Aliases

    The IP aliases used for heartbeating show up when you run AIX 5L commands such as netstat.

    Heartbeating over Disk

    You can configure a non-IP point-to-point heartbeating network, called a disk heartbeating network, over any shared disk in an enhanced concurrent mode volume group. Heartbeating over disk provides another type of non-IP point-to-point network for failure detection.

    Disk heartbeating networks provide an alternative to other point-to-point networks such as RS232 that have cable length restrictions, or TMSSA which require special disk adapter hardware and cabling. Heartbeating over disk does not require additional or specialized hardware, cabling or microcode; it can use any disk that is also used for data and for which volume groups and filesystems are included in an HACMP resource group.

    In a disk heartbeating network, two nodes connected to the disk periodically write heartbeat messages and read heartbeat messages (written by the other node) on a small, non-data portion of the disk. A disk heartbeating network, like the other non-IP heartbeating networks, connects only two nodes. In clusters with more than two nodes, use multiple disks for heartbeating. Each node should have a non-IP heartbeat path to at least one other node. If the disk heartbeating path is severed, at least one node cannot access the shared disk.

    You have two different ways for configuring a disk heartbeating network in a cluster:

  • You can create an enhanced concurrent volume group shared by multiple nodes in your cluster. Then you use the HACMP Extended Configuration SMIT path to configure a point-to-point pair of discovered communication devices.
  • or
  • You can start by creating a cluster disk heartbeating network, and then add devices to it using the Add Pre-Defined Communication Interfaces and Devices panel in SMIT.
  • The HACMP cluster verification utility verifies that the disk heartbeating networks are properly configured.

    For information on testing a disk heartbeating network for connectivity, see the section RSCT Command for Testing Disk Heartbeating in Appendix C in the Administration Guide.

    For information on checking the disk heartbeating network, see the Troubleshooting Guide.

    Heartbeating over Disk and Fast Method for Node Failure Detection

    With HACMP 5.4, you can reduce the time it takes for node failure to be realized throughout the cluster. If you have a disk heartbeating network configured, and specify a parameter for a disk heartbeating NIM, then when a node fails, HACMP 5.4 uses a disk heartbeating network to place a departing message on the shared disk so neighboring nodes are aware of the node failure within one heartbeat period. For more information, see the section Decreasing Node Fallover Time in this chapter.

    For information on enabling fast failure detection in HACMP, see the chapter on configuring NIMs in the Administration Guide.

    Designing the Network Topology

    The combination of IP and non-IP (point-to-point) networks that link cluster nodes and clients is called the cluster network topology. The HACMP software supports a large number of IP and point-to-point devices on each node to provide flexibility in designing a network configuration.

    When designing your network topology, ensure that clients have highly available network access to their applications. This requires that none of the following is a single point of failure:

  • The IP subsystem
  • A single network
  • A single NIC.
  • Eliminating the IP Subsystem as Single Points of Failure

    If the IP software fails on a node, you lose all IP communications to and from the node. You can increase availability by configuring non-IP point-to-point connections that directly link cluster nodes. This provides an alternate heartbeat path for the cluster, and it prevents the IP software from being a single point of failure.

    Non-IP network heartbeat rings protect the cluster from getting into an invalid state because of possible network contention issues. UDP packets that carry the heartbeat packet over IP networks will be dropped if there is a large amount of network contention on the wire where the heartbeating is occurring.

    For information about using non-IP point-to-point networks, see the section Planning Point-to-Point Networks in this chapter.

    Eliminating Networks as Single Points of Failure

    In a single-network setup, each node in the cluster is connected to only one network and has a single service interface available to clients. In this setup, the network is a single point of failure for the entire cluster, and each service interface is a single point of failure. The following diagram shows a single-network configuration:

    Single-Network, Single-NIC Setup 
    

    To eliminate the network as a single point of failure, configure multiple networks so that HACMP has multiple paths among cluster nodes. Keep in mind that if a client is connected to only one network, that network is a single point of failure for the client. In a multiple-network setup if one network fails, the remaining network(s) can still function to connect nodes and provide access for clients.

    The more networks you can configure to carry heartbeats and other information among cluster nodes, the greater the degree of system availability. Non-IP point-to-point networks are often used as an alternative heartbeat path. For information about planning point-to-point networks, see the section Planning Point-to-Point Networks in this chapter.

    The following diagram illustrates a dual-network setup with more than one path to each cluster node:

    Dual-Network Setup 
    
    Note: Hot-replacement of the dual-port Ethernet adapter used to configure two interfaces for one HACMP IP network is currently not supported.

    Planning Point-to-Point Networks

    You can also increase availability by configuring non-IP point-to-point connections that directly link cluster nodes. These connections provide:

  • An alternate heartbeat path for a cluster that uses a single TCP/IP-based network, and prevent the TCP/IP software from being a single point of failure
  • Protection against cluster partitioning.
  • For more information about cluster partitioning, see the section Cluster Partitioning in this chapter.

    You can configure heartbeat paths over the following types of networks:

  • Non-IP (RS232)
  • Disk heartbeating (over an enhanced concurrent mode disk)
  • Target Mode SSA
  • Target Mode SCSI.
  • Selecting a Layout for Point-to-Point Networks

    Because point-to-point networks play an important role in ensuring high availability of networks, select connections that create adequate paths in clusters that contain more than two nodes. The following table briefly lists the configuration types and describes the advantages or disadvantages of each:

    Type
    Description
    Considerations to Keep in Mind
    Mesh
    Connects each node in the cluster to all other nodes in the cluster
    Provides the most robust and reliable configuration
    Star
    Connects one node with all other nodes
    May create partitioned clusters when multiple failures occur simultaneously
    Ring or Loop
    Connects each node to directly adjacent neighbors

    The type of point-to-point networks you include in your network topology depends on the hardware available and the requirements for your cluster.

    Planning Serial Point-to-Point Networks

    When planning a serial (RS232) network, keep in mind the following:

  • If there are no serial ports available, and your planned HACMP configuration for that node uses an RS232 network, the configuration requires a serial NIC card.
  • All RS232 networks defined to HACMP are brought up by RSCT with a default of 38400 bps. The tty ports should be defined to AIX 5L as running at 38400 bps.
  • RSCT supports baud rates of 38400, 19200, 9600.

    Any serial port that meets the following requirements can be used for heartbeats:

  • The hardware supports use of that serial port for modem attachment.
  • The serial port is free for HACMP exclusive use.
  • Examples of processors with native serial ports that do not meet these conditions are S70, S7A, S80, and serial ports 1 and 2 in the F50, H50, and H70.

    Certain RS/6000 systems do not support the use of native serial ports.

    Note: HACMP supports serial ports that the hardware and the system software make available for application use. It is your responsibility to manage any modems or extenders between the ports.

    Refer to the hardware documentation and HACMP support announcements to determine whether your serial ports meet the requirements.

    The following figure shows four RS232 networks, in a ring configuration, connecting four nodes to provide full cluster non-IP connectivity:

    Serial Network Configuration in an HACMP Cluster 
    

    Planning Disk Heartbeating Networks

    Any shared disk in an enhanced concurrent mode volume group can support a point-to-point heartbeat connection. Each disk can support one connection between two nodes. The connection uses the shared disk hardware as the communication path.

    A disk heartbeating network in a cluster contains:

  • Two nodes
  • A node may be a member of any number of one disk heartbeating networks. A cluster can include up to 256 communications devices.
  • An enhanced concurrent mode disk that participates in only one heartbeat network.
  • Keep in mind the following points when selecting a disk to use for disk heartbeating:
    A disk used for disk heartbeating must be a member of an enhanced concurrent mode volume group. However, the volume groups associated with the disks used for disk heartbeating do not have to be defined as resources within an HACMP resource group. In other words, an enhanced concurrent volume group associated with the disk that enables heartbeating does not have to belong to any resource group in HACMP.
    You can convert an existing volume group to enhanced concurrent mode. For information about converting a volume group, see Chapter 12: Managing Shared LVM Components in a Concurrent Access Environment in the Administration Guide.
  • The disk should have fewer than 60 seeks per second at peak load. (Disk heartbeats rely on being written and read within certain intervals.)
  • Use the AIX 5L filemon command to determine the seek activity, as well as the I/O load for a physical disk.
    Typically, most disk drives that do not have write caches can perform about 100 seeks per second. Disk heartbeating uses 2 to 4 seeks.
    Disks that are RAID arrays, or subsets of RAID arrays, may have lower limits. Check with the disk or disk subsystem manufacturer to determine the number of seeks per second that a disk or disk subsystem can support.
    However, if you choose to use a disk that has significant I/O load, increase the value for the timeout parameter for the disk heartbeating network.
  • When SDD is installed and the enhanced concurrent volume group is associated with an active vpath device, ensure that the disk heartbeating communication device is defined to use the /dev/vpath device (rather than the associated /dev/hdisk device).
  • If a shared volume group is mirrored, at least one disk in each mirror should be used for disk heartbeating.
  • Make sure to set up heartbeating in this way when you plan to set the forced varyon option for a resource group. For more information about using HACMP forced varyon, see the section Using Quorum and Varyon to Increase Data Availability in Chapter 5: Planning Shared LVM Components.

    Planning Target Mode Networks

    Target mode SCSI and target mode SSA are also supported for point-to-point heartbeat communications. Each of these types of networks includes two nodes, a shared disk, and SCSI or SSA communications (as appropriate to the disk type).

    Extending the Distance of Point-to-Point Networks

    HACMP point-to-point networks can be extended over longer distances through the use of extender technologies:

    Network Type
    Extension Mechanism
    SCSI
    SCSI Bus Extenders
    SSA
    SSA Optical Extenders
    RS232
    • Optical line drivers
    • Short-haul modems
    • PTT-supplied RS232 lines

    When planning long-distance point-to-point network devices choose connections, such as fibre or copper cabling, within the operational tolerances at the distances required.

    When extending RS232 links, ensure that the device:

  • Supports the full RS232 protocol (all pins)
  • Is capable of supporting the distance required at a minimum line speed of 9600 bps (the default is 38400 bps).
  • Some RS232 networks that are extended to longer distances require that the baud rate be lower than the default of 38400 bps. For details, see the section Changing an RS232 Network Module Baud Rate in Chapter 13: Managing the Cluster Topology in the Administration Guide.
    The Failure Detection Rate may also need to be adjusted. For more information, see the section Setting Failure Detection Parameters in this chapter.

    Configuring Global Networks

    If you have multiple physical networks of the same type, each spanning only a portion of the cluster, you can group these networks into one global HACMP network. HACMP will route its heartbeat packets between the different networks, giving it another level of network redundancy, and thereby reducing the probability of cluster partitioning.

    Note: Networks configured to use any form of IP address takeover cannot be included in a global network.

    Creating global networks is useful on large SP systems, as each SP frame of nodes is usually set up as a different subnet on the SP Administrative Ethernet. Each of these subnets is then defined as a different HACMP network. Defining a global network that includes all these SP administrative Ethernets will avoid the possibility of cluster partitioning. For more information, see the section Planning for SP Networks in this chapter.

    Eliminating Network Interface Cards as a Single Point of Failure

    A network interface card (NIC) physically connects a node to a network. When configured with a single NIC per network, the NIC becomes a potential single point of failure. To remedy this problem, configure a node with at least two NICs for each network to which it connects. In the following figure, each cluster node has two connections to each network:

    Dual-Network, Dual-NIC Configuration 
    

    For more information about the actions HACMP takes in response to NIC failures, see Appendix B: Resource Group Behavior during Cluster Events in the Administration Guide.

    Note: Hot-replacement of the dual-port Ethernet adapter used to configure two interfaces for one HACMP IP network is currently not supported.

    Network Interface Functions

    When a node is configured with multiple connections to a single network, the network interfaces serve different functions in HACMP:

  • The service interface
  • The non-service interfaces.
  • You can also define a persistent node IP interface as an IP alias to a cluster network on a node.

    Service Interface

    A service interface is a communications interface configured with an HACMP service IP label. This interface serves as each node’s primary HACMP connection to each network. The service IP label is used in the following ways:

  • By clients to access application programs
  • For HACMP heartbeat traffic.
  • Non-service Interface

    A non-service interface is a communications interface with an HACMP non-service IP label that backs up a service interface. All client traffic is carried over the service interface. Non-service interfaces are hidden from client applications and carry only internal HACMP traffic. If a service interface fails, HACMP can move the service IP label onto a non-service interface. Using a non-service interface eliminates a network interface as a single point of failure.

    In addition, if a node fails, the cluster can use a non-service interface on another cluster node as a location for its service IP label when performing a resource group fallover.

    A node can have from zero to seven non-service interfaces for each network to which it connects. Your software configuration and hardware constraints determine the actual number of non-service interfaces that a node can support.

    Persistent Node IP Label

    A persistent node IP label is an IP alias that can be assigned to a specific node on a cluster network. A persistent node IP label:

  • Always stays on the same node (is node-bound)
  • Coexists on a NIC that already has a service or non-service IP label defined
  • Does not require installing an additional physical NIC on that node
  • Is not part of any resource group.
  • Assigning a persistent node IP label provides a node-bound address that you can use for administrative purposes, because a connection to a persistent node IP label always goes to a specific node in the cluster. You can have one persistent node IP label per network per node.

    As one of HACMP’s best practices, we recommend that you configure a persistent IP label for each cluster node. This is useful, for instance, if you must access a particular node in an HACMP cluster for purposes of running reports or for diagnostics. Having a persistent IP label configured has the advantage that HACMP can access the persistent IP label on the node despite individual NIC failures (provided there are spare NICs on the network).

    After a persistent node IP label is configured on a specified network node, it becomes available at boot time and remains configured even if HACMP is shut down on that node.

    You can create persistent node IP labels on the following types of IP-based networks:

  • Ethernet
  • Token Ring
  • FDDI
  • ATM LAN Emulator.
  • You cannot configure a persistent node IP label on the SP Switch, on ATM Classical IP, or on a serial cluster network.

    The following list describes how HACMP responds to a failure when a persistent node IP label is configured:

  • If a NIC that has a service IP label configured fails, and there is also a persistent label defined on this NIC, then the persistent label falls over to the same non-service interface to which the service IP label falls over.
  • If all NICs on the cluster network on a specified node fail, then the persistent node IP label becomes unavailable. A persistent node IP label always remains on the same network, and on the same node; it does not move between the nodes in the cluster.
  • For more information about persistent node IP labels/addresses, see the section Configuring HACMP Persistent Node IP Labels/Addresses in Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended) in the Administration Guide.

    IP Address Takeover via IP Aliases

    HACMP IP networks have IPAT via IP Aliases enabled by default. When HACMP is started on the node, the service IP label is aliased onto one of the non-service interfaces defined to HACMP. If that interface fails, the service IP label is aliased onto another interface if one is available on the same network. Hardware Address Takeover (HWAT) is not possible in this configuration. In order to use IPAT via IP Aliases, the network must support gratuitous ARP.

    For complete information on IPAT via IP Aliases, see Planning for IP Address Takeover via IP Aliases.

    Subnet Considerations for Persistent Node IP Labels

    When you configure a persistent node IP label on a cluster network configured with IP Address Takeover via IP Aliases, the IP address associated with a persistent IP label must be on a different subnet than any service address with which it may share an interface.

    In some situations you may need to configure a persistent IP label on the same subnet as the service IP label. In this case, to avoid problems with network packets sent from either of the addresses, consider configuring the distribution preference for service IP aliases. This preference is available in HACMP 5.1 and up and, if configured, lets you configure the type of the distribution preference suitable for the VPN firewall external connectivity requirements.

    For more information about the types of distribution preferences for service IP labels, see the section Types of Distribution for Service IP Label Aliases in this chapter.

    Note: The subnet considerations are different if you are planning to configure NFS filesystems. For information, see NFS Cross-Mounting and IP Labels in Chapter 5: Planning Shared LVM Components.

    IP Address Takeover via IP Replacement

    IP Address Takeover can be set up to use IP Replacement (by disabling the IPAT via IP Aliases option in SMIT). In this configuration, if a node fails, another node takes over the service IP label as part of a resource group takeover. This service IP label will be configured onto one of the takeover node’s non-service interfaces (on the appropriate network) in place of the non-service IP label.

    Hardware Address Takeover (HWAT) and IPAT via IP Replacement

    This configuration supports Hardware Address Takeover (HWAT) through alternate hardware addresses. If HWAT is not used, ensure that clients and network equipment get their ARP caches updated after IPAT occurs, and after fallback, if necessary.

    Subnet Considerations for Persistent Node IP Labels

    When you configure a persistent node IP label on a cluster network configured with IP Address Takeover via IP Replacement, the subnet you use must either be the subnet used by the service IP label, or unique to that network on that node.

    Planning for IP Address Takeover via IP Aliases

    You can configure IP address takeover on certain types of networks using the IP aliases network capabilities supported in AIX 5L. Assigning IP aliases to NICs allows you to create more than one IP label on the same network interface.

    HACMP allows the use of IPAT via IP Aliases with the following network types that support gratuitous ARP in AIX 5L:

  • Ethernet
  • Token Ring
  • FDDI
  • SP Switch1 and SP Switch2.
  • Note: IPAT via IP Aliases is not supported on ATM networks.

    During IP address takeover via IP aliases, when an IP label moves from one NIC to another, the target NIC receives the new IP label as an IP alias and keeps the original IP label and hardware address.

    Configuring networks for IPAT via IP Aliases simplifies the network configuration in HACMP. You configure a service address and one or more non-service addresses for NICs.

    This section contains the following topics:

  • Assigning IP Labels for IPAT via IP Aliases
  • Planning for Service IP Label Alias Placement
  • Types of Distribution for Service IP Label Aliases
  • Planning for Site-Specific Service IP Labels.
  • Assigning IP Labels for IPAT via IP Aliases

    IP address takeover via IP aliases is an alternative method of taking over the IP address that allows one node to acquire the IP label and the IP address of another node in the cluster using IP aliases.

    To enable IP Address Takeover via IP aliases, configure NICs to meet the following requirements:

  • At least one boot-time IP label must be assigned to the service interface on each cluster node.
  • Hardware Address Takeover (HWAT) cannot be configured for any interface that has an IP alias configured.
  • Subnet requirements:
  • Multiple boot-time addresses configured on a node should be defined on different subnets.
  • Service addresses must be on a different subnet from all non-service addresses defined for that network on the cluster node. This requirement enables HACMP to comply with the IP route striping function of AIX 5L, which allows multiple routes to the same subnet.
  • Service address labels configured for IP address takeover via IP aliases can be included in all non-concurrent resource groups.
  • Multiple service labels can coexist as aliases on a given interface.
  • The netmask for all IP labels in an HACMP network must be the same.
  • You cannot mix aliased and non-aliased service IP labels in the same resource group.
  • HACMP non-service labels are defined on the nodes as the boot-time address, assigned by AIX 5L after a system reboot and before the HACMP software is started. When the HACMP software is started on a node, the node’s service IP label is added as an alias onto one of the NICs that has a non-service label.

    When you configure IPAT via IP Aliases, the node’s NIC meets the following conditions:

  • The NIC has both the boot-time and service IP addresses configured, where the service IP label is an alias placed on the interface
  • Unlike in IPAT via IP Replacement, the boot-time address is never removed from a NIC
  • If the node fails, a takeover node acquires the failed node’s service address as an alias on one of its non-service interfaces on the same HACMP network.
  • Note: During a node fallover event, the service IP label that is moved is placed as an alias on the target node’s NIC in addition to any other service labels that may already be configured on that NIC.

    For example, if Node A fails, Node B acquires Node A’s service IP label. This service IP label is placed as an alias onto the appropriate NIC on Node B, and any other existing labels remain intact on Node B’s NIC. Thus, a NIC on Node B now receives client traffic directed to Node A’s service address. Later, when Node A is restarted, it comes up on its boot-time address(es) and attempts to reintegrate into the cluster by requesting that Node B release Node A’s service IP label.

    When Node B releases the requested service IP label, the alias for the service IP labels is deleted on Node B, and Node A once again puts it as an alias onto one of its non-service interfaces on the appropriate network.

    When using IPAT via IP Aliases, service IP labels are acquired using all available non-service interfaces. If there are multiple interfaces available to host the service IP label, the interface is chosen according to the number of IP labels currently on that interface. If multiple service IP labels are acquired and there are multiple interfaces available, the service IP labels are distributed across all the available interfaces.

    If you remove the boot-time address with the ifconfig delete command, even though the interface can have working aliased service IP labels on it, Topology Services detects a local NIC failure. This is done because the boot-time address that is being monitored for heartbeats is no longer available.

    Note: If you plan to have an NFS filesystem and use IPAT via IP Aliases in the cluster, see the section NFS Cross-Mounting and IP Labels in Chapter 5: Planning Shared LVM Components.

    For information about configuring networks to use IPAT via IP Aliases, see the section Configuring IP-Based Networks in Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended) of the Administration Guide.

    Planning for Service IP Label Alias Placement

    If you use IPAT via IP Aliases, you can configure a distribution preference for the placement of service IP labels that are configured in HACMP.

    HACMP lets you specify the distribution preference for the service IP label aliases. These are the service IP labels that are part of HACMP resource groups and that belong to IPAT via IP Aliases networks.

    A distribution preference for service IP label aliases is a network-wide attribute used to control the placement of the service IP label aliases on the physical network interface cards on the nodes in the cluster.

    We recommend using the distribution preference for IP aliases to address the following cluster requirements:

  • Firewall considerations
  • Cluster configurations that use VLANs (when applications are expecting to receive packets from a specific network interface)
  • Specific requirements for the placement of IP labels in the cluster.
  • Distribution Preference for Service IP Label Aliases: How It Works

    Configuring a distribution preference for service IP label aliases does the following:

  • Lets you customize the load balancing for service IP labels in the cluster, taking into account the persistent IP labels previously assigned on the nodes.
  • Enables HACMP to redistribute the alias service IP labels according to the preference you specify.
  • Allows you to configure the type of the distribution preference suitable for the VPN firewall external connectivity requirements.
  • Although the service IP labels may move to another network interface, HACMP ensures that the labels continue to be allocated according to the specified distribution preference. That is, the distribution preference is maintained during startup and the subsequent cluster events, such as a fallover, fallback or a change of the interface on the same node. For instance, if you specified the labels to be mapped to the same interface, the labels will remain mapped on the same interface, even if the initially configured service IP label moves to another node.
  • The distribution preference is exercised in the cluster as long as there are acceptable network interfaces available. HACMP always keeps service IP labels active, even if the preference cannot be satisfied.
  • Types of Distribution for Service IP Label Aliases

    You can specify in SMIT the following distribution preferences for the placement of service IP label aliases:

    Type of distribution preference
    Description
    Anti-collocation
    This is the default. HACMP distributes all service IP label aliases across all boot IP labels using a “least loaded” selection process.
    Collocation
    HACMP allocates all service IP label aliases on the same network interface card (NIC).
    Anti-collocation with persistent
    HACMP distributes all service IP label aliases across all active physical interfaces that are not hosting the persistent IP label. HACMP will place the service IP label alias on the interface that is hosting the persistent label only if no other network interface is available.
    If you did not configure persistent IP labels, HACMP lets you select the Anti-Collocation with Persistent distribution preference, but it issues a warning and uses the regular anti-collocation preference by default.
    Collocation with persistent
    All service IP label aliases are allocated on the same NIC that is hosting the persistent IP label. This option may be useful in VPN firewall configurations where only one interface is granted external connectivity and all IP labels (persistent and service) must be allocated on the same interface card.
    If you did not configure persistent IP labels, HACMP lets you select the Collocation with Persistent distribution preference, but it issues a warning and uses the regular collocation preference by default.

    The following rules apply to the distribution preference:

  • If there are insufficient interfaces available to satisfy the preference, HACMP allocates service IP label aliases and persistent IP labels to an existing active network interface card.
  • If you did not configure persistent labels, HACMP lets you select the Collocation with Persistent and Anti-Collocation with Persistent distribution preferences, but it issues a warning and uses the regular collocation or anti-collocation preferences by default.
  • You can change the IP labels distribution preference dynamically: the new selection becomes active during subsequent cluster events. HACMP does not interrupt the processing by relocating the currently active service IP labels at the time the preference is changed.
  • When a service IP label fails and another one is available on the same node, HACMP recovers the service IP label aliases by moving them to another network interface card on the same node. During this event, the distribution preference that you specified remains in effect.

    For information on configuring and changing the distribution preference for service IP labels, see the Administration Guide.

    Planning for Site-Specific Service IP Labels

    You can have a service IP label configurable on multiple nodes, associated with a resource group than can move between nodes or sites. Since an IP address that is valid at one site may not be valid at the other site due to subnet issues, you can associate a service IP label that is configurable on multiple nodes with a specific site. Site-specific service IP labels are configured in HACMP and can be used with or without XD-type networks. This label is associated with a resource group and is active only when the resource group is in online primary state at the associated site.

    Planning for IP Address Takeover via IP Replacement

    If you disable the default option to configure the network with IP Address Takeover via IP Aliases, you can use IP Address Takeover via IP Replacement. This method uses a service IP label and one or more non-service IP labels per node for a given network. At boot, AIX 5L comes up with a configured IP address on each NIC. When HACMP is started, it replaces non-service IP labels with service IP labels for the resource groups it brings online.

    If the NIC that is hosting the service IP label fails, and there is another non-service interface for that network on the node, HACMP will replace that non-service IP label with the service IP label, in order to maintain the service IP label.

    Each node on the network, including client nodes, maintains an ARP cache that maps each IP address to a particular hardware (MAC) address. After HACMP moves a service IP label and the address associated with it to a different NIC, these ARP cache entries must be updated so that the service address correctly matches the hardware address of the target NIC.

    AIX 5L does a gratuitous ARP when this reconfiguration occurs. This allows clients and network devices that support promiscuous listen mode to update their ARP caches. If your clients do not support promiscuous listen mode, you can either use clinfo to update their ARP caches, or use Hardware Address Takeover (HWAT).

    All non-concurrent resource groups can be configured with IP labels on IPAT via IP Replacement networks.

    If you are using an IPAT via IP Replacement network and plan on configuring a non-concurrent resource group, you may also consider using the node distribution policy for the resource group’s startup. For more information, see Node Distribution Policy in Chapter 6: Planning Resource Groups.

    Hardware Address Takeover with IPAT via IP Replacement

    Certain types of networks cannot process ARP cache updates. Also, some clients and network devices do not support promiscuous listen. To eliminate these ARP cache difficulties, HACMP uses the Hardware Address Takeover (HWAT) facility of AIX 5L. HWAT not only moves the service IP label, but also moves the underlying hardware (MAC) address of the target NIC so that it remains with the service IP label. This way, as both the IP address and the hardware address are moved, the ARP cache on the node remains correct. For more information, see the section Selecting an Alternate Hardware Address in this chapter.

    Note: Turn off flow control on the gigabit Ethernet adapters. This can cause network problems after an HACMP fallover.

    Planning for Other Network Conditions

    The most typical network planning consideration involve nameserving in an HACMP environment, and setting up cluster monitoring and failure detection.

    This section contains the following topics:

  • Using HACMP with NIS and DNS
  • Monitoring Clusters
  • Planning for VPN Firewall Network Configurations in HACMP
  • Setting Failure Detection Parameters
  • Setting Values for the Network Grace Period
  • Identifying Service Adapter Failure for Two-Node Clusters
  • Setting RS232 TTY Baud Rates
  • Planning Networks for Inter-Node Communication with Oracle.
  • Using HACMP with NIS and DNS

    Some of the commands used to troubleshoot network and interface problems after a failure require IP lookup to determine the IP address associated with a specified IP label.

    If NIS or DNS is in operation, IP lookup defaults to a nameserver system for name and address resolution. However, if the nameserver was accessed through an interface that has failed, the request does not complete, and eventually times out. This may significantly slow down HACMP event processing.

    To ensure that cluster event completes successfully and quickly, HACMP disables NIS or DNS hostname resolution by setting the following AIX 5L environment variable during service IP label swapping:

    NSORDER = local

    As a result, the /etc/hosts file of each cluster node must contain all HACMP defined IP labels for all cluster nodes.

    DNS Requests Sent from Non-HACMP Processes

    Disabling NIS or DNS hostname resolution is specific to the HACMP event script environment. HACMP sets the NSORDER variable to local when it attaches a service IP label and when it swaps IP labels on an interface.

    Other processes continue to use the default system name resolution settings (for example, applications outside of HACMP that require DNS IP address resolution). If these processes request IP lookup, then during the network interface reconfiguration events in HACMP the processes may still not be able to contact an external name server. The request to the DNS will succeed after HACMP completes the network interface reconfiguration event.

    Monitoring Clusters

    Each supported cluster network has a corresponding cluster network module that monitors all I/O to its cluster network. The network modules maintain a connection to each other in a cluster. The Cluster Managers on cluster nodes send messages to each other through these connections.

    Each network module sends periodic heartbeat messages to and receives periodic heartbeat messages from other network modules in the cluster to:

  • Monitor interfaces
  • Ensure connectivity to cluster peers
  • Report when a connection fails.
  • Currently, HACMP network modules support communication over the following types of networks:

  • Serial (RS232)
  • disk heartbeating (over enhanced concurrent mode disks)
  • Target-mode SCSI
  • Target-mode SSA
  • Ethernet
  • Token Ring
  • FDDI
  • SP Switch
  • ATM.
  • Planning for VPN Firewall Network Configurations in HACMP

    Starting with HACMP 5.1, HACMP lets you specify the distribution preference for the service IP label aliases. These are the service IP labels that are part of HACMP resource groups and that belong to IPAT via IP Aliasing networks.

    Certain VPN firewall configurations allow external connectivity to only one NIC at a time. If your firewall is configured this way, allocate all HACMP service and persistent IP labels on the same interface.

    To have HACMP manage the IP labels to satisfy the requirements of such a VPN firewall:

  • Specify the persistent IP label for each node in the cluster. The persistent IP label is mapped to an available interface on the selected network.
  • Use IPAT via IP Aliases for the network that contains the service IP labels (this is the default). That is, when configuring service IP labels as resources in a resource group, ensure that the Enable IP Address Takeover via IP Aliases field is set to Yes under the Configure HACMP Networks SMIT panel.
  • Specify the Collocation with Persistent distribution preference for the network containing the service IP labels. This ensures that all service IP label aliases are allocated on the same physical interface that is hosting the persistent IP label.
  • For an overview of types of distribution preferences you can specify in SMIT, see Types of Distribution for Service IP Label Aliases in this chapter. For configuration instructions, see the Administration Guide.

    Setting Failure Detection Parameters

    An important tuning parameter for network modules is the Failure Detection Rate, which determines how quickly a connection is considered to have failed.

    The Failure Detection Rate consists of two components:

  • Cycles to fail (cycle). The number of heartbeats missed before detecting a failure
  • Heartbeat rate (hbrate). The number of seconds between heartbeats.
  • The time needed to detect a failure can be calculated using this formula:

    (heartbeat rate) X (cycles to fail) X 2  
    

    The Failure Detection Rate can be changed for a network module in two ways:

  • Select the preset rates of slow, normal or fast
  • Change the actual components cycle or hbrate.
  • The preset values are calculated for each type of network to give reasonable results. Use the preset rate values (Slow, Normal or Fast) if you change your Failure Detection Rate. For information about customizing the Failure Detection Rate components, see the section Changing a Failure Detection Rate in this chapter.

    You may want to consider changing the Failure Detection Rate to:

  • Decrease fallover time
  • Keep node CPU saturation from causing false takeovers.
  • Decreasing Fallover Time

    The default setting for the failure detection rate is usually optimal. You can slightly speed up fallover by speeding up failure detection. However, the amount and type of customization you add to event processing has a much greater impact on the total fallover time. You should test the system for some time before deciding to change the failure detection speed of any network module.

    Decreasing Node Fallover Time

    HACMP 5.4 reduces the time it takes for a node failure to be realized throughout the cluster, while reliably detecting node failures.

    HACMP 5.4 uses disk heartbeating to place a departing message on a shared disk so neighboring nodes are aware of the failed node within one heartbeat period (hbrate). Remote nodes that share the disks, receive this message and broadcast that the node has failed. Directly broadcasting the node failure event greatly reduces the time it takes for the entire cluster to become aware of the failure compared to waiting for the missed heartbeats, and therefore HACMP can take over critical resources faster.

    The failure detection rates of Fast, Normal and Slow contain hbrates of 1, 2, or 3 seconds respectively. Therefore, if you are using disk heartbeating, the time for the neighbor nodes to determine if the node is down would be at most 1, 2 or 3 seconds, followed by the other cluster nodes immediately becoming aware of the failure.

    For example, using the formula to calculate the time needed to detect a failure:

    (heartbeat rate) X (cycles to fail) X 2

    For a heartbeat rate of 2 seconds and cycles to fail of 12, the adapter failure detection time is 48 seconds compared to the fast method of node failure detection rate of 4 seconds.

    Fast Node Failure Detection Prerequisites

    Full support for the fast node failure detection method requires:

  • HACMP 5.4
  • AIX v.5.3 (the latest available maintenance level)
  • All supported disks except for SSA disks. Fast method for node failure detection is not supported on SSA disks.
  • If you have an entire HACMP 5.4 cluster, but some nodes are not running AIX 5.3, you can still benefit from fast node failure detection method: If an AIX v.5.3 node fails, the neighboring node will react quickly to the failure (even if the neighboring node is running AIX v.5.2). If a node running AIX v.5.2 fails, the neighboring node will wait for the missing heartbeats before declaring its neighbor as failed.

    Fast node failure detection is not supported in a mixed-version HACMP cluster.

    For information about how to enable the fast method of failure detection in HACMP, see the section Changing the Failure Detection Rate of a Network Module, in the chapter on Managing the Cluster Topology in the Administration Guide.

    Eliminating False Takeovers

    If HACMP cannot get enough CPU resources to send heartbeats on IP and point-to-point networks, other nodes in the cluster assume the node has failed and initiate takeover of the node resources. Once this takeover process begins, it is critical that the failed node does not become active and begin using these resources again.

    To ensure a clean takeover, HACMP provides a Deadman Switch, which is configured to halt the unresponsive node one second before the other nodes begin processing a node failure event. The Deadman Switch uses the Failure Detection Parameters of the slowest network to determine at what point to halt the node. Thus, by increasing the amount of time before a failure is detected, you give a node more time in which to give HACMP CPU cycles. This can be critical if the node experiences saturation at times.

    To help eliminate node saturation, modify AIX 5L tuning parameters. For information about these tuning parameters, see the following sections:

  • Configuring Cluster Performance Tuning in Chapter 1: Troubleshooting HACMP Clusters in the Troubleshooting Guide.
  • Changing the Failure Detection Rate of a Network Module in Chapter 13: Managing the Cluster Topology in the Administration Guide.
  • Change Failure Detection Parameters only after these other measures have been implemented.

    Changing a Failure Detection Rate

    If you change the failure detection rate of a network module, keep in mind the following considerations:

  • Failure detection is dependent on the fastest network linking two nodes.
  • The failure rate of networks varies, depending on their characteristics.
  • Before altering the network module, assess how much time you want to elapse before a real node failure is detected by the other nodes and the subsequent takeover is initiated.
  • Faster heartbeat rates may lead to false failure detections, particularly on busy networks. For example, bursts of high network traffic may delay heartbeats and this may result in nodes being falsely ejected from the cluster. Faster heartbeat rates also place a greater load on networks. If your networks are very busy and you experience false failure detections, you can try slowing the failure detection speed on the network modules to avoid this problem.
  • Before customizing the Failure Detection Rate, change the rate from normal to slow (or fast). To customize it, change the hbrate and adjust the cycle values to custom values using the SMIT panel for changing tuning parameters to custom values.
  • The Failure Detection Rate for a particular network should be set equally for a given network across the cluster. The change must be synchronized across cluster nodes. The new values become active during a dynamic reconfiguration (DARE).
  • Whenever a change is made to any of the values that affect the failure detection time (failure cycle (FC), heartbeat rate (HB) or failure detection rate), the new value of these parameters is sent as output to the screen in the following message:
  • SUCCESS: Adapter Failure Detection time is now FC * HB * 2 or SS seconds
  • Setting Values for the Network Grace Period

    During IP Address Takeover (IPAT) operations, the network grace period is the time period in which node reachability is not computed for the network. This is used so that a node is not considered to be down while its NICs are undergoing IPAT. The grace period value needs to account for the time it takes for the interface to be reconfigured with the new address, and the time it takes for it to rejoin its interface membership group. When IPAT is used with HWAT, it usually takes longer for the operation to complete, so larger values of the grace period may have to be used.

    For more information about configuring these parameters, see the section Changing the Tuning Parameters to Predefined Values in Chapter 13: Managing the Cluster Topology in the Administration Guide.

    Identifying Service Adapter Failure for Two-Node Clusters

    In cluster configurations where there are networks that under certain conditions can become single adapter networks, it can be difficult for HACMP to accurately determine adapter failure. This is because RSCT Topology Services cannot force packet traffic over the single adapter to confirm its proper operation. (This shortcoming is less of an exposure if the network adapter is under heavy use. In this case, the inbound packet count continues to increase over the service adapter without stimulation from RSCT Topology Services).

    An enhancement to netmon, the network monitor portion of RSCT Topology Services allows for a more accurate determination of a service adapter failure. This function can be used in configurations that require a single service adapter per network.

    You can create netmon.cf configuration file that specifies additional network addresses to which ICMP ECHO requests can be sent.

    This file must exist at cluster startup—RSCT Topology Services scans the netmon.cf configuration file during initialization. When netmon needs to stimulate the network to ensure adapter function, it sends ICMP ECHO requests to each IP address. After sending the request to every address, netmon checks the inbound packet count before determining whether an adapter has failed.

    Creating the netmon.cf File

    The netmon.cf file must be placed in the /usr/sbin/cluster directory on all cluster nodes.

    Requirements for creating the file:

  • The netmon.cf file consists of one IP address or IP label per cable.
  • Include each IP address and its corresponding label for the netmon.cf file in the /etc/hosts file.
  • When selecting IP addresses (or hostnames) for the netmon.cf file, keep in mind ALL possible IP addresses that an interface might hold at any given time (boot IP addresses, service IP addresses used by HACMP, and other interfaces), and ensure that for each interface on the node, the netmon.cf file contains one or more targets that can be reached by those addresses.
  • For example, an adapter on a node that serves as a boot adapter when it is not holding a service address should be able to ping some targets in the netmon.cf file using that boot address.
    Note: Ensure that the names in the netmon.cf file are included in the /etc/hosts file. If this is not done, then when the NIM process (that is part of the RSCT Topology Services subsystem) attempts to determine the state of the local adapters, it may try to run the hostname resolution. The hostname resolution may in turn result in the process being blocked in cases when the network used for name resolution is unreachable. The blockage may result in longer adapter failure detection times, which will slow fallover operations.
  • A maximum of 30 IP addresses/labels can be defined in netmon.cf.
  • The following example shows a /usr/sbin/cluster/netmon.cf configuration file:

    180.146.181.119 
    steamer 
    chowder 
    180.146.181.121 
    mussel 
    

    Choosing IP Addresses for the netmon.cf File

    Guidelines for choosing the IP addresses to include in the netmon.cf file depend on whether the local interface is a service or a non-service interface:

    If the local interface is a service interface, it will verify that it is operational via point-to-point communication with the following interfaces:

  • Existing remote service interface(s) on the same logical network
  • One of the existing local non-service interfaces on the same logical network
  • IP addresses/hostnames in the netmon.cf file.
  • If the local interface is a non-service interface, it verifies that it is operational via point-to-point communication with the following interfaces:

  • Existing remote non-service(s) on the same logical network
  • One of the existing local interfaces on the same logical network
  • IP addresses/hostnames in the netmon.cf file.
  • The netmon.cf file should contain remote IP labels/addresses that are not in the cluster configuration that can be accessed from HACMP interfaces. Routers can also be used.

    Setting RS232 TTY Baud Rates

    The default baud rate is 38400 bps; however some modems or devices do not support a baud rate of 38400 bps. If this is the case for your situation, you can change the default by customizing the RS232 network module to read the desired baud rate (9600bps, 19200 bps, 38400 bps).

    For more information about configuring these parameters, see the section Changing an RS232 Network Module Baud Rate in Chapter 13: Managing the Cluster Topology in the Administration Guide.

    Planning Networks for Inter-Node Communication with Oracle

    ORACLE uses the private network attribute setting to select networks for Oracle inter-node communications. This attribute is not used by HACMP and will not affect HACMP in any way. The default attribute is public.

    Changing the network attribute to private makes the network Oracle-compatible by changing all interfaces to service (as well as changing the attribute in HACMPnetwork).

    After creating your cluster networks (either manually or using discovery), you can change the network attribute by following this SMIT path:

    Extended Configuration > Extended Topology Configuration > Configure HACMP Networks > Change/Show a Network in the HACMP Cluster.

    Select the network to be changed, then change the Network Attribute setting to private. Synchronize the cluster after making this change. See the Administration Guide for complete instructions.

    Rules for Configuring Private Networks

    Follow these steps to configure private networks for use by Oracle:

      1. Configure the network and add all interfaces. You cannot change the attribute if the network has no interfaces.
      2. Change the network attribute to private.
      3. Private networks must have either all boot or all service interfaces. If the network has all boot interfaces (the default when using discovery) HACMP converts these interfaces to service (Oracle only looks at service interfaces).
      4. Synchronize the cluster after changing the attribute.
    Note: Once you define the network attribute as private you cannot change it back to public. You have to delete the network and then redefine it to HACMP (it defaults to public).

    Planning for SP Networks

    An SP network requires special planning for HACMP networking. There are three SP-specific networks:

  • SP Ethernet. Each SP frame includes an Ethernet network that connects all SP nodes to the Control Workstation. This network is used for software installation and other administrative purposes. In addition, it is available to HACMP and recommended for use as a heartbeat network.
  • SP Switch. An SP frame can optionally contain a switch network that connects all SP nodes, providing a high-speed data connection. The SP Switch supports the TCP/IP protocol and is available to HACMP and client traffic.
  • Note: When using an SP Switch network, configure an additional network for HACMP. Otherwise, errors occur when synchronizing the cluster.

    For IP Address Takeover via IP Aliases, (the default) the SP Switch css0 base address should be configured as an HACMP base address. The Switch base address cannot be modified. For more information about the SP Switch network, see the section Planning for IPAT with the SP Switch Network in this chapter.

  • SP serial network. Every SP frame includes a serial network that connects all SP nodes to the Control Workstation. This network is used for administrative purposes and is not available for use by HACMP or external client traffic.
  • SP Planning Considerations

    When planning to use multiple networks in your SP cluster, keep in mind the following:

  • The SP Switch is a highly available network. Fault isolation and recovery is automatic.
  • Do not route client traffic over the SP Ethernet. This network is used by the SP for administrative traffic, such as netinstalls, which could be affected by heavy client traffic.
  • Applications that make use of the SP Switch network must be “tolerant” of brief network outages during switch initialization or switch faults. In general, most TCP/IP applications are not affected by switch initialization. There is no impact to HACMP other than it noticing this behavior.
  • Handling SP Network Failure

    Rare failures may result in an SP Switch outage on one or more nodes in the cluster. HACMP distinguishes between two types of network failure:

  • Local. A local network failure occurs when a node can no longer communicate over a particular network, but the network is still in use by other nodes. For example, this may happen if the cable between one node and the SP switch breaks.
  • Global. A global network failure occurs when all nodes lose the ability to communicate over a network (for example, if the SP Switch itself were powered off). In either case, HACMP uses selective fallover to handle the situation.
  • For more information, see the section Selective Fallover Caused by Local Network Failures in Appendix B: Resource Group Behavior During Cluster Events in the Administration Guide.

    Special SP Switch Failure Considerations

    In the event that the SP Switch itself is powered off, the error notification subsystem interprets the failure as an HPS_FAULT9_ERR event followed by an HPS_FAULT6_ERR event (fault service daemon terminated).

    Note that on some versions of the SP Switch, to recover from a major switch failure (power off, for example), you issue Eclock and Estart commands to bring the switch back online. The Eclock command runs rc.switch, which deletes the aliases HACMP needs for switch IP address takeover.

    If you are configuring IPAT via IP Replacement on the SP Switch, create an event script for either the network_up or the network_up_complete event to add back the aliases for css0.

    Planning for the SP Switch Network

    The following sections discuss considerations specific to the SP Switch network, especially regarding IP Address Takeover configuration. Note that the planning depends on whether you are using the SP Switch 1 or the SP Switch 2. The considerations listed in this section apply to both types of switches unless otherwise noted.

    Handling SP Switch Adapter Failure

    You cannot configure a non-service interface for the SP Switch Network if you only have css0. To eliminate this as a cluster single point of failure, use the AIX 5L Error Notification facility to promote an SP Switch adapter failure to a node failure.

    Planning for IPAT with the SP Switch Network

    The SP Switch can make use of IP address aliases with HACMP to permit IP address takeover (IPAT).

    You can configure IPAT on the SP Switch Network two ways. The following definitions are used in this manual to prevent confusion about which method of IPAT is discussed.

  • IPAT via IP Aliases on the SP Switch is defined the same way as it is defined for other aliased networks in HACMP. The css0 base IP label of the SP Switch interface must be used as an HACMP non-service IP label in an IPAT via IP Aliases configuration.
  • IPAT via IP Replacement on the SP Switch is supported without any changes to the interface definitions. The css0 base IP label of the SP Switch interface must not be used when you configure a resource group with IPAT via IP Replacement on the SP Switch.
  • The following figure is a general illustration of IPAT via IP Replacement on the SP Switch. In the figure, node 1 and node 2 are SP nodes. E1 and E2 are administrative ethernet IP addresses. The base1 and base2 labels reflect SP Switch base IP addresses. H1 and H2 are SP Switch alias HACMP service addresses. VG1 and VG2 are volume groups.

    Sample Two-Node Cluster Configuration on an SP with Switch 1 
    
    Note: The HACMP base addresses are not included in the preceding figure. The HACMP base addresses are also aliases; they are not the same as SP switch base IP addresses.

    Planning for IPAT via IP Aliases with the SP Switch

    If you are planning to configure an SP Switch network manually, and want to use IPAT via IP Aliases, note the following considerations:

  • Enable the Address Resolution Protocol (ARP) for the SP Switch network to use IPAT via IP Aliases on an SP Switch network.
  • Configure the base adapter address on the SP Switch network as a non-service IP label in HACMP.
  • Configure the service IP label on a different subnet than the non-service IP label to avoid errors during verification. If you have multiple service IP labels, they should all be on a different subnet than the non-service IP labels.
  • Ensure that HWAT is disabled for all adapters on this network.
  • Enable IP Address takeover via IP Aliases.
  • The SP Switch alias addresses configured for IPAT via IP Aliases can be included in non-concurrent resource groups.
  • For more information about IPAT via IP Aliases, see the section Converting an SP Switch Network to an Aliased Network in Chapter 13: Managing the Cluster Topology in the Administration Guide.

    Verifying Address Resolution Protocol (ARP) Settings for SP Switch

    From HACMP 4.5 on, ARP is enabled for SP Switch networks by default. Once you have installed HACMP, ensure that the network is configured to support gratuitous ARP in HACMP. This procedure is explained in the section on SP Switch Address Resolution Protocol (ARP) in the chapter on Configuring AIX 5L for HACMP in the Installation Guide.

    Configuring the SP Switch2

    SP Switch2 supports configurations with two switches, so that nodes can communicate over one or two switch planes. This provides improved communication performance and higher availability.

    If you have two css interfaces on a node, they must be configured into two separate logical networks. You cannot use one css interface as a service interface and another interface as a non-service on the same network, and fall over between them.

    Note: PSSP allows you to configure an Aggregate IP. This is an ml0 interface with its own IP address. This ml0 interface is not supported by HACMP.

    Avoiding SP Switch 2 Configuration Errors

    If you have two switch adapters, but definitions for only one switch network, then the definition will be taken as applying to css0. A Warning Message reminds the user during the adapter definition process. If you intend to use css0, then no action is required. Otherwise, follow the process described in the preceding example. If you do not follow these guidelines, the interface name is css0 by default.

    If you define multiple adapters without adding the subnet address of the non-service/service then synchronization will fail with the following message:

    ERROR: Multiple switch adapters defined on %s without base subnets. Add 
    base subnets to switch networks. 
    

    Completing the Network Worksheets

    In the following sections you fill out the following network planning worksheets:

  • TCP/IP Networks Worksheet
  • TCP/IP Network Interface Worksheet
  • Point-to-point Network Worksheet
  • Serial Interface Worksheet
  • Completing the TCP/IP Networks Worksheet

    The TCP/IP Networks Worksheet helps you organize the networks for an HACMP cluster. Print the TCP/IP Networks Worksheet from Appendix A: Planning Worksheets, and fill it out using the information in this section. Print as many copies as you need.

    To complete the TCP/IP Networks Worksheet:

      1. Enter a symbolic name for each network in the Network Name field.
    You use this value during the installation process when you configure the cluster. This name can be up to 32 characters long and can include alphanumeric characters and underscores. Do not begin the name with a number. Do not use the HACMP network type alone as a name; however, you can use the type plus a number. For example, use Ether1 instead of Ether.
    The Network Name is a symbolic value that identifies a network in an HACMP environment. Cluster processes use this information to determine which adapters are connected to the same physical network. Use any naming convention you prefer, but be consistent. If several interfaces share the same physical network, make sure that you use the same network name when defining these adapters.
      2. Identify the network type in the Network Type field (for example, Ethernet, Token Ring, and so forth).
    Note: For SP Switch networks, ARP must be enabled, and the network type must be hps.
      3. In the Netmask field, provide the network mask of each network. The network mask is site dependent.
    The HACMP software uses the subnet facility of TCP/IP to divide a single physical network into separate logical subnets. To use subnets, define a network mask for your system.
    An IP address consists of 32 bits. Some of the bits form the network address; the remainder form the host address. The network mask (or netmask) determines which bits in the IP address refer to the network and which bits refer to the host. For example:

    IP Address Example 
    
    In the preceding figure, the netmask is shown in both dotted decimal and binary format. A binary 1 indicates that a bit is part of the network address. A binary 0 indicates that a bit is part of the host address. In the IP Address Example figure, the network portion of the address occupies 24 bits; the host portion occupies 8 bits. Thus, addresses 10.10.10.1 and 10.10.10.2 would be on the same subnet; 10.10.20.1 would be on a different subnet, as it has a different network part in its address. It is convenient (but not necessary) to define a subnet on an octet boundary.
    Subnetting is relevant only for the local site. Remote sites view the network portion of the IP address by the network’s class.
    The HACMP software supplies a SMIT panel where you can add and remove subnets from the network using UNIX standard subnet syntax (for example, 192.9.200.0/24).
    For more information, see the section Managing Communication Interfaces in HACMP in Chapter 13: Managing the Cluster Topology in the Administration Guide.
    See the IBM AIX Communication Concepts and Procedures manual for further information about address classes. Also, ask your network administrator about the class and subnets used at your site.
      4. In the Node Names field, list the names of the nodes connected to each network. Refer to your cluster diagram.
      5. In the IP Address Takeover via IP Aliases field, the default is YES. Disable this option if you are not planning to use this feature.
    For more information, see the section IP Address Takeover via IP Aliases in this chapter.
      6. In the IP Address Offset for Heartbeating over IP Aliases field, enter the beginning address for HACMP to use to create heartbeat alias addresses.
    For more information, see the section Heartbeating over IP Aliases in this chapter.

    Completing the TCP/IP Network Interface Worksheet

    The TCP/IP Network Interface Worksheet helps you define the NICs connected to each node in the cluster. Print the TCP/IP Network Interface Worksheet from Appendix A: Planning Worksheets, and fill it out using the information in this section. Print as copy for each node.

    To complete the TCP/IP Network Interface Worksheet:

      1. Enter the node name in the Node Name field. You assigned this name in Chapter 2: Initial Cluster Planning. For each node, perform the following tasks for each network adapter configured on the node.
      2. Assign each interface an IP label and record it in the IP Label/Address field. This name can be up to 32 characters long and can include alphanumeric characters (but not a leading numeric), hyphens, and underscores.
    If the system hostname is the same as the interface label, do not use underscores since underscores in hostnames may not be allowed with some levels of BIND.
    For more information about IP labels, see the section IP Labels in TCP/IP Networks in this chapter.
      3. Enter the name of the network to which this interface is connected in the Network Interface field.
      4. Identify the interface function as service, non-service, or persistent in the Interface Function field.
      5. Record the IP address for each interface in the IP Address field.
    All service IP addresses for a network can share the same subnet. If your network uses heartbeat via IP aliases, your service and non-service IP addresses can all be on the same subnet, or on separate subnets.
    If your network is not using heartbeat via IP aliases:
  • On each node, every non-service IP address for a given network must be on a separate subnet. You can use the same set of subnets on every node.
  • If you have a resource group with the startup policy Online on Highest Available Node or Online Using Node Distribution Policy, fallover policy Fallover to the Next Priority Node in the List and fallback policy Never Fallback, the service IP address should be on the same subnet as one of the non-service IP addresses.
  • If you have a resource group with the startup policy Online on Home Node, fallover policy Fallover to the Next Priority Node, and fallback policy Fallback to Highest Priority Node, the service IP address must be on a different subnet from all non-service IP addresses.
    1. 6. Enter the NIC Hardware Address.
    If you are using hardware address swapping, enter in the NIC HW Address field the alternate hardware address for each service IP label. The hardware address is a 12 or 14-digit hexadecimal value. Usually, hexadecimal numbers are prefaced with “0x” (zero x) for readability. For readability, colons are usually used to separate the numbers in the hardware address. The colons are not part of the actual address, and should not be entered on the SMIT panel during configuration.
    For more information, see the section Defining Hardware Addresses in this chapter.
    Note: Hardware address swapping is not supported on ATM networks, SP Ethernet, and SP Switch.

    Completing the Point-to-Point Networks Worksheet

    The Point-to-Point Networks Worksheet helps you organize the point-to-point networks for your cluster. Print the Point-to-Point Networks Worksheet from Appendix A: Planning Worksheets, and fill it out using the information in this section.

    To complete the Point-to-Point Networks Worksheet:

      1. Enter the cluster name in the Cluster Name field.
      2. In the Network Name field, assign each network a symbolic name. Names can contain up to 32 characters alphanumeric characters and underscores. Do not begin the name with a number.
    The Network Name is a symbolic value that identifies a network in an HACMP environment. Cluster processes use this information to determine which interfaces are connected to the same physical network. Use any naming convention you prefer, but be consistent. If two interfaces share the same physical network, make sure that you use the same network name when defining these interfaces.
      3. Identify the network type in the Network Type field. For point-to-point networks, you can specify RS232, disk heartbeating (diskhb), Target Mode SCSI (tmscsi), or Target Mode SSA (tmssa).
    The Network Attribute field has a preassigned value.
      4. In the Node Names field, list the names of the nodes connected to each network. Refer to your cluster diagram.
      5. (For diskhb) In the Hdisk field, specify the disk used in the diskhb network.
      6. Use the Miscellaneous Data field to record any extra information about devices used to extend point-to-point links (for example, modem number or extender information).

    Completing the Serial Network Interface Worksheet

    The Serial Network Interface Worksheet in Appendix A: Planning Worksheets lets you define the network interfaces connected to each node in the cluster. Complete the following steps for each node on a separate worksheet

    To complete the Serial Network Interface Worksheet:

      1. Enter the node name in the Node Name field. You assigned this name when filling in the section Completing the TCP/IP Networks Worksheet in this chapter. Record the following information for each serial network configured on the node.
      2. Enter the number of the slot in which the serial interface is located in the Slot Number field.
    You will enter a value in the Interface Name field after you configure the adapter following the instructions in the relevant AIX 5L documentation. AIX 5L assigns an interface name to the interface when it is configured. The interface name is made up of two or three characters that indicate the type of interface, followed by a number that AIX 5L assigns in sequence for each interface of a certain type. For example, AIX 5L assigns an interface name such as tty0 for the first tty.
      3. Assign each interface a symbolic name and record it in the Interface Label field.
    Use a naming convention that helps identify the interface’s role in the cluster. For example, such as nodea-tty1.
      4. In the Network Name field, enter the name you assigned to the network in the Point-to-Point Network Worksheet.
      5. The appropriate values appear in the Interface Function field.

    Defining Hardware Addresses

    Note: You cannot use hardware address swapping if you have IP address takeover via IP aliases configured for the network. You cannot use Hardware Address Takeover on the SP Ethernet or SP Switch networks. Hardware address swapping is not supported on ATM networks.

    The hardware address swapping facility works in tandem with IP address takeover via IP Replacement. Hardware address swapping maintains the binding between an IP address and a hardware address. This eliminates the need to update the ARP cache of clients and network devices after an IP address takeover. This facility is supported for Ethernet, Token Ring, and FDDI adapters.

    Note that hardware address swapping takes about 60 seconds on a Token Ring network and up to 120 seconds on a FDDI network. These periods are longer than the usual time it takes for the Cluster Manager to detect a failure and take action, so you may need to adjust the tuning parameters for these networks. For more information, see the section Setting Failure Detection Parameters in this chapter.

    Selecting an Alternate Hardware Address

    This section provides information about hardware addressing for Ethernet, Token Ring, and FDDI network interface cards. Note that any alternate hardware address you define for a NIC should be similar in form to the default hardware address the manufacturer assigned to the NIC.

    To determine an adapter’s default hardware address, use the netstat -i command (when the networks are active).

    Using netstat

    To retrieve hardware addresses using the netstat -i command, enter:

    netstat -i | grep link 
    

    The command returns output similar to the following:

    	lo0   16896 link#1                          186303     0   186309     0     0 
    	en0   1500  link#2      2.60.8c.2f.bb.93      2925     0     1047     0     0 
    	tr0   1492  link#3      10.0.5a.a8.b5.7b    104544     0    92158     0     0 
    	tr1   1492  link#4      10.0.5a.a8.8d.79     79517     0    39130     0     0 
    	fi0   4352  link#5      10.0.5a.b8.89.4f     40221     0        1     1     0 
    	fi1   4352  link#6      10.0.5a.b8.8b.f4     40338     0        6     1     0 
    

    Using the arp Command

    Use the arp command to view the list of nodes, IP addresses, and associated hardware (MAC) addresses in a host’s ARP cache. For example:

    arp -a 
    flounder (100.50.81.133) at 8:0:4c:0:12:34 [ethernet] 
    cod (100.50.81.195) at 8:0:5a:7a:2c:85 [ethernet] 
    seahorse (100.50.161.6) at 42:c:2:4:0:0 [token ring] 
    pollock (100.50.81.147) at 10:0:5a:5c:36:b9 [ethernet] 
    

    This output shows what the host node currently believes to be the IP and MAC addresses for nodes flounder, cod, seahorse and pollock. (If IP address takeover occurs without Hardware Address Takeover (HWAT), the MAC address associated with the IP address in the host’s ARP cache may become outdated. You can correct this situation by refreshing the host’s ARP cache.)

    For more information, see the arp man page.

    Specifying an Alternate Ethernet Hardware Address

    To specify an alternate hardware address for an Ethernet interface, begin by using the first five pairs of alphanumeric characters as they appear in the current hardware address. Then substitute a different value for the last pair of characters. Use characters that do not occur on any other adapter on the physical network.

    For example, you could use 10 and 20 for node A and node B, respectively. If you have multiple adapters for hardware address swapping in each node, you can extend to 11 and 12 on node A, and 21 and 22 on node B.

    Specifying an alternate hardware address for adapter interface en0 in the output above thus yields the following value:

    Original address
    02608c2fbb93
    New address
    02608c2fbb10

    To define this alternate hardware address to the cluster environment, see the section Configuring Service IP Labels/Addresses in Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended) in the Administration Guide.

    Specifying an Alternate Token Ring Hardware Address

    To specify an alternate hardware address for a Token Ring interface, set the first two digits to 42, indicating that the address is set locally.

    Specifying an alternate hardware address for adapter interface tr0 in the output above thus yields the following value:

    Original address
    10005aa8b57b
    New address
    42005aa8b57b

    To define this alternate hardware address to the cluster environment, see the section Configuring Service IP Labels/Addresses in Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended) in the Administration Guide.

    Specifying an Alternate FDDI Hardware Address (for Service Labels)

    To specify an alternate FDDI hardware address, enter the new address into the Adapter Hardware Address field as follows, without any decimal separators:

      1. Use 4, 5, 6, or 7 as the first digit (the first nibble of the first byte) of the new address.
      2. Use the last 6 octets of the manufacturer’s default address as the last 6 digits of the new address.

    Here is a list of some sample valid addresses, shown with decimals for readability:

    40.00.00.b8.10.89 
    40.00.01.b8.10.89 
    50.00.00.b8.10.89 
    60.00.00.b8.10.89 
    7f.ff.ff.b8.10.89 
    

    Avoiding Network Conflicts

    Avoid network conflicts on the following:

  • Hardware Address. Each network adapter is assigned a unique hardware address when it is manufactured, which ensures that no two adapters have the same network address. When defining a hardware address for a network adapter, ensure that the defined address does not conflict with the hardware address of another network adapter in the network. Two network adapters with the same hardware address can cause unpredictable behavior within the network.
  • IP Addresses. Verification will notify you if there are duplicate IP addresses. Correct the duplicate address and resynchronize the cluster.
  • Adding the Network Topology to the Cluster Diagram

    You can now add the network topology to your cluster diagram.

    Sketch the networks to include all TCP/IP networks (including any TCP/IP point-to-point connections) and non-IP point-to-point networks. Identify each network by name and attribute. In the boxes in each node that represent slots, record the interface label.

    You can now add networking to the sample cluster diagram started in Chapter 1: Overview of the Planning Process.

    A sample network set up may include the use of five networks:

  • A Token Ring network, named clus1_TR, used to connect clients to the ten cluster nodes that run the customer database “front end” application. The Token Ring network allows client access.
  • An Ethernet network, named db_net, used to connect the four cluster nodes that run the database application, which handles the update of the actual database records. The Ethernet network db_net is not intended for client use.
  • The SP Ethernet is configured as a service network, named sp_ether. It is not available for client access.
  • The SP Switch network is configured as a service network and is also used for high speed data transfer between the front end nodes and back end nodes. For example, you may name the SP Switch network clus1_HPS.
  • To avoid corruption of critical database storage, the db_net nodes are also connected by a series of point-to-point networks. These networks provide additional protection against contention situations where IP heart beat packets are discarded.
  • The RSCT software monitors the status of all defined interfaces by sending heartbeats across the network.

    Where You Go from Here

    You have now planned the network topology for the cluster. The next step in the planning process is to lay out the shared disk configuration for your cluster, described in Chapter 4: Planning Shared Disk and Tape Devices.


    PreviousNextIndex