![]() ![]() ![]() |
Chapter 1: HACMP for AIX
This chapter discusses the concepts of high availability and cluster multi-processing, presents the HACMP cluster diagram, and describes an HACMP cluster from a functional perspective.
High Availability Cluster Multi-Processing for AIX
The IBM HACMP software provides a low-cost commercial computing environment that ensures quick recovery of mission-critical applications from hardware and software failures. HACMP has two major components: high availability (HA) and cluster multi-processing (CMP).
With HACMP software, critical resources remain available. For example, an HACMP cluster could run a database server program that services client applications. The clients send queries to the server program that responds to their requests by accessing a database, stored on a shared external disk.
This high availability system combines custom software with industry-standard hardware to minimize downtime by quickly restoring services when a system, component, or application fails. Although not instantaneous, the restoration of service is rapid, usually within 30 to 300 seconds.
In an HACMP cluster, to ensure the availability of these applications, the applications are put under HACMP control. HACMP takes measures to ensure that the applications remain available to client processes even if a component in a cluster fails. To ensure availability, in case of a component failure, HACMP moves the application (along with resources that ensure access to the application) to another node in the cluster.
HACMP and HACMP/ES
HACMP 5.4 includes all the features of the product that was previously referred to as HACMP/ES (Enhanced Scalability). The product previously referred to as HACMP (Classic) is no longer available.
Note: Prior to version 5.1, the HACMP for AIX software included four features: HAS and CRM with core filesets named cluster.base*; and ES and ESCRM with core filesets named cluster.es*. Starting with HACMP 5.1, the HAS, CRM and ES features were no longer available, and the ESCRM feature is now called HACMP.
To summarize, HACMP 5.4 has the following characteristics:
Includes all the features of ESCRM 4.5 in addition to the new functionality added in HACMP version 5.1 through 5.4. Takes advantage of the enhanced Reliable Scalable Cluster Technology (RSCT). RSCT provides facilities for monitoring node membership; network interface and communication interface health; and event notification, synchronization and coordination via reliable messaging. Can have up to 32 nodes on HACMP clusters with both non-concurrent and concurrent access. Is supported on AIX 5L versions 5.2 and 5.3. High Availability and Hardware Availability
High availability is sometimes confused with simple hardware availability. Fault tolerant, redundant systems (such as RAID) and dynamic switching technologies (such as DLPAR) provide recovery of certain hardware failures, but do not provide the full scope of error detection and recovery required to keep a complex application highly available.
A modern, complex application requires access to all of these components:
Nodes (CPU, memory) Network interfaces (including external devices in the network topology) Disk or storage devices. Recent surveys of the causes of downtime show that actual hardware failures account for only a small percentage of unplanned outages. Other contributing factors include:
Operator errors Environmental problems Application and operating system errors. Reliable and recoverable hardware simply cannot protect against failures of all these different aspects of the configuration. Keeping these varied elements—and therefore the application— highly available requires:
Thorough and complete planning of the physical and logical procedures for access and operation of the resources on which the application depends. These procedures help to avoid failures in the first place. A monitoring and recovery package that automates the detection and recovery from errors. A well-controlled process for maintaining the hardware and software aspects of the cluster configuration while keeping the application available. High Availability vs. Fault Tolerance
Fault tolerance relies on specialized hardware to detect a hardware fault and instantaneously switch to a redundant hardware component—whether the failed component is a processor, memory board, power supply, I/O subsystem, or storage subsystem. Although this cutover is apparently seamless and offers non-stop service, a high premium is paid in both hardware cost and performance because the redundant components do no processing. More importantly, the fault tolerant model does not address software failures, by far the most common reason for downtime.
High availability views availability not as a series of replicated physical components, but rather as a set of system-wide, shared resources that cooperate to guarantee essential services. High availability combines software with industry-standard hardware to minimize downtime by quickly restoring essential services when a system, component, or application fails. While not instantaneous, services are restored rapidly, often in less than a minute.
The difference between fault tolerance and high availability, then, is this: A fault tolerant environment has no service interruption but a significantly higher cost, while a highly available environment has a minimal service interruption. Many sites are willing to absorb a small amount of downtime with high availability rather than pay the much higher cost of providing fault tolerance. Additionally, in most highly available configurations, the backup processors are available for use during normal operation.
High availability systems are an excellent solution for applications that must be restored quickly and can withstand a short interruption should a failure occur. Some industries have applications so time-critical that they cannot withstand even a few seconds of downtime. Many other industries, however, can withstand small periods of time when their database is unavailable. For those industries, HACMP can provide the necessary continuity of service without total redundancy.
Role of HACMP
HACMP helps you with the following:
The HACMP planning process and documentation include tips and advice on the best practices for installing and maintaining a highly available HACMP cluster. Once the cluster is operational, HACMP provides the automated monitoring and recovery for all the resources on which the application depends. HACMP provides a full set of tools for maintaining the cluster while keeping the application available to clients. HACMP lets you:
Quickly and easily set up a basic two-node HACMP cluster by using the Two-Node Cluster Configuration Assistant. Set up an HACMP environment using online planning worksheets to simplify the initial planning and setup. Test your HACMP configuration by using the Cluster Test Tool. You can evaluate how a cluster behaves under a set of specified circumstances, such as when a node becomes inaccessible, a network becomes inaccessible, and so forth. Ensure high availability of applications by eliminating single points of failure in an HACMP environment. Leverage high availability features available in AIX 5L. Manage how a cluster handles component failures. Secure cluster communications. Set up fast disk takeover for volume groups managed by the Logical Volume Manager (LVM). Monitor HACMP components and diagnose problems that may occur. For a general overview of all HACMP features, see the IBM website:
http://www.ibm.com/servers/aix/products/ibmsw/high_avail_network/hacmp.html
For a list of new features in HACMP 5.4, see Chapter 8: HACMP 5.4: Summary of Changes.
Cluster Multi-Processing
Cluster multi-processing is a group of loosely coupled machines networked together, sharing disk resources. In a cluster, multiple server machines cooperate to provide a set of services or resources to clients.
Clustering two or more servers to back up critical applications is a cost-effective high availability option. You can use more of your site’s computing power while ensuring that critical applications resume operations after a minimal interruption caused by a hardware or software failure.
Cluster multi-processing also provides a gradual, scalable growth path. It is easy to add a processor to the cluster to share the growing workload. You can also upgrade one or more of the processors in the cluster to a more powerful model. If you were using a fault tolerant strategy, you must add two processors, one as a redundant backup that does no processing during normal operations.
The Availability Costs and Benefits Continuum
The following figure shows the costs and benefits of availability technologies.
As you can see, availability is not an all-or-nothing proposition. Think of availability as a continuum. Reliable hardware and software provide the base level of availability. Advanced features such as RAID devices provide an enhanced level of availability. High availability software provides near continuous access to data and applications. Fault tolerant systems ensure the constant availability of the entire system, but at a higher cost.
Enhancing Availability with the AIX 5L Software
HACMP takes advantage of the features in AIX 5L—the high-performance UNIX operating system.
AIX 5L adds new functionality to further improve security and system availability. This includes improved availability of mirrored data and enhancements to Workload Manager that help solve problems of mixed workloads by dynamically providing resource availability to critical applications. Used with the IBM eServer pSeries, HACMP can provide both horizontal and vertical scalability without downtime.
The AIX 5L operating system provides numerous features designed to increase system availability by lessening the impact of both planned (data backup, system administration) and unplanned (hardware or software failure) downtime. These features include:
Journaled File System and Enhanced Journaled File System Disk mirroring Process control Error notification. Journaled File System and Enhanced Journaled File System
The AIX 5L native filesystem, the Journaled File System (JFS), uses database journaling techniques to maintain its structural integrity. System or software failures do not leave the filesystem in an unmanageable condition. When rebuilding the filesystem after a major failure, AIX 5L uses the JFS log to restore the filesystem to its last consistent state. Journaling thus provides faster recovery than the standard UNIX filesystem consistency check (fsck) utility.
In addition, the Enhanced Journaled File System (JFS2) is available in AIX 5L. For more information, see the section Journaled File System and Enhanced Journaled File System in Chapter 3: HACMP Resources and Resource Groups.
Disk Mirroring
Disk mirroring software provides data integrity and online backup capability. It prevents data loss due to disk failure by maintaining up to three copies of data on separate disks so that data is still accessible after any single disk fails. Disk mirroring is transparent to the application. No application modification is necessary because mirrored and conventional disks appear the same to the application.
Process Control
The AIX 5L System Resource Controller (SRC) monitors and controls key processes. The SRC can detect when a process terminates abnormally, log the termination, pass messages to a notification program, and restart the process on a backup processor.
Error Notification
The AIX 5L Error Notification facility detects errors, such as network and disk adapter failures, and triggers a predefined response to the failure.
HACMP builds on this AIX 5L feature by providing:
Automatically created error notification methods for volume groups that you configure in HACMP. These error notification methods let HACMP react to certain volume group failures and provide recovery. HACMP also configures automatic error notification methods for those volume groups that are configured in AIX 5L (and are not configured in HACMP) but that contain the corresponding filesystems configured in HACMP. This ensures that the filesystems are kept highly available and allows HACMP to automatically respond to the volume group failures by recovering the filesystems.
For more information on error notification and how it is used in HACMP, see Eliminating Disks and Disk Adapters as a Single Point of Failure in Chapter 5: Ensuring Application Availability.
Error emulation function. It allows you to test the predefined response without causing the error to occur. You also have an option to automatically configure notification methods for a set of device errors in one step. Physical Components of an HACMP Cluster
HACMP provides a highly available environment by identifying a set of resources essential to uninterrupted processing. It also defines a protocol that nodes use to collaborate to ensure that these resources are available. HACMP extends the clustering model by defining relationships among cooperating processors where one processor provides the service offered by a peer should the peer be unable to do so.
As shown in the following figure, an HACMP cluster is made up of the following physical components:
Nodes Shared external disk devices Networks Network interfaces Clients. The HACMP software allows you to combine physical components into a wide range of cluster configurations, providing you with flexibility in building a cluster that meets your processing requirements.
This figure shows one example of an HACMP cluster. Other HACMP clusters could look very different—depending on the number of processors, the choice of networking and disk technologies, and so on.
Chapter 6: HACMP Cluster Configurations provides examples of cluster configurations supported by the HACMP software.
Nodes
Nodes form the core of an HACMP cluster. A node is a processor that runs AIX 5L, the HACMP software, and the application software.
In an HACMP cluster, up to 32 pSeries servers divided into LPARS, RS/6000 or pSeries standalone systems, systems that run Parallel System Support Program (PSSP), or a combination of these cooperate to provide a set of services or resources to other entities. Clustering these servers to back up critical applications is a cost-effective high availability option. A business can use more of its computing power while ensuring that its critical applications resume running after a short interruption caused by a hardware or software failure.
In an HACMP cluster, each node is identified by a unique name. A node may own a set of resources—disks, volume groups, filesystems, networks, network addresses, and applications. Cluster resources are discussed in detail in Chapter 3: HACMP Resources and Resource Groups. Typically, a node runs a server or a back end application that accesses data on the shared external disks. Applications are discussed in Chapter 5: Ensuring Application Availability.
A node in an HACMP cluster has several layers of software components. For the detailed description of these components, see the section Software Components of an HACMP Node in Chapter 4: HACMP Cluster Hardware and Software.
Shared External Disk Devices
Each node has access to one or more shared external disk devices. A shared external disk device is a disk physically connected to multiple nodes. The shared disk stores mission-critical data, typically mirrored or RAID-configured for data redundancy. A node in an HACMP cluster must also have internal disks that store the operating system and application binaries, but these disks are not shared.
Depending on the type of disk used, the HACMP software supports the following types of access to shared external disk devices—non-concurrent access and concurrent access.
In non-concurrent access environments, only one connection is active at any given time, and the node with the active connection owns the disk. When a node fails, the node that currently owns the disk leaves the cluster, disk takeover occurs and a surviving node assumes ownership of the shared disk. In concurrent access environments, the shared disks are actively connected to more than one node simultaneously. Therefore, when a node fails, disk takeover is not required. Note that in such environments, either all nodes defined in the cluster can be part of the concurrent access, or only a subset of cluster nodes can have access to shared disks. In the second case, you configure resource groups only on those nodes that have shared disk access.
The differences between these methods are explained more fully in the section Shared External Disk Access in Chapter 4: HACMP Cluster Hardware and Software.
Networks
As an independent, layered component of AIX 5L, the HACMP software is designed to work with any TCP/IP-based network. Nodes in an HACMP cluster use the network to:
Allow clients to access the cluster nodes Enable cluster nodes to exchange heartbeat messages Serialize access to data (in concurrent access environments). The HACMP software has been tested with Ethernet, Token-Ring, ATM, and other networks.
Types of Networks
The HACMP software defines two types of communication networks, characterized by whether these networks use communication interfaces based on the TCP/IP subsystem (TCP/IP-based) or communication devices based on non-TCP/IP subsystems (device-based).
TCP/IP-based network. Connects two or more server nodes, and optionally allows client access to these cluster nodes, using the TCP/IP protocol. Ethernet, Token-Ring, ATM, HP Switch and SP Switch networks are defined as TCP/IP-based networks. Device-based network. Provides a point-to-point connection between two cluster nodes for HACMP control messages and heartbeat traffic. Device-based networks do not use the TCP/IP protocol and, therefore, continue to provide communications between nodes in the event the TCP/IP subsystem on a server node fails. Target mode SCSI devices, Target Mode SSA devices, disk heartbeat devices, or RS232 point-to-point devices are defined as device-based networks.
Clients
A client is a processor that can access the nodes in a cluster over a local area network. Clients each run a “front end” or client application that queries the server application running on the cluster node.
The HACMP software provides a highly available environment for critical data and applications on cluster nodes. The HACMP software does not make the clients themselves highly available. AIX 5L clients can use the Cluster Information (Clinfo) services to receive notice of cluster events. Clinfo provides an API that displays cluster status information.
HACMP provides a cluster status utility, the /usr/es/sbin/cluster/clstat. It is based on Clinfo and reports the status of key cluster components—the cluster itself, the nodes in the cluster, the network interfaces connected to the nodes, and the resource groups on each node. The cluster status interface of the clstat utility includes web-based, Motif-based and ASCII-based versions.
See Cluster Information Program in Chapter 4: HACMP Cluster Hardware and Software, for more information about how Clinfo obtains cluster status information.
Goal of HACMP: Eliminating Scheduled Downtime
The primary goal of high availability clustering software is to minimize, or ideally, eliminate, the need to take your resources out of service during maintenance and reconfiguration activities.
HACMP software optimizes availability by allowing for the dynamic reconfiguration of running clusters. Most routine cluster maintenance tasks, such as adding or removing a node or changing the priority of nodes participating in a resource group, can be applied to an active cluster without stopping and restarting cluster services.
In addition, you can keep an HACMP cluster online while making configuration changes by using the Cluster Single Point of Control (C-SPOC) facility. C-SPOC makes cluster management easier, as it allows you to make changes to shared volume groups, users, and groups across the cluster from a single node. The changes are propagated transparently to other cluster nodes.
![]() ![]() ![]() |