![]() ![]() ![]() |
Chapter 6: HACMP Cluster Configurations
This chapter provides examples of the types of cluster configurations supported by the HACMP software.
Sample Cluster Configurations
The following sample cluster configurations are discussed in this chapter
Standby Configurations—These are the traditional redundant hardware configurations where one or more standby nodes stand idle, waiting for a server node to leave the cluster. Takeover Configurations—In these configuration, all cluster nodes do useful work, processing part of the cluster’s workload. There are no standby nodes. Takeover configurations use hardware resources more efficiently than standby configurations since there is no idle processor. Performance can degrade after node detachment, however, since the load on remaining nodes increases. Takeover configurations that use concurrent access use hardware efficiently and also minimize service interruption during fallover because there is no need for the takeover node to acquire the resources released by the failed node—the takeover node already shares ownership of the resources.
Cluster Configurations with Multi-Tiered Applications—In these configurations, one application depends on another application. These cluster configurations use dependent resource groups. Cluster Configurations with Resource Group Location Dependencies—In these configurations, related applications are configured to always stay on the same node or to always stay on a different node from other applications. These cluster configurations use location dependent resource groups. Cross-Site LVM Mirror Configurations for Disaster Recovery—In these geographically dispersed configurations, LVM mirroring replicates data between the disk subsystems at each of two sites for disaster recovery. Cluster Configurations with Dynamic LPARs—In these configurations, HACMP clusters use LPARs as cluster nodes. This lets you perform routine system upgrades through the dynamic allocation of system resources and redistribute CPU and memory resources to manage the application workload. This list is by no means an exhaustive catalog of the possible configurations you can define using the HACMP software. Rather, use them as a starting point for thinking about the cluster configuration best suited to your environment.
Standby Configurations
The standby configuration is a traditional redundant hardware configuration, where one or more standby nodes stand idle, waiting for a server node to leave the cluster.
The sample standby configurations discussed in this chapter show how the configuration is defined for these types of resource groups:
Standby Configurations: Example 1 shows resource groups with the Online on Home Node Only startup policy, Fallover to Next Priority Node in the List fallover policy and Fallback to Higher Priority Node in the List fallback policy. Standby Configurations: Example 2 shows resource groups with the startup Online Using Distribution Policy (network or node), fallover policy Next Priority Node in the List, and fallback policy Never Fallback. Concurrent resource groups require all nodes to have simultaneous access to the resource group and cannot be used in a standby configuration.
Standby Configurations: Example 1
The following figure shows a two-node standby configuration that uses resource groups with these policies:
Startup policy: Online on Home Node Only Fallover policy: Fallover to Next Priority Node in the List Fallback policy: Fallback to Higher Priority Node in the List. In the figure, a lower number indicates a higher priority:
In this setup, the cluster resources are defined as part of a single resource group. A nodelist is then defined as consisting of two nodes. The first node, Node A, is assigned a takeover (ownership) priority of 1. The second node, Node B, is assigned a takeover priority of 2.
At cluster startup, Node A (which has a priority of 1) assumes ownership of the resource group. Node A is the “server” node. Node B (which has a priority of 2) stands idle, ready should Node A fail or leave the cluster. Node B is, in effect, the “standby”.
If the server node leaves the cluster, the standby node assumes control of the resource groups owned by the server, starts the highly available applications, and services clients. The standby node remains active until the node with the higher takeover priority rejoins the cluster. At that point, the standby node releases the resource groups it has taken over, and the server node reclaims them. The standby node then returns to an idle state.
Extending Standby Configurations from Example 1
The standby configuration from the previously described example can be easily extended to larger clusters. The advantage of this configuration is that it makes better use of the hardware. The disadvantage is that the cluster can suffer severe performance degradation if more than one server node leaves the cluster.
The following figure illustrates a three-node standby configuration using the resource groups with these policies:
Startup policy: Online on Home Node Only Fallover policy: Fallover to Next Priority Node in the List Fallback policy: Fallback to Higher Priority Node in the List.
In this configuration, two separate resource groups (A and B) and a separate nodelist for each resource group exist. The nodelist for Resource Group A consists of Node A and Node C. Node A has a takeover priority of 1, while Node C has a takeover priority of 2. The nodelist for Resource Group B consists of Node B and Node C. Node B has a takeover priority of 1; Node C again has a takeover priority of 2. (A resource group can be owned by only a single node in a non-concurrent configuration.)
Since each resource group has a different node at the head of its nodelist, the cluster's workload is divided, or partitioned, between these two resource groups. Both resource groups, however, have the same node as the standby in their nodelists. If either server node leaves the cluster, the standby node assumes control of that server node’s resource group and functions as the departed node.
In this example, the standby node has three network interfaces (not shown) and separate physical connections to each server node’s external disk. Therefore, the standby node can, if necessary, take over for both server nodes concurrently. The cluster’s performance, however, would most likely degrade while the standby node was functioning as both server nodes.
Standby Configurations: Example 2
In the following standby configuration, the resource groups have these policies:
Startup policy: Online Using Distribution Policy (network-based or node-based) Fallover policy: Next Priority Node in the List Fallback policy: Never Fallback. This configuration differs from a standby configuration in which the ownership priority of resource groups is not fixed. Rather, the resource group is associated with an IP address that can rotate among nodes. This makes the roles of server and standby fluid, changing over time.
The following figure illustrates the one-for-one standby configuration that is described in this section:
At system startup, the resource group attaches to the node that claims the shared IP address. This node “owns” the resource group for as long as it remains in the cluster. If this node leaves the cluster, the peer node assumes the shared IP address and claims ownership of that resource group. Now, the peer node “owns” the resource group for as long as it remains in the cluster.
When the node that initially claimed the resource group rejoins the cluster, it does not take the resource group back. Rather, it remains idle for as long as the node currently bound to the shared IP address is active in the cluster. Only if the peer node leaves the cluster does the node that initially “owned” the resource group claim it once again. Thus, ownership of resources rotates between nodes.
Extending Standby Configurations From Example 2
As with the first example of the standby configuration, configurations from Example 2 can be easily extended to larger clusters. For example, in a one-for-two standby configuration from Example 2, the cluster could have two separate resource groups, each of which includes a distinct shared IP address.
At cluster startup, the first two nodes each claim a shared IP address and assume ownership of the resource group associated with that shared IP address. The third node remains idle. If an active node leaves the cluster, the idle node claims that shared IP address and takes control of that resource group.
Takeover Configurations
All nodes in a takeover configuration process part of the cluster’s workload. There are no standby nodes. Takeover configurations use hardware resources more efficiently than standby configurations since there is no idle processor. Performance degrades after node detachment, however, since the load on remaining nodes increases.
One-Sided Takeover
The following figure illustrates a two-node, one-sided takeover configuration. In the figure, a lower number indicates a higher priority.
This configuration has two nodes actively processing work, but only one node providing highly available services to cluster clients. That is, although there are two sets of resources within the cluster (for example, two server applications that handle client requests), only one set of resources needs to be highly available. This set of resources is defined as an HACMP resource group and has a nodelist that includes both nodes. The second set of resources is not defined as a resource group and, therefore, is not highly available.
At cluster startup, Node A (which has a priority of 1) assumes ownership of Resource Group A. Node A, in effect, “owns” Resource Group A. Node B (which has a priority of 2 for Resource Group A) processes its own workload independently of this resource group.
If Node A leaves the cluster, Node B takes control of the shared resources. When Node A rejoins the cluster, Node B releases the shared resources.
If Node B leaves the cluster, however, Node A does not take over any of its resources, since Node B’s resources are not defined as part of a highly available resource group in whose chain this node participates.
This configuration is appropriate when a single node is able to run all the critical applications that need to be highly available to cluster clients.
Mutual Takeover
The mutual takeover for non-concurrent access configuration has multiple nodes, each of which provides distinct highly available services to cluster clients. For example, each node might run its own instance of a database and access its own disk.
Furthermore, each node has takeover capacity. If a node leaves the cluster, a surviving node takes over the resource groups owned by the departed node.
The mutual takeover for non-concurrent access configuration is appropriate when each node in the cluster is running critical applications that need to be highly available and when each processor is able to handle the load of more than one node.
The following figure illustrates a two-node mutual takeover configuration for non-concurrent access. In the figure, a lower number indicates a higher priority.
The key feature of this configuration is that the cluster’s workload is divided, or partitioned, between the nodes. Two resource groups exist, in addition to a separate resource chain for each resource group. The nodes that participate in the resource chains are the same. It is the differing priorities within the chains that designate this configuration as mutual takeover.
The chains for both resource groups consist of Node A and Node B. For Resource Group A, Node A has a takeover priority of 1 and Node B has a takeover priority of 2. For Resource Group B, the takeover priorities are reversed. Here, Node B has a takeover priority of 1 and Node A has a takeover priority of 2.
At cluster startup, Node A assumes ownership of the Resource Group A, while Node B assumes ownership of Resource Group B.
If either node leaves the cluster, its peer node takes control of the departed node’s resource group. When the “owner” node for that resource group rejoins the cluster, the takeover node relinquishes the associated resources; they are reacquired by the higher-priority, reintegrating node.
Two-Node Mutual Takeover Configuration for Concurrent Access
The following figure illustrates a two-node mutual takeover configuration for concurrent access:
In this configuration, both nodes have simultaneous access to the shared disks and own the same disk resources. There is no “takeover” of shared disks if a node leaves the cluster, since the peer node already has the shared volume group varied on.
In this example, both nodes are running an instance of a server application that accesses the database on the shared disk. The application’s proprietary locking model is used to arbitrate application requests for disk resources.
Running multiple instances of the same server application allows the cluster to distribute the processing load. As the load increases, additional nodes can be added to further distribute the load.
Eight-Node Mutual Takeover Configuration for Concurrent Access
The following figure illustrates an eight-node mutual takeover configuration for concurrent access:
In this configuration, as in the previous configuration, all nodes have simultaneous—but not concurrent—access to the shared disks and own the same disk resources. Here, however, each node is running a different server application. Clients query a specific application at a specific IP address. Therefore, each application server and its associated IP address must be defined as part of a non-concurrent resource group, and all nodes that are potential owners of that resource group must be included in a corresponding nodelist.
Concurrent access resource groups are supported in clusters with up to 32 nodes in HACMP.
Cluster Configurations with Multi-Tiered Applications
A typical cluster configuration that could utilize parent/child dependent resource groups is the environment in which an application such as WebSphere depends on another application such as DB2.
Note: It is important to distinguish the application server, such as WebSphere, from the HACMP application server that you configure in HACMP by specifying the application server start and stop scripts.
In order to satisfy business requirements, a cluster-wide parent/child dependency must be defined between two or more resource groups. The following figure illustrates the business scenario that utilizes dependencies between applications:
Multi-Tiered Applications
Business configurations that use layered, or multi-tiered applications can also utilize dependent resource groups. For example, the back end database must be online before the application server. In this case, if the database goes down and is moved to a different node, the resource group containing the application server would have to be brought down and back up on any node in the cluster.
Environments such as SAP require applications to be cycled (stopped and restarted) anytime a database fails. An environment like SAP provides many application services, and the individual application components often need to be controlled in a specific order.
Another area where establishing interdependencies between resource groups proves useful is when system services are required to support application environments. Services such as cron jobs for pruning log files or initiating backups need to move from node to node along with an application, but are typically not initiated until the application is established. These services can be built into application server start and stop scripts. When greater granularity is needed, they can be controlled through pre- and post- event processing. Parent/child dependent resource groups allow an easier way to configure system services to be dependent upon applications they serve.
For an overview of dependent resource groups, see the section Resource Group Dependencies in Chapter 3: HACMP Resources and Resource Groups.
Cluster Configurations with Resource Group Location Dependencies
With HACMP 5.4, you can configure the cluster so that certain applications stay on the same node, on the same site, or on different nodes not only at startup, but during fallover and fallback events. To do this, you configure the selected resource groups as part of a location dependency set.
Publishing Model with Same Node and Different Nodes Dependencies
Consider this example: The XYZ Publishing company follows a business continuity model that involves separating the different platforms used to develop the web content. XYZ uses location dependency policies to keep some resource groups strictly on separate nodes and others together on the same node.
The Production database (PDB) and Production application (PApp) are hosted on the same node to facilitate maintenance (and perhaps the highest priority node for these resource groups has the most memory or faster processor). It also makes sense to set up a parent/child relation between them, since the application depends on the database. The database must be online for the application to function.The same conditions are true for the System Database (SDB) and the System application (Sapp) and for the QA Database (QADB) and the QA application (QAapp).
Since keeping the production database and application running is the highest priority, it makes sense to configure the cluster so that the three database resource groups stay on different nodes (make them an Online On Different Nodes dependency set), and assign the PDB resource group with the high priority. The SDB is the Intermediate priority and the QADB is the low priority.
The databases and their related applications are each configured to belong to an Online On Same Node dependency set.
HACMP handles these groups somewhat differently depending on how you configure startup, fallover, and fallback policies. It makes sense to have the participating nodelists differ for each database and application set to facilitate keeping these resource groups on the preferred nodes.
The figure below shows the basic configuration of the three nodes and six resource groups.
Resource Group Policies
For the sake of illustration of this case, all six resource groups might have the following behavioral policies:
Startup Policy: Online On First Available Node Fallover Policy: Fallover to Next Priority Node Fallback Policy: Never Fallback
Participating Nodes Location Dependency Parent/Child Dependency Online On The Same Node Dependent Groups:Online On Different Nodes Dependent set:
[PDB SDB QADB]Priority: PDB > SDB > QADBSee Appendix B in the Administration Guide for more examples of location dependencies and use cases.
Cross-Site LVM Mirror Configurations for Disaster Recovery
In HACMP 5.2 and up, you can set up disks located at two different sites for remote LVM mirroring, using a Storage Area Network (SAN). A SAN is a high-speed network that allows the establishment of direct connections between storage devices and processors (servers) within the distance supported by Fibre Channel. Thus, two or more distantly separated servers (nodes) located at different sites can access the same physical disks, which may be distantly separated as well, via the common SAN. These remote disks can be combined into volume groups, using C-SPOC.
The logical volumes in a volume group can have up to three mirrors or copies, for example, one mirror at each site. Thus the information stored on this logical volume may be kept highly available, and in case of a certain failures—for example, all nodes at one site, including the disk subsystem at that site—the remote mirror at another site will still have the latest information and the operations can be continued on that site.
The primary intent of this feature is to support two-site clusters where LVM mirroring through a SAN replicates data between the disk subsystem at each site for disaster recovery.
Another advantage of cross-site LVM mirroring is that after a site/disk failure and subsequent site reintegration, HACMP attempts to synchronize the data from the surviving disks to the joining disks automatically. The synchronization occurs in the background and does not significantly impact the reintegration time.
The following figure illustrates a cross-site LVM mirroring configuration using a SAN:
The disks that are connected to at least one node at each of the two sites can be mirrored. In this example, PV4 is seen by nodes A and B on Site 1 via the Fibre Channel Switch 1-Fibre Channel Switch 2 connection, and is also seen on node C via Fibre Channel Switch 2. You could have a mirror of PV4 on Site 1. The disks that are connected to the nodes on one site only (PV5 and PV6) cannot be mirrored across sites.
The disk information is replicated from a local site to a remote site. The speed of this data transfer depends on the physical characteristics of the channel, the distance, and LVM mirroring performance.
Cluster Configurations with Dynamic LPARs
The advanced partitioning features of AIX 5L v. 5.2 and up provide the ability to dynamically allocate system CPU, memory, and I/O slot resources (dynamic LPAR).
Using HACMP in combination with LPARs lets you:
Perform routine system upgrades through the dynamic allocation of system resources. When used with dynamic LPARs, HACMP can reduce the amount of downtime for well-planned systems upgrades by automating the transition of your application workload from one logical partition to another, so that the first logical partition may be upgraded without risk to the application. Effectively redistribute CPU and memory resources to manage the workload. Combining HACMP with dynamic LPAR lets you use customized application start and stop scripts to dynamically redistribute CPU and memory resources to logical partitions that are currently executing application workload, to further support application transition within a single frame. This way you maintain the processing power and resources necessary to support your applications, while minimal resources are devoted to upgrading, a less resource intensive task. Note: Do not have all your cluster nodes configured as LPARs within the same physical server. This configuration could potentially be a significant single point of failure.
The following example illustrates a cluster configuration that uses three LPARs:
LPAR #1 is running a back end database (DB2 UDB) LPAR #2 is running WebSphere Application Server (WAS) LPAR #3 is running as a backup (standby) for both the DB2 and WAS LPARs. This LPAR contains only minimal CPU and memory resources. When it is time to move either the DB2 or WAS application to the third LPAR (due to a planned upgrade or a resource failure in these LPARs, for instance), you can use customized application start and stop scripts in HACMP to automate the dynamic reallocation of CPU and memory from the primary LPAR to the standby LPAR. This operation allows the third LPAR to acquire the CPU and memory resources necessary to meet business performance requirements. When HACMP moves the resource group containing the application back to its home LPAR, the CPU and memory resources automatically move with it.
Note: In general, dynamic LPARs allow dynamic allocation of CPU, memory and I/O slot resources. HACMP and dynamic LPAR I/O slot resources are not compatible (although you can dynamically allocate I/O slot resources outside of HACMP cluster).
The following figure illustrates this cluster environment:
DLPARs and Capacity Upgrade on Demand
Capacity Upgrade on Demand (CUoD) is one of the features of Dynamic Logical Partitioning (DLPAR) on some of the pSeries IBM servers that lets you activate preinstalled but yet inactive processors as resource requirements change. The additional CPUs and memory, while physically present, are not used until you decide that the additional capacity you need is worth the cost. This provides you with a fast and easy upgrade in capacity to meet peak or unexpected loads.
HACMP 5.2 and up integrates with the Dynamic Logical Partitioning and CUoD functions. You can configure cluster resources in a way where the logical partition with minimally allocated resources serves as a standby node, and the application resides on another LPAR node that has more resources than the standby node.
![]() ![]() ![]() |