PreviousNextIndex

Appendix B: Resource Group Behavior during Cluster Events


This appendix describes how HACMP manages resource groups. It contains information on the following:

  • Resource Group Event Handling and Recovery
  • General Resource Group Event Processing Logic
  • Selective Fallover for Handling Resource Groups
  • Handling of Resource Group Acquisition Failures
  • Recovering Resource Groups when Nodes Join the Cluster
  • Handling of Resource Groups Configured with IPAT via IP Aliases
  • Examples of Location Dependency and Resource Group Behavior.
  • The information in this appendix may be especially useful for experienced HACMP users who are familiar with previous resource group handling by HACMP but may not be aware of changes made in recent releases.

    It is assumed that the reader is familiar with basic resource group fallover policies. These concepts are covered in the Concepts Guide and the Planning Guide.

    Overview

    Once you have planned and defined the cluster topology and resources, HACMP monitors the working cluster and takes actions to recover when failures are detected. HACMP monitors resources and launches events to handle resource groups. You do not need to specify these actions in the cluster; they are initiated automatically.

    HACMP manages resource groups by:

  • Moving only the resource groups that are affected by a failure of an individual resource to another node in the cluster.
  • Taking recovery actions when it fails to acquire resource groups on a node. This can happen when HACMP attempts to move a resource group to another node in the cluster, but fails to acquire it on that node. You can disable automatic recovery, if needed.
  • The sections below provide an overview of resource group events and describe when HACMP moves resource groups in the cluster, how the resource groups are placed on the nodes, and how to identify the causes of the underlying cluster events.

    Resource Group Event Handling and Recovery

    HACMP tracks the state of all cluster resources and manages the recovery according to the available backup resources. When multiple backup resources are available, HACMP can be configured to dynamically select the backup resources to use based on the current performance statistics (using a dynamic node priority policy). Event logging includes a detailed summary for each high-level event to help you understand exactly what actions were taken for each resource group during the handling of failures.

    Events and Resource Groups with Dependencies or Sites

    If either parent/child or location dependencies between any resource groups or sites are configured in the cluster, HACMP processes all events related to resource groups in the cluster with the use of resource_state_change trigger events that are launched for all resource groups for events where resource groups are affected.

    The Cluster Manager then takes into account all configured runtime policies, especially the configuration of dependencies and sites for resource groups, and the current distribution and state of resource groups on all nodes in order to properly handle any acquiring, releasing, bringing online or taking offline of resource groups.

    When events handle resource groups with dependencies or sites, a preamble is written to the hacmp.out log file listing the plan of sub_events for handling the resource groups.

    resource_state_change
    This trigger event is used for resource group recovery if resource group parent/child or location dependencies or sites are configured in the cluster. This action indicates that the Cluster Manager needs to change the state of one or more resource groups, or there is a change in the state of a resource managed by the Cluster Manager. This event runs on all nodes if one of the following occurs:
    • Application monitoring failure
    • Selective fallover for loss of volume group
    • Local network down
    • WAN failure
    • Resource Group Acquisition Failure
    • Resource Group Recovery on IP Interface Availability
    • Expiry of settling timer for a resource group
    • Expiry of fallback timer for a resource group
    While the event runs, the state of the resource group is changed to TEMP_ERROR or SECONDARY_TEMP_ERROR. This is broadcast to all nodes.
    NOTE: This is a place where you can add pre- or post-events for specific resources if needed.
    resource_state_change_complete
    This event runs on all nodes when the resource_state_change event has successfully completed (necessary recovery actions including release and acquire events have completed).

    Events for Moving Resource Groups

    HACMP may move resource groups as a result of recovery actions taken during the processing of events such as node_down and especially for resource_state_change:

    rg_move
    This event moves a specified resource group from one node to another.
    rg_move_complete
    This action indicates that the rg_move event has successfully completed.

    Resource Group Subevents and States

    Handling of individual resources during the processing of an event may include the following actions or resource group states. For example, when a filesystem is in the process of being unmounted and mounted it is taken offline and then released by one node. Then if there is an available backup node the filesystem will be acquired and brought online.

    RELEASING
    A resource group is being released either to be brought offline, or to be acquired on another node.
    ACQUIRING
    A resource group is being acquired on a node.
    ONLINE
    The resource group is online.
    OFFLINE
    The resource group is offline.
    ERROR
    The resource group is in an error state.
    TEMPORARY ERROR
    The resource group is in a temporary error state. It occurs, for instance, due to a local network failure or an application failure. This state informs the Cluster Manager to initiate an rg_move event for this resource group. Resource groups should not be in this state when the cluster is stable. See the section Resource Group Recovery when the Network or Interface is Up in this appendix.
    UNKNOWN
    The state of the resource group is unknown.
    ONLINE SECONDARY
    The resource group is online at the secondary site.
    ONLINE PEER
    The resource group is online on a peer node.
    ACQUIRING SECONDARY
    A resource group is being acquired on a node at the secondary site.
    RELEASING SECONDARY
    A resource group is being released on a node at the secondary site.
    ERROR SECONDARY
    The resource group is in error state at the secondary site.
    TEMPORARY ERROR SECONDARY
    The resource group is in a temporary error state at the secondary site.
    UNMANAGED
    You have stopped the cluster services without stopping the running applications. In this case:
    • HACMP is not managing the resources.
    • The previous state of the group was ONLINE.
    • The application and other resources may continue to be running on the node.

    Note: See the Planning Guide, Chapter 6, Planning Resource Groups, for more information on resource group behavior in clusters with sites.

    After the completion of an event, HACMP is aware of the state of resources and resource groups involved in the event. HACMP then analyzes the resource group information that it maintains internally and determines whether recovery events need to be queued for any of the resource groups. HACMP also uses status of individual resources in resource groups to print out a comprehensive event summary to the hacmp.out log file.

    For each resource group, HACMP keeps track of the nodes on which the resource group has tried to come online and failed. This information is updated when recovery events are processed. HACMP resets the nodelist for a resource group as soon as the resource group moves to the online or error states.

    When a resource group is in the process of being moved, application monitoring is suspended and resumed appropriately. The Application Monitor sees that the application is in recovery state while the event is being processed.

    resume_appmon
    This action is used by the Application Monitor to resume monitoring of an application.
    suspend_appmon
    This action is used by the Application Monitor to suspend monitoring of an application.

    Cluster Event Processing

    The resource group handling features add steps to the overall processing for an event:

      1. The Cluster Manager communicates with RSCT Group Services to obtain information about topology events, and consolidates information about events related to resource groups.
      2. The Cluster Manager performs event rollup and determines the actual cluster event to run.
      3. The Group Services protocol is run to get all cluster nodes to agree on the event (voting).
      4. The Cluster Manager starts up event scripts on the cluster nodes.
      5. Event scripts get information about resource groups to process for the event:
  • Get information from the HACMP Configuration Database and the Cluster Manager and determine resource groups to process for the event.
  • Get information about the nodes already tried and exclude these nodes from the default nodelist.
  • Exclude nodes with insufficient network interfaces (for resource groups that require network interfaces).
    1. 6. Check the node priority policy to prioritize the list of target nodes for a resource group.
      7. Event scripts process resource groups (bring them online/offline, etc.).
      8. Resource Group Manager internally marks resource groups as recoverable if failures are encountered during the “acquire” phase.
      9. Event scripts complete.
    Note: For more information on steps 5-9, and for information on which attributes and policies take precedence when the Cluster Manager determines which resource group to process first, see the section General Resource Group Event Processing Logic.
      10. The Cluster Manager gets return code from scripts.
      11. If the return code is 0, the event has completed (it may or may not have been successful); otherwise, return event_error (user intervention may be required to get the cluster back to a stable state).
      12. The Cluster Manager notes the resource groups affected by a local network failure event and marks affected resource groups as recoverable.
      13. The Cluster Manager notes the resource groups affected by a local network failure event (13) or by an acquiring error (8) and enqueues recovery events for each resource group in the recoverable state.
      14. End of event.

    General Resource Group Event Processing Logic

    You can specify various policies for resource groups in the cluster that affect the order in which resource group events take place. Such policies may include:

  • Specifying customized serial resource group processing. (This may be deprecated in a future release).
  • Requesting HACMP to move a resource group to a particular destination node or state. For more information, see the sections on migrating resource groups in Chapter 15: Managing Resource Groups in a Cluster.
  • Setting a dynamic node priority.
  • Specifying parent/child or location dependencies between resource groups
  • Customizing resource recovery for service IP labels and volume group resources (specifying fallover or notify)
  • Customizing resource group cross-site recovery (specifying fallover to other site or notify).
  • This section presents a high-level view of what actions take precedence in the resource group event processing process. The Cluster Manager examines the following variables to determine the order in which to handle events for resource groups. If all variables are equally true for two or more groups, groups are sorted according to the processing order specified on the nodes in the cluster.

      1. Resource group states (online, offline, error, unmanaged) determine which policies will be considered and, therefore, which resource groups will be taken for processing first. For instance, if the resource group is offline, the dynamic node priority set for this resource group is not considered. If the resource group is online, the Cluster Manager does not have to move it and does not perform look-ahead process to find an appropriate node.

    Also, in this step, resource groups’ dependencies are considered to determine which resource groups must be processed before other resource groups can be processed.

      2. Look-ahead for resource availability is considered. Nodes with insufficient network interfaces (for resource groups that require network interfaces) are excluded, and this affects the order in which events for resource groups will be handled.
      3. The node distribution policy for resource groups is considered.
      4. Dynamic node priority is considered.
      5. Participating nodelist for the particular resource group is considered.
      6. Startup, fallover and fallback settings of resource groups are considered: These settings include any previously configured delayed fallback timers and settling times for resource groups.
      7. Once the Cluster Manager decides which resource group to process, it considers the resource group dependencies, processing order settings and inter-site management policies. At this step, the Cluster Manager chooses the path for processing.

    Resource groups in clusters with dependencies or sites are processed in phases since more variables must be considered.

    Selective Fallover for Handling Resource Groups

    Selective fallover is a function of HACMP, which attempts to selectively move only a resource group that has been affected by an individual resource failure to another node in the cluster, rather than moving all resource groups. Selective fallover provides recovery for individual resource groups that are affected by failures of specific resources.

    Resources for which Selective Fallover is Used

    HACMP utilizes selective fallover in the case of a failure of several types of resources that can belong to a resource group:

  • Service IP labels
  • For more information, see the section Selective Fallover Caused by Network Interface Failures and the section Selective Fallover Caused by Local Network Failures in this appendix.
  • Applications
  • For more information, see the section Selective Fallover Caused by Application Failures in this appendix.
  • Communication Links.
  • For more information, see the section Selective Fallover Caused by a Communication Link Failure in this appendix.
  • Volume groups.
  • For more information, see the section Selective Fallover Caused by a Volume Group Loss in this appendix.

    You can customize the default selective fallover behavior to use a notification instead of a fallover for the following types of resources:

  • Service IP labels
  • Volume groups. This is the only resource type where customization also affects the secondary instance (if the resource group is replicated).
  • Selective Fallover and Resource Groups with Sites

    If you have configured replicated resource groups, HACMP by default tries to recover the failure of a secondary instance as well as the primary instance. Recovering the secondary instance does not affect the primary resource group.

    You can change the resource group recovery policy for replicated resource groups to allow (or not allow) the Cluster Manager to move the primary instance of resource groups to another site in cases where it can use selective fallover to avoid having the resource group go into ERROR state. If the primary instance is moved, then the secondary instances are placed on the opposite site.

    For information on customizing resource and cross-site resource group recovery, see Customizing Resource Recovery in Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended).

    Look-Ahead for Moving Resource Groups and Choosing Takeover Nodes

    HACMP determines the highest priority node using the resource group nodelist, dynamic node priority and persistent migration requests, as well as the availability of backup resources. For example, if a group contains a service IP label, HACMP looks at the status of the available interfaces on the backup nodes. If the resource group is part of a resource group parent/child or location dependency set, HACMP takes this into account also.

    HACMP does not move a resource group when there are no available backup resources. Instead, it simply takes the group offline from the current node. This result is clearly indicated in the event summary in the hacmp.out log file.

    Selective Fallover Caused by Network Interface Failures

    When a network interface with an HACMP service IP label fails and there are no other network interfaces available on the node on the same HACMP network, the affected applications on that node cannot run. If the service network interface is the last one available on the node, the network interface failure triggers a network failure event.

    HACMP distinguishes between two types of network failure, local and global. A local network failure occurs when a node can no longer communicate over a particular network, but the network is still in use by other nodes. A global network failure occurs when all nodes lose the ability to communicate over a network.

    HACMP uses the following formats for local and global network failure events:

    Local Network Failure Event
    network_down <node_name> <network_name>
    Global Network Failure Event
    network_down -1 <network_name>

    In the case of a local network failure, you may create a post-event to trigger a node_down event. While this has the desired effect of moving the resource group with the failed resource to another node, it has the undesired effect of moving all of the resource groups on the node to other nodes.

    Selective fallover uses this infrastructure to better handle network interface failures. You do not have to create a post-event to promote a local network failure to a node failure in this case. See the section below for more information on how HACMP handles network interface failures.

    You should not promote global network failures to node_down events as the global network event applies to all nodes and would result in a node down for all nodes.

    Actions Taken for Network Interface Failures

    HACMP takes the following actions in cases of network interface failures:

  • When a network interface with a service IP label fails, and there are no network interfaces available on the same node (therefore, a swap_adapter is not possible), it moves only the resource group associated with the failed service network interface to another node.
  • When a network interface fails and this can result in launching an rg_move for the affected resource group, a check for available network interfaces is made. The highest priority node with an available network interface attempts to acquire the resource group.
  • HACMP checks that a network interface is available on the node joining the cluster before releasing the resource group. If no network interfaces are available the resource group is not released.
  • In addition, see the section Resource Group Recovery when the Network or Interface is Up in this chapter for information on when HACMP makes an attempt to bring the resource group that went into an error state back online.
  • The above actions assume available nodes in the resource group definitions.

    The hacmp.out file contains messages informing you about cluster activity that results from selective fallover actions.

    Selective Fallover Caused by Local Network Failures

    When a local network failure event occurs, the Cluster Manager takes selective recovery actions for resource groups containing a service IP label connected to that network. The Cluster Manager tries to move only the resource groups affected by the local network failure event, rather than all resource groups on a particular node.

    Note: You do not need to create a post-event to promote a local network failure to a node failure in cases of network interface failures.

    For example, if you have two resource groups:

    RG1 - service label on network ether 
    RG2 - service label on network Token-Ring 
    

    If network Token-Ring fails, the Cluster Manager will move RG2; it will not touch RG1.

    Selective Fallover Caused by Application Failures

    When an application that is being monitored by Application Monitoring fails, HACMP attempts to move the resource group containing that application to another node. Only the affected resource group is moved.

    HACMP can only detect the application failure if an application monitor is configured. There are two types of application monitors:

  • Process monitors. You tell HACMP about a specific process to watch. When that process exits, HACMP initiates recovery.
  • Custom monitors. You provide your own method, which is called periodically by HACMP to check the health of the application. Custom monitors can be much more sophisticated and robust in their checking of application health.
  • Selective Fallover Caused by a Communication Link Failure

    An X.25 or SNA-over-X.25 communication link that is defined as a resource in a cluster resource group has a list of physical adapters on that node that it can use for reestablishing a connection.

    When clcommlinkd detects the failure of an X.25 link over which X.25 or SNA is running, the link falls over to another adapter on the same node if possible. If there are no other adapters available on the local node, the selective fallover action moves the resource group containing the link to another node.

    An rg_move event launched in this case logs the reasons why it was launched in the hacmp.out file. The hacmp.out file also indicates resource group status.

    Selective Fallover Caused by a Volume Group Loss

    Selective fallover can also be triggered when HACMP detects a volume group failure on a node containing that resource group. In other words, HACMP automatically reacts to a “loss of quorum” error associated with a volume group going offline on a cluster node.

    If a volume group in the resource group has dropped offline due to a loss of quorum error for the volume group on that node, HACMP selectively moves the resource group to another node.

    Conditions for Selective Fallover for Volume Group

    HACMP uses the selective fallover for volume group loss functionality under the following conditions:

  • HACMP monitors all volume groups that are included in the resource group, and all volume groups on which filesystems that are part of the resource group depend.
  • HACMP moves only resource groups that contain volume groups with volume groups for which the LVM_SA_QUORCLOSE error has been logged by the error daemon in the AIX 5L errpt on that node.
  • Note: HACMP does not react to any other type of volume group errors automatically. In these cases, you still need to configure customized error notification methods, or use AIX 5L Automatic Error Notification methods to react to volume group failures.

    Notes on the Error Notification Method for Volume Group Loss

    HACMP uses an Error Notification method to inform the Cluster Manager about the failure of a volume group. When using this error notification method:

  • Do not modify this error notification method. HACMP issues a warning and takes no action if you attempt to customize this notification method, or to use it to protect against the failure of other types of resources.
  • Synchronize the cluster after making changes to the cluster configuration. A notification script used for a volume group failure should correspond to the current configuration of cluster resources, otherwise HACMP issues a warning during verification and takes no action to selectively move the affected resource group.
  • Besides the errnotify entries created by HACMP for selective fallover, the errnotify ODM may also contain other entries related to the same AIX 5L error labels and resources. However, selective fallover provides the most effective recovery mechanism to protect a resource group from the failure of a single resource.
  • The notification method that is run in the case of a volume group failure provides the following information in the hacmp.out and clstrmgr.debug log files:
  • AIX 5L error label and ID
  • The name of the affected resource group
  • The node’s name on which the error occurred.
  • You can test the error notification methods generated by the selective fallover facility by emulating an error for each volume group in SMIT. To test error notification:
    1. 1. Enter smit hacmp
      2. In SMIT, select Problem Determination Tools > HACMP Error Notification > Emulate Error Log Entry, and press Enter.
      3. Select from the picklist the error notification object that was generated by the selective fallover facility for each volume group.

    For more information on how the error log emulation works, see Chapter 9: Configuring AIX 5L for HACMP in the Installation Guide.

    Handling of Resource Group Acquisition Failures

    HACMP uses event scripts to move resources around the HACMP cluster. HACMP differentiates certain types of failures in the event script. There are still fatal type errors where an error in the script logic or environment causes the script to fail, but now HACMP traps recoverable errors related to the processing of the resources. This allows HACMP to continue event processing and to try to bring the group online on the next available node.

    Attempts by HACMP to start or move a resource group may fail for a variety of reasons, such as busy or unavailable devices, or lack of disk space. HACMP may react to such failures by attempting to move the resource group to another node.

    In the event of a resource group acquisition failure on a particular node:

  • Not all resource group acquisition failures require immediate manual intervention. In some cases, a resource group will be successfully brought online on another node. However, the fact that a resource group acquisition failure occurred indicates a system problem that needs attention.
  • The Cluster Manager logs error messages when a node cannot acquire a resource group, and continues processing events so the cluster resources remain available.
  • HACMP 5.2 and up automatically attempts to activate resource groups in the ERROR state on the node during a node_up event. You cannot disable this functionality. If an attempt to recover a resource group in the ERROR state on a joining node is made but the resource group acquisition on the node fails, the non-concurrent resource group falls over to the next node in the nodelist, if one is available. If the concurrent resource group acquisition fails, the resource group remains in the ERROR state.
  • HACMP logs reported resource group acquisition failures (failures indicated by a non-zero exit code returned by a command) in hacmp.out. The information appears in an event summary that follows each top-level event’s details.
  • The event summary makes it easier for you to check the hacmp.out file for errors. Checking this log becomes more important, since the config_too_long console message will not be evident in every case where a problem exists.

    However, if you have previously configured a notification method for a config_too_long event, HACMP uses this method to notify you that the resource group went offline or remained in an error state.

    How HACMP Processes Resource Group Acquisition Failures

    If a node fails in an attempt to acquire a resource group during a fallover, HACMP marks the resource group “recoverable” and triggers an rg_move event to try to bring up the resource group on some other node. Note that a failure may occur during the acquisition phase of an rg_move, and this may cause a queue of rg_move events. The software goes through the queue until the resource group is successfully brought online, or until all possible owners have failed to acquire it, in which case the resource group is left in the ERROR state.

    Resource Group Recovery when the Network or Interface is Up

    When a local network failure occurs, HACMP determines whether any resource groups are affected by the failure. The affected resource groups are those that contain service labels defined on the failed network. In this case, HACMP checks whether such resource groups are online on a node, and attempts to move each affected resource group to another node by initiating an rg_move event for it.

    The rg_move event attempts to bring the resource group online on another node. If HACMP does not find available resources on any nodes to bring the resource group online, then the rg_move event leaves the resource group in an error state and the resource group goes offline and becomes unavailable.

    HACMP tries to bring resource groups in ERROR state online whenever a network interface becomes available. When bringing the affected resource groups online, HACMP does the following:

      1. If it finds resource groups that went to an error state due to a resource failure and contain a service IP label of a network interface that became available again, then it moves such resource groups to the rg_temp_error_state.
      2. Prior to running an rg_move for an affected resource group, HACMP determines possible candidate nodes for bringing the resource group online. If it cannot find any candidate nodes, then the resource group remains offline and unavailable.
      3. If HACMP finds candidate nodes, it initiates an rg_move event to attempt to bring the resource groups online.

    Disabling Automatic Recovery of Resource Groups

    When a local network failure occurs, you may need to replace the failed resource first, before letting HACMP automatically bring the affected resource group online. For instance, you may need to replace and test the network interface before bringing the affected resource group online.

    To avoid automatic recovery of a resource group if the resource group goes into an error state, take these steps:

      1. Enter smit hacmp
      2. In SMIT, select System Management (C-SPOC) > Resource Group and Application Management > Bring a Resource Group Offline and press Enter.
      3. Specify that this resource group must remain offline on a node.

    Remember to bring the resource group back online manually when needed.

    Recovering Resource Groups when Nodes Join the Cluster

    Prior to HACMP 5.2, when a node joined the cluster, it did not make an attempt to acquire any resource groups that had previously gone into an ERROR state on any other node. Such resource groups remained offline and required use of the Resource Group Migration utility, clRGmove, to manually bring them back online.

    In HACMP 5.2 and up, resource group recovery is improved. An attempt is made to automatically bring online the resource groups that are currently in the ERROR state. This further increases the chances of bringing the applications back online. When a node that is included in the nodelist for the resource group starts up, if the resource group is in the ERROR state on any node in the cluster, this node attempts to acquire the resource group. The node must be included in the nodelist for the resource group.

    The resource group recovery on node startup is different for non-concurrent and concurrent resource groups:

  • If the starting node fails to activate a non-concurrent resource group that is in the ERROR state, the resource group continues to selectively fall over to another node in the nodelist, if a node is available. In this case, HACMP uses selective fallover. The fallover action continues until all available nodes in the nodelist have been tried.
  • If the starting node fails to activate a concurrent resource group that is in the ERROR state, the concurrent resource group is left in the ERROR state.
  • Note: HACMP 5.2 and up automatically attempts to activate resource groups in the ERROR state on the node during a node_up event. You cannot disable this functionality. If an attempt to recover a resource group in the ERROR state on a joining node is made but the resource group acquisition on the node fails, the non-concurrent resource groups falls over to the next node in the nodelist, if one is available. If the concurrent resource group acquisition fails, the resource group remains in the ERROR state.

    Handling of Resource Groups Configured with IPAT via IP Aliases

    When you configure your HACMP cluster, you define certain IP labels/IP addresses (service addresses) to be kept highly available. These service addresses are typically the IP address used by clients to access the server application. HACMP keeps the IP address available to clients by moving the address between different network interfaces.

    With IPAT via IP Replacement there is a strict one-to-one mapping between IP address and network interface: Only one address can be placed on an interface at a given time. When HACMP moves the service IP address, it first removes the base or boot address from the interface, then puts the service address on the interface.

    This one-to-one relationship between addresses and interfaces requires physical cluster hardware—a local interface and a backup interface—for every highly available service address. This also limits the configuration options available for the HACMP cluster since resource group placement is bound to the availability of physical resources and recovery is not possible once all the physical resources are consumed.

    IP aliasing is a function of the TCP/IP stack where multiple IP addresses can be added to the same physical interface. When HACMP uses IP aliases for recovery, the base address for the interface does not change. HACMP recovers the service address by adding it as a second or alias address on the same interface. If you are using IPAT via IP Aliases, networks do not have the strict one-to-one relationship between interfaces and addresses. A single physical interface can host or back up multiple service addresses. This greatly improves the configuration flexibility and fallover options by requiring fewer physical resources to serve as backup. IPAT via IP Aliases is also faster as there are fewer commands required when moving addresses.

    Do not mix service IP labels that can be used in IPAT via Aliases with those that cannot in the same resource group. This restriction is enforced during verification of cluster resources.

    To control the placement of the service IP label aliases on the cluster node physical network interface cards, you can configure a distribution preference for the aliases of the service IP labels that are placed under HACMP control. See the section Distribution Preference for Service IP Label Aliases: Overview in Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended).

    Resource Group Behavior when Using IPAT via IP Aliases

    Networks configured to use IPAT via Aliases have service addresses (the IP address HACMP is to keep highly available) and base interfaces (the network interfaces to use to host the service address). There are no “standby” interfaces. When multiple interfaces are on the same network and the same node, they are all defined as boot.

    With IPAT via IP Aliases, the service address is added as an alias address on an available boot interface. This applies to the node where the resource group is first acquired, as well as to the node(s) that might acquire it later. When the resource group is released, the service address is removed from the interface, but this does not alter the base or boot address on the interface.

    The mechanics of IP address takeover on a network using aliases works the same way for all non-concurrent resource groups. While the mechanics are identical, IPAT via IP aliases does affect the initial startup and fallover placement of the resource group.

    The aliased service IP labels are distributed across all available boot interfaces. To facilitate even distribution of labels across all available IP interface cards, HACMP sorts all available interfaces by state and then by the number of aliased addresses already placed on the interface, and places the aliased labels accordingly. Note that this distribution is only done at fallover time, HACMP makes no attempt to redistribute the labels later, if another interface becomes active.

    Note: If you want HACMP to activate only a specific resource group on a node during startup among multiple resource groups that could potentially be acquired on this node, we recommend that you use the startup policy Online Using Node Distribution Policy.

    Resource Group Placement on Cluster Startup

    The presence of a service IP label defined on an network configured for IPAT via aliases in the resource group does not alter the resource group’s placement policy on initial cluster startup. That is, on initial cluster startup, the non-concurrent resource group configured with IPAT via IP Aliases is placed according to the defined startup policy.

    On the subsequent cluster startup, HACMP moves the resource group containing the service IP label onto a node with a boot interface that:

  • Is up
  • Has a different subnet than the IP label that is being moved.
  • In addition, HACMP follows these rules:

  • If multiple boot interfaces are found that are up and have different subnets, then HACMP moves the resource group onto the one that comes first in the alphabetically-sorted list of network interfaces configured on the node.
  • If the resource group uses Online Using Node Distribution Policy startup, the resource group is placed on a node that does not host another resource group.
  • Resource Group Placement on Fallover

    On fallover, if you have configured resource groups that contain aliased service IP labels, this allows having more than one non-concurrent resource group on the same node. Therefore, more than one resource group can be serviced by a node with a single physical interface.

    On fallover, HACMP moves the resource group containing the service IP label onto a node with a boot interface which:

  • Is up
  • Has a different subnet
  • Is preferably not hosting another service label (if available).
  • Comes first in the alphabetically-sorted list of network interfaces in the network configuration.
  • If multiple boot interfaces are found that are up, have different subnets, and are not hosting another service label, then HACMP moves the resource group onto the one that comes first in the alphabetically-sorted list of network interfaces configured on the node.
  • The key advantage of having resource groups configured with IPAT via IP Aliases is that, on fallover, more than one resource group can be serviced by a node with a single physical interface.

    Examples of Location Dependency and Resource Group Behavior

    This appendix contains scenarios that illustrate how location dependent resource groups are processed at startup and also how they are processed for various failure scenarios.

  • Publishing Model with Same Node and Different Nodes Dependencies
  • Publishing Model—Alternate Configuration
  • WAS/DB2 Cluster Model and Use Cases
  • Replicated WAS-Solution Model Use Cases
  • Replicated WAS-Solution, Configuration II
  • Replicated WAS-Solution, Configuration III
  • Publishing Model with Same Node and Different Nodes Dependencies

    The XYZ Publishing company follows a business continuity model that involves prioritizing the different platforms used to develop the web content. XYZ uses location dependency policies to keep some resource groups strictly on separate nodes and others together on the same node.

    The Production database (PDB) and Production application (Papp) are hosted on the same node to facilitate maintenance (and perhaps the highest priority node for these resource groups has the most memory or faster processor). It also makes sense to set up a parent/child relation between them, since the application depends on the database. The database must be online for the application to function.The same conditions are true for the System Database (SDB) and the System application (Sapp) and for the QA Database (QADB) and the QA application (QAapp).

    Since keeping the production database and application running is the highest priority, it makes sense to configure the cluster so that the three database resource groups stay on different nodes (make them an Online On Different Nodes dependency set), and assign the PDB resource group with the high priority. The SDB is the Intermediate priority and the QADB is the low priority.

    The databases and their related applications are each configured to belong to an Online On Same Node dependency set.

    HACMP handles these groups somewhat differently depending on how you configure startup, fallover, and fallback policies. It makes sense to have the participating nodelists differ for each database and application set to facilitate keeping these resource groups on the preferred nodes.

    The figure below shows the basic configuration of the three nodes and six resource groups.

    Publishing Model with Parent/Child and Location Dependencies 
    

    Resource Group Policies: Online on First Available Node

    For the following use case discussions, all six resource groups have the following policies:

  • Startup Policy: Online On First Available Node
  • Fallover Policy: Fallover to Next Priority Node
  • Fallback Policy: Never Fallback
  • Participating Nodes
    Location Dependency
    Parent/Child Dependency
    • PApp: 1, 2, 3
    • PDB: 1, 2, 3
    • SApp: 2, 3
    • SDB: 2, 3
    • QAApp: 3
    • QADB: 3
    Online On The Same Node Dependent Groups:
    • PApp with PDB
    • SApp with SDB
    • QAApp with QADB
    Online On Different Nodes Dependent set:
    [PDB SDB QADB]
    Priority: PDB > SDB > QADB
    • PApp (child) depends on PDB (parent)
    • SApp (child) depends on SDB (parent)
    • QAApp (child) depends on QADB (parent)
  • Use Case 1: Start Nodes in Numerical Order (Node1 First)

    Starting the nodes in numerical order we expect the Production resource groups to come online on Node 1, the System resource groups to come online on Node 2, and the QA resource groups to come online on Node 3. There is no contention.

    Node 1 is the highest priority node for resource groups PDB and PApp. The parent/child dependency dictates that PDB must be brought online prior to processing PApp. Therefore, HACMP processes the rg_move event to acquire PDB first and then it acquires PApp.

    Node 1 is not in the nodelist for any other groups. Even if it were, the Online on Different Nodes dependency would disallow any lower priority groups from coming online on this node.

    Consolidated View of Start Node Sequence: 1, 2, 3

    Step
    Node 1
    Node 2
    Node 3
    Start Node 1
    PApp: ONLINE
    PDB: ONLINE
    PApp:
    PDB:
    SApp:
    SDB:
    PApp:
    PDB:
    SApp:
    SDB:
    QAApp:
    QADB:
    Start Node 2
    PApp: ONLINE
    PDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE SDB: ONLINE
    PApp:
    PDB:
    SApp:
    SDB:
    QAApp:
    QADB:
    Start Node 3
    PApp: ONLINE
    PDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE SDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: OFFLINE SDB: OFFLINE
    QAApp: ONLINE
    QADB: ONLINE

    Use Case 2: Start Nodes Out of Order (Node 3)

    Precondition: Resource groups are offline, all nodes are offline
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 3
     
     
     
    2
     
     
     
    Acquire PDB
    3
     
     
     
    Acquire PApp
    Post-condition/
    Resource group states
     
    PDB
    PApp
     
    PDB:
    Papp
    SDB
    SApp
     
    PApp: ONLINE
    PDB: ONLINE
    SApp: ERROR
    SDB: ERROR
    QAApp: ERROR
    QADB: ERROR

    Node 3 is the lowest priority node for PDB and PApp, as well as for SDB and SApp. Node 3 is the highest priority node for QADB and QAApp. However, the PDB/PApp pair has the highest priority due to the Online On Different Nodes Dependency. Therefore, HACMP will acquire and start PDB on Node 3 and then process its child PApp. The other resource groups will go to the ERROR state based on the rule—these resource groups could have been brought online on Node 3 but were not acquired due to the Online On Different Nodes Dependency policy.

    Use Case 2 Continued: Start Nodes Out of Order (Node 2)

    Precondition: Node 3 is up; cluster and group states as at the end of the previous table.
    Step/Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 2
     
     
     
    2
     
     
     
    Release PApp
    3
     
     
     
    Release PDB
    4
     
     
    Acquire PDB
    Acquire SDB
    5
     
     
    Acquire PApp
    Acquire SApp
    Post-condition/ Resource group states
     
    PApp:
    PDB:
    PApp: ONLINE
    PDB: ONLINE
    SApp: OFFLINE
    SDB: OFFLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE
    SDB: ONLINE
    QAApp: ERROR
    QADB: ERROR

    Node 2 is the highest priority node for the SDB and SApp resource groups. However, the higher priority resource group in the Online On Different Nodes Dependency set is PDB. Therefore, PDB will fall over to this joining node while SDB and SApp will be acquired and started on Node 3. HACMP moves Papp to the same node with PDB because these two resource groups belong to an Online on Same Node dependency set. QA PDB is lower priority than SDB, so it stays in the ERROR state along with QAapp.

    When Node 1 comes up, PDB and Papp will fall over to Node 1, SDB and Sapp will fall over to Node 2, and the QA resource groups will be acquired and started on Node 3.

    Consolidated View of Start Nodes Out of Order: Sequence 3, 2, 1

    Step
    Node 1
    Node 2
    Node 3
    Start Node 3
    PApp:
    PDB:
    PApp:
    PDB:
    SApp:
    SDB:
    PApp: ONLINE
    PDB: ONLINE
    SApp: ERROR
    SDB: ERROR
    QAApp: ERROR
    QADB: ERROR
    Start Node 2
    PApp:
    PDB:
    PApp: ONLINE
    PDB: ONLINE
    SApp: OFFLINE
    SDB: OFFLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE
    SDB: ONLINE
    QAApp: ERROR
    QADB: ERROR
    Start Node 1
    PApp: ONLINE
    PDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE
    SDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: OFFLINE
    SDB: OFFLINE
    QAApp: ONLINE
    QADB: ONLINE

    Use Case 3: Fallover of Resource Groups due to Node Failure

    Precondition: All nodes are ONLINE. Resource groups PDB and PApp are online on Node 1, SDB and SApp are online on Node 2, QAApp and QADB are online on Node 3
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Comments
    1
    Node 1 crash.
     
    Release SApp
    Release QAApp
     
    2
     
     
    Release SDB
    Release QADB
    QAApp and QDB go to ERROR state.
    3
     
     
    Acquire PDB
    Acquire SDB
     
    4
     
     
    Acquire PApp
    Acquire SApp
     
    Post-condition/Resource Group states
     
    PApp:
    PDB:
    PApp: ONLINE
    PDB: ONLINE
    SApp: ERROR
    SDB: ERROR
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE
    SDB: ONLINE
    QAApp: ERROR
    QADB: ERROR
     

    When Node 1 fails, HACMP releases SApp and SDB and QADB and QAapp and moves the highest priority resource group PDB and its Same Node dependency partner and child PApp to Node 2 and likewise moves the System groups to Node 3. The QA groups are left with nowhere to go; they go into ERROR state).

    Use Case 4: Fallover of Resource Groups—Network Goes Down During Fallover

    Precondition: All nodes are ONLINE. Resource groups PDB and PApp are online on Node 1, SDB and SApp are online on Node 2, QAApp and QADB are online on Node 3. All applications use the app_network,
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Comments
    1
    Node 1 crash.
     
    Release SApp
    Release QAApp
     
    2
     
     
    Release SDB
    Release QADB
     
    3
     
     
    Acquire PDB
    Acquire SDB
    QADB goes into ERROR state
    4
    app_network
    down Node 2
     
     
     
    app_network failure.
    5
     
     
    Acquire PApp
    Acquire SApp
    PApp and SApp go to the ERROR state (network not available)
    6
     
     
    resource_state_
    change event
    resource_state_
    change event
    Triggers rg_move events
    7
     
     
    Release PDB
    Release SApp
     
    8
     
     
     
    Release SDB
     
    9
     
     
    Acquire SDB
    Acquire PDB
     
    10
     
     
    Acquire SApp
    Acquire PApp
     
    Post-condition/ Resource group states
     
    PApp:
    PDB:
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ERROR
    SDB: ONLINE
    PApp: ONLINE
    PDB: ONLINE
    SApp: ERROR
    SDB: ERROR
    QAApp: ERROR
    QADB: ERROR
     

    In step 5, PApp goes to the ERROR state directly, instead of going through the acquisition phase since the Cluster Manager knows that the network required for PApp on Node 2 is currently down. This is in contrast to an acquisition failure.

    In step 6, the event queue gets a resource_state_change event on the queue, which gets voted and queues additional ACQUIRE/RELEASE events.

    In steps 7 & 8: SApp goes to the ERROR state due to network failure.

    Publishing Model—Alternate Configuration

    This model consists of three pairs of parent and child resource groups (total of six resource groups) and three nodes in the HACMP cluster. The applications (PApp, SApp and QAApp) are Online On The Same Node with their corresponding databases (PDB, SDB and QADB.) All databases (which are parent resource groups also) are Online On Different Nodes from the other databases.

    The only difference between the original Publishing Model configuration and this alternate Publishing Model is the resource group’s startup preferences. This section uses the Online On Home Node Only startup policy whereas the original Publishing Model configuration uses Online On First Available Node as the startup policy for the resource groups.

    Alternate Publishing Model: Startup on Home Node Only 
    

    Resource Group Policies: Online on Home Node Only

    All six resource groups have the following policies:

  • Startup Policy: Online On Home Node Only—this is different from the previous set of use cases.
  • Fallover Policy: Fallover to Next priority node
  • Fallback Policy: Never Fallback
  • Participating Nodes
    Location Dependency
    Parent/Child Dependency
    PApp: 1, 2, 3
    PDB: 1, 2, 3
    SApp: 2, 3
    SDB: 2, 3
    QAApp: 3
    QADB: 3
    Online On The Same Node Dependent Groups:
    • PApp is with PDB
    • SApp is with SDB
    • QAApp is with QADB
    Online On Different Nodes Dependent set:
    [PDB SDB QADB]
    Priority: PDB>SDB>QADB
    PApp (child) depends on PDB (parent)
    SApp (child) depends on SDB (parent)
    QAApp (child) depends on QADB (parent)
  • Use Case 1: Start Lowest Priority Node (Node 3)

    Precondition: All resource groups are offline, all nodes are offline
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 3
     
     
     
    2
     
     
     
    Acquire QADB
    3
     
     
     
    Acquire QAApp
    Post-condition/Resource group states
     
    PApp:
    PDB:
    Papp
    PDB:
    SApp:
    SDB:
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: OFFLINE SDB: OFFLINE
    QAApp: ONLINE
    QADB: ONLINE

    Node 3 is the home node for resource groups QAApp and QADB. Although PDB and PApp have higher priority as defined by the Online On Different Nodes Dependency set, during cluster startup the startup policy allows only the QAApp and QADB resource groups to come online on Node 3. Therefore, the higher priority resource groups remain in the OFFLINE state at startup.

    Use Case 2: Start Second Node (Node 2)

    Precondition: Node 3 is up; cluster and group states as at the end of the previous use case.
    Step/Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 2
     
     
     
    2
     
     
    Acquire SDB
     
    3
     
     
    Acquire SApp
     
    Post-condition/ Resource group states
     
    PApp:
    PDB:
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE
    SDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: OFFLINE
    SDB: OFFLINE
    QAApp: ONLINE
    QADB: ONLINE

    Node 2 is the highest priority node for SDB and SApp resource groups. Since the startup policy for the resource groups is Online On Home Node, these resource groups will be started even though PDB and PApp are the highest priority resource groups.

    Consolidated View of Start Node Sequence: 3, 2, 1

    Step
    Node 1
    Node 2
    Node 3
    Start Node 3
    PApp:
    PDB:
    PApp:
    PDB:
    SApp:
    SDB:
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: OFFLINE
    SDB: OFFLINE
    QAApp: ONLINE
    QADB: ONLINE
    Start Node 2
    PApp:
    PDB:
    PApp: ONLINE
    PDB: ONLINE
    SApp: OFFLINE
    SDB: OFFLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: OFFLINE
    SDB: OFFLINE
    QAApp: ONLINE
    QADB: ONLINE
    Start Node 1
    PApp: ONLINE
    PDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: ONLINE
    SDB: ONLINE
    PApp: OFFLINE
    PDB: OFFLINE
    SApp: OFFLINE
    SDB: OFFLINE
    QAApp: ONLINE
    QADB: ONLINE

    WAS/DB2 Cluster Model and Use Cases

    This model contains a DB2 database, a WAS application that depends on DB2 and four WebSphere Applications. The parent/child dependency for this model is that DB2 should be available prior to activating the WAS and WebSphere (WS#) applications depend on the availability of WAS.

    The location dependency of the resource groups is that DB2 and WAS should not be activated on the same node and WAS is Online On The Same Node Dependent with WS4 (see figure) and DB2 is Online On The Same Node with WS1, WS2 and WS3. The location dependency for the WS# is purely artificial in this example. However, this is a configuration where one of the nodes is fine-tuned for DB2 (hence will be the highest priority node for DB2) and the other one is fine-tuned for WAS. They both have a common backup node, which can host only one of the two groups at a time.

    WAS/DB2 Cluster with Location and Parent/Child Dependencies 
    

    Resource Group Policies

    All resource groups have the following policies:

  • Startup Policy: Online On First Available Node
  • Fallover Policy: Fallover to Next priority node
  • Fallback Policy: Never Fallback
  • Participating Nodes
    Location Dependency
    Parent/Child Dependency
    DB2 [2, 3]
    WS1 [2, 3]
    WS2 [2, 3]
    WS3 [2, 3]
    WS4 [1, 3]
    WAS [1, 3]
    Online On The Same Node Dependent Groups:
    1. DB2, WS1, WS2, WS3
    2. WAS, WS4
    Online On Different Nodes Dependent Groups:
    • DB2, WAS
    1. WS1, WS2, WS3 and WS4 (children) depend on WAS (parent)
    2. WAS (child) depends on DB2 (parent)

    Use Case 1: Start First Node (Node 1)

    Precondition: All resource groups are offline, all nodes are offline
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 1
     
     
     
    Parent/child dependency not met.
    2
     
    WAS: ERROR
     
    3
     
    WS4: ERROR
     
    Post-condition/ Resource group states
     
    WAS: ERROR WS4: ERROR
    DB2:
    WS1:
    WS2:
    WS3:
    WAS:
    DB2:
    WS1:
    WS2:
    WS3:
    WS4

    WAS and WS4 could have started on Node 1, but the parent resource group DB2 is still in the offline state. Therefore, WAS and WS4 are put in the ERROR state.

    Use Case 2: Start Second Node (Node 2)

    Precondition: Cluster state as in the post-condition from the above use case.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 2
     
     
     
    2
     
     
    Acquire DB2
     
    3
     
    Acquire WAS
     
     
    4
     
    Acquire WS4
    Acquire WS1, WS2, WS3
     
    Post-condition/ Resource group states
     
    WAS:ONLINE
    WS4: ONLINE
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS:
    DB2:
    WS1:
    WS2:
    WS3:
    WS4:

    Node 2 starts DB2 (the parent RG), which in turn triggers processing of WAS (child of DB2). Finally all the grandchildren are started on their respective nodes.

    Consolidated View of Start Node Sequence 1, 2, 3

    Step
    Node 1
    Node 2
    Node 3
    Start node 1
    WAS: ERROR
    WS4: ERROR
    DB2:
    WS1:
    WS2:
    WS3:
    WAS:
    DB2:
    WS1:
    WS2:
    WS3:
    WS4:
    Start node 2
    WAS: ONLINE
    WS4: ONLINE
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS:
    DB2:
    WS1:
    WS2:
    WS3:
    WS4:
    Start node 3
    WAS: ONLINE
    WS4: ONLINE
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS: OFFLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: OFFLINE

    Use Case 3: Start Nodes Out of Order (Node 3)

    Precondition: All cluster nodes and resource groups are in the offline state.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Comments
    1
    Start Node 3
     
     
     
     
    2
     
     
     
    Acquire DB2
     
    Post-condition/ Resource group states
     
    WAS:
    DB2:
    WS4
    WS1:
    WS2:
    WS3:
    WAS: ERROR
    DB2: ONLINE
    WS1: ERROR
    WS2: ERROR
    WS3: ERROR
    WS4: ERROR
     

    Node 3 is a participating node for all the resource groups. However, WAS and DB2 cannot coexist on the same node. DB2— being a parent—is started on Node 3, which means that WAS cannot be started on the same node. Since WAS is not online none of the children of WAS can come online on Node 3.

    Use Case 4: Start Second Node Out of Order (Node 2)

    Precondition: Cluster and RG states as at the end of the previous use case.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 2
     
     
     
    2
     
     
     
    Release DB2
    3
     
     
    Acquire DB2
     
    4
     
     
     
    Acquire WAS
     
     
     
    Acquire WS1, WS2, WS3
    Acquire WS4
    Post-condition/ Resource group states
     
    WAS:
    WS4
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS: ONLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: ONLINE

    Node 2 is the higher priority node for DB2. Therefore DB2 falls back to Node 2 and WAS (Online On Different Nodes Dependency set) can now be acquired on Node 3.

    Use Case 5: Start Third Node (Node)1

    Precondition: Cluster and RG states as at the end of the previous use case.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    1
    Start Node 1
     
     
     
    2
     
     
    Release WS1, WS2, and WS3
    Release WS4
    3
     
    Acquire WAS
     
     
    4
     
    Acquire WS4
     
     
    5
     
     
    Acquire WS1, WS2 and WS3
     
    Post-condition/ Resource group states
     
    WAS: ONLINE
    WS4: ONLINE
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS: OFFLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: OFFLINE

    All groups are now online.

    Consolidated View of Start Node Sequence: 3, 2, 1

    Step
    Node 1
    Node 2
    Node 3
    Start Node 3
    WAS:
    WS4:
    DB2:
    WS1:
    WS2:
    WS3:
    WAS: ERROR
    DB2: ONLINE
    WS1: ERROR
    WS2: ERROR
    WS3: ERROR
    WS4: ERROR
    Start Node 2
    WAS:
    WS4:
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS: ONLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: ONLINE
    Start Node 1
    WAS: ONLINE
    WS4: ONLINE
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS: OFFLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: OFFLINE

    Use Case 6: Acquisition Failure Example

    Precondition: Node 1 is offline and all resource groups are ONLINE on Nodes 2 and 3.
       
    Node 1
    Node 2
    Node 3
     
     
     
    WAS:
    WS4:
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3:ONLINE
    WAS: ONLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: ONLINE
     
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Comments
    1
    Node_up 1
     
     
     
     
    2
     
     
    Release WS1 WS2 WS3 Release
    Release WS4
     
    3
     
     
     
    Release WAS
     
    4
     
    Acquire WAS
     
     
    Acquisition Failure for WAS
    5
     
    rg_move WAS
     
     
    Normal rg_move event
    6
     
     
     
    Acquire WAS
     
    7
     
     
    Acquire WS1 WS2 WS3 Acquire
    Acquire WS4
     
    Post condition/ Resource group states
     
    WAS:OFFLINE
    WS4: OFFLINE
    DB2: ONLINE WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS: ONLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: ONLINE
     

    As Node 1 joins the cluster, WAS attempts to fallback but gets the acquisition failure. The acquisition failure launches a resource_state_change event; this triggers an rg_move event, which moves WAS to its original node.

    Use Case 7: Resource Group Move Due to Local Network Down (DB2 failure)

    Precondition: All resource groups are ONLINE and all nodes are ONLINE.

     

       
    Node 1
    Node 2
    Node 3
     
     
     
    WAS:
    WS4:
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3:ONLINE
    WAS: ONLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: ONLINE
     
     
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Comments
    1
    Local network failure for DB2
     
     
     
     
    2
     
    Release WS4
    Release WS1 WS2 WS3
     
     
    3
     
    Release WAS
     
     
     
    4
     
     
    Release DB2
     
     
    5
     
     
     
    Acquire DB2
     
    6
     
    Acquire WAS
     
     
     
    7
     
    Acquire WS4
     
    Acquire WS1 WS2 WS3
     
    Post condition/ Resource group states
     
    WAS: OFFLINE
    WS4: OFFLINE
    DB2: ONLINE
    WS1: ONLINE
    WS2: ONLINE
    WS3: ONLINE
    WAS: ONLINE
    DB2: OFFLINE
    WS1: OFFLINE
    WS2: OFFLINE
    WS3: OFFLINE
    WS4: ONLINE
     

    The local network down for DB2 (the parent group) results in release and acquire of the entire stack of the resource groups.

    Replicated WAS-Solution Model Use Cases

    This model contains a DB2 database and a WAS application that depends on DB2; DB2 in turn depends on an LDAP directory server. Note that DB2 normally does not depend on LDAP, but this scenario is used for our discussion.

    The parent/child dependency for this model is that LDAP should be available before activating DB2 and DB2 should be available prior to activating the WAS.

    The location dependency of the resource groups is that DB2 and WAS should not be activated on the same node. LDAP can coexist with either of the applications.

    The model has a backup site as well, and the constraint that, for performance reasons, all three resource groups should reside on the same site at any given time. In other words, the three resource groups involved have the dependency Online On The Same Site.

    All resource groups are of equal priority. There is an indirect priority implied by the parent/child dependency configuration, not something the administrator explicitly specified.

    Replicated WAS-Solution, Configuration I

    Replicated WAS Solution with Parent/Child, Different Nodes and Same Site Dependencies 
    

    Resource Group Policies

    All resource groups have the following policies:

  • Startup Policy: Online On First Available Node
  • Fallover Policy: Fallover to Next priority node
  • Fallback Policy: Never Fallback
  • Site Policy: Prefer Primary Site
  • Participating Nodes
    Location Dependency
    Parent/Child Dependency
    DB2 [1, 2, 3, 4]
    LDAP [1, 2, 3, 4]
    WAS [1, 2, 3, 4]
    Online On The Same Site Dependent Groups:
    • DB2, LDAP, WAS
    Online On Different Nodes Dependent Groups:
    • DB2, WAS
    1. DB2 (child) depends on LDAP (parent.)
    2. WAS (child) depends on DB2 (parent.)

    Use Case 1: Start First Node (Node 1)

    Precondition: All resource groups are offline, all nodes are offline
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start Node 1
     
     
     
     
     
    2
     
    LDAP Primary instance is acquired
     
     
     
     
    3
     
    DB2 Primary Instance is acquired
     
     
     
     
    4
     
    WAS goes into global ERROR state
     
     
     
    Startup policy has been met, but no node available
    Post-condition/ Resource group states
     
    LDAP: ONLINE
    DB2: ONLINE
    WAS: ERROR
     
     
     
     

    According to the resource group startup preferences, all resource groups could have come online on Node 1. LDAP came online first, DB2 second, which was dictated by the resource group parent/child dependency configuration. Since WAS and DB2 are Online On Different Nodes dependent, WAS could not come online, therefore it went into global ERROR state.

    WAS going into ERROR state at this point might not be intuitive at first.

    Use Case 2: Start Second Node (Node 2)

    Precondition: See Post-condition for Use Case 1
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start Node 2
     
     
     
     
     
    2
     
     
    WAS Primary Instance acquired
     
     
     
    Post condition/ Resource group states
     
    LDAP: ONLINE
    DB2: ONLINE
    WAS: ONLINE
     
     
     

    WAS is already in a pre-event ERROR state and has all of its dependencies met. Its parent resource groups are ONLINE. Node 2 is a candidate node because it is in the same site as
    Node 1 (the node hosting DB2 and LDAP) therefore, WAS Primary instance is acquired.

    Use Case 3: Start Third Node (Node 3)

    Precondition: See Post-condition for previous use case
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start
    Node 3
     
     
     
     
    All secondary instances are acquired in parallel
    2
     
     
     
     
    Acquire:
    DB2,
    LDAP, and WAS Secondary
     
    Post condition/ Resource group states
     
    LDAP: ONLINE
    DB2: ONLINE
     
    WAS: ONLINE
     
    LDAP: Secondary
    DB2: Secondary
    WAS: Secondary
     

    All secondary instances are acquired on the node that is started on the secondary site. Secondary instances do not repel each other; i.e., the Online On Different Nodes and Online On The Same Site policies that have been configured for the primary instances do not affect the behavior of the secondary instances. The secondary instances are acquired on the first node that joins the cluster.

    Use Case 4: Start Fourth Node (Node 4)

    Precondition: See Post-condition for previous use case
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start
    Node 4
     
     
     
    no action
     
    2
     
     
     
     
    Acquire:
    DB2,
    LDAP, and WAS Secondary
     
    Post-condition/ Resource group states
     
    LDAP: ONLINE
    DB2: ONLINE
    WAS: ONLINE
     
    LDAP: Secondary
    DB2: Secondary
    WAS Secondary
     

    No action is taken on any of the instances on any of the resource groups.

    Use Case 5: User-requested rg_move Entire Stack to Opposite Site

    Precondition: See Post-condition for previous use case
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    rg_move Same Site Dependent stack to
    Site 2
     
     
     
     
     
    2
     
     
    Release WAS Primary
     
     
     
    3
     
    Release DB2 Primary
     
     
     
     
    4
     
    Release LDAP Primary
     
     
     
     
    5
     
     
     
    Release:
    LDAP, DB2, WAS Secondary
     
     
    6
     
    Acquire:
    LDAP, DB2, WAS Secondary
     
     
     
     
    7
     
     
     
    Acquire LDAP Primary
     
     
    8
     
     
     
    Acquire DB2 Primary
     
     
    9
     
     
     
     
    Acquire WAS Primary
     
    Post condition/ Resource group states
     
    LDAP: Secondary
    DB2: Secondary
    WAS Secondary
     
     
    LDAP: ONLINE
    DB2: ONLINE
     
    WAS: ONLINE

    The primary instances are released prior to releasing the secondary instances, so that pending writes can be flushed to the secondary site.

    During acquisition, the secondary instances are acquired prior to acquiring the primary instances, so that the primary instances can mirror as soon as they come online. HACMP is responsible for placing the primary instances on site_2, taking into consideration their parent/child and location dependencies. The secondary instances are placed on the opposite site.

    The only constraint raised by resource behavior is that the primary instance MUST be released prior to releasing the secondary instance.

    Use Case 6: DARE Change Nodelist of All Resource Groups

    In this use case, we examine the steps used to handle the case when the node list of every resource group is changed, so that they are forced to move to the opposite site. The expected result, as well as the steps that need to be taken to get to that result, is similar to use case 5.

    The node list in the old configuration is [1, 2, 3, 4] is changed to [4, 3, 2, 1]. Since the site policy is Prefer Primary Site, the DARE operation should move all resource groups to their new highest priority site (if at all possible), and distribute the resource groups on that site within the constraints of their various policies, such as parent/child and location dependency policies.

    Precondition: See Post-condition for use case 4; all resource groups online on site1, and all nodes are available
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    DARE IN changed node list
     
     
     
     
     
    2
     
     
    Release WAS Primary
     
     
     
    3
     
    Release DB2 Primary
     
     
     
     
    4
     
    Release LDAP Primary
     
     
     
     
    5
     
     
     
    Release: LDAP, DB2, WAS Secondary
     
     
    6
     
    Acquire:
    LDAP, DB2, WAS Secondary
     
     
     
     
     
     
     
     
     
    Acquire:
    LDAP
     
     
     
     
     
     
    Acquire:
    DB2
     
     
     
     
     
     
     
    Acquire WAS
    Post-condition/ Resource group states
     
    LDAP: Secondary
    DB2: Secondary
    WAS Secondary
     
    LDAP: ONLINE
    DB2: ONLINE
     
    WAS: ONLINE
     

    It is determined that all the resource groups can potentially be hosted on their new highest priority site; this is determined by taking into consideration the list of available nodes as well as the list of available interfaces. Then HACMP will try to move the entire stack to the opposite site.

    The primary instances are released prior to releasing the secondary instances, so that pending writes can be flushed to the secondary site.

    During acquisition, the secondary instances are acquired prior to acquiring the primary instances, so that the primary instances can mirror as soon as they come online. HACMP is responsible for placing the primary instances on site_2, taking into consideration their parent/child and location dependencies. The secondary instances are placed on available nodes at the opposite site.

    Use Case 7: Node 4 fails

    Precondition: See Post-condition for use case 5 in this section
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Node 4 fails
     
     
     
     
     
    2
     
     
     
     
    Release WAS Primary
     
    3
     
     
     
    Release DB2 Primary
     
     
    4
     
     
     
    Release LDAP Primary
     
     
    5
     
    Release:
    LDAP, DB2, and WAS Secondary
     
     
     
     
    6
     
     
     
    Acquire:
    LDAP, DB2, and WAS Secondary
     
     
    7
     
    Acquire LDAP Primary
     
     
     
     
    8
     
    Acquire DB2 Primary
     
     
     
     
     
     
     
    Acquire WAS Primary
     
     
     
    Post-condition/ Resource group states
     
    LDAP: ONLINE
    DB2: ONLINE
    WAS: ONLINE
     
    LDAP: Secondary
    DB2: Secondary
    WAS Secondary
     

    Since the node hosting WAS failed, the entire stack can be moved over to the opposite site—at least in this case. The primary instances are released prior to releasing the secondary instances, so that pending writes can be flushed to the secondary site.

    During acquisition, the secondary instances are acquired prior to acquiring the primary instances, so that the primary instances can mirror as soon as they come online. HACMP is responsible for placing the primary instances on site_2, taking into consideration their parent/child and location dependencies. The secondary instances are placed on available nodes on the opposite site.

    Issues are much the same as for the previous use case. This use case just reiterates the difficulties in handling the placement of the secondary instance, when the primary instance is not yet online.

    Replicated WAS-Solution, Configuration II

    Replicated WAS Solution II with Dependencies 
    

    Resource Group Policies

    All resource groups have the following policies (note that Startup and Fallback policies changed from Configuration I):

  • Startup Policy: Online On Home Node
  • Fallover Policy: Fallover to Next priority node
  • Fallback Policy: Fallback to Higher Priority Node
  • Site Policy: Prefer Primary Site
  • Participating Nodes
    Location Dependency
    Parent/Child Dependency
    DB2 [1, 2, 3, 4]
    LDAP [1, 2, 3, 4]
    WAS [2, 1, 3, 4]
    Online On The Same Site Dependent Groups:
    • DB2, LDAP, WAS
    Online On Different Nodes Dependent Groups:
    • DB2, WAS
    1. DB2 (child) depends on LDAP (parent.)
    2. WAS (child) depends on DB2 (parent.)

    Use Case 1: Start First Node (Node 2)

    Precondition: All resource groups are offline, all nodes are offline
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start
    Node 2
     
     
     
     
     
    2
     
     
    WAS goes into global ERROR state
     
    Startup policy has been met, but parents not online
    Post-condition/ Resource group states
     
     
    WAS: ERROR
     
     
     

    Since the WAS startup preference has been met, but its parent resource groups are not yet online, WAS goes into global ERROR state.

    Use Case 2: Start Second Node (Node 4)

    Precondition: See post-condition for the previous use-case
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start
    Node 4
     
     
     
     
     
    2
     
     
     
     
    Acquire:
    DB2, LDAP, and WAS Secondary
    Acquisition in parallel
    Post-condition/ Resource group states
     
     
    WAS: ERROR
     
    DB2: Secondary
    LDAP: Secondary
    WAS: Secondary
     

    The secondary instances are acquired as soon as possible. The startup preference, as well as parent/child and location dependencies, only applies to the primary instance. In a later use case we see that the fallback preference does not apply to the secondary instance either.

    Use Case 3: Start Third Node (Node 3)

    Precondition: See Post-condition for the previous use case.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start Node 3
     
     
     
     
     
    2
     
     
     
    No action
     
     
    Post-condition/ Resource group states
     
     
    WAS: ERROR
     
    DB2: Secondary
    LDAP: Secondary
    WAS: Secondary
     

    The secondary instances do not fall back to the higher priority node. Fallback would mean an outage in mirroring.

    Use Case 4: Bring Business Operation Online on Backup Site

    This use case illustrates what the administrator, as well as HACMP, has to do to bring business operations online on the secondary site. Node 1 on the primary site is not expected to ever join, all resource groups should be brought online on the secondary site, and since Node 2 is available we establish mirroring with the primary site.

    Precondition: See post-condition for the previous use-case
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    rg_move online WAS on Node 4
    /rg_move online entire stack on backup site
     
     
     
     
     
    2
     
     
     
     
    Release DB2, WAS and LDAP Secondary
     
     
     
     
    Acquire LDAP, DB2, and WAS Secondary
     
     
    Parallel Operation
     
     
     
     
    Acquire LDAP Primary
     
     
     
     
     
     
    Acquire DB2 Primary
     
     
     
     
     
     
     
    Acquire WAS Primary
     
    Post-condition/ Resource group states
     
     
    DB2: Secondary
    LDAP: Secondary
    WAS: Secondary
    DB2: ONLINE
    LDAP: ONLINE
    WAS: ONLINE
    WAS: ONLINE
     

    As part of one user-requested rg_move operation, complex resource group movements have to happen in the correct order for both instances. The secondary instances are first brought offline on the site that will serve as the business operation site, then the secondary instances are brought online on the site that will serve as the backup site, after which the primary instances are placed on their new site, taking into consideration their parent/child and location dependency constraints.

    If acquisition failures occur, the resource groups do not migrate back to their primary site, instead they go into ERROR state.

    Replicated WAS-Solution, Configuration III

    WAS Solution III - Concurrent Resource Group 
    

    This configuration is similar to the previous one, except that the DB2 resource group is a concurrent resource group and WAS and DB2 can come online on the same node.

    Resource Group Policies

    WAS and LDAP resource groups have the following policies:

  • Startup Policy: Online On Home Node
  • Fallover Policy: Fallover to Next priority node
  • Fallback Policy: Fallback to Higher Priority Node
  • Site Policy: Prefer Primary Site
  • DB2 has the following polices:

  • Startup Policy: Online On All Available Nodes
  • Fallover Policy: Bring Offline (On Error Node)
  • Fallback Policy: Never Fallback
  • Site Policy: Prefer Primary Site
  • Participating Nodes
    Location Dependency
    Parent/Child Dependency
    DB2 [1, 2, 3, 4]
    LDAP [1, 2, 3, 4]
    WAS [2, 1, 3, 4]
    Online On The Same Site Dependent Groups:
    1. DB2 LDAP WAS
    2. LDAP and DB2 should be on same node.
    1. DB2 (child) depends on LDAP (parent.)
    2. WAS (child) depends on DB2 (parent.)
  • Use Case 1: Start First Node (Node 2)

    Precondition: All resource groups are offline, all nodes are offline.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start Node 2
     
    Acquire WAS
     
     
    Goes to ERROR
    2
     
     
    Acquire DB2
     
     
    Goes to ERROR
    Post-condition/ Resource group states
     
    WAS:
    DB2:
    LDAP:
    WAS:ERROR
    DB2: ERROR
    LDAP: OFFLINE
    WAS:
    DB2:
    LDAP:
    WAS:
    DB2:
    LDAP:
     

    The home node for LDAP is Node 1. Since the resource group startup policy is Online On Home Node, LDAP could not be acquired on Node 2 at node join. Since LDAP is the parent group and is not in the ONLINE state, DB2 and WAS are put to ERROR state when the node joins.

    Use Case 2: Start First Node (Node 1)

    Precondition: As in the post condition of the previous use case.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start Node 1
     
     
     
     
     
    2
     
    Acquire LDAP
     
     
     
     
     
     
    Acquire DB2
    Acquire DB2
     
     
     
     
     
     
    Acquire WAS
     
     
     
    Post condition/ Resource group states
     
    WAS: OFFLINE WAS: ONLINE
    DB2: ONLINE
    LDAP: ONLINE
    DB2: ONLINE
    LDAP: OFFLINE
    WAS:
    DB2:
    LDAP:
    WAS:
    DB2:
    LDAP:
     

    When Node 1 starts, LDAP is started on Node 1 and Node 2 starts WAS and DB2 since the dependency condition is met.

    Use Case 3: Start First Node (Node 3)

    Precondition: As in the post condition of the previous use case.
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Start node 3
     
     
     
     
     
    2
     
     
     
    Acquire WAS_SEC DB2_SEC LDAP_SEC
     
     
    Post condition/ Resource group states
     
    WAS: OFFLINE WAS: ONLINE
    DB2: ONLINE
    LDAP: ONLINE
    DB2: ONLINE
    LDAP: OFFLINE
    WAS: Secondary DB2: Secondary
    LDAP:
    Secondary
    WAS:
    DB2:
    LDAP:
     

    Starting a node in the secondary instance activates all the secondary instances of the resource groups. Starting Node 4 does not affect the states of any resource groups. However, Node 4 acquires DB2 in ONLINE_SECONDARY state.

    Use Case 4: Fallover of DB2 on Node 1

    Precondition: All nodes in the cluster are ONLINE and the state of the resource groups is as follows:
       
    Node 1
    Node 2
    Node 3
    Node 4
     
       
    WAS: OFFLINE
    DB2: ONLINE
    LDAP: ONLINE
    WAS: ONLINE
    DB2: ONLINE LDAP: OFFLINE
    WAS: Secondary
    DB2: Secondary
    LDAP: Secondary
    WAS: OFFLINE
    DB2: Secondary
    LDAP: OFFLINE
    Step/ Qualifier
    Action
    Node 1
    Node 2
    Node 3
    Node 4
    Comments
    1
    Disk adapter Failure on Node 1
     
     
     
     
     
    2
     
    Release WAS
     
     
     
     
     
     
    Release DB2
    Release DB2
     
     
     
     
     
    Release LDAP
     
     
     
     
     
     
     
    Acquire LDAP
     
     
     
     
     
     
    Acquire DB2
     
     
     
     
     
     
    Acquire WAS
     
     
     
    Post-
    condition/ Resource group states
     
    WAS: OFFLINE
    DB2: OFFLINE
    LDAP: OFFLINE
    WAS: ONLINE
    DB2: ONLINE
    LDAP: ONLINE
    WAS: Secondary DB2: Secondary
    LDAP: Secondary
    WAS: OFFLINE
    DB2: Secondary
    LDAP: OFFLINE
     

    Since LDAP and DB2 should be online on the same node, although DB2 is a concurrent resource group, a failure on Node 1 results in moving LDAP (Online On The Same Node Dependent) to Node 2. Because LDAP is a parent the result is all the resource groups are bounced as a result of adapter failure.


    PreviousNextIndex