Table of Contents
Chapter 1: Administering an HACMP Cluster
Options for Configuring an HACMP Cluster
Configuring HACMP Using the Standard Configuration Path
Configuring HACMP Using the Extended Configuration Path
Verifying and Synchronizing the Configuration
Starting and Stopping Cluster Services
Maintaining Shared Logical Volume Manager Components
Managing Cluster Resource Groups
Managing Users and Groups in a Cluster
Managing Cluster Security and Inter-Node Communications
Understanding the /usr/es/sbin/cluster/etc/rhosts File
Saving and Restoring HACMP Cluster Configurations
Additional HACMP Maintenance Tasks
Troubleshooting an HACMP Cluster
Maintaining Highly Available Applications
AIX 5L Files Modified by HACMP
Chapter 2: Administering a Cluster Using WebSMIT
Configuring HACMP Using WebSMIT
Configuring and Managing Nodes and Networks in WebSMIT
Configuring and Managing Resources in WebSMIT
Viewing the Cluster Components
Viewing Cluster Configuration Information in WebSMIT
Viewing HACMP Documentation in WebSMIT
Chapter 3: Configuring an HACMP Cluster (Standard)
Prerequisite Tasks for Using the Standard Path
Assumptions and Defaults for the Standard Path
Steps for Configuring a Cluster Using the Initialization and Standard Configuration Path
Configuring a Two-Node Cluster, or Using Smart Assists
Configuring Applications with the General Configuration Smart Assist
Defining HACMP Cluster Topology (Standard)
Configuring HACMP Resources (Standard)
Configuring Application Servers
Configuring HACMP Service IP Labels/Addresses
Configuring Volume Groups, Logical Volumes, and Filesystems as Cluster Shared Resources
Configuring Concurrent Volume Groups, Logical Volumes, and Filesystems
Configuring HACMP Resource Groups (Standard)
Creating HACMP Resource Groups Using the Standard Path
Configuring Resources in Resource Groups (Standard)
Resource Group Configuration Considerations
Assigning Resources to Resource Groups (Standard)
Verifying and Synchronizing the Standard Configuration
Procedure to Verify and Synchronize the HACMP Configuration
Viewing the HACMP Configuration
Additional Configuration Tasks
Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended)
Understanding the Extended Configuration Options
Steps for Configuring an HACMP Cluster Using the Extended SMIT Menu
Discovering HACMP-Related Information
Configuring Cluster Topology (Extended)
Configuring HACMP Networks and Heartbeat Paths
Configuring Communication Interfaces/Devices to HACMP
Configuring Heartbeating over Disk
Configuring HACMP Persistent Node IP Labels/Addresses
Configuring Node-Bound Service IP Labels
Configuring HACMP Global Networks
Configuring HACMP Network Modules
Configuring Topology Services and Group Services Logs
Configuring HACMP Resources (Extended)
Configuring Service IP Labels as HACMP Resources
Configuring HACMP Application Servers
Configuring Volume Groups, Logical Volumes, and Filesystems as Resources
Configuring Concurrent Volume Groups, Logical Volumes, and Filesystems as Resources
Configuring Multiple Application Monitors
Steps for Configuring Multiple Application Monitors
Configuring Tape Drives as HACMP Resources
Configuring AIX 5L Fast Connect
Configuring Highly Available Communication Links
Configuring SNA-Over-LAN Communication Links
Configuring X.25 Communication Links
Configuring SNA-Over-X.25 Communication Links
Notes on Application Service Scripts for Communication Links
Chapter 5: Configuring HACMP Resource Groups (Extended)
Limitations and Prerequisites for Configuring Resource Groups
Steps for Configuring Resource Groups in SMIT
Dynamic Node Priority Policies
Configuring Resource Group Runtime Policies
Configuring Dependencies between Resource Groups
Considerations for Dependencies between Resource Groups
Steps to Configure Dependencies between Resource Groups
Configuring Resource Groups with Dependencies
Configuring Processing Order for Resource Groups
Reconfiguration, Startup, and Shutdown of WLM by HACMP
Configuring a Settling Time for Resource Groups
Defining Delayed Fallback Timers
Assigning a Delayed Fallback Policy to a Resource Group
Using the Node Distribution Startup Policy
Adding Resources and Attributes to Resource Groups Using the Extended Path
Steps for Adding Resources and Attributes to Resource Groups (Extended Path)
Customizing Inter-Site Resource Group Recovery
Enabling or Disabling Selective Fallover between Sites
Relinquishing Control over NFS Filesystems in an HACMP Cluster
NFS Exporting Filesystems and Directories
Forcing a Varyon of Volume Groups
When HACMP Attempts a Forced Varyon
Avoiding a Partitioned Cluster
Verification Checks for Forced Varyon
Chapter 6: Configuring Cluster Events
Considerations for Pre- and Post-Event Scripts
Using Shell Environment Variables in Pre- and Post-Event Scripts
event_error Now Indicates Failure on a Remote Node
Parallel Processing of Resource Groups Affects Event Processing
Dependent Resource Groups and the Use of Pre- and Post-Event Scripts
Configuring Pre- and Post-Event Commands
Configuring Pre- and Post- Event Processing
Configuring User-Defined Events
Changing or Showing User-Defined Events
Tuning Event Duration Time Until Warning
Changing Event Duration Time Until Warning
Configuring a Custom Remote Notification Method
Defining a Remote Notification Method
Changing or Removing a Custom Remote Notification Method
Chapter 7: Verifying and Synchronizing an HACMP Cluster
Automatic Verification and Synchronization
Understanding the HACMP Cluster Verification Process
Cluster Verification during a Dynamic Cluster Reconfiguration Event
Parameters Automatically Corrected
Understanding the Detailed Phases of Verification
Verifying the HACMP Configuration Using SMIT
Verifying and Synchronizing a Cluster Configuration
Verifying and Synchronizing the Cluster Configuration
Running Corrective Actions during Verification
Managing HACMP File Collections
Default HACMP File Collections
Options for Propagating an HACMP File Collection
Using SMIT to Manage HACMP File Collections
Adding a Custom Verification Method
Changing or Showing a Custom Verification Method
Removing a Custom Verification Method
Chapter 8: Testing an HACMP Cluster
Launching the Cluster Test Tool
Modifying Logging and Stopping Processing in the Cluster Test Tool
Understanding Automated Testing
Setting up Custom Cluster Testing
Creating a Custom Test Procedure
Specifying Parameters for Tests
Network Tests for an IP Network
Network Interface Tests for IP Networks
Network Tests for a Non-IP Network
Running Custom Test Procedures
Launching a Custom Test Procedure
Criteria for Test Success or Failure
Recovering the Control Node after Cluster Manager Stops
How to Avoid Manual Intervention
Customizing the Types of Information to Collect
Adding Data from hacmp.out to the Cluster Test Tool Log File
Fixing Problems when Running Cluster Tests
Cluster Test Tool Stops Running
Control Node Becomes Unavailable
Cluster Does Not Return to a Stable State
Testing Does Not Progress as Expected
Chapter 9: Starting and Stopping Cluster Services
A Note on Application Monitors
Procedure for Starting Cluster Services
Modifying the Startup of Cluster Services
Procedure for Stopping Cluster Services
Stopping HACMP Cluster Services without Stopping Applications
Abnormal Termination of Cluster Manager Daemon
AIX 5L Shutdown and Cluster Services
Stopping HACMP Cluster Services and RSCT
Maintaining Cluster Information Services
Enabling Clinfo for Asynchronous Event Notification
Chapter 10: Monitoring an HACMP Cluster
Periodically Monitoring an HACMP Cluster
Automatic Cluster Configuration Monitoring
Tools for Monitoring an HACMP Cluster
Monitoring a Cluster with HAView
HAView Installation Requirements
HAView File Modification Considerations
Tivoli NetView Hostname Requirements for HAView
Viewing Clusters and Components
Obtaining Component Details in HAView
Customizing HAView Polling Intervals
Removing a Cluster from HAView
Using the HAView Cluster Administration Utility
Monitoring Clusters with Tivoli Distributed Monitoring
Cluster Monitoring and Cluster Administration Options
Using Tivoli to Monitor the Cluster
Using Tivoli to Perform Cluster Administration Tasks
Uninstalling HACMP-Related Files from Tivoli
Monitoring Clusters with clstat
Viewing clstat in ASCII Display Mode
Viewing clstat in X Window System Display Mode
Viewing clstat with a Web Browser
A Note on Application Monitors
Displaying an Application-Centric Cluster View
Measuring Application Availability
Planning and Configuring for Measuring Application Availability
Configuring and Using the Application Availability Analysis Tool
Using Resource Groups Information Commands
Using HACMP Topology Information Commands
Monitoring Cluster Services on a Node
Monitoring Cluster Services on a Client
Size of /var Filesystem May Need to Be Increased
/tmp/clsmuxtrmgr.debug Log File
/usr/es/sbin/cluster/history/cluster.mmddyyyy File
/var/hacmp/clcomd/clcomd.log File
/var/hacmp/clcomd/clcomddiag.log File
/var/hacmp/clverify/clverify.log File
/var/hacmp/log/clutils.log File
/var/ha/log/grpsvcs.<filename> File
/var/ha/log/topsvcs.<filename> File
Chapter 11: Managing Shared LVM Components
Understanding C-SPOC and Its Relation to Resource Groups
Updating LVM Components in an HACMP Cluster
Lazy Update Processing in an HACMP Cluster
Forcing an Update before Fallover
Maintaining Shared Volume Groups
Understanding Active and Passive Varyon in Enhanced Concurrent Mode
Collecting Information on Current Volume Group Configuration
Importing Shared Volume Groups
Creating a Shared Volume Group with C-SPOC
Setting Characteristics of a Shared Volume Group
Mirroring a Volume Group Using C-SPOC
Unmirroring a Volume Group Using C-SPOC
Synchronizing Volume Group Mirrors
Synchronizing a Shared Volume Group Definition
Adding a Logical Volume to a Cluster Using C-SPOC
Setting Characteristics of a Shared Logical Volume Using C-SPOC
Changing a Shared Logical Volume
Removing a Logical Volume Using C-SPOC
Synchronizing LVM Mirrors by Logical Volume
Maintaining Shared Filesystems
Journaled Filesystem and Enhanced Journaled Filesystem
Creating Shared Filesystems with C-SPOC
Adding the Filesystem to an HACMP Cluster Logical Volume
Changing a Shared Filesystem in HACMP Using C-SPOC
Removing a Shared Filesystem Using C-SPOC
Adding a Disk Definition to Cluster Nodes Using C-SPOC
Removing a Disk Definition on Cluster Nodes Using C-SPOC
Using SMIT to Replace a Cluster Disk
Managing Data Path Devices with C-SPOC
Configuring Cross-Site LVM Mirroring
Steps to Configure Cross-Site LVM Mirroring
Showing and Changing Cross-Site LVM Mirroring Definition
Removing a Disk from a Cross-Site LVM Mirroring Site Definition
Troubleshooting Cross-Site LVM Mirroring
Chapter 12: Managing Shared LVM Components in a Concurrent Access Environment
Understanding Concurrent Access and HACMP Scripts
Maintaining Concurrent Access Volume Groups
Activating a Volume Group in Concurrent Access Mode
Determining the Access Mode of a Volume Group
Restarting the Concurrent Access Daemon (clvmd)
Verifying a Concurrent Volume Group
Maintaining Concurrent Volume Groups with C-SPOC
Creating a Concurrent Volume Group on Cluster Nodes Using C-SPOC
Converting Volume Groups to Enhanced Concurrent Mode
Listing All Concurrent Volume Groups in the Cluster
Importing a Concurrent Volume Group with C-SPOC
Extending a Concurrent Volume Group with C-SPOC
Enabling or Disabling Cross-Site LVM Mirroring
Removing a Physical Volume from a Concurrent Volume Group with C-SPOC
Mirroring a Concurrent Volume Group Using C-SPOC
Unmirroring a Concurrent Volume Group Using C-SPOC
Synchronizing Concurrent Volume Group Mirrors
Maintaining Concurrent Logical Volumes
Listing All Concurrent Logical Volumes in the Cluster
Adding a Concurrent Logical Volume to a Cluster
Removing a Concurrent Logical Volume
Setting Characteristics of a Concurrent Logical Volume
Chapter 13: Managing the Cluster Topology
Reconfiguring a Cluster Dynamically
Requirements before Reconfiguring
Managing Communication Interfaces in HACMP
Configuring Communication Interfaces/Devices to the Operating System on a Node
Updating HACMP Communication Interfaces/Devices with AIX 5L Settings
Swapping IP Addresses between Communication Interfaces Dynamically
Replacing a PCI Hot-Pluggable Network Interface Card
Changing the Configuration of Cluster Nodes
Adding a Cluster Node to the HACMP Configuration
Removing a Cluster Node from the HACMP Configuration
Changing the Name of a Cluster Node
Changing the Configuration of an HACMP Network
Converting an HACMP Network to use IP Aliasing
Establishing Default and Static Routes on Aliased Networks
Converting an SP Switch Network to an Aliased Network
Controlling Distribution Preferences for Service IP Label Aliases
Changing the Configuration of Communication Interfaces
Configuring Multiple Logical Interfaces on the Same ATM NIC
Adding HACMP Communication Interfaces/Devices
Removing a Communications Interface from a Cluster Node
Managing Persistent Node IP Labels
Configuring Persistent Node IP Labels/Addresses
Changing Persistent Node IP Labels
Deleting Persistent Node IP Labels
Changing the Configuration of a Global Network
Adding an HACMP Network to a Global Network
Removing an HACMP Network from a Global Network
Changing the Configuration of a Network Module
Understanding Network Module Settings
Resetting the Network Module Tunable Values to Defaults
Behavior of Network Down on Serial Networks
Changing the Failure Detection Rate of a Network Module
Changing an RS232 Network Module Baud Rate
Changing the Configuration of a Site
Synchronizing the Cluster Configuration
Dynamic Reconfiguration Issues and Synchronization
Releasing a Dynamic Reconfiguration Lock
Chapter 14: Managing the Cluster Resources
Dynamic Reconfiguration: Overview
Reconfiguring a Cluster Dynamically
Requirements before Reconfiguring
Dynamic Cluster Resource Changes
Reconfiguring Application Servers
Changing an Application Server
Removing an Application Server
Changing or Removing Application Monitors
Suspending and Resuming Application Monitoring
Changing the Configuration of an Application Monitor
Removing an Application Monitor
Reconfiguring Service IP Labels as Resources in Resource Groups
Steps for Changing the Service IP Labels/Addresses Definitions
Changing Distribution Preference for Service IP Label Aliases
Viewing Distribution Preference for Service IP Label Aliases
Reconfiguring Communication Links
Changing Communication Adapter Information
Removing a Communication Adapter from HACMP
Changing Communication Link Information
Removing a Communication Link from HACMP
Reconfiguring Tape Drive Resources
Removing a Tape Device Resource
Reconfiguring Resources in Clusters with Dependent Resource Groups
Reconfiguring Resources and Topology Dynamically
Making Dynamic Changes to Dependent Resource Groups
Cluster Processing During DARE in Clusters with Dependent Resource Groups
Synchronizing Cluster Resources
Chapter 15: Managing Resource Groups in a Cluster
Reconfiguring Cluster Resources and Resource Groups
Changing Resource Group Processing Order
Resource Group Ordering during DARE
Changing the Configuration of a Resource Group
Changing Resource Group Attributes
Changing a Dynamic Node Priority Policy
Changing a Delayed Fallback Timer Policy
Showing, Changing, or Deleting a Settling Time Policy
Changing a Location Dependency between Resource Groups
Changing a Parent/Child Dependency between Resource Groups
Displaying a Parent/Child Dependency between Resource Groups
Removing a Dependency between Resource Groups
Adding or Removing Individual Resources
Reconfiguring Resources in a Resource Group
Forcing a Varyon of a Volume Group
Requirements before Migrating a Resource Group
Migrating Resource Groups with Dependencies
Migrating Resource Groups Using SMIT
Migrating Resource Groups from the Command Line
Special Considerations when Stopping a Resource Group
Migrating Resource Groups with Replicated Resources
Customizing Inter-Site Resource Group Recovery
Chapter 16: Managing User and Groups
Requirements for Managing User Accounts in an HACMP Cluster
Managing User Accounts across a Cluster
Listing Users On All Cluster Nodes
Adding User Accounts on all Cluster Nodes
Changing Attributes of User Accounts in a Cluster
Removing User Accounts from a Cluster
Managing Password Changes for Users
Prerequisites for Allowing Users to Change Passwords
Allowing Users to Change Their Own Passwords
Configuring the Cluster Password Utility
Changing Passwords for User Accounts
Changing the Password for Your Own User Account
Listing Groups on All Cluster Nodes
Adding Groups on Cluster Nodes
Changing Characteristics of Groups in a Cluster
Removing Groups from the Cluster
Chapter 17: Managing Cluster Security
Configuring Connection Authentication
Setting the HACMP Security Mode
Setting Up Cluster Communications over a VPN
Configuring Message Authentication and Encryption
About Configuring Message Authentication and Encryption
Configuring Message Authentication and Encryption using Automatic Key Distribution
Configuring Message Authentication and Encryption using Manual Key Distribution
Changing the Security Authentication Mode
Troubleshooting Message Authentication and Encryption
Chapter 18: Saving and Restoring Cluster Configurations
Relationship between the OLPW Cluster Definition File and a Cluster Snapshot
Information Saved in a Cluster Snapshot
Defining a Custom Snapshot Method
Changing or Removing a Custom Snapshot Method
Creating (Adding) a Cluster Snapshot
Dynamic Changes and Cluster Snapshots
Appendix B: Resource Group Behavior during Cluster Events
Appendix C: HACMP for AIX Commands
Appendix D: RSCT: Resource Monitoring and Control Subsystem
Appendix E: Using DLPAR and CUoD in an HACMP Cluster