![]() ![]() ![]() |
Appendix C: GPFS Cluster Configuration
This appendix describes how to configure, manage, and remove an IBM General Parallel File System (GPFS) version 2.3 cluster within an HACMP cluster environment, using the HACMP SMIT interface.
You can also configure and manage GPFS clusters using the native GPFS command set (mmxxx). Do not mix these two methods of configuring and managing a GPFS cluster; choose one or the other. For complete documentation on GPFS including restrictions and how to use the native GPFS command set, see the following URL:
http://www.ibm.com/servers/eserver/clusters/software/gpfs.html
Before configuring GPFS, the hardware must be set up as required by GPFS and all restrictions imposed by GPFS apply. This appendix contains these sections:
Overview
GPFS 2.3 provides concurrent high speed file access to applications executing on multiple systems that are part of an HACMP cluster. It offers:
High-performance filesystem for IBM UNIX® clusters capable of supporting multi-terabytes of storage within a single filesystem Shared-disk filesystem where every GPFS cluster node can have concurrent read/write access to the same file High availability through automatic recovery from node and disk failures. GPFS in an HACMP Cluster Environment
HACMP and AIX 5L provide the operating and administrative environment for GPFS 2.3 in a cluster environment. GPFS uses some part of the CLVM component of AIX 5L and also has its own data replication method.
GPFS uses the topology framework of an HACMP cluster: Nodes, networks, and network interfaces used by a GPFS cluster are configured as part of the HACMP cluster. Only one GPFS cluster can be configured per HACMP cluster.
GPFS Nodes
Each node in the GPFS cluster can be defined as belonging to a GPFS nodeset. A GPFS cluster can contain multiple nodesets. All nodes within a GPFS nodeset have access to the same GPFS filesystems and must have access to all the directly-attached disks that form part of the GPFS cluster.
Note: When using the HACMP SMIT panels to configure a GPFS cluster, all the cluster nodes are included in one GPFS nodeset.
GPFS cluster nodes can host HACMP cluster resources as well as GPFS.
GPFS Network and Network Interfaces
One IP network is defined for communication between GPFS cluster nodes. This network should not be used for service by HACMP. You can either use node-bound service IP interfaces on a network or base interfaces on networks using IPAT via IP Aliases.
Note: When you are configuring GPFS on SP nodes running HACMP using the HACMP SMIT method, you will get an error if you configure the SP Switch network for use by GPFS. To use GPFS on SP nodes, configure another IP network in HACMP that either has node-bound service addresses defined on it, or uses IPAT via IP Aliases.
GPFS Disk Devices
The disk devices are not defined to the HACMP cluster as HACMP resources; they are defined only for use by GPFS. If you include the GPFS disk, volume group, or filesystem in an HACMP resource group, verification reports this as an error.
Planning for a GPFS Cluster
See the GPFS 2.3 manuals for extensive information on planning for GPFS in HACMP clusters.
Basic Configuration Considerations for GPFS Clusters
Basic considerations for planning the GPFS set up:
GPFS uses an IP network to connect all of the nodes. This is typically a LAN with sufficient bandwidth (minimum 100MB/sec bandwidth) available for GPFS control traffic. The IP network and network interfaces are configured within the HACMP cluster. GPFS requires invariant network connections; network interfaces supporting TCP/IP socket connections for GPFS cannot be configured for Hardware Address Takeover (HWAT). Use a single service network interface per network, or base interfaces on networks using IPAT via IP Aliases.
GPFS requires that the SSA or Fibre Channel disk devices be configured and directly attached on all of the nodes that will mount the filesystems. The disks, volume groups, and filesystems must not be configured within the HACMP cluster resource groups. Tuning the HACMP Cluster for GPFS
Tune the cluster before you configure GPFS. Components to consider include:
Communications I/O
Set the network option ipqmaxlen to 512 (the default is 128). This parameter controls the number of incoming packets that can exist on the IP interrupt queue.
no -o ipmaxlen=512
Since this option must be modified at every reboot, it is suggested to place it at the end of one of the system startup files, such as the /etc/rc.net shell script.
Disk I/O
Set the SSA disk I/O option max_coalesce parameter to allow the SSA device driver to coalesce requests that have been broken up to satisfy LVM requirements:
chdev -1 hdiskX - a max_coalesce=0x40000
Security
Ensure you have an ~/.rhosts file on every node in the GPFS cluster. You must be the root user.
Installing GPFS
Before you install GPFS, install the correct levels of the following software:
AIX 5L HACMP RSCT. You can configure a GPFS Cluster using HACMP SMIT. To use this capability, be sure to install the cluster.es.cfs.rte fileset.
For GPFS installation instructions, see the GPFS version 2.3 documentation.
Configuring a GPFS Cluster Using HACMP SMIT
A Configure GPFS entry is available on the System Management (C-SPOC) SMIT panel if you have installed the GPFS fileset (cluster.es.cfs.rte). Selecting this item leads to the following menu of actions available for configuring and manipulating a GPFS cluster and the GPFS filesystems:
Configure a GPFS Cluster - create and configure a GPFS cluster List All GPFS Filesystems - List all configured GPFS filesystems Add GPFS Filesystem - Add a GPFS filesystem to an existing GPFS cluster Remove GPFS Filesystem - Remove existing GPFS filesystem(s) Remove a GPFS Cluster - Remove a GPFS cluster and all filesystems. Note: The use of the HACMP SMIT panels is limited to GPFS clusters created and configured using these panels. You cannot use the Add or Remove function on any previously configured GPFS cluster or GPFS filesystem. However, the List option will list all GPFS filesystems, no matter the method used to configure them.
Prerequisites and Limitations
Using the Configure GPFS panels simplifies the process of creating and removing the GPFS cluster components. However, certain conditions apply:
You can have one GPFS cluster per HACMP cluster. No previously configured GPFS cluster may exist in the HACMP cluster. All nodes in the HACMP cluster will be included in the GPFS cluster created by HACMP SMIT. If you are using Fibre Channel connected disks, you must have at least three nodes in the cluster (for GPFS quorum). This is a GPFS limitation. You must have configured and synchronized an HACMP cluster with topology that includes one IP-based network that includes all nodes, with node-bound service IP label/addresses (for exclusive use of GPFS) or non-service interfaces on networks using IPAT via IP Aliases.
All nodes must be up and running HACMP. Step 1: Create the HACMP and GPFS Cluster Topology
Use the HACMP SMIT panels to configure the cluster name, nodes, networks and network interfaces that will be used by the GPFS cluster. For information about how to configure the topology from HACMP SMIT, see Chapter 4: Configuring HACMP Cluster Topology and Resources (Extended) in the Administration Guide.
You must include at least one IP-based network that includes all nodes, with node-bound service IP label/addresses (for exclusive use of GPFS) or non-service interfaces on networks using IPAT via IP Aliases.
Step 2: Create the GPFS Cluster Using SMIT
From the main HACMP menu, select System Management (C-SPOC) > Configure GPFS > Configure a GPFS Cluster.
The software performs the following steps to create and configure the GPFS cluster:
Check that no GPFS cluster is currently configured Check that an HACMP network with a node-bound service IP label/address on each node is available for the GPFS network. If none, check for non-service interfaces on networks using IPAT via IP Aliases.
Select the first node listed as the primary server Create a nodeset with the name HAgpfs01 that includes all the HACMP cluster nodes Run the mmcrcluster command to create the GPFS cluster Verify and synchronize the GPFS cluster configuration. Notes on the GPFS Configuration Created Using SMIT
If you have configured more than one network that meets the criteria for the GPFS cluster, the networks are sorted by name and the first one in the list is picked as the GPFS network. (HACMP checks for a network with node-bound service IP labels/addresses first.)
The GPFS network interface information is stored along with the GPFS Cluster ID. HACMP uses this information to recognize the presence of a GPFS cluster in the HACMP environment.
The GPFS cluster created by using the HACMP SMIT panels will always contain all the HACMP cluster nodes. The nodeset is named HAgpfs01.
The first node listed is selected as a primary GPFS cluster data server node.
Step 3: Add the GPFS Filesystem
To add the GPFS filesystem:
1. Enter smit hacmp
2. In SMIT, select System Management (C-SPOC) > Configure GPFS and press Enter.
3. Select the Add a GPFS Filesystem SMIT panel and press Enter.
4. Fill in the fields as follows to set up the GPFS filesystem:
5. Press Enter. The GPFS daemon is started if it is not already running. The network shared disk is created on the chosen hdisk. The filesystem is created and mounted on each cluster node.
If there are any problems, error messages are printed to the screen.
HACMP Cluster Operations and GPFS
As administrator of the HACMP cluster with GPFS, be aware of the effect on GPFS when you make any changes to the cluster.
GPFS assumes that IP addresses, node names, and hostnames remain constant. With this in mind, if you make HACMP cluster topology configuration changes, take care that GPFS is not affected. For example:
Before changing a cluster node’s name, IP address, or hostname, remove the node from the cluster, make the changes, then add it back to the cluster. You can use a DARE operation to remove the node and add it back in. If a network interface fails, quorum may be affected if the gpfs daemon stops on a node. GPFS may not be available. GPFS does not support the use of backup network interfaces; therefore HACMP cannot take corrective action. Using the HACMP SMIT panels to configure and manipulate the GPFS cluster ensures that these actions are done properly and issues a warning if quorum is affected. The verification utility reports errors if the GPFS network attributes are changed.
GPFS Cluster Dynamic Reconfiguration Operations
If you configured the GPFS cluster using the HACMP SMIT method, you can make the following changes to HACMP cluster topology while GPFS is active (following the HACMP DARE rules):
You can add or remove a cluster node as part of a dynamic reconfiguration. These operations will cause the GPFS cluster to be reconfigured to reflect the change. During reconfiguration the GPFS cluster is stopped temporarily, then restarted after the node is added or deleted. If you delete the GPFS primary node, the software will reassign the primary. Adding a node dynamically to the HACMP configuration also adds it to the GPFS cluster configuration. However, you then have to mount the GPFS filesystem on the new node the first time when cluster services are started.
Swap a GPFS service network interface using the Swap Network Adapter option on the Cluster Communications Adapter Management SMIT panel if there is a non-service network interface configured. The GPFS filesystem may be unavailable for a short time. Use the PCI Hot Plug Replace Network Adapter option on the Cluster Communications Adapter Management SMIT panel to replace a GPFS service network interface. The verification utility reports errors if you have performed an operation that affects the GPFS cluster.
Verifying the Modified GPFS Cluster
You do not need to run verification after you first create the GPFS cluster. However, you should verify and synchronize after making any change to HACMP configuration.
Running the verification utility on a GPFS cluster:
Checks that networks, network interfaces, and disk access are all correctly configured for GPFS Prints a message reminding you that the GPFS filesystem must be manually mounted on the new node(s) after starting the HACMP cluster services. If you are adding node(s) to a cluster that previously had two nodes, the GPFS daemon must be stopped on all nodes and started with multi-node quorum. An onscreen message indicates that the GPFS filesystem will be unavailable during this time. Prints an informational message that the removed node will be automatically deleted from the GPFS cluster configuration and GPFS filesystems will not be available on that node Prints a WARNING message if you deleted a node and the number of remaining nodes in the GPFS cluster causes a quorum issue. (On dynamic reconfiguration only) If there are fewer nodes in the GPFS cluster than in the HACMP node, a forward check is made to ensure that the new node(s) contains a network interface of the same kind, on the same subnet and network that is configured for use by GPFS. Listing GPFS Filesystems Using HACMP SMIT
To list GPFS filesystems:
1. Enter smit hacmp
2. In SMIT, select System Management (C-SPOC) > Configure GPFS > List All GPFS Filesystems. The system lists the information for all the GPFS filesystems files found in /etc/filesystems, similar to the following sample output:
============================================================ Filesystem Name: gpfs1 Filesystem Mount Dir: /gpfs1 Filesystem Details: flag value description ---- -------------- ------------------------------------------------------------ -s roundRobin Stripe method -f 8192 Minimum fragment size in bytes -i 512 Inode size in bytes -I 16384 Indirect block size in bytes -m 1 Default number of metadata replicas -M 1 Maximum number of metadata replicas -r 1 Default number of data replicas -R 1 Maximum number of data replicas -a 1048576 Estimated average file size -n 32 Estimated number of nodes that will mount file system -B 262144 Block size -Q none Quotas enforced none Default quotas enabled -F 33792 Maximum number of inodes -V 5.01 File system version. Highest supported version: 5.01 -z no Is DMAPI enabled? -d gpfs0lv;gpfs1lv Disks in file system -A yes Automatic mount option -C gpfsns1 GPFS nodeset identifier -E no Exact mtime default mount option -S no Suppress atime default mount optionRemoving a GPFS Filesystem Using HACMP SMIT
If you configured the GPFS filesystem using HACMP SMIT, you can remove it with the GPFS Cluster Remove GPFS Filesystem option.
To remove a GPFS filesystem:
1. Enter smit hacmp
2. In SMIT, select System Management (C-SPOC) > Configure GPFS and press Enter.
3. Select the Remove GPFS Filesystem option.
4. Enter the following information and press Enter:
Note: Although the filesystems are deleted, the network shared disks and volume groups used by the GPFS filesystems are left untouched. Remove them manually if you do not want them. See the GPFS documentation.
Removing a GPFS Cluster Using HACMP SMIT
If you configured the GPFS cluster using HACMP SMIT, you can remove it with the Remove a GPFS Cluster option. First remove all GPFS filesystems. You can force-delete the GPFS cluster even when there are active filesystems, by setting the Remove All GPFS Filesystems option to TRUE.
To force-delete a GPFS cluster:
1. Enter smit hacmp
2. In SMIT, select System Management (C-SPOC) > Configure GPFS.
3. Select the Remove a GPFS Cluster option.
4. Choose whether to force-delete the active filesystems or not and press Enter.
A message appears warning you that this action will remove all GPFS cluster information and all GPFS filesystems. SMIT prompts you to confirm your selections.
5. To complete the removal, press Enter again.
If a cluster node is not available when the mmdelnode command runs, GPFS cannot remove its configuration files. In this case, you will have to manually remove the files from that node. See the GPFS documentation for the procedure.
Removing the HACMP Cluster Definition of a GPFS Cluster
Remove the GPFS cluster definition before removing the HACMP cluster definition. If you remove the HACMP cluster definition first, the GPFS cluster definition still exists and you will have to manually delete it from the configuration database.
Troubleshooting the GPFS Cluster
The /usr/sbin/cluster/utilities/clgetesdbginfo log file includes GPFS logs for debugging purposes.
If you use the HACMP SMIT panels to create the GPFS cluster, then when you take a cluster snapshot the GPFS configuration file /var/mmfs/gen/mmsdrfs is saved in the HACMP cluster snapshot .info file. However, restoring and applying snapshots does not create a GPFS cluster. You must recreate it following the procedures outlined here after restoring the HACMP cluster.
GPFS ODM (HACMPgpfs) Data
If the HACMP GPFS fileset is installed, the HACMPgpfs ODM is added to the HACMP cluster configuration. This ODM is used to store the configuration data that GPFS uses for its operation. The ODM stores one data entry per one HACMP node. This ODM consists of the following fields:
HACMPgpfs:
{ gpfs_clusterid [20]; gpfs_version [25]; nodeset_name [10] gpfs_network [33]; identifier [32]; interfacename [33]; nodename [33]; }References
Sizing and Tuning GPFS: SG24-5610 GPFS in an RS/6000 Cluster: SG24-6035 IBM GPFS Concepts, Planning, and Installation Guide: GA22-7453 IBM GPFS for AIX 5L Administration and Programming Reference: SA22-7452 AIX 5L Performance Management Guide
![]() ![]() ![]() |