Scribd
Upload
51 views6 pages

GPFS Cluster Setup on AIX Systems

Install the GPFS software on both nodes using installp, create a cluster with mmcrcluster designating one node as the manager and quorum, add the second node with mmaddnode, create NSDs across both nodes' disks by defining disk stanzas in a file and using mmcrnsd.

Uploaded by

raja rajan
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
51 views6 pages

GPFS Cluster Setup on AIX Systems

Install the GPFS software on both nodes using installp, create a cluster with mmcrcluster designating one node as the manager and quorum, add the second node with mmaddnode, create NSDs across both nodes' disks by defining disk stanzas in a file and using mmcrnsd.

Uploaded by

raja rajan
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

/usr/lpp/mmfs/bin

mmlsconfig
mmlscluster
mmstartup -a
mmgetstate
mmcrnsd -F /tmp/[Link] -v no
mmlsnsd� & �mmlspv
mmcrfs /mountpoint lvname nsd-diskname�

mmmount all
mmlsfs� �mmdf
mmdf nsd-dskname (gpfslv2)
mmadddisk
1 crnsd
[Link]
mmadddisk gpfslv2 -F /tmp/disk.file6
mmlsdisk lvname
mmdeldisk gpfslv2 -F /tmp/disk.file6

mmexportfs all -o /tmp/[Link]


to cronfimr or cross check mmlsnsd

� MPIO on AIX = hdisk


� SDD (non-MPIO) on AIX = vpath
� PowerPath by EMC = hdiskpower
� HDLM by Hitatchi = dlmfdrv
mmimportfs all -i /tmp/[Link]

[Link]
General+Parallel+File+System+(GPFS)/page/
Install+and+configure+a+GPFS+cluster+on+AIX

Objectives
Verify the system environment
Create a GPFS cluster
Define NSD's
Create a GPFS file system
You will need

Requirements for this lab (not necessarily GPFS minimum requirements):

Two AIX 6.1 or 7.1 operating systems (LPARs)


Very similar to Linux installation. AIX LPP packages replace the Linux RPMs, some
of the administrative commands are different.
At least 4 hdisks

Step 1: Verify Environment

Verify nodes properly installed


Check that the operating system level is supported

On the system run oslevel

Check the GPFS


FAQ:[Link]
gpfs_faqs/[Link]
Is the installed OS level supported by GPFS? Yes No
Is there a specific GPFS patch level required for the installed OS? Yes No
If so what patch level is required? ___________
Verify nodes configured properly on the network(s)
Write the name of Node1: ____________
Write the name of Node2: ____________
From node 1 ping node 2
From node 2 ping node 1

If the pings fail, resolve the issue before continuing.


Verify node-to-node ssh communications (For this lab you will use ssh and scp for
secure remote commands/copy)
On each node create an ssh-key. To do this use the command ssh-keygen; if you don't
specify a blank passphrase, -N, then you need to press enter each time you are
promoted to create a key with no passphrase until you are returned to a prompt. The
result should look something like this:
# ssh-keygen -t rsa -N "" -f $HOME/.ssh/id_rsa
Generating public/private rsa key pair.
Created directory '/.ssh'.
Your identification has been saved in /.ssh/id_rsa.
Your public key has been saved in /.ssh/id_rsa.pub.
The key fingerprint is:
7d:06:95:45:9d:7b:7a:6c:64:48:70:2d:cb:78:ed:61
root@node1
On node1 copy the $HOME/.ssh/id_rsa.pub file to $HOME/.ssh/authorized_keys
# cp $HOME/.ssh/id_rsa.pub $HOME/.ssh/authorized_keys
From node1 copy the $HOME/.ssh/id_rsa.pub file from node2 to /tmp/id_rsa.pub
# scp node2:/.ssh/id_rsa.pub /tmp/id_rsa.pub
Add the public key from node2 to the authorized_keys file on node1
# cat /tmp/id_rsa.pub >> $HOME/.ssh/authorized_keys
Copy the authorized key file from node1 to node2
# scp $HOME/.ssh/authorized_keys node2:/.ssh/authorized_keys
To test your ssh configuration ssh as root from node 1 to node1 and node1 to node2
until you are no longer prompted for a password or for addition to the known_hosts
file.

node1# ssh node1 date


node1# ssh node2 date
node2# ssh node1 date
node2# ssh node2 date
Supress ssh banners by creating a .hushlogin file in the root home directory
# touch $HOME/.hushlogin
Verify the disks are available to the system

For this lab you should have 4 disks available for use hdiskw-hdiskz.
Use lspv to verify the disks exist
Ensure you see 4 unused disks besides the existing rootvg disks and/or other volume
groups.

Step 2: Install the GPFS software

On node1

Locate the GPFS software in /yourdir/gpfs/base/


# cd /yourdir/gpfs/base/
Run the inutoc command to create the table of contents, if not done already
# inutoc .
Install the base GPFS code using the installp command
# installp -aXY -d/yourdir/gpfs/base all
Locate the latest GPFS updates in /yourdir/gpfs/fixes/
# cd /yourdir/gpfs/fixes/
Run the inutoc command to create the table of contents, if not done already
# inutoc .
Install the GPFS PTF updates using the installp command
# installp -aXY -d/yourdir/gpfs/fixes all
Repeat Steps 1-7 on node2. On node1 and node2 confirm GPFS is installed using the
lslpp command
# lslpp -L gpfs.\*
the output should look similar to this

Fileset Level State Type Description (Uninstaller)


----------------------------------------------------------------------------
[Link] [Link] A F GPFS File Manager
[Link] [Link] A F GPFS Server Manpages and
Documentation
[Link] [Link] A F GPFS GSKit Cryptography Runtime
[Link].en_US [Link] A F GPFS Server Messages U.S. English
Note 1: The above example is from GPFS V4.1 Express Edition. The important part is
the base, docs and msg filesets are present.

If you have GPFS Standard Edition, you should also have the following:

[Link] [Link] A F GPFS Extended Features


If you have GPFS Advanced Edition, in addition to [Link], you should
also have the following entry:
[Link] [Link] A F GPFS Cryptographic Subsystem

Note2: The [Link] fileset is used by the Power 775 HPC cluster only and there is
no need to install this fileset on any other AIX cluster. This fileset does not
ship on the V4.1 media.

Confirm the GPFS binaries are in your $PATH using the mmlscluster command
# mmlscluster
mmlscluster: This node does not belong to a GPFS cluster.
mmlscluster: Command failed. Examine previous error messages to determine cause.
Note: The path to the GPFS binaries is: /usr/lpp/mmfs/bin

Step 3: Create the GPFS cluster

For this exercise the cluster is initially created with a single node. When
creating the cluster make node1 the primary configuration server and give node1 the
designations quorum and manager. Use ssh and scp as the remote shell and remote
file copy commands.

*Primary Configuration server (node1): __________


*Verify fully qualified path to ssh and scp:
ssh path__________
scp path_____________

Use the mmcrcluster command to create the cluster


# mmcrcluster -N node1:manager-quorum --ccr-disable -p node1 \
-r /usr/bin/ssh -R /usr/bin/scp
Thu Mar 1 09:04:33 CST 2012: mmcrcluster: Processing node node1
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
Use the mmchlicense command to designate licenses as needed.
Run the mmlscluster command again to see that the cluster was created
# mmlscluster

===============================================================================
| Warning: |
| This cluster contains nodes that do not have a proper GPFS license |
| designation. This violates the terms of the GPFS licensing agreement. |
| Use the mmchlicense command and assign the appropriate GPFS licenses |
| to each of the nodes in the cluster. For more information about GPFS |
| license designation, see the Concepts, Planning, and Installation Guide. |
===============================================================================

GPFS cluster information


========================

GPFS cluster name: [Link]


GPFS cluster id: 13882390374179224464
GPFS UID domain: [Link]
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers:


-----------------------------------

Primary server: [Link]


Secondary server: (none)

Node Daemon node name IP address Admin node name


Designation
-----------------------------------------------------------------------------------
------------
1 [Link] [Link] [Link]
quorum-manager

Set the license mode for the node using the mmchlicense command. Use a server
license for this node.
# mmchlicense server --accept -N node1

The following nodes will be designated as possessing GPFS server licenses:


[Link]
mmchlicense: Command successfully completed

Step 4: Start GPFS and verify the status of all nodes

Start GPFS on all the nodes in the GPFS cluster using the mmstartup command
# mmstartup -a
Check the status of the cluster using the mmgetstate command
# mmgetstate -a

Node number Node name GPFS state


------------------------------------------
1 node1 active

Step 5: Add the second node to the cluster

One node 1 use the mmaddnode command to add node2 to the cluster
# mmaddnode -N node2
Confirm the node was added to the cluster using the mmlscluster command
# mmlscluster
Use the mmchcluster command to set node2 as the secondary configuration server
# mmchcluster -s node2
Set the license mode for the node using the mmchlicense command. Use a server
license for this node.
# mmchlicense server --accept -N node2
Start node2 using the mmstartup command
# mmstartup -N node2
Use the mmgetstate command to verify that both nodes are in the active state
# mmgetstate -a

Step 6: Collect information about the cluster

Now we will take a moment to check a few things about the cluster. Examine the
cluster configuration using the mmlscluster command

What is the cluster name? ______________________


What is the IP address of node2? _____________________
What date was this version of GPFS "Built"? ________________

Hint: look in the GPFS log file: /var/adm/ras/[Link]

Step 7: Create NSDs

You will use the 4 hdisks.

Each disk will store both data and metadata


The NSD server field (ServerList) can be left blank if both nodes have direct
access to the shared LUNs.
On node1 create the directory /yourdir/data
Create a disk stanza file /yourdir/data/[Link] using your favorite text
editor.

The format for the file is:


%nsd: device=DiskName
nsd=NsdName
servers=ServerList
usage={dataOnly | metadataOnly | dataAndMetadata | descOnly}
failureGroup=FailureGroup
pool=StoragePool

You only need to populate the fields required to create the NSD's, in this example
all NSD's use the default failure group and pool definitions.

%nsd:
device=/dev/hdisk1
nsd=mynsd1
usage=dataAndMetadata

%nsd:
device=/dev/hdisk2
nsd=mynsd2
usage=dataAndMetadata

Create the NSD's using the mmcrnsd command:


mmcrnsd -F /yourdir/data/[Link]
Note: hdisk numbers will vary per system.
Create the NSD's using the mmcrnsd command
# mmcrnsd -F /yourdir/data/[Link]

Step 8: Collect information about the NSD's

Now collect some information about the NSD's you have created.

Examine the NSD configuration using the mmlsnsd command


What mmlsnsd flag do you use to see the operating system device (/dev/hdisk?)
associated with an NSD? _______

Step 9: Create a file system

Now that there is a GPFS cluster and some NSDs available you can create a file
system. In this section we will create a file system.

Set the file system blocksize to 64kb


Mount the file system at /gpfs
Create the file system using the mmcrfs command
# mmcrfs /gpfs fs1 -F [Link] -B 64k
Verify the file system was created correctly using the mmlsfs command
# mmlsfs fs1
Will the file system automatically mounted when GPFS starts? _________________

Mount the file system using the _mmmount_ command


# mmmount all -a
Verify the file system is mounted using the df command
# df -k
Filesystem 1024-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 65536 6508 91% 3375 64% /
/dev/hd2 1769472 465416 74% 35508 24% /usr
/dev/hd9var 131072 75660 43% 620 4% /var
/dev/hd3 196608 192864 2% 37 1% /tmp
/dev/hd1 65536 65144 1% 13 1% /home
/proc - - - - - /proc
/dev/hd10opt 327680 47572 86% 7766 41% /opt
/dev/fs1 398929107 398929000 1% 1 1% /gpfs
Use the mmdf command to get information on the file system.
# mmdf fs1
How many inodes are currently used in the file system? ______________

The output of these commands:


mmauth
mmgetstate -a
mmlscluster
mmlsconfig
mmlsdisk
mmlsfileset
mmlsfs
mmlspolicy
mmlsmgr
mmlsnode -a
mmlsnsd
mmlssnapshot
mmremotecluster
mmremotefstsstatus 2. The contents of the /var/adm/ras/[Link].* file (on all
nodes in the cluster)

Common questions

Powered by AI

Setting a GPFS file system block size to 64KB optimizes performance for workloads with specific I/O characteristics and affects the space allocation and efficiency of data access. It's implemented during file system creation using the `mmcrfs` command with the `-B` flag, like `mmcrfs /gpfs fs1 -F diskdesc.txt -B 64k`. This configuration should match the I/O patterns for improved system performance, making it crucial for system administrators to understand workload needs before setting the block size .

To add a node to an existing GPFS cluster, use the `mmaddnode` command on the current node, specifying the new node (`-N node2`). Then, use `mmchcluster` to set the new node as a secondary configuration server (`-s node2`). Assign a server license to the new node using `mmchlicense server --accept -N node2`. Start the GPFS on the new node using `mmstartup -N node2` and verify both nodes are active with `mmgetstate -a` .

To verify node-to-node SSH communications in a GPFS cluster setup on AIX, each node must create an SSH key pair using the command `ssh-keygen`. The public key from each node is then added to the `authorized_keys` file of the other node. This enables password-less SSH access between the nodes. The verification process involves using the SSH command to connect from one node to itself and the other node, ensuring no password prompts occur, which confirms SSH configurations are correct .

To verify the availability of unused disks for NSD creation, use the `lspv` command to list physical volumes, ensuring there are 4 available disks (e.g., hdiskw-hdiskz) besides existing volume groups like rootvg. This ensures that there are sufficient resources for creating NSDs without affecting existing system configurations .

Creating a Network Shared Disk (NSD) involves several steps: first, prepare a disk stanza file describing the NSDs, specifying parameters such as device, NSD name, and usage type. For example, an entry might be `%nsd: device=/dev/hdisk1 nsd=mynsd1 usage=dataAndMetadata`. Next, use the `mmcrnsd` command with the -F flag to create the NSD based on this file. Once created, verify the configuration with the `mmlsnsd` command to ensure the NSDs are properly set up .

Not having proper GPFS license designations violates the GPFS licensing agreement, potentially leading to legal and operational issues. The implications include cluster operation restrictions and potential legal action due to breach of the licensing agreement. To resolve this, use the `mmchlicense` command to assign the correct licenses to all nodes. For instance, assigning a server license using `mmchlicense server --accept -N <node_name>` ensures compliance and avoids these issues .

To ensure all GPFS nodes are in active state, use the `mmstartup -a` command to start GPFS on all nodes. Then, use the `mmgetstate -a` command to check each node's status; the output should show all nodes as 'active'. Any discrepancies should be investigated using cluster log files and configuration checks to diagnose potential issues .

When creating a GPFS cluster, ensure all nodes hold the correct GPFS server license designations to comply with the licensing agreement, as improper configuration can lead to legal issues and operational limitations. Use the `mmchlicense` command to assign licenses and `mmlscluster` to verify the correct setup. Additionally, designate a primary configuration server and possibly a secondary for redundancy, using commands like `mmcrcluster` and `mmchcluster` accordingly, while continuously checking compliance .

To ensure GPFS licenses are properly designated, use the `mmchlicense` command to set the appropriate GPFS license mode for each node. For example, the command `mmchlicense server --accept -N node1` is used to assign a server license to node1. It's important that all nodes have the correct license designation, as failing to do so violates the GPFS license agreement .

To verify GPFS installation on AIX nodes, use the `lslpp` command with `-L` to list installed GPFS filesets, ensuring files like `gpfs.base`, `gpfs.docs.data`, and `gpfs.msg.en_US` are present. Confirm their correct version and state. Additionally, ensure the GPFS binaries are accessible in `$PATH` by attempting to run commands like `mmlscluster`, which should not produce an error if installation is successful .

You might also like