Differences

This shows you the differences between two versions of the page.

--- gpfs:gpfs_operation [2021/01/27 09:40]
manu created
+++ gpfs:gpfs_operation [2021/11/24 12:10] (current)
manu [Remove a node from a cluster]
@@ Line 1: / Line 1: @@
 ====== GPFS operations ======
-For the cluster Oracle in AIX 5.3 32bits, the latest GPFS supported version is 3.3.
+===== Add a new disk to a filesystem or create a new filesystem =====
-A snapshot don't need the same space as intial filesystem. If it's full, then it die.
+==== Create NSD (Network Shared Disk) disks (format on GPFS format) ====
+Before adding a disk to a GPFS filesystem, you have to give him som parameters:
+  * name : logical name corresponding to the filesystem, This name can contain only the following characters: 'A' through 'Z', 'a' through 'z', '0' through '9', or '_' (the underscore). All other characters are not valid.
+  * failuregroup : important setting, Maximum 3 different ID per filesystem, it correspond to copies of data. If no copy is required, then only one failure group...Use for example 2 for the first copy and 3 for the second copy. mmlsdisk will give you the currents failuregroup IDs
+  * usage : use always dataAndMetadata, except if you want to make tuning, so you could put metadata on high speed disks
+  * pool : always use system (except for advanced users)
+=== identify the SAN disks ===
+In non multipathing, ex VMware RDM you can use
+<cli prompt='#'>
+[root@gpfs01 ~]# lsscsi -s
+[0:2:0:0]    disk    IBM      ServeRAID M1115  2.13  /dev/sda    298GB
+[0:2:1:0]    disk    IBM      ServeRAID M1115  2.13  /dev/sdb    198GB
+[1:0:0:0]    cd/dvd  IBM SATA DEVICE 81Y3676   IBD1  /dev/sr0        -
+[7:0:0:0]    disk    IBM      2145             0000  /dev/sdii   274GB
+[7:0:0:1]    disk    IBM      2145             0000  /dev/sdc   21.4GB
+[7:0:0:2]    disk    IBM      2145             0000  /dev/sdd   1.09TB
+...
+</cli>
+List UUID serial corresponding to
+<cli prompt='#'>
+[root@gpfs01 ~]# ll /dev/disk/by-id/
+...
+lrwxrwxrwx 1 root root   10 Sep 30 01:30 dm-uuid-mpath-36005076300810163a00000000000006a -> ../../dm-4
+lrwxrwxrwx 1 root root   10 Sep 30 01:30 dm-uuid-mpath-36005076300810163a00000000000006b -> ../../dm-5
+...
+lrwxrwxrwx 1 root root   10 Sep 30 15:05 wwn-0x60050764008181c46800000000000058 -> ../../sdml
+lrwxrwxrwx 1 root root   10 Sep 30 15:05 wwn-0x60050764008181c46800000000000059 -> ../../sdmm
+</cli>
+For multipathing devices use
+<cli prompt='#'>
+[root@gpfs01 ~]# multipath -ll | egrep "mpath|size" | paste -d " "  - -
+mpathcu (360050764008181c46800000000000042) dm-126 IBM     ,2145             size=256G features='1 queue_if_no_path' hwhandler='0' wp=rw
+mpathbp (360050764008181c46800000000000030) dm-23 IBM     ,2145             size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
+...
+</cli>
+Give the right device name after rescan, identify your disk and device name, use **dm-xx**:
+<cli prompt='#'>
+[root@gpfs01 scripts]# rescan-scsi-bus.sh -a
+[root@gpfs01 ~]# multipath -ll | egrep "mpath|size" | paste -d " "  - -
+...
+mpathbd (360050764008181c46800000000000023) dm-47 IBM     ,2145                   size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
+</cli>
+POOL: system    (default)
+=== List failure group ===
+Check FailureGroup : A is 2 and B is 3
+<cli prompt='#'>
+[root@gpfs01 ~]# mmlsdisk gpfs01
+disk         driver   sector     failure holds    holds                            storage
+name         type       size       group metadata data  status        availability pool
+------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------
+GPFS_NSD_DATA_B_08 nsd         512           3 Yes      Yes   ready         up           system
+...
+GPFS_NSD_DATA_A_13 nsd         512           2 Yes      Yes   ready         up           system
+</cli>
+=== identify NSD in use and free disks (new disks) ===
+NSD in use
+<cli prompt='#'>
+[root@gpfs01 ~]# mmlsnsd -X | grep gpfs01-hb |  awk '{print $3}' |  sort
+/dev/dm-10
+/dev/dm-11
+/dev/dm-12
+/dev/dm-13
+...
+</cli>
+List all disks
+<cli prompt='#'>
+[root@gpfs01 ~]# multipath -ll | egrep "mpath|size" | paste -d " "  - - | tr ' ' '\n' | grep 'dm-' | sed 's/^/\/dev\//' | sort
+/dev/dm-50
+/dev/dm-51
+...
+</cli>
+**Difference**
+  multipath -ll | egrep "mpath|size" | paste -d " "  - - | tr ' ' '\n' | grep 'dm-' | sed 's/^/\/dev\//' | sort > /tmp/disk_all.txt
+  mmlsnsd -X | grep gpfs01-hb |  awk '{print $3}' |  sort > /tmp/disk_nsd.txt
+  sdiff -sw100 /tmp/disk_all.txt /tmp/disk_nsd.txt
+=== Build NSD file ===
+Create a text file containing a list of NSD disks, to add, and their characteristics.
+<cli prompt='#'>
+[root@gpfs01 scripts]# cat list.disks_CESSHARE.txt
+%nsd:
+device=/dev/dm-47
+nsd=GPFS_NSD_CESSHARE_A_01
+servers=gpfs01-hb,gpfs02-hb
+usage=dataAndMetadata
+failureGroup=2
+pool=system
+</cli>
+Create the NSD (network shared disk), and also verify the disk
+<cli prompt='#'>
+[root@gpfs01 ~]# mmcrnsd -F list.disks_CESSHARE.txt -v yes
+mmcrnsd: Processing disk dm-8
+mmcrnsd: Propagating the cluster configuration data to all
+  affected nodes.  This is an asynchronous process.
+</cli>
+Now disk is formated, but free, not attached to a filesystem
+<cli prompt='#'>
+[root@gpfs01 ~]# fdisk -l /dev/dm-47
+...
+Disk /dev/dm-47: 21.5 GB, 21474836480 bytes, 41943040 sectors
+...
+Disk label type: gpt
+Disk identifier: 7A94FA63-6A6C-4001-89E8-E36D00B3F66E
+#         Start          End    Size  Type            Name
+           48     41942991     20G  IBM General Par GPFS:
+</cli>
+<cli prompt='#'>
+[root@gpfs01 ~]# mmlsnsd -L
+ File system   Disk name    NSD volume ID      NSD servers
+---------------------------------------------------------------------------------------------
+ cesshared01lv GPFS_NSD_CESSHARE01 0A0113A15B0BFD87   gpfs01-hb,gpfs02-hb
+...
+ (free disk)   GPFS_NSD_CESSHARE_A_01 0A0113A15E2EB417   gpfs01-hb,gpfs02-hb
+[root@gpfs01 ~]# mmlsnsd -X
+ Disk name    NSD volume ID      Device         Devtype  Node name                Remarks
+---------------------------------------------------------------------------------------------------
+ GPFS_NSD_CESSHARE01 0A0113A15B0BFD87   /dev/dm-9      dmm      gpfs01-hb                server node
+ GPFS_NSD_CESSHARE01 0A0113A15B0BFD87   /dev/dm-2      dmm      gpfs02-hb                server node
+ GPFS_NSD_CESSHARE_A_01 0A0113A15E2EB417   /dev/dm-47     dmm      gpfs01-hb                server node
+ GPFS_NSD_CESSHARE_A_01 0A0113A15E2EB417   /dev/dm-47     dmm      gpfs02-hb                server node
+.....
+</cli>
+==== Add the NSD disk to a filesystem ====
+=== Add NSD to a current filesystem ===
+First list unused disks
+<cli prompt='#'>
+[root@gpfs01 ~]# mmlsnsd -F
+ File system   Disk name    NSD servers
+---------------------------------------------------------------------------
+ (free disk)   GPFS_NSD_CESSHARE_A_01 gpfs01-hb,gpfs02-hb
+</cli>
+Create a stanza file like for NSD
+<cli prompt='#'>
+[root@gpfs01 scripts]# cat list.disks_CESSHARE.txt
+%nsd:
+device=/dev/dm-47
+nsd=GPFS_NSD_CESSHARE_A_01
+servers=gpfs01-hb,gpfs02-hb
+usage=dataAndMetadata
+failureGroup=2
+pool=system
+</cli>
+Now add your disk to the filesystem, and rebalance blocs (if 2 copies of data are required, the a second copy will be done)
+<cli prompt='#'>
+[root@gpfs01 ~]# mmadddisk /dev/cesshared01lv -F list.disks_CESSHARE.txt -r
+The following disks of cesshared01lv will be formatted on node gpfs02:
+    GPFS_NSD_CESSHARE_A_01: size 20480 MB
+Extending Allocation Map
+Checking Allocation Map for storage pool system
+Completed adding disks to file system cesshared01lv.
+mmadddisk: Propagating the cluster configuration data to all
+  affected nodes.  This is an asynchronous process.
+Restriping /dev/cesshared01lv ...
+Scanning file system metadata, phase 1 ...
+% complete on Mon Jan 27 11:28:13 2020
+Scan completed successfully.
+Scanning file system metadata, phase 2 ...
+% complete on Mon Jan 27 11:28:13 2020
+Scan completed successfully.
+Scanning file system metadata, phase 3 ...
+% complete on Mon Jan 27 11:28:13 2020
+Scan completed successfully.
+Scanning file system metadata, phase 4 ...
+% complete on Mon Jan 27 11:28:13 2020
+Scan completed successfully.
+Scanning user file metadata ...
+.00 % complete on Mon Jan 27 11:28:13 2020  (     65792 inodes with total        404 MB data processed)
+Scan completed successfully.
+Done
+</cli>
+Check number of copy of a file --> only 1 copy of data and metadata ! We will a a second copy using **mmchfs** command, and then restripe to copy the data and metadat on second failuregroup disks, and the second step to optimize data placement:
+<cli prompt='#'>
+[root@gpfs01 connections]# mmlsattr /CESshared/ha/nfs/ganesha/gpfs-epoch
+  replication factors
+metadata(max) data(max) file    [flags]
+------------- --------- ---------------
+(  2)   1 (  2) /CESshared/ha/nfs/ganesha/gpfs-epoch
+[root@gpfs01 connections]# mmchfs cesshared01lv -m 2 -r 2
+[root@gpfs01 connections]# mmrestripefs  cesshared01lv -R
+Scanning file system metadata, phase 1 ...
+% complete on Mon Jan 27 12:50:45 2020
+Scan completed successfully.
+...
+.00 % complete on Mon Jan 27 12:50:46 2020  (     65792 inodes with total        808 MB data processed)
+Scan completed successfully.
+[root@gpfs01 connections]# mmlsattr /CESshared/ha/nfs/ganesha/gpfs-epoch
+  replication factors
+metadata(max) data(max) file    [flags]
+------------- --------- ---------------
+(  2)   2 (  2) /CESshared/ha/nfs/ganesha/gpfs-epoch    [unbalanced]
+[root@gpfs01 connections]#
+</cli>
+Optimize data placement
+<cli prompt='#'>
+[root@gpfs01 connections]# mmrestripefs  cesshared01lv -b
+Scanning file system metadata, phase 1 ...
+% complete on Mon Jan 27 12:51:56 2020
+Scan completed successfully.
+...
+.00 % complete on Mon Jan 27 12:51:57 2020  (     65792 inodes with total        808 MB data processed)
+Scan completed successfully.
+[root@gpfs01 connections]# mmlsattr /CESshared/ha/nfs/ganesha/gpfs-epoch
+  replication factors
+metadata(max) data(max) file    [flags]
+------------- --------- ---------------
+(  2)   2 (  2) /CESshared/ha/nfs/ganesha/gpfs-epoch
+</cli>
+=== Add NSD to a new filesystem ===
+This a an example of creation of a filesystem with the previously defined NSD disk, bock size 512K, 2 copies of data and metadata, quota is enable,
+<cli prompt='#'>
+[root@gpfs01 connections]# mmcrfs cesshared01lv -F list.disks_CESSHARE.txt -B 512K -m 2 -r 2 -Q yes -T /CESshared -v yes -D nfs4 -k nfs4 -A yes
+</cli>
+<cli prompt='#'>
+[root@gpfs01 connections]# mmmount cesshared01lv
+</cli>
+===== Remove a disk =====
+To delete GPFS_NSD_DATA01 from file system gpfs01 and rebalance the files across the remaining disks, issue this command:
+<cli prompt='#'>
+[root@gpfs01 ~]# mmlsdisk gpfs01
+disk         driver   sector     failure holds    holds                            storage
+name         type       size       group metadata data  status        availability pool
+------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------
+GPFS_NSD_DATA01 nsd         512           2 Yes      Yes   ready         up           system
+GPFS_NSD_DATA02 nsd         512           2 Yes      Yes   ready         up           system
+GPFS_NSD_DATA03 nsd         512           2 Yes      Yes   ready         up           system
+GPFS_NSD_DATA04 nsd         512           2 Yes      Yes   ready         up           system
+GPFS_NSD_DATA05 nsd         512           2 Yes      Yes   ready         up           system
+GPFS_NSD_DATA06 nsd         512           2 Yes      Yes   ready         up           system
+GPFS_NSD_DATA07 nsd         512           2 Yes      Yes   ready         up           system
+[root@gpfs01 ~]# mmdeldisk gpfs01 GPFS_NSD_DATA01
+</cli>
+Now you are able to delete NSD GPFS_NSD_DATA01 from the GPFS cluster, first check if disk is free, then issue this command:
+<cli prompt='#'>
+[root@gpfs01 scripts]# mmlsnsd -F
+ File system   Disk name    NSD servers
+---------------------------------------------------------------------------
+ (free disk)   GPFS_NSD_DATA01 gpfs01-hb,gpfs02-hb
+[root@gpfs01 ~]# mmdelnsd GPFS_NSD_DATA01
+</cli>
-Here we only explain commands only for snapshot on a separarte LV. Both LV (source and snapshot) must be an the same volume group
 ==== Remove a node from GPFS cluster: ====
@@ Line 532: / Line 808: @@
 /dev/orafs1
 /dev/orafs2
+</cli>
+===== Remove a node from a cluster =====
+<cli prompt='#'>
+# mmchnsd "GPFS_NSD_M_B_0002:prscale-b-01"
+# mmchnode --noperfmon -N prscale-b-02
+# mmchnode --ces-disable -N prscale-b-02
+# mmperfmon config update --collectors prscale-b-02
+# mmchnode --nonquorum -N prscale-b-02
+# mmchnode --nomanager -N prscale-b-02
+# mmdelnode -N prscale-b-02
 </cli>

wiki

User Tools

Site Tools

Differences

Page Tools