===== SVC performance statistics ===== To get the stats, first of all you need to make sure that statistics collection is turned on. To check this, use the lssystem command (or lscluster for code prior to 6.3): svcinfo lssystem\\ About 15 lines down you should see the following:\\ statistics_status on\\ statistics_frequency 15\\ If statistics_status is off, then you need to switch it on. The second line is the frequency at which statistics will be collected. This is specified in minutes and can be between 1 and 60. To turn this on, or change the frequency, use the startstats command: svctask startstats -interval intervalBear in mind if you use and interval of 1 minute, you will get a lot of stats, but doing so will make it easier to catch small spikes in performance. A maximum of 16 set of perfomance collections are kept on SVC node, collect it on configuration node, into the directory: /dumps/iostats/ SVC and V7000 Metrics currently included: DISK Blocks/sec DISK I/O DISK Response Time MDISK Blocks/sec MDISK I/O MDISK Response Time NODE CPU Busy NODE Port bytes/sec VDISK Blocks/sec VDISK I/O VDISK Response Time http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3177 If you don't have a tool like TPC to keep an historic of performance data, the SVC, and Storwize also, will keep statistics for 5 minutes. You can collect all the perf statistics and store it on a DB, by using **SVCMON** which is a free tools from IBM developers. https://www.ibm.com/developerworks/mydeveloperworks/blogs/svcmon/?lang=en http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3177 SVCMON is can run on a Windows or Linux server. For Windows, the following component are needed:\\ - Windows OS\\ - ActivePerl\\ - PostgreSQL\\ - PuTTY\\ - msxsl.exe === Installation SVCMON on linux === svcmon on Linux requires the following components.\\ - Linux OS\\ - Perl\\ - PostgreSQL\\ - ssh\\ - xsltproc {{:storage:svcmon1.png?nolink&|Final GPFS on SAN picture.}} * **Install required packages:**\\ (for redhat use yum, zypper for opensuse, apt-get for debian and ubuntu)\\ sudo apt-get install perl\\ sudo apt-get install libdbd-mysql-perl libdbd-pg-perl libdbi-perl\\ sudo apt-get install perl-modules\\ sudo apt-get install liburi-perl\\ sudo apt-get install postgresql\\ sudo apt-get install xsltproc\\ sudo apt-get install libxml-simple-perl\\ sudo apt-get install unzip\\ sudo apt-get install libgd-graph-perl\\ * **Create a user for svcmon:** sudo useradd svcmon -d /home/svcmon -m\\ sudo passwd svcmon\\ sudo vi /etc/group\\ svcmon:x:1003:postgres * **Create the base folders for svcmon** It would be better to create a filesystem /svcmon\\ [root@sspc bin]# ls -alR /svcmon /svcmon: drwxr-xr-x 2 root root 4096 Nov 13 10:52 backup drwxr-xr-x 2 root root 4096 Nov 13 10:54 bin drwxr-xr-x 2 root root 4096 Nov 13 10:54 config drwxr-xr-x 2 root root 4096 Nov 16 05:59 iostats drwxr-xr-x 2 root root 4096 Nov 13 10:52 report * **Download and unzip svcmon into /svcmon/bin** * **aaa** perl svcdb.pl --create --dbhost localhost --dbname svcdb --dbuser postgres --dbpwd Passw0rd taken uit te voeren als user postgres: postgres@svcmon:~$ createuser svcmon postgres@svcmon:~$ psql psql (8.4.7) Type "help" for help. postgres=# alter user svcmon password 'Passw0rd'; alter role postgres=# \q taken uit te voeren als user svcmon: ssh-keygen cd /svcmon/bin/ svcmon@svcmon:/svcmon/bin$ perl svcdb.pl --create --dbhost localhost --dbname svcdb --dbuser svcmon --dbpwd xxxxxxxxxxxx niet vergeten om op de SVC de public key te importeren voor de user svcmon!! ook minstens 1 keer connecteren via ssh als user svcmon: svcmon@svcmon:/svcmon/bin$ ssh admin@svc1.vmm.be **Start/stop script** $ ls -l /etc/init.d/svcperf -rwx------ 1 root root 983 Apr 4 14:33 /etc/init.d/svcperf $ sudo cat /etc/init.d/svcperf #! /bin/bash ### BEGIN INIT INFO # Provides: svcperf # Required-Start: $remote_fs $syslog postgresql # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: svc performance data collector ### END INIT INFO set -e SVCPERFBINDIR=/svcmon/bin case "$1" in start) sleep 20 echo "" echo "Starting svc performance data collector..." su - svcmon -c ". /etc/profile; cd $SVCPERFBINDIR; nohup perl svcperf.pl -svc svc1.vmm.be --dbuser svcmon --dbpwd xxxxxxxxxxxx --interval 5 &" echo "" ;; stop) SVCPERFPID=$(ps -ef | grep "perl svcperf.pl" | grep -v grep | awk '{print $2}') if [ "X$SVCPERFPID" != "X" ] then kill $SVCPERFPID fi ;; restart) $0 stop sleep 2 $0 start ;; status) ps -ef | grep "perl svcperf.pl" | grep -v grep echo "" su - svcmon -c ". /etc/profile; cd $SVCPERFBINDIR; perl svcdb.pl --list --dbuser svcmon --dbpwd xxxxxxxxxxxx" ;; *) echo "Usage: $0 {start|stop|restart|status}" exit 1 esac exit 0 $ sudo update-rc.d svcperf defaults **cron** $ sudo grep svcmon /vmm/crons/root # reload the cron every hour for user svcmon, but only if there is a new version 00 * * * * if [ /vmm/crons/svcmon -nt /var/spool/cron/crontabs/svcmon ] ; then crontab -u svcmon /vmm/crons/svcmon; fi $ sudo cat /vmm/crons/svcmon # m h dom mon dow command # daily clean up 01 00 * * * /svcmon/bin/cron/svc-cleanup.sh > /svcmon/logs/svc-cleanup/svc-cleanup.log.$(date +\%d) 2>&1 # create daily overview of the SVC config 01 00 * * * /svcmon/bin/cron/svc-config.sh # create hourly SVC performance report 07 * * * * /svcmon/bin/cron/svc-report-hourly.sh # create daily SVC performance report 08 00 * * * /svcmon/bin/cron/svc-report-daily.sh # create weekly SVC performance report #09 00 * * 1 /svcmon/bin/cron/svc-report-weekly.sh # create monthly SVC performance report #10 00 1 * * /svcmon/bin/cron/svc-report-monthly.sh # backup svcdb performance report database 04 04 * * * /svcmon/bin/cron/svc-backupdb.sh **cron-scripts** $ ls -l /svcmon/bin/cron/ total 28 -rwx------ 1 svcmon svcmon 128 Apr 5 11:33 svc-backupdb.sh -rwx------ 1 svcmon svcmon 1844 Apr 6 11:22 svc-cleanup.sh -rwx------ 1 svcmon svcmon 210 Apr 5 11:33 svc-config.sh -rwx------ 1 svcmon svcmon 703 Apr 5 11:34 svc-report-daily.sh -rwx------ 1 svcmon svcmon 706 Apr 5 11:34 svc-report-hourly.sh -rwx------ 1 svcmon svcmon 853 Apr 5 11:34 svc-report-monthly.sh -rwx------ 1 svcmon svcmon 721 Apr 5 11:34 svc-report-weekly.sh $ sudo cat /svcmon/bin/cron/svc-backupdb.sh #!/bin/bash pg_dump svcdb | gzip > /vmm/backup/postgresql/svcdb.gz 2>/svcmon/logs/svc-backupdb/svc-backupdb.log.$(date +"%d") $ sudo cat /svcmon/bin/cron/svc-cleanup.sh #!/bin/bash #set -x # delete old logs and reports find /svcmon/out/config -type f -mtime +365 -print -exec rm -f {} \; find /svcmon/logs -type f -mtime +365 -print -exec rm -f {} \; find /svcmon/logs/report-hourly -type f -mtime +10 -print -exec rm -f {} \; find /svcmon/out/report-hourly -mindepth 1 -maxdepth 1 -type d -mtime +8 -print -exec rm -rf {} \; find /svcmon/out/report-daily -mindepth 1 -maxdepth 1 -type d -mtime +365 -print -exec rm -rf {} \; #find /svcmon/out/report-weekly -mindepth 1 -maxdepth 1 -type d -mtime +52 -print -exec rm -rf {} \; #find /svcmon/out/report-monthly -mindepth 1 -maxdepth 1 -type d -mtime +12 -print -exec rm -rf {} \; #psql svcdb -c "select * from svc_cpu where timestamp < '2011-04-02 00:00:00';" # delete old records psql svcdb -c "delete from svc_cpu where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';" psql svcdb -c "delete from svc_node where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';" psql svcdb -c "delete from svc_cache where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';" psql svcdb -c "delete from svc_partition where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';" psql svcdb -c "delete from svc_port where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';" psql svcdb -c "delete from svc_vdsk where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';" psql svcdb -c "delete from svc_mdsk where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';" # defrag/vacuum database psql svcdb -c "vacuum full svc_node;" psql svcdb -c "vacuum full svc_node;" psql svcdb -c "vacuum full svc_cache;" psql svcdb -c "vacuum full svc_partition;" psql svcdb -c "vacuum full svc_port;" psql svcdb -c "vacuum full svc_vdsk;" psql svcdb -c "vacuum full svc_mdsk;" $ sudo cat /svcmon/bin/cron/svc-config.sh #!/bin/bash cd /svcmon/bin CONFIGLOG=/svcmon/logs/svc-config/svc-config.log.$(date +"%d") date > $CONFIGLOG perl svcconfig.pl --svc svc1.vmm.be --dir /svcmon/out/config/ >> $CONFIGLOG 2>&1 date >> $CONFIGLOG $ sudo cat /svcmon/bin/cron/svc-report-hourly.sh #!/bin/bash #set -x SVCREPORTDATE=$(date --date="-1hour" +"%Y%m%d%H00") SVCREPORTOUTDIR=/svcmon/out/report-hourly/$SVCREPORTDATE SVCREPORTLOG=/svcmon/logs/report-hourly/report-hourly-$SVCREPORTDATE date >>$SVCREPORTLOG 2>&1 cd /svcmon/bin mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="-1hour" +"%Y-%m-%d %H:00:00")" --for 60 --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1 chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 $ sudo cat /svcmon/bin/cron/svc-report-daily.sh #!/bin/bash #set -x SVCREPORTDATE=$(date --date="-1day" +"%Y%m%d0000") SVCREPORTOUTDIR=/svcmon/out/report-daily/$SVCREPORTDATE SVCREPORTLOG=/svcmon/logs/report-daily/report-daily-$SVCREPORTDATE date >>$SVCREPORTLOG 2>&1 cd /svcmon/bin mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="-1day" +"%Y-%m-%d 00:00:00")" --for 1440 --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1 chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 $ sudo cat /svcmon/bin/cron/svc-report-weekly.sh #!/bin/bash #set -x SVCREPORTDATE=$(date --date="monday-1week" +"%Y%m%d0000") SVCREPORTOUTDIR=/svcmon/out/report-weekly/$SVCREPORTDATE SVCREPORTLOG=/svcmon/logs/report-weekly/report-weekly-$SVCREPORTDATE date >>$SVCREPORTLOG 2>&1 cd /svcmon/bin mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="monday-1week" +"%Y-%m-%d 00:00:00")" --for 10080 --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1 chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 $ sudo cat /svcmon/bin/cron/svc-report-monthly.sh #!/bin/bash #set -x SVCREPORTDATE=$(date --date="-1month" +"%Y%m010000") SVCREPORTOUTDIR=/svcmon/out/report-monthly/$SVCREPORTDATE SVCREPORTLOG=/svcmon/logs/report-monthly/report-monthly-$SVCREPORTDATE NUMBEROFDAYSINMONTH=$(date --date="-$(date +"%d")day" +"%d") NUMBEROFMINUTESINMONTH=$(expr $NUMBEROFDAYSINMONTH \* 1440) date >>$SVCREPORTLOG 2>&1 cd /svcmon/bin mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="-1month" +"%Y-%m-01 00:00:00")" --for $NUMBEROFMINUTESINMONTH --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1 chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1 date >>$SVCREPORTLOG 2>&1 Usage svcmon collect statistics on SVC each 5 minutes and put it into the postgresql database. every hour, day, week, month, a report is generated. daily, the current SVC configuration stored. the /svcmon/out directory via apache offered next. useful commands manual database query svcmon@svcmon:/$ psql svcdb svcdb=# select version(); version ---------------------------------------------------------------------------------------------------- PostgreSQL 8.4.7 on i486-pc-linux-gnu, compiled by GCC gcc-4.4.real (Debian 4.4.5-8) 4.4.5, 32-bit (1 row) svcdb=# select * from svc_cpu where timestamp < '2011-04-02 00:00:00'; svcdb=# select * from svc_node where timestamp < '2011-04-01 14:00:00'; svcdb=# select * from svc_cache where timestamp < '2011-04-01 14:00:00'; svcdb=# select * from svc_partition where timestamp < '2011-04-01 14:00:00'; svcdb=# select * from svc_port where timestamp < '2011-04-01 14:00:00'; svcdb=# select * from svc_vdsk where timestamp < '2011-04-01 14:00:00'; svcdb=# select * from svc_mdsk where timestamp < '2011-04-01 14:00:00'; report manually draw (2 hours of data from April 1, 2011 1:19:32 p.m. you, written to dir /OUTDIR) svcmon@svcmon:/svcmon/bin$ perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "2011-04-01 13:19:23" --for 120 --dir /OUTDIR/ oplijsten welke svc data momenteel in de database terug te vinden is svcmon@svcmon:/svcmon/bin$ perl svcdb.pl --list --dbuser svcmon --dbpwd xxxxxxxxxxxx Database backup files in /svcmon/backup/ directory ... total 110664 drwxrwx--- 2 svcmon postgres 21 Apr 4 15:55 . drwxr-xr-x 4 root root 35 Apr 4 15:55 .. -rw-r--r-- 1 svcmon svcmon 113317183 Apr 6 04:05 svcdb.gz Current database (localhost, svcdb, svcmon) has ... Cluster: From: To: SVC1 2011-04-04 00:00:10 2011-04-06 12:07:48 **status from svcperf.pl** $ sudo /etc/init.d/svcperf status svcmon 23401 1 0 11:45 pts/3 00:00:36 perl svcperf.pl -svc svc1.vmm.be --dbuser svcmon --dbpwd xxxxxxxxxxxx --interval 5 Database backup files in /svcmon/backup/ directory ... total 110664 drwxrwx--- 2 svcmon postgres 21 Apr 4 15:55 . drwxr-xr-x 4 root root 35 Apr 4 15:55 .. -rw-r--r-- 1 svcmon svcmon 113317183 Apr 6 04:05 svcdb.gz Current database (localhost, svcdb, svcmon) has ... Cluster: From: To: SVC1 2011-04-04 00:00:10 2011-04-06 13:27:41