wiki

User Tools

Site Tools

Trace: • svcmon
storage:svcmon

SVC performance statistics

To get the stats, first of all you need to make sure that statistics collection is turned on. To check this, use the lssystem command (or lscluster for code prior to 6.3):

svcinfo lssystem
About 15 lines down you should see the following:

statistics_status on
statistics_frequency 15

If statistics_status is off, then you need to switch it on. The second line is the frequency at which statistics will be collected. This is specified in minutes and can be between 1 and 60. To turn this on, or change the frequency, use the startstats command:

svctask startstats -interval intervalBear in mind if you use and interval of 1 minute, you will get a lot of stats, but doing so will make it easier to catch small spikes in performance. A maximum of 16 set of perfomance collections are kept on SVC node, collect it on configuration node, into the directory: /dumps/iostats/ 

SVC and V7000 Metrics currently included:
DISK Blocks/sec
DISK I/O
DISK Response Time

MDISK Blocks/sec
MDISK I/O
MDISK Response Time

NODE CPU Busy
NODE Port bytes/sec

VDISK Blocks/sec
VDISK I/O
VDISK Response Time

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3177

If you don't have a tool like TPC to keep an historic of performance data, the SVC, and Storwize also, will keep statistics for 5 minutes.

You can collect all the perf statistics and store it on a DB, by using SVCMON which is a free tools from IBM developers.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/svcmon/?lang=en

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3177

SVCMON is can run on a Windows or Linux server. For Windows, the following component are needed:
- Windows OS
- ActivePerl
- PostgreSQL
- PuTTY
- msxsl.exe

Installation SVCMON on linux

svcmon on Linux requires the following components.
- Linux OS
- Perl
- PostgreSQL
- ssh
- xsltproc

Final GPFS on SAN picture.

  • Install required packages:

(for redhat use yum, zypper for opensuse, apt-get for debian and ubuntu)
sudo apt-get install perl
sudo apt-get install libdbd-mysql-perl libdbd-pg-perl libdbi-perl
sudo apt-get install perl-modules
sudo apt-get install liburi-perl
sudo apt-get install postgresql
sudo apt-get install xsltproc
sudo apt-get install libxml-simple-perl
sudo apt-get install unzip
sudo apt-get install libgd-graph-perl

  • Create a user for svcmon:

sudo useradd svcmon -d /home/svcmon -m
sudo passwd svcmon
sudo vi /etc/group

svcmon:x:1003:postgres

  • Create the base folders for svcmon

It would be better to create a filesystem /svcmon

[root@sspc bin]# ls -alR /svcmon
/svcmon:
drwxr-xr-x  2 root root 4096 Nov 13 10:52 backup
drwxr-xr-x  2 root root 4096 Nov 13 10:54 bin
drwxr-xr-x  2 root root 4096 Nov 13 10:54 config
drwxr-xr-x  2 root root 4096 Nov 16 05:59 iostats
drwxr-xr-x  2 root root 4096 Nov 13 10:52 report
  • Download and unzip svcmon into /svcmon/bin
  • aaa
perl svcdb.pl --create --dbhost localhost --dbname svcdb --dbuser postgres --dbpwd Passw0rd

taken uit te voeren als user postgres: 

postgres@svcmon:~$ createuser svcmon
postgres@svcmon:~$ psql
psql (8.4.7)
Type "help" for help.

postgres=# alter user svcmon password 'Passw0rd';
alter role
postgres=# \q

taken uit te voeren als user svcmon: 

ssh-keygen
cd /svcmon/bin/
svcmon@svcmon:/svcmon/bin$ perl svcdb.pl --create --dbhost localhost --dbname svcdb --dbuser svcmon --dbpwd xxxxxxxxxxxx

niet vergeten om op de SVC de public key te importeren voor de user svcmon!!

ook minstens 1 keer connecteren via ssh als user svcmon: 

svcmon@svcmon:/svcmon/bin$ ssh admin@svc1.vmm.be

Start/stop script 

$ ls -l /etc/init.d/svcperf 
-rwx------ 1 root root 983 Apr  4 14:33 /etc/init.d/svcperf

$ sudo cat /etc/init.d/svcperf 
#! /bin/bash
### BEGIN INIT INFO
# Provides:             svcperf
# Required-Start:       $remote_fs $syslog postgresql
# Required-Stop:        $remote_fs $syslog
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    svc performance data collector
### END INIT INFO

set -e
SVCPERFBINDIR=/svcmon/bin

case "$1" in
  start)
        sleep 20
        echo ""
        echo "Starting svc performance data collector..."
        su - svcmon -c ". /etc/profile; cd $SVCPERFBINDIR; nohup perl svcperf.pl -svc svc1.vmm.be --dbuser svcmon --dbpwd xxxxxxxxxxxx --interval 5 &"
        echo ""
        ;;
  stop)
        SVCPERFPID=$(ps -ef | grep "perl svcperf.pl" | grep -v grep | awk '{print $2}')
        if [ "X$SVCPERFPID" != "X" ]
        then
                kill $SVCPERFPID
        fi
        ;;
  restart)
        $0 stop
        sleep 2
        $0 start
        ;;
  status)
        ps -ef | grep "perl svcperf.pl" | grep -v grep
        echo ""
        su - svcmon -c ". /etc/profile; cd $SVCPERFBINDIR; perl svcdb.pl --list --dbuser svcmon --dbpwd xxxxxxxxxxxx"
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit 0
$ sudo update-rc.d svcperf defaults

cron 

$ sudo grep svcmon /vmm/crons/root 

# reload the cron every hour for user svcmon, but only if there is a new version
00 * * * *      if [ /vmm/crons/svcmon -nt /var/spool/cron/crontabs/svcmon ] ; then crontab -u svcmon /vmm/crons/svcmon; fi

$ sudo cat /vmm/crons/svcmon 

# m h  dom mon dow   command

# daily clean up
01 00 * * *     /svcmon/bin/cron/svc-cleanup.sh > /svcmon/logs/svc-cleanup/svc-cleanup.log.$(date +\%d) 2>&1

# create daily overview of the SVC config
01 00 * * *     /svcmon/bin/cron/svc-config.sh

# create hourly SVC performance report
07 * * * *      /svcmon/bin/cron/svc-report-hourly.sh

# create daily SVC performance report
08 00 * * *     /svcmon/bin/cron/svc-report-daily.sh

# create weekly SVC performance report
#09 00 * * 1     /svcmon/bin/cron/svc-report-weekly.sh

# create monthly SVC performance report
#10 00 1 * *     /svcmon/bin/cron/svc-report-monthly.sh

# backup svcdb performance report database
04 04 * * *     /svcmon/bin/cron/svc-backupdb.sh

cron-scripts 

$ ls -l /svcmon/bin/cron/
total 28
-rwx------ 1 svcmon svcmon  128 Apr  5 11:33 svc-backupdb.sh
-rwx------ 1 svcmon svcmon 1844 Apr  6 11:22 svc-cleanup.sh
-rwx------ 1 svcmon svcmon  210 Apr  5 11:33 svc-config.sh
-rwx------ 1 svcmon svcmon  703 Apr  5 11:34 svc-report-daily.sh
-rwx------ 1 svcmon svcmon  706 Apr  5 11:34 svc-report-hourly.sh
-rwx------ 1 svcmon svcmon  853 Apr  5 11:34 svc-report-monthly.sh
-rwx------ 1 svcmon svcmon  721 Apr  5 11:34 svc-report-weekly.sh

$ sudo cat /svcmon/bin/cron/svc-backupdb.sh 

#!/bin/bash

pg_dump svcdb | gzip > /vmm/backup/postgresql/svcdb.gz 2>/svcmon/logs/svc-backupdb/svc-backupdb.log.$(date +"%d")

$ sudo cat /svcmon/bin/cron/svc-cleanup.sh 

#!/bin/bash
#set -x

# delete old logs and reports
find /svcmon/out/config -type f -mtime +365 -print -exec rm -f {} \;
find /svcmon/logs -type f -mtime +365 -print -exec rm -f {} \;
find /svcmon/logs/report-hourly -type f -mtime +10 -print -exec rm -f {} \;
find /svcmon/out/report-hourly -mindepth 1 -maxdepth 1 -type d -mtime +8 -print -exec rm -rf {} \;
find /svcmon/out/report-daily -mindepth 1 -maxdepth 1 -type d -mtime +365 -print -exec rm -rf {} \;
#find /svcmon/out/report-weekly -mindepth 1 -maxdepth 1 -type d -mtime +52 -print -exec rm -rf {} \;
#find /svcmon/out/report-monthly -mindepth 1 -maxdepth 1 -type d -mtime +12 -print -exec rm -rf {} \;

#psql svcdb -c "select * from svc_cpu where timestamp < '2011-04-02 00:00:00';"

# delete old records
psql svcdb -c "delete from svc_cpu where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';"
psql svcdb -c "delete from svc_node where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';"
psql svcdb -c "delete from svc_cache where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';"
psql svcdb -c "delete from svc_partition where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';"
psql svcdb -c "delete from svc_port where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';"
psql svcdb -c "delete from svc_vdsk where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';"
psql svcdb -c "delete from svc_mdsk where timestamp < '$(date --date="-2months-2days" +"%Y-%m-%d 00:00:00")';"

# defrag/vacuum database
psql svcdb -c "vacuum full svc_node;"
psql svcdb -c "vacuum full svc_node;"
psql svcdb -c "vacuum full svc_cache;"
psql svcdb -c "vacuum full svc_partition;"
psql svcdb -c "vacuum full svc_port;"
psql svcdb -c "vacuum full svc_vdsk;"
psql svcdb -c "vacuum full svc_mdsk;"
$ sudo cat /svcmon/bin/cron/svc-config.sh 

#!/bin/bash

cd /svcmon/bin
CONFIGLOG=/svcmon/logs/svc-config/svc-config.log.$(date +"%d")
date > $CONFIGLOG
perl svcconfig.pl --svc svc1.vmm.be --dir /svcmon/out/config/ >> $CONFIGLOG 2>&1
date >> $CONFIGLOG
$ sudo cat /svcmon/bin/cron/svc-report-hourly.sh 

#!/bin/bash

#set -x

SVCREPORTDATE=$(date --date="-1hour" +"%Y%m%d%H00")
SVCREPORTOUTDIR=/svcmon/out/report-hourly/$SVCREPORTDATE
SVCREPORTLOG=/svcmon/logs/report-hourly/report-hourly-$SVCREPORTDATE

date >>$SVCREPORTLOG 2>&1
cd /svcmon/bin
mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="-1hour" +"%Y-%m-%d %H:00:00")" --for 60 --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1
chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1
find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1
$ sudo cat /svcmon/bin/cron/svc-report-daily.sh 

#!/bin/bash
#set -x

SVCREPORTDATE=$(date --date="-1day" +"%Y%m%d0000")
SVCREPORTOUTDIR=/svcmon/out/report-daily/$SVCREPORTDATE
SVCREPORTLOG=/svcmon/logs/report-daily/report-daily-$SVCREPORTDATE

date >>$SVCREPORTLOG 2>&1
cd /svcmon/bin
mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="-1day" +"%Y-%m-%d 00:00:00")" --for 1440 --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1
chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1
find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1
$ sudo cat /svcmon/bin/cron/svc-report-weekly.sh 

#!/bin/bash
#set -x

SVCREPORTDATE=$(date --date="monday-1week" +"%Y%m%d0000")
SVCREPORTOUTDIR=/svcmon/out/report-weekly/$SVCREPORTDATE
SVCREPORTLOG=/svcmon/logs/report-weekly/report-weekly-$SVCREPORTDATE

date >>$SVCREPORTLOG 2>&1
cd /svcmon/bin
mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="monday-1week" +"%Y-%m-%d 00:00:00")" --for 10080 --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1
chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1
find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1
$ sudo cat /svcmon/bin/cron/svc-report-monthly.sh 
#!/bin/bash
#set -x

SVCREPORTDATE=$(date --date="-1month" +"%Y%m010000")
SVCREPORTOUTDIR=/svcmon/out/report-monthly/$SVCREPORTDATE
SVCREPORTLOG=/svcmon/logs/report-monthly/report-monthly-$SVCREPORTDATE
NUMBEROFDAYSINMONTH=$(date --date="-$(date +"%d")day" +"%d")
NUMBEROFMINUTESINMONTH=$(expr $NUMBEROFDAYSINMONTH \* 1440)

date >>$SVCREPORTLOG 2>&1
cd /svcmon/bin
mkdir $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
chmod 777 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "$(date --date="-1month" +"%Y-%m-01 00:00:00")" --for $NUMBEROFMINUTESINMONTH --dir $SVCREPORTOUTDIR/ >>$SVCREPORTLOG 2>&1
chmod 775 $SVCREPORTOUTDIR >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1
find $SVCREPORTOUTDIR/ -name '*csv' -exec sudo -u postgres /bin/bzip2 {} \; >>$SVCREPORTLOG 2>&1
date >>$SVCREPORTLOG 2>&1

Usage

svcmon collect statistics on SVC each 5 minutes and put it into the postgresql database.

every hour, day, week, month, a report is generated. daily, the current SVC configuration stored.

the /svcmon/out directory via apache offered next. useful commands manual database query 

svcmon@svcmon:/$ psql svcdb

svcdb=# select version();

                                              version
----------------------------------------------------------------------------------------------------
 PostgreSQL 8.4.7 on i486-pc-linux-gnu, compiled by GCC gcc-4.4.real (Debian 4.4.5-8) 4.4.5, 32-bit
(1 row)

svcdb=# select * from svc_cpu where timestamp < '2011-04-02 00:00:00';
svcdb=# select * from svc_node where timestamp < '2011-04-01 14:00:00';
svcdb=# select * from svc_cache where timestamp < '2011-04-01 14:00:00';
svcdb=# select * from svc_partition where timestamp < '2011-04-01 14:00:00';
svcdb=# select * from svc_port where timestamp < '2011-04-01 14:00:00';
svcdb=# select * from svc_vdsk where timestamp < '2011-04-01 14:00:00';
svcdb=# select * from svc_mdsk where timestamp < '2011-04-01 14:00:00';

report manually draw (2 hours of data from April 1, 2011 1:19:32 p.m. you, written to dir /OUTDIR) 

svcmon@svcmon:/svcmon/bin$ perl svcreport.pl --cluster SVC1 --dbuser svcmon --dbpwd xxxxxxxxxxxx --from "2011-04-01 13:19:23" --for 120 --dir /OUTDIR/

oplijsten welke svc data momenteel in de database terug te vinden is 

svcmon@svcmon:/svcmon/bin$ perl svcdb.pl --list --dbuser svcmon --dbpwd xxxxxxxxxxxx

Database backup files in /svcmon/backup/ directory ...
total 110664
drwxrwx--- 2 svcmon postgres        21 Apr  4 15:55 .
drwxr-xr-x 4 root   root            35 Apr  4 15:55 ..
-rw-r--r-- 1 svcmon svcmon   113317183 Apr  6 04:05 svcdb.gz

Current database (localhost, svcdb, svcmon) has ...
Cluster:        From:                   To:
SVC1            2011-04-04 00:00:10     2011-04-06 12:07:48

status from svcperf.pl 

$ sudo /etc/init.d/svcperf status

svcmon   23401     1  0 11:45 pts/3    00:00:36 perl svcperf.pl -svc svc1.vmm.be --dbuser svcmon --dbpwd xxxxxxxxxxxx --interval 5

Database backup files in /svcmon/backup/ directory ...
total 110664
drwxrwx--- 2 svcmon postgres        21 Apr  4 15:55 .
drwxr-xr-x 4 root   root            35 Apr  4 15:55 ..
-rw-r--r-- 1 svcmon svcmon   113317183 Apr  6 04:05 svcdb.gz

Current database (localhost, svcdb, svcmon) has ...
Cluster:        From:                   To:
SVC1            2011-04-04 00:00:10     2011-04-06 13:27:41
storage/svcmon.txt · Last modified: 2021/01/01 21:25 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: GNU Free Documentation License 1.3
GNU Free Documentation License 1.3 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki