wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Brocade problem determination ======

//For info// if ask for supportsave : CRA (Challenge Response Authentication) is chosen as "NO" or "N" and the issue is still seen and/or root access is not available 

===== SAN switch cleanup files =====

<cli prompt='>'>

FLEX-A1-BLUE:root> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root             377M  247M  110M  69% /
/dev/hda3             193M   23M  161M  12% /core_files
/dev/hda1             377M  277M   80M  78% /mnt
FLEX-A1-BLUE:root>
FLEX-A1-BLUE:root> supportsave -R
Removing all core and FFDC files!
SupportSave completed (Duration : 0 minutes 1 seconds).
FLEX-A1-BLUE:root> cleanup

This utility will remove obsoleted files on the local CP.

Be aware the tool will remove all unauthorized code under
following directories on BOTH partitions:

        /bin
        /sbin
        /lib
        /fabos
        /root
        /usr
        /core_files

Note that all the core files will be removed as well.
In case you want to save any of your private code or core,
please copy them before running this command.

Do you want to continue [Y]: Y
Checking //bin, please wait ...
Checking //sbin, please wait ...
Checking //usr, please wait ...
--Remove //usr/share/zoneinfo/Etc/GMT
--Remove //usr/share/zoneinfo/Etc/GMT+0
--Remove //usr/share/zoneinfo/Etc/UTC
--Remove //usr/share/zoneinfo/Europe/Luxembourg
--Remove //usr/share/zoneinfo/UTC
--Remove //usr/apache/bin/httpd.0
--Remove //usr/local/mib_indexes/0
--Remove //usr/local/snmpd.conf
Checking //fabos, please wait ...
--Remove //fabos/lib/libconfig_pharos.so.1.0
--Remove //fabos/man/cat7/AN-1001.7m.gz
--Remove //fabos/man/cat7/AN-1002.7m.gz
...
FLEX-A1-BLUE:root> supportsave -R
Removing all core and FFDC files!
SupportSave completed (Duration : 0 minutes 1 seconds).
FLEX-A1-BLUE:root> cleanup

This utility will remove obsoleted files on the local CP.

Be aware the tool will remove all unauthorized code under
following directories on BOTH partitions:

        /bin
        /sbin
        /lib
        /fabos
        /root
        /usr
        /core_files

Note that all the core files will be removed as well.
In case you want to save any of your private code or core,
please copy them before running this command.

Do you want to continue [Y]: Y
Checking //bin, please wait ...
Checking //sbin, please wait ...
Checking //usr, please wait ...
--Remove //usr/share/zoneinfo/Etc/GMT
--Remove //usr/share/zoneinfo/Etc/GMT+0
--Remove //usr/share/zoneinfo/Etc/UTC
--Remove //usr/share/zoneinfo/Europe/Luxembourg
--Remove //usr/share/zoneinfo/UTC
--Remove //usr/apache/bin/httpd.0
--Remove //usr/local/mib_indexes/0
--Remove //usr/local/snmpd.conf
Checking //fabos, please wait ...
--Remove //fabos/lib/libconfig_pharos.so.1.0
--Remove //fabos/man/cat7/AN-1001.7m.gz
--Remove //fabos/man/cat7/AN-1002.7m.gz
...
--Remove /mnt/fabos/users/admin/.ssh/authorized_keys
--Remove /mnt/fabos/users/admin/.ssh/authorized_keys.admin
--Remove /mnt/fabos/users/admin/.ssh/authorizedKeys.tar
--Remove /mnt/fabos/webtools/bin/web.conf.0
--Remove /mnt/fabos/webtools/bin/httpd.conf.0
--Remove /mnt/fabos/webtools/htdocs/serverstatus.html
--Remove /mnt/fabos/webtools/htdocs/0.weblinker.fcg
Checking /mnt/lib, please wait ...
--Remove /mnt/lib/modules/default/modules.ieee1394map
--Remove /mnt/lib/modules/default/modules.pcimap
--Remove /mnt/lib/modules/default/modules.usbmap
--Remove /mnt/lib/modules/default/modules.ccwmap
--Remove /mnt/lib/modules/default/modules.isapnpmap
--Remove /mnt/lib/modules/default/modules.inputmap
--Remove /mnt/lib/modules/default/modules.ofmap
--Remove /mnt/lib/modules/default/modules.seriomap
--Remove /mnt/lib/modules/default/modules.alias
--Remove /mnt/lib/modules/default/modules.symbols
Checking /mnt/root, please wait ...
--Remove /mnt/root/.ssh/id_rsa
--Remove /mnt/root/.ssh/id_rsa.pub
Checking /mnt/core_files, please wait ...
Checking /mnt/etc/fabos/rbac, please wait ...
--Remove /mnt/etc/fabos/rbac/dynamic.tmp
Finish cleanup of /mnt

FLEX-A1-BLUE:root> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root             377M  235M  123M  66% /
/dev/hda3             193M   16M  167M   9% /core_files
/dev/hda1             377M  264M   93M  74% /mnt
</cli>

===== SSH not working, unable to firmwaredownload =====

===== SSH not working, unable to firmwaredownload =====

<cli prompt='>'>
sansw01:admin> firmwaredownload -p sftp 10.10.10.10,root,/export/software/firmwares/san/v8.2.3c1_pha,Mypasswd

Server IP: 10.10.10.10, Protocol IPv4
Checking system settings for firmwaredownload...
Failed to access sftp://root:************@10.10.10.10//export/software/firmwares/san/v8.2.3c1_pha/release.plist
The server is inaccessible or firmware path is invalid. Please make sure the server name/IP address and the firmware path are valid, the protocol and authentication are supported. It is also possible that the RSA host key could have been changed and please contact the System Administrator for adding the correct host key.
</cli>

<cli prompt='>'>
sansw01:admin> seccryptocfg --default -type SSH -force
Terminating all SSH/SCP sessions running
</cli>

Then retry the download

Additionnaly you can disable IPsec
<cli>
ipsecConfig --disable
</cli>

===== Defect Gbic module =====


porterrshow

sfpshow <port_no> --> check RX and TX value at end (uW values)

  * **RX** value is low (compare with others Gbics with same speed), then Gbic is defect on **host side**, or cable (rare time from Gbic on SAN switch)
  * **Tx** low value problem comes from **Brocade switch Gbic** (or potentially from Gbic slot), or cable

**Example** here with lower RX Power, compare to the same port with no problems, so the problem is related to host side, or cable
<cli prompt='>'>
sansw01:FID128:admin> sfpshow -all
</cli>
Or
<cli prompt='>'>
sansw01:FID128:admin> sfpshow 12/34
Identifier:  3    SFP
Connector:   7    LC
Transceiver: 7004404000000000 4,8,16_Gbps M5 sw Short_dist
RX Power:    -7.1    dBm (193.7uW)   31.6   uW  1258.9 uW  31.6   uW   794.3  uW
TX Power:    -3.1    dBm (486.5 uW)  125.9  uW  1258.9 uW  251.2  uW   794.3  uW


sansw01:FID128:admin> sfpshow 12/33
Identifier:  3    SFP
Connector:   7    LC
Transceiver: 7004404000000000 4,8,16_Gbps M5 sw Short_dist
RX Power:    -3.4    dBm (453.3uW)   31.6   uW  1258.9 uW  31.6   uW   794.3  uW
TX Power:    -3.1    dBm (488.8 uW)  125.9  uW  1258.9 uW  251.2  uW   794.3  uW
</cli>

reset error statistics:\\
**statsclear** --> useful to see new errors


===== POD license not assigned or reserved yet =====

Use the following commands to change licences port assignation
<cli>
licenseport –release <portnumber>
licenseport –reserve <portnumber> 
licenseport --show
</cli>

===== How to find the source of CRCs in a Brocade SAN =====

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009263

Use the command **porterrshow** (or portstatsshow)

If error on a port in section **CRC err**, and **CRC g_eof** is null, then this port is connected to a switch which produce errors, check on this switch to find the problem.

If error on a port in section **CRC err**, and **CRC g_eof** error counters are both incrementing, so the root source is with the attached device’s transmitter or the path from the sending device. 

=== Frames tx/rx N/A counters representing the number of frames transmitted: ===

  * **Enc_in:** 8bit/10bit encoding errors inside frame. Words inside of frames are encoded, if this encoding is corrupted or an error is detected, enc_in is generated. Minimum compliance with the link bit error rate specification on a link continuously receiving frames would cause approximately one error every 20 minutes. Reinitialisation/reboots of the associated Nx-port can also cause these errors. Everything hitting the wire is encoded using 8/10b encoding. The Bit Error Rate (BER) formula is BER= Nerr/Nbits. The BER is calculated by comparing the transmitted sequence of bits to the received bits and counting the number of errors. The ratio of how many bits received in error over the number of total bits received is the BER. This measured ratio is affected by many factors including: signal to noise, distortion, and jitter.
  * **Crc_err:** crc errors - A mathematical formula generates counters at sending port. Receiving port uses the same formula to check and compare. Statistically, crc_err and enc_out errors together imply a GBIC/SFP problem. Also see bad_eof below. CRC and ENC_IN are pointing to a SFP and/or ASIC issue. ENC_out may be seen on loops connecting to a fabric (FMC for example) if a disk is changed or the loop initializes for any other reason. This loop initialization may not be noticeable from ONTAP. Therefore, it is important to know to what a connection is being made and what is to be expected of this connection.  Generally speaking CRC_errs indicate an issue with the SFP.
  * **Too_long:** FC frames are 2148 bytes maximum (frames that were longer than the FC maximum - SOF+header+2112bytes+CRC+EOF). If an eof is corrupted or data generation is incorrect, a too_long error is reported.
  * **Too_short:** The too_short error statistics counter is incremented whenever a frame, bounded by an SOF and EOF is received, and the number of words between the SOF and EOF is less than 7 words (6 words header plus 1 word CRC), i.e. 38 bytes (not 48) including the SOF and EOF. This could be caused by the transmitter or an unreliable link.
  * **Bad_eof:** After a loss of synchronization error, continuous-mode alignment allows the receiver to re-establish word alignment at any point in the incoming bit stream while the receiver is operational. If such a re-alignment occurs, detection of the resulting error condition is dependent upon higher level functions (such as invalid CRC or missing EOF).
  * **Enc_out:** 8bit/10bit encoding errors occurred in words (ordered sets) outside of the FC frame. Words outside of frames are encoded. If this encoding is corrupted or an error is detected, enc_out is generated. It indicates a problem if it increments faster than the link-bit error rate allows, approximately once every 20 minutes for 1 Gbit/s. Statistically, enc_out errors on their own imply a cable/connector problem. Enc_out errors and crc_err together imply a GBIC/SFP problem. Such errors are also expected every time a user brings a port down and up (reboot host, power-cycle storage subsystem, unplug/plug cable or portdisable/portenable, and so on). Such errors will also be generated on a link which has a 1Gbit/s port connected to a 2Gbit/s port when autonegotiation is turned off. Crc and enc_in are more likely to be an SFP and/or ASIC issue. Enc_out is more likely sfp/cable. Also, if connecting to a disk loop (FMC), it's more likely to see them rising, which may not necessarily indicate an issue. To spot a possible issue, investigate other counters that are not part of portErrShow.
  * **Disc c3:** Discard class 3 errors could be generated by the switch when devices send frames without FLOGIing first or with an invalid destination. This error is just reporting that a discard occurred. A frame can be discarded for a number of reasons; Timeout, destination unreachable, zone discard, or other reasons for discard. Most of the time you will see timeout, which means a frame is longer than E_D_TOV in the buffer. Disc/c3 is not trivial to troubleshoot as it is not always the port discarding the frame that is causing the issue.
  * **Link-fail:** If a port remains in the LR Receive State for a period of time greater than a timeout period (R_A_TOV), a link reset protocol timeout will be detected, which results in a link failure condition (enter the NOS transmit state). The link failure also indicates that loss of signal or loss of sync lasting longer than the R_ATOV value was detected while not in the offline state.
  * **Loss sync:** Synchronization failures on either bit or transmission-word boundaries are not separately identifiable and cause loss-of-synchronization errors. Such errors are also expected every time a user brings a port down and up ( reboot host, power-cycle storage subsystem, unplug/plug cable or portdisable/portenable, and so on).
  * **Loss sig:** Occurs when a signal is transmitted but none is being received on the same port. Such errors are also expected every time a user brings a port down and up (reboot host, power-cycle storage subsystem, unplug/plug cable or portdisable/portenable, and so on).
  * **Frjt:** If the fabric cannot process a class 2 frame, an F_RJT is returned
  * **Frbsy:** If a fabric cannot deliver a class 2 frame within E_D_TOV a F_BSY will be returned.
  * **c3-timeout tx:** The number of transmit class 3 frames discarded at the transmission port due to timeout (platform- and port-specific).  This indicates an issue with the device connected to the switch.
  * **c3-timeout rx:** The number of receive class 3 frames received at this port and discarded at the transmission port due to timeout (platform- and port-specific).  This indicates an issue with the port on the switch.

//Note:// These errors should always be seen in relation to each other and in relation to the device that is being connected. There is a difference between a Loop with 28 disks being connected and a HBA in fabric mode.  Additionally, CRCs by themselves with no other errors likely have a different cause than CRCs that are accompanied by enc_out errors.
===== Buffer credit problem =====

Check the parameters **tim64_txcrd_z** (Time BB_credit zero) and **stat64_inputBuffersFull** (Occasions on which input buffers are full). 
<cli prompt='>'>
besw32:admin> portstats64show 3/5
...
tim64_rdy_pri	7 226 622
tim64_txcrd_z	14 338 091 729
stat64_rateTxFrame	69 017
...
stat64_inputBuffersFull  20
</cli>

Check the port buffer usage, on each port you can check if you have enough buffer credit if the parameter **stat64_inputBuffersFull** is equal to zero, and/or **tim64_txcrd_z** else you have to increase the buffercredit on this port, and if it's an ISL (E-port), add also buffercredit on the paired switch. 
<cli prompt='>'>
SWSAN1:admin> portbuffershow
User  Port   Lx   Max/Resv Buffer Needed     Link   Remaining
Port  Type  Mode  Buffers  Usage  Buffers  Distance  Buffers
--------------------------------------------------------------
  0     E     -      -       16       24       10km
  1           -      -        0       -        -
  2           -      -        0       -        -
  3     F     -      -       16       -        -          76
--------------------------------------------------------------
</cli>

Change the buffercredit value for the port, it's **diruptive** (connection needs to be redundant to keep your host online.
<cli prompt='>'>
besw32:admin> portcfgfportbuffers --enable 3/5 24
</cli>

Do not forget to clear the stats after changing the buffer credit value
<cli prompt='>'>
besw32:admin> portstatsclear 3/5
</cli>

Or **statsclear** 

===== Brocade SAN unable to connect using ssh with public keys =====

After importkeys (using sshutil importpubkey), the SAN switch ask for a password!
 
On the SAN switch first connect using **root** account, and list the **authorized_keys** file rights.
<cli prompt='>'>
[root@labaix] /root> ssh root@labsan01
root@labsan-blue's password:
Disclaimer for Root and Factory Accounts Usage!
LABSAN01:FID128:root>  ps -ef | grep ssh
root     18474     1  0 12:36 ?        00:00:00 /usr/sbin/sshd
root     18475 18474  2 12:36 ?        00:00:00 sshd: root@pts/0
root     18567 18482  0 12:37 pts/0    00:00:00 grep ssh

LABSAN01:FID128:root> cd /fabos/users/admin/.ssh

LABSAN01:FID128:root> ls -l
total 28
-rw-r--r--   1 root     admin       10240 Apr 12 20:05 authorizedKeys.tar
-rw-r--r--   1 root     admin         398 Apr 12 20:05 authorized_keys
-rw-------   1 root     admin         398 Apr 12 20:05 authorized_keys.admin
-rw-------   1 root     admin         796 Apr 12 19:38 authorized_keys.lpardeploy
-rw-r--r--   1 root     admin         134 Jul 14  2016 environment
</cli>

Now try a ssh connection in debug mode using a user defined on the SAN and with ssh public keys from your lab server for example admin
<cli prompt='>'>
[root@labaix] /root> ssh -vv admin@labsan01
OpenSSH_6.0p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Failed dlopen: /usr/krb5/lib/libkrb5.a(libkrb5.a.so):   0509-022 Cannot load module /usr/krb5/lib/libkrb5.a(libkrb5.a.so).
        0509-026 System error: A file or directory in the path name does not exist.

debug1: Error loading Kerberos, disabling Kerberos auth.
...
debug2: key: /root/.ssh/id_rsa (20080bf8)
debug2: key: /root/.ssh/id_dsa (0)
debug2: key: /root/.ssh/id_ecdsa (0)
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug1: Next authentication method: password
admin@labsan01's password:
</cli>

A password is required 

So now connect again to the san switch as root and change the righits to known_hosts file for my admin user:
<cli prompt='>'>
[root@labaix] /root> ssh root@labsan01
root@labsan-blue's password:
Disclaimer for Root and Factory Accounts Usage!

LABSAN01:FID128:root> cd /fabos/users/admin/.ssh
LABSAN01:FID128:root>  chmod 644 authorized_keys.admin

LABSAN01:FID128:root> ls -l
total 28
-rw-r--r--   1 root     admin       10240 Apr 12 20:05 authorizedKeys.tar
-rw-------   1 root     admin         398 Apr 12 20:05 authorized_keys
-rw-r--r--   1 root     admin         398 Apr 12 20:05 authorized_keys.admin
-rw-------   1 root     admin         796 Apr 12 19:38 authorized_keys.lpardeploy
-rw-r--r--   1 root     admin         134 Jul 14  2016 environment
</cli>

And now retry a connection as admin:
<cli prompt='>'>
[root@labaix] /root> ssh -vv admin@labsan01
...
debug2: key: /home/admin/.ssh/id_rsa (2004f6a8)
debug2: key: /home/admin/.ssh/id_dsa (0)
debug2: key: /home/admin/.ssh/id_ecdsa (0)
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/admin/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 4a:c6:ac:83:9f:26:b7:9e:0e:b2:21:b6:23:c1:94:cd
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
...
LABSAN01:FID128:admin>
</cli>

===== Brocade SAN other ssh problems =====

First step:
To configure a user for public key authentication:
<cli prompt='>'>
switch:admin> sshutil allowuser username
Allowed user has been successfully changed to username.
</cli>

If not enough,

Second step:

If you have other problems, you can do an **scp** of the file /etc/sshd_config (on brocade), to be able to modify it on a UNIX machine, and then do an scp back to the SANS swicth, then kill the sshd process as **root**, it will start again (do not use kill -9). You 'll be disconnected, reconnect again.
<cli prompt='>'>
[root@labaix] /root> ssh root@labsan01
root@labsan-blue's password:
Disclaimer for Root and Factory Accounts Usage!
LABSAN01:FID128:root>  ps -ef | grep ssh
root     18474     1  0 12:36 ?        00:00:00 /usr/sbin/sshd
root     18475 18474  2 12:36 ?        00:00:00 sshd: root@pts/0
root     18567 18482  0 12:37 pts/0    00:00:00 grep ssh

LABSAN01:FID128:root> kill 18474
</cli>

As root check the files permissions
<cli prompt='>'>
CURB04:FID128:root> cd /fabos/users/admin/
CURB04:FID128:root> ls -l
total 16
-rw-r--r--   1 admin    admin         507 May 24  2018 .bash_logout
-rw-r--r--   1 admin    admin          27 May 24  2018 .inputrc
-rw-r--r--   1 admin    admin        1347 May 24  2018 .profile
drwxr-xr-x   2 admin    admin        4096 Nov  7  2018 .ssh/
CURB04:FID128:root> cd .ssh/
CURB04:FID128:root> ls -l
total 24
-rw-r--r--   1 root     root        10240 Nov  7  2018 authorizedKeys.tar
-rw-------   1 root     root          790 Nov  7  2018 authorized_keys
-rw-------   1 admin    admin         790 Feb  9  2018 authorized_keys.admin
-rw-r--r--   1 admin    admin         134 May 24  2018 environment
</cli>

check authorized key file for admin
wiki

User Tools

Site Tools

Page Tools
