wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== AIX disk queue depth tuning for performance ======

http://www.iozone.org/

Use perpmr to check perf issues:\\
https://public.dhe.ibm.com/aix/tools/perftools/perfpmr/perf72/
  ==> check the **.out** files 
  
===== disk =====

Disk queue depth tuning. This is important for multi-user, multi-threaded applications doing a lot of IO.

If a disk is too busy (%tm_act) near 100%, increase the queue_depth
<cli prompt='#'>
[root@aix01]/root# iostat 2

System configuration: lcpu=64 drives=30 ent=5.00 paths=240 vdisks=0

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        434.0               23.7   9.9   66.3      0.1   5.1  102.5

Disks:         % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0            0.0       0.0       0.0          0         0
hdisk99          17.0     972.0      65.0          0      1944
hdisk94           7.5     200.0      29.5          0       400
hdisk100         12.5     812.0      59.5          0      1624
hdisk93          47.5     3661.5     234.5          0      7323
hdisk89          98.0     1569.5     253.5          0      3139
</cli>

Check the current value used for disk hdisk89
<cli prompt='#'>
[root@aix01]/root# echo scsidisk hdisk89 | kdb | grep queue_depth
    ushort queue_depth  = 0x20;        < In Hex.
[root@aix01]/root# echo "ibase=16 ; 20" | bc
          32
</cli>

<cli prompt='#'>
[root@aix01]/root# lsattr -El hdisk89 -a queue_depth
queue_depth 32 Queue DEPTH True
</cli>

Change the value online if possible, to 48, or 64. If it doesn't apply online, use the option **-P** to apply in ODM, and reboot
<cli prompt='#'>
[root@aix01]/root# chdev -l hdisk89 -a queue_depth=48
</cli>

===== LPAR CPU =====

Best practice is to have a ratio between Virtual CPU and entitled capacity, not higher than 2.\\
//Example, bad practice://  --> besides, we are here exceeding the Virtual CPUs to entitled CPUs ratio by 3.2 - and it should not be higher the 2:
  Entitled Capacity:   5.00
  Online Virtual CPUs: 16
  
  
==== Purpose ====

The purpose of this document is to describe how IOs are queued the disk device driver and the adapter device driver and SDDPCM and to explain how these can be tuned to increase performance. And this also includes the use of VIO. This information is also useful for AIX LPARs using other multi-path code as well.

== Where this stuff fits in the IO stack ==

Following is the IO stack from the application to the disk:

Application
File system (optional)
LVM device driver (optional)
SDDPCM or other multi-path driver (if used)
hdisk device driver
adapter device driver
interconnect to the disk
Disk subsystem
Disk

Note that even though the disk is attached to the adapter, the hdisk driver code is utilized before the adapter driver code. So this stack represents the order software comes into play over time as the IO traverses the stack.

== Why do we need to simultaneously submit more than one IO to a disk? ==

This improves performance. And this would be performance from an application's point of view. This is especially important with disk subsystems where a virtual disk (or LUN) is backed by multiple physical disks. In such a situation, if we only could submit a single IO at a time, we'd find we get good IO service times, but very poor throughput. Submitting multiple IOs to a physical disk allows the disk to minimize actuator movement (using an "elevator" algorithm) and get more IOPS than is possible by submitting one IO at a time. The elevator analogy is appropriate. How long would people be waiting to use an elevator if only one person at a time could get on it? In such a situation, we'd expect that people would wait quite a while to use the elevator (queueing time), but once they got on it, they'd get to their destination quickly (service time). So submitting multiple in-flight IOs to a disk subsystem allows it to figure out how to get the most throughput and fastest overall IO service time.

Theoretically, the IOPS for a disk is limited by queue_depth/(average IO service time). Assuming a queue_depth of 3, and an average IO service time of 10 ms, this yields a maximum throughput of 300 IOPS for the hdisk. And for many applications this may not be enough thruput.

== Where are IOs queued? ==

As IOs traverse the IO stack, AIX needs to keep track of them at each layer. So IOs are essentially queued at each layer. Generally, some number of in flight IOs may be issued at each layer and if the number of IO requests exceeds that number, they reside in a wait queue until the required resource becomes available. So there is essentially an "in process" queue and a "wait" queue at each layer (SDDPCM are a little more complicated).

At the file system layer, file system buffers limit the maximum number of in flight IOs for each file system. At the LVM device driver layer, hdisk buffers limit the number of in flight IOs. SDDPCM on the other hand does not queue IOs before sending them to the disk device driver. The hdisks have a maximum number of in flight IOs that's specified by its queue_depth attribute. And FC adapters also have a maximum number of in flight IOs specified by num_cmd_elems. The disk subsystems themselves queue IOs and individual physical disks can accept multiple IO requests but only service one at a time. Here are an ESS hdisk's attributes:

<cli prompt='#'>
# lsattr -El hdisk33
PR_key_value   none                      Reserve Key                                True
location       Location                  Label                                      True
lun_id         0x5515000000000000        Logical Unit Number ID                     True
lun_reset_spt  yes                       Support SCSI LUN reset                     True
max_transfer   0x40000                   N/A                                        True
node_name      0x5005076300c096ab        FC Node Name                               False
pvid           none                      Physical volume identifier                 False
q_type         simple                    Queuing TYPE                               True
qfull_dly      20                        delay in seconds for SCSI TASK SET FULL    True
queue_depth    20                        Queue DEPTH                                True
reserve_policy single_path               Reserve Policy                             True
rw_timeout     60                        READ/WRITE time out value                  True
scbsy_dly      20                        delay in seconds for SCSI BUSY             True
scsi_id        0x620713                  SCSI ID                                    True
start_timeout  180                       START unit time out value                  True
ww_name        0x5005076300cd96ab        FC World Wide Name                         False
</cli>
The default queue_depth is 20, but can be changed to as high as 256 for ESS, DS6000 and DS8000. One can display allowable values with:

<cli prompt='#'>
# lsattr -Rl hdisk33 -a queue_depth
1...256 (+1)
</cli>

indicating the value can be anywhere from 1 to 256 in increments of 1. One can use this command to see any allowable value for attributes which can be changed (showing a value of True in the last field of "lsattr -El <device>" for the device using:
<cli prompt='#'>
# lsattr -Rl <device> -a <attribute>
</cli>

Here's a FC adapter's attributes:
<cli prompt='#'>
# lsattr -El fcs0
bus_intr_lvl    65703            Bus interrupt level                                  False
bus_io_addr     0xdec00          Bus I/O address                                      False
bus_mem_addr    0xe8040000       Bus memory address                                   False
init_link       al               INIT Link flags                                      True
intr_priority   3                Interrupt priority                                   False
lg_term_dma     0x800000         Long term DMA                                        True
max_xfer_size   0x100000         Maximum Transfer Size                                True
num_cmd_elems   200              Maximum number of COMMANDS to queue to the adapter   True
pref_alpa       0x1              Preferred AL_PA                                      True
sw_fc_class     2                FC Class for Fabric                                  True
</cli>
The default queue depth (num_cmd_elems) for FC adapters is 200 but can be increased up to 2048 for most adapters.

Both the hdisk and adapter drivers have an "in process" and "wait" queues. Once the queue limit is reached, the IOs wait until an IO completes, freeing up a slot in the service queue. The in process queue is also sometimes referred to as the "service" queue

It's worth mentioning, that many applications will not generate many in flight IOs, especially single threaded applications that don't use asynchronous IO. Applications that use asynchronous IO are likely to generate more in flight IOs.

== What tools are available to monitor the queues? ==

For AIX, one can use iostat (at AIX 5.3 or later) and sar (5.1 or later) to monitor the hdisk driver queues. The **iostat -D** command generates output such as:
<cli prompt='#'>
# iostat -D
hdisk6    xfer: %tm_act  bps  tps bread  bwrtn
                    4.7 2.2M 19.0   0.0  2.2M
          read: rps avgserv minserv maxserv timeouts fails
                0.0     0.0     0.0     0.0        0     0
         write:   wps avgserv minserv maxserv timeouts fails
                 19.0    38.9     1.1   190.2        0     0
         queue: avgtime mintime maxtime avgwqsz avgsqsz sqfull
                   15.0     0.0    83.7     0.0     0.0    136
</cli>

Here, the avgwqsz is the average wait queue size, and avgsqsz is the average service queue size. The average time spent in the wait queue is avgtime. The sqfull value has changed from initially being a count of the times we've submitted an IO to a full queue, to now where it's the rate of IOs per second submitted to a full queue. The example report shows the prior case (a count of IOs submitted to a full queue), while newer releases typically show decimal fractions indicating a rate. It's nice that iostat -D separates reads and writes, as we would expect the IO service times to be different when we have a disk subsystem with cache. The most useful report for tuning is just running "iostat -D" which shows statistics since system boot, assuming the system is configured to continuously maintain disk IO history (run # lsattr -El sys0, or smitty chgsys to see if the iostat attribute is set to true). And the author's favorite iostat command flags are "iostat -RDTl <interval> <#intervals>".

From the application's point of view, the length of time to do an IO is its service time plus the time it waits in the hdisk wait queue.

The sar -d command changed at AIX 5.3, and generates output such as:
<cli prompt='#'>
# sar -d
16:50:59     device    %busy    avque    r+w/s    Kbs/s   avwait   avserv
16:51:00     hdisk1      0      0.0        0        0      0.0      0.0
             hdisk0      0      0.0        0        0      0.0      0.0
</cli>

The avwait and avserv are the average times spent in the wait queue and service queue respectively. And avserv here would correspond to avgserv in the iostat output. The avque value changed; at AIX 5.3, it represents the average number of IOs in the wait queue, and prior to 5.3, it represents the average number of IOs in the service queue.

SDDPCM provides the "pcmpath query devstats" and "pcmpath query adaptstats" commands to show hdisk and adapter queue statistics. You can refer to the SDDPCM manual for syntax, options and explanations of all the fields. Here's some devstats output for a single LUN:
<cli prompt='#'>
# pcmpath query devstats
Device #: 0
=============
Total Read Total Write Active Read Active Write Maximum
I/O: 29007501 3037679 1 0 40
SECTOR: 696124015 110460560 8 0 20480

Transfer Size: <= 512 <= 4k <= 16K <= 64K > 64K
21499 10987037 18892010 1335598 809036
</cli>

and here's some adaptstats output:
<cli prompt='#'>
# pcmpath query adaptstats
Adapter #: 0
=============
Total Read Total Write Active Read Active Write Maximum
I/O: 439690333 24726251 7 0 258
SECTOR: 109851534 960137182 608 0 108625
</cli>

Here, we're mainly interested in the Maximum field which indicates the maximum number of IOs submitted to the device since system boot.

For SDDPCM, if the Maximum value equals the hdisk's queue_depth, then the hdisk driver queue was filled during the interval, and increasing queue_depth is usually appropriate.

One can similarly monitor adapter IOPS with # iostat -at <interval> <# of intervals> and for adapter queue information, run # iostat -aD, optionally with an interval and number of intervals. For FC adapters, the fcstat command provides information on the adapter queue and resource use, and can tell us if we need to increase its queue sizes.

For adapter queues, the fcstat command is used and is discussed below.

How to tune

First, one should not indiscriminately just increase these values. It's possible to overload the disk subsystem or cause problems with device configuration at boot. So the approach of adding up the hdisk's queue_depths and using that to determine the num_cmd_elems isn't necessarily the best approach. Instead, it's better to use the maximum number of submitted IOs to each device for tuning. When you increase the queue_depths and number of in flight IOs that are sent to the disk subsystem, the IO service times are likely to increase, but throughput will also increase. If IO service times start approaching the disk timeout value, then you're submitting more IOs than the disk subsystem can handle. If you start seeing IO timeouts and errors in the error log indicating problems completing IOs, then this is the time to look for hardware problems or to make the pipe smaller.

A good general rule for tuning queue_depths, is that one can increase queue_depths until IO service times start exceeding 15 ms for small random reads or writes or one isn't filling the queues. Once IO service times start increasing, we've pushed the bottleneck from the AIX disk and adapter queues to the disk subsystem. Two approaches to tuning queue depth are 1) base the queue depths on actual IO requests your application generate or 2) use a test tool to see what the disk subsystem can handle and tune the queues based on what the disk subsystem can handle. The ndisk tool (part of the nstress package available on the internet at http://www-941.ibm.com/collaboration/wiki/display/WikiPtype/nstress) can be used to stress the disk subsystem to see what it can handle. The author's preference is to tune based on your application IO requirements, especially when the disk is shared with other servers.

For tuning, we can categorize the situation into four categories:

    We're filling up the queues and IOs are waiting in the hdisk or adapter drivers
    We're not filling up the queues, and IO service times are good
    We're not filling up the queues, and IO service times are poor
    We're not filling up the queues, and we're sending IOs to the storage faster than it can handle and it loses the IOs


We want to tune the queues to be in either situation 2 or 3. If we're in situation 3, that indicates a bottleneck beyond the hdisk driver which will typically be in the disk subsystem itself, but could also be in the adapter driver or SAN fabric.

Situation 4 is something we do want to avoid. All disks and disk subsystem have limits regarding the number of in-flight IOs they can handle, mainly due to memory limitations to hold the IO request and data. When the storage loses IOs, the IO will eventually time out at the host, recovery code will be used and resubmit the IO, but in the meantime transactions waiting on that IO will be stalled. This isn't a desirable situation, as the CPU ends up doing more work to handle IOs than necessary. If the IO eventually fails, then this can lead to an application crash or worse. So be sure to check your storage documentation to understand its limits.

Then after running your application during peak IO periods look at the statistics and tune again.


Regarding the qdepth_enable parameter for SDD, the default is yes which essentially has SDD handling the IOs beyond queue_depth for the underlying hdisks. Setting it to no results in the hdisk device driver handling them in its wait queue. In other words, with qdepth_enable=yes, SDD handles the wait queue, otherwise the hdisk device driver handles the wait queue. There are error handling benefits to allowing SDD to handle these IOs, e.g., if using LVM mirroring across two ESSs. With heavy IO loads and a lot of queueing in SDD (when qdepth_enable=yes) it's more efficient to allow the hdisk device drivers to handle relatively shorter wait queues rather than SDD handling a very long wait queue by setting qdepth_enable=no. In other words, SDD's queue handling is single threaded where each hdisk driver has its own thread. So if error handling is of primary importance (e.g. when LVM mirroring across disk subsystems) then leave qdepth_enable=yes. Otherwise, setting qdepth_enable=no more efficiently handles the wait queues when they are long. Note that one should set the qdepth_enable parameter via the datapath command as it's a dynamic change that way (using chdev is not dynamic for this parameter).

If error handling is of concern, then it's also advisable, assuming the disk is SAN switch attached, to set the fscsi device attribute fc_err_recov to fast_fail rather than the default of delayed_fail, and also change the fscsi device dyntrk attribute to yes rather than the default of no. These attributes assume a SAN switch that supports this feature.


== What are reasonable average IO service times? ==

What is good or reasonable is somewhat a factor of the technology of the storage and the storage cache sizes. Assuming no IOs are queued to a disk, a typical read will take somewhere from 0 to 15 ms, or so, depending on how far the actuator has to travel (seek time), how long it takes the disk to rotate to the right sector (rotation time), and how long it takes to read the data (transfer time). Then the data must move from the storage to the host. Typically the time is dominated by seek time + rotation time, though for large IOs transfer time also can be significant. Sometimes the data will be in disk subsystem read cache, in which case the IO service time is around 1 ms. Typically for large disk subsystems that aren't overloaded, IO service times will average around 5-10 ms. When small random reads start averaging greater than 15 ms, this indicates the storage is getting busy.

Writes typically go to write cache (assuming it exists) and then these average typically less than about 2.5 ms. But there are exceptions. If the storage is synchronously mirroring the data to a remote site, writes can take much longer. And if the IO is large (say 64 KB or larger) then the transfer time becomes more significant and the average time is slightly worse. If there's no cache, then writes take about the same as reads.

If the IO is large block sequential, then besides the increased transfer time, we expect IOs to queue at the physical disk, and IO service times to be much longer on average. E.G., if an application submits 50 IOs (say 50 64 KB IOs reading a file sequentially) then the first few IOs will have reasonably good IO service times, while the last IO will have had to wait for the other 49 to finish first, and will have a very long IO service time.

IOs to SSDs are typically less than 1 ms, and for SSDs in disk subsystems, typically less than 2 ms, and on occasion higher.

Tuning the FC adapter num_cmd_elems

The fcstat command is perhaps the easiest tool to look for blocked IOs in the adapter's queues, e.g.:
<cli prompt='#'>
# fcstat fcs0

FIBRE CHANNEL STATISTICS REPORT: fcs0
...
FC SCSI Adapter Driver Information
No DMA Resource Count: 0
No Adapter Elements Count: 104848
No Command Resource Count: 13915968
...
</cli>
The values for "No Adapter Elements Count" and "No Command Resource Count" are the number of times since boot that an IO was temporarily blocked due to an inadequate num_cmd_elems attribute value. Non-zero values indicate that increasing num_cmd_elems may help improve IO service times. Of course if the value increments slowly, then the improvement may be very small, while quickly incrementing values means tuning is more likely to have a measurable improvement in performance.

Like the hdisk queue_depth attribute, changing the num_cmd_elems value requires stopping use of the resources or a reboot.


== Queue depths with VSCSI VIO ==

When using VIO, one configures VSCSI adapters (for each virtual adapter in a VIOS, known as a vhost device, there will be a matching VSCSI adapter in a VIOC). These adapters have a fixed queue depth that varies depending on how many VSCSI LUNs are configured for the adapter. There are 512 command elements of which 2 are used by the adapter, 3 are reserved for each VSCSI LUN for error recovery and the rest are used for IO requests. Thus, with the default queue_depth of 3 for VSCSI LUNs, that allows for up to 85 LUNs to use an adapter: (512 - 2) / (3 + 3) = 85 rounding down. So if we need higher queue depths for the devices, then the number of LUNs per adapter is reduced. E.G., if we want to use a queue_depth of 25, that allows 510/28= 18 LUNs. We can configure multiple VSCSI adapters to handle many LUNs with high queue depths. each requiring additional memory. One may have more than one VSCSI adapter on a VIOC connected to the same VIOS if you need more bandwidth.

Also, one should set the queue_depth attribute on the VIOC's hdisk to match that of the mapped hdisk's queue_depth on the VIOS.

For a formula, the maximum number of LUNs per virtual SCSI adapter (vhost on the VIOS or vscsi on the VIOC) is =INT(510/(Q+3)) where Q is the queue_depth of all the LUNs (assuming they are all the same).

Note that to change the queue_depth on an hdisk at the VIOS requires that we unmap the disk from the VIOC and remap it back, or a simpler approach is to change the values in the ODM (e.g. # chdev -l hdisk30 -a queue_depth=20 -P) then reboot the VIOS.

For LV VSCSI hdisks, where multiple VIOC hdisks are created from a single VIOS hdisk, then one may take a dedicated resource, shared resource or an in between approach to the VIOS hdisk queue slots. See the section below entitled Further theoretical thoughts on shared vs. dedicated resources.


== Queue depths with NPIV VIO ==

When using NPIV, we have virtual FC adapters (vFC) and real FC adapters, and often have multiple vFCs tied to a single real FC adapter.

If you increase num_cmd_elems on the virtual FC (vFC) adapter, then you should also increase the setting on the real FC adapter.

You can use the fcstat command for both the virtual adapter as well as the real adapter for tuning purposes.


A special note on the FC adapter max_xfer_size attribute

This attribute for the fscsi device, which controls the maximum IO size the adapter device driver will handle, also controls a memory area used by the adapter for data transfers. When the default value is used (max_xfer_size=0x100000) the memory area is 16 MB in size. When setting this attribute to any other allowable value (say 0x200000) then the memory area is 128 MB in size. At AIX 6.1 TL2 or later a change was made for virtual FC adapters so the DMA memory area is always 128 MB even with the default max_xfer_size. This memory area is a DMA memory area, but it is different than the DMA memory area controlled by the lg_term_dma attribute (which is used for IO control). The default value for lg_term_dma of 0x800000 is usually adequate.

So for heavy IO and especially for large IOs (such as for backups) it's recommended to set max_xfer_size=0x200000.

The fcstat command can also be used to examine whether or not increasing num_cmd_elems or max_xfer_size could increase performance
<cli prompt='#'>
# fcstat fcs0
...
FC SCSI Adapter Driver Information
No DMA Resource Count: 0
No Adapter Elements Count: 0
No Command Resource Count: 0
</cli>
This shows an example of an adapter that has sufficient values for num_cmd_elems and max_xfer_size. Non zero value would indicate a situation in which IOs queued at the adapter due to lack of resources, and increasing num_cmd_elems and max_xfer_size would be appropriate.

Note that changing max_xfer_size uses memory in the PCI Host Bridge chips attached to the PCI slots. The salesmanual, regarding the dual port 4 Gbps PCI-X FC adapter states that "If placed in a PCI-X slot rated as SDR compatible and/or has the slot speed of 133 MHz, the AIX value of the max_xfer_size must be kept at the default setting of 0x100000 (1 megabyte) when both ports are in use. The architecture of the DMA buffer for these slots does not accommodate larger max_xfer_size settings"

If there are too many FC adapters and too many LUNs attached to the adapter, this will lead to issues configuring the LUNs. Errors will look like:
<cli prompt='#'>
LABEL: DMA_ERR
IDENTIFIER: 00530EA6

Date/Time: Mon Mar 3 10:48:09 EST 2008
Sequence Number: 863
Machine Id: 00C3BCB04C00
Node Id: p595back
Class: H
Type: UNKN
Resource Name: PCIDMA
Resource Class: NONE
Resource Type: NONE
Location:

Description
UNDETERMINED ERROR

Probable Causes
SYSTEM I/O BUS
SOFTWARE PROGRAM
ADAPTER
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
BUS NUMBER
FFFF FFFF 9000 00E4
CHANNEL UNIT ADDRESS
0000 0000 0000 018B
ERROR CODE
0000 0000 1000 0003
</cli>

So if you get these errors, you'll need to change the max_xfer_size back to the default value. Also note that if you are booting from SAN, if you encounter this error, you won't be able to boot, so be sure to have a back out plan if you plan to change this and are booting from SAN.

Further theoretical thoughts on shared vs. dedicated resources

The astute reader will have considered the fact that typically we have many hdisk drivers sharing multiple adapters and adapter drivers, thus, the FC queue slots are a shared resource for the hdisk drivers:



Thus, it's possible to ensure that we never fill the adapter queues, by making SUM(hdisk0 queue_depth, hdisk1 queue_depth, ... hdiskM queue_depth) <= SUM (fcs0 num_cmd_elems, fcs1 num_cmd_elems, ... fcsN num_cmd_elems). This assumes that IO are evenly spread across the adapters. And most multi-path code does balance IOs across the adapters (or at least can).

Though often, environments have many more hdisks than FC ports, and ensuring we won't fill the adapter drivers can lead to small values for queue_depth, and full queues on the hdisk drivers.

So there is the dedicated resource approach, the shared resource approach, and in between dedicated and shared. Taking this simple example where Q represents the queue depth for the device driver:



This would be considered a dedicated resource approach, where 10 of the adapter driver queue slots are dedicated to each hdisk driver. Here we know we'll never submit an IO to a full queue on the adapter driver.

Alternatively:



This would be considered a shared resource approach where the 10 adapter queue slots could be filled up from a single hdisk driver.

And here's an example of something in between:



Here, there will always be at least 5 queue slots available in the adapter driver for either hdisk driver.


There are pros and cons to each approach. The benefit of the dedicated resource approach is that the resources allocated will always be available but typically there will be fewer resources available to each user of the resource (here the resource we're considering is the adapter queue slots, and the users of the resource are the hdisk drivers). The benefit of the shared resource approach is that we'll have more resources for an individual user of the resource when it needs it and it will be able to get greater thruput than in the dedicated resource approach. The author generally prefers a shared resource approach, as generally provides the best thruput and price performance.

Note that this situation of shared resources occurs in several possible ways beyond hdisk drivers using adapter drivers. It is also involved when:

    Several LV VSCSI hdisks for a single hdisk on a VIOS
    Several vFC adapters using a single real FC adapter
    Several LPARs using the same disk subsystem

==== Best practice for FC adapter ====

If you have VIOS, first change it on VIOS, and reboot, then change it on LPARs (else your LPAR won't boot)
<cli prompt='#'>
# chdev -l fcs0 -a num_cmd_elems=2048 -P
# chdev -l fcs0 -a max_xfer_size=0x200000 -P

# chdev -l fscsi0 -a fc_err_recov=fast_fail -P
# chdev -l fscsi0 -a dyntrk=yes -P
</cli>


<code>
I/O:
from the kernel traces:
Top 10 Worst I/Os by Elapsed Time_
              FC   pblock     KB     strategy                bstart             SCSI_intr    Total   TYPE        HDISK
              --   ------     --     --------                ------             ---------    -----   ----        -----
8000002D00000002 6AAF0E30      8     2.352922    0.002     2.352924    6.552     2.359476    6.554   READ      hdisk66
8000002D00000002  D73D5F0      8     0.799024    0.002     0.799026    6.348     0.805374    6.350   READ      hdisk76
8000002D00000002 2D44A200    256     2.589451    0.002     2.589453    6.199     2.595652    6.201   READ      hdisk68
8000002D00000000 79739254    121     1.327328    0.151     1.327479    6.045     1.333524    6.196  WRITE      hdisk69
8000002D00000002 26C4CDB0    128     1.799651    0.001     1.799652    6.057     1.805709    6.058  WRITE      hdisk66
8000002D00000002 24ACF510    128     1.799907    0.000     1.799907    5.814     1.805721    5.814  WRITE      hdisk68
8000002D00000002 79737D75  138.5     1.105284    0.597     1.105881    5.118     1.110999    5.715  WRITE      hdisk69
8000002D00000003 2D543A00    256     1.329969    0.003     1.329972    5.634     1.335606    5.637   READ      hdisk65
8000002D00000003  7D3F440      8     0.799197    0.002     0.799199    5.557     0.804756    5.559   READ      hdisk74
8000002D00000002 78951D75  138.5     1.102060    0.000     1.102060    5.427     1.107487    5.427  WRITE      hdisk73
--> we are just above to say the SCSI times are not low enough.
--> the SCSI times should be lower or around 3 mili seconds, for the WRITE operations even lower.
--> this need to be discussed with the storage support team.

File system level:
/var/adm/syslog/warning.log:
Oct 5 12:05:07  kern:warn|warning unix: JFS2 sync time exceeded 600 seconds for global sync

--> the jfs2 buffers need to be increased:
--> # ioo -p -o j2_dynamicBufferPreallocation=256

NW:
--> the NW is OK, except the VEA buffers are exceeded:
ent0       Buffer Allocations
           Tiny Small   Med   Lrg  Huge
BufSz(byt)  512  2048 16384 32768 65536
MaxPossibl 4096  4096  2048   128    64
---------------------------------------
     Min:  2048  2048   256    64    64
     Max:  4096  4096   512   128   128
  Allocd:  2048  2048   256    64    64
Reg_Befr:  2048  2048   256    64    64
Reg_Aftr:  2048  2047   256    64    64
Reg_Dlta:     0    -1     0     0     0
    MaxA:  2048  2138   512   128    64
MaxA>Min:     -     *     *     *     -

--> Please do as follows:
--> # chdev -l ent0 -a max_buf_small=4096 -a min_buf_small=2560 -P
--> # chdev -l ent0 -a max_buf_medium=896 -a min_buf_medium=640 -P
--> # chdev -l ent0 -a max_buf_large=128 -a min_buf_large=128 -P

</code>
wiki

User Tools

Site Tools

Page Tools
