Analysis of StorageTek 9176 Fibre Channel Disk Array

A series of timing tests were run on our StorageTek 9176 fibre disk array. The results of these tests are discussed in this report. Results include showing the original data and comparison graphs. The configuration of the various components and how the tests were conducted are also shown.

Configuration

        StorageTek 9176:
        Fibre disk array
        2 controllers, each configured with a 128 MB cache buffer
        24 72 GB drives (plus 1 hot spare)
        3 Raid Groups were created, all using Raid 5, each with one LUN
        Controller A:  10 disks (614 GB) -- Referred to as "Large" below
        Controller B:  8 disks (478 GB) -- Referred to as "Medium"  below
                       6 disks (341 GB) -- Referred to as "Small" below
        Cache Block Size = 16 KB
        Segment Storage Size = 256 KB
        Enabled Read & Write Caching

        Host System:
        IBM Enterprise Server H70 with Processors running AIX 4.3.3
        2 GB memory
        3 Emulex LP8000 Fibre Channel Adapters, 256 KB TRANSMIT queue size

Assumptions:

• All tests were run on a fairly quiet system. HPSS processes were running, but under a minimal load.
• Because of previous experience in the group, each LUN had some basic settings assigned to it before testing. Both read caching and write caching were enabled and write caching with mirroring was turned off.
• The reading and writing of large files was the basis for these tests. All reads and writes used a file size of 4 GB. This assured the flushing of all system caches.
• The UNIX utility dd was used for all I/O. Reads were done from the raw disk (e.g., rhdisk10) to /dev/null. Writes were from /dev/zero to the raw logical volume (e.g., rlv10). The block sizes tested within dd were: 16 KB, 32 KB, 64 KB, 128 KB, 256 KB, 512 KB, 1024 KB, and 2048 KB. Initial reads tests were also done with a block size of 4096 KB, but the differences between 2048 KB and 4096 KB were negligible. Thus, further testing using the 4096 KB block size was dropped.

        Example:
                read:   dd if=/dev/hdisk10 of=/dev/null bs=256k count=16000
                write:  dd if=/dev/zero of=/dev/rlv10 bs=128k count=32000

• Most tests were run three times and an average figure was derived and used for analysis. Little difference was noted between the multiple runs. Thus, later tests were frequently done with only one timing run.
• Multiple reads or writes were launched 5 seconds apart. For the concurrent reads and writes (3 or 10) to more than one LUN, the I/O operations were alternated between the LUNs. For example, a read to LUN A, followed by a read to LUN B, etc.

Results & Analysis:

The StorageTek array is controlled with the "SANtricity Storage Manager" GUI. The GUI is fairly easy to use and had fairly good on-line documentation.
1. Storage Array Cache Block Size:
   The most critical setting on the STK controller is the cache block size for the storage array. There are two possible choices, 4 KB and 16 KB. The 16 KB cache block size was significantly better. For some dd block sizes tested, performance was almost 3 times faster for single reads. NOTE: Reconfiguration of the STK will reset the block size to the default 4 KB.

 

 


 

2. Storage Array read ahead cache multiplier:
   Initial read tests of single and multiple reads showed that changing the STK read ahead cache multiplier had a negligible effect. Cache multipliers tested were 8, 32, 1024, and 8192. Little difference in performance was noted and further testing was limited to testing with a read ahead cache multiplier of 8.

 

 


 

3. dd Block Size and Raid Group Size:
   The dd block size used for reading and writing data was important. As the graph shows, a block size of 64 KB improves performance by 45% over 32. Improvement continued to increased to a block size of 256. These rates are consistent with the 256 KB buffer size of the Emulex adapters.. The graph also compares the differences between the size of the raid groups (6 and 10 disks per raid group). Minimal difference (except for the 512 KB) was noticed.

 

 



The next graph shows that similar performance was noted on the different dd block sizes for both reading and writing data. Also, write performance is not significantly less than reading data.


 

4. Multiple Fibre Channel Controller, adapters, and PCI buses:
   As the next series of graphs show, multiple reads and writes to a Raid Group filled the available bandwidth. For example, reads and writes across a single controller peaked at around 100 MB/sec. This limitation was probably governed by the limitation of a fibre channel connection. The slightly better than 100 MB/sec performance was probably due to the fact that our reads and writes were staggered.

 

 



Further testing shows that we can improve our performance if we concurrently read data from two controllers connected to one PCI bus. Now our peak performance is around 130 MB/sec which is approximately the speed of our PCI bus. Connecting the two controllers to two different PCI buses, peaked our performance at 187 MB/sec.