Cisco MDS 9509 iSCSI Performance Results

1. Overview

The purpose of this report is to document the performance results of the I/O tests done on the Cisco MDS 9509 Multilayer Director Switch. The primary focus of the I/O performance test was to measure the end-to-end iSCSI performance, in terms of I/O bandwidth in MB/second, and the scalability of the MDS 9509 switch. The tests included single-thread and multi-thread I/O tests with different file sizes and I/O sizes. The evaluation period was from January 2004 to March 2004.

2. Test Configuration

We used NERSC PIORAW benchmarks to measure the performance. To measure the end-to-end (host-to-storage controller) performance, PIORAW generates sequential I/O requests (reads and writes) to a small file. All the tests were performed using the raw device interface (e.g., /dev/raw/raw1) to avoid any caching effect on the host system.

The block sizes used in the tests include: 1 KB, 4 KB, 16 KB, 64 KB, 256 KB, 1 MB, 4 MB, and 16 MB.

For multi-stream parallel I/O tests, the aggregate bandwidth was calculated as the total number of bytes read or written divided by the longest elapsed time.

All tests were run on a quite system. No other activities were present on either the clients or the storage when the tests were running.

2.1 Linux Host Configuration

Two types of hosts were used during the tests: P3 nodes and P4 nodes:

• P3 Node Configuration:

• Dual 1 GHz Pentium III CPUs, Intel Server Board STL2 motherboards

• 1 GB 133 MHz ECC memory

• RedHat 7.3 with 2.4.18-10smp kernel

• Intel PRO/1000 XT 133 MHz PCI-X Gigabit Ethernet NIC

• P4 Node Configuration:

• Dual 2.2 GHz Xeon Pentium IV CPUs, SuperMicro mother board

• 2 GB 133 MHz ECC memory

• RedHat 7.3 with 2.4.18-10smp kernel

• Intel PRO/1000 XT 133 MHz PCI-X Gigabit Ethernet NIC

2.2 Storage Configuration

Two storage systems were used in the tests: the Yotta Yotta NetStorager GSX 2400 system and the DataDirect Networks (DDN) S2A 8500 system.

Yotta Yotta NetStorager GSX 2400

• 4 control blades, each has 4 GB memory

• Each control blade has 4 2Gb/s FC ports, two for the front-end host connections and two for the backend disk connections

• 2 disk enclosures, each disk enclosure is split into two set of 8 disks

• 8 4-way RAID-0 LUNs, with each LUN exported to one front-end host port

DataDirect Networks (DDN) S2A 8500

• Two controllers, each has separate 2560 MB memory (only one controller was used in the test)

• Each controller has 4 2Gb/s FC ports for host connectivity

• 20 back-end Fibre-Channel loops with 74 FC disks (64 data disks, 8 parity disks, and 2 spares)

• 8 8+1 RAID-3 LUNs, with each LUN exported to one front-end host port

2.3 Testbed Configuration

3. TEST RESULTS

3.1 SingleiSCSI Target Performance (One iSCSI initiator, One iSCSI Target)

Objective: To measure iSCSI single target end-to-end performance (a single host accessing a single iSCSI target device)

Test Setup:

• Clients: P4 nodes, Linux 2.4.18-10smp

• Switches: Extreme 7i

• Switch Topology: client and iSCSI GigE port are on the same 7i switch

• Storage: Yotta Yotta GSX 2400

• # of FC Ports and Speed: 1 2Gb/s FC port

• LUN Configuration: 1 4-way Yotta Yotta RAID-0 LUN (accessed as /dev/raw/raw1)

• I/O Size: 1 KB - 16 MB

• # of Client: 1

• # of I/O Processes per Client: 1 to 10

• Benchmark: PIORAW

Two Tests:

Test 1 was to measure iSCSI performance with increased number of I/O processes on a single host, using an I/O size of 256 KB, to see if we would saturate a single iSCSI interface with a single host.

Test 2 was to measure iSCSI performance of different I/O sizes.

Test Result 1 (Varying # of I/O processes on the host):

Test 1 results show the iSCSI performance of a single iSCSI target device accessing from a single host using different number of I/O processes, using an I/O size of 256 KB.

Test 1 results indicate that, on a single host, the performance of a single iSCSI target did not increase very much after 8 I/O processes. These results show that, for the I/O size of 256 KB, the best iSCSI performance of a single iSCSI target device on a single host is about 70 MB/s for writes and 45 MB/s for reads. These numbers are lower than what we saw before with similar tests run on the Cisco SN5428 switch for the read/write performance was close to 90 MB/s.

Test 2 Result (Varying I/O Size):

Test 2 Results show the performance of a single iSCSI target device, accessing from a single host using different I/O Sizes. The results show the best write performance was 83 MB/s, and was achieved at the I/O size of 64 KB and the best read performance was 45 MB/s for I/O sizes larger than 256 KB.

Test 2 results indicate that there was a sweet spot for writes at 64 KB and 256 KB (or about 30% performance drop for I/O sizes that are larger than 1MB). The reason for these sweet spots (or performance drop for larger I/O sizes) is not clear. This behavior does not seem to be related to the underlying storage device though. In earlier tests we did with the same storage device, the Yotta Yotta GSX 2400, we constantly saw better performance for larger I/O sizes.

For I/O sizes that are larger than 1 MB, the performance of a single iSCSI target device on a single Linux host is about 55 MB/s for writes and 45 MB/s for reads. Both the read performance and the write performance were much lower than the GigE line speed (which is about 125 MB/s). From talking to Cisco, it seems that we need to have more than one iSCSI target devices to achieve a better performance over a single GigE interface.

3.2 iSCSI Performance (One iSCSI initiator, Multiple iSCSI Target Devices)

Objective: To measure multi-LUN iSCSI performance, with a single host accessing multiple iSCSI target devices

Test Setup:

• Clients: P4 nodes, Linux 2.4.18-10smp

• Switches: Extreme 7i

• Switch Topology: client and iSCSI GigE port are on the same 7i switch

• Storage: DDN S2A 8500

• # of FC Ports and Speed: 4 2Gb/s FC port

• LUN Configuration: 4 DDN RAID-3 LUNs

• I/O Size: 1 KB - 16 MB

• # of Client: 1

• # of I/O Processes per Client: 16

• Benchmark: PIORAW

Test Result:

Test Summary:

The results show that, on a single host, the iSCSI performance did increase when more iSCSI target devices were accessed from the same GigE interface. The best iSCSI performance achieved by a single host with four iSCSI target devices was about 80.29 MB/s for writes (with 64 KB I/O size) and 72.08 MB/s for reads (with 16 KB I/O size).

These results clearly indicate that a host needs more than a few iSCSI target devices to saturate a single GigE interface with the MDS 9509 switch. Even with four LUNs, these numbers are still lower than what we saw on the Cisco SN5428 switch with only a single iSCSI target, and the performance was close to 90 MB/s for both reads and writes.

3.3 Cisco MDS 9505 iSCSI I/O Scalability (Multiple iSCSI initiators and targets)

Objective: To measure the iSCSI performance scalability with increased number of iSCSI initiators (clients)

Test Setup:

• Clients: P4 nodes, Linux 2.4.18-10smp

• Switches: Extreme 7i

• Switch Topology: client and iSCSI GigE port are on the same 7i switch

• Storage: DataDirect Networks S2A 8500

• # of FC Ports and Speed: 4 2Gb/s ports

• LUN Configuration: 4 8+1 RAID-3 LUNs

• # of GigE Ports and Speed: 4 1Gb/s ports

• I/O Size: 1 KB - 16 MB

• # of Client: 1 to 8

• # of I/O Processes per Client: 8

• Benchmark: PIORAW

Test Result:

Result Summary:

The results show the MDS 9509 iSCSI scalability when the number of iSCSI initiators (client) increased. The aggregate iSCSI performance did increase when more iSCSI initiators were added. However, the aggeregate write performance stopped to increase after four initiators and remained at around 250 MB/s. The best aggregate performance was achieved with 7 iSCSI initiators. The best aggregate performance was about 467 MB/s which was very close to the aggregate bandwidth of 4 GigE interfaces. However, there were some performance drops in the cases of 4, 6 and 8 initiators. The reason for these performance drops is not clear.

3.4 iSCSI Test with Multiple GigE Switches (Bridging Overhead)

Objective: To measure the switch bridging overhead when multiple switches are used

Test Setup:

• Clients: P4 nodes, Linux 2.4.18-10smp

• Switches: 2 Extreme 7i and a Dell PowerConnect 6024

• Inter-switch links: 4-port trunk between two 7i switch and between a 7i and the Dell switch

• Switch Topology: 4 clients on one 7i switch, 4 iSCSI GigE ports on the Dell Switch, and these two switches were connected by a second 7i switch

• Storage: DDN S2A 8500

• # of FC Ports and Speed: 4 2Gb/s FC ports

• LUN Configuration: 4 8+1 RAID-3 LUNs (/dev/raw/raw1 - /dev/raw/raw4)

• Zone Configuration: no zones – each host sees 4 LUNs

• I/O Size: 1 KB to 16 MB

• # of Client: 4

• # of I/O Processes per Client: 8 (2 processes per raw device)

• Benchmark: PIORAW

Test Result:

Result Summary:

When iSCSI is deployed in a large cluster, it is expected to have more than just one switch. The iSCSI traffic may need to hop through several switches. The purpose of the test was to better understand how bridging may affect iSCSI performance for sequential I/O’s. Comparing these results against the results in Section 3.3, the write performance was about the same but the read performance was lower for I/O sizes that are larger than 256 KB. More investigation is needed to understand whether the performance drop was caused by any of the switches or by the overhead of hopping through several switches.

3.5 MDS 9509 FC Performance Baseline

Objective: To measure MDS 9509 FC performance as a baseline for the iSCSI performance

Test Setup:

• Client: Linux 2.4.18-10smp, Dual 2.2 GHz Xeon P4

• FC HBA: QLA2340

• # of FC Ports and Speed: 3 2Gb/s FC port

• LUN Configuration: 4 DDN 8+1 RAID-3 LUNs, all LUNs exported to all hosts

• I/O Size: 1 KB to 16 MB

• File Size: 32 MB

• # of Client: 1 - 3

• # of I/O Processes per Client: 16

• Benchmark: PIORAW

Result Summary:

The above results show the MDS 9509 end-to-end FC performance, with each host accessing (reading and writing) 4 DDN RAID-3 LUNs simultaneously. These results were the sequential I/O performance, for reading and writing of a 32 MB file, using different I/O sizes. Since the memory cache on the controller is 2 GB, the content of the 32 MB file can be cached entirely in the controller cache. With the file content cached, the performance would show the best possible transfer rate of the FC connections between the hosts and the FC ports on the storage controller.

The results indicate that both read and write performance increased when the I/O size increased. Once the I/O size reached 1 MB, the performance stayed at the same level. The results show a near linear scalability for writes with I/O sizes larger than 256 MB. However, for reads, only the test with the I/O size of 64 KB showed a near linear scalability. For three clients, the read performance reached to 593 MB/s at 64 KB and then dropped to 507 MB/s for I/O sizes larger than 256 KB.

This was the first time we used the MDS 9509 switch and the DDN S2A8500 storage system, we had no prior experience on their performance. In order to understand whether the performance drop a MDS 9509 switch issue or a problem with the DDN S2A8500 storage, more tests need to be run using a different switch or a different storage system.

These numbers provide the baseline FC performance for the iSCSI performance test.