Preliminary Analysis of iSCSI
January, 2002
As part of the Probe testbed environment, we tested a Cisco SN 5420 iSCSI router.
The SN 5420 Storage Router provides servers with IP access to storage
through SCSI routing using the iSCSI protocol.
The iSCSI protocol is a protocol for
encapsulating SCSI requests and responses over IP.
With SCSI routing, servers use an IP network to access storage
as if the servers were directly attached to the storage devices.
The results of our testing of the iSCSI router in this document are preliminary.
They are preliminary, because our testing was not as rigorous as we would like.
We only tested the box for a week, and much of that time was spent on solving
various problems we had with the systems we were connecting the SN 5420 to.
Further problems we encountered are described below.
System Configurations
-
A Fibre Disk Array was connected to a host computer over the internet using
a Cisco SN 5420.
-
We tested with two different host platforms.
We used a Sun E250 with 2 400MHz UltraSPARC-II processors and
.5 GB memory and a Linux Red Hat 7.2 based dual 1.4 GHz Athlon machine with 1GB
memory.
We would like to have used an AIX machine
but the AIX drivers were not available at the time of our
testing.
-
For disk storage, we used a Sun T3 fibre channel disk array with stripping across
8 disks and a ninth for parity.
-
We tested the SN 5420 over a private Gigabit ethernet
and over a WAN.
In the later case,
the SN 5420 and Sun T3 disk array was located in Berkeley and the host computer
was located in Oakland.
The network connection between the two sites is a busy OC12 connection.
Software Configurations
-
All tests used a 500MB file.
We rotated among five copies of this file to avoid using a cached copy.
-
We used the Unix utility dd for reading and writing
and ran dd with various block sizes.
-
In the tests with the Sun host, we also tested various HiWater sizes
(1MB (default), 8MB, and 32MB).
HiWater is the TCP transmit and receive window size that is set in
iSCSI Sun /etc/NuScsiTcp.conf file.
Analysis/Results
-
Initially, we tried connecting a 6.2 Red Hat Linux(linux 2.2 based kernel) system to the i
SCSI box.
While the documentation implied this would work,
we were unable to successfully connect until we upgraded the Linux box to Redhat 7.2(a lin
ux 2.4 based kernel).
-
The Sun configuration consistently outperformed the Linux configuration for reading
data.
The opposite was true for writing, where the Linux configuration consistently outperformed
the Sun writes.
Unlike with the "Reads", the dd block size had some impact on the Sun writes.
This was less true with the Linux writes, except for bs=64k. The figures were so high,
that we must be caching the writes.
-
The performance figures for the WAN configuration was not very good.
The factors that may have contributed to this are:
- There was no time to do any network tuning.
- We were having problems with the T3 disk used for these timings
-
We got approximately 40% performance degradation using the iSCSI
switch over connecting the Sun T3 over fibre directly to the host.
-
The configuration of the Sun is critical to performance. Our initial
tests were done on a Sun where we got both poor and inconsistent
results. For example,
-
On our read tests (bs=256) we saw a degradation of performance when we
read a file successive times. The degradation was less with the higher
HiWater values.
-
HiWater=1MB, went from 8.75MB/sec to 3MB/sec over 5 runs.
-
HiWater=8MB, went from 15.88MB/sec to 10.94MB/sec over 4 runs.
-
HiWater=32MB, went from 29MB/sec to 10.48MB/sec over 7 runs.
-
Better performance was seen with higher values of the TCP transmit and
receive window size. The initial value (i.e., highest rate) and the
average across the various runs are:
-
HiWater=1MB - initial run is 8.75MB/sec, average of five runs is
4MB/sec
-
HiWater=8MB - initial run is 15.88MB/sec, average of four runs
is 2.65MB/se
-
HiWater=32MB - initial run is 20.49MB/sec, average of seven runs
is 10.37MB/se
-
A few test were run varying the HiWater mark. It seemed to have little
effect over the private ethernet.
Performance Timings: Fibre vs. iSCSI
|
READ PERFORMANCE: dd if=/t3/500mb if=/dev/null bs=<value>
|
| | bs=64k | bs=128k | bs=256k | bs=512k |
bs=1024k | bs=2048k |
|---|
| | MB/sec | MB/sec | MB/sec | MB/sec |
MB/sec | MB/sec |
|---|
| Sun/with iSCSI | 26.59 | 26.79 | 26.52 |
27.19 | 26.96 | 26.97 |
| /with Fibre | 59.11 | 59.65 |
58.45 |
59.78 | 56.93 | 55.72 |
| Linux/with iSCSI | 17.26 | 17.69 | 17.78 |
17.97 | 17.97 | 17.97 |
| /with Fibre | 24.15 | 24.47 | 2
4.36 | 24.87 |
25.09 | 25.29 |
| Remote Linux/with ISCSI | 2.64 | 2.88 | 3.03 |
3.19 | 3.14 | 3.20 |
|
WRITE PERFORMANCE: dd if=/dev/zero of=/t3/<file> bs=<value>
|
| Sun/with iSCSI | 17.64 | 17.78 | 18.81 |
20.10 | 21.92 | 22.81 |
| /with Fibre | 34.79 |
34.56 | 35.71 | 44.70 | 49.79 | 48.59 |
| Linux/with iSCSI | 62.05 | 33.39 | 31.53 |
30.01 | 30.08 | 33.33 |
| /with Fibre | 94.47 | 57.3 |
57.93 | 57.74 | 58.06 | 57.68 |
| Remote Linux/with ISCSI | 6.70 | 5.73 | 5.68 |
5.53 | 5.55 | 5.57 |
|
Two Simultaneous Reads
|
| | bs=256k | bs=256k | Aggregate=256k |
|---|
| | MB/sec | MB/sec | MB/sec |
|---|
| Sun/with iSCSI | 16.74 | 16.54 | 33.28 |
| /with Fibre | 32.99 | 33.16 | 6
6.15 |
| Linux/with iSCSI | 11.31 | 11.29 | 22.60 |
| /with Fibre | 16.3 | 16.32 | 32
.62 |
| Remote Linux/with ISCSI | 2.47 | 2.17 | 4.64 |
