TOPIC:

Hi!

I set up ha-iscsi on a xenserver 6.2 and after some benchmarks, i found that iscsi block devices are way (10x at least) slower than their backing devices according to bonnie++ benchmarks (attached). (even without drbd 'under' the iscsi)

I managed to improve it a litle bit by adjusting /sys/block/sdX/queue/max_sectors_kb to 128 instead of 1024 and setting scheduler to noop.

The XenServer installation is almost clean and new, only with some Citrix patches/updates installed. Do you know about any way to debug it?

Thanks,
Lefty

Can you provide detailed steps on your test procedure? We can try the same on our development servers. We have not used bonnie++. In simple file transfer write tests with dd we see a negligible difference in write performance when comparing iscsi disk on the master with dom0 writing to local scsi disk.

Also, can you elaborate on your HW specs and whether you are using SW or HW RAID? What is the link speed for your iscsi interface?

I did the tests again with dd:
dd if=/dev/zero of=testf bs=10000 count=100000

results:
- lvm on sw 3drive-raid0 on regular server 7200rpm hdd (iscsi backing device):
1000000000 bytes (1,0 GB) copied, 10,3351 seconds, 96,8 MB/s

- iscsi
1000000000 bytes (1,0 GB) copied, 100,8 seconds, 9,9 MB/s

- iscsi with max_sectors_kb 128 instead of 1024
1000000000 bytes (1,0 GB) copied, 195,857 seconds, 5,1 MB/s

iSCSI server is running on the same server and drbd is disconnected from the other. (and the iSCSI is that slow even when not running on top of the drbd but on top of lvm volume.)

I also encountered these errors yesterday:

Aug 7 08:47:50 xenserver-gama kernel: [57702.380555] end_request: I/O error, dev sdf, sector 263377024
Aug 7 08:47:50 xenserver-gama kernel: [57702.380562] Buffer I/O error on device dm-4, logical block 9535488
Aug 7 08:47:50 xenserver-gama kernel: [57702.380567] lost page write due to I/O error on dm-4
Aug 7 08:47:50 xenserver-gama kernel: [57702.397253] sd 16:0:0:10: [sdf] Unhandled error code
Aug 7 08:47:50 xenserver-gama kernel: [57702.397258] sd 16:0:0:10: [sdf] Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Aug 7 08:47:50 xenserver-gama kernel: [57702.397262] sd 16:0:0:10: [sdf] CDB: Read(10): 28 00 0f 9b 77 60 00 00 08 00

If you didn't manage to replicate it, I can give you access to that machine. As I said, it's almost clear installation of XenServer 6.2.

Performance on our servers is significantly better that your results.
Below results are from the same VM. First running on the master and then the slave. In both cases the backing store is iscsi running on the local hosts as described in our how-to iscsi-ha.

The HW in this test uses 10K scsi disks with HW RAID 1+0 and bonded 2Gbps ethernet for replication and slave access to iscsi storage on the master.

Is it possible you are using SW raid? If so, that coupled with slower disks and extra RAID processing in dom0 may be degrading your performance.

Here are our test results:

From a VM running on the master backed by iscsi-ha
[root@test1 tmp]# dd if=/dev/zero of=testf bs=10000 count=10000
10000+0 records in
10000+0 records out
100000000 bytes (100 MB) copied, 0.81666 s, 122 MB/s

From same VM as above when running on slave backed by iscsi-ha
[root@test1 tmp]# dd if=/dev/zero of=testf bs=10000 count=10000
10000+0 records in
10000+0 records out
100000000 bytes (100 MB) copied, 1.34623 s, 74.3 MB/s

We use SW raid, but the performance hit should be around 1%. Write to that SW raid partition is 100MB/s, write to iscsi volume (from dom0) is 5-10MB/s even without iscsi-ha enabled. Maybe we should dedicate more CPU resources to dom0 or there is a bug in CentOS tgtd...

I will try to run the tests on another XenServers with hw and sw raids and compare.