iscsi ≠Speed when it comes to DataBases

Recently a quite large client contacted the HostMedic agency regarding some issues they are having within their network at their datacenter.

They are using VMWare – and have what appears to be an excellent setup – until you step under the hood and tinker for a little while.    This client has done everything “right” – purchasing a SAN from Dell (the axiom line) as well as using Gigabit networking – Their SAN is connected to the network sharing its drives using the iSCSI protocol.

What this client and I discovered the hard way is that it does not matter how fast the disks are – or how many disks you have if your pipe to them is slower than the pipe sticking out of a molasses tree.

I recreated the network locally as best as I could – and then turned on Bonnie. Bonnie is an excellent disk performance utility.   After compiling the latest version, 1.96, (which allows you to measure lag time and even includes a multi-threaded mode) it quickly became apparent that their read times were quite high – interestingly enough higher than their write times (go figure).

The upper limit of “Gigabit Ethernet” is (theoretically) about 125MByte/sec.  Sadly – there is no real way for this client to get more due to their expensive iSCSI device not allowing more than one connection to parallel the data – so of course 125MByte/sec is all you get…   My testing in the real world gave roughly 40mbps.    OUCH -  So copying roughly 250GB of databases would take the better part of 5+ hours to complete.

I ran to google and asked the question: Can you get high throughput with iSCSI with GbE on ESX?

The answer is YES.  But there are some complications, and some configuration steps that are not immediately apparent.

You need to understanding some iSCSI fundamentals, some Link Aggregation fundamentals, and know some ESX internals – none of which are immediately obvious to most folks (including myself).

A few take homes here:

• Ethernet link aggregation isn’t worth crap in iSCSI environments
• iSCSI HBA’s don’t buy you much other than boot-from-SAN in ESX,
• The most common configuration (ESX software iSCSI) is limited to about 100 MB/s per iSCSI target over one-gigabit Ethernet.
• Adding multiple iSCSI targets adds performance across the board, but configurations will vary according to your array.
• Maximum per-target performance comes from guest-side software iSCSI, which can make use of multiple Ethernet links to push each array as fast as it can go.
• accessing iSCSI directly from guest VM’s gives better performance than accessing it via the hypervisor layer.
  
• If you are serious about iSCSI in your production environment, it’s valuable to do a bit of investigation learning, and it’s important to do a little engineering during design.    Of course iSCSI is easy to connect and begin using, but like many technologies which excel in terms of their simplicity the default options and parameters may not be robust enough to provide an iSCSI infrastructure which can support your organization.
• Think about Flow-Control (should be set to receive on switches and transmit on iSCSI targets)
• Enable spanning tree protocol with either RSTP or portfast enabled
• Filter / restrict bridge protocol data units on storage network ports
• Configure jumbo frames (always end-to-end – otherwise you will get fragmented crap)
• Use Cat6 cables rather than Cat5/5e – the better the cable the better the connection. PERIOD
• Consider cross-stack Etherchannel trunking for your configuration.
• Investigate your switch – every switch has its good and bad – such as the amount of port buffers, ram, cpu, etc …

(Strongly) Recommended Additional Reading

1. Scott Lowe has done an excellent job talking on ESX networking.   Start with his recap here: http://blog.scottlowe.org/2008/12/19/vmware-esx-networking-articles/
2. Read the vendor’s documentation!
   • START HERE - VMware: iSCSI SAN Configuration Guide
   • EMC Celerra: VMware ESX Server Using EMC Celerra Storage Systems – Solutions Guide
   • EMC CLARiiON: VMware ESX Server Using EMC CLARiiON Storage Systems – Solutions Guide
   • EMC DMX: VMware ESX Server Using EMC Symmetrix Storage Systems – Solutions Guide
   • NetApp: NetApp & VMware Virtual Infrastructure 3 : Storage Best Practices (Vaughn is proud to say this is the most popular NetApp TR)
   • HP/LeftHand: LeftHand Networks VI3 field guide for SAN/iQ 8 SANs
   • Dell/EqualLogic:
     • Network Performance Guidelines
     • VMware Virtual Infrastructure 3.x Considerations, Configuration and Operation Using an Equallogic PS Series SAN

My advice is – stay away from iSCSI for databases – and go with direct storage – iSCSI is not ready for performance-sensitive applications out of the box.   I would suggest that anyone considering iSCSI with VMware should feel confident that their deployments can provide high performance and high availability as long as theyunderstand the “one link max per iSCSI target” ESX iSCSI initiator behavior.