If you've read any of my other posts, you know I'm running OpenFiler as an iSCSI backend for VMware ESXi 4.1.
There are some issues with running it in this manner, and I hope to write out some more instructions later about setting up to use SCST rather than IETD. However, this is for anyone out there who might be trying to get better performance out of your iSCSI infrastructure... hopefully this will help you avoid my "doh!" moment.
If you haven't already done so, find a time to bring your environment down long enough to turn on jumbo frames on your switches. Your VMs and the customers who use them will thank you, not by saying anything, but by not complaining that the performance is really slow.
After having done so, my average write latencies have gone from triple digits to double digits, and my throughput has roughly doubled. Also, my Openfiler system has gone from load averages that were around 1 to around 4 - 5 (4 is a full load for my system).
So here's a friendly reminder to avoid my face-palm moment X[ and get some decent performance out of your system =D