So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.
In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode. This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.
I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO. Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO. VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine. I then tested an fresh install of a system, since this is highly IO intensive.
Needless to say, I was very surprised to see the performance improvement. Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting). I also saw the memory on the OpenFiler system being used for caching, which was another win.
I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode. There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time. Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.
Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem... the cache was using all the RAM on the box.
My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each. Needless to say, I'm shopping for RAM right now =D.
(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).
So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure. In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K. More to come once I have more RAM =D