iSCSI Performance, round 2

So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.

In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode.  This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.

I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO.  Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO.  VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine.  I then tested an fresh install of a system, since this is highly IO intensive.

Needless to say, I was very surprised to see the performance improvement.  Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting).  I also saw the memory on the OpenFiler system being used for caching, which was another win.

I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode.  There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time.  Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.

Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem...  the cache was using all the RAM on the box.

My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each.  Needless to say, I'm shopping for RAM right now =D.

(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).

So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure.  In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K.  More to come once I have more RAM =D

Setting up OpenFiler 2.3 for VMWare ESXi storage

so I'm looking to set up openfiler as a storage backend for VMWare ESXi, just wanted to include some notes here, since the examples I've found in the forums isn't what I wanted to set up (disk sizes were fairly small).

So, main thing to remember is that OpenFiler doesn't handle the automated partitioning of the hard drive you're installing it on.  Best info I've found on this (without purchasing the Manual) is on Greg Porter's Wiki.  He has much excellent info on Openfiler, and I would highly recommend reading through his info.  This list here is just intended to be a quick checklist of things that need to be done to get this set up.

really rough run-down:

1. Get disk set up in RAID 5 on Dell 2850 (or hardware of your choice), preferably in hardware RAID rather than software RAID.
2. Boot from Openfiler 2.3 install disk (x86_64 in this case.... what, you haven't downloaded it already? (=  )
3. Using Graphical install, manually partition the disk as follows:
       /boot : 100MB ext3, Fixed size, Force Primary
       / (root): 2048MB ext3, Fixed size, Force Primary
      swap : 2048MB swap, Fixed size, Force Primary
4. Finish install (time, root password, etc.)
5. update, update, update (I've had some issues with this from the webgui; however, per the last entry of this forum, using "conary updateall" from the command line worked just fine.)
6. see Greg Porter's Wiki about the iSCSI reboot issue to prevent your stores from vanishing from the network upon reboot.  Last thing you want is for your system to boot and your services to fail.

At this point, you should be able to log in to OpenFiler, configure your shares, and get rolling.