So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.
In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode. This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.
I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO. Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO. VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine. I then tested an fresh install of a system, since this is highly IO intensive.
Needless to say, I was very surprised to see the performance improvement. Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting). I also saw the memory on the OpenFiler system being used for caching, which was another win.
I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode. There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time. Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.
Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem... the cache was using all the RAM on the box.
My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each. Needless to say, I'm shopping for RAM right now =D.
(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).
So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure. In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K. More to come once I have more RAM =D
This is somewhat of an old post, anyways... Your box will always use 100% (well, all that is left over) of your RAM for cache. That is just the way Linux cache works. It doesn't mean that you low on RAM, as soon as another process requires RAM, cache wil release it. So the more RAM the better, but you will almost always see it as used, even though it's actually available to the system.
ReplyDeleteRight. The issue wasn't that all the RAM was used as much as the fact that I only had 2GB. I've since increased the memory in both boxes, and the performance has increased as well.
DeleteAs far as I can tell, this isn't the Linux system itself as much as Openfiler reporting the cache it has. They could be the same, I'm not sure.
Great that I found you! For you as for an entrepreneur and for your followers who are probably business owners as well such article about cloud technology as virtual data room comparison may be interesting.
ReplyDeleteHey Rick, thanks for the link!
DeleteThere are several different competing ideas related to "cloud" technology; one of my main reasons for doing this at the time was not just cost but also to stay current on technology stacks.
Reading back over this post, I would not use Openfiler. I'm in the middle of a move right now, so everything is packed up. I'll work on putting my thoughts down on virtual data rooms and cloud computing in the future.