Cobbler Install on CentOS 6.2

Cobbler - not the kind you put peaches in, this is an automated install tool

Here's the quick and dirty to get it installed and the web interface working:

1. CentOS 6.2 install
• Basic Server install option
• as root, run "setenable 0" to turn selinux to permissive (without this, selinux caused me many headaches with the "cobbler check" command later)
• as root, run "vi /etc/selinux/config" and change the SELINUX=enforcing to SELINUX=permissive.  This keeps it in permissive mode over reboots.
• optional: set up a local user with wheel access, enable wheel sudo access, and set /etc/ssh/sshd_config with "PermitLocalRootLogin without-password"
2. add EPEL repo
1. point browser to: http://fedoraproject.org/wiki/EPEL
2. right-click, copy link
3. on CentOS system (I connect through putty and change to root at this point), run
• rpm -ivh <SHIFT+INSERT> (last two keys will paste the link from step 2)
3. Install Cobbler
1. yum -y install cobbler cobbler-web koan policycoreutils-python
2. service cobblerd start
3. service httpd start
4. cobbler check
1. resolve all reported issues (I had about 10)
4. Configure Cobbler-Web
1. see cobbler-web wiki page, just remember to try http if https fails

I think you might be able to skip step 3.4 and do that after step 4 if you'd like to have the web gui, since it is available there, but I don't know if you can resolve all the issues from there.

Kudos to Mike DeHaan for a really helpful config checker; wish all software came with something like that.

iSCSI Performance, round 2

So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.

In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode.  This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.

I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO.  Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO.  VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine.  I then tested an fresh install of a system, since this is highly IO intensive.

Needless to say, I was very surprised to see the performance improvement.  Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting).  I also saw the memory on the OpenFiler system being used for caching, which was another win.

I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode.  There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time.  Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.

Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem...  the cache was using all the RAM on the box.

My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each.  Needless to say, I'm shopping for RAM right now =D.

(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).

So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure.  In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K.  More to come once I have more RAM =D

Ansible setup

Ansible - def.  1. super-luminal (aka, faster than light)
                        2. system managment automation program on github you wished you were running

Ansible is set up to be very simple, and runs over ssh.  Here are my notes from trying to get it installed and working on Centos 6.2, using the "Running from Checkout" instructions found at http://ansible.github.com/gettingstarted.html, which gets you version 0.5.  The RPM from EPEL provides version 0.3.

here's my super quick instructions, the few issues I ran into mentioned below:

1. start with CEntOS 6.2
2. sudo su - root or su - root
3. install needed packages
1. # rpm -ivh http://mirror.pnl.gov/epel/6/i386/epel-release-6-7.noarch.rpm
2. # yum -y install python PyYAML python-jinja2 python-paramiko
3. # exit
4. add ansible
1. $ git clone git://github.com/ansible/ansible.git
2. $ cd ./ansible 
3. $ source ./hacking/env-setup
5. configure hosts
1. $ echo "127.0.0.1" > ~/ansible_hosts 
2. $ export ANSIBLE_HOSTS=~/ansible_hosts
6. and test:
• $ ansible all -m ping -u dewey.garwood
  127.0.0.1 | success >> {
      "ping": "pong"
  }

you should note the following errors will occur if you aren't paying attention:

• if you go looking for paramiko, yum wont find it; you have to use python-paramiko
• without the -u option in the test command (step 6), ansible tries to use the root user to log in and you end up with:
• $ ansible all -m ping --ask-pass
  SSH password:
  127.0.0.1 | FAILED => FAILED: Authentication failed.

iSCSI perfomance

If you've read any of my other posts, you know I'm running OpenFiler as an iSCSI backend for VMware ESXi 4.1.

There are some issues with running it in this manner, and I hope to write out some more instructions later about setting up to use SCST rather than IETD.  However, this is for anyone out there who might be trying to get better performance out of your iSCSI infrastructure... hopefully this will help you avoid my "doh!" moment.

If you haven't already done so, find a time to bring your environment down long enough to turn on jumbo frames on your switches.  Your VMs and the customers who use them will thank you, not by saying anything, but by not complaining that the performance is really slow.

After having done so, my average write latencies have gone from triple digits to double digits, and my throughput has roughly doubled.  Also, my Openfiler system has gone from load averages that were around 1 to around 4 - 5 (4 is a full load for my system).

So here's a friendly reminder to avoid my face-palm moment X[ and get some decent performance out of your system =D

Minor format tweaks to blog

aka: how to make your background image stay put using CSS

Someone mentioned that it would be nice if the background would stay put on my blog, so it was always there, rather than just at the top.

Since I'm in the process of learning html and css, figured I would see if I could do something about that.  Care to guess which CSS section I'm learning about right now? =-D

Before:
body {
background: #000000 url(<image_url_here>)  repeat-x scroll top center /* Credit for photo here */;
}

After:
body {
background: #000000 url(<image_url_here>)  repeat-x fixed top center /* Credit for photo here */;

}


I hope this makes the main blog a bit easier to read, and not seem like you're Lost in Space™ (weeeoooo!) when you scroll down.

I haven't been able to get the mobile working yet, so if you're looking at this on a too-smart-for-your-own-good phone and you know how to fix it, drop me a comment, please.  Or be patient; I should be there in a few more chapters :)

Also, I want to take this time to highly recommend www.murach.com.  They publish books that are excellent tools for learning technology, and are worth their weight in gold.  You won't find a better book for getting up to speed on a topic quickly, provided that they have a book that covers what you're looking for.

So, just in case anyone from murach.com is reading this, a few topics I'd like to request:
Perl, Python, Apache Administration, and testing automation.

In the meantime, if you're interested in those topics, stay tuned, I'll probably end up with something to "leak".