Configuring MantisBT

Once you're done Installing MantisBT, the web interface is available.  However, in order to create/reset user accounts, you need to have email configured so emails can be sent out.  I did the following:

vi /opt/mantisbt-1.2.11/config_defaults_inc.php
search for "g_smtp_host"
change the following settings (I used one of my existing email accounts):

    $g_phpMailer_method = PHPMAILER_METHOD_SMTP;
    $g_smtp_host = 'smtp.example.com';
    $g_smtp_username = 'account@mailserver.com';
    $g_smtp_password = 'THIS_IS_CLEAR_TEXT';
    $g_smtp_connection_mode = 'ssl';
    $g_smtp_port = 465;

At this point, I was able to send email to my account and get logged in with an account other than the default administrator account.

While MantisBT did well at the install, the configuration interface leaves a few things to be desired.  First, there are settings beyond getting the system to use the web server or database that cannot be configured in the web gui.  Secondly, while all of the possible settings are well documented, grep is your friend; there is no clear indication where they are located.

Total score for ease of administrative configuration: 3/10
Major failing: unable to edit full config from the web interface.

Installing MantisBT

from a fresh CentOS 6 install (don't forget yum -y update and a reboot), run the following commands:

1. cd /tmp
2. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz/download
3. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz.digests/download
4. md5sum -c mantisbt-1.2.11.tar.gz.digests
5. tar -xzf mantisbt-1.2.11.tar.gz
6. mv mantisbt-1.2.11/ /opt/mantisbt-1.2.11
7. chown -R apache:apache /opt/mantisbt-1.2.11
8. ls -n /opt/mantisbt-1.2.11/ /var/www/html/mantisbt
9. yum -y install httpd mysql mysql-server php php-mysql
10. vi /etc/php.ini
11. change line 946 to match:
• date.timezone = "America/New_York" (should match cat /etc/sysconfig/clock output)
12. chkconfig httpd on
13. chkconfig mysqld on
14. service httpd start
15. service mysqld start
16. mysqladmin -u root password 'newPassword'
17. mysqladmin -u root -p -h localhost.localdomain password 'newPassword' (enter newPassword from above when prompted)
18. iptables -I INPUT 5 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
19. iptables -I INPUT 6 -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
20. now open your browser, and point it to:
• http://yourServerOrIPHere/mantisbt/admin/install.php
21. add newPassword from above in the password field
22. click the "Install" button.

Total time with Cobbler from PXE boot to creating your first project: ~20 - 25 mins.  This includes the time to install the OS from scratch, update it, reboot, and go through the commands above manually.  For those interested in specs, this was run on my setup.

Once you're done here, you'll want to start Configuring MantisBT.

Total score for ease of install: 8/10

My testing/working environment

Below is the current config that I have set up for all the testing/installing, etc. that I do as part of this blog.

2x Dell 1950 2x  Xeon 5130 @ 2.00GHz w/ 8GB RAM and 4x 73GB 10K 2.5" SAS HDD in RAID 5
runs: VMware ESXi 4.1 build-260247

2x Dell 2850 2x  Xeon DC @ 2.80GHz w/ 8GB RAM and 6x 146GB 10K 3.5" SCSI HDD in RAID 5
runs: OpenFiler 2.3 x86_64


Connecting the two is a Linksys SGE2000 running Jumbo Frames for 1GbE connections, and I'm serving up the storage from the 2850 as iSCSI storage to the 1950.

Finally, I have an APC SMX1500RM2U SmartUPS protecting everything from power bumps (I would get some occasionally that weren't enough to power off the systems, but were enough to make them reboot).  It provides about 5-7 mins of run-time currently.

Everything runs off a single 15A 120V circuit in my basement.  Total cost: ~$2600 - $3000.  This assumes you already have cooling for your room.

Cobbler DHCP config

I'm working on installing Cobbler so that I have a reliable way to automate the install of my linux systems.  Following my first post will get the software installed, so now I was ready to move on to actually getting a PXE boot to install from Cobbler.

Looked up the directions here, and the steps are pretty straight forward.  If you're continuing from my previous post, it would look something like this:

1. insert your dvd with the distro
2. yum -y install dhcp
3. mount /dev/dvd /media
4. cobbler import --path=/media --name=CentOS6 (this takes a while, press [CTRL+Z], then run bg [ENTER] to send it to the background)
5. vi /etc/cobbler/dhcp.template
6. edit per your network, mine looked like this:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

7. service cobblerd restart
8. cobbler sync

All is well, we're now ready to PXE boot clients, and Cobbler will..... what's this?

running: dhcpd -t -q
received on stdout:
received on stderr:
dhcpd -t failed
Exception occured: <class 'cobbler.cexceptions.CX'>
Exception value: 'cobbler trigger failed: cobbler.modules.sync_post_restart_services'
Exception Info:
  File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run
    rc = self._run(self)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 184, in runner
    return self.remote.api.sync(self.options.get("verbose",False),logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 701, in sync
    return sync.run()
   File "/usr/lib/python2.6/site-packages/cobbler/action_sync.py", line 155, in run
    utils.run_triggers(self.api, None, "/var/lib/cobbler/triggers/sync/post/*", logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 918, in run_triggers
    raise CX("cobbler trigger failed: %s" % m.__name__)

!!! TASK FAILED !!!

hmm.. well, we can see that there is an issue with dhcpd, lets try to start the service on it's own:

[root@OPS8-Cobbler ~]# service dhcpd start
Starting dhcpd:                                            [FAILED]
[root@OPS8-Cobbler ~]#

Here's the log from /var/log/messages while trying to start the service above:

Jul  2 17:38:48 OPS8-Cobbler dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: All rights reserved.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Jul  2 17:38:48 OPS8-Cobbler dhcpd: /etc/dhcp/dhcpd.conf line 19: semicolon expected.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:      option domain-name-servers 68.87.85.102 68.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:                                                ^
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Configuration file errors encountered -- exiting
Jul  2 17:38:48 OPS8-Cobbler dhcpd:

Looking at /etc/dhcp/dhcpd.conf, I realized that I'd left a comma out of the template config.  It should look like the following in /etc/cobbler/dhcp.template:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102, 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

Now we restart the service and run a sync....

[root@OPS8-Cobbler ~]# service cobblerd restart

Stopping cobbler daemon:                                   [  OK  ]

Starting cobbler daemon:                                   [  OK  ]

[root@OPS8-Cobbler ~]# cobbler sync

task started: 2012-07-02_174015_sync

task started (id=Sync, time=Mon Jul  2 17:40:15 2012)

...

rendering DHCP files

generating /etc/dhcp/dhcpd.conf

...

running: dhcpd -t -q

received on stdout:

received on stderr:

running: service dhcpd restart

received on stdout: Starting dhcpd: [  OK  ]

...

*** TASK COMPLETE ***

[root@OPS8-Cobbler ~]#

much better.

A simple PXE test confirms that the service is working and providing addresses successfully.  However, I wasn't able to get the PXE menu to load on a network boot...  turns out that I didn't have the proper Firewall settings.  You can test by stopping your firewall (service iptables stop) and trying again; watch out for tcp/udp differences!!  After using the information under Firewall on this page  I was able to get the PXE boot menu to load and test the install.

Cobbler Install on CentOS 6.2

Cobbler - not the kind you put peaches in, this is an automated install tool

Here's the quick and dirty to get it installed and the web interface working:

1. CentOS 6.2 install
• Basic Server install option
• as root, run "setenable 0" to turn selinux to permissive (without this, selinux caused me many headaches with the "cobbler check" command later)
• as root, run "vi /etc/selinux/config" and change the SELINUX=enforcing to SELINUX=permissive.  This keeps it in permissive mode over reboots.
• optional: set up a local user with wheel access, enable wheel sudo access, and set /etc/ssh/sshd_config with "PermitLocalRootLogin without-password"
2. add EPEL repo
1. point browser to: http://fedoraproject.org/wiki/EPEL
2. right-click, copy link
3. on CentOS system (I connect through putty and change to root at this point), run
• rpm -ivh <SHIFT+INSERT> (last two keys will paste the link from step 2)
3. Install Cobbler
1. yum -y install cobbler cobbler-web koan policycoreutils-python
2. service cobblerd start
3. service httpd start
4. cobbler check
1. resolve all reported issues (I had about 10)
4. Configure Cobbler-Web
1. see cobbler-web wiki page, just remember to try http if https fails

I think you might be able to skip step 3.4 and do that after step 4 if you'd like to have the web gui, since it is available there, but I don't know if you can resolve all the issues from there.

Kudos to Mike DeHaan for a really helpful config checker; wish all software came with something like that.

iSCSI Performance, round 2

So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.

In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode.  This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.

I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO.  Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO.  VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine.  I then tested an fresh install of a system, since this is highly IO intensive.

Needless to say, I was very surprised to see the performance improvement.  Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting).  I also saw the memory on the OpenFiler system being used for caching, which was another win.

I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode.  There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time.  Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.

Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem...  the cache was using all the RAM on the box.

My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each.  Needless to say, I'm shopping for RAM right now =D.

(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).

So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure.  In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K.  More to come once I have more RAM =D

Ansible setup

Ansible - def.  1. super-luminal (aka, faster than light)
                        2. system managment automation program on github you wished you were running

Ansible is set up to be very simple, and runs over ssh.  Here are my notes from trying to get it installed and working on Centos 6.2, using the "Running from Checkout" instructions found at http://ansible.github.com/gettingstarted.html, which gets you version 0.5.  The RPM from EPEL provides version 0.3.

here's my super quick instructions, the few issues I ran into mentioned below:

1. start with CEntOS 6.2
2. sudo su - root or su - root
3. install needed packages
1. # rpm -ivh http://mirror.pnl.gov/epel/6/i386/epel-release-6-7.noarch.rpm
2. # yum -y install python PyYAML python-jinja2 python-paramiko
3. # exit
4. add ansible
1. $ git clone git://github.com/ansible/ansible.git
2. $ cd ./ansible 
3. $ source ./hacking/env-setup
5. configure hosts
1. $ echo "127.0.0.1" > ~/ansible_hosts 
2. $ export ANSIBLE_HOSTS=~/ansible_hosts
6. and test:
• $ ansible all -m ping -u dewey.garwood
  127.0.0.1 | success >> {
      "ping": "pong"
  }

you should note the following errors will occur if you aren't paying attention:

• if you go looking for paramiko, yum wont find it; you have to use python-paramiko
• without the -u option in the test command (step 6), ansible tries to use the root user to log in and you end up with:
• $ ansible all -m ping --ask-pass
  SSH password:
  127.0.0.1 | FAILED => FAILED: Authentication failed.

iSCSI perfomance

If you've read any of my other posts, you know I'm running OpenFiler as an iSCSI backend for VMware ESXi 4.1.

There are some issues with running it in this manner, and I hope to write out some more instructions later about setting up to use SCST rather than IETD.  However, this is for anyone out there who might be trying to get better performance out of your iSCSI infrastructure... hopefully this will help you avoid my "doh!" moment.

If you haven't already done so, find a time to bring your environment down long enough to turn on jumbo frames on your switches.  Your VMs and the customers who use them will thank you, not by saying anything, but by not complaining that the performance is really slow.

After having done so, my average write latencies have gone from triple digits to double digits, and my throughput has roughly doubled.  Also, my Openfiler system has gone from load averages that were around 1 to around 4 - 5 (4 is a full load for my system).

So here's a friendly reminder to avoid my face-palm moment X[ and get some decent performance out of your system =D

Minor format tweaks to blog

aka: how to make your background image stay put using CSS

Someone mentioned that it would be nice if the background would stay put on my blog, so it was always there, rather than just at the top.

Since I'm in the process of learning html and css, figured I would see if I could do something about that.  Care to guess which CSS section I'm learning about right now? =-D

Before:
body {
background: #000000 url(<image_url_here>)  repeat-x scroll top center /* Credit for photo here */;
}

After:
body {
background: #000000 url(<image_url_here>)  repeat-x fixed top center /* Credit for photo here */;

}


I hope this makes the main blog a bit easier to read, and not seem like you're Lost in Space™ (weeeoooo!) when you scroll down.

I haven't been able to get the mobile working yet, so if you're looking at this on a too-smart-for-your-own-good phone and you know how to fix it, drop me a comment, please.  Or be patient; I should be there in a few more chapters :)

Also, I want to take this time to highly recommend www.murach.com.  They publish books that are excellent tools for learning technology, and are worth their weight in gold.  You won't find a better book for getting up to speed on a topic quickly, provided that they have a book that covers what you're looking for.

So, just in case anyone from murach.com is reading this, a few topics I'd like to request:
Perl, Python, Apache Administration, and testing automation.

In the meantime, if you're interested in those topics, stay tuned, I'll probably end up with something to "leak".