BitBucket: git familiar with Atlassian's repo service

I've recently started using a new hosted git provider: Atlassian's BitBucket.  While most everyone out there is probably using github, there were two things I noticed about Bitbucket that I liked:

1. They have made https work for checking in with git version 1.7.1 (default yum install git from CentOS 6.x)
2. They allow the option to have private repositories

While I am a huge supporter of open source software, until now, I wasn't aware of any public cloud options for having a private repo without setting up a billed account (other than installing and managing your own server).  Simply put, I'd rather not add yet another copy of whatever tutorial I'm working through to the web.  (I will however, write about it :) )  Not everything should be public.

I'm also interesting in hearing what you have to say about this.  Will you be migrating everything as soon as you can?  Will you be sticking with GitHub?  Some mix of the two?  What arguments could or should be considered for paying for this kind of service?  I do have to say that I'm glad there is another option, since competition leads to better products and services for those using them.

Configuring MantisBT

Once you're done Installing MantisBT, the web interface is available.  However, in order to create/reset user accounts, you need to have email configured so emails can be sent out.  I did the following:

vi /opt/mantisbt-1.2.11/config_defaults_inc.php
search for "g_smtp_host"
change the following settings (I used one of my existing email accounts):

    $g_phpMailer_method = PHPMAILER_METHOD_SMTP;
    $g_smtp_host = 'smtp.example.com';
    $g_smtp_username = 'account@mailserver.com';
    $g_smtp_password = 'THIS_IS_CLEAR_TEXT';
    $g_smtp_connection_mode = 'ssl';
    $g_smtp_port = 465;

At this point, I was able to send email to my account and get logged in with an account other than the default administrator account.

While MantisBT did well at the install, the configuration interface leaves a few things to be desired.  First, there are settings beyond getting the system to use the web server or database that cannot be configured in the web gui.  Secondly, while all of the possible settings are well documented, grep is your friend; there is no clear indication where they are located.

Total score for ease of administrative configuration: 3/10
Major failing: unable to edit full config from the web interface.

Installing MantisBT

from a fresh CentOS 6 install (don't forget yum -y update and a reboot), run the following commands:

1. cd /tmp
2. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz/download
3. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz.digests/download
4. md5sum -c mantisbt-1.2.11.tar.gz.digests
5. tar -xzf mantisbt-1.2.11.tar.gz
6. mv mantisbt-1.2.11/ /opt/mantisbt-1.2.11
7. chown -R apache:apache /opt/mantisbt-1.2.11
8. ls -n /opt/mantisbt-1.2.11/ /var/www/html/mantisbt
9. yum -y install httpd mysql mysql-server php php-mysql
10. vi /etc/php.ini
11. change line 946 to match:
• date.timezone = "America/New_York" (should match cat /etc/sysconfig/clock output)
12. chkconfig httpd on
13. chkconfig mysqld on
14. service httpd start
15. service mysqld start
16. mysqladmin -u root password 'newPassword'
17. mysqladmin -u root -p -h localhost.localdomain password 'newPassword' (enter newPassword from above when prompted)
18. iptables -I INPUT 5 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
19. iptables -I INPUT 6 -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
20. now open your browser, and point it to:
• http://yourServerOrIPHere/mantisbt/admin/install.php
21. add newPassword from above in the password field
22. click the "Install" button.

Total time with Cobbler from PXE boot to creating your first project: ~20 - 25 mins.  This includes the time to install the OS from scratch, update it, reboot, and go through the commands above manually.  For those interested in specs, this was run on my setup.

Once you're done here, you'll want to start Configuring MantisBT.

Total score for ease of install: 8/10

My testing/working environment

Below is the current config that I have set up for all the testing/installing, etc. that I do as part of this blog.

2x Dell 1950 2x  Xeon 5130 @ 2.00GHz w/ 8GB RAM and 4x 73GB 10K 2.5" SAS HDD in RAID 5
runs: VMware ESXi 4.1 build-260247

2x Dell 2850 2x  Xeon DC @ 2.80GHz w/ 8GB RAM and 6x 146GB 10K 3.5" SCSI HDD in RAID 5
runs: OpenFiler 2.3 x86_64


Connecting the two is a Linksys SGE2000 running Jumbo Frames for 1GbE connections, and I'm serving up the storage from the 2850 as iSCSI storage to the 1950.

Finally, I have an APC SMX1500RM2U SmartUPS protecting everything from power bumps (I would get some occasionally that weren't enough to power off the systems, but were enough to make them reboot).  It provides about 5-7 mins of run-time currently.

Everything runs off a single 15A 120V circuit in my basement.  Total cost: ~$2600 - $3000.  This assumes you already have cooling for your room.

Cobbler DHCP config

I'm working on installing Cobbler so that I have a reliable way to automate the install of my linux systems.  Following my first post will get the software installed, so now I was ready to move on to actually getting a PXE boot to install from Cobbler.

Looked up the directions here, and the steps are pretty straight forward.  If you're continuing from my previous post, it would look something like this:

1. insert your dvd with the distro
2. yum -y install dhcp
3. mount /dev/dvd /media
4. cobbler import --path=/media --name=CentOS6 (this takes a while, press [CTRL+Z], then run bg [ENTER] to send it to the background)
5. vi /etc/cobbler/dhcp.template
6. edit per your network, mine looked like this:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

7. service cobblerd restart
8. cobbler sync

All is well, we're now ready to PXE boot clients, and Cobbler will..... what's this?

running: dhcpd -t -q
received on stdout:
received on stderr:
dhcpd -t failed
Exception occured: <class 'cobbler.cexceptions.CX'>
Exception value: 'cobbler trigger failed: cobbler.modules.sync_post_restart_services'
Exception Info:
  File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run
    rc = self._run(self)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 184, in runner
    return self.remote.api.sync(self.options.get("verbose",False),logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 701, in sync
    return sync.run()
   File "/usr/lib/python2.6/site-packages/cobbler/action_sync.py", line 155, in run
    utils.run_triggers(self.api, None, "/var/lib/cobbler/triggers/sync/post/*", logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 918, in run_triggers
    raise CX("cobbler trigger failed: %s" % m.__name__)

!!! TASK FAILED !!!

hmm.. well, we can see that there is an issue with dhcpd, lets try to start the service on it's own:

[root@OPS8-Cobbler ~]# service dhcpd start
Starting dhcpd:                                            [FAILED]
[root@OPS8-Cobbler ~]#

Here's the log from /var/log/messages while trying to start the service above:

Jul  2 17:38:48 OPS8-Cobbler dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: All rights reserved.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Jul  2 17:38:48 OPS8-Cobbler dhcpd: /etc/dhcp/dhcpd.conf line 19: semicolon expected.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:      option domain-name-servers 68.87.85.102 68.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:                                                ^
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Configuration file errors encountered -- exiting
Jul  2 17:38:48 OPS8-Cobbler dhcpd:

Looking at /etc/dhcp/dhcpd.conf, I realized that I'd left a comma out of the template config.  It should look like the following in /etc/cobbler/dhcp.template:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102, 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

Now we restart the service and run a sync....

[root@OPS8-Cobbler ~]# service cobblerd restart

Stopping cobbler daemon:                                   [  OK  ]

Starting cobbler daemon:                                   [  OK  ]

[root@OPS8-Cobbler ~]# cobbler sync

task started: 2012-07-02_174015_sync

task started (id=Sync, time=Mon Jul  2 17:40:15 2012)

...

rendering DHCP files

generating /etc/dhcp/dhcpd.conf

...

running: dhcpd -t -q

received on stdout:

received on stderr:

running: service dhcpd restart

received on stdout: Starting dhcpd: [  OK  ]

...

*** TASK COMPLETE ***

[root@OPS8-Cobbler ~]#

much better.

A simple PXE test confirms that the service is working and providing addresses successfully.  However, I wasn't able to get the PXE menu to load on a network boot...  turns out that I didn't have the proper Firewall settings.  You can test by stopping your firewall (service iptables stop) and trying again; watch out for tcp/udp differences!!  After using the information under Firewall on this page  I was able to get the PXE boot menu to load and test the install.

Cobbler Install on CentOS 6.2

Cobbler - not the kind you put peaches in, this is an automated install tool

Here's the quick and dirty to get it installed and the web interface working:

1. CentOS 6.2 install
• Basic Server install option
• as root, run "setenable 0" to turn selinux to permissive (without this, selinux caused me many headaches with the "cobbler check" command later)
• as root, run "vi /etc/selinux/config" and change the SELINUX=enforcing to SELINUX=permissive.  This keeps it in permissive mode over reboots.
• optional: set up a local user with wheel access, enable wheel sudo access, and set /etc/ssh/sshd_config with "PermitLocalRootLogin without-password"
2. add EPEL repo
1. point browser to: http://fedoraproject.org/wiki/EPEL
2. right-click, copy link
3. on CentOS system (I connect through putty and change to root at this point), run
• rpm -ivh <SHIFT+INSERT> (last two keys will paste the link from step 2)
3. Install Cobbler
1. yum -y install cobbler cobbler-web koan policycoreutils-python
2. service cobblerd start
3. service httpd start
4. cobbler check
1. resolve all reported issues (I had about 10)
4. Configure Cobbler-Web
1. see cobbler-web wiki page, just remember to try http if https fails

I think you might be able to skip step 3.4 and do that after step 4 if you'd like to have the web gui, since it is available there, but I don't know if you can resolve all the issues from there.

Kudos to Mike DeHaan for a really helpful config checker; wish all software came with something like that.

iSCSI Performance, round 2

So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.

In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode.  This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.

I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO.  Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO.  VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine.  I then tested an fresh install of a system, since this is highly IO intensive.

Needless to say, I was very surprised to see the performance improvement.  Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting).  I also saw the memory on the OpenFiler system being used for caching, which was another win.

I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode.  There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time.  Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.

Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem...  the cache was using all the RAM on the box.

My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each.  Needless to say, I'm shopping for RAM right now =D.

(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).

So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure.  In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K.  More to come once I have more RAM =D