Generating a record checksum in mysql

I was recently faced with the following challenge at work:

A database process could be improved by processing diffs between two sets of data if there was a checksum that could be calculated for the record, which would subsequently be used to check if a record had changed.

The database was on mysql, and there was already a process in the workflow that calculated a unique key based on other fields in each record.

basic calculation of the unique key was as follows:

key = SHA1(concat(field1, field2));

so I wanted something similar, only for all fields that I cared about changing:

checksum = SHA1(concat(field1, field2, ..., field{n}));

however, the problem with concat in mysql is that it returns null if one of the fields in it is null.

Another method that is similar is concat_ws and looks like this

checksum = SHA1(concat_ws(',', field1, field2, ..., fieldN));

that first character in the quotes before the fields is the delimiter for concat_ws, and would normally produce strings like:

"field1value,field2value,...,fieldNvalue"

but this doesn't work if the delimiter is changed in the future (all records would be updated because all checksums would be different).  What I needed was a string that didn't have a delimiter in it.  Here's what I used:

checksum = SHA1(concat_ws('', field1, ..., fieldN));

by leaving the delimiter blank, I was able to get a string that looked like:

"field1valuefield2value...fieldNvalue"

so why use concat_ws in all this?  Because it ignores (drops) nulls by default.  Because of this, I didn't have to write any additional code to handle that case.

End result is the checksum was calculated in the same amount of time that the unique key was generated, and the diff processing it allowed provided a nearly 80% reduction in the amount of time to process the data.

Eclipse plugins I've found helpful

List of eclipse plugins I've found helpful:

getting eclipse installed on centos:
 http://www.if-not-true-then-false.com/2010/linux-install-eclipse-on-fedora-centos-red-hat-rhel/

Included in Eclipse releases
Juno Releases: http://download.eclipse.org/releases/juno/
Juno Updates: http://download.eclipse.org/eclipse/updates/4.2

Kepler Releases: http://download.eclipse.org/releases/kepler
Kepler Updates: http://download.eclipse.org/eclipse/updates/4.3


Not included in Eclipse releases:
Android Development Toolkit (ADT): https://dl-ssl.google.com/android/eclipse/ aptana studio3: http://download.aptana.com/studio3/plugin/install
checkstyle: http://eclipse-cs.sf.net/update

eCobertura:http://ecobertura.johoop.de/update
FindBugs:http://findbugs.cs.umd.edu/eclipse
PMD: http://pmd.sourceforge.net/eclipse
subclipse: http://subclipse.tigris.org/update_1.8.x (if I need to go back to svn)
Cucumber-JVM: http://cucumber.github.com/cucumber-eclipse/update-site (mostly used for gherkin .feature file syntax highlighting; I use Spinach in a Ruby stack for running the tests)

Speeding up file open/close in Eclipse 4.3(Kepler) on CentOS 6.5

I recently started a new job, and have been using CentOS 6.5 for my laptop while working as a QA Automation engineer.  One of the issues I have run into is that eclipse (main IDE that I use) was taking a long time to both open and close files.

Come to find out that a bug that was initially reported in 2008 is still wreaking havok, namely that gtk+ hangs when looking for network printers using CUPS.  This is significant because it doesn't just affect eclipse, but has also been reported to affect firefox, openoffice/LibreOffice, and other applications where you would want to print.

If you want to confirm that this affects you, start eclipse with the following arguements:

eclipse -vmargs -Dorg.eclipse.swt.internal.gtk.disablePrinting

then, open and close files in eclipse and observe performance.  For me, files were taking approx 10-30 seconds to open, and up to 30 seconds to close, for *every file*.  Running with this option dropped the open/close time to <= 5 sec per file.

Since this impacts more than just eclipse, I wanted to make sure this was resolved on my system.  The best workaround I have found is the following, mentioned at the debian bug report listed below.  Steps are as follows:

1. sudo vi /etc/cups/cupsd.conf
1. comment out "Listen /var/run/cups/cups.sock"
2. sudo vi /etc/cups/client.conf
1. add "ServerName 127.0.0.1"
3. sudo service cups restart

after this, I was able to open eclipse as normal, without the -vmargs argument and without the delay in opening/closing files.  Less time waiting on the computer, more time being productive. :-)

see the links below if you are interested in the nitty gritty.

debian bug report that lists work around:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=594997

eclipse bug report of the issue with the eclipse vmargs workaround
https://bugs.eclipse.org/bugs/show_bug.cgi?id=275072

linked eclipse bug with more info:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=215234

eclipse faq entry with short discussion of issue:
http://www.eclipse.org/swt/faq.php#printOnGTKHangs

Linux printing to a Windows Shared Printer - update

This is an update to my previous post about getting my printer working.

Basic set-up was a network print server that was already set up and working with Windows (XP and 7).  Goal was to add the printer to a Linux system without affecting any existing set-up.  I tried CUPS, but ultimately used LPD/LPR to get it working.

My previous post was for RHEL/CEntOS 5.  This update relates to RHEL/CEntOS 6, and can be done from the Gnome Desktop.  These instructions assume that you already know the host IP you're using as your print server

In the following instructions, enter the root password if you are prompted for authentication:

1. Click System > Administration > Printing
2. Click the "New" button
3. Enter password if prompted
4. Click the "Do it later" button to change your firewall settings (you shouldn't need to change them)
5. Enter Root password if prompted
6. Expand "Network Printer" in the left tree
7. Select "LPD/LPR Host or Printer" in the left tree
8. In the right side, enter your information
• (in my case, Host: 192.168.x.x, Queue: Samsung_2010)
9. Click "Forward"
10. Select your printer make from the database and click "Forward"
11. Select your printer model from the left tree, and the driver from the right tree
12. Click "Forward"
13. Enter a unique name for the computer (line #1), a human-readable name (line #2) and a location (line #3)
14. Click "Apply"
15. When prompted, click "Yes" to print a test page

You should be set at this point!

SQLite Database Browser on CentOS6

Working through Michael Hartl's Ruby on Rails Tutorial, chapter 6 calls for the SQLite Database Browser.

Just wanted to put a note out there for anyone building this on CentOS6: make sure you check the version of qt that your qmake command runs against.

The below commands show how to check which version you're using, and run the correct one to build Sqlite Database Browser.

check installed qt versions
[user@dev-workstation sqlitebrowser]$ which qmake
/usr/lib64/qt-3.3/bin/qmake
[user@dev-workstation sqlitebrowser]$ ls /usr/lib64/qt*
/usr/lib64/qt-3.3:
bin  include  lib  mkspecs  phrasebooks  plugins  translations

/usr/lib64/qt4:
bin  mkspecs  phrasebooks  plugins  q3porting.xml

confirm qt4/bin/qmake is there
[user@dev-workstation sqlitebrowser]$ ls /usr/lib64/qt4/bin/qmake
/usr/lib64/qt4/bin/qmake

commands to run to build Sqlite Database Browser
[user@dev-workstation sqlitebrowser]$ /usr/lib64/qt4/bin/qmake
[user@dev-workstation sqlitebrowser]$ make

after building, move to your ~/bin folder and enjoy
[user@dev-workstation sqlitebrowser]$ cp sqlitebrowser ~/bin/
[user@dev-workstation sqlitebrowser]$ which sqlitebrowser
~/bin/sqlitebrowser
[user@dev-workstation sqlitebrowser]$ sqlitebrowser &
[user@dev-workstation sqlitebrowser]$

hope this helps. :)

Setting up Ruby on Rails dev environment

We are working on a ruby app for one of my CS classes,  and found that it was easier to set up a virutal linux system than it was to try to get ruby working on windows (mainly because we don't have any stick time on that, and spinning the VM was cheaper).

The principle behind our decision to do this was that it's easier to get help from the community when you stick with the main focus of the community.  See an app out there that is primarily linux, but you're a windows admin?  It's easier for you to get help by installing and learning linux than it is to try to make it work under windows, even if this is a possibility.  I've had to learn this the hard way at work, and it was much easier to move forward with the software when I finally bit the bullet and went all in on linux.

The argument against this goes something like: "we have experience in the company with 'blah' and this can be made to run on 'blah', so we'd like to leverage our existing expertise..." etc.  Unless you're looking to become the pioneers in making it work, you're better off just sticking with the primary environment the software was built in.

The place this really matters is smaller companies.  Most of you out there probably use windows for your day-to-day work, because it's familiar, but the software you'd like to use runs on linux, which you don't really know.  If you're trying to avoid the cost of having it hosted (read "managed") by another company, learn linux.

I'm interested to know what others think of this, if they have experiences that are similar to mine, or if you think I'm nuts.  (OK, you're right on that last part :-D ).

Here's what we did to get our ruby environment set up.  If anyone has feedback on these instructions, I'm all ears.  Have fun.

(it should be noted that the instructions below were used in conjunction with http://ruby.railstutorial.org/book.  The part on bitbucket and heroku were based on having already published an app per that tutorial).

installing CentOS 6.3 on virtual box
===========

install virtual box
    for windows:     http://download.virtualbox.org/virtualbox/4.2.6/VirtualBox-4.2.6-82870-Win.exe
    for mac:         http://download.virtualbox.org/virtualbox/4.2.6/VirtualBox-4.2.6-82870-OSX.dmg

setup VM virtual box config
    Create Virtual Machine
        click new

        name virtual system (ie: Centos6 x64)
        if not already selected, select "Linux" and "Red Hat 64 bit"
        click next

        set for 1024 MB RAM
        create a virtual hard drive now, click create
        VDI, click next
        Dynamically allocated, click next
        set for 16GB, click create
  
    click settings
  
    Configure Network Connections
        select "Network" in left tree
        set as follows:
            Adatper 1:
                Enabled
                Attached to: Bridged Adapter
                Name: (your wired ethernet connection)
            Adapter 2:
                Enabled
                Attached to: Bridged Adapter
                Name: (you wireless ethernet connection)

    Configure iso for install at boot
        click settings
        select "Storage" in left tree
        select CD drive ("Empty") under Storage Tree on right
        click CD icon at right of "IDE Secondary Master" drop down
        select "Choose a virtual CD/DVD disk file"
        find the CentOS6 DVD iso
      
    click ok to confirm VM settings

config/install CentOS OS
    start vm
    click in vm screen to capture input
    click enter to start install
  
    once CentOS 6 logo loads, you can set one of the following:
        right-CTRL+F = full screen
        right-CTRL+C = scaled
        (I find scaled set to full screen works the best)
      
    proceed with install:
        click next (3x)
        basic storage device, click next
        click "yes, discard any data" (this is for the virtual disk)
        click configure network (lower left)
            for each adapter:
                double-click on name
                select "connect automaticaly"
                click "Apply"
            click "close"
        change hostname if desired (ie: centos6-dev)
      
        time zone = America/Denver
        uncheck "System clock uses UTC"
        click next
      
        set root password, click next
  
        select "use all space", click next
        click "write changes to disk"
      
        select "Software Development Workstation"
        click next (this starts installing packages)
      
        let install run
  
        reboot
      
        click forward
        "yes, I agree", click forward
        create your user, click forward
        check "Synchronize date and time over the network", click forward (allow ntp to start)
        click ok when prompted by kdump
        click finish
      
To open firefox:
    click icon
  
To open Terminal:
    Applications > System Tools > Terminal

  
set up sudo access and harden ssh login
    $ su - root (enter root password when prompted)
    allow wheel access to root via sudo
        # visudo
        arrow down to the "wheel" group
        press [i]
        for the line with %wheel  ALL=(ALL)  ALL, delete the # sign at the beginning
        press [ESC][:][X][ENTER]
    give yourself wheel group membership
        # usermod -G wheel <username>
        confirm
        # id <username> wheel(10) should show up in the list of groups
    change #PermitRootLogin yes to PermitRootLogin without-password
        # vi /etc/ssh/sshd_config
        arrow down and over
        press [i]
        use arrows, backspace and keys to change line
        press [ESC][:][X][ENTER]
        # service sshd restart

    what this will do is allow you to log in and switch users as needed, and prevents someone from logging in as root unless they are using ssh keys
  
update the system
    as root:
        # yum -y update
    as your user
        $ sudo yum -y update
    reboot

      
Install Eclipse Juno:
    http://www.if-not-true-then-false.com/2010/linux-install-eclipse-on-fedora-centos-red-hat-rhel/
  
Install Eclipse plugins:
    eclipse marketplace
    aptana studio
    dynamic languages toolkit - Ruby Development tools
    dynamic languages toolkit - core framework
    eclipse EGit
    Eclipse JGit
    eclipse web developer tools
  
Import the project:
    download the repo from bitbucket
    (this can be done from the terminal view in the eclipse web perspective)
    $ cd ~/workspace
    $ git clone https://$user_name@bitbucket.org/$user_name/$app_name.git
    $ cd $app_name
    $ git checkout master

Import the project to eclipse with this project:
    choose ruby perspective
    window > show view > project explorer
    method 1 (may not work):
        expand local filesystem, browse /path/to/git/advising_app
        right-click on advising app, choose "Promote to project"
        open App Explorer, and you should see the project
    method 2 (known to work:
        right click in project explorer, select import
        select General > Existing Project into workspace
        browse to advising_app folder
        click check box by folder on left side
        click finish
      
To open a CLI terminal in Eclipse:
    From Web perspective, it will be in the bottom portion of your screen
    From Ruby (or any other perspective):
        Window menu > Show View > click Other > Studio > Terminal
  
    If you would like a terminal editor (same place as files), click the bigger icon in the view
  
Some of the next steps might need to be run from a system terminal rather than an eclipse terminal, I can't remember which

Install Ruby on Rails environment
    Install RVM:
        $ \curl -L https://get.rvm.io | bash -s stable
    Install Ruby, RubyGems, Rails, etc.
        see section 1.2.2 of http://ruby.railstutorial.org/book
      
Install Heroku toolbelt:
    $ wget -qO- https://toolbelt.heroku.com/install.sh | sh
    $ cd ~/workspace/advising_app
    $ heroku login
    enter your heroku account credentials when prompted

BitBucket: git familiar with Atlassian's repo service

I've recently started using a new hosted git provider: Atlassian's BitBucket.  While most everyone out there is probably using github, there were two things I noticed about Bitbucket that I liked:

1. They have made https work for checking in with git version 1.7.1 (default yum install git from CentOS 6.x)
2. They allow the option to have private repositories

While I am a huge supporter of open source software, until now, I wasn't aware of any public cloud options for having a private repo without setting up a billed account (other than installing and managing your own server).  Simply put, I'd rather not add yet another copy of whatever tutorial I'm working through to the web.  (I will however, write about it :) )  Not everything should be public.

I'm also interesting in hearing what you have to say about this.  Will you be migrating everything as soon as you can?  Will you be sticking with GitHub?  Some mix of the two?  What arguments could or should be considered for paying for this kind of service?  I do have to say that I'm glad there is another option, since competition leads to better products and services for those using them.

Configuring MantisBT

Once you're done Installing MantisBT, the web interface is available.  However, in order to create/reset user accounts, you need to have email configured so emails can be sent out.  I did the following:

vi /opt/mantisbt-1.2.11/config_defaults_inc.php
search for "g_smtp_host"
change the following settings (I used one of my existing email accounts):

    $g_phpMailer_method = PHPMAILER_METHOD_SMTP;
    $g_smtp_host = 'smtp.example.com';
    $g_smtp_username = 'account@mailserver.com';
    $g_smtp_password = 'THIS_IS_CLEAR_TEXT';
    $g_smtp_connection_mode = 'ssl';
    $g_smtp_port = 465;

At this point, I was able to send email to my account and get logged in with an account other than the default administrator account.

While MantisBT did well at the install, the configuration interface leaves a few things to be desired.  First, there are settings beyond getting the system to use the web server or database that cannot be configured in the web gui.  Secondly, while all of the possible settings are well documented, grep is your friend; there is no clear indication where they are located.

Total score for ease of administrative configuration: 3/10
Major failing: unable to edit full config from the web interface.

Installing MantisBT

from a fresh CentOS 6 install (don't forget yum -y update and a reboot), run the following commands:

1. cd /tmp
2. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz/download
3. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz.digests/download
4. md5sum -c mantisbt-1.2.11.tar.gz.digests
5. tar -xzf mantisbt-1.2.11.tar.gz
6. mv mantisbt-1.2.11/ /opt/mantisbt-1.2.11
7. chown -R apache:apache /opt/mantisbt-1.2.11
8. ls -n /opt/mantisbt-1.2.11/ /var/www/html/mantisbt
9. yum -y install httpd mysql mysql-server php php-mysql
10. vi /etc/php.ini
11. change line 946 to match:
• date.timezone = "America/New_York" (should match cat /etc/sysconfig/clock output)
12. chkconfig httpd on
13. chkconfig mysqld on
14. service httpd start
15. service mysqld start
16. mysqladmin -u root password 'newPassword'
17. mysqladmin -u root -p -h localhost.localdomain password 'newPassword' (enter newPassword from above when prompted)
18. iptables -I INPUT 5 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
19. iptables -I INPUT 6 -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
20. now open your browser, and point it to:
• http://yourServerOrIPHere/mantisbt/admin/install.php
21. add newPassword from above in the password field
22. click the "Install" button.

Total time with Cobbler from PXE boot to creating your first project: ~20 - 25 mins.  This includes the time to install the OS from scratch, update it, reboot, and go through the commands above manually.  For those interested in specs, this was run on my setup.

Once you're done here, you'll want to start Configuring MantisBT.

Total score for ease of install: 8/10

My testing/working environment

Below is the current config that I have set up for all the testing/installing, etc. that I do as part of this blog.

2x Dell 1950 2x  Xeon 5130 @ 2.00GHz w/ 8GB RAM and 4x 73GB 10K 2.5" SAS HDD in RAID 5
runs: VMware ESXi 4.1 build-260247

2x Dell 2850 2x  Xeon DC @ 2.80GHz w/ 8GB RAM and 6x 146GB 10K 3.5" SCSI HDD in RAID 5
runs: OpenFiler 2.3 x86_64


Connecting the two is a Linksys SGE2000 running Jumbo Frames for 1GbE connections, and I'm serving up the storage from the 2850 as iSCSI storage to the 1950.

Finally, I have an APC SMX1500RM2U SmartUPS protecting everything from power bumps (I would get some occasionally that weren't enough to power off the systems, but were enough to make them reboot).  It provides about 5-7 mins of run-time currently.

Everything runs off a single 15A 120V circuit in my basement.  Total cost: ~$2600 - $3000.  This assumes you already have cooling for your room.

Cobbler DHCP config

I'm working on installing Cobbler so that I have a reliable way to automate the install of my linux systems.  Following my first post will get the software installed, so now I was ready to move on to actually getting a PXE boot to install from Cobbler.

Looked up the directions here, and the steps are pretty straight forward.  If you're continuing from my previous post, it would look something like this:

1. insert your dvd with the distro
2. yum -y install dhcp
3. mount /dev/dvd /media
4. cobbler import --path=/media --name=CentOS6 (this takes a while, press [CTRL+Z], then run bg [ENTER] to send it to the background)
5. vi /etc/cobbler/dhcp.template
6. edit per your network, mine looked like this:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

7. service cobblerd restart
8. cobbler sync

All is well, we're now ready to PXE boot clients, and Cobbler will..... what's this?

running: dhcpd -t -q
received on stdout:
received on stderr:
dhcpd -t failed
Exception occured: <class 'cobbler.cexceptions.CX'>
Exception value: 'cobbler trigger failed: cobbler.modules.sync_post_restart_services'
Exception Info:
  File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run
    rc = self._run(self)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 184, in runner
    return self.remote.api.sync(self.options.get("verbose",False),logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 701, in sync
    return sync.run()
   File "/usr/lib/python2.6/site-packages/cobbler/action_sync.py", line 155, in run
    utils.run_triggers(self.api, None, "/var/lib/cobbler/triggers/sync/post/*", logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 918, in run_triggers
    raise CX("cobbler trigger failed: %s" % m.__name__)

!!! TASK FAILED !!!

hmm.. well, we can see that there is an issue with dhcpd, lets try to start the service on it's own:

[root@OPS8-Cobbler ~]# service dhcpd start
Starting dhcpd:                                            [FAILED]
[root@OPS8-Cobbler ~]#

Here's the log from /var/log/messages while trying to start the service above:

Jul  2 17:38:48 OPS8-Cobbler dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: All rights reserved.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Jul  2 17:38:48 OPS8-Cobbler dhcpd: /etc/dhcp/dhcpd.conf line 19: semicolon expected.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:      option domain-name-servers 68.87.85.102 68.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:                                                ^
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Configuration file errors encountered -- exiting
Jul  2 17:38:48 OPS8-Cobbler dhcpd:

Looking at /etc/dhcp/dhcpd.conf, I realized that I'd left a comma out of the template config.  It should look like the following in /etc/cobbler/dhcp.template:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102, 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

Now we restart the service and run a sync....

[root@OPS8-Cobbler ~]# service cobblerd restart

Stopping cobbler daemon:                                   [  OK  ]

Starting cobbler daemon:                                   [  OK  ]

[root@OPS8-Cobbler ~]# cobbler sync

task started: 2012-07-02_174015_sync

task started (id=Sync, time=Mon Jul  2 17:40:15 2012)

...

rendering DHCP files

generating /etc/dhcp/dhcpd.conf

...

running: dhcpd -t -q

received on stdout:

received on stderr:

running: service dhcpd restart

received on stdout: Starting dhcpd: [  OK  ]

...

*** TASK COMPLETE ***

[root@OPS8-Cobbler ~]#

much better.

A simple PXE test confirms that the service is working and providing addresses successfully.  However, I wasn't able to get the PXE menu to load on a network boot...  turns out that I didn't have the proper Firewall settings.  You can test by stopping your firewall (service iptables stop) and trying again; watch out for tcp/udp differences!!  After using the information under Firewall on this page  I was able to get the PXE boot menu to load and test the install.

Cobbler Install on CentOS 6.2

Cobbler - not the kind you put peaches in, this is an automated install tool

Here's the quick and dirty to get it installed and the web interface working:

1. CentOS 6.2 install
• Basic Server install option
• as root, run "setenable 0" to turn selinux to permissive (without this, selinux caused me many headaches with the "cobbler check" command later)
• as root, run "vi /etc/selinux/config" and change the SELINUX=enforcing to SELINUX=permissive.  This keeps it in permissive mode over reboots.
• optional: set up a local user with wheel access, enable wheel sudo access, and set /etc/ssh/sshd_config with "PermitLocalRootLogin without-password"
2. add EPEL repo
1. point browser to: http://fedoraproject.org/wiki/EPEL
2. right-click, copy link
3. on CentOS system (I connect through putty and change to root at this point), run
• rpm -ivh <SHIFT+INSERT> (last two keys will paste the link from step 2)
3. Install Cobbler
1. yum -y install cobbler cobbler-web koan policycoreutils-python
2. service cobblerd start
3. service httpd start
4. cobbler check
1. resolve all reported issues (I had about 10)
4. Configure Cobbler-Web
1. see cobbler-web wiki page, just remember to try http if https fails

I think you might be able to skip step 3.4 and do that after step 4 if you'd like to have the web gui, since it is available there, but I don't know if you can resolve all the issues from there.

Kudos to Mike DeHaan for a really helpful config checker; wish all software came with something like that.

iSCSI Performance, round 2

So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.

In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode.  This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.

I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO.  Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO.  VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine.  I then tested an fresh install of a system, since this is highly IO intensive.

Needless to say, I was very surprised to see the performance improvement.  Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting).  I also saw the memory on the OpenFiler system being used for caching, which was another win.

I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode.  There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time.  Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.

Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem...  the cache was using all the RAM on the box.

My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each.  Needless to say, I'm shopping for RAM right now =D.

(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).

So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure.  In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K.  More to come once I have more RAM =D

Ansible setup

Ansible - def.  1. super-luminal (aka, faster than light)
                        2. system managment automation program on github you wished you were running

Ansible is set up to be very simple, and runs over ssh.  Here are my notes from trying to get it installed and working on Centos 6.2, using the "Running from Checkout" instructions found at http://ansible.github.com/gettingstarted.html, which gets you version 0.5.  The RPM from EPEL provides version 0.3.

here's my super quick instructions, the few issues I ran into mentioned below:

1. start with CEntOS 6.2
2. sudo su - root or su - root
3. install needed packages
1. # rpm -ivh http://mirror.pnl.gov/epel/6/i386/epel-release-6-7.noarch.rpm
2. # yum -y install python PyYAML python-jinja2 python-paramiko
3. # exit
4. add ansible
1. $ git clone git://github.com/ansible/ansible.git
2. $ cd ./ansible 
3. $ source ./hacking/env-setup
5. configure hosts
1. $ echo "127.0.0.1" > ~/ansible_hosts 
2. $ export ANSIBLE_HOSTS=~/ansible_hosts
6. and test:
• $ ansible all -m ping -u dewey.garwood
  127.0.0.1 | success >> {
      "ping": "pong"
  }

you should note the following errors will occur if you aren't paying attention:

• if you go looking for paramiko, yum wont find it; you have to use python-paramiko
• without the -u option in the test command (step 6), ansible tries to use the root user to log in and you end up with:
• $ ansible all -m ping --ask-pass
  SSH password:
  127.0.0.1 | FAILED => FAILED: Authentication failed.

iSCSI perfomance

If you've read any of my other posts, you know I'm running OpenFiler as an iSCSI backend for VMware ESXi 4.1.

There are some issues with running it in this manner, and I hope to write out some more instructions later about setting up to use SCST rather than IETD.  However, this is for anyone out there who might be trying to get better performance out of your iSCSI infrastructure... hopefully this will help you avoid my "doh!" moment.

If you haven't already done so, find a time to bring your environment down long enough to turn on jumbo frames on your switches.  Your VMs and the customers who use them will thank you, not by saying anything, but by not complaining that the performance is really slow.

After having done so, my average write latencies have gone from triple digits to double digits, and my throughput has roughly doubled.  Also, my Openfiler system has gone from load averages that were around 1 to around 4 - 5 (4 is a full load for my system).

So here's a friendly reminder to avoid my face-palm moment X[ and get some decent performance out of your system =D

Minor format tweaks to blog

aka: how to make your background image stay put using CSS

Someone mentioned that it would be nice if the background would stay put on my blog, so it was always there, rather than just at the top.

Since I'm in the process of learning html and css, figured I would see if I could do something about that.  Care to guess which CSS section I'm learning about right now? =-D

Before:
body {
background: #000000 url(<image_url_here>)  repeat-x scroll top center /* Credit for photo here */;
}

After:
body {
background: #000000 url(<image_url_here>)  repeat-x fixed top center /* Credit for photo here */;

}


I hope this makes the main blog a bit easier to read, and not seem like you're Lost in Space™ (weeeoooo!) when you scroll down.

I haven't been able to get the mobile working yet, so if you're looking at this on a too-smart-for-your-own-good phone and you know how to fix it, drop me a comment, please.  Or be patient; I should be there in a few more chapters :)

Also, I want to take this time to highly recommend www.murach.com.  They publish books that are excellent tools for learning technology, and are worth their weight in gold.  You won't find a better book for getting up to speed on a topic quickly, provided that they have a book that covers what you're looking for.

So, just in case anyone from murach.com is reading this, a few topics I'd like to request:
Perl, Python, Apache Administration, and testing automation.

In the meantime, if you're interested in those topics, stay tuned, I'll probably end up with something to "leak".

Method for cooking down Pumpkin

Hello again.  Been a while, been busy, and thought I'd write something unrelated to systems administration.

If you find this helpful, leave a comment.  Thanks for reading!

One of the things that has always frustrated me around Halloween was throwing out the pumpkin that was carved just a night or two ago.  If it's a reasonable size to carve, then you're talking about throwing out at least a couple of cans for each pumpkin.

If you've ever tried to cook pumpkin before, you know the amount of work involved, scraping out the insides (and getting seeds if you enjoy eating them), then cutting up the pumpkin to cook it, and trying to find a way to get the peel off without burning your fingers (I really hate that part), and then trying to turn it into puree.  So after trying several different things, here's what my wife and I have come up with.

This method minimizes the amount of work you'll have to put in, as well as any burns you might receive from handling hot pumpkin.

Stats for one pumpkin (reasonable carving size)
Total cook time: 2 hours
Total prep time: 1.5 hours w/ seeds, 1 hour w/o seeds
Total seeds: 1 cup (approx)
Total pumpkin yield: 1 quart

Step one: Cut pumpkin in half, seed, and scrape out stringy insides.

yes, getting the seeds can be a bit of a slimy mess, but if you enjoy eating them like I do, it's worth it.  For about 15 mins of work, you end up with about a cup of seeds per pumpkin, and they're easier to get out than sunflower seeds.

If you've carved your pumpkin, you've already gone through the process of cleaning out the inside, so just cut the pumpkin in half.

Step Two: Cut pumpkin into strips no more than 1" thick.

I find that holding the pumpkin with the outer shell towards you and pushing down on the handle end of the knife works well.  I also use the largest knife we have when doing this work.  Also, cutting a strip single that has the stem and the stub where the flower was (bottom) make it easy to remove these.

Step Three: With a vegetable peeler, remove the outer shell.

When I finally thought to do this, I was surprised how easy it was.  It's a bit more like peeling carrots than potatoes, and removes the shell quickly without much effort.  You'll want the peeler at an angle, rather than the whole blade flat on the pumpkin, or it will be harder to get started; once started, it's pretty easy to get under the shell.

Step Four: In a 6 qt pot, put in 1/2 cup water (enough to cover the bottom about 1/4"), and place the pumpkin in.  Cook covered for 1 hour over med-low heat.  Pumpkin is cooked when it cuts easily with a fork.

This helps to remove the water.  You'll start with 1/2 cup, but you might have to drain it a few times to avoid having it boil over.  You many also want to cut it into smaller pieces to get it into the pot (4-6" strips).

Step Five: Pack the pumpkin in a blender, mashing out as much water as possible.  Then, puree the pumpkin.

You can actually fit 1 whole pumpkin in a blender that holds a quart.  It is preferable to have a blender that also has a dispenser on the bottom, since this is the easiest way to get the pureed pumpkin out.  I use a potato masher to press the pumpkin in.  Also, you'll want to get as much water out now as you can, before you puree the pumpkin.

Step Six: Cook puree uncovered over med-low heat to remove water, stirring occasionally, until it makes a paste about the consistency of semi-thick oatmeal. (about 1 hour)

Your pumpkin is now ready to use in recipes (pie, scones, oatmeal, cookies, butter, etc.)

VTP on Cisco Switches in a Small Company (aka: my network just drops)

Sorry it's been a while.  Here's the most recent fun bang-head-here problem I was able to resolve.

Situation:

3 Cisco Switches in an office.  1x 3750, 2x 2960S

Every so often at random intervals, the network connections for all the clients would just vanish; connectivity through the main switch was fine (used Zenoss to monitor, only reported failure of the switches, and a printer beyond them), but couldn't get to any of the clients, and they couldn't use the network, let alone the internet.

Troubleshooting:
I tried everything I could think of to identify this problem.  checked spanning tree, checked logging to see if I could catch it (this was one of those really random problems, highly unpredictable), made sure the VLANs were set correctly, had Zenoss pulling snmp data for interface utilization % on the trunks, etc.

what I noticed was the following: graphs didn't show any vertical breaks, so the interfaces never went down, even though the network connections would drop.  This meant the switch was up, and there was no problem with the physical wiring, as well as the power to the switches.

after asking someone else more knowledgeable than me, he pointed me in the direction of VTP settings.

What I learned (they probably cover this in CCNA 101): VTP is a proprietary Cisco protocol used to simplify VLAN management on many many switches (think triple digits or higher), allowing Network admins to manage them all from one point.  Makes sense, cuts down the amount of mistakes and time to configure a switch fabric.  My problem was that the three devices that were installed in the company had not been configured correctly, and since they were all non-configured when they were added, they all became servers.  Apparently, they couldn't decide which switch was the authoritative switch, and when the switch designated as the true master would change, all the VLANs would be deleted off these switches, and then added back.  Net result was the switches looked like they were going down.  Highly unpredictable, highly annoying (to everyone).

Resolution:

Set the switches to VTP transparent mode.  commands were really as simple as:

log in
config t
vtp mode transparent
write mem

some things to remember are to check your vtp status to see where you are on a given switch (show vtp status), and that you need to make sure you are not using vtp pruning when you make the change.  The change does not prevent you from connecting to the switch (some reported a delay, but I didn't experience one), but if vtp pruning is in place, it can cause problems getting your clients to connect as you change switches in the environment.  Since the environment I'm in is so small, I just set vtp transparent, since I could set the vlans on those switches, and they would still forward vtp packets.

info that I used included the following:
https://supportforums.cisco.com/thread/2029581 (be sure to read the whole forum thread)
http://www.cisco.com/en/US/tech/tk389/tk689/technologies_tech_note09186a0080094c52.shtml (main page about VTP configuration and what it is and does)
http://www.cisco.com/warp/public/473/vtp_flash/
(this really helped with my understanding of VLAN Trunking Protocol, VTP; the first problem discussed is exactly what I was facing, called Problem #1, of all things)

something else I learned (again, probably CCNA 101) was that a good protection technique on making changes where you might possibly lose connectivity to a switch is to start with the following before you make your change:

reload in {mmm|hhh:mm}
<make your change>
reload cancel (after change is complete)

this allows you to work, and if something happens that you can no longer connect to the switch, it will reload the config that worked before you started.

I'm sure there are more knowledgeable networking folks out there, but this was how I solved this problem for the time being.  Simply putting this out there for anyone who could use it; like me when I run into this problem again. (=

Setting up SNMP on OSX 10 Xserve via SSH

setting up snmpd on MAC Server OS X via ssh

verify /usr/sbin/snmpd exists
 ls /usr/sbin/snmpd

verify /usr/share/snmp/snmpd.conf exists
 ls /usr/share/snmp/snmpd.conf

if snmpd.conf doesn't exist, run:
 /usr/bin/snmpconf -i (-i is required to write the file to the correct location)

    set the following options:
 
 default = all
 1 (snmpd.conf)
 1 (access control setup)
  3 v1/2c ro community name
   <ro_community>
  f
 4 (Agent operating mode)
  2 (system user agent runs as)
   root
  f
 5 (system information setup)
  1 (physical location of system)
   <system_location>
  f
 6 (trap destinations)
  2 (v2c trap receiver)
   <monitoring_system_ipaddr>
   [ENTER]
   [ENTER]
  3 (v2c inform receiver)
   <monitoring_system_ipaddr>
   [ENTER]
   [ENTER]
  5 (default trap sink community)
   <ro_community>
  f
 f
    q


starting snmpd:
 /usr/sbin/snmpd

restart snmpd:
 kill -HUP <pid>

finally, to make sure it runs at boot time:
=======================================================================================
from: http://scott.wallace.sh/2009/12/04/enabling-snmp-in-mac-os-x-10-6-snow-leopard/
---------------------------------------------------------------------------------------
Under Snow Leopard there is a slight change to the way services are enabled.
-w       Overrides the Disabled key and sets it to false. In previous versions, this
         option would modify the configuration file. Now the state of the Disabled key
         is stored elsewhere on-disk.

So, to enable the SNMP daemon correctly:
$ sudo launchctl load -w /System/Library/LaunchDaemons/org.net-snmp.snmpd.plist
=======================================================================================

Configuring DNS Servers on OS X via SSH

Need to set up/change the DNS servers on a Mac OS X system, using SSH.  Command to use is networksetup.  For instance:

$ networksetup /?

for all it's gory details.

What concerns me today is just DNS config for the system.  So, without further adiou, commands are in bold:

user:~ localhost$ networksetup -listallnetworkservices
An asterisk (*) denotes that a network service is disabled.
Ethernet 1
Ethernet 2
*Built-in Serial Port (1)
FireWire
user:~ localhost$ networksetup -getdnsservers Ethernet\ 1  (observe your character escape sequences)
192.168.1.10
192.168.1.9
user:~ localhost$ networksetup -getdnsservers Ethernet\ 2
192.168.1.10
192.168.1.9
user:~ localhost$ sudo networksetup -setdnsservers Ethernet\ 1 10.75.66.2
Password:
user:~ localhost$ networksetup -getdnsservers Ethernet\ 1
10.75.66.2
user:~ localhost$ sudo networksetup -setdnsservers Ethernet\ 2 10.75.66.2
user:~ localhost$ networksetup -getdnsservers Ethernet\ 2
10.75.66.2
user:~ localhost$