Generating a record checksum in mysql

I was recently faced with the following challenge at work:

A database process could be improved by processing diffs between two sets of data if there was a checksum that could be calculated for the record, which would subsequently be used to check if a record had changed.

The database was on mysql, and there was already a process in the workflow that calculated a unique key based on other fields in each record.

basic calculation of the unique key was as follows:

key = SHA1(concat(field1, field2));

so I wanted something similar, only for all fields that I cared about changing:

checksum = SHA1(concat(field1, field2, ..., field{n}));

however, the problem with concat in mysql is that it returns null if one of the fields in it is null.

Another method that is similar is concat_ws and looks like this

checksum = SHA1(concat_ws(',', field1, field2, ..., fieldN));

that first character in the quotes before the fields is the delimiter for concat_ws, and would normally produce strings like:

"field1value,field2value,...,fieldNvalue"

but this doesn't work if the delimiter is changed in the future (all records would be updated because all checksums would be different).  What I needed was a string that didn't have a delimiter in it.  Here's what I used:

checksum = SHA1(concat_ws('', field1, ..., fieldN));

by leaving the delimiter blank, I was able to get a string that looked like:

"field1valuefield2value...fieldNvalue"

so why use concat_ws in all this?  Because it ignores (drops) nulls by default.  Because of this, I didn't have to write any additional code to handle that case.

End result is the checksum was calculated in the same amount of time that the unique key was generated, and the diff processing it allowed provided a nearly 80% reduction in the amount of time to process the data.

Eclipse plugins I've found helpful

List of eclipse plugins I've found helpful:

getting eclipse installed on centos:
 http://www.if-not-true-then-false.com/2010/linux-install-eclipse-on-fedora-centos-red-hat-rhel/

Included in Eclipse releases
Juno Releases: http://download.eclipse.org/releases/juno/
Juno Updates: http://download.eclipse.org/eclipse/updates/4.2

Kepler Releases: http://download.eclipse.org/releases/kepler
Kepler Updates: http://download.eclipse.org/eclipse/updates/4.3


Not included in Eclipse releases:
Android Development Toolkit (ADT): https://dl-ssl.google.com/android/eclipse/ aptana studio3: http://download.aptana.com/studio3/plugin/install
checkstyle: http://eclipse-cs.sf.net/update

eCobertura:http://ecobertura.johoop.de/update
FindBugs:http://findbugs.cs.umd.edu/eclipse
PMD: http://pmd.sourceforge.net/eclipse
subclipse: http://subclipse.tigris.org/update_1.8.x (if I need to go back to svn)
Cucumber-JVM: http://cucumber.github.com/cucumber-eclipse/update-site (mostly used for gherkin .feature file syntax highlighting; I use Spinach in a Ruby stack for running the tests)

Speeding up file open/close in Eclipse 4.3(Kepler) on CentOS 6.5

I recently started a new job, and have been using CentOS 6.5 for my laptop while working as a QA Automation engineer.  One of the issues I have run into is that eclipse (main IDE that I use) was taking a long time to both open and close files.

Come to find out that a bug that was initially reported in 2008 is still wreaking havok, namely that gtk+ hangs when looking for network printers using CUPS.  This is significant because it doesn't just affect eclipse, but has also been reported to affect firefox, openoffice/LibreOffice, and other applications where you would want to print.

If you want to confirm that this affects you, start eclipse with the following arguements:

eclipse -vmargs -Dorg.eclipse.swt.internal.gtk.disablePrinting

then, open and close files in eclipse and observe performance.  For me, files were taking approx 10-30 seconds to open, and up to 30 seconds to close, for *every file*.  Running with this option dropped the open/close time to <= 5 sec per file.

Since this impacts more than just eclipse, I wanted to make sure this was resolved on my system.  The best workaround I have found is the following, mentioned at the debian bug report listed below.  Steps are as follows:

1. sudo vi /etc/cups/cupsd.conf
1. comment out "Listen /var/run/cups/cups.sock"
2. sudo vi /etc/cups/client.conf
1. add "ServerName 127.0.0.1"
3. sudo service cups restart

after this, I was able to open eclipse as normal, without the -vmargs argument and without the delay in opening/closing files.  Less time waiting on the computer, more time being productive. :-)

see the links below if you are interested in the nitty gritty.

debian bug report that lists work around:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=594997

eclipse bug report of the issue with the eclipse vmargs workaround
https://bugs.eclipse.org/bugs/show_bug.cgi?id=275072

linked eclipse bug with more info:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=215234

eclipse faq entry with short discussion of issue:
http://www.eclipse.org/swt/faq.php#printOnGTKHangs

Linux printing to a Windows Shared Printer - update

This is an update to my previous post about getting my printer working.

Basic set-up was a network print server that was already set up and working with Windows (XP and 7).  Goal was to add the printer to a Linux system without affecting any existing set-up.  I tried CUPS, but ultimately used LPD/LPR to get it working.

My previous post was for RHEL/CEntOS 5.  This update relates to RHEL/CEntOS 6, and can be done from the Gnome Desktop.  These instructions assume that you already know the host IP you're using as your print server

In the following instructions, enter the root password if you are prompted for authentication:

1. Click System > Administration > Printing
2. Click the "New" button
3. Enter password if prompted
4. Click the "Do it later" button to change your firewall settings (you shouldn't need to change them)
5. Enter Root password if prompted
6. Expand "Network Printer" in the left tree
7. Select "LPD/LPR Host or Printer" in the left tree
8. In the right side, enter your information
• (in my case, Host: 192.168.x.x, Queue: Samsung_2010)
9. Click "Forward"
10. Select your printer make from the database and click "Forward"
11. Select your printer model from the left tree, and the driver from the right tree
12. Click "Forward"
13. Enter a unique name for the computer (line #1), a human-readable name (line #2) and a location (line #3)
14. Click "Apply"
15. When prompted, click "Yes" to print a test page

You should be set at this point!

SQLite Database Browser on CentOS6

Working through Michael Hartl's Ruby on Rails Tutorial, chapter 6 calls for the SQLite Database Browser.

Just wanted to put a note out there for anyone building this on CentOS6: make sure you check the version of qt that your qmake command runs against.

The below commands show how to check which version you're using, and run the correct one to build Sqlite Database Browser.

check installed qt versions
[user@dev-workstation sqlitebrowser]$ which qmake
/usr/lib64/qt-3.3/bin/qmake
[user@dev-workstation sqlitebrowser]$ ls /usr/lib64/qt*
/usr/lib64/qt-3.3:
bin  include  lib  mkspecs  phrasebooks  plugins  translations

/usr/lib64/qt4:
bin  mkspecs  phrasebooks  plugins  q3porting.xml

confirm qt4/bin/qmake is there
[user@dev-workstation sqlitebrowser]$ ls /usr/lib64/qt4/bin/qmake
/usr/lib64/qt4/bin/qmake

commands to run to build Sqlite Database Browser
[user@dev-workstation sqlitebrowser]$ /usr/lib64/qt4/bin/qmake
[user@dev-workstation sqlitebrowser]$ make

after building, move to your ~/bin folder and enjoy
[user@dev-workstation sqlitebrowser]$ cp sqlitebrowser ~/bin/
[user@dev-workstation sqlitebrowser]$ which sqlitebrowser
~/bin/sqlitebrowser
[user@dev-workstation sqlitebrowser]$ sqlitebrowser &
[user@dev-workstation sqlitebrowser]$

hope this helps. :)

Setting up Ruby on Rails dev environment

We are working on a ruby app for one of my CS classes,  and found that it was easier to set up a virutal linux system than it was to try to get ruby working on windows (mainly because we don't have any stick time on that, and spinning the VM was cheaper).

The principle behind our decision to do this was that it's easier to get help from the community when you stick with the main focus of the community.  See an app out there that is primarily linux, but you're a windows admin?  It's easier for you to get help by installing and learning linux than it is to try to make it work under windows, even if this is a possibility.  I've had to learn this the hard way at work, and it was much easier to move forward with the software when I finally bit the bullet and went all in on linux.

The argument against this goes something like: "we have experience in the company with 'blah' and this can be made to run on 'blah', so we'd like to leverage our existing expertise..." etc.  Unless you're looking to become the pioneers in making it work, you're better off just sticking with the primary environment the software was built in.

The place this really matters is smaller companies.  Most of you out there probably use windows for your day-to-day work, because it's familiar, but the software you'd like to use runs on linux, which you don't really know.  If you're trying to avoid the cost of having it hosted (read "managed") by another company, learn linux.

I'm interested to know what others think of this, if they have experiences that are similar to mine, or if you think I'm nuts.  (OK, you're right on that last part :-D ).

Here's what we did to get our ruby environment set up.  If anyone has feedback on these instructions, I'm all ears.  Have fun.

(it should be noted that the instructions below were used in conjunction with http://ruby.railstutorial.org/book.  The part on bitbucket and heroku were based on having already published an app per that tutorial).

installing CentOS 6.3 on virtual box
===========

install virtual box
    for windows:     http://download.virtualbox.org/virtualbox/4.2.6/VirtualBox-4.2.6-82870-Win.exe
    for mac:         http://download.virtualbox.org/virtualbox/4.2.6/VirtualBox-4.2.6-82870-OSX.dmg

setup VM virtual box config
    Create Virtual Machine
        click new

        name virtual system (ie: Centos6 x64)
        if not already selected, select "Linux" and "Red Hat 64 bit"
        click next

        set for 1024 MB RAM
        create a virtual hard drive now, click create
        VDI, click next
        Dynamically allocated, click next
        set for 16GB, click create
  
    click settings
  
    Configure Network Connections
        select "Network" in left tree
        set as follows:
            Adatper 1:
                Enabled
                Attached to: Bridged Adapter
                Name: (your wired ethernet connection)
            Adapter 2:
                Enabled
                Attached to: Bridged Adapter
                Name: (you wireless ethernet connection)

    Configure iso for install at boot
        click settings
        select "Storage" in left tree
        select CD drive ("Empty") under Storage Tree on right
        click CD icon at right of "IDE Secondary Master" drop down
        select "Choose a virtual CD/DVD disk file"
        find the CentOS6 DVD iso
      
    click ok to confirm VM settings

config/install CentOS OS
    start vm
    click in vm screen to capture input
    click enter to start install
  
    once CentOS 6 logo loads, you can set one of the following:
        right-CTRL+F = full screen
        right-CTRL+C = scaled
        (I find scaled set to full screen works the best)
      
    proceed with install:
        click next (3x)
        basic storage device, click next
        click "yes, discard any data" (this is for the virtual disk)
        click configure network (lower left)
            for each adapter:
                double-click on name
                select "connect automaticaly"
                click "Apply"
            click "close"
        change hostname if desired (ie: centos6-dev)
      
        time zone = America/Denver
        uncheck "System clock uses UTC"
        click next
      
        set root password, click next
  
        select "use all space", click next
        click "write changes to disk"
      
        select "Software Development Workstation"
        click next (this starts installing packages)
      
        let install run
  
        reboot
      
        click forward
        "yes, I agree", click forward
        create your user, click forward
        check "Synchronize date and time over the network", click forward (allow ntp to start)
        click ok when prompted by kdump
        click finish
      
To open firefox:
    click icon
  
To open Terminal:
    Applications > System Tools > Terminal

  
set up sudo access and harden ssh login
    $ su - root (enter root password when prompted)
    allow wheel access to root via sudo
        # visudo
        arrow down to the "wheel" group
        press [i]
        for the line with %wheel  ALL=(ALL)  ALL, delete the # sign at the beginning
        press [ESC][:][X][ENTER]
    give yourself wheel group membership
        # usermod -G wheel <username>
        confirm
        # id <username> wheel(10) should show up in the list of groups
    change #PermitRootLogin yes to PermitRootLogin without-password
        # vi /etc/ssh/sshd_config
        arrow down and over
        press [i]
        use arrows, backspace and keys to change line
        press [ESC][:][X][ENTER]
        # service sshd restart

    what this will do is allow you to log in and switch users as needed, and prevents someone from logging in as root unless they are using ssh keys
  
update the system
    as root:
        # yum -y update
    as your user
        $ sudo yum -y update
    reboot

      
Install Eclipse Juno:
    http://www.if-not-true-then-false.com/2010/linux-install-eclipse-on-fedora-centos-red-hat-rhel/
  
Install Eclipse plugins:
    eclipse marketplace
    aptana studio
    dynamic languages toolkit - Ruby Development tools
    dynamic languages toolkit - core framework
    eclipse EGit
    Eclipse JGit
    eclipse web developer tools
  
Import the project:
    download the repo from bitbucket
    (this can be done from the terminal view in the eclipse web perspective)
    $ cd ~/workspace
    $ git clone https://$user_name@bitbucket.org/$user_name/$app_name.git
    $ cd $app_name
    $ git checkout master

Import the project to eclipse with this project:
    choose ruby perspective
    window > show view > project explorer
    method 1 (may not work):
        expand local filesystem, browse /path/to/git/advising_app
        right-click on advising app, choose "Promote to project"
        open App Explorer, and you should see the project
    method 2 (known to work:
        right click in project explorer, select import
        select General > Existing Project into workspace
        browse to advising_app folder
        click check box by folder on left side
        click finish
      
To open a CLI terminal in Eclipse:
    From Web perspective, it will be in the bottom portion of your screen
    From Ruby (or any other perspective):
        Window menu > Show View > click Other > Studio > Terminal
  
    If you would like a terminal editor (same place as files), click the bigger icon in the view
  
Some of the next steps might need to be run from a system terminal rather than an eclipse terminal, I can't remember which

Install Ruby on Rails environment
    Install RVM:
        $ \curl -L https://get.rvm.io | bash -s stable
    Install Ruby, RubyGems, Rails, etc.
        see section 1.2.2 of http://ruby.railstutorial.org/book
      
Install Heroku toolbelt:
    $ wget -qO- https://toolbelt.heroku.com/install.sh | sh
    $ cd ~/workspace/advising_app
    $ heroku login
    enter your heroku account credentials when prompted

BitBucket: git familiar with Atlassian's repo service

I've recently started using a new hosted git provider: Atlassian's BitBucket.  While most everyone out there is probably using github, there were two things I noticed about Bitbucket that I liked:

1. They have made https work for checking in with git version 1.7.1 (default yum install git from CentOS 6.x)
2. They allow the option to have private repositories

While I am a huge supporter of open source software, until now, I wasn't aware of any public cloud options for having a private repo without setting up a billed account (other than installing and managing your own server).  Simply put, I'd rather not add yet another copy of whatever tutorial I'm working through to the web.  (I will however, write about it :) )  Not everything should be public.

I'm also interesting in hearing what you have to say about this.  Will you be migrating everything as soon as you can?  Will you be sticking with GitHub?  Some mix of the two?  What arguments could or should be considered for paying for this kind of service?  I do have to say that I'm glad there is another option, since competition leads to better products and services for those using them.