Installing MantisBT

from a fresh CentOS 6 install (don't forget yum -y update and a reboot), run the following commands:

1. cd /tmp
2. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz/download
3. wget http://sourceforge.net/projects/mantisbt/files/mantis-stable/1.2.11/mantisbt-1.2.11.tar.gz.digests/download
4. md5sum -c mantisbt-1.2.11.tar.gz.digests
5. tar -xzf mantisbt-1.2.11.tar.gz
6. mv mantisbt-1.2.11/ /opt/mantisbt-1.2.11
7. chown -R apache:apache /opt/mantisbt-1.2.11
8. ls -n /opt/mantisbt-1.2.11/ /var/www/html/mantisbt
9. yum -y install httpd mysql mysql-server php php-mysql
10. vi /etc/php.ini
11. change line 946 to match:
• date.timezone = "America/New_York" (should match cat /etc/sysconfig/clock output)
12. chkconfig httpd on
13. chkconfig mysqld on
14. service httpd start
15. service mysqld start
16. mysqladmin -u root password 'newPassword'
17. mysqladmin -u root -p -h localhost.localdomain password 'newPassword' (enter newPassword from above when prompted)
18. iptables -I INPUT 5 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
19. iptables -I INPUT 6 -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
20. now open your browser, and point it to:
• http://yourServerOrIPHere/mantisbt/admin/install.php
21. add newPassword from above in the password field
22. click the "Install" button.

Total time with Cobbler from PXE boot to creating your first project: ~20 - 25 mins.  This includes the time to install the OS from scratch, update it, reboot, and go through the commands above manually.  For those interested in specs, this was run on my setup.

Once you're done here, you'll want to start Configuring MantisBT.

Total score for ease of install: 8/10

Cobbler DHCP config

I'm working on installing Cobbler so that I have a reliable way to automate the install of my linux systems.  Following my first post will get the software installed, so now I was ready to move on to actually getting a PXE boot to install from Cobbler.

Looked up the directions here, and the steps are pretty straight forward.  If you're continuing from my previous post, it would look something like this:

1. insert your dvd with the distro
2. yum -y install dhcp
3. mount /dev/dvd /media
4. cobbler import --path=/media --name=CentOS6 (this takes a while, press [CTRL+Z], then run bg [ENTER] to send it to the background)
5. vi /etc/cobbler/dhcp.template
6. edit per your network, mine looked like this:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

7. service cobblerd restart
8. cobbler sync

All is well, we're now ready to PXE boot clients, and Cobbler will..... what's this?

running: dhcpd -t -q
received on stdout:
received on stderr:
dhcpd -t failed
Exception occured: <class 'cobbler.cexceptions.CX'>
Exception value: 'cobbler trigger failed: cobbler.modules.sync_post_restart_services'
Exception Info:
  File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run
    rc = self._run(self)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 184, in runner
    return self.remote.api.sync(self.options.get("verbose",False),logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 701, in sync
    return sync.run()
   File "/usr/lib/python2.6/site-packages/cobbler/action_sync.py", line 155, in run
    utils.run_triggers(self.api, None, "/var/lib/cobbler/triggers/sync/post/*", logger=self.logger)
   File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 918, in run_triggers
    raise CX("cobbler trigger failed: %s" % m.__name__)

!!! TASK FAILED !!!

hmm.. well, we can see that there is an issue with dhcpd, lets try to start the service on it's own:

[root@OPS8-Cobbler ~]# service dhcpd start
Starting dhcpd:                                            [FAILED]
[root@OPS8-Cobbler ~]#

Here's the log from /var/log/messages while trying to start the service above:

Jul  2 17:38:48 OPS8-Cobbler dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: All rights reserved.
Jul  2 17:38:48 OPS8-Cobbler dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Jul  2 17:38:48 OPS8-Cobbler dhcpd: /etc/dhcp/dhcpd.conf line 19: semicolon expected.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:      option domain-name-servers 68.87.85.102 68.
Jul  2 17:38:48 OPS8-Cobbler dhcpd:                                                ^
Jul  2 17:38:48 OPS8-Cobbler dhcpd: Configuration file errors encountered -- exiting
Jul  2 17:38:48 OPS8-Cobbler dhcpd:

Looking at /etc/dhcp/dhcpd.conf, I realized that I'd left a comma out of the template config.  It should look like the following in /etc/cobbler/dhcp.template:

subnet 10.1.2.0 netmask 255.255.255.0 {
     option routers             10.1.2.1;
     option domain-name-servers 68.87.85.102, 68.87.69.150;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.1.2.100 10.1.2.254;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                10.1.2.26;
}

Now we restart the service and run a sync....

[root@OPS8-Cobbler ~]# service cobblerd restart

Stopping cobbler daemon:                                   [  OK  ]

Starting cobbler daemon:                                   [  OK  ]

[root@OPS8-Cobbler ~]# cobbler sync

task started: 2012-07-02_174015_sync

task started (id=Sync, time=Mon Jul  2 17:40:15 2012)

...

rendering DHCP files

generating /etc/dhcp/dhcpd.conf

...

running: dhcpd -t -q

received on stdout:

received on stderr:

running: service dhcpd restart

received on stdout: Starting dhcpd: [  OK  ]

...

*** TASK COMPLETE ***

[root@OPS8-Cobbler ~]#

much better.

A simple PXE test confirms that the service is working and providing addresses successfully.  However, I wasn't able to get the PXE menu to load on a network boot...  turns out that I didn't have the proper Firewall settings.  You can test by stopping your firewall (service iptables stop) and trying again; watch out for tcp/udp differences!!  After using the information under Firewall on this page  I was able to get the PXE boot menu to load and test the install.

Cobbler Install on CentOS 6.2

Cobbler - not the kind you put peaches in, this is an automated install tool

Here's the quick and dirty to get it installed and the web interface working:

1. CentOS 6.2 install
• Basic Server install option
• as root, run "setenable 0" to turn selinux to permissive (without this, selinux caused me many headaches with the "cobbler check" command later)
• as root, run "vi /etc/selinux/config" and change the SELINUX=enforcing to SELINUX=permissive.  This keeps it in permissive mode over reboots.
• optional: set up a local user with wheel access, enable wheel sudo access, and set /etc/ssh/sshd_config with "PermitLocalRootLogin without-password"
2. add EPEL repo
1. point browser to: http://fedoraproject.org/wiki/EPEL
2. right-click, copy link
3. on CentOS system (I connect through putty and change to root at this point), run
• rpm -ivh <SHIFT+INSERT> (last two keys will paste the link from step 2)
3. Install Cobbler
1. yum -y install cobbler cobbler-web koan policycoreutils-python
2. service cobblerd start
3. service httpd start
4. cobbler check
1. resolve all reported issues (I had about 10)
4. Configure Cobbler-Web
1. see cobbler-web wiki page, just remember to try http if https fails

I think you might be able to skip step 3.4 and do that after step 4 if you'd like to have the web gui, since it is available there, but I don't know if you can resolve all the issues from there.

Kudos to Mike DeHaan for a really helpful config checker; wish all software came with something like that.

iSCSI Performance, round 2

So after turning on Jumbo frames (see my last post about this), I was able to get wonderful speed through the network, but I was having an issue with the storage server at this point; load averages were too high, and none of the RAM on the box was being used for caching.

In reading through the OpenFiler forums, I'd seen people referring to using iSCSI (a blockIO type technology) with fileIO transfer mode.  This didn't make sense to me, but I decided to try it with a new storage system I'd brought online.

I'd already mapped the LUN on the new system in the same was as the old system: iSCSI, write-back, blockIO.  Since there wasn't anything riding on this one, I just unmapped the LUN, and remapped it with write-back/ fileIO.  VMware didn't bat an eyelash at it (I didn't take the iSCSI service offline) and was able to browse the datastore just fine.  I then tested an fresh install of a system, since this is highly IO intensive.

Needless to say, I was very surprised to see the performance improvement.  Read and write latencies are now in the single digits, and I had a sustained network transfer during the install of 233Mbps, or 23.3% of my 1GbE connection (info based on VMware's performance reporting).  I also saw the memory on the OpenFiler system being used for caching, which was another win.

I immediately shut down my other 9 VMs and flipped my other system to fileIO tranfer mode.  There was no data loss (again, VMware didn't even notice the change), and I brought up the systems, first two at the same time, and then all the rest at the same time.  Latencies stayed in the single digits during the boot, and everything came up as if it was on dedicated hardware.

Also, the load averages on the OpenFiler system had dropped back to where they were before, but I noticed another problem...  the cache was using all the RAM on the box.

My OpenFiler systems are DELL 2850s, and when I bought them, I'd only gotten them with 2GB of RAM each.  Needless to say, I'm shopping for RAM right now =D.

(ps: I'm using BBU on the PERC cards in the Dells, and I have all my systems on a UPS as well).

So there you have it: iSCSI can be done cheaply and perform well enough to run your virtual infrastructure.  In this case, I'm currently running 10 VMs on a DELL 2850 and a DELL 1950, and total cost to me to set this up was under $2K.  More to come once I have more RAM =D

Ansible setup

Ansible - def.  1. super-luminal (aka, faster than light)
                        2. system managment automation program on github you wished you were running

Ansible is set up to be very simple, and runs over ssh.  Here are my notes from trying to get it installed and working on Centos 6.2, using the "Running from Checkout" instructions found at http://ansible.github.com/gettingstarted.html, which gets you version 0.5.  The RPM from EPEL provides version 0.3.

here's my super quick instructions, the few issues I ran into mentioned below:

1. start with CEntOS 6.2
2. sudo su - root or su - root
3. install needed packages
1. # rpm -ivh http://mirror.pnl.gov/epel/6/i386/epel-release-6-7.noarch.rpm
2. # yum -y install python PyYAML python-jinja2 python-paramiko
3. # exit
4. add ansible
1. $ git clone git://github.com/ansible/ansible.git
2. $ cd ./ansible 
3. $ source ./hacking/env-setup
5. configure hosts
1. $ echo "127.0.0.1" > ~/ansible_hosts 
2. $ export ANSIBLE_HOSTS=~/ansible_hosts
6. and test:
• $ ansible all -m ping -u dewey.garwood
  127.0.0.1 | success >> {
      "ping": "pong"
  }

you should note the following errors will occur if you aren't paying attention:

• if you go looking for paramiko, yum wont find it; you have to use python-paramiko
• without the -u option in the test command (step 6), ansible tries to use the root user to log in and you end up with:
• $ ansible all -m ping --ask-pass
  SSH password:
  127.0.0.1 | FAILED => FAILED: Authentication failed.

Making X work on RHEL/CEntOS 5 after VMWare P2V Import

Used the VMWare Standalone Converter running on my local machine to import a RHEL/CEntOS 5 Linux system, and afterward, was greeted with the following (this is cli, the gui had it's own errors):

-----snip-----
[root@qa01 ~]# startx
xauth:  creating new authority file /root/.serverauth.9175
xauth:  creating new authority file /root/.Xauthority
xauth:  creating new authority file /root/.Xauthority


X Window System Version 7.1.1
Release Date: 12 May 2006
X Protocol Version 11, Revision 0, Release 7.1.1
Build Operating System: Linux 2.6.18-164.6.1.el5 x86_64 Red Hat, Inc.
Current Operating System: Linux qa01.localdomain 2.6.18-164.11.1.el5 #1 SMP Wed Jan 6 13:26:04 EST 2010 x86_64
Build Date: 16 November 2009
Build ID: xorg-x11-server 1.1.1-48.67.el5_4.1
        Before reporting problems, check http://wiki.x.org
        to make sure that you have the latest version.
Module Loader present
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Fri May 20 12:28:27 2011
(==) Using config file: "/etc/X11/xorg.conf"
(EE) No devices detected.

Fatal server error:
no screens found
XIO:  fatal IO error 104 (Connection reset by peer) on X server ":0.0"
      after 0 requests (0 known processed) with 0 events remaining.
[root@qa01 ~]#
-----snip-----


Some digging online led me to http://www.vmware.com/pdf/osp_install_guide.pdf

I realize it probably makes more sense to add the yum repo, but since I was in a hurry, I just pulled the files manually and did a local install.  The files I needed are listed below:

Files are located at: http://packages.vmware.com/tools/esx/4.1/rhel5/x86_64/

IMPORTANT!!! BUILD NUMBERS ARE CRITICAL!!!
      Make sure you get the build number for your version of VMWare and Guest OS

   vmware-tools-nox-8.3.2-257589.el5.x86_64.rpm

   vmware-tools-8.3.2-257589.el5.x86_64.rpm
   vmware-tools-common-8.3.2-257589.el5.x86_64.rpm

   vmware-open-vm-tools-8.3.2-257589.el5.x86_64.rpm
   vmware-open-vm-tools-common-8.3.2-257589.el5.x86_64.rpm
   vmware-open-vm-tools-nox-8.3.2-257589.el5.x86_64.rpm
   vmware-open-vm-tools-xorg-utilities-8.3.2-257589.el5.x86_64.rpm
   vmware-open-vm-tools-kmod-8.3.2-257589.el5.x86_64.rpm
   vmware-open-vm-tools-xorg-drv-mouse-12.6.4.0-0.257589.el5.x86_64.rpm
   vmware-open-vm-tools-xorg-drv-display-10.16.7.0-0.257589.el5.x86_64.rpm

Commands as follows:

wget http://packages.vmware.com/tools/esx/4.1/rhel5/x86_64/<package_name>
    (yes, this has to be done for each rpm)

wget http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

rpm --import VMWARE-PACKAGING-GPG-RSA-KEY.pub (this saves you from the --no-gpg-check)

yum localinstall <big_list_of_all_rpms_use_tab_complete>


So, now that you're done with all that, it's time to reboot.  Yes, this is needed (remember that kmod rpm you just installed?).  After the reboot, everything works with X just fine.

And now things work virtually like they did before.  (=

Setting up OpenFiler 2.3 for VMWare ESXi storage

so I'm looking to set up openfiler as a storage backend for VMWare ESXi, just wanted to include some notes here, since the examples I've found in the forums isn't what I wanted to set up (disk sizes were fairly small).

So, main thing to remember is that OpenFiler doesn't handle the automated partitioning of the hard drive you're installing it on.  Best info I've found on this (without purchasing the Manual) is on Greg Porter's Wiki.  He has much excellent info on Openfiler, and I would highly recommend reading through his info.  This list here is just intended to be a quick checklist of things that need to be done to get this set up.

really rough run-down:

1. Get disk set up in RAID 5 on Dell 2850 (or hardware of your choice), preferably in hardware RAID rather than software RAID.
2. Boot from Openfiler 2.3 install disk (x86_64 in this case.... what, you haven't downloaded it already? (=  )
3. Using Graphical install, manually partition the disk as follows:
       /boot : 100MB ext3, Fixed size, Force Primary
       / (root): 2048MB ext3, Fixed size, Force Primary
      swap : 2048MB swap, Fixed size, Force Primary
4. Finish install (time, root password, etc.)
5. update, update, update (I've had some issues with this from the webgui; however, per the last entry of this forum, using "conary updateall" from the command line worked just fine.)
6. see Greg Porter's Wiki about the iSCSI reboot issue to prevent your stores from vanishing from the network upon reboot.  Last thing you want is for your system to boot and your services to fail.

At this point, you should be able to log in to OpenFiler, configure your shares, and get rolling.

DRAC4 Password Reset

So, building out a system for my work, and need DRACs for Out-of-Band-Management.  Found some Dell Poweredge systems on ebay for cheap (2850's and 6850's) and tried to access the DRAC cards with the default login....

didn't work.  )=

Looking for information on the web has taken most of the last week, since Dell's docs don't make it clear where the tools you need are, and what needs to be installed for them to work.

Goal: have Out-Of-Band Management on a storage system (Dell 2850 running OpenFiler).

I'm eventually going to run Openfiler on this system, but have installed CentOS 5.5 x86_64 to be able to complete this reset


Someone else ran into the same problem (used Dell's tools on windows), and the command that eventually resolved this for them was RACADM.  (see forum here).

in following the link on that forum, the CD that installed was not the correct one (the CD they list is for the initial install, I was looking for something I could install on an existing system).

So, here's the route that I went (racadm tools on CEntOS/RHEL 5.X):
(for 2850, this has been used to reset a DRAC 4/I)

Get Dell's RACADM installed on the system (instructions here)

Now, you can reset the card and be on your merry way:
      racadm racresetcfg (resets the DRAC)
      watch -n 60 racadm getsysinfo (lets you know when the DRAC has finished the reset)
      racadm setniccfg -s <rac ip addr> <netmask> <gateway>

Open a browser, point it at http://<RAC ip addr>, accept the certificate, and log in with the dell DRAC default login of username: root, password: calvin

I should also mention that to get the DRAC reset, these commands need to be run locally on the system, not over the network (however, you can connect over the network after the reset has been performed).

Hopefully this will save some folks headaches related to getting into the DRAC cards they have.  Overall process should take about 30-60 mins.

Installing RACADM for Dell DRAC 4/I on RHEL/CEntOS

Had several places this has become useful.  Follow the steps below to get Dell's RACADM tools installed on a RHEL/CEntOS 5.x system

UPDATE(2011.06.25) - use the LIVE CD when doing this, and you don't have to install CEntOS to fix this (ie: fix a system that already has something installed on it).  See notes at the end for additional commands.

Install CEntOS5.x Server and Server GUI from CD/DVD

install firefox with yum:
       yum install firefox
download the Dell OpenManage Deployment Toolkit:
       wget http://ftp.us.dell.com/sysman/dtk_3.5_new_43_Linux.iso

create a directory, mount the iso there and cd to that directory:
      mkdir /mnt/dtk_3.5_new_43_Linux
      mount -o loop -t iso9660 /path/to/dtk_3.5_new_43_Linux.iso /mnt/dtk_3.5._new_43_Linux/

      cd /mnt/dtk_3.5_new_43_Linux/
from here, run the following command to install the racadm tools:
       yum --nogpgcheck localinstall RPMs/x86/smbios-utils-bin-2.2.26-3.1.el5.i386.rpm RPMs/noarch/srvadmin-omilcore-6.5.0-1.385.1.el5.noarch.rpm RPMs/x86/srvadmin-racsvc-6.5.0-1.154.1.el5.i386.rpm RPMs/x86/libsmbios-2.2.26-3.1.el5.i386.rpm RPMs/x86/srvadmin-racadm4-6.5.0-1.154.1.el5.i386.rpm

UPDATE(2011.06.25) -  run the following commands to reset the DRAC without installing the OS

service racsvc start
locate racadm (it's in /opt/dell/something/i/dont/remember)
/opt/dell/rest/of/path/racadm racresetcfg

even though this will scream that it can't access the card, the card should be reset when you reboot

Setting up OTRS on CEntOS 5.5

OTRS (Open Ticket Request System) is a great open source ticketing system with a pretty clean interface, written entirely in Perl.  Below are some notes from setting this up on Centos 5.5, see the website above for full install instructions.

some things to remember:

run /opt/otrs/bin/otrs.checkModules to verify that everything is installed correctly, RPMForge yum repo can help with Perl packages

use generic agent to automagically move tickets/delete tickets.  This works great for deleting stuff in the junk folder.

set up 2.4.9, not 3.0 (the interface was significantly changed in 3.0, not used to it yet.  I think there was another reason for this as well, but I can't remember this right now).

remember to set up mysqld and httpd with chkconfig --levels 2345 <daemon> on

Setting up Mach build environment

So, clearly taking me a bit longer to get this going than I thought it would.  So here's the first tidbit I can give pretty quickly: I'm working on setting up a build environment using mach (stands for Make A CHroot).  This is a nifty tool to work in a clean build environment, and can be found here.  Basically, it will pull the packages you need to be able to get a clean build for whatever software you're working with.  If you are missing a dependancy, it will pull it automagically, if it can be found in the repos.  It also allows you to be able to build software for any version that is supported (ls /etc/mach/dists.d/ to get an idea).

Anyway, I'm just going through setting this up again, on a Centos 5 system that I'm planning on using for builds, and wanted to make a few notes, since the documentation still makes you dig a bit:

Steps followed so far:

1. set up yum for epel repo
2. run: yum install mach
3. add build user(s) to the mach group (ie: usermod -G mach <username>)
4. run: mach -r <root> setup build
5. go check irc or something (approx 5 mins)

remember that the root name needs to have an extension after the architechture, since that is what it is looking for in the /etc/mach/dists.d/ config files (separate file for each major os version).  For instance, the base centos 5 root that I just created wouldn't work as "centos-5-i386", it had to be "centos-5-i386-os".

Also, something else that should be mentioned is that it doesn't look like this package currently is able to perform builds for ubuntu/debian packages, but it does support apt-get, so maybe a config file could be created?