I am currently installing a cluster of old PIII boxes at BTH. Every machine has 4-20GB of harddisk and 512MB memory. Since I was tired of doing all the machine installation by myself (Debian, NFS, NIS, SGE, Ganglia, Java, …), I searched for a better ‘out-of-the-box’ solution. The Rocks cluster distribution provides everything I expected. You install a front node with all the software packages, and the dumb compute nodes install themself over PXE. The great thing is that all relevant cluster stuff is already integrated, so you get a full-fledged SGE+Ganglia+Globus Head+(their)NIS+MPI cluster in one day:

www.rocksclusters.org

Now the bad part: The documentation is lousy, like in all purely academical projects. My main problem was the age of the machines – the pure amount of software installed normally expects at least 1GB of RAM and 10GB of harddisk. Here is my set of experiences:

• Give the front node at least 1GB. With 512MB, you get an obscure VFS error message during frontend installation, since the ramdisk gets full. The compute nodes work fine with 512MB.
• insert-ethers is only needed for the first time the compute node is connected to the cluster. After the MAC address is registered, it will reinstall always from PXE. So you can have endless rounds to fix the problem with this particular compute node.
• If the compute nodes have too small hard disks, switch to manual partitioning. Rocks expects a root partition, a swap partition, and a partition mounted under /state/partition1. With SGE, Globus, Ganglia, Java, and HPC roll the root partition needs at least 3GB with Rocks 4.3.0. The activation of manual compute node partitioning is described here.
• New cluster users are created as described here.
  You need to consider that all compute nodes must be up and running to receive the update immediately.
• Resist the temptation to install everything from the Net. It takes ages, and in my case the SGE installation on the frontend was incomplete afterwards, which leaded to another round of frontend renewal (in case, look here). Burn the CD’s or the DVD.

We upgraded our SGE cluster machines to Debian testing release, which contains beside other things the latest 2.6 libc:

marie# dpkg -l|grep libc6
ii libc6 2.6.1-1+b1 GNU C Library: Shared libraries
ii libc6-i686 2.6.1-1+b1 GNU C Library: Shared libraries


Especially on the master node, this package version for libc is needed to install libmotif3 from the Debian repositories, which is a prerequisite for the qmon tool.

After upgrading everything, the SGE daemons refused to be started by the old /etc/init.d/ scripts, while the direct start of the binaries still worked. I figured out that the Sun shell scripts rely on the output from SGE_ROOT/util/arch, which returned the following on our updated installation:

UNSUPPORTED-lx24-GLIBC-2.6-x86

With bash -x arch, I got the following execution dump:


...
++ strings /lib/libc.so.6
++ grep 'GNU C Library'
+ libc_string='GNU C Library stable release version 2.6.1, by Roland McGrath et al.'
+ '[' 0 -ne 0 ']'
++ echo GNU C Library stable release version 2.6.1, by Roland McGrath et al.
++ tr ' ,' '\n'
++ grep '2\.'
++ cut -f 2 -d .
+ libc_version=6
+ case $libc_version in
+ unsupported=UNSUPPORTED-
+ lxrelease=24-GLIBC-2.6
+ ARCH=UNSUPPORTED-lx24-GLIBC-2.6-x86
...


I am not a shell script expert, but fetching the libc version by using strings on the binary looks a little bit weird.

Anyway, looking for the libc_version variable in the arch script brought up the following:


# verify the GNU C lib version
# For an alternative means to determine GNU C lib version see
# http://www.gnu.org/software/libc/FAQ.html#s-4.9
case $lxmachine in
amd64)
libc_string=`strings /lib64/libc.so.6 | grep "GNU C Library"`
;;
ia64)
libc_string=`strings /lib/libc.so.6.1 | grep "GNU C Library"`
;;
*)
libc_string=`strings /lib/libc.so.6 | grep "GNU C Library"`
;;
esac

# retrieving libc version failed
if [ $? -ne 0 ]; then
unsupported="UNSUPPORTED-"
lxrelease="${lxrelease}-GLIBC"
else
libc_version=`echo $libc_string | tr ' ,' '\n' | grep "2\." | cut -f 2 -d "."`
case $libc_version in
2)
unsupported="u"
;;
3|4|5)
;;
*)
unsupported="UNSUPPORTED-"
lxrelease=24-GLIBC-2.${libc_version}
esac
fi
;;


As you can see in the last part, all libc minor versions until 2.5 are considered to be known. I couldn’t find any problem statement for libc2.6 on the SGE web pages, so I added the missing “|6)”, and everything works now.