History / Status

• Complete - out of date

1. Overview

Building a Linux cluster is a time consumming and difficult process. There many ways of setting up a cluster. Each methode has its pluses and minuses.

The objective of this howto is to guide the reader on setting up a Mosix cluster with diskless nodes. The setup is based on K12ltsp Project. This should provide an easily scalable system.

1.1 About K12ltsp

K12ltsp was chosen for the cluster. Its a solid distribution for the beginner as well as the advanced user. It simplifies the cluster by installing LTSP during the server setup.

1.2 About Linux Terminal Server

Please see www.ltsp.org

1.3 About Mosix

Mosix is a patch to the linux kernel which allows a cluster of linux machines to act as one large computer. From a programming standpoint this allows the programmer to write software as if it is running on an SMP machine. Just fork and forget.

An example of what you can is as follows. Lets say you are rendering a 3D animation. The renderer we'll be using is povray. A script can be used to do the following.

1. Check to see how many nodes there are. 2. See how many cpus are in each node. 3. Calculate the total number of CPUs. 4. fork off a povray process for each CPU. 5. Wait for a process to end and fork off a new one if necessary. 6. take the individual files and make an avi file 7. encode the avi file to your favorite compression standard. 8. all done.

1.4 About Etherboot

[ to be written ]

2. Requirements

2.1 Software Requirements

The software you'll need is the following: - K12ltsp.iso 2.0.1 - Mosix 1.57 - MPI (optional) - PVM (optional) - Linux kernel 2.4.17 (from www.kernel.org)

2.2 Hardware Requirements

The following hardware guidelines should be followed. The hardware listed below are minimum requirements. The kernel setup later will require at least a pentium pro.

Server

There is a lot of I/O tasks it will handle. A dual processor system is recommended. This is the computer you should spend some money on.

- Pentium (pro, II, III, 4) class CPU (dual CPUs is recommended) - or celeron cpu - minimum 128M RAM (256M is recommended) - hard drive of at least 4Gig (SCSI perfered) - cdrom and floppy - video card - what you need depends on if you'll be using the server locallly or remotely. - 2 network cards, one must be 100base-t - sound card (nice)

Nodes

- Some type of intel CPU. at least a pentium pro class - 64Meg RAM (128Meg recommended) - floppy drive - 100base-t network card - video card (needed during troubleshooting) - keyboard mouse monitor (to use node as xterminal)

Other - Network switch 100mbit - cabling

I do not recommend using 100base-t hub. A switch provides full duplex operation. You need as much bandwidth to the server you can get.

3. Hardware Installation and configuration

3.1 Server

Assemble and configure your server hardware. Be sure you can successfully boot the linux CD. At this point you can go to the section on installing the software on the server. While the server is installing software you can build and configure the nodes.

3.2 Nodes

Assemble and configure your nodes. Be sure each node can boot from a dos floppy.

[ I haven't worked with PXE yet ]

3.3 Network

If you are doing custom cabling do that now. (installing linux can take some time :)

3.4 The Final hardware setup

Now that you have all these computers, where are you going to put them? The best setup for your hardware is storage racks. Did I mention that logging into a node is a fringe benni for my use. Most of you are setting the equipment up in a lab.

4. Software Installation and Configuration

This section will cover the installation of the software on the server and the nodes. The items that will take the most time are installing linux, updating the packages, compiling the 2.4.17 kernel, compiling the 2.4.17 kernel with mosix. Hopefully your are reading this section while building the nodes.

4.1 Server

The server is where most of the software installs will occure.

3.1.1 Installing K12ltps

Boot the CD. K12ltsp.org provides good instructions to guide you. If it does not automatically boot check your bios settings for boot devices. Agree to stuff that comes up.

I'm assuming that your hardware and software configuration are the defaults as recommended by K12ltsp.org

Finish the rest of the installation steps.

3.1.2 Booting for the first time

Boot you newly installed linux system. Be sure everything is working correctly. Set everything up the way you like it. Also make sure you can connect to the internet. This is required for package updating.

At this point check for the latest updates. Update all the installed packages except the kernel.

WARNING: KERNEL UPDATING FROM THE UPDATE MANAGER DOESN'T WORK. I DON'T CARE WHAT ANYBODY TELLS YOU. BESIDES WE'RE GOING TO MAKE OUR OWN KERNEL ANYWAY!.

Reboot your system to be sure everything went ok. You never know when an installed package is going to currupt something.

3.1.3 k12ltsp system checkout

Be sure that your nodes boot. Do not continue with the MOSIX install until your setup works.

4. MOSIX setup

4.1 Getting stuff together

Download the following files:

mosix 1.5.7 from www.mosix.org kernel 2.4.17 from www.kernel.org initrd_kit from www.ltsp.org

4.2 Install the software

Unpack the packages into the /usr/src/ directory. From the K12ltsp cd install the kernel sources rpm. This will give you the default RedHat kernel config file.

I like to unpack things in a temp directory. so.

> su > cd /usr/src > mkdir tmp

Copy the files you downloaded to /usr/src/tmp.

> cd /usr/src/tmp > tar -xzf linux_kernel-2.4.17.tar.gz > tar -xzf MOSIX-1.5.7.tar.gz > tar -xzf ltsp_initrd_kit-3.0.1-i386.tgz

If everything looks good than lets move unpacked stuff to /usr/src.

> mv MOSIX-1.5.7 /usr/src/ > mv ltsp_initrd_kit /usr/src/ > mv linux /usr/src/linux-2.4.17

Now we need to install a few more packages.

Insert the k12ltsp cd 2

> rpm -i /mnt/cdrom/RedHat/RPMS/kernel-sources-2.4.9-31.i386.rpm > rpm -i /mnt/cdrom/RedHat/RPMS/kernel-doc-2.4.9-31.i386.rpm

4.3 Bug fixes and cleanup

The following items need to edited or fixed.

type:

> chmod a+x /usr/src/MOSIX-1.5.7/inst/add_kernel_to_grub

The above script was not set to be executable

> mkdir /usr/local/man

This man directory doesn't exist

4.4 Installing mosix on the server

This is where the fun part begins :)

Fist we want to create a place to store our kernel configs.

> cd /usr/src > mkdir kernel-configs

Lets get a kernel config file for a starting point.

> cd /usr/src/linux-2.4.9-31/configs > cp kernel-2.4.9-i686-smp.config /usr/src/kernel-configs/kernel-2.4.17-smp.config

Copy our config file into the kernel directory

> cd /usr/src/ > cp kernel-configs/kernel-2.4.17-smp.config linux-2.4.17/.config

Lets get the MOSIX install going.

> cd /usr/src/MOSIX-1.5.7 > ./install.mosix

Accept all the defaults. When the kernel configurator comes up be sure to enable MOSIX, mfs, and dfsa. If you have compiled kernels before, get rid of the device support you don't need. Once you are done save the config file and exit. Now let the installer do its thing.

Lets setup the mosix.map file. Use your favorite editor and type in the following:

2. MOSIX map file

1 192.168.0.254 1 2 192.168.0.1 253 <<EOF>>

I like the server to be node 1 and the clients to be nodes 2 through 253. This is a bit overkill on the number of nodes but I wanted to keep it consistant with the distrobution setup.

when mosix finishes do the following

> cp /usr/src/linux-2.4.17/System.map /boot/System-2.4.17-mosix.map > mkinitrd /boot/initrd-2.4.17-mosix.img 2.4.17

Mosix is bad and clobbered the grub.conf file. So lets fix it.

> cp /etc/grub.conf /boot/grub/grub.conf > rm -f /etc/grub.conf > ln -sf /boot/grub/grub.conf /etc/grub.conf

Mosix didn't add the initrd entry so we have to.

> pico /boot/grub/grub.conf # or your favorite editor

Add the following line to the mosix configuration

	initrd /initrd-2.4.17-mosix.img

reboot the system.

Boot to your new mosix kernel. Test your server and make sure nothing got broken. Before continuing make sure your clients still boot.

4.5 Seting up mosix for the clients

First lets clean up the kernel directory. We're also remembering to save our config file :)

> cp /usr/src/linux-2.4.17/.config /usr/src/kernel-configs/mosix-2.4.17.config > cd /usr/src/linux-2.4.17 > make mrproper

Lets get our default ltsp config file

> cp /usr/src/ltsp_initrd_kit/config.2.4.9-ltsp-5 .config

Now on to compiling the kernel

> make xconfig

Enable the mosix stuff and save and exit.

Now we need to add extra version info to the kernel makefile

> pico Makefile

Change the EXTRAVERSION line to read:

EXTRAVERSION = ltsp

Save and exit. Remember to remove the extra version info when you are done compiling ltsp kernels

Now we compile the kernel

> make dep > make bzImage > make modules > make modulae_install

Lets save a copy of our config file.

> cp .config /usr/src/kernel-configs/mosix-ltsp-2.4.17.config

Now we need to setup the kernel for ltsp. LTSP provides a script for this.

> cd /usr/src/ltsp_initrd_kit > pico buildk

Edit this file. Go to the end of the file. Comment out the last prepare_kernel line. Edit the first one to read the following:

prepare_kernel /usr/src/linux-2.4.17 2.4.17ltsp

Save the file and type the following.

> ./buildk > cp /lib/modules/2.4.17ltsp /opt/ltsp/i386/lib/modules/

PXE NOTE: I have not worked with PXE yet.

Our kernel has now been installed. We need to edit our dhcpd.conf file.

> pico /etc/dhcpd.conf

Add the following line above the trick from Peter comment.

option host-name = concat( "ws" , binary-to-ascii( 10, 8, "", substring( reverse( 1, leased-address), 0, 1)));

Mosix needs the hostname set on each client. DHCPD does not pass the hostname when you set up everything you're supposed to. Now edit the filename parameter to point to the new kernel.

filename "/lts/vmlinux-2.4.17ltsp";

Now save and quit. Lets restart dhcpd

> service dhcpd restart

Mosix isn't completely setup at this point but we should be sure our new kernel boots. At this point boot a client and make sure everything is working ok. If something goes wrong than you'll prob have to fiddle with the kernel config options and build a new kernel.

Everything worked! GREAT! The hard part is over. Now we just edit and copy a few files :)

Fist lets copy the user programs into the ltsp directory tree. Type:

> cp /sbin/setpe /opt/ltsp/i386/sbin/ > cp /sbin/tune /opt/ltsp/i386/sbin/ > cp /bin/mosrun /opt/ltsp/i386/bin/ > cp /usr/bin/mon /opt/ltsp/i386/usr/bin/ > cp /usr/bin/mosctl /opt/ltsp/i386/usr/bin/ > cp /usr/bin/migrate /opt/ltsp/i386/usr/bin/ > cp /bin/touch /opt/ltsp/i386/bin/

Copy our mosix.map file to ltsp. Remember to edit both files if you make changes.

> cp /etc/mosix.map /opt/ltsp/i386/etc/

Copy the hosts file. The one ltsp generates won't work with mosix.

> rm /opt/ltsp/i386/etc/hosts > cp /etc/hosts /opt/ltsp/i386/etc/

Now for the mosix startup script

> cp /etc/rc.d/init.d/mosix /opt/ltsp/i386/etc/rc.mosix

We need a mfs mount point. so:

> mkdir /opt/ltsp/i386/mfs

Now to edit some files.

> pico /opt/ltsp/i386/etc/fstab

Add the following line

none /mfs mfs dfsa=1 0 0

Save and exit.

> pico /opt/ltsp/i386/stc/rc.local

At the end of the file add the following lines

1. mosix startup section
2. we don't want any terminal processes to migrate

echo 1 > /proc/mosix/admin/lstay

1. start mosix

/etc/rc.mosix start

1. mount mfs filesystem. doesn't work when done earlier

mount /mfs

1. end mosix startup

Save and exit

We are done! OK now boot a couple of clients. Type:

> mon

Look for your nodes to show up in the monitor.

Enjoy your new cluster.

5. Testing and Checkout

5.1 Using seti@home to test the cluster

I use seti@home for cluster testing. Its very cpu intensive. But at times it does I/O which requires it to be migrated back to the server.