openMosix General Instructions

Kernel Compilation

Always use pure vanilla kernel-sources from e.g. www.kernel.org to compile an openMosix kernel! Be sure to use the right openMosix version depending on the kernel-version. Do not use the kernel that comes with any Linux-distribution; it won't work.

Download the actual version of openMosix and untar it in your kernel-source directory (e.g. /usr/src/linux-2.4.16). If your kernel-source directory is other than "/usr/src/linux-[version_number]" at least the creation of a symbolic link to "/usr/src/linux-[version_number]" is required. Now apply the patch using the patch utility:

patch -Np1 < openMosix1.5.2moshe

This command displays now a list of patched files from the kernel-sources. Enable the openMosix-options in the kernel-configuration e.g.

...
CONFIG_MOSIX=y
# CONFIG_MOSIX_TOPOLOGY is not set
CONFIG_MOSIX_UDB=y
# CONFIG_MOSIX_DEBUG is not set
# CONFIG_MOSIX_CHEAT_MIGSELF is not set
CONFIG_MOSIX_WEEEEEEEEE=y
CONFIG_MOSIX_DIAG=y
CONFIG_MOSIX_SECUREPORTS=y
CONFIG_MOSIX_DISCLOSURE=3
CONFIG_QKERNEL_EXT=y
CONFIG_MOSIX_DFSA=y
CONFIG_MOSIX_FS=y
CONFIG_MOSIX_PIPE_EXCEPTIONS=y
CONFIG_QOS_JID=y
...

and compile it with:

make dep bzImage modules modules_install

After compilation install the new kernel with the openMosix options within you boot-loader e.g. insert an entry for the new kernel in /etc/lilo.conf and run lilo after that.

Reboot and your openMosix-cluster(node) is up!

hpc.map

Syntax of the /etc/hpc.map file Before starting openMosix there has to be a /etc/hpc.map configuration file (on each node) which must be equal on each node. The hpc.map contains three space separated fields:

openMosix-Node_ID               IP-Address(or hostname)          Range-size

An example hpc.map could look like this:

1       node1   1
2       node2   1
3       node3   1
4       node4   1

or

1       192.168.1.1     1
2       192.168.1.2     1
3       192.168.1.3     1
4       192.168.1.4     1

or with the help of the range-size these both examples are equal with:

1       192.168.1.1     4

openMosix "counts-up" the last byte of the ip-address of the node according to its openMosix-ID. (if you use a range-size greater than 1 you have to use ip-addresses instead of hostnames)

If a node has more than one network-interfaces it can be configured with the ALIAS option in the range-size field (which is equal to set the range-size to 0) e.g.

1       192.168.1.1     1
2       192.168.1.2     1
3       192.168.1.3     1
4       192.168.1.4     1
4       192.168.10.10   ALIAS

Here the node with the openMosix-ID 4 has two network-interfaces (192.168.1.4 + 192.168.10.10) which are both visible to openMosix.

Always be sure to run the same openMosix version AND configuration on each of your Cluster nodes!

Start openMosix with the "setpe" utility on each node : setpe -w -f /etc/hpc.map Execute this command (which will be described later on in this HOWTO) on every node in your openMosix cluster. Installation finished now, the cluster is up and running :)

MFS

At first the CONFIG_MOSIX_FS option in the kernel configuration has to be enabled. If the current kernel was compiled without this option recompilation with this option enabled is required. Also the UIDs and GUIDs in the cluster must be equivalent. The CONFIG_MOSIX_DFSA option in the kernel is optional but of course required if DFSA should be used. To mount MFS on the cluster there has to be an additional fstab-entry on each nodes /etc/fstab.

for DFSA enabled:

mfs_mnt         /mfs            mfs     dfsa=1          0 0

for DFSA disabled:

mfs_mnt          /mfs           mfs     dfsa=0          0 0

the syntax of this fstab-entry is:

[device_name]           [mount_point]   mfs     defaults        0 0

After mounting the /mfs mount-point on each node, each nodes file-system is accessible through the /mfs/[openMosix_ID]/ directories.

With the help of some symbolic links all cluster-nodes can access the same data e.g. /work on node1

on node2 :      ln -s /mfs/1/work /work
on node3 :      ln -s /mfs/1/work /work
on node3 :      ln -s /mfs/1/work /work
...

Now every node can read+write from and to /work !

The following special files are excluded from the MFS:

the /proc directory

special files which are not regular-files, directories or symbolic links e.g. /dev/hda1

Creating links like:

ln -s /mfs/1/mfs/1/usr         

or

ln -s /mfs/1/mfs/3/usr

is invalid.

The following system calls are supported without sending the migrated process (which executes this call on its home (remote) node) going back to its home node:

read, readv, write, writev, readahead, lseek, llseek, open, creat, close, dup, dup2, fcntl/fcntl64, getdents, getdents64, old_readdir, fsync, fdatasync, chdir, fchdir, getcwd, stat, stat64, newstat, lstat, lstat64, newlstat, fstat, fstat64, newfstat, access, truncate, truncate64, ftruncate, ftruncate64, chmod, chown, chown16, lchown, lchown16, fchmod, fchown, fchown16, utime, utimes, symlink, readlink, mkdir, rmdir, link, unlink, rename

Here are situations when system calls on DFSA mounted filesystems may not work:

different mfs/dfsa configuration on the cluster-nodes

dup2 if the second file-pointer is non-DFSA

chdir/fchdir if the parent dir is non-DFSA

pathnames that leave the DFSA-filesystem

when the process which executes the system-call is being traced

if there are pending requests for the process which executes the system-call