updated parallelknoppix

[mcreel]

There's a new version of parallelknoppix available. This is a remaster that will let you create a cluster for parallel computing in very little time. If you use this you should definitely update to this version.

Screenshot

More info

Changes:
* removed packages to get image size down (~460MB now)
* configuration is now XDIALOG based: click, click
* package updates (lam, octave, and a few others)
* has a working example of GNU Octave running in parallel, using MPITB

Happy T-day, Michael

[mcreel]

16-Dec-2004: new version based on Knoppix 3.7
* same functionality, but newer packages
* image size is somewhat larger (550MB) - too lazy to trim fat, but now you can play frozen-bubble!
* the script to copy to HD for remastering has been improved a bit

[A. Jorge Garcia]

Wow, Michael, I just looked at your site and find your remaster quite amazing! Now, let me get this straight. Your remaster is unlike clusterKNOPPIX and QUANTIAN as its not openMOSIX based. So, am I correct in assuming this is like a Beowulf implementation? Is that what the LAM-MPI and MPICH packages are designed for? Please excuse my noobishness (that's a word?), but the only clustering I've done is based on MOSIX.

I think I would like to investigate MPI programming some more, however. I'd like to give my java students a cool final project involving clustering an app that they themselves have written from scratch. I have not found java support for MOSIX, however. So my students have run existing apps like KANDEL and POVRAY but not written their own code to solve a problem of their own choosing. We did play with C and MPI once using the fork() method, but the learning curve was a bit much for my students.

I do see some research on the web regarding the use of java with MPI or PVM that looks intriguing. So I think I'll try out your distro some more. What is astounding about your remaster - asside from its unique approach in the KNOPPIX remaster world (ie: no MOSIX) - is that yours is the only one that has been updated to a KNOPPIX 3.7 base! I don't know what the other remasters are waiting for, but they seem to be sitting around and not supporting their projects of late.

What do you think of all this? Do you think I could use your CD with java? What I'd like to do is install your system to hdd as a more standard linux rw fs and add java functionality to it. Have you used knoppix-installer with your system? BTW, I see that your ISO is down to about 500MB. So there should be plenty of room for adding to it if I want to do that too. Just curious, what did you remove? I see you removed all but en lang support, that must save a lot of space! I suppose you got rid of games too, is that correct? What else, if I may ask?

BTW, thanx Michael Creel, for all your work!

TIA,
AJG

[mcreel]

You're right, it facilitates greatly setting up a distributed cluster environment, and once that's done, LAM/MPI or MPICH can be used in the ordinary way. The openmosix approach used by ClusterKnoppix and Quantian is an alternative that would probably work just as well for "embarassingly parallelizable" problems, but I _suspect_ not as well for problems that have some serial parts interspersed with the parallelized parts. Maybe a comparison would be an interesting project.

About Java, I've never used it. But if you can use LAM/MPI or MPICH with Java, then I guess parallelknoppix would be useful.

Good luck, let me know if you would like packages added. Michael

By the way, there is a newer version than the one mentioned above. It includes the Ganglia monitor so you can access a web page to monitor the cluster, and it has some menu entries for some scripts for remastering. Still need to update the tutorial, though.

[A. Jorge Garcia]

Hi Michael,

Thanx for your response! You are correct, it is very difficult for my students to write code for an openMOSIX environment (as a 2-4 week final programming project after a year of learning java). However, the MPI approach looks promissing now that there's some java interfaces for MPI and PVM. BTW, do you include PVM on your ISO? PVM also looks very teachable.

Regarding jClusters, you may want to take a look at:

http://www.buyya.com
http://www.ipd.uka.de/JavaParty
http://freshmeat.net/projects/jcluster
http://www.srg.csis.hku.hk/homepage/srg/jessica.htm
http://www.npac.syr.edu/projects/pcrc/mpiJava
http://www.hpjava.org/
http://www.cs.virginia.edu/~ajf2j/jpvm.html

Oh, I did see your new website and I was referring to your latest version when I wrote the post above - that would be version 2005.0110, right?

However, I'm still a little vague on some points. There was no such thing as clusters when I was in college in the 70s! So I'm teaching myself all this stuff from scratch right now. I have a lot of catching up to do. This whole clustering field sounds so useful and interesting, I feel that I've only just scratched the surface!

Anyway, I'm not clear on whether this is a Beowulf cluster you have implemented and what that means to me and my programming students! Also, would there be any problem in my using knoppix-installer to make a hdd installed cluster. I would like a little more permanent solution than a CD cluster. Also, I cannot get PXE to work in my lab. This is how I got around the PXE problem for openMOSIX using clusterKNOPPIX or QUANTIAN....

TIA,
AJG

[mcreel]

Yes, PVM is on the ISO, but I have not tried it out. I do know that MPICH and LAM/MPI work properly.

The web page has the link to the latest version.

Exactly what makes a cluster a Beowulf, and whether the distinction is important or not, is not perfectly clear to me. But I think that parallelknoppix sets up a Beowulf-style cluster. I don't think that the issue is too important. Basically, parallelknoppix will set up a cluster to allow subsequent MPI work, using LAM, MPICH, and probably PVM.

The main characteristic of parallelknoppix is that it lets you set up a Linux cluster on IA32 machines, without any permanent alterations to them. I use it at my university to do cluster work in a computer room that is ordinarily used by students, running windows. Also I have a special version that lets me collect junky old computers and incorporate them into my personal cluster in my office without having to install anything on them. Makes it easy to maintain and expand the cluster.

For permanent multi-user clusters, there are better solutions out there. Parallelknoppix is not secure, so an untrusted or incompetent user could do damage to any computer in the cluster. That limits the multi-user attractiveness. For permanence for a single user, it's what I use.

M.

[A. Jorge Garcia]

Sorry, repeated post...

[A. Jorge Garcia]

OK, Michael, that's great! I think I'll try javaPVM first. When I have some time, I really want to play with your CD. I think I'm going to have to hdinstall it, however, as I can't get PXE to work. Security is not an issue. These machines use only local static IP addresses and are not accessible from off campus.

One of the reasons openMOSIX appealed to me is all the monitoring software that made it easy to see that the cluster was actually doing something! I saw on your website that Ganglia is now on your ISO, that's great too! Of course, lets not forget the auto load balancing for certain apps that can migrate over MOSIX as well as nodes being added automatically to the cluster....

BTW, I see you have a demo the uses OCTAVE. I'm not familiar with that application. I know its one of the QUANTIAN suite of apps along with R and some Stats and CAS software. Could you describe what OCTAVE does and what your demo should show?

Regards,
A. Jorge Garcia
http://calcpage.tripod.com

[UnderScore]

One of the reasons openMOSIX appealed to me is all the monitoring software that made it easy to see that the cluster was actually doing something! I saw on your website that Ganglia is now on your ISO, that's great too!

Maybe I have been out of the clustering loop too long but I figure I can contribute something helpful.

I setup & presented a Bewowulf cluster that used PVM for my SysAdmin2 class. I posted my slides online at my webpage http://www.virtualacuity.com/james/projects.html. In particular I used xpvm, a PVM monitoring tool for X, and it allowed me to see how the cluster was operating. In my slides, slide number 17 shows full utilization of the cluster across 5 nodes. The slides are now dated & old (this was before mosix was forked into OpenMosix) but the images are proof that it works.
I hope this helps.
James

[A. Jorge Garcia]

Hey, James, that was a really good presentation. Would you mind very much if I used in class? I was going to make some sort of powerpoint presentation on this, but you stole the cake!

Thanx,
AJG