Customizing for server use - a bad idea?

[Capricorny]

I have been impressed with the performance of 64 bits 6.4.4, and running it on a fast machine with adequate hardware setup, I think it could make a good server for development/testing purposes. In such cases, it is not all-important to mimic the production situation extremely well, but it should be close enough to estimate actual performance in common use situations and identify weak points, bottlenecks etc.

As a concrete example, I think about running remastered Knoppix in 64-bits mode on a machine with core i7-2630QM processor, 16GB of RAM and a poor man's install on a 115 GB SSD disk. One of the reasons to do it this way, is to simplify making variations on the server setup, which can be quite a lot of work. Having it on a USB stick, we can easily try it out on new hardware configurations, share among developers etc.

I started out using Scientific Linux 6 (a RedHat Enterprise Linux clone) for the actual project, but found out that it took too much administrative work to set up the development system - we would, in fact set up something that could be more or less directly transferred into production use.

Are there, for example 64-bits problems/complexities involved that make this a bad idea?

[Capricorny]

To update: So far, everything looks good. Using USB3 and external SSD disk is "fast enough" for many purposes, and when needed, the whole setup can easily be transferred. I have also found that even with USB2, the performance is good enough at least for testing when running with a generous amount of RAM.

On a HP notebook, a kernel bug keeps cropping up keeping one of the cores completely busy, but the remaining three cores are able to run services nonetheless. As this is not something to live with in a testing situation, it is very practical just to hook up the SSD disk to another machine not suffering from the bug.

One uncertain thing right now, is the best Java version to use. Knoppix comes with OpenJDK


root@Microknoppix:/store/KNOPPIX # java -version
java version "1.6.0_18" OpenJDK Runtime Environment (IcedTea6 1.8.7) (6b18-1.8.7-2~squeeze1)
OpenJDK Server VM (build 14.0-b16, mixed mode)


I think I will give it a try, as it seems to be compatible http://www.jboss.com/products/platforms/application/supportedconfigurations/

But my synaptic can't tell me of any jBoss 5 .deb-package yet, so I think I must use a tarball for that.

[klaus2008]

But my synaptic can't tell me of any jBoss 5 .deb-package yet, so I think I must use a tarball for that.Look at http://wiki.debian.org/JBossPackaging Maybe you can use an already existing package.

[Capricorny]

Look at http://wiki.debian.org/JBossPackaging Maybe you can use an already existing package.
I have looked at it before. They don't seem to work with JBoss 6 yet, and I have jBoss 6 final working right out of the box, so it really doesn't matter much.
jBoss is a moving target, so it is not easy to create something remotely stable (over release versions) from it.

[Capricorny]

Somewhat unexpectedly, I am already through with all major program installations, and even without the bzip step, the compressed isofs is below 4GB. Dropping bzipping, compressing isofs doesn't take very long, so there is not really any pressing need for using squashfs. But, squashfs is in many ways a simpler approach. The squashfs image is some 200 MB smaller than the cloop variant.

The purge list is, so far, not very long. I used dpkg-query with sorting to isolate the biggest resource hogs, and didn't have to throw out any important tools to get enough space.



26956 frozen-bubble-data
28712 kde-l10n-it
30612 etoys
31156 extremetuxracer-data
33108 enigma-data
33224 freepats
35984 qt4-demos
38520 kde-l10n-fr
39668 kde-l10n-de
46912 kdewallpapers
52616 kde-l10n-es
60760 pinyin-database
62844 openvas-plugins-dfsg
65260 tuxpaint-stamps-default
86044 neverball-data
98064 gcompris-data
120892 wesnoth-1.8-data
136684 wesnoth-1.8-music
203404 crossfire-maps


apt-get remove `dpkg-query -l | grep i18n | grep kde | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep wesnoth | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep crossfire | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep gcompris | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep neverball | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep kde-l10n | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep kdewallpapers | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep openvas | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep pinyin | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep extremetuxracer | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep tuxpaint | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep frozen-bubble | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep etoys | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep enigma | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep oxygen | cut -d' ' -f3`
apt-get remove --purge `dpkg-query -l | grep qt4-demos | cut -d' ' -f3`


The most important added packages are OracleXE, Vmware Workstation, Bibble, R w/several extras and jBoss 6. After deborphan and apt-get clean before remastering, only about 0.6 of 4 GB persistent store is used now, after the third remastering, so a fourth iteration will probably not be necessary. As a regexp, the process can be written

purge (upgrade install remaster)+

There has been few package/version conflicts, but most of the big extra packages for R require the full build process, and this has often been something of an hassle. This should be improved upon.

[Capricorny]

Seems like I may have to adjust.



purge (upgrade install remaster)+
Think it shall be


purge (purge? upgrade install remaster)+As after another remastering, to include Groovy, Grails and some more servers, the image is full. To enter new packages now, some have to go out. So there I am, with a very efficient packet ---package-wise. And it is useful - maybe a bit too much of a success.

Because the need for 64-bits programs has become really urgent. For example, R is far more efficient for number crunching on larger datasets than I had imagined from my small-sample use - but then I must use 64-bits versions. Which I will have to compile myself, as the install architecture is 32-bit.

But then goes the semi-automated package handling - so we may lose about as much as we win by using Knoppix. So, I don't think I will go for alternative zero:

0. Compile the critical packages 64-bits.

I see two viable alternatives, maybe not too bad any of them:

1. Using Debian's live tools in live-helper to create a live version of a 64-bits Debian install with the package selection from Knoppix. http://www.pendrivelinux.com/create-your-own-live-linux-cd-or-usb-distribution/
(http://www.pendrivelinux.com/create-your-own-live-linux-cd-or-usb-distribution/)
2. Assembling a pure (well, almost at least) 64-bits version of Knoppix.

[dinosoep]

what I (was) planning on doing is installing some stuff on other data.img's. They could be on another storage medium and can be used by mounting the other servers.
I also don't get why the image is full you say you have a 115gb hard disk, that can't possibly be filled all the way :p
you can also switch squashfs? thats compressed a tad bit better.

and I've heard some things about knoppix 64 bit? there is a topic about that somewhere

[Capricorny]

I have been using several loop-mounted uncompressed data images, but I try to get everything into one 4GB cloop image and one 4GB loop image. Then everything can be handled with FAT32 USB sticks etc, just like the non-remastered versions. Knoppix mounts several cloop images if they are available, so it's not really a fundamental problem to have the first filled up, but I think it is easier to stick with one.

I am also doing the squashfs compression now, but in the first place, I only use the cloop image. Changing one parameter at a time. 64-bits kernel runs very nicely for me, the problems with 64 bits is that applications must be built with 64 bits libraries to run as such.

[Capricorny]

I now have a pure 64-bits Knoppix version up and running, created as a hybrid of Knoppix6.4.4 w 2.6.37-64 kernel and Debian 6.0.1. It runs as a full HD version, and it will take some time, testing and installing of new programs before I (try to) remaster it to make a package. In the first version, changes from basic Knoppix will be kept at a minimum, thus postponing the planned transfer to squashfs. Comfortably fitting on a 8GB pendrive/SD-card, including a grub /boot partition, features:

Ca 3.5-4GB cloop image, most of DVD-Knoppix apart from games, some international support and duplicated utilities.
3 GB persistent store, filled up 0.5-1 GB initially. (Database files under /var etc).
Basics: SAMBA, sshd,vncserver etc
Full choice of databases - MySQL,PostgreSQL,SQLite,OracleXE-11g (beta)
Virtualization: Qemu, VirtualBox, VMware Player (Possibly also Workstation w/o license, if VMware permits)
Apache/PHP: Full LAMP stack
A range of compilers and development tools
Java: Eclipse,TomCat, jBoss, Groovy, Grails etc
Computing tools: WXMaxima, 64-bits R with many Debian packages, Octave
I'm happy to include other things on request. Preferably from Debian packages (eventually create a package) As there has been no response about cooperation so far, I think I will create a Wiki entry for the 64-bits version here, and detail methods, scripts used etc. So far, I haven't needed to compile anything at all, and I'm not sure about the need for things like SVN setup etc for this task.

Doing too much programming/compilation might in fact indicate wrong orientation, I think.

There will be no CD image - not worth it, too limited. Will try to find a place a DVD ISO for download, but that may be far off in time. Could also send pendrives by mail.

A simple way to test it, may be from a fast (USB3) pendrive. In fact, also uncompressed, the whole system will fit fine on a 16GB drive.