Full 64 bits Knoppix - a project for this site?

[Capricorny]

While running Knoppix with the 64 bits kernel version works very well for me, Knoppix is still, basically, a 32 bits system. And, according to Klaus K, it will probably stay that way for a while. Maintaining 4 (64/32*CD/DVD) versions instead of 4 would probably more than double his work (according to him, and I think he should know), and I don't think that is a sensible use of developer resources, even if he had been willing.

For some purposes, we may need "pure" 64 bits. Video/image editing, running virtualization software, doing data analysis on large data sets are some examples.
It turns out that creating a 64 bits version is probably much more of an administrative than a development task. If we make a minimum of modifications from standard Knoppix, we can mostly substitute Debian's 64-bits packages, and most of the Knoppix-specific software is architecture-independent scripts. So, it may go with just a few new compiles, and I tend to think that most of the work involved will be to set up some efficient administration scripts. Having assembled all the necessary packages, it should be mostly plain remastering sailing to create a new cloop/squashfs image. But one has to try it to tell.

While it might be tempting to do a lot with setup and package selection in this context, I would try to do as little as possible. For my own use, it would imply adding some more scientific programs, a bit like Dirk Eddenbuettel's Quantian, and remove quite a few games and little used programs, to free up space for program adding and remastering.

Also, such a version could have two cloop/squashfs images: One with Open Source programs, and one with non-free software. There is still support for this in the init process.

While a more experimental version could well be called "Geek-Knoppix", as suggested before, the 64 bits version might perhaps be called Server Knoppix, because server use will be an important application area.

I would like to stress that this is not any kind of fork or fundamentally new derivative. I think it should work exactly like standard Knoppix whenever possible, and that additions should not modify the ordinary ways of doing things. We might add a few scripts and other programs, and some new cheatcodes, but that should be all. In practice, a user could install both 32 and 64 bits versions on, for example, a USB harddisk, and then choose which to use at bootup, just like 32- and 64 bits kernel versions are used today.

I don't know exactly how much work this will be, but I imagine it won't have to be a very huge task,so a few volunteers ought to suffice.

Or am I completely mistaken?

[Capricorny]

Small update:
I'm now finished with the first round of Knoppix package selection installation on a Debian 6.0.1-64 system with LXDE/Openbox. The majority of packages could be found in repositories, but there will probably be quite a bit of compiling necessary here, too. Knoppix and std Debian differed more in the package selection than I would have expected. And I already miss the simple backup with Knoppix!

It may be a coincidence, but I have had quite a bit of hardware-related trouble with std Debian - on a machine that runs Knoppix without a glitch.
And things run a bit faster on 64-bits, though differences depend very much on tasks. I have noticed between 0 and 30% speedup so far.

[Capricorny]

Can report success so far.
Procedure:

1. Installed Debian 6.0.1 w/LXDE-openbox: A
2. HD-Installed Knoppix 6.4.4 32-bits kernel CD version: B
3. HD-Installed Knoppix 6.4.4 64-bits kernel remastered DVD version: C
4. Compared packages, installed 64-bits versions of all available packages installed in B on A
5. Set up a 18GB partition with Reiserfs: D
6. Transferred B-system to D by rsync
7. Updated D-system from A by rsync
8. Updated /boot on D from C: 64-bits kernel as vmlinuz
9. Updated D-system /etc, /lib/modules, /usr/src from C by rsync
10. Updated D-system /etc/alternatives from A by rsync. (To get the 64-bit library links right.) Added grub entry for D in common /boot, using basically copy of C-entry for D

So, essentially, this was carried out by a mix n' match of three OS installations, resulting in a fourth. If 64-bits support had not been broken in CD version of Knoppix 6.4.4, it might have been enough with two, but it was quite nice to be able to check with the full Knoppix version on HD. And starting with the CD version gave me fewer packages to install (or miss) in 64-bits versions.

There are a few minor editing details left out in the above list.

Here are the grub entries:

# entry created by 0wn

title KNOPPIX 64-kernel HD install DVD
root (hd0,7)
kernel /boot/vmlinuz root=/dev/sda8 rootwait lang=us apm=power-off nomce libata.force=noncq tz=localtime loglevel=1 ramdisk_size=100000 lang=en keyboard=no  nosound vt.default_utf8=0 apm=power-off initrd=minirt.gz nomce libata.force=noncq loglevel=1 tz=localtime rw

# ---

title KNOPPIX 64 FULL Devel vs 2
root (hd0,5)
kernel /boot/vmlinuz root=/dev/sda6 rootwait lang=us apm=power-off nomce libata.force=noncq tz=localtime loglevel=1 ramdisk_size=100000 lang=en keyboard=no  nosound vt.default_utf8=0 apm=power-off initrd=minirt.gz nomce libata.force=noncq loglevel=1 tz=localtime rw

As for the Knoppix vs Debian questions: No, Knoppix is not Debian. Knoppix is far, far better. Basic Debian is still stuck in the "being everything to everybody" sure-to-fail paradigm. That is one important reason why Ubuntu has strayed unneccesarily far from its parent.

Installed system size is now 5.8 GB, that is with full 64-bits R and VMware workstation and MySQL installed. Plus lots of unneccessary Debian 6.0.1 stuff - but it does no harm, I think. Typically, the 64-bits versions are somewhat bigger, I think the compressed kernels: 3.2 vs 2.9 MB, may be representative.

[Capricorny]

The system seems to work well in daily use, but I have of course not been able to test it thoroughly. Now I have installed almost all packages from the DVD version, except for the games category, and system size is now ca 13 GB, whereof ca 1GB in /var etc. With duplication of OpenOffice/LibreOffice and some packages left over from the Debian 6.0.1 install, "necessary" space for a DVD-like version with enhancements seems to be ca 12GB, I think that is a 10-20% increase over 32-bits version.

This means that it's probably hard to fit everything in one 4GB cloop image, but we may use as many such as we like, and everything will still fit on a 8GB pendrive.

Still lacking is jBoss and a few Java packages (too old versions in Debian repos), but I have installed Oracle XE11g beta. Converted from rpm with alien - seemed to work well, but haven't tested. The only major problem has been with postgresql, can't get it to install.

So, I'll soon try the first remastering now.

[klaus2008]

This means that it's probably hard to fit everything in one 4GB cloop image, but we may use as many such as we like, and everything will still fit on a 8GB pendrive.

I think that Knoppix 6.4.4 supports up to 10 cloop images (KNOPPIX or KNOPPIX0 ... KNOPPIX9 in the "$knoppix_dir"). But how do you decide which files are to be put in a certain cloop image? Does it matter in which order they are loaded by the init script?

The only major problem has been with postgresql, can't get it to install.

If you could explain more details someone might be able to help you a bit further.

[Capricorny]

I think that Knoppix 6.4.4 supports up to 10 cloop images (KNOPPIX or KNOPPIX0 ... KNOPPIX9 in the "$knoppix_dir"). But how do you decide which files are to be put in a certain cloop image? Does it matter in which order they are loaded by the init script?

If you could explain more details someone might be able to help you a bit further.

The cloop handling is automatic upon init, has been able to handle several cloops for a long time, just we haven't normally used it lately. As everything gets merged in /UNIONFS, I don't think it matters very much how we partition the loops. As the 4GB limit is just because of FAT32 (+ evt DVDs), I think I will start just creating one image, and proceed from there. Maybe the best candidate for a partition is to keep /usr/share by itself - currently that one is 6.2GB, almost half the whole system footprint. Second largest is /usr/lib, at 4.2GB. SO, one of those will have to be handled separately, I think. The only large installed program currently outside the package system is VMware, and it wouldn't help enough to place that on its own.

I didn't implicitly ask for help with postgresql, just noted there are problems with the latest 64-bits debian package(s). There are several ways to sort that out, but as I try to make this whole enterprise as easily reproducible as possible, I try to stick to the standard downloadable packages, latest versions, as much as possible. Postgresql is there mainly for completeness, as this is a server-oriented version, I don't presently use it myself.

What I could need advice about, it that .gvfs - it's kind of special Gnome directory, and even root has permission denied on it!!!

du: cannot access `/home/knoppix/.gvfs': Permission denied

What's going on here? I don't know the underlying fs. Reiserfs, at all, maybe a cleanup will help.

[Capricorny]

One way to divide packages, is to do it by categories, "Section" in Debian-speak. Doing it incrementally, one may add packages by category, and at one point where the compressed image attains a critical size, break off and compress it. Then add a large (say 6-10GB) persistent store, continue with installs, which then will be done to the persistent store, and at the end compress this as a KNOPPIX2 cloop image. Repeating the process as necessary.

This is very actual with the version I work on now, dividing into basics and sci/math. So that most people would live well with one image, the first, but the second is stuffed with tools. I have just reached ca 20 GB installed size, after adding a majority of 64-bits sci/math Debian packages, to test this out. With one image first, but then with two or more.

[Capricorny]

There are 4787 packages installed currently. So I think 5000 packages is kind uf upper practical limit for what is in any sense reasonable to pre-install in this context.
The compressed cloop image, non-optimized compression, is 7.8GB. Including almost 1GB /var.
Which means that we can easily divide into two <4GB images, where the second will be for the specially interested. (Virtualization, Oracle XE, math, science)
So, it seems, 64 bits doesn't mean that much increase in system size.

[Capricorny]

So far, I haven't been able to get through minirt.gz init with the 64-bits cloop images, neither a big (7.8 GB) or a small (2.4GB) one. Clearly, there is a problem in fs mounting, for init complains about not finding /bin/mount - when I can run mount perfectly well from the cloop image. This happened when it was created in the 64 bits version.

[Capricorny]

It seems that the best thing to do, remastering-wise, is to create two cloop images, one basic with programs similar to the standard DVD, and one with additions for specially interested. Would make two (at least) DVDs, but after copying the cloop images to writable media, they will both be mounted automatically on init if present.

The main hurdle now, is that even with minirt.gz updated with 64-bits busybox, it does not seem to be able to run programs off the UNIONFS mount when the cloop is created "purely" 64-bits. Running init in debug mode, I can manually aufs-mount UNIONFS, but I can't link to it and run programs from it from busybox. When I chroot into it, all seems to work well, but that's not the way minirt init is set up. I have tested both with small (2.4GB) and medium (4GB) cloops. No way. And it's not the format in itself: Creating the cloop with 32-bits system gave the same result, so the trouble may be kind of analogous to 32/64-bits ELF formats blues.

Maybe a kernel recompile would help? I have posted about this on the Knoppix-devel list.