Project: Merging Knoppix and Debian Live to get 64 bit Knoppix derivative?

[Capricorny]

Some years ago, I merged Knoppix 6.4.4 and Debian Live to get a pure 64 bit Knoppix version. It was done by a lot of handwork and some guesswork, and 64 bit busybox then had kind of persistent bug preventing it from working quite correctly in 64 bit Knoppix init, but the result worked fine in practice. When I later looked into upgrading this merger, I found that Knoppix and Debian Live had then grown apart, so the perspective was doing an increasing amount of handwork over and over again with each release. For my needs, it has been easier to use standard Knoppix w/64 bit kernel, and on demand occasionally run a pure 64 bit distro.

Looking into the Debian live init at that time, I simply thought Klaus K's approach was so much better. Therefore, there was never a question for me about the starting point: Standard Knoppix was to be updated with 64 bits Debian. It worked quite well, but it became too hard to repeat.

Debian Live has moved very far since then, and today I wonder if a better approach might be to "knoppify" Debian Live, swapping its init with a Knoppix derivative. I already routinely use squashfs in Knoppix instead of cloop in remastering, so adapting to the Debian Live environment should be no great problem today.

Any views on this?

With an upcoming major Knoppix update, quite a few of us will probably work with remastering, modifying init etc anyway. Therefore, I think it may be worthwhile to consider a 64 bit version at the same time. Unless Klaus K surprises us all with a 64 bit Knoppix version. But I haven't seen any hints of that.

[utu]

.
Greetings, Capricorny.

I, for one, would love to see derivative Knoppix material presented
as part of this forum. I would expect KK himself might be very interested
in discussions of alternative approaches.

I would also like to take this opportunity to thank you for your
contributions to this forum's wiki material. I was late in recognizing
relatively recent contributions to the wiki by Werner & yourself.

[dwstarke]

Would this be a one-off or would you maintain and improve a separate distro? Would you take each successive Knoppix and Debiam Live release and "Re-Knoppify" it or will you do that once and fork off of Knoppix?

[Capricorny]

Would this be a one-off or would you maintain and improve a separate distro? Would you take each successive Knoppix and Debiam Live release and "Re-Knoppify" it or will you do that once and fork off of Knoppix?

I really don't know what may turn out to be practical. If the modifications are mostly made in the init, it should be possible to adapt to new Debian releases fairly simply, as long as they don't make huge changes to the default init setup. In Knoppix, quite a few users have routinely patched the init for different purposes, like using squashfs, and there has usually not been a lot of new challenges with a new release. So, "Re-Knoppification" could at best reduce to a few patches. In which case it can be routinely carried out and patches posted. But it could also involve a lot of work, in which case it might be too much work to carry out with every new release. For me, even, say, a 64 bit "Knoppification" every third release would help a lot, as one can (in many cases) easily run more updated OS versions in VMs. I wouldn't call it a fork in any case, just a modest Knoppix-inspired derivative of Debian Live.

[Capricorny]

After having worked with Debian Live (7.6) LXDE for a short while, I have some initial observations:

* CD++ sized starting point is much more practical than I thought. A 900 MB squashfs image contains basic system, LibreOffice, Gimp and Iceweasel, in standard Debian versions (that's the basic philosophy) + a few accessories. No bloat, but quite useful out of the box.

* System setup is awkward, but usable. You can use the "iso/hybrid" image, which can be bare-bones copied onto USB media cp <iso.image> /dev/sdb. Then a read-only iso file system is created, so there is no direct configuration possible. But you can use e.g. gparted to create a partition on the rest of the stick, and set up a persistence loop file, live-rw, there. With boot parameter persistence, this loop image will be union-mounted and used just like knoppix-data.img in Knoppix. You may also use a separate home image.

* While I have been using and reastering Knoppix for years without an ordinary hard disk install, the methods are geared towards working from such an install. Which is no big deal to make, I use a 12 GB partition, it's more than enough. I ran qemu-kvm from that to setup the USB stick, using apt-get and synaptic in the VM.

* 64 bits and nobloat makes a bit of a difference - not very much, but it just feels more right. (And I need it for R, Oracle XEand VMware Workstation)

Debian Live work is very much geared towards making tools, so a partial Knoppification may not be that hard. First step for me will be setting up a complete system, then trying to remaster it.

[utu]

Greetings, Capricorny.

I was favorably impressed by Debian Live and its CD++ size as noted in
my prior blurb at:
http://knoppix.net/forum/threads/31009-64-bit-LXDE-Debian-LiveUSB
But, I find Debian Live documentation very obtuse. I don't think I
would have a clue if it weren't for its similarity to Knoppix.

As you know, I'm fond of Klaus K's init that allows 'stacking' some
read-only images (at your own peril, of course). Debian Live doesn't
have that. If it had the absence of systemd, I might defect to it.
I don't recall if Debian Live's init is bash-readable like Knoppix;
I doubt that it is.

And, I'm not sure any of my apps are 64-bit, except maybe for UEFI
when using my win8 machine.

[Capricorny]

Yes, utu, I think there is little reason to discuss whether the Knoppix init and overall setup is better than current Debian Live. It's miles ahead. And I don't really understand why the Live guys haven't bothered to pick up more tricks from Klaus K.

That said, I think Klaus K could do a few things differently, too. First and foremost, it is the bloat problem. When it was difficult or impossible to do extensive new installs and changes on a Knoppix system, the package selection was all-important, and should be as broad as possible. But nowadays, good sized permanent storage and easy standard Debian program installs and updates makes customizations and program-wise updates a breeze, so for lots of serious users, the package selection of Debian Live would be a better starting point for customizations.

There is also a need for "Debianization" of the Knoppix customizations. I think I know why Klaus K hasn't done much of this, but it would surely make Knoppix less of a one-man venture. I also think it would make debugging life quite a lot easier for Klaus K himself.

With the very smart 32/64 kernel hybrid solution, Knoppix with the 64 bits kernel in several respects behaves as a 64 bit system, so for instance, with 16 GB RAM, I can run a bunch of virtual machines simultaneously. Therefore, pure 64 bits is mostly for specialized use. But today, that may include image processing, like video editing, RAW photo image development etc. Which can greatly benefit from "pure" 64 bits. So I think Klaus K could support 3(4) versions with relatively little extra work. (Which he also surely could get help with):

1. Pure 32 bit CD. Every now and then. For backwards compatibility etc. Here standard packages may have to be trimmed down to fit.
2. Debian Live package selection, 32 and 64 bit. <1000 MB ISO image with LXDE. Bare-bones, explicitly with little support to expect. In fact, I think we may adapt the Debian Live tools for this - that's the 'Knoppification' hopefully to come.
3. The usual 32 bit hybrid DVD version of today.

I don't think we should talk about "defection" as long as we must go elsewhere to do things not supported in Knoppix. The question is whether we will come back. I surely hope so.

[Capricorny]

I am quite a bit relieved to report that I have managed to coerce Debian Live into working quite similarly to a Knoppix Poor Man's Install.
Which means that I can use it with some degree of efficiency.
I byte-copied the ISO image onto a USB stick, created an ext2 partition on the rest of that stick, and created a live-rw file image there with an ext2 file system on it. Did some installs on it, it worked quite fine under qemu-kvm (necessarily 64 bit version).

So far, so good, but what about Poor Man's installs? I copied the /live directory from the ISO file onto a FAT32 partition (/dev/sda1), and the live-rw file onto an ext3 partition (/dev/sda8 ). Then I created a /boot/deb760_live directory on my boot partition (here /dev/sda6), copied vmlinuz and initrd.img from the live-directory there, and modified old legacy grub's menu.lst with an entry for Debian Live:


title Debian 7.6 64 bits live sda1
root(hd0,5)
kernel (hd0,5)/boot/deb760_live/vmlinuz boot=live config quiet splash initrd=(hd0,5)/boot/deb760_live/initrd.img persistence keyboard-layouts=no
initrd (hd0,5)/boot/deb760_live/initrd.img

And it worked! Debian complained it didn't find the persistence file at /dev/sda2 as it expected, but got around to mounting it from /dev/sda8 anyway. It passed by a file with the same name at /dev/sda1 - that's fine with me. Documentation says that several such files may be mounted, provided they are in separate directories and have different, compatible, mount points defined.

One of the great attractions with Knoppix is that you can always do a remastering from the version you use. Debian Live takes an almost diametrically opposite approach, making the creation of new images extremely simple, but with little interest in using them and remastering them - it seems to be more designed as a vehicle for Debian installs. We'll see how this works out.

The iso-hybrid image doesn't work on USB in many BIOSes, and unetbootin which is recommended as an alternative for creating bootable sticks, crashed for me. So I'll rather work around that, using Knoppix methods, I think.

[utu]

One of the great attractions with Knoppix is that you can always do a remastering from the version you use. Debian Live takes an almost diametrically opposite approach, making the creation of new images extremely simple, but with little interest in using them and remastering them - it seems to be more designed as a vehicle for Debian installs.
I concur in this assessment. Another feature of Knoppix that is unique & valuable, not to be lost in adapting to Debian-Live.

[dwstarke]

Is an iso available? TIA

[Capricorny]

Is an iso available? TIA

Actually, the basic idea in Debian-Live seems to be that you roll your own. I have used the standard 7.6 LXDE Live ISO from July as starting point, but I may try to create a new ISO from scratch myself.
I could use the package list from my current running system - but the three most important additions for me, VMware Workstation, Oracle XE 11g and SqlDeveloper, are not available as .deb packages. The two last were converted from .rpm by alien.

The somewhat fundamentalist Debian approach also shows up in hardware problems. I couldn't get Wifi to work on a Asus N53S laptop, but found out that I could fix it by downloading and installing the firmware-iwlwifi package. It works perfectly, but I have to start it manually! With non-free components, wireless is disabled by default!!! While there are no such problems with more politically correct drivers. ;-)

[Capricorny]

Well - this has worked very well so far.
I write this from a remastered Live-Debian with no persistent storage, but with all previously installed packages crammed into a 4.2 GB filesystem.squashfs file.
It has all worked on the first try. Looks very promising, as I won't have to exceed the 4GB limit of FAT32 do set up my 64 bit servers..

[Capricorny]

The details

Here, I work "somewhere" with a few GB free space. In this case, 15 GB would be enough.
The persistent store is /media/sda8/live-rw, and the compressed image file is /mnt-system/live/filesystem.squashfs. Those names are default Debian-live names.
I run this from Knoppix, so the files are not in use. I union-mount them, copy the file system to a directory copyfs, and compress this into a squashfs image to replace the original one.
If you insist on running from the ISO, as it is copied onto sticks, you get problems replacing the squashfs image. You may have to create a new ISO image in some way.
But I have simply copied the /live directory (corresponds to the KNOPPIX directory) onto a hard drive, and set up good old legacy grub to boot it.


#!/bin/bash
#
[ -d sqfs ] || mkdir sqfs ;
[ -d persfs ] || mkdir persfs
[ -d aufs ] || mkdir aufs
[ -d copyfs ] || mkdir copyfs
#
sudo mount -o ro,loop /mnt-system/live/filesystem.squashfs sqfs
sudo mount -o loop /media/sda8/live-rw persfs
sudo mount -t aufs -o br:persfs:sqfs none aufs
#
#
sudo rm -rf copyfs/*
sudo rsync -axu aufs/* copyfs
#
sudo mksquashfs copyfs filesystem_rem.squashfs

You may want to keep the old squashfs image as a backup before copying in the new one.
Next setup a new persistent image



cd /media/sda8
sudo mv live-rw live-rw-0 # Or whatever version number
sudo dd if=/dev/zero of=live-rw bs=1M count=3950
sudo mkfs.ext2 -F live-rw


And I still can't see the great advantages of working in a chroot environment, glad I was able to completely avoid it.. :-)

[Capricorny]

Debian-Live has a terrible way to handle overlays. All instances of live-rw encountered are sandwich-mounted together, even if the advertised label mechanism is used. So, basically, for safety one should use just one at a time, renaming the others.

How bad is this in practice? It turns out, it doesn't have to be that bad. With frequent remasterings, we don't really need much persistent storage for everyday use, and we can mount volumes to get what else we need.

Right now, I write this in a kvm window running Debian-Live without persistence in a Knoppix 7.0.5 host, with two other kvm instances simultaneously running from the same image, one with persistent store, the other without.
I run them with no modification, with the same command


kvm -m 1024 /dev/sda
to get the grub menu of the host, and I choose the same (!) alternative, Live-Debian without persistence, several times.

Under kvm, I mount and modify (different) disk partitions on the host. Typically, I can run one instance for database, another for application server (e.g. JBoss), and maybe a third for mailserver.

This is what I need to develop applications with realistic testing on just one machine.
While it was not possible to run kvm in the virtual machine, I could run VMware there..

Performance isn't hopeless, I may try remastering in a VM again.
There is a definite need for knoppification of several aspects of this live-version, but the need is not desperate.

[utu]

Debian-Live has a terrible way to handle overlays. All instances of live-rw encountered are sandwich-mounted together, even if the advertised label mechanism is used. So, basically, for safety one should use just one at a time, renaming the others.

How bad is this in practice? It turns out, it doesn't have to be that bad. With frequent remasterings, we don't really need much persistent storage for everyday use, and we can mount volumes to get what else we need.

Have you decoded Debian-Live's minirt.gz?

[Capricorny]

Have you decoded Debian-Live's minirt.gz?
Not so far this time around. I didn't like the init at all when working on a 64 bit version from the Knoppix side 3 years ago.
So, eventually, it may be knoppified, but it's not something I look forward to.
Obvious modifications are introductions of parameters fromhd, livedir and live-rw-dir.
I don't know if it is worthwhile to implement the full Knoppix overlay setup. Because there is a general mechanism in place to mount overlays on given mount points. Which in many ways is more precise.

[utu]

Debian-Live has a terrible way to handle overlays.

Describing a process as 'bad' is not informative.
I suspect in this case, as in Knoppix's case, the process of combining
separate images is not bad, per se, but hazardous -unless cerain conditions are met.

I would like to know if in general one might find a working Debian-Live to
contain several squashfs images or devices that 'add-up' to a working
union of the expansions of these images. This should be obvious from the
decoded minirt.gz.

AFAIK, Klaus K has never issued any rules for combining images in Knoppix,
yet he continues to provide and use a latent capability to up to stack seven
additional images together in Knoppixes 6s & 7s.

I think the general rule in both Knoppix and Debian-Live might be that
so long as the expanded images do not land on the same locations with
different meanings, things will be ok. Segregating the information
contained onto separate devices may be one way to assure that the union
of expanded images will not be informationally corrupt.

I routinely make a read-only Knoppix 1 which contains all my /home tweaks
and all my added programs. My only current difficulty is in treating a
troublesome link which must be re-established after every new image is
added. This is without the precaution of providing /home its separate device.
Simplistically, adding things works ok, but removing things is problematic
in the union of expanded filesystem images.

I'd rather start with a CD-size iso and add things, 'cause all I need works really
well on 2 Gb SanDisk LiveUSBs. Now that 8 Gb Cruzers are so inexpensive,
I make DVD-size LiveUSBs and just add fewer things, but it's sure
wasteful bringing in a lot of stuff I never use, anyway.

I'm more interested in Debian-Live's live-build on-line capability to
define and create really tailor-made minimal isos. There's currently no
similar magic in Knoppix.

[Capricorny]

Utu, believe just what you want to, I am using this the way it is advertised to work, Which does not work. Terrible is no exaggeration. I could also add "pathetic" for the ways the same types of errors keep on cropping up over and over again. And that the Debian standard kernel is at 3.2.0 - which in itself contributes to an own set of problems, not the least in networking.
Suffice to say that Klaus K has very good reasons for his policies with Knoppix. If you doubt, just give Debian-live a try, really using what are presented as the features.
Debian-live is designed for multiple overlays, but I doubt many use it. There are lots of bugs, and they are not benign.
The worst thing, is that you never know when and why things break. It has to do with all the promises that are not kept. For axample, you drop the overlay anywhere, the system just will find it. True? Not quite. I think I have experienced most of the possible combinations of errors during the last 24 hours. Overlay isn't mounted, it is mounted, but not used (that's a really terrible one), it is mounted and crashed, it is sandwich-mounted, it is not correctly updated, it is mounted, but not read correctly.

None of those problems were ever encoutered during 8 intensive years with Knoppix.

[utu]

the Debian standard kernel is at 3.2.0
I must admit, that's a bit of a shock.

[Capricorny]

I must admit, that's a bit of a shock.

Of course there have been a few upgrades to that version, but.. On the other hand, come on, it's still not three years old ;-)


user@live-debian:/media/sda8$ uname -a
Linux live-debian 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u1 x86_64 GNU/Linux
user@live-debian:/media/sda8$

The basic philosophy, their "paradigm", is worse, though. How can anyone hit on the idea to use partitions for persistent storage? Even on a small USB stick you would like to be able to place some files there too, outside of the image file. And the raw-copy-iso "installation" method makes it impossible to tweak parameters, while the creation of more user-friendly install versions is delegated to unetbootin et al. I'm not very fond of that bunch, they have been less than reliable to me.

There are a lot of comical aspects, too. Using gparted to partition the rest of the USB stick after raw copy of the ISO image, it is equipped with a label that Debian truthfully copies to the device name on automount of the stick.
Resulting in a mount point that the OS is often not able to handle correctly parsing commands, creating errors&problems.

So I have had to manually unmount, and remount using the more prosaic /media/sdb2. Problems gone.

[Capricorny]

The Debian-Live approach also provides some good opportunities. First and foremost, the very easy remastering makes it efficient to run with no persistent store quite often. For instance, upon remastering, I just start without the store, and the image I run is union-mounted with the persistence file and copied&compressed. Then I have to boot something else to copy the new squashfs image in place, but that's all that has to be done outside the system.

Boot the new image without persistence, create a new persistence file.
For further adaptations, the new persistence file can be mounted in an instance run under kvm, programs can be added and settings tweaked for an eventual new remastering.

Storing the series of squashfs-compressed images, one can always go back in this process if needed.