remaster knoppix with squashfs

[wangji]

KnoppixRemastering-with-squashfs-guidance.
Foreword.
This is not a howto ;there are so many good knoppixremasteringhowto around ,so I only discuss the main points with enough details so anyone could
switch from cloop-to-squashfs remastering.
Introduction.
Let me put aside zisofs,zftree ,but only give here a comparison of cloop and squashfs :compression ratio and runtime .Let me call
knoppix.clp the usual knoppix image obtained with cloopcompress the isofile (or mkiso piped thru createcompressfs )
knoppix.sqf the knoppix image of the same structure using mksquashfs.
compression ratio sqfs 689655Kb in 34 minutes
cloop 729163Kb in 43.7minutes (a small gain of 5% and 10 min in favor of sqfs ) Source=2082Mb
runtime
In order to avoid errors induced by bad behaviour of cdrom-reading (aleas due to dust ,mechanics & feedback-control )the runtime measures are
done directly on running knoppix-image sitting on hard drive in both compression cases .In both cases mkzftree of a huge /usr is measured.
runtime-sqfs 27min25s (985.3Mb/1769Mb)
runtime-cloop 27min59s (985Mb/1785Mb)
Almost no difference in runtime.So we must consider other factors to favor the one or the other.
The cloop are compiled in modules ,so don't touch the kernel ;but in fact it is almost impossible -without too much contorsion if ever possible-to mount
the clooped image with another-version-kernel.One needs to boot with the same-version-numbered-kernel to be able to insmod cloop !
The squashfs has to be compiled with patched kernel ;but any other-version-patched-kernel can "mount loopback" the image.Now,each time one changes
the kernel,one needs at least patch modifying the cmdline-length from 256 to 512 characters (a line in /linux/include/asm-i386/setup.h ) to keep
Klaus'idea of having a flexible command line .For amateur-developer-remastering the last solution makes life easy !
Philosophy
To switch from cloop to squashfs just replace the make-image. and of course the loopback mount the image during booting phase
Requirements
1 Get the kernel and the squashfs patch (squashfs.sourcefore.net ; latest version 2.0-alpha) .Modify setup.h as said ,patch with squashfs,compile.
compile the "mksquashs-tools" directory to get the binary "mksquashfs" and put in /usr/bin .
2 Get a copy of /KNOPPIX to some /remaster ;replace in /remaster the following directories to reflect the new kernel
/boot /lib/modules/2.6.x /usr/src/linux-2.6.x (this is standard kernel playing game ) and any eventual add/remove apps !
3 create the image: mksquashfs /remaster /outside-remaster-dir/knoppix
This line replaces the mkisofs ...| create_compressed_fs ...>knoppix.
4 modify linuxrc (from miniroot.gz or initrd.gz ) by commenting out the lines
(after scsi driver,at the paragraph "try to find cdrom" )
#insmod -f /modules/cloop.*o file=/cdrom/KNOPPIX/KNOPPIX
#mountit /dev/cloop /KNOPPIX "-o ro" || FOUND_KNOPPIX=""
insert there a line
mount /cdrom/KNOPPIX/KNOPPIX /KNOPPIX -o loop ,ro || FOUND_KNOPPIX=""
This mounts the image as squashfs tree.The rest is independant of compression mechanism.
Do not forget to change vmlinuz (new) when building the isocd.Some other version uses insmod cloop and losetup the loopback file ,it 's quite similar.
The Fun
By remaking the sqfs images of diferents knoppix-like-distro (cebit-3.4,quantian-openmosix,pollix,kurumin,physicstools...) with kernel-2.6.6 and put them
all in a /KNOPPIX ext2-directory ,I am able to boot any of them with just a boot-cd-15Mb and typing ...a label !
Have fun,

[covici]

I would love to do this, but I cannot compile cloop for my kernel, how did you copy the compressed file system to uncompress it so you could play with it?

Thanks.

[wangji]

just boot the cd you want; at boot prompt ,type
knoppix 2 to get into unix_console
you are on a ramfs ,in which /KNOPPIX contains the whole
uncompressed files (mepis or berry would be in /linux or /berry
directories)
just #cp -a /KNOPPIX/* /where you want to dir/
in this way the uncomprssion is done by the original author of the cd's cloop !!! and you avoid the problem o mismatching between
your cloop_compiled and the original one !
just mksquashfs th at structure and you are done
(almost)

[covici]

Hi. OK, I have now run into the problem that the builtin mount does not except -o loop option -- how did you get around this -- did you use a different mount somehow?

[wangji]

I suspect you compile your kernel with loopbackfs as module !!
you have to insmod-modprobe loop-module
If ,it's still the old kernel,which distro you startoff to play with ?
Look at .config around filesystem
in my case,when compile 2.6.6 I just have loopback as builtin !

[covici]

I mean that you cannot say mount -o loop because mount does not acceppt loop as an option so no matter how you compile the kernel the built in mount is not going to work (at least that is what it looks like if you look at the code in mount.c in the ash-knoppix). So what would be the command line that would work or did you use another mount command?


I suspect you compile your kernel with loopbackfs as
module !!
you have to insmod-modprobe loop-module
If ,it's still the old kernel,which distro you startoff to play with ?
Look at .config around filesystem
in my case,when compile 2.6.6 I just have loopback as builtin !

[wangji]

this is why I ask which distro you play with !
If you pick up the miniroot.gz from prometeusX_iso or physicstools-squashfs_iso then that ash allows you to do so !
the busybox from peanut.gz allows you to play with fun

[covici]

I see, I was still using the KNOPPIX ash and that is my problem -- I didn't use any distro but compiled my kernels from vanilla sources -- I can fix the loop.o problem and if I can find those ash things I am done!

Do you have a link to the ones you mentioned, this would be great!

Thanks so much for your help.

[wangji]

http://sf.gds.tuwien.ac.at/b/br/brproject/PrometeusX.iso

1 ftp://ftpthep.physik.uni-mainz.de/pub/physicstools/
2 ftp://ftp.ibiblio.org/pub/linux/distributions/physictools/
3 *http://www.vnlug.org/knoppix-vi/physicstools-1.0.iso
* *http://www.vnlug.org/knoppix-vi/physicstools-1.0.iso.md5
* *http://www.vnlug.org/knoppix-vi/physicstools-1.0.lsm
(I am not sure the 3rd contains squashfs;but the two first are ok)
if on hurry,or your connection is slow,have an email_add_temporary somewhere and put it here I will send there
miniroot.gz(around 1Mb gz),or just ash (945kb non-gz)

[covici]

My Email address is covici@ccs.covici.com if you could send the miniroot.gz this would be very good.

In the meantime, I did try busybox which has a lot of nice things in it, but there is something wrong when it tries to exec static/ash it hangs and this is true even by hand, so their exec is broke and this may mean that iit is unusable.

Thanks.

[kelmo]

Thanks wangji, the mods worked well. The squashfs image of the KNOPPIX filesystem is ~5% smaller than that of cloop.

Busybox was simple to compile and worked well.

Rebuilt the miniroots with symbolic links to busybox(static) instead of ash, and with the loop module.

The following changes to linuxrc did the trick for Knoppix 3.4.


$INSMOD /modules/loop.o >/dev/null 2>&1
mount -t squashfs /$1/$KNOPPIX_DIR/$KNOPPIX_NAME /KNOPPIX -o loop,ro ||


Only problem:-
The cd is not ejected before shutdown. Where do you suppose cloop is mentioned in other scripts? SysV-init scripts?

[wangji]

hi kelmo,
this cdeject at halt is handled by /etc/init.d/knoppix-halt
it has nothing to do with cloop !!!I am not 100% sure ,but I suspect rather the problem of "cd not eject" related ...with
kernel-2.6.x_cdATAPI_cd_fake_scsi.
Just to put a hint : if you allow dma at boot for cdreader,some
knoppix_like distro don't boot at all (no matter which compression used) the /dev/scdi ...not detected and you hang .
(This is not the kernel fault but rather ...poor mechanics design,
-price is so low !!-errors bursts -irq,dusts...-and perhaps too weak
Reed Solomon implementations ,-not more than 5 errors per sector_read-.).In those specific cases just type at boot:
knoppix nodma ...
and the miracle operates
But of course I do not help you solve the cd ejection !!!
Type as root : cdrecord -eject dev=x,x,x
works,even if you had boot a livecd,
that's a very poorman solution.

[kelmo]

No big problem. I really like the simplicity of remastering using squashfs.

I wish a debian packager would put the latest squashfs-tools in a .deb.

Wangji, can you give me any tips of what options can be used to 'optimize' the compressed filesystem with mksquashfs. At the moment I am using the basic command:-

mksquashfs {source} {target}

This works fine, but I'd like to experiment with sorting the filesystem to squeeze every last bit of performance out of it.

Thanks

[wangji]

I am sure I do not know the real answer !! but I try to summarises my "feelings" .In older version ,ie squashfs-1.3 the
indexing pointer scheme was clever and allow very fast stuffs ;
but if you compress the huge_knoppix_2.2Gb then you get into trouble : that's why version 2 from Philip Lougher ! but I understand then it compresses almost in blocks "exactly as cloop"
(with probably variations ,tree structure...giving this ~5% diff)
Arranging big files at the beginning may help readout response time ,but that is related to how iso files,cdreader behave(in case of cloop) I believe you gain nothing with squashfs (we have to ask Philip at his forum).The true point I like mksquashfs is that you
are not bound to put the whole stuff in a "remastering_directory" since you can put exclude_options (I am using that way a lot)If you need response time,I believe mkztree
should be faster (however poor compression ,since block size vary a lot depending on your applications !!! remember zisofs,was included long time ago into kernel -politically correct put aside !!-)