Squashfs-ed knoppix

[kl522]

As I learn more about knoppix, I am beginning to form this idea of using squashfs instead of cloop. This idea is not new. Apparently there is a distro already doing it, finnix. But why use squashfs instead of cloop :-

1. Squashfs is already in the kernel. There is no need to maintain a separate cloop module source code, which can potentially be incompatible with newer kernels.

2. I ***THINK*** squashfs dynamically uses a memory chunk for decompression, whereis cloop allocates a huge chunk ( currently at least 1G ) of memory to store the uncompressed image. According to finnix, it needs only 192MB, so squashfs is more memory efficient.

I guess I will try out this idea of one of these days.

[kl522]

Read this for the claim about knoppix needing at least 1 GB of memory :-

http://www.finnix.org/Frequently_Asked_Questions#What_are_the_difference s_between_Finnix_and_Knoppix.3F

While at it, I am still very uncertain if the claim made of finnix auther is true. Nevertheless I found that knoppix 6.2 mounted a 1G tmpfs, but the tmpfs is only used if there is no persistent store. When using persistent store, the tmpfs is not used, but it's remained mounted.

[krishna.murphy]

I suppose for people with lots of processor cycles to spare, it might make sense. Squashfs IS used, AFAIK, in the CD version of Knoppix. But for the many who have less CPU and more disk space (for swap), it's probably better to use cloop. Let us know what you determine experientially!

Cheers!
Krishna :mrgreen:

[kl522]

I think the claim that cloop needs at least I G memory is likely to be over-stated. A quick check on cloop source code revealed that it also decompresses the file system on-the-fly basis.

The 1G of /ramdisk mounted, used when one does not have persistent store, is something which the actual usage will grow as it is used. The 1G /ramdisk is mounted, but not used when there is a persistent store. And therefore even though that's a "bug", it does no harm.

In terms of CPU cycles, there is no apparent difference between squashfs and cloop, because both uses the same "on-the-fly" decompression and buffered blocked strategies, and also same compression algorithm.

Therefore the only different between squashfs and cloop is that squashfs has long been accepted into the kernel, it needs no extra maintenance. Cloop is currently maintained separatelly by the author of KNOPPIX.

[krishna.murphy]

I think the claim that cloop needs at least I G memory is likely to be over-stated. A quick check on cloop source code revealed that it also decompresses the file system on-the-fly basis.

The 1G of /ramdisk mounted, used when one does not have persistent store, is something which the actual usage will grow as it is used. The 1G /ramdisk is mounted, but not used when there is a persistent store. And therefore even though that's a "bug", it does no harm.

In terms of CPU cycles, there is no apparent difference between squashfs and cloop, because both uses the same "on-the-fly" decompression and buffered blocked strategies, and also same compression algorithm.

Therefore the only different between squashfs and cloop is that squashfs has long been accepted into the kernel, it needs no extra maintenance. Cloop is currently maintained separatelly by the author of KNOPPIX.

I don't think Klaus would have become the maintainer of cloop for anything less than a substantial reason; too much work for no real benefit. I don't know what that reason is, but I know it must exist. The "phantom GB of ramdisk" aspect of cloop is probably intentional, perhaps to make possible very fast growth of the actual filesystem, or some such. Regardless of all that, if one had some reason to avoid using the compressed filesystems (perhaps speed?), how difficult would it be to leave that out in a re-master? Just curious, and I thought one of y'all might have the answer.

Cheers!
Krishna :mrgreen:

[kl522]

I don't think Klaus would have become the maintainer of cloop for anything less than a substantial reason; too much work for no real benefit. I don't know what that reason is, but I know it must exist.

That's what we call "belief", it's not based on facts. ;)
I think it's all historical. Cloop existed long time ago. Klaus is not the creator. He is just the maintainer. He needed it at that moment. So now nobody else except him has "vested" interest in it. He is using it simply because he is familiar with it. Kind of sentimental value thingie. As a matter of fact, it's a easy swap to use squashfs.



The "phantom GB of ramdisk" aspect of cloop is probably intentional, perhaps to make possible very fast growth of the actual filesystem, or some such. I believe that you said this because you did not look at the code. The ram disk has nothing to do with cloop. The ram disk is created for unionsfs with cloop, to allow users to have write access to the otherwise read-only cloop. And it's left mounted unused when the user has a persistent store. If you look at the code you will agree with me, it's a bug.

[kl522]

Below is a comparison of some of the interesting aspects of squashfs vs cloop when running on the ***SAME*** machine with 4G physical memory. Some details have been snipped for clarity :-

When using cloop kernel driver :-



# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 94G 33G 61G 36% /mnt-system
/dev/cloop 4.0G 4.0G 0 100% /KNOPPIX
/dev/loop0 2.4G 940M 1.5G 39% /KNOPPIX-DATA
....

# cat /proc/meminfo
MemTotal: 3588864 kB
MemFree: 3005164 kB
Buffers: 112860 kB
Cached: 303956 kB
SwapCached: 0 kB
Active: 220000 kB
Inactive: 294008 kB
....

# ls -al /mnt-system/KNOPPIX/KNOPPIX
-rwxrwxrwx 1 root root 1546036015 2010-06-16 22:36 /mnt-system/KNOPPIX/KNOPPIX
Here is the corresponding information when using squashfs :-


# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 94G 33G 61G 36% /mnt-system
/dev/loop0 1.4G 1.4G 0 100% /KNOPPIX
/dev/loop1 2.4G 939M 1.5G 39% /KNOPPIX-DATA
....
# cat /proc/meminfo
MemTotal: 3588864 kB
MemFree: 2927872 kB
Buffers: 178880 kB
Cached: 329248 kB
SwapCached: 0 kB
Active: 187076 kB
Inactive: 414508 kB
....
# ls -al /mnt-system/KNOPPIX/KNOPPIX.sq
-rwxrwxrwx 1 root root 1453662208 2010-06-16 23:09 /mnt-system/KNOPPIX/KNOPPIX.sq
Summary :-
1. cloop registers the need to use 4G of virtual memory on /dev/cloop, whereas squashfs registers the need for 1.4 G of virtual memory on /dev/loop0

2. squashfs compressed image is smaller by 100M when using the same compression algorithm and block size.

Cheers.

[kl522]

Following what I have posted previously, I noticed that the kernel 2.6.34 has included the LZMA support for squashfs, it will be a matter of small delay, you will also see the squashfs-tools to OFFICIALLY include LZMA support. Actually squashfs using LZMA has existed a long while ago, and has been actively used by the embedded systems ( particularly OpenWRT ) so there is nothing so new about this.

Cloop also support LZMA compression.

As for me, LZMA is not particularly useful to me, because the compression takes too long. However, for a distro or a LiveCD, it is something particularly useful, meaning for the same amount of space ( say a CDROM ), you can squeeze in additional 10-20 % of additional programs into it.

[Capricorny]

Have you made any more observations using squashfs with Knoppix? If there are no clear problems or potential drawbacks, I think, if only for the sake of uniformity, we should encourage the use of squashfs. My impression from what I have read is similar to your observation: In practice, the difference isn't that big, but having the necessary modules in the standard kernel makes things a bit easier.

I'd like to make a remastering now, with updated packages and using squasfs, to have an updated system with everything needed and a 4GB persistent image on a 8GB flash drive. Now I need 16.. :)

[kl522]

As you have noticed, I tried out the idea of squashfs. Eversince, I have been using it without a single problem. There is actually no downside, but there are a few ( perhaps not very significant ) upsides. For summary, they are :-

1. stock kernel module (vs patched module ).
2. less virtual memory required.
3. slightly better compression.

To create the compress image, this is what I use :-


mksquashfs /mnt/knx/source/KNOPPIX /mnt/knx/master/KNOPPIX/KNOPPIX.sq -b 262144 -noappend
For comparison, this is what I use for cloop image :-


mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
-hide-rr-moved -cache-inodes -pad /mnt/knx/source/KNOPPIX \
| nice -5 /usr/bin/create_compressed_fs - 262144 > /mnt/knx/master/KNOPPIX/KNOPPIX
You notice that for both cases, I don't sort the files according to 'frequent use', but that does not mean it is not important. I just don't bother to do it.

Last but not least, I have modified the initrd.gz to mount squashfs accordingly :-



indknoppix(){
local dev
for dev in "$@"; do
[ -b "$dev" ] || continue
message -n -e "\r${CRE}${BLUE}${SEARCHINGFOR} $DISTRO in: ${MAGENTA}$dev${NORMAL} "
trymount "$dev" /mnt-system >/dev/null 2>&1 || continue
if [ -r /mnt-system/"$knoppix_dir"/KNOPPIX -o -r /mnt-system/"$knoppix_dir"/KNOPPIX.sq ]; then
message -e "\r${CRE}${GREEN}$DISTRO ${FOUNDAT}: ${MAGENTA}$dev${NORMAL} "
return 0
fi
umount /mnt-system
done
return 1
}
....
mountknoppix(){
local k dev dir count=0 RC=0
[ -b /dev/cloop ] || mknod m 644 /dev/cloop b 240 0
[ -f /mnt-system/"$knoppix_dir"/modules/squashfs.ko ] && \
insmod /mnt-system/"$knoppix_dir"/modules/squashfs.ko
[ -d /sys/module/cloop ] || insmod /mnt-system/"$knoppix_dir"/modules/cloop.ko preload=32 || return 2
[ -d /sys/module/aufs ] || insmod /mnt-system/"$knoppix_dir"/modules/aufs.ko || return 3
if checkbootparam squashfs && [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] ]
then
mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
elif [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] -a \
! -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] ]
then
mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
else
for k in /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] \
/mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx][0-9]; do
.....
I retained the cloop booting capability for comparison purposes. When both images are present, I use a cheatcode to select the preferred one. It was kind of elaborate but now I can safely say, it is unnecessary to retain the cloop image at all.

Have fun.

[dinosoep]

I have tried myself to squashfs my knoppix cd install on usb and created knoppix.sq but I do not have inirt.gz but a look at syslinux.cfg learned that it used minirt.gz.
I modified that the way you did in the above post but I somehow messed up pretty badly.
first there was a syntax error while booting so I reedited the init script and now it's completely screwed up :p

is someone so kind to share their (m)inirt.gz with me?

[Forester]

is someone so kind to share their (m)inirt.gz with me?

Do you still have available
a) a working Linux and
b) Knoppix on the CD you used to install to USB in the first place ?

If so, pop the CD in and mount it under /media/sr0. You should find what you are looking for under /media/sr0/boot/syslinux/minirt.gz.

If the Linux you are running is the Knoppix on you CD, then the CD is already mounted so look under /mnt-system/boot/syslinux/minirt.gz.


in the above post

Which post ?

[dinosoep]

forester, look at the previous page of this forum :p
yes I have still a working linux install and the original iso file. I want however the modified version that uses squashfs instead of cloop

[kl522]

forester, look at the previous page of this forum :p
yes I have still a working linux install and the original iso file. I want however the modified version that uses squashfs instead of cloop

Try this :-

http://www.fileserve.com/file/s9TyZW8

The said file is modified from some older version of Knoppix - for my own use - so I do not claim its suitability for general use. I also have other mods for other reasons which you might want to ignore.

[dinosoep]

kl522, I saw your post a little to late.
I've been messing around with minirt.gz and I now have a "working" version for 6.4.3
my mountknoppix is now:


mountknoppix(){
local k dev dir count=0 RC=0
[ -b /dev/cloop ] || mknod m 644 /dev/cloop b 240 0
grep -q squashfs /proc/filesystems || insmod /mnt-system/"$knoppix_dir"/modules/squashfs.ko || return 4
grep -q cloop /proc/devices || insmod /mnt-system/"$knoppix_dir"/modules/cloop.ko preload=32 || return 2
grep -q aufs /proc/filesystems || insmod /mnt-system/"$knoppix_dir"/modules/aufs.ko || return 3

if checkbootparam squashfs && [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] ]
then
mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
elif [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] -a \
! -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] ]
then
mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
else
for k in /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] \
/mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx][0-9]; do
[ -r "$k" ] || continue
check_knoppix "$k"
if [ $count = 0 ]; then
dev=/dev/cloop ; dir=/KNOPPIX
else
dev=/dev/cloop$count ; dir=/KNOPPIX$count
[ -b $dev ] || mknod -m 644 $dev b 240 $count
[ -d $dir ] || mkdir -m 755 $dir
fi
if ! losetup $dev $k >/dev/null 2>&1 || ! mount -r -t iso9660 $dev $dir >/dev/null 2>&1; then
# Verbose error message
echo -n -e "\n\r${RED}${BROKENIMAGE}: $k ${ON} "; mount | grep /mnt-system; dmesg | tail | grep cloop; echo -n "${NORMAL}"
# Clean up for next retry
for dev in /dev/cloop*; do
umount -f "$dev" >/dev/null 2>&1
losetup -d "$dev" >/dev/null 2>&1
done
return 4
fi
let count++
done
fi
return 0
}

I'll see how long it'll run and if there turns out to be some problems I'm going to use your older version kl522 :)

[utu]

@ kl522
Great thread. Subscribing.

[Forester]

forester, look at the previous page of this forum :p

Sorry, my mistake. With you now. Interesting thread. Will subscribe too.

Pity the FAQ on the finix site seems to have disappeared. (Post #2, this thread).

Puzzled by the early assertions about virtual memory size. I don't see how that can be deduced from the output of df -h. That reports the file system size, how much is used and how much is free. Nothing, as far as I know, about virtual memory sizes. One has to be careful with these figures when it comes to compressed file systems. It seems cloop reports the size of the file system before compression, squashfs after compression.

[Forester]

@ kl522
Great thread.

Spot On ! Cracking ! Champion ! Works a treat !

It's late (for me), Need sleep. Will write more tomorrow.

[Forester]

I recommend this !

I remastered the DVD edition of 6.4.3 (with a few location modifications). It's big.

The cloop compressed file system was made with the -b 'try all and keep the best" option.
The cloop method requires either 9 Gb of virtual memory I don't have or a 9 Gb temporary file.

First the size of the KNOPPIX file produced:


cloop: 3.85 Gb
squashfs: 3.76 Gb
Second the time taken:


cloop: 94 minutes
squashfs: 9 minutes
Third temporary disk space required:


cloop: 19 Gb
squashfs: 10 Gb
That's what they call a Grand Slam.

And for me there's a bonus. Boot time on my laptop is about 60 s with either solution. Take my USB stick into work where the whacky set up I have with the Virtual Machine makes for a very slow boot ...

Boot time at work:


cloop: ~ 4 minutes
squashfs: < 2 minutes
The udev probing is a lot quicker with squashfs at work. I've no idea why but it's a very welcome result.

The mksquahfs tool I had to install (from the Debian Squeeze repository) listed zlib1g as a 'depends on' package so I was surprised to see such a difference in size. Fundamentally, how good the compression is depends on the compression library used underneath. lzma is supposed to compress better than bzip2 which is supposed to produce compresed files 20 % smaller than good old zlib (aka gzip). It is also supposed to be fast to decompress. So there are good things to come.

It may be that Knoppix has stuck to cloop for sentimental reasons but it may just have been one less change to worry about at the time Knoppix 6 first came out two years ago. I don't think squashfs was part of the standard kernel then. If cloop reappears with support for lzma I'll be surprised.

As for the outrageous claim that Knoppix requires at least 1 Gb of memory and cloop is to blame ...

... the original article has gone but I found a precis at https://www.experts123.com/q/what-are-the-differences-between-finnix-and-knoppix.html and offending text is ....


Finnix can be copied to RAM and run in only 192MB; Knoppix requires at least 1GBDoh ! Finnix has no X, no desktop, no mega Windoze-style applications with user-friendly OK buttons so of course it is small and can be loaded into a modest amount of memory and, of course, when Finnix compares itself with Knoppix is used the toram cheatcode, which loads the entire uncompressed Knoppix iso into memory and comes out with the obvious statement that Knoppix requires at least 1 Gb of memory. By that same reasoning I need 10 Gb of memory to run by Knoppix DVD but my laptop doesn't have that much memory.

The claim is about as useful as stating that oranges and bananas aren't the same thing because bananas take a lot less time to peel and suddenly all sorts of people are deducing that this is why monkeys are smart and that drinking orange juice is bad for your health.

Now I hope squashfs has come a long way since I first met it. My first Knoppix was 3.6. It ran like a dream but I was a newbie and someone told me to install Ubuntu to hard disk. I couldn't install Breezy from LiveCD. A year later I couldn't Dapper. A year later I couldn't install Feisty until I tried the alternative (aka Debian) installer. All those squashfs error I had had nothing to do with my hardware or my media - it was Ubuntu's user friendly way of telling me I didn't have enough memory.

Perhaps Knoppix has a reason to stick with cloop after all.

[Werner P. Schulz]

The cloop compressed file system was made with the -b 'try all and keep the best" option.
...
Second the time taken:
Code:
cloop: 94 minutes
squashfs: 9 minutes
... building the cloop compressed file system without the "-b" parameter works about twelve times faster. And the effect by using the "-b" parameter isn't worth to waste so much time for only a little better compression.

[kl522]

Perhaps Knoppix has a reason to stick with cloop after all.

You guys are diplomatic ! :)
I think it is strictly sentimental value and perhaps better way of saying is, for backward compatibility.

Each time I wanted to compile cloop, I have to look out for patches for it. Kernel 2.6.35, .36, .37, .38. Every version there is something to tweak. Perhaps the Linux kernel is to blame, and perhaps squashfs kernel source also requires changes, but hey, someone has already modified it for me !

If one checks kernel 2.6.38, it has LZMA2 for SQUASHFS ( aka XZ compresion). This fella claims it compresses better than lzma :-

http://chakra-project.org/bbs/viewtopic.php?id=4145

Cheers.

p/s: I am quite surprised about the time taken difference. Perhaps the mksquashfs was not done with lzma compression. But hey, without lzma it is already smaller than the '-b' of cloop ?

[utu]

I would like to see a comparison of the time it takes to boot
a LiveUSB (better yet, a LiveSDCard) made from a
Knoppix 6.4.x LiveCD, with the only difference being
between cloop and squashfs. Times for both, that is.

[kl522]

Read post #19 from Forester. He did it already. Basically on a fast computer, there is no appreciable difference. In a virtual machine, cloop is slower.

[utu]

@ kl522

Thanks. I was hoping for a lot more difference.
You should be aware of some history, if you're not:

google <squashfs cloop knoppix 2004 klaus>

Thread from 1/25/2005
http://www.knoppix.net/forum/threads/16266-smaller-and-a-little-bit-faster-knoppix-3.7-%28experimental%29

Thread from 11/12/2005
http://www.knoppix.net/forum/threads/21403-cloop-sucks-squashfs-rocks

[kl522]

There maybe some truth in some of the older threads. However, the then cloop and the then squashfs might be different now. For software/hardware, 6months or 1 year is already a big difference. So I am not too sure of their relevance in todays view.

Technical analysis might be useful. But unless it is subtantiated with real life data, otherwise it is academic. I dare say in order to see the significant and conclusive difference in terms of performance, one will have to carefully design experiments to observe them. Otherwise at a gross level, especially on today's hardware, it will be hard to notice any difference.

And for usage history, I put my bet on squashfs. You see them in almost all of the embedded linux devices today. These devices have much more stringent CPU and memory contraints than a typical notebook/desktop.

[utu]

@ kl522

I would like to sign on to krishna's post #5 and
suggest Klaus K. probably has good reasons for lagging
the squashfs effort. Among other reasons is Ubuntu
and Fedora are plowing this ground. KK is a one-man
effort AFAIK.

This is not to diminish your effort or Forester's,
merely to suggest that we keep our discussions relative
to computer metrics, and not spend any time on long-
distance prognostications on ulterior motives.

Foresters decompress times on 'a machine at work'
seemed attractive. What were its parameters?
How does that machine compare to my Laptop/SDCard rig?

[kl522]

Since you use the word "probably", so it is belief-based. Period.

The time difference in compression is fully explanable, as I have already mentioned.

Likely Forester did not use lzma compression for squashfs. Lzma has the behaviour that it takes a long time to compress but decompresses very fast. When he uses '-b' for cloop, that will result in using lzma and gzip. The gzip time is insignificant compared to lzma.

But the thing is even without lzma, squashfs results in smaller image - if that experiment carried out by Forester is correct. ( In my posts sometime ago, I use gzip for both cloop and squashfs, it also proved squashfs has smaller image). Once you use lzma-squashfs, you will see about another 20 % smaller image.

[Forester]

Ladies, gentlemen, please. "Calm down. It's only a commercial". One set of results does not prove anything and should not be used to jump to conclusions.

The improvement in boot time at work does not prove that squashfs is faster than cloop. It is an unexplained side affect. The slow part of the boot is the udevprobing. Why should that be hitting the compressed file system ? After the green bar has gone as far as it will, the spinner shows the system is still working. It spins for a long time with cloop but not with squashfs. Comments in /etc/init.d/knoppix-autoconfig suggest to me the boot is waiting for i/o activity to die down. What i/o I don't know - the VirtualBox console indicator shows no USB i/o at this time.

I did say (perhaps not clearly) that I used the Squeeze version of mksquashfs, which depends (have a look on the Debian repository web-site) on a gzip library, not an lzma library. Ergo, I used gzip compression. The same web-site shows the Sid version of mksquashfs, depends on several compression library packages, including liblzo2-2, which supports lzma and lzma2 compression.

kl522 says that Linux kernel 2.6.38 contains lzma compression but I am using Knoppix 6.4.3, which runs atop Linux kernel 2.6.36. I infer that had I used lzma compression, I would not have been able to boot my squashed file system.

There are some that say cloop requires more memory than squashfs. Perhaps, I don't know but the arguments for this that I have read so far appear specious to me.

Linux manages the disk cache. If memory isn't required for anything else, Linux will use it to cache disk contents but as soon as memory is required for something else it will free up disk cache. This is why, with two otherwise identical machines, the one with the more memory will appear to run faster. It is also why, at the time, Linux 'ran faster' than Windows 98.

The disk cache management is independent of squashfs and cloop. Because cloop is a loop device, data gets cached twice - once before and again after decompression (also true of knoppix-data.img but without the decompression). This is given as 'proof' that cloop requires more memory than squashfs. Poppy-cock.

Somehow, somewhere, squashfs must be buffering (aka caching) data before decompression. If its cache is too small, it will have to 'hit the disk' more often. Might use less memory, but that would make it slower.

Block size might be more significant and might give different results for different users. I used 64 Kb blocks from the cloop example on the Wiki. The squashfs man page says its default is 128 Kb. kl522's examples appear to be using 256 kb.

Now, if you are starting up mega Windoze-like applications that require tens of Mb to display an OK button, big blocks are probably going to make your app start-up faster. How often do you start these programs ? Once per session, so you don't need the disk cache.

If you are a sentimental old Unix fuddy-duddy who is reluctant to say goodbye to the power and productivity of the old command line interface, you want the disk cache to cache the commands you use a lot. The very idea that typing 'pwd' might cause squashfs to go off and read a 256 kb block and decompress it into a memory 'block' twice that size in order to run a program of only 25 kb in size is embarassing.

[Forester]

mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX

There is a problem with the patches used to mount the squashfs file system.

With the vanilla Knoppix, the KNOPPIX and KNOPPIX-DATA file systems are mounted on /dev/cloop0 and /dev/loop0 respectively. Code elsewhere in the /init script 'knows' that KNOPPIX-DATA is on /dev/loop0.

With squashed Knoppix, both file systems are on loop devices. The -o loop=/dev/loop1 attempts to ensure that KNOPPIX is /dev/loop1, leaving /dev/loop0 available for KNOPPIX-DATA in order to be backwards compatible.

This does not work as df -h on a squashed Knoppix systems shows. Why ? The /init script runs BusyBox mount, not the ordinary mount we all know and love. The output of /bin/busybox mount --help includes:


-o OPT:
loop Ignored (loop devices are autodetected)The only solution is to 'fix' all the references to /dev/loop0 throughout the /init script.

But it works for me so where's the problem ? Long may it continue to do so. If you are using an ordinary persistent store I think you're OK: /dev/loop0 is only referenced on error paths. If you try to use an encrypted persistent store I think you might be in trouble.

Attached to this post are two patches: generalise.txt fixes the /dev/loop0 problem; squashfs.txt implements the boot squashed-Knoppix feature that is the proper subject of this thread.

If you have already implemented a boot squashed-Knoppix feature using either kl522's or dinosoep's examples as a guide, you don't need squashfs.txt - that's just my implementation of the same thing.

If you look at generalise.txt you may think it looks way too complex for what it is. There is a reason for this. Many of the lines that needed changing are also lines that needed changing for the implementation of the knoppix_data cheat code. In generalise.txt is a combo-patch that meets the needs of both squashed-Knoppix and (a re-issue) of the knoppix_data cheat code patch.

[Forester]

I don't want to get into the war of the compression speed comparison ...


You want just wanna be an innocent bystander who lobs the odd grenade from time to time.



Which is more established compression method ?

Maybe my knowledge is skewed but here is what I know :-

1. Squashfs has long existed if not as long as cloop compression.
2. Squashfs is accepted into stock kernel a few versions back but cloop is still an external patch.
3. You can find squashfs in almost all embedded devices, including your typical home routers, home ADSL modems, home media players, home appliances such as TVs and so on. If you count the number of seats ( linux OS ) uses squashfs compared to cloop, squashfs is many many times more widespread than cloop..
4. Just run this utility on the compression file system :-
[code]


Are you in embedded software development ?

My screwed knowledge is that the vast majority of embedded devices are too dumb to have either an operating system or a filesystem while high-end house hold gadgets like games consoles, smart phones and iWhatevers don't run Linux. There is a large class of low-end embedded systems run the [appalling] uClinux which don't use [I]squashfs but use cramfs instead, which is so bad that you would have to be a lot more than merely 'sentimental' to prefer it. The assertion that you can find squashfs in almost all embedded devices is your first grenade.

For those who know nothing of embedded systems, you've usually got flash but it is not the same as USB or picture card flash. It's mtd flash and you might prefer a jffs2 file system for your persistent store as it does wear levelling. Depending on whether your flash is nand or nor and whether your upgrades are slow 'over the air' jobs or a quick updates from CD or USB, you might or might not put all your applications on the jffs2. That's what my last lot did, bless them.

You've still got the equivalent of minitrt.gz but usually it is embedded in the Linux kernel image. This is the initial root file system that Knoppix uses to get going. It uses it to mount the KNOPPIX image, your knoppix-data.img, join the two into a UNIONFS and then chroot to the UNIONFS leaving minirt behind like the first stage of a moon rocket.

The Knoppix (and desktop systems in general) minitrt.gz is a minimal Linux 'cos desktop distributions have to work on lots of different hardware. In embedded systems you often tailor the kernel and minirt.gz for a particular system and just use that. You don't need a second stage KNOPPIX image, just a minirt.gz and a knoppix-data.img. So using embedded systems in a discussion about compression methods for the KNOPPIX image is grenade number 2.

My last lot had a box with lots of option cards. The option cards ran uClinux 2.6.20 with cramfs for a root file system, the box ran Linux 2.6.20 with squashfs but not because it came with the stock kernel. The kernel was too old and management had no intention of upgrading just because. So even if the majority of embedded Linux devices use squashfs (and that may be so), they don't do it because squashfs is mainline, rather squashfs is now mainline because its use is so widespread. Grenade number 3.

I was tasked with getting rid of uClinux and cramfs. It took a recent stock kernel, built a initramfs with BusyBox and a few extra packages and I had a working system. Yes, the entire root file system was in RAM but there was plenty of it and this fixed many of the performance problems. An embedded system with no squashfs. In fact, no compressed file systems at all.

Both box and cards either used the old, deprecated initrd (don't think so, no /initrc) or used an initramfs arrangement but with squashfs or cramsfs instead of a compressed cpio image to be read into a tmpfs. Either way, once again, not what I think most people think of as 'stock kernel'.

Time for me to toss few grenades back (I'm a non-combatant too).

The Wikipedia page on initrd and initramfs has a description that makes the old initrd sound a lot like cloop in design. KK was using cloop for his iso image before Debian tried using cramfs with initrd. I don't think Debian ever tried squashfs. Probably squashfs was still 'in development' at the time.

The squashfs web-site freely admits that its mainlining was only possible because the Consumer Electronics Linux Form paid for it. Or in other words, big companies suddenly realised their products and reputations were dependent on a ropely old bit of free software maintained by some kernel developer wanabee. :) No, I'm sure Phillip Lougher is a very nice bloke and a good egg and all that but you can see why big companies would not want to be dependent on something maintained, in his spare time, by an eccentric German university professor. Besides KK cannot be bought.

Finanlly, look up cloop on the Debian package web-site. It's available. It's not maintained by KK. It is available for all platforms. Unstable sid even has a port for m68k. So someone out there is using it with Debian on all sorts of platforms. It's not Knoppix 'cos was just i386 (and only recent amd64). So someone knows something they are not telling us.

[utu]

Austrian, I think.

[Forester]

Austrian, I think.

The professor, yes, the university, no.

[DirkS]

mksquashfs /mnt/knx/source/KNOPPIX /mnt/knx/master/KNOPPIX/KNOPPIX.sq -b 262144 -noappend
...



mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
-hide-rr-moved -cache-inodes -pad /mnt/knx/source/KNOPPIX \
| nice -5 /usr/bin/create_compressed_fs - 262144 > /mnt/knx/master/KNOPPIX/KNOPPIX

Well, this already shows the fundamental difference: SquashFS is a filesystem on its own, it reads a directory structure with a bunch of files from someplace and compresses them while putting them into a container.

Cloop, on the other hand, does not care about the underlying filesystem. It provides a sort of "compressing shell" around anything that is thrown at it. It could even be a cpio archive or a tar file.

With cloop you could do the following:
- Make a full bitwise backup copy of any partition, including boot sector and whatnot, using dd.
- After compressing it with cloop, you get a rather small image that provides read access while "staying compressed" and still contains everything the original partition did, true to the bit.

You cannot do that with sqashfs, you only get the part that is underlined above. The italic part will never be possible with sqashfs.

It could be arguable whether the possibility to have a bit-copy of the original filesystem at hand has any value in the use case of a live-CD/DVD. Maybe not. But I believe that cloop has some value above that: It is a general compressor/decompressor and, in the special case of a mountable filesystem, provides read access to the compressed thing (plus providing write protection by design).

Therefore, for me it would be a major pity to drop cloop altogether.

Have fun :)
Dirk

[kl522]

It could be arguable whether the possibility to have a bit-copy of the original filesystem at hand has any value in the use case of a live-CD/DVD. Maybe not. But I believe that ....

You are not even convinced yourself and yet you are trying to convince us. And finally you use the word 'believe' .....

You should give us an example of what you described is useful in the context of a live-CD/DVD which the purpose is to squeeze has many programs as possible into the distro.
And that better be something cannot be done by squashfs.

And finally I said before, if Klaus is serious about wanting to continue to use cloop, he should work on including his patch into the mainstream kernel.

[DirkS]

You are not even convinced yourself ...Well, I am convinced that my use case can only be satisfied by cloop, never by squashfs. That's why I don't want to loose it altogether.

In terms of a live-CD/DVD: If space efficiency is the killer argument, squashfs is obviously the best. Maybe you can use it even without encountering problems that don't show up with cloop. Maybe there are good reasons to not use it for Knoppix, I don't know. Also I don't remember having ever remastered a Knoppix DVD, I used cloop for something else. So... *shrug*


if Klaus is serious about wanting to continue to use cloop, :D Klaus Knopper is the one who decides whether to continue to use cloop or switch to squashfs, not you.


he should work on including his patch into the mainstream kernel. I'm on his side about this. The patch is easily added, the so it's not worth the fuss. It works like this since years, so why put a lot of extra effort on oneself, for very little benefit?

(Besides, things of much higher general interest than squashfs or cloop have been rejected by the kernel developers, like Con Kolivas' BFS - but meanwhile some distributions patch their default kernels with it, even though it's not in mainline and most likely never will be. (And since I use BFS myself for several years now, I can tell you it makes a noticeable difference.)
Being in mainline is not a sign of quality whatsoever, it just means that the developer managed to win a fight of some sort. )

[Capricorny]

Well, I am convinced that my use case can only be satisfied by cloop, never by squashfs. That's why I don't want to loose it altogether.

In terms of a live-CD/DVD: If space efficiency is the killer argument, squashfs is obviously the best. Maybe you can use it even without encountering problems that don't show up with cloop. Maybe there are good reasons to not use it for Knoppix, I don't know. Also I don't remember having ever remastered a Knoppix DVD, I used cloop for something else. So... *shrug*

:D Klaus Knopper is the one who decides whether to continue to use cloop or switch to squashfs, not you.

I'm on his side about this. The patch is easily added, the so it's not worth the fuss. It works like this since years, so why put a lot of extra effort on oneself, for very little benefit?

(Besides, things of much higher general interest than squashfs or cloop have been rejected by the kernel developers, like Con Kolivas' BFS - but meanwhile some distributions patch their default kernels with it, even though it's not in mainline and most likely never will be. (And since I use BFS myself for several years now, I can tell you it makes a noticeable difference.)
Being in mainline is not a sign of quality whatsoever, it just means that the developer managed to win a fight of some sort. )

I'd like to expand a little on this:

1. I don't think an ordinary comparison of cloop and squashfs is very fruitful - different tools, for different, but in the case of Knoppix overlapping, sets of jobs. I also think it is a bad idea to scrap cloop altogether, it seems to be completely(?) file system agnostic, that can be extremely useful.

2. For quite a few uses, among them the practically important remastering of USB or poor man's HD installs, squashfs may have an edge. While being in mainline is surely not a quality sign per se, in my view it offers some guarantee against being too bad for too long time. That may in fact be relevant in changing times. But the most important thing, is that using squashfs, we can use new or alternate kernel versions without having to update cloop. And while it is by no means a pure Knoppix project, cloop is a tool for special purposes, and will probably stay that way, as squashfs in many 'standard' cases will look like the natural default choice.

3. I think Knoppix has not got the use it deserves, and one of the reasons is clearly the development model, with Klaus Knopper "The Upstream Vendor" (TUV) and very little basic development being done downstream. What we are discussing, is different alternatives for system solutions, and the existence of different alternative solutions can only be a good thing. While I think it may be true that the Knoppix project would benefit from switching to squashfs, I don't think it is necessary now or in the near future.

4. I think Knoppix would benefit from a development process more analogous to kernel development, where Klaus Knopper is the Linus T equivalent of "benevolent dictator", free to include or reject different kinds of "patches" for inclusion in the basic Knoppix package.

5. Creating a squashfs-based derivative with a more specialized package selection (e.g. a hybrid of standard Knoppix and Quantian) could be a good way of experimenting. It could also have some standard non-free software pre-installed, like vmware Workstation and Oracle XE, subject to activation (e.g. by license number). Non-free software is one of the facts of life in many professional uses fo Linux.

[dinosoep]

I have a litle problem:
today I ditched up my usb with knoppix on just for fun and messed around with it.
Don't ask me how but I ruined my minirt.gz
I have no linux on my laptop installed so I can't patch it myself again and this is needed as my usb install uses squashfs :p

Can someone share his?

[Werner P. Schulz]

... please contact me; I can send you minirt.gz from 6.4.4

listings@wp-schulz.de

[dinosoep]

thanks a lot werner.
for the people who want squashfs without problems(or with the same issue as me :p) go here : http://dl.dropbox.com/u/15024434/initrd.gz

[Capricorny]

It seems that Klaus K prefers cloop because of versatility and well proven good properties. But that's of course "past experience", not really present. He has his good reasons, but I think testing out squashfs should be encouraged, as eventual problems and improvements will have a much broader impact than for cloop, and fixes therefore are more likely to happen.

We should aim for a clean extension of the mounting function in minirt init, to cope with squashfs. Klaus K is of course free to reject such a patch, but if we prove it to be useful, I don't think it will be rejected for very long. (Reminiscent of Kanotix forked off because of reluctance to HD installs... not much later we had 0wn...)

When running cloop compression without optimization, it goes almost as fast as squashfs, so compression time isn't that much of an issue. But, the resulting difference in compressed size amounts to about the last GB of programs stuffed into memory from a DVD size image, and that is practically relevant. With an ever increasing footprint of the basic system functions and utilities, it tends to get more rather than less relevant with time, too. I see this very clearly with my pure 64-bits version of 6.4.4.

In the 64-bits context, it should be noted that there have been some adaptation problems with cloop, and I'm not quite sure they are alle honed out by now, Klaus' latest cloop patch being from July 6, and it seemed to make some difference for 64-bits remastering. Where I can still not make busybox (1.1.17, amd64, Debian package, 1.1.18 compiled from source doesn't run at all) get the mounting right, even if I can do it manually in debug mode. I should add that using a freshly compiled 2.6.39.2 64 bits kernel did not help for this bug, only a few others :)

I'm going to try the squashfs alternative here, and if that goes through, it will be a strong argument pro squashfs, I think. I would also like to add that the way cloop is used in current Knoopix, it's really not file system agnostic, rather we have a (harmless at best) step via isofs, which introduces another set of potential problems. Maybe small and/or irrelevant, but most likely, not empty.

We should, at least, know what may, eventually, be to learn from other successful live distros, in particular Debian-based. Like Ubuntu live and Kanotix. BTW, I looked at the init for the Debian 6.0.1 live CD showcasing LXDE - think Knoppix is far better, and it wasn't a very good user experience either - I just needed it as a starting point for "pure" 64-bits Knoppix.

[Capricorny]

In another post, I have uploaded and documented a minirt init modification handling cloop and squashfs transparently for the user, just using cheatcode "squashfs". It's for pure 64-bits, but the modification for 32-bits is trivial.

IMHO, present day squashfs seems to be about as stable, efficient and versatile as we need for this kind of use, and its widespread and growing use indicates others think the same.

So, with a "uni-minirt", KNOPPIX may still be distributed as cloop image, but subsequent remasterings will not have to unpack minirt.gz to make modifications in order to turn to squashfs. I think lots of remasterers will prefer squashfs, as it gives (at least) slightly simpler and faster workflow with at least as good results. OTOH. I can't see anything right now forcing us to use squashfs.

And, just for the record: This is not about the practicality or usability of cloop per se, but about the actual Knoppix use case.