Page 1 of 4 123 ... LastLast
Results 1 to 10 of 41

Thread: Squashfs-ed knoppix

Hybrid View

  1. #1
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423

    Squashfs-ed knoppix

    As I learn more about knoppix, I am beginning to form this idea of using squashfs instead of cloop. This idea is not new. Apparently there is a distro already doing it, finnix. But why use squashfs instead of cloop :-

    1. Squashfs is already in the kernel. There is no need to maintain a separate cloop module source code, which can potentially be incompatible with newer kernels.

    2. I ***THINK*** squashfs dynamically uses a memory chunk for decompression, whereis cloop allocates a huge chunk ( currently at least 1G ) of memory to store the uncompressed image. According to finnix, it needs only 192MB, so squashfs is more memory efficient.

    I guess I will try out this idea of one of these days.

  2. #2
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Read this for the claim about knoppix needing at least 1 GB of memory :-

    http://www.finnix.org/Frequently_Ask...and_Knoppix.3F

    While at it, I am still very uncertain if the claim made of finnix auther is true. Nevertheless I found that knoppix 6.2 mounted a 1G tmpfs, but the tmpfs is only used if there is no persistent store. When using persistent store, the tmpfs is not used, but it's remained mounted.

  3. #3
    Moderator Moderator
    Join Date
    Jan 2010
    Location
    Asheville, NC, USA
    Posts
    528
    I suppose for people with lots of processor cycles to spare, it might make sense. Squashfs IS used, AFAIK, in the CD version of Knoppix. But for the many who have less CPU and more disk space (for swap), it's probably better to use cloop. Let us know what you determine experientially!

    Cheers!
    Krishna

  4. #4
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    I think the claim that cloop needs at least I G memory is likely to be over-stated. A quick check on cloop source code revealed that it also decompresses the file system on-the-fly basis.

    The 1G of /ramdisk mounted, used when one does not have persistent store, is something which the actual usage will grow as it is used. The 1G /ramdisk is mounted, but not used when there is a persistent store. And therefore even though that's a "bug", it does no harm.

    In terms of CPU cycles, there is no apparent difference between squashfs and cloop, because both uses the same "on-the-fly" decompression and buffered blocked strategies, and also same compression algorithm.

    Therefore the only different between squashfs and cloop is that squashfs has long been accepted into the kernel, it needs no extra maintenance. Cloop is currently maintained separatelly by the author of KNOPPIX.

  5. #5
    Moderator Moderator
    Join Date
    Jan 2010
    Location
    Asheville, NC, USA
    Posts
    528
    Quote Originally Posted by kl522 View Post
    I think the claim that cloop needs at least I G memory is likely to be over-stated. A quick check on cloop source code revealed that it also decompresses the file system on-the-fly basis.

    The 1G of /ramdisk mounted, used when one does not have persistent store, is something which the actual usage will grow as it is used. The 1G /ramdisk is mounted, but not used when there is a persistent store. And therefore even though that's a "bug", it does no harm.

    In terms of CPU cycles, there is no apparent difference between squashfs and cloop, because both uses the same "on-the-fly" decompression and buffered blocked strategies, and also same compression algorithm.

    Therefore the only different between squashfs and cloop is that squashfs has long been accepted into the kernel, it needs no extra maintenance. Cloop is currently maintained separatelly by the author of KNOPPIX.
    I don't think Klaus would have become the maintainer of cloop for anything less than a substantial reason; too much work for no real benefit. I don't know what that reason is, but I know it must exist. The "phantom GB of ramdisk" aspect of cloop is probably intentional, perhaps to make possible very fast growth of the actual filesystem, or some such. Regardless of all that, if one had some reason to avoid using the compressed filesystems (perhaps speed?), how difficult would it be to leave that out in a re-master? Just curious, and I thought one of y'all might have the answer.

    Cheers!
    Krishna

  6. #6
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by krishna.murphy View Post
    I don't think Klaus would have become the maintainer of cloop for anything less than a substantial reason; too much work for no real benefit. I don't know what that reason is, but I know it must exist.
    That's what we call "belief", it's not based on facts.
    I think it's all historical. Cloop existed long time ago. Klaus is not the creator. He is just the maintainer. He needed it at that moment. So now nobody else except him has "vested" interest in it. He is using it simply because he is familiar with it. Kind of sentimental value thingie. As a matter of fact, it's a easy swap to use squashfs.

    The "phantom GB of ramdisk" aspect of cloop is probably intentional, perhaps to make possible very fast growth of the actual filesystem, or some such.
    I believe that you said this because you did not look at the code. The ram disk has nothing to do with cloop. The ram disk is created for unionsfs with cloop, to allow users to have write access to the otherwise read-only cloop. And it's left mounted unused when the user has a persistent store. If you look at the code you will agree with me, it's a bug.

  7. #7
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423

    cloop and squashfs comparison

    Below is a comparison of some of the interesting aspects of squashfs vs cloop when running on the ***SAME*** machine with 4G physical memory. Some details have been snipped for clarity :-

    When using cloop kernel driver :-
    Code:
    # df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2              94G   33G   61G  36% /mnt-system
    /dev/cloop            4.0G  4.0G     0 100% /KNOPPIX
    /dev/loop0            2.4G  940M  1.5G  39% /KNOPPIX-DATA
    ....
    
    # cat /proc/meminfo 
    MemTotal:        3588864 kB
    MemFree:         3005164 kB
    Buffers:          112860 kB
    Cached:           303956 kB
    SwapCached:            0 kB
    Active:           220000 kB
    Inactive:         294008 kB
    ....
    
    # ls -al /mnt-system/KNOPPIX/KNOPPIX
    -rwxrwxrwx 1 root root 1546036015 2010-06-16 22:36 /mnt-system/KNOPPIX/KNOPPIX
    Here is the corresponding information when using squashfs :-
    Code:
    # df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2              94G   33G   61G  36% /mnt-system
    /dev/loop0            1.4G  1.4G     0 100% /KNOPPIX
    /dev/loop1            2.4G  939M  1.5G  39% /KNOPPIX-DATA
    ....
    # cat /proc/meminfo
    MemTotal:        3588864 kB
    MemFree:         2927872 kB
    Buffers:          178880 kB
    Cached:           329248 kB
    SwapCached:            0 kB
    Active:           187076 kB
    Inactive:         414508 kB
    ....
    # ls -al /mnt-system/KNOPPIX/KNOPPIX.sq
    -rwxrwxrwx 1 root root 1453662208 2010-06-16 23:09 /mnt-system/KNOPPIX/KNOPPIX.sq
    Summary :-
    1. cloop registers the need to use 4G of virtual memory on /dev/cloop, whereas squashfs registers the need for 1.4 G of virtual memory on /dev/loop0

    2. squashfs compressed image is smaller by 100M when using the same compression algorithm and block size.

    Cheers.

  8. #8
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Following what I have posted previously, I noticed that the kernel 2.6.34 has included the LZMA support for squashfs, it will be a matter of small delay, you will also see the squashfs-tools to OFFICIALLY include LZMA support. Actually squashfs using LZMA has existed a long while ago, and has been actively used by the embedded systems ( particularly OpenWRT ) so there is nothing so new about this.

    Cloop also support LZMA compression.

    As for me, LZMA is not particularly useful to me, because the compression takes too long. However, for a distro or a LiveCD, it is something particularly useful, meaning for the same amount of space ( say a CDROM ), you can squeeze in additional 10-20 % of additional programs into it.

  9. #9
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Have you made any more observations using squashfs with Knoppix? If there are no clear problems or potential drawbacks, I think, if only for the sake of uniformity, we should encourage the use of squashfs. My impression from what I have read is similar to your observation: In practice, the difference isn't that big, but having the necessary modules in the standard kernel makes things a bit easier.

    I'd like to make a remastering now, with updated packages and using squasfs, to have an updated system with everything needed and a 4GB persistent image on a 8GB flash drive. Now I need 16..

  10. #10
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    As you have noticed, I tried out the idea of squashfs. Eversince, I have been using it without a single problem. There is actually no downside, but there are a few ( perhaps not very significant ) upsides. For summary, they are :-

    1. stock kernel module (vs patched module ).
    2. less virtual memory required.
    3. slightly better compression.

    To create the compress image, this is what I use :-
    Code:
    mksquashfs /mnt/knx/source/KNOPPIX /mnt/knx/master/KNOPPIX/KNOPPIX.sq -b 262144 -noappend
    For comparison, this is what I use for cloop image :-
    Code:
    mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
    -hide-rr-moved -cache-inodes -pad /mnt/knx/source/KNOPPIX \
     | nice -5 /usr/bin/create_compressed_fs - 262144 > /mnt/knx/master/KNOPPIX/KNOPPIX
    You notice that for both cases, I don't sort the files according to 'frequent use', but that does not mean it is not important. I just don't bother to do it.

    Last but not least, I have modified the initrd.gz to mount squashfs accordingly :-

    Code:
    indknoppix(){
     local dev
     for dev in "$@"; do
      [ -b "$dev" ] || continue
      message -n -e "\r${CRE}${BLUE}${SEARCHINGFOR} $DISTRO in: ${MAGENTA}$dev${NORMAL}   "
      trymount "$dev" /mnt-system >/dev/null 2>&1 || continue
      if [ -r /mnt-system/"$knoppix_dir"/KNOPPIX -o -r /mnt-system/"$knoppix_dir"/KNOPPIX.sq ]; then
       message -e "\r${CRE}${GREEN}$DISTRO ${FOUNDAT}: ${MAGENTA}$dev${NORMAL}       "
       return 0
      fi
      umount /mnt-system
     done
     return 1
    }
    ....
    mountknoppix(){
     local k dev dir count=0 RC=0
     [ -b /dev/cloop ] || mknod m 644 /dev/cloop b 240 0
     [ -f /mnt-system/"$knoppix_dir"/modules/squashfs.ko ] && \
       insmod /mnt-system/"$knoppix_dir"/modules/squashfs.ko
     [ -d /sys/module/cloop ] || insmod /mnt-system/"$knoppix_dir"/modules/cloop.ko preload=32 || return 2
     [ -d /sys/module/aufs ]  || insmod /mnt-system/"$knoppix_dir"/modules/aufs.ko || return 3
     if checkbootparam squashfs && [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] ]
     then 
       mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
     elif [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] -a \
       ! -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] ]
     then
       mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
     else
     for k in /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] \
              /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx][0-9]; do
      .....
    I retained the cloop booting capability for comparison purposes. When both images are present, I use a cheatcode to select the preferred one. It was kind of elaborate but now I can safely say, it is unnecessary to retain the cloop image at all.

    Have fun.

Page 1 of 4 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Vintage 5362 IBM System/36 Mini-Computer Mainframe 5291 2, CRT Terminal DM12N501 picture

Vintage 5362 IBM System/36 Mini-Computer Mainframe 5291 2, CRT Terminal DM12N501

$349.99



IBM Type 4869 External 5 1/4in Floppy Disk Drive Mainframe Collection - UNTESTED picture

IBM Type 4869 External 5 1/4in Floppy Disk Drive Mainframe Collection - UNTESTED

$75.00



Vintage Hewlett Packard HP 2100S Microprogrammable Computer System Mainframe #2 picture

Vintage Hewlett Packard HP 2100S Microprogrammable Computer System Mainframe #2

$1499.99



Rare Vintage Texas Instruments Silent 700 Data Terminal + phone hookup picture

Rare Vintage Texas Instruments Silent 700 Data Terminal + phone hookup

$145.00



1 lot of 8 vintage IBM 9316s, picture

1 lot of 8 vintage IBM 9316s,

$150.00



National Semiconductor SC/MP Low Cost Development System w/ Cards Manuals picture

National Semiconductor SC/MP Low Cost Development System w/ Cards Manuals

$975.00



NOS Vintage CRAY X-MP  SCALE MODEL Supercomputer Black Brown picture

NOS Vintage CRAY X-MP SCALE MODEL Supercomputer Black Brown

$347.60



Vintage Apple Mac Macintosh 512K Case & Working CRT Only Chasis Mainframe picture

Vintage Apple Mac Macintosh 512K Case & Working CRT Only Chasis Mainframe

$99.99



Sage MAS 90 Software for Windows Financial Reporting Mainframe Collection picture

Sage MAS 90 Software for Windows Financial Reporting Mainframe Collection

$360.99



NCR Mechanical Keyboard USB Wired 01027100 Beige Mainframe Collection picture

NCR Mechanical Keyboard USB Wired 01027100 Beige Mainframe Collection

$290.99