Re-master your flash installation

[kl522]

The title is kind of confusing. If you have already installed knoppix on flash, and you
have enabled a persistent store, meaning you could go ahead with adding, upgrading
and customizing packages, why do you still need to re-master knoppix ?

Reasons as follows :-

1. When you upgrade your files/packages, the original copy resided on the
read-only compressed image is still there. The system exposes you to the ones on
the persistent store, and hides away the ones on the compressed image.

This is a waste of space, if you have upgraded many files/packages, for
every file/package in the persistant store, there is a counterpart in the
compressed image. Re-master will delete the hidden copies.

Even when there is no compressed image counterpart, due to you added some
new interesting packages, by moving your added packages to the compressed
image, you also free up the (uncompressed) space on the persistent store.

2. Another reason to re-master is that you want to maintain a clean separation between
experimental things verses the "stable" portion of your customization, ie you want to
move the stable and proven things to the compressed image, so that you can later
proceed to add your experimental things to the persistent store. In case you
are not happy with the new experimental things you added, you can always erase it
from your persistent store, or even you can wipe out the entire persistence store.
If you have previously re-mastered your flash installation, deleting things from the
persistent store is an easy, clear-cut and safe operation; because even if you delete
everything, you will still have the last snapshot of a fully working, stable and proven,
and yet already fully customized version of knoppix to start with. There is no need
to falling back to original uncustomized knoppix.

The end result of a flash re-master is not a new copy of CD or DVD, as you have already
booted off from flash, CD or DVD is irrelevant. The end result of flash re-master is just a
new copy of compressed image KNOPPIX.

This is how I re-master my flash installation, in brief. Note there is no "chroot" needed :-

Find a directory or a partition where you have a few GB of space :-
# mkdir -p /lots_of_free_space/source/KNOPPIX
# mkdir -p /lots_of_free_space/master/KNOPPIX
# cd /lots_of_free_space/source/KNOPPIX
# cp -a /dev .
# cp -a /UNIONFS/bin .
# cp -a /UNIONFS/etc .
# cp -a /UNIONFS/lib .
# cp -a /UNIONFS/sbin .
# cp -a /UNIONFS/usr .
# cp -a /UNIONFS/var .
# cp -a /UNIONFS/opt .
# cp -a /UNIONFS/home .
# cp -a /UNIONFS/root .
( Whatever you want it to appear in the new compressed image, you copy it there )
.....

# cd ..
# mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
-hide-rr-moved -cache-inodes -pad /lots_of_free_space/source/KNOPPIX \
| nice -5 /usr/bin/create_compressed_fs - 262144 > /lots_of_free_space/master/KNOPPIX/KNOPPIX

After a long time, you will get a compressed image at /lots_of_free_space/master/KNOPPIX/KNOPPIX,
which you can use it to replace your /mnt-system/KNOPPIX/KNOPPIX.

You can see that the steps above are quite simple. This last step of replacing existing KNOPPIX
compressed image must be perform "offline", ie you should perform it when you are not
using /mnt-system/KNOPPIX/KNOPPIX.

There is a challenge, because once you have booted in knoppix, you are already using
/mnt-system/KNOPPIX/KNOPPIX. So either you boot from a CD with your flash detached
and then put the flash back after you have booted so that you can overwrite it, OR you
could configure /mnt-system/boot/syslinux/syslinux.cfg to add the cheatcode "debug"
in it. At the first shell prompt, the /mnt-system/KNOPPIX/KNOPPIX is not mounted yet,
this is when you could manually perform this somewhat "risky" task.

Basically the steps will be roughly like this :-

If your free space which contains the compressed image resides on a separate partition :-
# mount /dev/sdaX /lot_of_free_space
If you have enough space, you can rename KNOPPIX for backup purposes :-
# mv /mnt-system/KNOPPIX/KNOPPIX /mnt-system/KNOPPIX/KNOPPIX.bak
Now overwrite it with the re-mastered copy :-
# cp /lots_of_free_space/master/KNOPPIX/KNOPPIX /mnt-system/KNOPPIX
You can skip mounting knoppix-data.img by renaming it,
# mv /mnt-system/KNOPPIX/knoppix-data.img /mnt-system/KNOPPI/knoppix-data.img.bak
If using the cheatcode "debug", once done, you type "exit" to proceed with the rest of booting
to verify that you new image is working. You will have to exit 6 times before it will be booted
completely.

If you are happy with it, you can shutdown and boot another time into "debug" shell, this time
you can "restore" your knoppix-data.img and free up the space in there :-

# mv /mnt-system/KNOPPIX/knoppix-data.img.bak /mnt-syste/KNOPPIXknoppix-data.img
# mkdir /tmp/tmp
# mount -o loop /mnt-systsem/KNOPPIX/knoppix-data.img /tmp/tmp
( Now you can delete whichever files you want to delete to free up the space, as
they have already been transferred to the compressed image ).
# cd /tmp/tmp
# rm -rf bin sbin lib etc var usr .......

Once you have finished :-
# cd /
# umount -d /tmp/tmp
Again, if you are using "debug" cheatcode, you have to type 'exit' 6 times before you
will be booted completely with the freed-up persistant store.

In any case, the steps above involved will be somewhat tricky to the Linux newbies. I wish there
is a cheatcode called "upgrade=/dev/sda4:/master/KNOPPIX/KNOPPIX" which will automatically
perform the upgrade. I will leave this as an exercise for those interested to do so by modifying minirt.gz.

Last note: the "debug" cheatcode is a very useful thing. As you can use it to rename/move the
knoppix-data.img ( or knoppix-data.aes ) for the purpose of resizing.

[kl522]

# cd ..
# mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
-hide-rr-moved -cache-inodes -pad /lots_of_free_space/source/KNOPPIX \
| nice -5 /usr/bin/create_compressed_fs - 262144 > /lots_of_free_space/master/KNOPPIX/KNOPPIX

After a long time, you will get a compressed image at /lots_of_free_space/master/KNOPPIX/KNOPPIX,
which you can use it to replace your /mnt-system/KNOPPIX/KNOPPIX.


Maybe I was too agressive with the blocksize 262144 ? Most web site only recommends 65536.
I suspect blocksize 262144 might be related to some intermittent problem which I am facing.

[Capricorny]

I really appreciate that you share this! :)
The first rule about remastering is, don't do it. But it's not the last rule...

I would like to postulate a few more good (IMHO) reasons for remastering, I fully agree with yours.
3. Bloat. The DVD contains gigabytes (uncompressed) of programs we will probably never run. While some essentials (every man his own!) are missing.
4. Bugfixes made permanent, moved from "user space".
5. Extra package install/updating made more systematic.
6. Smaller total system footprint. Will always be relevant, if not so important economically as storage gets cheaper.
7. Possible to make several cloop images. Could be useful for things like heavy development tools.
8. Could even try other approaches than through cloop.

I imagine that a good way of doing the full KNOPPIX upgrade, would be to copy the newly made compressed image to, for example, an external USB HD, and do the "field testing" on that. Speaking for myself, I'm pretty sure I would want to make quite a few adjustments.

[dinosoep]

I created an account just for this tutorial.
I tried your method of remastering as I like the way it works.
But did however made the mistake of trying to do the source/master directory on a ntfs filesystem. this caused all persmissions to be lost while copying and after the remaster nearly everything was broken.
Of course I was going to rage on this tutorial :p when I realized it maybe couldve been because it was ntfs and took another memorystick and formatted it to an ext2 filesystem. this did the job :)

is there a way to not break anything while still copying to a ntfs filesystem?

[Forester]

dinosoep wonders ...


is there a way to not break anything while still copying to a ntfs filesystem? Well, yes there is. Have you noticed how Knoppix uses (compressed) loop devices a lot ? You can think of these as filesystems-in-a-file instead of filesystems-on-a-partition. You can use the same trick to create an Linux file system on an ntfs partition.

There are four steps:

1. Mount your ntfs partition under Knoppix (if not already mounted). That will probably be something like:


mount /media/sda12. Create a large file on it. How big depends but for remastering the Knoppix CD let's guess 4 Gb is enough. You need something like:


dd bs=4K count=1M if=/dev/zero of=/media/sda1/big.one3. Create a file system in the big file:


mkfs -t ext3 -F /media/sda1/temp/big.one4. Mount the file as a loop back device:


mount -o loop /media/sda1/temp/big.one /lots_of_free_spaceYou may need to sudo one or more of the commands and, of course, adapt the parameters to your particular situation. The mkfs command takes many parameters and some may be more useful in this situation than others.

[Harry Kuhman]

Well, yes there is. Have you noticed how Knoppix uses (compressed) loop devices a lot ? You can think of these as filesystems-in-a-file instead of filesystems-on-a-partition....

As a moderator, I passed the above post, as it is not spam and on topic.

However, as a user and someone who has been on this site for nearly 8 years and seen over and over again the reports of disaster after writing to NTFS, I couldn't just pass it and not warn others that it is not a universal opinion. Yea, you can create a file system within another file system under Linux. But to do so you still need to write to the physical file system. Not just to create it but during all of the writing processes. I know of no reason to think that this is safer than just writing to NTFS (which is unfortunately still not safe).

If you want another file system on your hard disk consider shrinking the main partition, or use that "back up" partition that the vendor wasted disk space to create (after making a backup disc), and create a second Linux or FAT partition there, or even just add another inexpensive hard disk. An extra partition can be a Linux partition or a FAT partition (A FAT partition will be subjected to the 4GB-1 file size but lets you transfer files easily between Windows and Linux). But if you write to a NTFS partition, have very good backups of everything and keep making backups, the effects of corruption are often not noticed for some time after the damage is done.

[kl522]

However, as a user and someone who has been on this site for nearly 8 years and seen over and over again the reports of disaster after writing to NTFS, I couldn't just pass it and not warn others that it is not a universal opinion. Yea, you can create a file system within another file system under Linux. But to do so you still need to write to the physical file system. Not just to create it but during all of the writing processes. I know of no reason to think that this is safer than just writing to NTFS (which is unfortunately still not safe).


This subject has been mentioned quite many times. 8 years of observation is quite a long time, especially in the world of IT. I am not sure how extensive your observation is substantiated with facts, as software version changes, some bad experience in the past with bugged software may not be applicable to newer versions of software.

The fact is over the recent years, the use of NTFS (for read and write access ) in Linux has got wide spread usage and penetration. If you look at embedded devices, quite many of them are Linux based, they read/write usb flashes and SATA harddisk. I came to know quite many of the media players are Linux based. I also know that Samsung TV is Linux-based too. These devices read/write on NTFS file system (and FAT32) !

I cannot help but to think that read/write of NTFS on Linux has gotten to a usable degree of safety - if one does not use those fancy features of NTFS. But if indeed your observation - substantiated with facts - does indicate that NTFS writing is still unsafe, perhaps it is a problem unique to Knoppix ? I am wondering ......

[Capricorny]

I think the creation of an image file, organizing a file system on it and loop-mounting that file system, may be relatively safe on NTFS. But, as NTFS is a rather swiftly and erratically moving object, I can't really understand how anyone can be quite sure everything will always go well. Even if everything works perfectly today, there is a chance that the next service pack, or even bugfix patch, may introduce problems. Therefore, I like to make a distinction between simple tasks like creating, updating and reading an image file, and more general file system use. In particular, if all reading/writing to the image file is to be done under Linux, one may mostly avoid inconsistencies etc between Windows and Linux NTFS implementations.

[kl522]

I agree with you that creation of an image file, organizing a file system on it and loop-mounting that file system is considered relatively safe - with a catch I shall explain below.

I was reading the ntfs-3g write up, they mentioned this in the Wiki :-


NTFS-3G supports all operations for writing files: files of any size can be created, modified, renamed, moved, or deleted on NTFS partitions. Transparent compression (http://en.wikipedia.org/wiki/NTFS#File_compression) is supported, but there is no support for encryption (http://en.wikipedia.org/wiki/Encrypting_File_System). Support to modify access control lists (http://en.wikipedia.org/wiki/Access_control_list) and permissions (http://en.wikipedia.org/wiki/File_system_permissions) is available.
From what I read, if you want to use NTFS-3g to fix access control list and permission on a Windows system, you might not succeed because of the catchy support of ACL and permission. However, if you want to implement what Knoppix needed, ie compressed loop mount a file on NTFS, it would appear to be safe.

On the other hand ......

I did mention before for Knoppix 6.2.1 ( which I did not check if it is still valid after that ), it is in my humble opinion that knoppix does not shutdown the persistent file system properly, more so for FUSE. If you look at the shutdown script, basically it did not manage to unmount the UNIONFS/aufs2. The FUSE user space driver is terminated before the persistent file system got unmounted. If FUSE keeps buffer, the buffer content is it flushed ? I am not sure. Specifically I noticed when using vanilla Knoppix 6.2.1 with a ext2 persistent store loop mounted on NTFS, I noticed the system had to fsck the persistent store, and each time there is inconsistencies and it requires manual intervention. That was Knoppix 6.2.1 anyway .....

[dinosoep]

sorry for replying this late, I hoped that knoppix forums would sent me a mail when someone reacted.
this was exactly what I was looking for :D
I have been writing from linux mint to a ntfs partition a lot and I got no problems whatsoever so I think writing is getting more and more stable to ntfs

[Forester]

Has this thread been renamed ? I like the new title. Lots. It is exactly what I was looking for.

Are the instructions for the CD or the DVD ? For the CD you may need /lots_of_free_space but for the DVD you need /super_lots_of_free_space. Like about 5 times as much so you've got to plan ahead.

I tried following the instructions with the DVD (6.4.3 EN). The 'main' command did not scale so well. It failed claiming I was out of virtual memory. The 'main' command (taken I see from http://www.knoppix.net/wiki/Knoppix_Remastering_Howto)


mkiosfs ... | create_compressed_fs ... > KNOPPIX/KNOPPIXis a pipe, which is usually good. Usually the right-hand side consumes data as fast as the left-hand side produces it. The pipe means no (large) temporary files are needed and it all runs faster as a result.

However, a look at the output of create create_compressed_fs -h, particularly the description of the -s flag and the stuff at the end, suggests the pipe is a bad idea: the right-hand side is going to buffer its entire input before it processes anything.

If you are remastering the CD, you might have enough real memory and/or enough swap (the HowTo describes how to create a 1 Gb swap) but for remastering the DVD you're looking at 8 Gb or so. I have no other use for an 8 Gb swap partition, so I split the main command in two and used a temporary file (a bit like using the -f option of create_compressed_fs).

The 'main' command examples all prefix the create_compressed_fs with nice -5. This too I find strange. The right hand side runs at a lower priority that the left, so you would expect data to pile up in the pipe even if you use the -s or -f options of create_compressed_fs. The right hand side takes a long time to run and uses a lot of processor. The intention of the nice -5 is run the compression at a low priority so you can do other 'stuff' while you wait. But it uses up a lot of memory too so doing other 'stuff' may not be all that great. Me, I left off the nice -5 and got in some zeds.

The instructions do cover in detail how to meet the challenge of getting the new KNOPPIX image onto the USB and how to clean up the contents of the old persistent store. However, I had to read it three times before it was clear that you don't have to do it the 'risky' bit booting Knoppix with the debug cheatcode.

So, for the benefit of those new to Knoppix and remastering, once you have created the new KNOPPIX image:
- shutdown your PC / laptop;
- remove the USB;
- boot your Knoppix Live CD (the one you installed to USB with the first place);
- insert the USB stick again;
- now follow the instructions.

Also for the benefit of those new to Knoppix and remastering:
- you should run all the commands from a 'root shell'
- if you don't, you'll have to prefix most commands with sudo
- even so, the command with the '>' may fail.

Those new to Knoppix and remastering may also be a little nonplussed by the suggestion that what they put on the new image is an entirely a free choice:
- update/remove/install software until you are happy/fed up/get a life;
- you don't want /home in the new image (that's your stuff)
- you don't want /var in the new image (stuff that changes all the time);
- take /home and /var from the old image
- it is a reasonable bet everything else should go in the new image
- until you know better
- when you clean up your knoppix-data.img, delete everything except /home and /var.

I working my way towards describing how much disk space you (might) need to remaster the DVD. To do that I'm going to have to show you how I did it. I do everything a directory created for the purpose and always write to a subdirectory. Fewer disasters that way. The subdirectories may be real directories or symbolic links or mount points (my favourite). That's not important at this stage. I put all the scary commands into a script so that next time I remaster, I won't have to type them in or copy/paste them from a web page. Fewer disasters that way.


#!/bin/bash

command=$1; shift;
operand=$1; shift;

case "${command} ${operand}" in
("copy source")
excludes="--exclude=/home --exclude=/lost+found --exclude=/var";
sudo rsync -ax --delete ${excludes} /UNIONFS/ knoppix_source;
sudo rsync -ax /KNOPPIX/home /KNOPPIX/var knoppix_source;
sudo cp -p /KNOPPIX/etc/fstab knoppix_source/etc;
;;
("make isofs")
sudo chmod 1000:1000 knoppix_tmpiso;
sudo mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" -hide-rr-moved -cache-inodes -pad knoppix_source > knoppix_tmpiso/knoppix.iso
;;
("compress isofs")
mkdir -p knoppix_master/KNOPPIX;
sudo create_compressed_fs -b -B 65536 knoppix_tmpiso/knoppix.iso knoppix_master/KNOPPIX/KNOPPIX;
echo "Now reboot to CD";
;;
("backup usb")
mkdir -p knoppix_backup/KNOPPIX;
cp -p knoppix_usb/KNOPPIX/KNOPPIX knoppix_usb/KNOPPIX/knoppix_data knoppix_backup/KNOPPIX;
;;
("copy master")
cp -p knoppix_master/KNOPPIX/KNOPPIX knoppix_usb/KNOPPIX;
;;
("mount data")
mkdir -p knoppix_data;
sudo mount -o loop knoppix_usb/KNOPPIX/knoppix-data.img knoppix_data;
;;
("prune data")
cd knoppix_data;
sudo rm -fr $(ls -ignore=home --ignore=var);
cd ..;
;;
("umount data")
sudo umount knoppix_data;
rmdir knoppix_data;
;;
(*)
echo oops;
;;
esac
So, meet the directories:

knoppix_source - where the filesystem for the new image is assembled. It has to be a Linux filesystem. It must be at least the size of /UNIONFS.
Budget 10 Gb.

knoppix_tmpiso - where the uncompressed iso image goes. It can't be a FAT filesystem as it takes a single file bigger than 4 Gb. The iso file is slightly smaller than its source.
Budget 10 Gb.

knoppix_master - where the compressed iso image (the new Knoppix image) goes. This can be a FAT filesystem, since the image will be copied to a FAT filesystem on USB. This also means it can't be bigger than 4 Gb (the original 6.4.3 is 3.5 Gb).
Budget 4 Gb.

knoppix_backup - back up the old Knoppix image and knoppix-data.img goes here. I'm using a 1 Gb knoppix-data.img. Again, can be any filesystem, even FAT.
Budget 5 Gb.

knoppix_usb - the USB installation that is being remastered is under here.
This already exists on the USB so ...
Budget 0 Gb.

That's 29 Gb total but if you use just one large Linux partition and delete stuff as you go along you can do it in 20 Gb. I don't know what you'd need to remaster the CD but I guess 4 Gb would be enough.

Where does all this disk space come from ?

I'm lucky in that I when I remaster I don't have ntfs between me and my hard drive. I only have lvm2 so I can whip up a new filesystem in less time that it takes to type this.

Many (me included) first try Knoppix because they want to try Linux but do not want to touch Windows. If you are one of those, my advice is don't touch Windows (aka "Don't try this at work"). Find another way or find something else to do.

I don't know how inexpensive hard disks or USB sticks are in $ but my advice would be to buy a USB stick big enough to do job. That's a 32 Gb stick for the DVD and may be only a 4 Gb for the CD.

You would need to plan ahead. Figure out what goes in which partition, how big each partition must be, partition and format the USB all before you install Knoppix to USB. Me, I'm already on my third iteration.

[Forester]

I hoped that knoppix forums would sent me a mail when someone reacted.

Go to the top of the page. Click on the caret to the right of Thread Tools and select Subscribe to this thread... from the drop down list.

[dinosoep]

sudo chmod 1000:1000 knoppix_tmpiso; didn't work, chmod 777 knoppix_tmpiso did however
thank you for the script, I'm currently trying it out :-)

[Forester]

sudo chmod 1000:1000 knoppix_tmpiso; didn't work, chmod 777 knoppix_tmpiso did however
thank you for the script, I'm currently trying it out :-)

Oops, you're quite right. I meant:


sudo chown 1000:1000 knoppix_tmpiso I often type chmod when I mean to type chown. :oops:

The idea is to make knoppix_tmpiso writeable without using sudo, so either what I meant to type or what you did type will do.

[Forester]

In any case, the steps above involved will be somewhat tricky to the Linux newbies. I wish there
is a cheatcode called "upgrade=/dev/sda4:/master/KNOPPIX/KNOPPIX" which will automatically
perform the upgrade. I will leave this as an exercise for those interested to do so by modifying minirt.gz.


If you organise thing right, you can use cheat codes that are already there (assuming Knoppix 6.4.3 or equivalent).

Suppose the Knoppix you have just remastered is on filesystem /dev/sdb1 under the directory KNOPPIX. Suppose further that you arranged for the new remastered Knoppix to be on device /dev/sda1 under the directory /home/yourself/remaster. This directory contains both the new KNOPPIX file and the cleaned-up knoppix-data.img (or aes) file. I don't think it has to contain all the other little files normally found in the KNOPPIX directory but it wouldn't hurt.

You can now reboot to the remastered knoppix for test purposes with the cheat codes:


fromhd=/dev/sda1 knoppix_dir=home/yourself/remasterThe directory /home/yourself/remaster has to be a real directory (avoid symbolic links and mount points). A nice feature of this approach is that /dev/sda1 doesn't have to be a vfat file system so long as it is a) visible at boot time and b) of a type the init script understands. This means it could (you didn't get this from me) be your ntfs Windows 7 main disk.

If the new remastered Knoppix seems to be working OK you can use it to copy the new KNOPPIX file over the old one:



mount /dev/sdb1 /media/sdb1
cp -pf /mnt-system/home/yourself/remaster/KNOPPIX /media/sdb1/KNOPPIX/
Then clean up the old knoppix-data.img:



mkdir knoppix_tmp
sudo mount -o loop=/dev/loop1 /media/sdb1/KNOPPIX/knoppix-data.img knoppix_tmp/
cd knoppix_tmp
sudo rm -fr $(ls --ignore=home --ignore=var);
Finally, reboot Knoppix from your USB stick 'as normal'.

[Capricorny]

If you organise thing right, you can use cheat codes that are already there (assuming Knoppix 6.4.3 or equivalent).
....

fromhd=/dev/sda1 knoppix_dir=home/yourself/remasterThe directory /home/yourself/remaster has to be a real directory (avoid symbolic links and mount points). A nice feature of this approach is that /dev/sda1 doesn't have to be a vfat file system so long as it is a) visible at boot time and b) of a type the init script understands. This means it could (you didn't get this from me) be your ntfs Windows 7 main disk.


I wouldn't recommend newbies experimenting with remastering on Win7 NTFS partitions. I think it is much safer to shrink the Win partition on a disk, set up an ext3 partition there, and do everything on "native Linux".

Apart from that, I think the method looks very promising. I will try out something like this myself now, as I feel forced to do it: Upgrading all packages on Knoppix 6.4.4 DVD today, filled up about 2.6 of total 4 GB persistent store. Therefore, people should not attempt it without really in need of it. OR, as a first step in remastering.

I'll start out using internal hard disk, then I may try with USB-sticks - but I think a better external storage for remastering would be an external SSD drive with USB3 connection.

[Capricorny]

To follow up, I think it may be an advantage to do as much as possible before starting the remastering process. That implies doing package installs, upgrades etc the "ordinary" way as much as possible. But how to, within 4GB space? No way there, but we don't have to confine ourselves to FAT32 and DVD limitations.

I just tried out expanding knoppix-data.img to 8.5 GB, and that works well on ext3. I place the Knoppix package on an ext3 partition and run from there, starting from USB with the fromhd= cheatcode. I'm not sure if there will be some kind of performance problems with large loop-mounted images, but I haven't seen any so far. So I will try with an even larger persistent store, allowing me to do more comprehensive installs and compile large programs.

If we, after cleaning up and remastering, get outside the 4GB limits, there are several ways to still use USB sticks as basis. The simplest is to format the stick to ext2/ext3, install (for example) GRUB on it and just dump the Knoppix files to it. Somewhat more elaborate would be to split the compressed packages on two volumes, or work with several uncompressed volumes - that has worked just fine for me.

I think using Linux file systems on sticks is very practical. For exchange of data with other platforms, we can set up a smaller FAT (or even NTFS) partition on the stick, and have that partition automatically mounted on boot-up. Typically, a 16 GB stick could be partitioned into 15 GB ext2 and 1 GB FAT32, the ext2 volume: 4 GB compressed, 10 persistent and 1 spare.

[kl522]

I just tried out expanding knoppix-data.img to 8.5 GB, and that works well on ext3. I place the Knoppix package on an ext3 partition and run from there, starting from USB with the fromhd= cheatcode. I'm not sure if there will be some kind of performance problems with large loop-mounted images, but I haven't seen any so far. So I will try with an even larger persistent store, allowing me to do more comprehensive installs and compile large programs.


I have used ext3 persistent store for quite sometime now. The only performance problem with large ext3 and ext2 file system ( not specifically to loop-mounted ones ) is when you have to do file system check. The file system check time is really slow when compared to ext4.

[Capricorny]

... The file system check time is really slow when compared to ext4.

I use a 16.5 GB ext2 persistent store, for testing now, seems like checking takes some time.. But, it seems to work. Full system upgrade went well. Which is not obvious, as, for example, I found that dpkg halts in updates on one missing newline in one file list - regardless of dependencies. Now is the time for purging/pruning. Is there any available hinting for this? The packages list, like in http://ftp.knoppix.nl/os/Linux/distr/knoppix/DVD/packages-dvd.txt doesn't give too much clue, it seems. Is there any way to pick out the games here? Here are some deletion candidates among the 40+ MB packages. Are any of these strictly necessary or very useful?


dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -n ......
....
42884 context
44452 wireshark-common
45644 lmodern
46912 kdewallpapers
49972 gnome-games-data
51448 scribus-ng
52616 kde-l10n-es
59692 evolution-common
60760 pinyin-database
62844 openvas-plugins-dfsg
65260 tuxpaint-stamps-default
68852 kdebase-workspace-data
85476 mingw32
85972 inkscape
86044 neverball-data
98064 gcompris-data
101889 linux-image-2.6.37-64
120892 wesnoth-1.8-data
136684 wesnoth-1.8-music
203404 crossfire-maps
In particular, identifying game packages in a simple way would be nice.

[kl522]

I use a 16.5 GB ext2 persistent store, for testing now, seems like checking takes some time.. But, it seems to work. Full system upgrade went well.

16.5 GB persistent store is considered a huge persistent store but it is still probably not near to a point where file system check is unbearably slow. I used to have a 130 G ext3 file system which file system check takes unbearably long time to complete. I have since migrated to ext4. I am glad that I did. 130 GB is already slow, I can't imagine having a 1 TB ext3 file system !

As I am writing, I did a small mod to my minirt.gz to support ext4. I think it is about time it should do that.

Cheers.

[Capricorny]

130 GB is already slow, I can't imagine having a 1 TB ext3 file system !

As I am writing, I did a small mod to my minirt.gz to support ext4. I think it is about time it should do that.


Very interesting if we could use ext4 - but I found that time-wise, 8GB persistent is quite OK. And I really don't think we need more when remastering. We definitely need more than 4 for upgrades and new installs, but using 8GB after remastering, we can do an awful lot of things.

Very much to my surprise, DVD remastering, mostly following Forester's procedure, succeeded at the first try. Of course not perfectly, as I tried out alternatives and got a few things somewhat wrong. And the compressed KNOPPIX image was only 3.2 GB, with Oracle XE 10g and a lot of R packages installed. Removed a few games etc - there seems to be ample room for things like vmware workstation and a couple of servers - in fact, the USB VFAT stick concept could be enough for most uses.

I had 4GB RAM and 10 GB swap - clearly enough. And, according to top, the I5-430M ran at 370% CPU during compression, indicating that create_compressed_fs parallelized nicely.

[kl522]

Very interesting if we could use ext4 - but I found that time-wise, 8GB persistent is quite OK. And I really don't think we need more when remastering.

True and fully agree.

However there is another subtle need to have minirt.gz supporting ext4, ie when the persistent store itself is a file sitting on an ext4 filesystem. Right now this configuration is not supported, one cannot have a poor-man install on an ext4 file system.

[Capricorny]

However there is another subtle need to have minirt.gz supporting ext4, ie when the persistent store itself is a file sitting on an ext4 filesystem. Right now this configuration is not supported, one cannot have a poor-man install on an ext4 file system.

That is a rather serious limitation for me. The hardware and file system agnosticism of Knoppix is one of the great advantages, making many system details mostly an efficiency issue.

And the need for efficient 100+ GB volumes is clear - for many, if not most, users.

Would it really be enough to tweak minirt.gz for Knoppix - because support is integrated otherwise?

I have a couple more questions. First, I noticed that the ISO9660 standard was frequently violated when creating the iso file system. Some file names got changed too - what is the expected effect of this? Anything to do about it?

Second, I come back to the cloop issue. Is there anything new on this front? Seems to me that for our special purpose, we don't need the particular features of cloop (being file system agnostic maybe the most important contrast to squashfs, which is a file system?). I am going to make some comments on this in the squashfs thread.

[kl522]

I am going to make some comments on this in the squashfs thread.
By all means please go ahead to do it. However I will not be joining you there. I better keep my mouth shut on this topic. Everything I want to say about squashfs is already said. I have no rights whatsoever to comment on the choice of it, only the author can decide on the choice. Wow, this is indeed like trying to meddle into people's choice of religion. No way a third person can make any comments on your choice of religion, don't you think so ? :)

[Capricorny]

By all means please go ahead to do it. However I will not be joining you there. I better keep my mouth shut on this topic. Everything I want to say about squashfs is already said. I have no rights whatsoever to comment on the choice of it, only the author can decide on the choice. Wow, this is indeed like trying to meddle into people's choice of religion. No way a third person can make any comments on your choice of religion, don't you think so ? :)

Exactly! :-)

But luckily, the assumptions and assertions are to a large degree testable here.
And if all the steps necessary for remastering are gathered into a two-step routine, with deletion/pruning and (eventually) upgrading is the first step, I don't think many will care if squashfs is being used, as long as everything works and the process can be iterated. With expanded persistent store, no program installations need to be part of the remastering process, and this is also better practice - everything that gets compressed should be well tested first.

One feature that might come in handy in this respect, is an option to make a backup of persistent store upon system shutdown. It is much more natural to do this after running the system, and, eventually, before using the system files for upgrading, than at bootup time.

Another feature, could be union-mounting the system to remaster as a separate file system from the running Knoppix - then any risk of open files or inconsistencies can be avoided. With squashfs at least, this should be possible.

[kl522]

That is a rather serious limitation for me. The hardware and file system agnosticism of Knoppix is one of the great advantages, making many system details mostly an efficiency issue.

And the need for efficient 100+ GB volumes is clear - for many, if not most, users.

Would it really be enough to tweak minirt.gz for Knoppix - because support is integrated otherwise?


Since I am not the author I could only ***SPECULATE*** why this is not supported. For the author, it is trial to change minirt.gz to support ext4 and yet it is not supported probably :-

1. The author does not believe in ext4, thinking it might be unsafe.
( Well, this is unlikely to be true because even the much more dangerous configuration of poor-man install on NTFS is at the least "installable" ).

2. The code for support for various file system has long been written and nobody has provided the feedback for the need for ext4 or rather, the author is distant enough to be able to hear about the need for ext4 from anyone yet.
( This is highly likely because of the way KNOPPIX development has been taking it's course all the years - a solo project ).

[Capricorny]

2. The code for support for various file system has long been written and nobody has provided the feedback for the need for ext4 or rather, the author is distant enough to be able to hear about the need for ext4 from anyone yet.
( This is highly likely because of the way KNOPPIX development has been taking it's course all the years - a solo project ).

I think there may be a couple of more reasons for ext4 support somewhat lagging. First, out-of-the-box support for it has not been universal until recently. For instance, Red Hat Enterprise Linux only added it in the last release a couple of months ago. Second, it is just in line with the add-in philosophy of Knoppix that the no-alternatives situations are handled first: People using ext4 today, mostly know how to create partitions, and they are, mostly, in position to do it for their KNOPPIX interests. The same does not apply to most NTFS users. And the highest priority for any general purpose distro can not be to cater for the highly interested with skills to do the modifications themselves. A "standard remastering" based on the work presented in this forum could make such issues less personal, I think. Any name for such a Knoppix derivative? Geek-knoppix maybe? ;-)

[kl522]

I think there may be a couple of more reasons for ext4 support somewhat lagging. First, out-of-the-box support for it has not been universal until recently. For instance, Red Hat Enterprise Linux only added it in the last release a couple of months ago.

I would like to agree with you but I can't. :) Knoppix has a lot of chance to include new changes recently. Just watch how many new releases have taken place recently ? If it is something in the to-do list, it would have been included. And this particular ext4 thingie, it's simply trivial. Takes only 2 minutes. It ***WAS*** not on the to-do list. I am not sure about it now. :)

[Capricorny]

And this particular ext4 thingie, it's simply trivial. Takes only 2 minutes. It ***WAS*** not on the to-do list. I am not sure about it now. :)
Well, rather than speculating, we can proceed to produce that geek-knoppix. Then Klaus K is free to include or drop any changes in his next upstream release. And, if you look at his communication with Ryumbeke, he can be very positive towards suggestions and modifications. Also, I think the direction of development in 6.X is the opposite of secteric.

My take on some of the main features:
* ext4-support plus, eventually, other non-suppeorted fs in general use.
* Thorough purging of non-essentials, like space-consuming games. Users can install them themselves.
* Addition of a series of packages that are important for some users. My own standard examples: R and some Java packages.
* Maybe addition of some packages/libraries to facilitate the installation of some programs/extensions that are not included.
* Use of squashfs, at least as one version, keeping the cloop tools.
* Tools for setting up for GRUB-booting.
* Backup of persistent store + safe system transfer. (Copying mounted and constantly modified persistent store is not entirely safe, it seems.)
* Remastering tools.
* If possible, some tool for collecting package use statistics - to aid in inclusion/exclusion of packages.