Ramdisk and partial remastering in Knoppix 7

[utu]

.
Another un-advertised new feature in Knoppix 7

Klaus K has included a new 'ramdisk' element in Knoppix 7 for serial
collection of LiveUSB program changes that may be individually compacted
which may then be expanded and added in overlay fashion to the basic
operating system at boot.

A brief description of the use 'ramdisk' to achieve an alternative to
wholesale re-mastering occurs on debian-knoppix at:
http://lists.debian.org/debian-knoppix/2012/06/msg00005.html

[Werner P. Schulz]

Be aware, this only works if you don't have peristent memory.

[utu]

Greetings, Werner.

I'm thinking that the KNOPPIX1, KNOPPIX2, etc idea might be a good way to segment the
remastering that one does over time. Into chunks. Some programs here, Some personal tweaks there.
A way to organize things.

Klaus's choices of which directories to save is a puzzlement to me.

For one thing, with an 8 Gb LiveUSB, I mostly can just save the whole overlay as a backup.
My needs for re-mastering have almost disappeared.

I can see where Klaus can use larger chunks of compressed material,
for example a set of non-free programs to go with the CeBIT DVD as KNOPPIX1.

It is a new feature worth calling attention-to.
But it's not all that different than the old update.tar.gz approach.

[utu]

.
First, a historic note:

In post #1 I called some attention to Klaus K's recent murmurings about using
'ramdisk' as an alternative to some of the more recent re-mastering schemes that
use lots of resources and time.
You may be interested to know this is not something new or even new to Knoppix.
Just take a look at:
http://www.oreillynet.com/sysadmin/blog/2005/03/knoppix_38_and_unionfs_wow_jus.html

Now, for something closer to home:

In posts elsewhere on this forum I have touted a little scheme of I think-of as
'partial backups'. I try to keep track of the tweaks I use and where they go, so
I can collect these in a file update*.tar.gz. I then use the cheatcode note about
how to get this file overlaid on KNOPPIX-DATA and become a part of
knoppix-data.img. I've used this all through Knoppix 6 to back-up my LiveUSB when
I ruin my knoppix-data.img by some inadvertent mistake. Maybe a little clumsy, but
it serves the purpose. my tweaks in update*.tar.gz amount to about 50 Mb which
I store on /mnt-system/ in-between uses.

It turns out, I can read this 50 Mb update*.tar.gz into the Knoppix 7 ramdisk and
produce an equivalent 30 Mb KNOPPIX1. I have to do some juggling with reboots
and using the knoppix noimage cheatcode to accomplish this, but it works.
There are two hidden bonuses in this approach. First, you needn't be very
fastidious about noting your tweaks and where they go. Ramdisk does this for you.
It only retains the directories, and contents thereof it needs to record differences
from an initial compressed image. Second, after all the manipulations you may
once again re-install a persistence file to record retain any additional
changes you might decide to investigate on a more tentative basis than the
ramdisk approach.

You don't need a second system to accomplish all this. I bootstrapped my 8 Gb
Knoppix 7.0.2 LiveUSB from an update*.tar.gz state to a ramdisk-modified version
with persistence using only two special commands and the cheatcode 'knoppix noimage'.
A second system would be a good idea as a backup, but this is just to let you
know it's possible to do without.

The two commands that are crucial are: To transfer update*.tar.gz to /ramdisk/;


sudo tar -xzf /mnt-system/update*.tar.gz - C /ramdisk and, To create the file KNOPPIX1 you will need to complete this task:


sudo mkisofs -R -U home etc KNOPPIX-DATA var (or whatever the ramdisk dirs were) \
| create_compressed_fs -B 131072 -m - - >/tmp/KNOPPIX1I'll work out a turn-by-turn list eventually,

but the first step is rebooting with knoppix noimage to bring in the ramdisk;
next read update*.tar.gz into ramdisk;
next get rid of your original knoppix-data.img and KNOPPIX-DATA;
next make your directory list and issue the second command to produce KNOPPIX1;
next move KNOPPIX1 into the directory that has KNOPPIX and formerly held
KNOPPIX-DATA as well; and
last, but not least, reboot and re-establish persistence, when prompted to do so.

I may have thrown in a few re-boots here and there along the way to see how
things were going. I'll have to make some notes. But the preceeding gives you
the broad outline.

I think there is a sequel to this that Klaus K has defined which describes
how to combine compressed images, but I've not progressed that far yet.

I'm interested to hear the experiences of others with Knoppix 7's ramdisk.

[utu]

.
I have developed a number of changes to Knoppix 7.0.2 and
used the following procedure to save these changes to KNOPPIX1 and
re-establish persistence:

1. In a normal, bare Knoppix 7.0.2, I establish a LiveUSB with
persistence and develop certain changes and try them out, noting
all the steps necessary.
2. I reboot with the cheatcode 'knoppix noimage' and repeat all
the same necessary change steps.
3. I use Klaus K's formula to capture the ramdisk information in
KNOPPIX1, using all the directories I note using 'ls' in ramdisk and no others.
4. I transfer KNOPPIX1 into the same /mnt-system/KNOPPIX/ directory
as the original KNOPPIX file itself.
5. I delete knoppix-data.img, reboot and re-establish persistence.
6. I am pleased that KNOPPIX1 us now doing its job.

Here's what is not going right for me:
I go through steps 1 through 5 with some different changes and
develop and install KNOPPIX2, but without the success in step 6.

The simplest instances of this are to simply use Install-Components or
Aptitude to bring in Flash or ntfs-3g. The ntfs-3g exercise is the one
Klaus K uses to showcase the stacking of KNOPPIX1, KNOPPIX2,.. etc.
I presume this is supposed to work, but I can't seem to make it so.

Any suggestions as to what to look for are welcome.

[utu]

.
To stack permanent serial compressed image upgrades to KNOPPIX, use the magic incantation,
assuming KNOPPIX1 was the previous upgrade:

cd /
sudo mkisofs -R -U KNOPPIX-DATA | create_compressed_fs -B 131072 -m - - > /mnt-system/\
KNOPPIX/KNOPPIX2

(Sorry this line a little too long for this forum, so used the '\' sign to show it here.)

Then reboot to see if it all works like it did in the previous persistence file.
If so, move knoppix-data.img from /mnt-system/KNOPPIX/ to /mnt-system/ temporarily and reboot.
Re-establish persistence and again verify that KNOPPIX2 is doing what if ought.
When you are really sure KNOPPIX2 and persistence are ok, remove knoppix-data.img.

Dont count on PCManFM to remove things; use root and rm to do this.
init's mountunion() looks like you may have up to ten such upgrades, if you have enough space
on your LiveUSB

[utu]

.
When you are really sure KNOPPIX2 and persistence are ok, remove knoppix-data.img.


By this I mean the 'old' knoppix-data.img in /mnt-system/,
not the 'new' one in /mnt-system/KNOPPIX/.

[utu]

.
Then reboot to see if it all works like it did in the previous persistence file.
USB

This first reboot should be with 'knoppix noimage' or 'knoppix64 noimage'
cheatcode, depending on what kernel you were last using.
This allows you to temporarily ignore the current knoppix-data.img without moving or removing it.

On the next reboot, after moving the initial knoppix-data.img, the absence of this
.img is what triggers a request to (re-)establish a persistence file.

[klaus2008]

I'm interested to hear the experiences of others with Knoppix 7's ramdisk.
Hello utu!

When I read Klaus Knopper's post on the mailing list I decided to try the remastering procedure from the live system myself.

I use a kind of Poor man's install of Knoppix V 7.0.2 inside a VirtualBox 4.1.16. I copied the content of the DVD to the directory knx702 and use Grub as bootloader.

I think that a typical session looks as follows.

1. Boot with "noimage" cheat code.

2. Open root terminal.

3. Issue the following commands.
apt-get update
apt-get install flashplugin-nonfree
apt-get clean Of course, there are many more useful packages that are not included in the public version of Knoppix 7.0.2 since they are not free like unrar for example.

4. Create the archive file KNOPPIX1 from the ramdisk. The options used for makeisofs are the usual ones taken from the "Remastering howto". I use a temporary file on the virtual HDD since I do not have more than 1 GB of RAM available inside my virtual machine. I exclude some directories with files I consider not useful for the archive. Maybe there are more directories.

mkisofs -R -U -V "KNOPPIX1 filesystem" -P "KNOPPIX1" \
-hide-rr-moved -cache-inodes -no-bak -pad \
-x /ramdisk/var/lib/apt/lists -x /ramdisk/dev \
-x /ramdisk/var/log -x /ramdisk/var/run /ramdisk | nice -5 \
/usr/sbin/create_compressed_fs -b -B 65536 \
-f /mnt-system/knx702/isotemp - /mnt-system/knx702/KNOPPIX/KNOPPIX15. Reboot and enjoy Flash videos in Iceweasel.

Comments and suggestions are always welcome.

[utu]

Greetings. Klaus2008.
Nice to hear from you.

Glad to see someone carry-through on Klaus K's example.
I thought his example was hard to follow, myself.
Also, it left me puzzled on how to carry-on for subsequent images.

You are using a lot of options, I'd like to know more about each of these.
In some of my prior efforts I tried to cast out anything that had Cache or cache
in its name.

I like this one of Klaus K's idea because I can do it all within the resources
of my 8 Gb SDHC. No additional system, virtual or otherwise, no additional
workspace needed, no hours-long processing needed.

As an unexpected plus, I just cloned a new 8 Gb SDHC, and would-you-believe?
The clone has KNOPPIX1, KNOPPIX2 and persistence right away, ready to go.

Anxious to hear how you progress thru KNOPPIX2, KNOPPIX3, etc.

[klaus2008]

You can read more about the options I used in the man page of mkisofs ( 8 ) (http://linux.die.net/man/8/mkisofs)

-cache-inodes
Cache inode and device numbers to find hard links to files. If mkisofs finds a hard link (a file with multiple names), then the file will only appear once on the CD. This helps to save space on the CD. The option -cache-inodes is default on UNIX like operating systems. Be careful when using this option on a filesystem without unique inode numbers as it may result in files containing the wrong content on CD.
-hide-rr-moved
Rename the directory RR_MOVED to .rr_moved in the Rock Ridge tree. It seems to be impossible to completely hide the RR_MOVED directory from the Rock Ridge tree. This option only makes the visible tree better to understand for people who don't know what this directory is for. If you need to have no RR_MOVED directory at all, you should use the -D option. Note that in case that the -D option has been specified, the resulting filesystem is not ISO9660 level-1 compliant and will not be readable on MS-DOS. See also NOTES section for more information on the RR_MOVED directory.
-no-bak
Do not include backup files files on the iso9660 filesystem. If the -no-bak option is specified, files that contain the characters '~' or '#' or end in '.bak' will not be included (these are typically backup files for editors under unix).
-pad
Pad the end of the whole image by 150 sectors (300 kB). If the option -B is used, then there is a padding at the end of the iso9660 partition and before the beginning of the boot partitions. The size of this padding is chosen to make the first boot partition start on a sector number that is a multiple of 16.

The padding is needed as many operating systems (e.g. Linux) implement read ahead bugs in their filesystem I/O. These bugs result in read errors on one or more files that are located at the end of a track. They are usually present when the CD is written in Track at Once mode or when the disk is written as mixed mode CD where an audio track follows the data track.

To avoid problems with I/O error on the last file on the filesystem, the -pad option has been made the default.
-x path
Exclude path from being written to CDROM. path must be the complete pathname that results from concatenating the pathname given as command line argument and the path relative to this directory. Multiple paths may be excluded. Example:

mkisofs -o cd -x /local/dir1 -x /local/dir2 /local NOTE: The -m and -x option description should both be updated, they are wrong. Both now work identical and use filename globbing. A file is excluded if either the last component matches or the whole path matches. At least -pad seems to be unnecessary with new versions of mkisofs.

[utu]

Thanks, klaus2008. This helps a lot.

I still am confused over man mkisofs and the die.net version of mkisofs(8). For example, the B option seems to be quite different. Is there really much hazard for the less experienced Knoppix user, such as myself, to disregard many-if not all of these options, and just do something 'simple' like the following ?
sudo mkisofs -R -U KNOPPIX-DATA | create_compressed_fs -B 131072 -m - - > /mnt-system/KNOPPIX/KNOPPIX2 On the excludes, will wild-card specs work? I've found these handy to lighten things up. Such as the excludes in tar:
--exclude=*/Cache/* --exclude=*/cache/*

[Capricorny]

Hello utu!

When I read Klaus Knopper's post on the mailing list I decided to try the remastering procedure from the live system myself.

I use a kind of Poor man's install of Knoppix V 7.0.2 inside a VirtualBox 4.1.16. I copied the content of the DVD to the directory knx702 and use Grub as bootloader.

I think that a typical session looks as follows.

1. Boot with "noimage" cheat code.

2. Open root terminal.

3. Issue the following commands.
apt-get update
apt-get install flashplugin-nonfree
apt-get clean Of course, there are many more useful packages that are not included in the public version of Knoppix 7.0.2 since they are not free like unrar for example.

4. Create the archive file KNOPPIX1 from the ramdisk. The options used for makeisofs are the usual ones taken from the "Remastering howto". I use a temporary file on the virtual HDD since I do not have more than 1 GB of RAM available inside my virtual machine. I exclude some directories with files I consider not useful for the archive. Maybe there are more directories.

mkisofs -R -U -V "KNOPPIX1 filesystem" -P "KNOPPIX1" \
-hide-rr-moved -cache-inodes -no-bak -pad \
-x /ramdisk/var/lib/apt/lists -x /ramdisk/dev \
-x /ramdisk/var/log -x /ramdisk/var/run /ramdisk | nice -5 \
/usr/sbin/create_compressed_fs -b -B 65536 \
-f /mnt-system/knx702/isotemp - /mnt-system/knx702/KNOPPIX/KNOPPIX15. Reboot and enjoy Flash videos in Iceweasel.

Comments and suggestions are always welcome.

Just to expand a bit on this:
1. I wonder if VMware Workstation/Player could be an even better platform for this kind of work? And Qemu is rather versatile in my experience, running KNOPPIX directly.
2. Re Utu's question about KNOPPIX2, I also wonder if rolling auxiliary images together might be a better alternative than juggling with several of them. But maybe there isn't any easy way to do this, short of complete remastering?
3. Could setting up a script, which handles the alternatives of writing /ramdisk to an image file or to a cloop, be a rather universal solution to most problems encountered running from USB sticks?
4. Is there any principal difference between cloop and squashfs in this context? I would think squashfs would be even simpler, avoiding the file system setup pipe. Or am I mistaken?

[utu]

.
This is not to take away from heretofore 'standard' re-mastering procedure(s) but to
highlight a subtle refinement in using Knoppix that accomplishes much of what one hopes to
achieve by re-mastering, or making backups of some kind. This alternative is elegant
in minimizing the resources required in re-mastering, including time.
I'm re-wording here to collapse into one post what previously spread over several posts.

Assuming for example that KNOPPIX1 was the previous compressed image update,
to stack another compressed image update with the previous, use the following magic incantation:

cd /
sudo mkisofs -R -U KNOPPIX-DATA | create_compressed_fs -B 131072 -m - - > /mnt-system/KNOPPIX/KNOPPIX2

Then reboot, using 'knoppix noimage' or to see if it all works like it did using the previous
persistence file.

If so, move knoppix-data.img from /mnt-system/KNOPPIX/ to /mnt-system/ temporarily and reboot,
this time without the 'noimage' cheatcode. This should allow you to re-establish persistence.
Then, re-verify that KNOPPIX2 is doing what if ought. When you are sure KNOPPIX2 and persistence
are ok, remove the 'old' knoppix-data.img in /mnt-system/, not the 'new' one in /mnt-system/KNOPPIX/.

You may use PCManFM to move things around, but don't count on PCManFM to remove things;
use root and rm to remove files.
init's mountunion() looks like you may have up to ten such upgrades, if you have enough space
on your LiveUSB

Recapitulation: We can have a full-DVD-size LiveUSB on an 8 Gb Class 10 SDHC device. This device
may be serially updated with compressed image additions without requiring any additional workspace;
without requiring any additional system, real or virtual, to manage this process; and without
any significant time expenditure for these performing these updates.
The convenience of temporary persistence may be maintained, while greatly reducing the magnitude
of the hazard in occasional inadvertent ruination of this 'persistent' image, since these
successive updates preserve well-tested material in more safely situated in more permanent
compressed images.

[Capricorny]

.
Recapitulation: We can have a full-DVD-size LiveUSB on an 8 Gb Class 10 SDHC device. This device
may be serially updated with compressed image additions without requiring any additional workspace;
without requiring any additional system, real or virtual, to manage this process; and without
any significant time expenditure for these performing these updates.
The convenience of temporary persistence may be maintained, while greatly reducing the magnitude
of the hazard in occasional inadvertent ruination of this 'persistent' image, since these
successive updates preserve well-tested material in more safely situated in more permanent
compressed images.

For more heavy installs. I will strongly recommend using a 16 GB stick, for in this scheme nothing is ever purged, which means, for example, that you will get several instances of the apt data, essentially one on each KNOPPIX[N] image. In my own use case, I have by now purged about 2GB of programs/data, and added about 3 GB on 7.0.2. Of course, this is "real world" use of Knoppix, and most users might do less, but if I had used this method, I would have had about 6GB total in cloop images by now, and I advise people to be defensive with regard to the space overhead of different copy/compress/backup methods.

It does not go on indefinitely, for the vast majority of uses, 16GB media will be enough. I have also experimented with >4GB persistent image, for example using several (uncompressed) image files, but I think an adaptation of the method presented here is better in most cases.
I think this is also a nice illustration of what I have come to think is the basic rule of remastering: If you don't know exactly why you need it, and why the alternatives are indequate, don't do it.

I'd like to mention one way this can bu used for (sort of) incremental backups: If you call the KNOPPIX[1|2|3...] images something else, for example knoppix-20120620.cloop, and store it at your backup location, while you continue to use the same knoppix-data.img at next boot, instead of forcing creation of a new image, you may cloop-mount this image and recreate the knoppix-data.img of that day from it. They may of course grow ever larger, but if you purge your persistent image, they may also shrink.

Myself, I make tarballs (.tgz files) of the knoppix-data.img for backups, but this way, I don't have the chance to start afresh with the persistent image without remastering - like I can if I decide to use the cloop backup as KNOPPIX1 and start a new .img file. I should add that I don't use the persistent store for much data - they are stored on a volume mounted on /store in my setup, and this volume is stable across OS versions and backed up separately. Some of it dates back to Slackware days of 1995..

[Capricorny]

.
This is not to take away from heretofore 'standard' re-mastering procedure(s) but to
highlight a subtle refinement in using Knoppix that accomplishes much of what one hopes to
achieve by re-mastering, or making backups of some kind. This alternative is elegant
in minimizing the resources required in re-mastering, including time.


A full remastering of a Poor Man's Install using squashfs needs about 12GB temporary space, for example on a Windows partition, and it can take 15-20 minutes. For precise timings of an actual process, see my squashfs report, #7 in http://knoppix.net/forum/threads/29862-Simple-remastering-procdure-for-7.0.2?p=127051&viewfull=1#post127051 So while minimization is obviously correct, the savings achieved may be less than impressive. IMHO, the relevant reasons for using this kind of scheme is to improve workflow, backup safety etc. That a full remastering can be done with a minimum of work, doesn't mean one should do it. On the contrary, the easier it is, the more careful one should be about checking out alternatives.

[utu]

.
I've had great success in successively updating my Knoppix 7.0.2 DVD-size LiveUSB
using the process of stacking successive compressed images mentioned by Klaus K in
the link noted in my Post #1. This is a LiveUSB on an 8 Gb Class 10 SDHC made using
a Knoppix 7.0.2 LiveCD. At present, I have 230 Mb unused of my 500 Mb persistence,
and have 3.1 GB unused in /mnt-system.

I consider success in this instance is in being able to efficiently save whatever
small increments I develop in my LiveUSB, and to do so in such a fashion that
the occasional inadvertent ruination of tweaks stored only temporarily is 'no big deal'.

My KNOPPIX1, in 31 Mb, contains the bulk of my usual tweaks, except flash;
my KNOPPIX2, in 85 Mb, contains flash, an ntfs revision and Hidden Gems.
My current KNOPPIX-DATA has only a few new tweaks, so it hardly qualifies yet
for moving to the safety of KNOPPIX3.
These two new compressed images added about eight seconds to my boot-up time.

If one starts out with a LiveUSB and persistence, one may disregard ramdisk as a
necessary special case. The little program I use for this progression is as follows.
I have only chosen to automate a few of the steps, leaving several simple procedures
that require some judgment to the process as outlined in my Post #14.



#!/bin/bash
# Declare Integer count
#
# ~/text/smite: remember to do 'chmod +x backup' & 'chown root:root smite'
# Activate in /home/knoppix/text as sudo ./smite
#
IMGDIR=/mnt-system/KNOPPIX/; count=1
cd $IMGDIR; for dir in KNOPPIX[0-9]; do let count++; done
UPDIMG=$IMGDIR'KNOPPIX'$count

# No more than 9 images allowed
if [ "$count" -gt 9 ]; then echo 'Too many images; exiting.'; exit 0; fi

cd /KNOPPIX-DATA/; sudo mkisofs -x '*[Cc]ache*' -x '*.wine*' -R -U etc home root usr var| \
create_compressed_fs -B 131072 -m - - > $UPDIMG

exit 0
So far, it is my impression that for the purpose of progressing one's LiveUSB by
stacking successive images one may disregard a lot of the mkisofs options that are
required to define an iso for a bootable CD or for handling non-Linux files.
The exclude option is one that IS essential, to allow editing of what's ultimately saved.

I'd be pleased to hear of anyone else's experience with this stacking idea.

[Capricorny]

Just one technical remark about number of images handled. I haven't tried the cloops, but for ordinary loop the busybox version used seems to be limited in number. Has anybody tried how many will actually be mounted of KNOPPIX..KNOPPIX9? All, as we should hope for, or less?

BTW, I wonder how a subversion repo would work for persistent store backup with the modest amount of data/files we have there after remastering. I think about trying it myself, but if anybody already has, it would be interesting to hear about the experiences. The rationale is that it is inefficient to back up the image file often when the actual changes made are only to a limited number of files.

[utu]

init's mountunion() looks like you may have up to ten such upgrades, if you have enough space on your LiveUSB
From an earlier post.

[Capricorny]

Init's mountunion() has no influence on busybox's capabilitites, I would think.