An alternative to re-mastering for CD-size Knoppix flash disks.

[utu]

.
Greetings, Knoppix LiveUSB fans.

Attached please find two png screen snapshots to introduce you to
a script to manage Knoppix compressed image updates and persistence
files. The snapshots show two menus the script stackimu uses to
control the 'management' processes. The script is provided in
gzipped format.

If you are interested, I hope you will find or prepare a clone of a
recent Knoppix LiveUSB with persistence and try it out. Preferably
a LiveUSB derived from a Knoppix 7+ LiveCD, since that's what I use
all the time.

If you don't know what this is all about, you might first read a rambly
'Draft' within the Stacked.Images folder. You can read this 'Draft'
with leafpad alone if you wish, or with stackimu.

To get the whole picture you will need a working LiveUSB clone of Knoppix 7+
with a persistence file containing some added program material you would
like to safeguard by turning it into a compressed image update.

To do so, download the Stacked.Images.####.tar.gz file to your clone's
/home/knoppix directory, unzip it there with the command tar xvfz *.gz,
cd into your new Stacked.Images folder and there execute stackimu as knoppix
user with the command ./stackimu.

If you want to check the *tar.gz download, use the command,
echo "5e7142945d562c186cfe6c37683ee878 Stacked.Images.1106.tar.gz" | md5sum -c
and, note there must be exactly TWO spaces between the checksum and the filename.

As knoppix user examine stackimu's menus and text files and find out what
it is designed to do. After studying this material, if you want to try it
out more seriously, quit as knoppix user and begin again as root with the
command sudo ./stackimu.

I've used this technique myself for some time now on a LiveUSB made from
a Knoppix LiveCD built on a 2 Gb SDHC USB device and it suits my needs.
I hope you will find it useful as well. I would not be surprised to hear
that it may fail in some DVD-size situations, and certainly for any case
that a cloop tries to exceed 2 Gb.

Preview post does not clearly confirm *tar.gz upload, but here goes.

[utu]

If you are a cut-and-paste person, the previous should have read:



If you want to check the *tar.gz download, use the command,
echo "5e7142945d562c186cfe6c37683ee878 Stacked.Images.1106.tar.gz" | md5sum -c
and, note there must be exactly TWO spaces between the checksum and the filename.
Forum editor eats extra whitespace, you know.

[Blacksimon]

Hi utu,
you're doing a great job.
Stackimu only works for cd-size knoppix ? if I try with dvd-size could happen some kind of error ?

[utu]

Greetings, Blacksimom.

I'm no expert and don't use the DVD version very much, but I think
the problem is in the way cloops and iso9660 work together that makes
for complications with cloops larger than 2 Gb.

My program doesn't have the fine structure necessary to handle the DVD
case, but I think if one stays with CD-size Knoppix that no problems
will arise. The DVD complexity is what defines the generalized
remastering case.

From what I've read, the 'failure' which occurs with cloops larger
than 2 Gb is that file locations beyond a certain number 'wrap around'
and make inappropriate references. KK suggests you have to do a
file-by-file md5sum check to see this sometimes, the errors are
subtle, but deadly.

If you really want to work with DVD-size situations, I think you
might get into some serious trouble using stackimu.

I think stackimu makes it a virtue out of having ONLY a 2 to 4 GB
USB to work with, since I think (at least hope) this implicitly
gets around the filesize/cloop size considerations by its very
nature.

[Werner P. Schulz]

In my opinion it isn't useful to save the software device (the file knoppix-data.img). You have to store the content of this device, the folders and files within this.

For example: you have a persistent memory of 4GB and and the used size within this is 300 MB. Why store the whole knoppix-data.img and not only the content within this?

"knoppix-data.img" looks like a file, but it isnt't a file, it is a device with folders and files within it.

[utu]

Greetings, Werner.

You should read some of the text in 'Draft', Werner.
As I point out, I only save the .img long enough to make sure the Knoppix[n]
replacement is doing its thing.
You may, and I do, get rid of the .img when I'm confident it went ok.
I don't find it convenient to compress the .img, either, and may delete
this option in some upgrade.

I am not clear on your insistence on .img as a 'device'.
I thought the notion of unix was that everything might be thought of as a file.
Please be more explicit on why you make this distinction.
Otherwise this comes across as simply a cavil.

[Werner P. Schulz]

You should read some of the text in 'Draft', Werner.I studied it carefully.


I am not clear on your insistence on .img as a 'device'.
Creating a persistent memory includes three steps:


make a file of max 4 GB (knoppix-data.img)
format it with ext2 filesystem
mount this device (/dev/loop0) on /KNOPPIX-DATA

In my example (CD flash installation on a 4 GB USB stick and 3.1 GB persistent memory) you can see, this device (persistent memory) works like a drive; within this drive 479 MB are used and 2.6 GB are available.

Why, for Heaven's sake, shall I save the whole 3.1 GB device and not only the content of 479 MB of this drive? Apart from doing so, there is no space left on the USB stick to go your way. And last but not least, it doesn't make sense, not to store a backup on a different medium.

[utu]

Why, for Heaven's sake
Greetings, Werner.

I would worry about the 479 Mb of persistent store that you have, should
your knoppix-data become corrupted. In that case you will probably
lose whatever the difference is between that 479 Mb and whatever you
have in your backup. I don't make daily backups, myself. Perhaps you do.
I just 'Install to flash disk' after a compressed image update and make a
clone after my last compressed image update. I do not operate with 479 Mb
of stuff in 'temporary' condition. Also, there would be, for me, an
un-necessary overhead in time on mounting 3 Gb persistence on every boot.

Whenever I have a 'significant' amount in persistent store, I prefer
to convert that to a compressed image equivalent which is less likely
to get written-over, and which can be recalled faster on boot. Much less
un-necessary re-writes to USB if you care about that. I'm not sure there
is a 'wear problem' for these re-writes.

Here's my stuff for comparison. I can tell I have 8.4 Mb diffence between
my last backup and the present. My 'backup' is in KNOPPIX1,2,3. I have a
clone as well. When I boot, it takes less time to bring in 485 Mb
persistent store, than for 3 Gb which I'd never use if I had it. I also know
the 8.4 Mb is mostly .mozilla cache stuff which I don't worry about.

Thanks for the nice comparison. I like what I've got better.
stackimu only costs 20-30 Kb to make this possible.

[utu]

Incidentally,
df over-reports the contents of /KNOPPIX-DATA by a factor of two or more relative to du in my case.
I think du is correct.

[Werner P. Schulz]

I would worry about the 479 Mb of persistent store that you have, should
your knoppix-data become corrupted.No, there is nothing corrupted!

The size of the persistent memory is 3.1 GB and not 479 MB! 479 MB is the value of the used space within this drive. With
du -sh /KNOPPIX-DATA/I get the value of 465 MB.



Also, there would be, for me, an
un-necessary overhead in time on mounting 3 Gb persistence on every boot.In my case


boot with persistent memory of 3.1 GB: 1min 4 sec
boot without persistent memory 58 sec

[utu]

I get the value of 465 MB.

The difference between this value and its counterpart in your 'backup'
is what you have at risk IF your knoppix-data.img becomes corrupted.



In my case


boot with persistent memory of 3.1 GB: 1min 4 sec
boot without persistent memory 58 sec

My time to on-line is 28 seconds.


I appreciate your comparative numbers a lot.

[Werner P. Schulz]

The difference between this value and its counterpart in your 'backup'
is what you have at risk IF your knoppix-data.img becomes corrupted.I have no backup within my USB stick. Where do you see it?

[utu]

Why, for Heaven's sake, shall I save the whole 3.1 GB device and not only the content of 479 MB of this drive? Apart from doing so, there is no space left on the USB stick to go your way. And last but not least, it doesn't make sense, not to store a backup on a different medium.

On a different medium, I suppose.
Did you misplace it?

[Werner P. Schulz]

1. screenshot: Here you can see the situation before I purged "xpdf"

2. screenshot: Now are the deletion-marks in /KNOPPIX-DATA/ (/dev/loop0) for the two files xpdf and xpdf.real. This is the proper way, Knoppix will use persistent memory.

3. screenshot: After running your script "xpdf" is reastablished?!
The deletion-marks now within /KNOPPIX1/ (/dev/cloop1) doesn't work.

You cannot use /dev/cloop1 like /dev/loop0.

[utu]

Greetings, Werner.

I really do appreciate that you are concerned, and I respect your expertise,
but I don't know what you are trying to say here.

You can see that I am operating with Knoppix and three compressed images somehow
or other. I can't offer an IT proof of any kind. It just works. I may be
missing something here, but it's not obvious to me what it might be.

I gave you a reference to KK's suggestion along these lines. Did you look into
that? Is there any problem with my interpretation of his idea there?

Respectfully.

[utu]

Post 14 is unclear to me in its message, but I'll take a shot at it.

I describe an algorithm that is useful to my purposes which packages
compressed images of program material which collects and add capabilities
to my OS over time. The collection has two good qualities. First it
makes the saved material less likely to be inadvertently over-written.
Secondly it stores this material more compactly than otherwise.
Its an algorithm.

It's not good or bad. It's a definition. You have to add your own
criteria to describe it as better or worse than some other definition.
It does have its own peculiarities, as do all algorithms.

A peculiarity of this particular algorithm is that it demonstrates
how adding stored material and removing stored material in any OS
are not necessarily reversible inverse operations.

If you object to this as a 'shortcoming' of this algorithm, you must
surely be upset with Synaptic sometimes when you try to un-install
a previously installed program and end up in 'dependency hell'.

I find it a very minor restriction for my purposes to avoid situations
which would require _removing_ previouly compressed and saved material.

I expect it will take more resources to eliminate this _perceived_ flaw.
I find the restriction less onerous than the thought of bringing to
bear additional resources to eliminate this restriction.

[Blacksimon]

Hi Utu,
my point of view about the use of Stackimu is that it can be useful for:
- speed up the boot process
- decrease the size of knoppix-data.img
- compress images of personal customizations
But these customizations (loaded in one or more dev/cloop) may be only the most important ones and will not be changed for whole time that you will use the OS chosen.
For everything else you have to use the knoppix-data.img to store data as usual.
Feel free to tell me if I'm wrong in interpreting.
So I'm not sure Stackimu can be useful to me as it is for you, but by ideas of others can always create something interesting...

bye

[utu]

Greetings, Blacksimon

Thanks for your comments. Glad to discuss this with you.

My primary purpose in using stackimu is safeguarding my personal customizations.
It's one of several back-up methods of Klaus K's I've looked at.
It is my current method of coping against the occasional knoppix-data.img disaster.
I keep my 'used' fraction of knoppix-data.img small and its default total at ~500 Mb.

There are several things that just come along for free:
stuff stores more compactly; 2.5:1 compression.
stuff takes less time booting up; 30 second boot to on-line.
the idea works with minimal total device capacity; 2 Gb SDHC is only about half-full

[Capricorny]

My own experience is that full remastering is so quick and simple for me, that I rather do that than creating incremental modificatons. As for persistent store backup, I would guess that rsync -axu to another loop-mounted image should work just fine. In case a full file copy is deemed to take to long. If day-to-day modifications are not too extensive, using a subversion (or git) repository is also something to think about - then you will also be able to go back in time to any specific time you have checked in.

As Werner noted, use/not use of persistent store does not seem to affect boot-up times very much (DVD version). So in my use case, time would not be an important factor for choice of method.

[utu]

General Observation:

The title
'An alternative to remastering for Knoppix CD-Size LiveUSBs'
seems to offend some of the remastering cognoscenti.

Perhaps a better title might have been
'A minimalist script to transcribe Knoppix cloop customizations'

My focus is on saving my customizations and my style is to work with
adequate, but reasonably minimal resources. Remastering, as purely defined,
would be a poorer alternative to achieve these ends.

The latent capability within Knoppix to use overlaid compressed images
is under-utilized for its worth, in my opinion.


Specifically, in regard to stackimu as presented for download in Post #1:

1. I have downloaded the material posted on Knoppix.net, checked and found that
Stacked.Images (1106) is working as I intended.

2. Knoppix has a built-in (un-explained) minimum 200 Mb value for persistent store.
A recent Synaptic update brought in 287 Mb to /var; this came close to over-running a
350 Mb knoppix-data.img allotment originally in use. I'm now using 500 Mb as my
(default) persistence file size.

3. There may be a surprise at KNOPPIX8, if one hasn't provided enough cloops.
The Knoppix standard is to provide cloop[0-7]. linux cloop.cloop_max=10 (or more) at boot
would provide cloop8 and cloop9. mountknoppix and mountunion in /init might need to be re-
defined as well, beyond KNOPPIX[0-9], to further increase the number of updates beyond 9.
For the time being, my stackimu update limit is set to provide only 7 updates,
in accordance with the KNOPPIX 7 current default definitions.

4. 'View Contents of /KNOPPIX-DATA' is more informative if defined with a replacement
for the $V4 stackimu line which reads:
$V4) TL=$V4; du -m --max-depth=2 --exclude=.wh* $KDF \
| sort -g -r | sed 's_/KNOPPIX-DATA_._' > SF;;
This improved the View of /*DATA Option is to allow assessment of specifically where
additional material is coming from and to serve as a guide in editing of what's stored
by prepare_update.

5. Added 'View Option, df -h device assignments'. Note that KNOPPIX[n] files are reported
in the expanded size in Mbs in df reports, and in condensed size in Mbs in du reports.
Therefor, the size of contents of the /*DATA directory may appear more than twice their
size in df -h than in du -h.

6. Removed Main Option 6 to provide a compacted, off-line, prior-knoppix-data.img as a
belt-and-suspenders back-up option, since it seemed both clumsy and unnecessary
in practice.