USB Remasterig Using Minimum Real Estate

[Forester]

USB remastering traditionally requires lots of temporary file space. More than most people have spare on their USB sticks leading to the temptation to use nfts file space.

I've refined the process and managed to cut the temporary file space down to zero (and speed up the process considerably as a result). The assumption is that you start with a Knoppix image file and a persistent store file and finish with a new pair of files. You need to test the new pair before deleting the old so you need at least enough flash file storage for both but that is all the file storage you need.

The first USB remaster I did comprised four stages:

1. Prepare a new master file system. This is largely a copy of /UNIONFS.
2. Generate an ISO image from the master file sytem
3. Generate a cloop image by compression of the ISO image
4. Copy the cloop image to the USB

The obvious first refinement was to generate the cloop image directly on the USB filesystem.

Most people already combine the second and third stages by using a pipe but I couldn't because I was remastering the DVD MAXI edition and just ran out of (virtual) memory. I suspect somehow you could have enough virtual memory if you ran a 64-bit kernel but even swap space is temporary file space and I didn't want that. Instead I used squashfs that generates a compressed file system in one pass.

Note: you do not need to use squashfs if you remaster the CD edition. You just need a machine with 1 or 2 Gb of RAM.

The final refinement is to avoid creating a complete new master file system. I did this by feeding squashfs a list of directories thus:

sudo mksquashfs ${dirs} $DST/KNOPPIX/KNOPPIX.sq -b 65536 -noappend

The equivalent for the CD edition might be:

sudo genisoimage ${options} ${dirs} | sudo create_compressed_fs -B 65536 - $DST/KNOPPIX/KNOPPIX

Repeat: you do not need to use squashfs if you remaster the CD edition.



Which directories are fed to the remastering and which are not ?

I deliberately kept that simple: all directories under /UNIONFS except (possibly) etc, var, home, dev and lost+found.

lost+found should be empty. It isn't wanted anyway so I deleted anything in /UNIONFS/lost+found and then ignored it.

dev isn't really needed except as a back-up in case knoppix-autoconfig goes wrong. It is very small, so I included it in the remastering.

home is my stuff so it is not in the remastering. It stays in knoppix-data.img but needs a tidy up.

var is stuff that changes anyway so there seems little point including it in a remastering of a USB install (CD remaster yes). Unfortunately, it's rather big because I've updated the package lists. I could delete all those but they'd just come back next time I want to install anything. The best I could do was sudo rm /var/cache/apt/archives/*.deb to recover 50 Mb of space in knoppix-data.img.

etc is the difficult one. It contains a lot of files that are put there when packages are installed that I've never been anywhere near. These should go into the remaster. On the other hand there are files that have been changed since installation, many of which contain information about my machines and network, including passwords, that should not go into the remaster.

Really a second level sort is needed. Since etc isn't very big, this could be done in memory using /tmp. However, I decided since etc isn't very big and comprises a lot of little files that probably don't compress and decompress efficiently, I'd keep it out of the remaster and copy all of it into the new knoppix-data.img.

For etc, var and home, I included in the remaster the directory contents from /KNOPPIX so what was on the original DVD is carried forward.

Script ? Of course I have a script. I've remastered twice this way. The whole process, from start to rebooting with a new KNOPPIX.sq and knoppix-data.img takes less than 20 minutes. Not bad.

[utu]

Just like Fermat's last theorem.

[Forester]

Just like Fermat's last theorem.

I've had a word with my mate Fermat and he apologises. He latest theoretical script is only proven for one of nine Lagrangian points and for the least general point at that. He has ordered more UFDs so that he can test the hypothesis in Cartesian space.

[utu]

I think you may safely ignore the Noetherian domain
in cases where only wireless use is anticipated.

[Forester]

Attached is Fermat's Last Script that proves the hypothesis that it is possible to remaster a Knoppix USB installation without the need for temporary storage. It would be nice to just say QED and leave it at that but would not be terribly helpful.

The script assumes two USB installations, one of which is in use. It will determine automatically which is in use and overwrite the Knoppix image and persistent data of the other. The user has to declare where to find the two installations by editing the script. This should be a one time edit of the script.

There are three usage scenarios:

1. The good, where the USB installations are on different UFDs (aka pendrives, USB sticks ...). It is easy to lose (temporarily) one USB stick but two at once takes genius. Good also because this scenario does not require cheat codes.

2. The bad, where the cheat code fromhd= is used to boot USB installations on different file systems on the same USB stick. The redeeming feature of this arrangement is you're free to use proper file systems but otherwise the complexity is more that most people can be bothered with.

3. The ugly, where the cheat code knoppix_dir= is used to boot USB installations in different directories on the same file system on the same USB stick. This may be the scenario most would try first. Trials using a 'generic' USB were a disaster. Trials using a branded USB were successful. Both sticks were brand new. You have been warned.

The script contains instructions on how to declare your installations and requires an edit to run for the first time. Cheat codes not included.

After first time edit, the script will perform one of five actions when invoked correctly. Two are concerned with remastering the knoppix image and three with the persistent store. If you use still use knoppix.sh you will have to copy it yourself. It is suggested you consider using /etc/rc.local instead.

The Knoppix image can be remastered with either of the actions:

cloop image
squashfs image

Creation of the squashfs image is entirely at your discretion but does require the squashfs-tools package (not included). It is easy to convert between the two images and to compare execution times so you can pick the faster if you dare.

Note to boot a squashfs image requires prior modification of the /init script as discussed in the thread http://www.knoppix.net/forum/threads...hfs-ed-knoppix. Patches you can apply to /init are attached to post #29.

The actions:

create knoppix-data.img <size>
create knoppix-data-aes <size>

may (again at your discretion) be used to create a new persistent store for the remastered installation. These options allow the size or encryption of the persistent store to be modified during remastering (and yes, it is faster than using the backup / restore cheatcodes).

Finally the action:

prune knoppix-data

is used to copy /home etc. to the persistent store of the remastered installation.

The persistent store actions requires the loop device fix (not included), which is best done as a one off before remastering for the first time:

sudo rm -fr /dev/loop
sudo rm -fr /lib/udev/devices/loop

Since the upload facility of the forum limits file extensions, the attachment is named doremaster.txt. After download it is suggested you rename it to doremaster and make it executable.

[utu]

@ Forester

I'm getting ready for the plunge, but I'd like to clarify a few points.
I'm assuming the following:

1. Both USB's are assumed to be knoppix LiveUSB's of the same version,
in good working order. Presumably the working knoppix will have a persistent
store already, the target may or may not.

2. The two LiveUSBs while of the same version, may or may not be equally
well-developed from the same initial state.

3. The physicality of the two liveUSBs is immaterial. If one is a 2 Gb
SD Card and the other is a 4 Gb 'regular' USB, that's ok. And, noone cares
about the underlying format details for either, either.


I'm puzzled over the FLIP and FLOP
4. I would feel more comfortable saying that FLIP IS the working LiveUSB
and FLOP IS the target, with my own specifications. How do I know this
won't work out the other way?

Are we ok so far?

[dinosoep]

and what if I want to remaster but put the new knoppix image in /mnt ?
I know knoppix is on /dev/sdb1/knoppix but I do not know what to put in flop if I want it in mnt considering there is no knoppix install over there.
would be great if you clarify it a bit more

[utu]

And another thing, Forester:

I am further presuming that

1....The objective is to re-manufacture the KNOPPIX compressed image of
the target incorporating all previous knoppix-data.img improvements of
the source LiveUSB into it. And, that a new persistent store contains
only its minimum essential content, that is, nothing added by the user.
.....As a test, if one deferred to add persistent store to the target,
presumably the target would be as fully developed as the source, with
all its persistent store baggage, and not requiring any time to mount
a knoppix-data.img.


Since no good deed goes unpunished here, I offer this

2. If you've done all this, it would seem a relatively small matter
conceptually speaking, NOT to require a working LiveUSB to start with
for the target. Might there be some way to tap into a process already
in place on the master to prepare a carcass on the target on which to
hang the new files you have prepared. If not, you've left me with the
tedious task of going back to my LiveCD to prepare such carcasses.
I'm guessing I can omit the persistent store part of this tedium.

[Forester]

Gentlemen,

Think of the glass as half full, not half empty. The beer tastes better that way, though I admit many beers benefit from a slice of lemon →

Are you any good at drawing lines ? There are two kinds of lines: those you draw and those drawn for you. The second are usually called deadlines. Yes, you tick all the feature boxes but you know most only work downhill with a following wind.

Work is work. This is fun. I drew the line where I knew I could fill in all the details up to the line. That's what make me an amateur.

I started with a perfectly good script that did what I needed: squashfs image and prune knoppix-data for the bad scenario. Not a script that many would find useful. I've added the two other scenarios I can think of, the two potentially useful create actions, though I for one will probably never encrypt my data, and add the obvious cloop image I have no use for. I believe you and many others can now get the utility from script that I got out of the original.

Yes, there is stuff that can be added and I hope it will be. Each time I add a feature, I more there is to test (and retest) and the chances I miss something that hurts someone else goes up. Describing it clearly becomes harder and the chances that folks who never used the simple version are bewildered and confused into self-harm goes up. Why add features I don't need and that I don't know anyone else wants ?

This is an Open Source environment. That means someone has an idea, implements it, tests it, publishes it and then others try it, test it, discuss it, criticise it, amend it, extend it and you get a result that is better than one man can do (even with the supernatural help of a dead Frenchman).

utu and dinosoep, you have already raised some interesting points. I'll give feedback presently but it's time for my nightly TV fix.

[Forester]

and what if I want to remaster but put the new knoppix image in /mnt ?
I know knoppix is on /dev/sdb1/knoppix but I do not know what to put in flop if I want it in mnt considering there is no knoppix install over there.
would be great if you clarify it a bit more

I am not sure I understand your problem. Here's my understanding of how things work ...

Linux knows about 'block devices' and partitions on them (e.g sdxn). You can find out which it knows about by typing in cat /proc/partitions. For each of these Linux creates a device file in /dev (e.g. /dev/sdxn). For each of these (with a file system), Knoppix adds a entry in /etc/fstab and creates directory under /media so you can mount the file system with just mount /media/sdxn.

The doemaster script uses this to mount and umount file systems so you don't have to. You are not expected to use /mnt so you need to explain a bit more what you think you want so I can see what you mean.

For most people, the (only) hard drive is sda and their USB stick is sdb. Stick in a second USB and it will be sdc. So declare FLIP and FLOP to be /dev/sdb1/KNOPPIX and /dev/sdc1/KNOPPIX as the example in the script shows.

If you have two USB sticks plugged in when you boot and use the fromhd= cheat code to boot from /dev/sdc1/KNOPPIX, the script will spot this and remaster onto /dev/sdb1/KNOPPIX.

Does that help ?