USB Remasterig Using Minimum Real Estate

[Forester]

USB remastering traditionally requires lots of temporary file space. More than most people have spare on their USB sticks leading to the temptation to use nfts file space.

I've refined the process and managed to cut the temporary file space down to zero (and speed up the process considerably as a result). The assumption is that you start with a Knoppix image file and a persistent store file and finish with a new pair of files. You need to test the new pair before deleting the old so you need at least enough flash file storage for both but that is all the file storage you need.

The first USB remaster I did comprised four stages:

1. Prepare a new master file system. This is largely a copy of /UNIONFS.
2. Generate an ISO image from the master file sytem
3. Generate a cloop image by compression of the ISO image
4. Copy the cloop image to the USB

The obvious first refinement was to generate the cloop image directly on the USB filesystem.

Most people already combine the second and third stages by using a pipe but I couldn't because I was remastering the DVD MAXI edition and just ran out of (virtual) memory. I suspect somehow you could have enough virtual memory if you ran a 64-bit kernel but even swap space is temporary file space and I didn't want that. Instead I used squashfs that generates a compressed file system in one pass.

Note: you do not need to use squashfs if you remaster the CD edition. You just need a machine with 1 or 2 Gb of RAM.

The final refinement is to avoid creating a complete new master file system. I did this by feeding squashfs a list of directories thus:


sudo mksquashfs ${dirs} $DST/KNOPPIX/KNOPPIX.sq -b 65536 -noappendThe equivalent for the CD edition might be:


sudo genisoimage ${options} ${dirs} | sudo create_compressed_fs -B 65536 - $DST/KNOPPIX/KNOPPIXRepeat: you do not need to use squashfs if you remaster the CD edition.



Which directories are fed to the remastering and which are not ?

I deliberately kept that simple: all directories under /UNIONFS except (possibly) etc, var, home, dev and lost+found.

lost+found should be empty. It isn't wanted anyway so I deleted anything in /UNIONFS/lost+found and then ignored it.

dev isn't really needed except as a back-up in case knoppix-autoconfig goes wrong. It is very small, so I included it in the remastering.

home is my stuff so it is not in the remastering. It stays in knoppix-data.img but needs a tidy up.

var is stuff that changes anyway so there seems little point including it in a remastering of a USB install (CD remaster yes). Unfortunately, it's rather big because I've updated the package lists. I could delete all those but they'd just come back next time I want to install anything. The best I could do was sudo rm /var/cache/apt/archives/*.deb to recover 50 Mb of space in knoppix-data.img.

etc is the difficult one. It contains a lot of files that are put there when packages are installed that I've never been anywhere near. These should go into the remaster. On the other hand there are files that have been changed since installation, many of which contain information about my machines and network, including passwords, that should not go into the remaster.

Really a second level sort is needed. Since etc isn't very big, this could be done in memory using /tmp. However, I decided since etc isn't very big and comprises a lot of little files that probably don't compress and decompress efficiently, I'd keep it out of the remaster and copy all of it into the new knoppix-data.img.

For etc, var and home, I included in the remaster the directory contents from /KNOPPIX so what was on the original DVD is carried forward.

Script ? Of course I have a script. I've remastered twice this way. The whole process, from start to rebooting with a new KNOPPIX.sq and knoppix-data.img takes less than 20 minutes. Not bad.

[utu]

Just like Fermat's last theorem.

[Forester]

Just like Fermat's last theorem.

I've had a word with my mate Fermat and he apologises. He latest theoretical script is only proven for one of nine Lagrangian points and for the least general point at that. He has ordered more UFDs so that he can test the hypothesis in Cartesian space.

[utu]

I think you may safely ignore the Noetherian domain
in cases where only wireless use is anticipated.

[Forester]

Attached is Fermat's Last Script that proves the hypothesis that it is possible to remaster a Knoppix USB installation without the need for temporary storage. It would be nice to just say QED and leave it at that but would not be terribly helpful.

The script assumes two USB installations, one of which is in use. It will determine automatically which is in use and overwrite the Knoppix image and persistent data of the other. The user has to declare where to find the two installations by editing the script. This should be a one time edit of the script.

There are three usage scenarios:

1. The good, where the USB installations are on different UFDs (aka pendrives, USB sticks ...). It is easy to lose (temporarily) one USB stick but two at once takes genius. Good also because this scenario does not require cheat codes.

2. The bad, where the cheat code fromhd= is used to boot USB installations on different file systems on the same USB stick. The redeeming feature of this arrangement is you're free to use proper file systems but otherwise the complexity is more that most people can be bothered with.

3. The ugly, where the cheat code knoppix_dir= is used to boot USB installations in different directories on the same file system on the same USB stick. This may be the scenario most would try first. Trials using a 'generic' USB were a disaster. Trials using a branded USB were successful. Both sticks were brand new. You have been warned.

The script contains instructions on how to declare your installations and requires an edit to run for the first time. Cheat codes not included.

After first time edit, the script will perform one of five actions when invoked correctly. Two are concerned with remastering the knoppix image and three with the persistent store. If you use still use knoppix.sh you will have to copy it yourself. It is suggested you consider using /etc/rc.local instead.

The Knoppix image can be remastered with either of the actions:



cloop image
squashfs image
Creation of the squashfs image is entirely at your discretion but does require the squashfs-tools package (not included). It is easy to convert between the two images and to compare execution times so you can pick the faster if you dare.

Note to boot a squashfs image requires prior modification of the /init script as discussed in the thread http://www.knoppix.net/forum/threads/28484-Squashfs-ed-knoppix. Patches you can apply to /init are attached to post #29.

The actions:


create knoppix-data.img <size>
create knoppix-data-aes <size>may (again at your discretion) be used to create a new persistent store for the remastered installation. These options allow the size or encryption of the persistent store to be modified during remastering (and yes, it is faster than using the backup / restore cheatcodes).

Finally the action:


prune knoppix-datais used to copy /home etc. to the persistent store of the remastered installation.

The persistent store actions requires the loop device fix (not included), which is best done as a one off before remastering for the first time:


sudo rm -fr /dev/loop
sudo rm -fr /lib/udev/devices/loop
Since the upload facility of the forum limits file extensions, the attachment is named doremaster.txt. After download it is suggested you rename it to doremaster and make it executable.

[utu]

@ Forester

I'm getting ready for the plunge, but I'd like to clarify a few points.
I'm assuming the following:

1. Both USB's are assumed to be knoppix LiveUSB's of the same version,
in good working order. Presumably the working knoppix will have a persistent
store already, the target may or may not.

2. The two LiveUSBs while of the same version, may or may not be equally
well-developed from the same initial state.

3. The physicality of the two liveUSBs is immaterial. If one is a 2 Gb
SD Card and the other is a 4 Gb 'regular' USB, that's ok. And, noone cares
about the underlying format details for either, either.


I'm puzzled over the FLIP and FLOP
4. I would feel more comfortable saying that FLIP IS the working LiveUSB
and FLOP IS the target, with my own specifications. How do I know this
won't work out the other way?

Are we ok so far?

[dinosoep]

and what if I want to remaster but put the new knoppix image in /mnt ?
I know knoppix is on /dev/sdb1/knoppix but I do not know what to put in flop if I want it in mnt considering there is no knoppix install over there.
would be great if you clarify it a bit more :D

[utu]

And another thing, Forester:

I am further presuming that

1....The objective is to re-manufacture the KNOPPIX compressed image of
the target incorporating all previous knoppix-data.img improvements of
the source LiveUSB into it. And, that a new persistent store contains
only its minimum essential content, that is, nothing added by the user.
.....As a test, if one deferred to add persistent store to the target,
presumably the target would be as fully developed as the source, with
all its persistent store baggage, and not requiring any time to mount
a knoppix-data.img.


Since no good deed goes unpunished here, I offer this

2. If you've done all this, it would seem a relatively small matter
conceptually speaking, NOT to require a working LiveUSB to start with
for the target. Might there be some way to tap into a process already
in place on the master to prepare a carcass on the target on which to
hang the new files you have prepared. If not, you've left me with the
tedious task of going back to my LiveCD to prepare such carcasses.
I'm guessing I can omit the persistent store part of this tedium.

[Forester]

Gentlemen,

Think of the glass as half full, not half empty. The beer tastes better that way, though I admit many beers benefit from a slice of lemon → :)

Are you any good at drawing lines ? There are two kinds of lines: those you draw and those drawn for you. The second are usually called deadlines. Yes, you tick all the feature boxes but you know most only work downhill with a following wind.

Work is work. This is fun. I drew the line where I knew I could fill in all the details up to the line. That's what make me an amateur.

I started with a perfectly good script that did what I needed: squashfs image and prune knoppix-data for the bad scenario. Not a script that many would find useful. I've added the two other scenarios I can think of, the two potentially useful create actions, though I for one will probably never encrypt my data, and add the obvious cloop image I have no use for. I believe you and many others can now get the utility from script that I got out of the original.

Yes, there is stuff that can be added and I hope it will be. Each time I add a feature, I more there is to test (and retest) and the chances I miss something that hurts someone else goes up. Describing it clearly becomes harder and the chances that folks who never used the simple version are bewildered and confused into self-harm goes up. Why add features I don't need and that I don't know anyone else wants ?

This is an Open Source environment. That means someone has an idea, implements it, tests it, publishes it and then others try it, test it, discuss it, criticise it, amend it, extend it and you get a result that is better than one man can do (even with the supernatural help of a dead Frenchman).

utu and dinosoep, you have already raised some interesting points. I'll give feedback presently but it's time for my nightly TV fix.

[Forester]

and what if I want to remaster but put the new knoppix image in /mnt ?
I know knoppix is on /dev/sdb1/knoppix but I do not know what to put in flop if I want it in mnt considering there is no knoppix install over there.
would be great if you clarify it a bit more :D

I am not sure I understand your problem. Here's my understanding of how things work ...

Linux knows about 'block devices' and partitions on them (e.g sdxn). You can find out which it knows about by typing in cat /proc/partitions. For each of these Linux creates a device file in /dev (e.g. /dev/sdxn). For each of these (with a file system), Knoppix adds a entry in /etc/fstab and creates directory under /media so you can mount the file system with just mount /media/sdxn.

The doemaster script uses this to mount and umount file systems so you don't have to. You are not expected to use /mnt so you need to explain a bit more what you think you want so I can see what you mean.

For most people, the (only) hard drive is sda and their USB stick is sdb. Stick in a second USB and it will be sdc. So declare FLIP and FLOP to be /dev/sdb1/KNOPPIX and /dev/sdc1/KNOPPIX as the example in the script shows.

If you have two USB sticks plugged in when you boot and use the fromhd= cheat code to boot from /dev/sdc1/KNOPPIX, the script will spot this and remaster onto /dev/sdb1/KNOPPIX.

Does that help ?

[dinosoep]

I know for sure I've set my flip/flop wrong, They were /dev/sdb1/KNOPPIX and /dev/loop2 respectively.
For some reason it just gave me "error on line 61" and then something with the = sign.
went to line 61 and removed it :p

then it always gave me this:


Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.


So I went to see where grep was used, it was where you were trying to figure out what directory was in use and what was free.
So I commented out the ifs and simply made it


DST="${FLOP}";


at my great suprise that actually started to create a squashfs filesystem at
/mnt-system/
guess my flop is kinda wrong.

But hey, at least I got it running with my messing around.
I do think that that equal sign is not supposed to be there on line 61 and that those grep errors where caused because of my wrong flipflops?
I'm going to read it when my brain is relaxed :p and try to understand it completely although It won't be that big of a deal because the script is well comented :D

[utu]

@ Forester

I think I understand what you intend to do here,
but I find the user interface most unfriendly.

I'd be glad to set some variables and make some
choices, but none I try seem to be correct.

I suggest perhaps you make it more clear which
lines need attention.

I've prepared my 2nd USB but I'll wait to see
how the discussion goes before just tweaking away.

A good example for cloop, not-encrypted and pruned
with /dev/sdb1 as master and dev/sdc1 as target
would be most helpful.

[Forester]

I've seen plenty of your posts and you guys aren't that dumb. :)

You're winding me up. :p

Fermat says there are advantages to being dead. :mrgreen:

[utu]

Progress so far:

doremaster cloop image creates a fatter KNOPPIX compressed file.
doremaster create knoppix-data.img <anything> doesn't work.
doremaster prune knoppix-data doesn't work, with or without
a persistent store.

target LiveUSB still boots, but doesn't appear to benefit from
its larger size.

doremaster create knoppix-data.img <...> does not recognize an input
such as 2000 or 2000 Mb or <2000> or <2000 Mb>.

[dinosoep]

"you guys aren't that dumb" xd
indeed I have learned :D
after using the script a new knoppix.sq was created like I said but as it was just a test, I decided to delete it. But for some strange reason the original knoppix.sq was removed. so after a reboot it couldn't find it anymore (logical)

Now comes the smart part: I had a backup :)
Kinda saved my day as you can imagine :p
Maybe I should decide to simply try to understand the script befvore I try some more stuff with it.
and forester, don't overestimate me :cool:

[Forester]

after using the script a new knoppix.sq was created like I said but as it was just a test, I decided to delete it. But for some strange reason the original knoppix.sq was removed. so after a reboot it couldn't find it anymore (logical)


Oh dear. Your knoppix.sq is mounted so you can see it at /mnt-system/KNOPPIX. There is nothing to stop you mounting the file system again at some other location. If you did this, you'd have what might appear to be two knoppix.sq files but when you delete one the other vanishes as well. That may be what happened here. I don't know. I wasn't there.

You used /dev/loop2. Why did you think that was a good idea ? It reminds of when I was 15 I stuck the points if a pair of scissors into the main sockets on the bench in front me in the middle of a biology lesson to just to see what would happen. I didn't hear the bang. I was told afterwards it was very loud.



Now comes the smart part: I had a backup :)


Yep, you're learning. That's why I went and bought two USB sticks for test purposes.



Maybe I should decide to simply try to understand the script before I try some more stuff with it.


Careful, you might grow wisdom. The number of times I've decided I need software that does X, googled a bit, skimmed through a man page and then spent the rest of night/week/month trying to the convince the software I was right and it was wrong .... I've learnt at least to try to use the software the way it was intended and then decide if it meets my needs or not.

Think long term. Today you may remaster from your favourite stick to a backup and then use the backup stick for a while. Next time you would remaster from the backup to your favourite stick. Whatever you call your sticks, you alternate between them. That's why I called them flip-flop. I could have called them ping-pong but if I'd wanted to be obscure I'd have called them Hunky Dory or Castor and Pollux.



and forester, don't overestimate me :cool:


I don't. I just don't know what to estimate. I once showed an old lady how to send and receive e-mail. She had a daughter in Australia and that does things to people. The first problem was she didn't know what 'click the left mouse button' meant. At least she said so, I got useful feedback and we could move forward. Here's what doesn't work: feedback of the form "XYZ didn't work" with no other information but the clear implication that someone is stupid.

Now, were you try prune knoppix-data and report you an got error of the form:


losetup: could not find any free loop deviceI would suggest that it is just possible you forgot to remove /dev/loop etc. as the instructions in post #5 recommended. But you haven't and neither has utu so I can't make that suggestion can I ?

[utu]

forester@didact.edu (forester@didact.enu)

Please put me on your mailing list for doremaster 2.0.
I am declaring victory forthwith and retiring from
this scavenger hunt.

I'm sure there was a pony in there somewhere.

[Capricorny]

When working with external storage and doing lots of writing, I always try to put in a USB harddisk in the mix. I use both "ready-made" and "home-made", typically mounting a disk from a harddisk upgrade into a cabinet. Then I can use several partitions and file systems on one drive, and typically leave the USB "master install" read-only until the new version has been testen.