Unofficial cheatcodes to backup and restore persistent data
Attached are patches to /init that implement cheatcodes to backup and restore the Knoppix persistent store image inspired by ideas from utu.
Persistent store on a USB stick or a picture card is great but these are easily lost or the file system damaged so frequent backups are a good idea. Easy, just drag and drop onto your hard drive next time you aren't running Knoppix. If that is just about never, then backup costs two reboots. The drag and drop method also requires a chunk of hard drive space the size of the persistent store, which is not always available. As utu showed, it should be possible to backup and restore persistent store using Knoppix provided you get in early enough in the boot sequence. You could boot with the debug cheatcode and do it all by hand. Instead here are a couple of cheat codes to do the job safely.
The first example might be backup to hard drive, the second to the USB stick / picture card. The first is to be preferred. You can choose where to put the backup. If you don't specify the full /aDevice/someDirectory/theFilename, the cheat code will try to fill in the missing parts from the template /mnt-system/$knoppix_dir/knoppix-data.tgz.
The backup is reasonably straight forward though rather slow. How slow depends on how much data is in your persistent store rather than how big the store is. The result is a compressed tar file that will be a lot smaller than the persistent store itself.
The restore has a surprise. It will first delete your current persistent store on the grounds that you probably need to restore because your persistent store is corrupt and cannot be used. This means the first you know about it is the 'you don't have a persistent store yet' screen pops up and if this takes you by surprise you won't type in the size quick enough and you'll have to press ctrl-alt-del to reboot and try again.
Both backup and restore seem to take a unreasonably long time when compared with similar operations done from Knoppix proper. Perhaps the BusyBox version of tar and zip are unable to exploit the multi-core capabilities of modern processors.
If you use the two cheat codes at the same time, the backup happens first. You can use this to resize your persistent store or to convert your persistent store to/from encrypted, password protected, format.
The backup file always has the extension .tgz (aka .tar.zip) and can be unpacked with tar -xzf.
The backup is not encrypted and is not password protected. The extension .zip is forseen for that but the implementation is left as an exercise for the reader.
There are two attachments to this post. They are text file patches for the /init script in the sense of the (Linux) User Command patch(1). To apply them, you would have to first unpack /mnt-system/boot/syslinux/minirt.gz. See the Wiki page on remastering.
There are two parts to the patch. The second, backup_data.txt, actually implements the new cheat codes but requires a pair of more general purpose functions implemented in the first patch, mountbypath.txt. These patches, by design, not to overlap with the (original) patch for the knoppix_data cheat code.
A most impressive response to my wishlists. Thank you very much.
Several comments and questions come to mind. For extra credit, of course.
It seems to me you have answered a much larger question as well:
how to add 'unofficial cheatcodes' to Knoppix.
Have you invoked these patches yourself?
Couldn't these same functions appear as well on the LXDE menu,
perhaps with a few additional strokes?
I have a good example on deletions. I 'deleted' adriane & orca from
my 6.4.4, thinking I'd get a few bytes for other things. I forgot
these were permanently in the big KNOPPIX compressed image.
Now I'd have to 'add them back', so to speak.
Clever double entendre here.
I also noted that the tar/gz and untar/gz operations took a lot
longer than I would have expected, considering how fast
the KNOPPIX compressed file comes up.
I presumed this might get me into the cloop vs whatever catfight,
so I didn't bring it up earlier.
This question will allow you to calibrate my expertise:
Never having used 'patch' before, can I apply this command
in a console on a bash script, or do I have to be working
with source code in a special editor, preparing for a
re-compile?
I've done lots of one line edits to bash scripts, but
no real structural mods.
In order to avoid a re-compilation of source code, as
an alternative might I use the patch script to guide
a re-arrangement of the bash code just using a simple-
minded edit inside an ordinary text editor?
A most impressive response to my wishlists. Thank you very much.
Several comments and questions come to mind. For extra credit, of course.
A most impressive list of questions. I'm sorry, but I don't need extra course credits: I'm far too old.
Originally Posted by utu
It seems to me you have answered a much larger question as well:
how to add 'unofficial cheatcodes' to Knoppix.
Very carefully.
Originally Posted by utu
Have you invoked these patches yourself?
Yes. I don't have a comprehensive test suite or an army of test engineers. So no guarantees.
Originally Posted by utu
Couldn't these same functions appear as well on the LXDE menu,
perhaps with a few additional strokes?
In my opinion, no. You should not back up an active file system and you certainly can't restore to one. The cheat codes work directory on /KNOPPIX-DATA before the UNIONFS is created.
Originally Posted by utu
And, your time zone is what ?
All this is done in MOT (My Own Time). I surmise you are on GST (German Student Time).
The clever Wiki software tells me all times are in GMT from which I infer vBulletin™ have deduced my time zone from my IP address. Scary.
Originally Posted by utu
I also noted that the tar/gz and untar/gz operations took a lot
longer than I would have expected, considering how fast
the KNOPPIX compressed file comes up.
Yes I noted that too. Remember compression generally takes a lot longer than decompression. The cheat codes are using /bin/busybox tar, not the real thing. It might be the real thing can make full use of your multi-core CPU and Giga bytes of memory while the BusyBox version can't. I don't know.
You could try BusyBox with lzma compression (that would please kl522) or you could try running the real tar from the /init script. KK has some scary stuff in /init that runs real programs in their own chroot environment. Test your mettle and all that.
Originally Posted by utu
This question will allow you to calibrate my expertise:
No, you don't need a special editor to create a patch. The command line will do just fine.
The simple theory is you generate the patch file by comparing the original file with the changed file thus:
Code:
diff -Naur script.org script > script.patch
You can later apply the patch to an unaltered copy of script with:
Code:
patch < script.patch
I find this works quite well for saving my changes to 'system' configuration files and then reapplying my changes to new configuration files after an OS upgrade or something.
The diff program generates differences with a few lines of context either side and the patch program uses the context, not just the lines numbers, to figure out where the patch should go. Great if someone has changed the file so all the lines numbers have changed. Not so great if they've change the same lines so the context is different. You get a conflict that basically you have to resolve by hand.
That's the basics. It can get a lot more involved than that but since patches are how Open Source software developers get stuff done I guess knowing a little about diff and patch is
Here's a new pair of patches for the backup and restore persistent data. I've added the go faster stripe you complained was missing. I find the backup is 3-4 times as fast as before but restore is just as slow as it was. I suspect how long it takes is, for me, now dictated by how fast the software can read from / write to my USB stick.
Did you know you can remove old patches with the -R option (assuming you've still got them) and then you can apply new ones as before.
I must admit I've not done MY homework as promised. Tax time arriveth, you know.
I am taking up our quest again now, however.
To recoup, here's what I think is the current state of affairs:
you have invented patches to give Knoppix 6.4.4 two new cheatcodes, backup
and restore. These two capabilities, in effect, automate the utilzation of the
capability already in 6.4.4 which manages a file update*.tar.gz in a manner to
allow the backing-up and restoration-of the knoppix-data.img file.
These newer offerings utilize squashfs to speed things up beyond the capabilities
of the initial offerings. Only the backup speed improved significantly, but this
may be due to limitations in writing to the USB. Please confirm or correct me here.
In an earlier exchange, you indicated these capabilites could not be realized as
menu items, but only as cheatcodes. I'd like to re-visit this in consideration
of how I might like to apply all this magic. I really would like backup to be a
menu choice to be used during normal on-line use of 6.4.4. Restore I would use
as a cheatcode in addition to my normal DEFAULT set of cheatcodes.
The meaning of backup here is to create a tar.gz of a snapshot of KNOPPIX-DATA,
using squashfs, since it offers an advantage; but do this as a menu choice,
or as a console command, but not as a cheatcode, per se. This I realize is a
complication of your approach to backup. And, I know you are hesitant on doing
the back-up on-line, but I'm not sure it's a show-stopper.
The meaning of restore might be simply to find such a snapshot and restore it
simply per the normal update* process of 6.4.4, since squash seems not to offer
any improvement; do this as a cheatcode to be appended to a normal default.
This might actually be a simplification of your approach to restore.
I need to review for myself if there is any complication in appending a cheatcode
to my normal DEFAULT set of cheatcodes; I don't recall.
I have in the back of my mind how nice Win7 allows system image snapshots, and how
handy these are when something goes funny.
The faster restore is a holy grail for another day. I don't know why the
KNOPPIX file can be unpacked so fast, but not the knoppix-data.img; doesn't make
sense, or it's just not fair, somehow.
I wish I could say the confusion is all mine and apologise profusely.
Originally Posted by utu
To recoup, here's what I think is the current state of affairs:
you have invented patches to give Knoppix 6.4.4 two new cheatcodes, backup
and restore.
Yes. I'd rather say developed or even elaborated than invented.
Originally Posted by utu
These two capabilities, in effect, automate the utilzation of the
capability already in 6.4.4 which manages a file update*.tar.gz in a manner to
allow the backing-up and restoration-of the knoppix-data.img file.
No. I don't use the update*.tar.gz facility. It is not appropriate.
Originally Posted by utu
These newer offerings utilize squashfs to speed things up beyond the capabilities
of the initial offerings.
No. This has nothing to do with squashfs.
The first offering used the BusyBox with tar+zlib. The new offering contrives to the full tar + gzip programs available to use from the Knoppix desktop. This cleverness uses facilities KK already uses himself in /init. to run programs not available in BusyBox or minirt, most notably to handle the creation of encrypted knoppix-data.aes files.
Originally Posted by utu
Only the backup speed improved significantly, but this
may be due to limitations in writing to the USB. Please confirm or correct me here.
Yes. That is what I wrote.
Originally Posted by utu
I really would like backup to be a menu choice to be used during normal on-line use of 6.4.4.
My first question on this forum was how do I add an item to the LXDE menu. As I recall your answer was "I'd like to know too".
You can tar + gzip /KNOPPIX-DATA any time you like. Even a dodgy backup is better than none.
For me, the trouble with a menu item is I'd forget to use it and after a week I'd forget it was there. The same is true for a cheat code but less so, if you know what I mean.
Originally Posted by utu
Restore I would use as a cheatcode in addition to my normal DEFAULT set of cheatcodes.
Yes. There is little choice here. You can, naturally, extract individual files from the backup if that's what you need to do.
Originally Posted by utu
The meaning of backup here is to create a tar.gz of a snapshot of KNOPPIX-DATA,
using squashfs, since it offers an advantage; but do this as a menu choice,
or as a console command, but not as a cheatcode, per se.
You repeat yourself. No. This has nothing to do with squashfs.
It has to do with 'snapshot' . Think of a snapshot as a digital photograph of your persistent data. You want a clean crisp photo taken while the subject is not moving to avoid blurring of the details. Using a cheat code is the only way of ensuring the photo is taken before the subject starts moving. Backup from menu item means the subject is moving and you won't find out if some crucial detail is blurred until you, out of necessity, have to restore.
Originally Posted by utu
This I realize is a complication of your approach to backup. And, I know you are hesitant on doing the back-up on-line, but I'm not sure it's a show-stopper.
No. Taking a shapshot from the command line is so simple I do it everyday but I boot Debian to ensure I take a clean snapshot of Knoppix. Developing the user space shell script I use from Debian was a darn sight easier than adding cheat codes to /init.
Originally Posted by utu
The meaning of restore might be simply to find such a snapshot and restore it
simply per the normal update* process of 6.4.4, since squash seems not to offer
any improvement; do this as a cheatcode to be appended to a normal default.
This might actually be a simplification of your approach to restore.
No. There is nothing normal about this or the update* process. Using the update* process involves more error prone manual intervention than a simple cheat code but that is not the reasons using update* is a bad idea. The reason it to do with ensuring deleted files stay deleted.
If someone were to use these cheat codes on a regular basis, I'd suggest they add new boot targets to syslinux.cfg. Instead of typing in knoppix backup_data=/dev/sda6 at the boot: prompt and getting it wrong half the time, they'd type in knoppix_with_backup or something shorter they can easily remember.
Originally Posted by utu
I need to review for myself if there is any complication in appending a cheatcode
to my normal DEFAULT set of cheatcodes; I don't recall.
See previous remark.
Originally Posted by utu
I have in the back of my mind how nice Win7 allows system image snapshots, and how
handy these are when something goes funny.
That's a system function more akin to saving and restoring from the file system journal. What I have done is a simple user space add-on.
File system journal recovery is for hard drives where the risk of file system corruption through user error is greater than the risk of damage or loss of the hardware medium.
What I have done is intended as much as a protection against the loss or temporary misplacement of my USB stick as it is against the risk that I remove the USB stick from its slot without first shutting down Knoppix.
Originally Posted by utu
The faster restore is a holy grail for another day. I don't know why the
KNOPPIX file can be unpacked so fast, but not the knoppix-data.img; doesn't make
sense, or it's just not fair, somehow.
Umm. Nonsense. Unpacking or decompression is always faster than packing or compression. Reading from a USB stick is always faster than writing.
The KNOPPIX file is read and unpacked but only the bits needed, not the whole thing. By definition, backup / restore have to read / write all the bits of persistent store.
Here are my (unrepresentative times):
~ 60 s copy of 1 Gb knoppix-data.img file from USB to hard drive under Debian 6.
~ 30 s tar + gzip of knoppix-data.img file from USB to hard drive under Debian 6.
~ 30 s backup with revised cheat code from USB to hard drive
~ 90 s backup with original cheat code from USB to hard drive
My knoppix-data.img file is currently two thirds full and its backup is almost a ¼ Gb.
The Debian tar + gzip is faster than the cheat code but not when you take into account the time needed to boot and shutdown Debian.
The backup takes several minutes at work because I've a virtual machine with a virtual USB 1.1 controller that is very much slower than a real USB 2.0 controller. The go faster stripe has no effect where I need it most.
Reply With Quote
Originally Posted by utu
