a new & improved, self-contained Knoppix personal backup

[utu]

.
I posted a simple backup program on this forum a while back at:
http://www.knoppix.net/forum/threads/29332-Self-Contained-Backup-for-Persistent-Store?highlight=backup
I've refined it a bit. Mine now requires only about 77M, about half the space it formerly required.
Not only is this a nice backup, you can use the same procedure to re-establish all your
preferences on the next upgrade of Knoppix, as I just did.

Its use is as follows. More on this in the original reference, if needed.

When you make a new LiveUSB, don't use every last MB for persistence;
leave a little in /mnt-system, say 200MB or so. That is, just devote 200MB LESS to persistence.

After you have modified your new Knoppix LiveUSB somewhat, add 'backup' to your /home/knoppix directory; then, occasionally, as root, exercise 'backup' to put an update.tar.gz file into /mnt-system. Whenever you modify your new system, do another backup to keep current.

If you ever spoil* your knoppix-data.img, delete knoppix-data.img; move update.tar.gz
from the mnt-system into the same directory that knoppix-data.img was in and reboot;
re-establish the persistence file when prompted. Assuming Knoppix is again functioning,
either delete update.tar.gz now or move it back into the /mnt-system directory; don't
leave it in the same directory as knoppix-data.img. Make a new backup soon in any event.

Here's the little program:


#!/bin/bash
#
# 'backup'
#
# This script captures personal folders, adjustments, choices & tweaks,
# but does NOT account for program additions or removals
# relative to the LiveUSB's initial configuration;
# e.g., changes achieved using Synaptic.
#
# Note: do 'chown root:root backup' & 'chmod +x backup'.
cd /
echo -e 'Tarring data to update.tar in /tmp..\c'
tar -cf /tmp/update.tar KNOPPIX-DATA/home/
tar -rf /tmp/update.tar KNOPPIX-DATA/etc/
tar -rf /tmp/update.tar KNOPPIX-DATA/root/
tar -rf /tmp/update.tar mnt-system/boot/syslinux/syslinux.cfg
echo '..Done.'
#
echo 'Zipping update.tar to update.tar.gz in /tmp.'
echo -e 'Patience; this may take a little time..\c'
cd /tmp #tar -tf update.tar > tar.lst
gzip update.tar; echo '..Done.'
#
echo -e 'Moving update.tar.gz to /mnt-system..\c'
mv update.tar.gz /mnt-system; echo '..Done.'
#
echo 'All done.'
#
exit 0
____________________________

* I consider knoppix-data.img 'spoiled' either when my LiveUSB won't boot, or when the OS
doesn't seem to act as it should. I've spoiled my share.
In my experience this has usually been a result of one of three errors on my part:
1..Changing the plugged status of the USB while it is doing a read or write operation;
2..Bringing in too much additional program content via Synaptic; or
3..Making some OS change as ROOT that didn't turn out right.

[Werner P. Schulz]

If you ever spoil* your knoppix-data.img, delete knoppix-data.img; move update.tar.gz from the mnt-system into the same directory that knoppix-data.img was in and reboot;Before this step you have to boot Knoppix with cheatcode "knoppix noimage" to avoid problems with persistent memory.

At next boottime (without any cheatcode) I run in trouble. Knoppix recognize the file update.tar.gz and uncompress it and also ask for new persistent memory. But it doesn't start to LXDE. I can only change to a terminal and reboot. I didn't find a explanation for this curious behaviour.

Greetings Werner * http://www.wp-schulz.de/knoppix/summary.html
Own Rescue-CD with Knoppix (Knoppix V6.7.1 remaster)

[Werner P. Schulz]

A little suggestion
STOR=update$(date +'%m%d%y').tar.gz
cd /
tar -czf /mnt-system/$STOR KNOPPIX-DATA/home/ KNOPPIX-DATA/root/ \
mnt-system/boot/syslinux/syslinux.cfg

[utu]

Greetings, Werner.

I hope the goblins have returned your lxmenu to your top-line-lxpanel by now.

Sorry you are having problems with my little backup program.
I can't duplicate your complaint here.
The steps are:

1. Use my program prepare /mnt-system/update.tar.gz.
2. Move update.tar.gz to /mnt-system./KNOPPIX temporarily.
3a If the system is behaving, leave /mnt-system/KNOPPIX/knoppix-data.img where it is
and reboot.
Screen should indicate both knoppix-data.img and update.tar.gz being read-in.
When system comes up, move update.tar.gz one level back to /mnt-system for safe-keeping.
3b If the system is not behaving, either delete knoppix-data.img or move it out of
/mnt-system/KNOPPIX/ and reboot.
Screen should first ask you to re-establish persistence (my words). Do so.
As I recall, once persistence is established, update.tar.gz is read in.
If that's not the case, reboot and things should proceed as in 3a.

I appreciate your refinements to my crude programming and will incorporate these
to some extent. I sometimes leave things 'spread out' so I can more easily remember
and/or change them. I am mindful of the elegance of brevity, but my short-term memory
isn't so good as it once was.

I have been lucky up to now not to get my update*'s confused. It is time I used a
technique such as yours to keep them distinctly identified. Thanks for the idea.

[Werner P. Schulz]

If the system is not behaving, either delete knoppix-data.img or move it out of
/mnt-system/KNOPPIX/ and reboot.If I do so, I lose free space for persistent memory (=pm):

a) flash-disk Installation of my CD on a 4GB USB stick
b) after first boot Knoppix offers 3.3GB for pm, I selected 2GB
c) deleted 'knoppix-data.img'
d after next boot Knoppix offers 1.3GB for pm, I selected 1GB
e) deleted 'knoppix-data.img'
f) after next boot Knoppix offers 0.3GB for pm, I selected 0.2GB

Therefore you can only delete 'knoppix-data.img' after booting with cheatcode "knoppix noimage"

Step 2 and 3a isn't necessary. The contents of your just created update.tar.gz is allready within the old knoppix-data.img.

Now I did it in single steps and all went well:
1. create 'update tar.gz' within '/mnt-system'
2. reboot (cheatcode "noimage") and delete 'knoppix-data.img'
3. reboot and create new pm
4. reboot and move 'update.tar.gz' to '/mnt-system/KNOPPIX/'
5. reboot and move 'update.tar.gz' to '/mnt-system' or store it anywhere else

Greetings Werner * http://www.wp-schulz.de/knoppix/summary.html
Own Rescue-CD with Knoppix (Knoppix V6.7.1 remaster)

[utu]

Hi, Werner

A couple of fine points:

1..I prefer to do the tarring and zipping in the system-&-persistence side of the USB,
so the /mnt-system side only has to be as big as the zipped file.
Remember, I usually use just a 2 Gb USB.
So, I tar to /tmp , gzip there and mv the result to /mnt-system.

2..I'm not following you yet with the noimage cheatcode.
I guess if I've ever gotten an unreasonably small value offered
for persistence, I may have just started over & reformatted
the whole USB.

3...Most of the required space is for the browser's purposes.
I've just changed to IceWeasel 7 and my storage has gone from
77Mb to 44Mb. I wonder if you can confirm this.

[utu]

An update on the 'update' program
.
Here's an update using Werner's dating method, even in the status lines.
I've included a listing of tarred files, mailed to root, also suitably dated.

I now backup ~124 Mb into 48 Mb of tgz on /mnt-system.
IceWeasel alone accounts for 91 Mb of the 124 Mb it might be noted.



#!/bin/bash
#
# new.backup: do 'chmod +x new.backup' & 'chown root:root new.backup'
#
cd /
STOR=update$(date +'%m%d%y').tar
echo -e 'Tarring data to '$STOR' in /tmp..\c'
tar -cf /tmp/$STOR KNOPPIX-DATA/home/ KNOPPIX-DATA/etc/ \
KNOPPIX-DATA/root/ mnt-system/boot/syslinux/syslinux.cfg
echo '..Done.'
cd /tmp
STORZ=$STOR.gz
echo 'Zipping to '$STORZ' in /tmp.'
tar -tf $STOR > ~/$STOR.lst
echo -e 'Patience; this may take a little time..\c'
gzip $STOR; echo '..Done.'
#
echo -e 'Moving '$STORZ' to /mnt-system..\c'
mv $STORZ /mnt-system; echo '..Done.'
#
echo 'All done.'
#
exit 0

[Werner P. Schulz]

I changed the procedure a little:

1. create 'update tar.gz' within '/mnt-system'
2. reboot (cheatcode "noimage") and delete 'knoppix-data.img'
3. reboot, create new persistent memory and

tar -xzf /mnt-system/update.tar.gz -C /4. reboot

The way in your update on the 'update'
over '/tmp' are only for clarity reasons but without any benefit.

Greetings Werner * http://www.wp-schulz.de/knoppix/summary.html
Own Rescue-CD with Knoppix (Knoppix V6.7.1 remaster)

[utu]

Hello, again, Werner.

Just to make sure we see eye to eye on this, I see two different scenarios for using this backup idea:

1..After making some personal additions, and otherwise the LiveUSB with persistence seems to be working ok;
2..After making some personal additons, and the LiveUSB with persistence isn't behaving properly.

In case 1, we want to preserve BOTH our personal stuff & the PROGRAM changes inherent in persistent store.
In case 2, we want to preserve our personal stuff, but we will have to sacrifice PROGRAM changes & start a new persistent store, which will not include any program changes outside of home/, root/, etc/ and syslinux.cfg.

I think this means we may only want to use the noimage cheatcode for case 2; specifically NOT for case 1.
For myself, dealing with case 2, I'd try the noimage first; if it gives a reasonable choice for persistent store
that's the way to go. If not, then I'd repartition the USB with GParted and make a new LiveUSB from scatch,
then add the update disregarding any use of noimage (in this specific instance).

[Werner P. Schulz]

If you boot Knoppix with cheatcode "noimage" you can only see persistent memory in the (compressed) file '/mnt-system/KNOPPIX/knoppix-data.img'. If you boot without this cheatcode you can also see the persistent memory in the (decompressed) directory '/KNOPPIX-DATA/'.

In "case 1" from you, all is done by Knoppix.

You can change personal settings in '/home/knoppix/' or '/root/', you can install or deinstall programs in '/usr/' and the program-settings in '/etc/'. But in reality Knoppix writes all this stuff in '/KNOPPIX-DATA/' using "UNIONFS".

To prevent later trouble you can do a backup (your "update.tar.gz") from your personal stuff, which is a part of 'KNOPPIX-DATA/', and leave out the remaining of '/KNOPIX-DATA/'.

In "case 2" you can
a) only delete persistent memory alone, or
b) reformat the whole USB flash-drive and reinstall Knoppix. But why this expenditure?

[utu]

If you boot Knoppix with cheatcode "noimage" you can only see persistent memory in the (compressed) file '/mnt-system/KNOPPIX/knoppix-data.img'. If you boot without this cheatcode you can also see the persistent memory in the (decompressed) directory '/KNOPPIX-DATA/'.

I would not have expected this to be the case.

I would expect the cheatcode noimage to suppress the un-compressed .img file which has both the errors and the new program material in it, leaving you with just the initially installed compressed KNOPPIX file.

The .img file is not compressed. That would take up too much time on shut-down, for one thing.
I can see that the cheatcode noimage may indeed serve a useful purpose, but I'd not describe it the way you have.
And, if the .img is ok, I don't want to change it.

[Werner P. Schulz]

The file 'knoppix-data.img' or in case of encryption 'knoppix-data.aes' is always compressed! Have a look at it with midnightcommander.

If you boot Knoppix, it is mounted (and decompressed) via '/dev/loop0' to '/KNOPPIX-DATA'. If you boot with cheatcode "noimage" nothing happens with '..img' or '..aes'

Greetings Werner * http://www.wp-schulz.de/knoppix/summary.html
Own Rescue-CD with Knoppix (Knoppix V6.7.1 remaster)

[utu]

.
I've re-considered the exact terms of my comments, noting the following:

My KNOPPIX-DATA is about........ 385 Mb
My knoppix-data.img is about..... 943.7 Mb
My original CD's KNOPPIX is........ 722.4 Mb.

I think the CD's KNOPPIX is compressed; KNOPPIX-DATA is un-compressed; and

I expect that knoppix-data.img must be some combination of the two not requiring
any wholesale compression of the entire uncompressed UNION at shutdown.

[Werner P. Schulz]

The size of 'knoppix-data.img' remains always as big as the size of persistent memory you created once upon a time. The size of '/KNOPPIX-DATA' depends of all your work within your flash-disk Installation with persistent memory and changes with each action.

I think the CD's KNOPPIX is compressed; KNOPPIX-DATA is un-compressed;If you boot Knoppix, the (compressed) file '/mnt-system/KNOPPIX/KNOPPIX' is mounted (and decompressed) via '/dev/cloop' to '/KNOPPIX' - readonly.

Now you have
a) '/KNOPPIX-DATA', the persistent memory - read- and writeable
b) '/KNOPPIX', the filesystem image - only readable.

UNIONFS lays this two parts one over the other and you think, you have only one filesystem. Because you can not write to '/KNOPPIX' all your changes are written to '/KNOPPIX-DATA'.

If you install a program, the files of this program are now in '/KNOPPIX-DATA/usr/'; if you purge a program you find a deletion item in '/KNOPPIX-DATA/usr/'. And all of '/KNOPPIX-DATA' is stored via '/dev/loop0' in 'knoppix-data.img'. In 'knoppix-data.img' is nothing of '/KNOPPIX'.

[kl522]

The file 'knoppix-data.img' or in case of encryption 'knoppix-data.aes' is always compressed! Have a look at it with midnightcommander.

If you boot Knoppix, it is mounted (and decompressed) via '/dev/loop0' to '/KNOPPIX-DATA'. If you boot with cheatcode "noimage" nothing happens with '..img' or '..aes'


Sorry to intrude into this discussion. All other things do not interest me, but knoppix-data.img or knoppix-data.aes is always ***UNCOMPRESSED***. I don't use midnightcommander, but if midnightcommander tells you that it is compressed, then throw midnightcommander away ! Knoppix-data.img is just a normal EXT2/3/4 file system, which can mount it yourself using /dev/loop0 and it is read-write. Again, a perfectly usual read/write EXT file system ! There is nothing unusual about knoppix-data.img.

If it is compressed, it is not so easily becoming writable !

[utu]

I am trying to put some distance between making a simple backup and
purging the persistent store. Most of the time, one can make a little backup
and not even think about re-doing persistence.

I'm glad to learn there are other ways to go about re-doing persistence, but
I don't have to do that very often. So here's my take on knoppix-data.img,
but I consider this somewhat a distraction from the thread itself.

To purge a failed knoppix-data.img, it must be un-mounted first, then deleted.

One way to achieve this is to launch another working linux system, such as your
LiveCD and to delete the knoppix-data.img on an inert LiveUSB.

The cheatcode 'noimage' used in launching a LiveUSB must keep its knoppix-data.img
from being mounted, allowing one to delete it by means of the LiveUSB, without
recourse to another linux system for this purpose. I've not used it, but I can
(now) appreciate its usefulness.

In either case, the purged LiveUSB should provide the opportunity to (re)establish
a persistent file at each boot, until such time as one has in fact been
established. However, re-establishing persistence in this way purges ALL
changes inherent in the purged knoppix-data.img, both personal material and
other material not contained in home/, root/, etc/ and syslinux.cfg.

The personal backup will re-instate only the personal material. Other material
must be additionally re-constituted. The original backup program cited earlier
backs up ALL of KNOPPIX-DATA, not just the personal material if that is desired.
The storage overhead for this is considerably more, and to my mind not as efficient
a use of resources as this later 'personal version'.

So long as the LiveUSB seems to be properly functioning, there is no need to
purge-and-replace knoppix-data.img. Personal backups may be performed over and over
without recourse to purging and re-establishing persistence so long as
the OS seems to be performing as it should.

[Werner P. Schulz]

Sorry to intrude into this discussion. All other things do not interest me, but knoppix-data.img or knoppix-data.aes is always ***UNCOMPRESSED***.Thank you for the correction, you are right. (And I thought, nobody else read this thread.)

[Werner P. Schulz]

Personal backups may be performed over and over
without recourse to purging and re-establishing persistence so long as
the OS seems to be performing as it should.Full acknowledge. Therefore my proposal to insert a date-item
update$(date +'%m%d%y').tar.gz

[utu]

Hi, Werner

Not to belabor the point, but if the system sees a file is to be
saved with the same name as one already 'in situ' it asks if you
want to 'write over' the earlier file. I first became aware of
this when it seemed to spoil my status prompts.

Your idea allows making backups unique wrt DAYs. Making several
in a given day I still observe this phenomenon. I might prefer
to swap YEAR for HOUR, retaining the same brevity but allowing
for more changes per DAY of the medium.

[Werner P. Schulz]

... no problem

STOR=update$(date +'%m%d%H').tar.gz
or
STOR=update$(date +'%m%d-%H%M').tar.gz

[utu]

Further refinements to the personal backup idea.
.
I've incorporated all of Werner's excellent suggestions, and since
I found I never referred to the tar listing, that's been omitted.

In examining the remaining contributors to the 'personal backup', I found
that not only is .mozilla the real space hog, but its caches are its
biggest contributors. It is possible to clear at least one of these caches
with a quick trip to the IceWeasel Edit>Preferences GUI before making
a backup, resulting in a great reduction in the size of the stored file.

A revised listing is as follows:


#!/bin/bash
#
# new.backup2: do 'chmod +x new.backup2' & 'chown root:root new.backup2'
# Before backup, Go to Edit>Prefs on IceWeasel & 'Clear Now' the Off-Line cache.
#
cd /; STOR=/mnt-system/update$(date +'%m%d%H').tar.gz
echo -e 'Compressing data; patience, this may take a little time..\c'
tar -czf $STOR KNOPPIX-DATA/home/ KNOPPIX-DATA/etc/ \
KNOPPIX-DATA/root/ mnt-system/boot/syslinux/syslinux.cfg
echo '.Done.'; echo 'Restore using the command, tar -xzf '$STOR' -C /'
#
exit 0 Now if some clever person here will inform me of some appropriate API,
assuming there is one, we might automate that effort as well. This is
to say that I haven't been able to 'exclude' from the tar inclusion
the Cache & OfflineCache directories from ~/.mozilla/firefox/*.default/
which are the culprits I'd like to exclude.

[Werner P. Schulz]

Werner's excellent suggestionsPlease, don't exaggerate :-)

A suggestion with exclude something:

cd /; STOR=/mnt-system/update$(date +'%m%d%H').tar.gz
echo -e 'Compressing data; patience, this may take a little time..\n'
tar -cz --exclude=*/Cache/* --exclude=*/OfflineCache/* \
-f $STOR KNOPPIX-DATA/home/ KNOPPIX-DATA/etc/ \
KNOPPIX-DATA/root/ mnt-system/boot/syslinux/syslinux.cfg

[utu]

.
With that refinement, my backup only takes 46 Mb now.
Thanks a heap & best regards.