Using a Read-Only Persistence Capability in Knoppix LiveUSBs

[utu]

.
There is an intriguing comment by Klaus Knopper regarding an otherwise
little-advertised capability of Knoppix to bring in additional cloop-
compressed overlays and to combine their contents with that of the master
KNOPPIX compressed file at boot. Klaus uses this feature to add non-free
material, which CeBIT is licensed to distribute, to CeBIT versions
of KNOPPIX liveCDs, LiveDVDs and their isos. This may be found at:
http://lists.debian.org/debian-knoppix/2012/06/msg00005.html

AFAICT, Knoppix can handle, at present, eight overlays,
KNOPPIX0, KNOPPIX1,...KNOPPIX7. The LiveCD iso has only KNOPPIX, aka
KNOPPIX0. That leaves, for a LiveCD-sized LiveUSB, KNOPPIX1, KNOPPIX2,
...KNOPPIX7 unassigned and available.

For rather-permanent, unchanging parts of one's persistence, there are
at least three reasons one might want to utilize some of these unassigned
overlay positions, assuming one uses a LiveUSB with persistence.
1. A read-only version of the persistence material is safeguarded
against inadvertent write-over;
2 A read-only version may be compressed to 40% its ultimate size after
boot, with the same scant time penalty with which the master KNOPPIX files
are brought in; and
3. A read-only version will decrease the amount of writes associated
with its contained material over that which would occur if the material
were both read/write and uncompressed. There is a wear consideration
peculiar to USB write cycling.

Prior to Knoppix 7, I had added and/or changed a number of files and
programs beyond the standard files which come with the standard LiveCD iso.
With the advent of Knoppix 7.0, I converted these to a cloop-compressed
KNOPPIX1 and added that alongside the master KNOPPIX file in
/mnt-system/KNOPPIX/. I have developed a few additonal such additions
using Knoppix 7 since it first came out. I've recently converted these
additional changes to KNOPPIX2. I am now looking forward to further
adapting these efforts to an upcoming Knoppix 7.1. Two Screenshots, mtab
and df-h illustrate this; see Attached Images below.

I used a small bash program to facilitate and position the cloop-compressed-
read-only KNOPPIX1 and KNOPPIX2. The essential part of this program uses
Klaus K's magic formula, in essence, and for example:


cd /;
sudo mkisofs -R -U KNOPPIX-DATA | create_compressed_fs -B 131072 -m - - > /mnt-system/KNOPPIX/KNOPPIX1
One may account for the content of Knoppix changes in two groups:
1. The totality of files in /home/knoppix/ plus a small number of
individual files that may have been changed or added.
These changes may be recorded with a small bash script, like Keep*.
These changes may be restored with another small bash script, like Restore*.
2. The totality of files added via apt or Synaptic, may similarly be
expressed by another small bash script, like Uncommon.dpkg*.

*These or similar small programs, may be added to /etc/profile as follows:


Keep() { # Aid to backing-up unique user files
cd /; KEEP=/mnt-system/keep$(date +"%m%d%H").tar.gz
echo -e 'Compressing data; patience, this may take a little time..\c'
tar -cz --exclude *gvfs* --exclude *cache* \
-f $KEEP home/knoppix/ \
etc/profile etc/rc.local etc/X11/Xsession.d/45* etc/syslog-knoppix.conf \
etc/chromium/ mnt-system/boot/syslinux/syslinux.cfg
echo ".Done."; echo "Restore using the command, tar -xzf '$KEEP' -C /"
}
Restore() { # Restore $PEEK given the date, $1 in %m%d%H format
PEEK="/mnt-system/keep$1.tar.gz"
echo "Restoring '$PEEK files to /."; tar -xzf $PEEK -C /
}
Uncommon.dpkg() { # Packages uncommon to dpkg.current & "dpkg.$1"
dpkg --get-selections > /tmp/dpkg.current
comm -3 --nocheck-order /tmp/dpkg.current "dpkg.$1" | less | tee uncommon
}

[Blacksimon]

I think I will use this method with the upcoming knoppix 7.1. Until now I had not really considered this possibility.
Safe and compress overlay's size sound good.

[Blacksimon]

I used this to prevent errors:


cd /
sudo mkisofs --exclude *gvfs* --exclude *cache* -R -U KNOPPIX-DATA | create_compressed_fs -B 131072 -m - - > /mnt-system/KNOPPIX/KNOPPIX1

[utu]

Greetings, Blacksimon.

Excluding .gvfs may be a good idea; it may louse up the command.
This annoying file may or may not be hanging around to cause trouble.
I suspect it got into my distro when I added a program which uses
some gnome material. I complained to Klaus K, but he was not
sympathetic with calling it a bug. Its a feature, apparently.
An i/o re-direct command might be a more professional way of
handling the way that .gvfs trips up a command. Maybe some Bash
expert will suggest something appropriate here.

FWIW, I think .gvfs trips things up using a console command or as
part of an /etc/profile function, but as part of an executable shell
script, I don't think it gives (me) the same trouble. I can't explain
why this should be the case.

The only reason to exclude [Cc]ache files is to save the space.
I don't think they'd cause trouble unless you are a real miser on
space, like myself.

[utu]

.
I get into trouble with .gvfs while in /home/knoppix.
I usually execute my bash script from a root lxterminal.
My theory is that /home/knoppix is polluted with .gvfs, while / is not.
For example, try ls -a | grep .gvfs as root in both places.

[Werner P. Schulz]

Excluding .gvfs may be a good idea; it may louse up the command.
This annoying file may or may not be hanging around to cause trouble.'~/.gvfs' isn't a ordinary file; it is the mountpoint for GVFS. Therefore you cannot delete or copy it.

http://en.wikipedia.org/wiki/GVFS

[utu]

Greetings, Werner, nice to hear from you.

I've been reading about this (.gvfs) some.

It turns out, as you say, that .gvfs is a mount point put there by some program,
possibly LibreOffice, which I use. In my case the .gvfs, as a file, never has any contents;
it does show up as in an mtab entry related to fuse.

One post I read suggests unmount and delete and see what happens.
The notion being is that if some program needs to mount something it will re-
establish the mount point. I'm trying this route at present.

FWIW, Ubuntu and others are re-routing this mount point to what we would call /run/knoppix/.gvfs,
all with no more explanation than for the earlier home directory location.
Users have been perplexed by .gvfs for as far back as 2008 I've noticed.

[utu]

.
It turns out that .gvfs is re-established in /home/knoppix within
a few minutes or hours of normal operation using at most: pcmanfm,
lxterminal, libreoffice, firefox 19, leafpad and this forum's editor.

From this I conclude that removing .gvfs probably causes no harm, but also
solves no problem. Using the exclude option for scripts in /home/knoppix
will prevent triggering error messages.

The exclude option may not be necessary for scripts making overlays from
/KNOPPIX-DATA, if .gvfs is not in this part Knoppix. One may assess the
situation, for all of Knoppix with the commands: updatedb; locate .gvfs

[Blacksimon]

Greetings, Utu.
in my case:
options --exclude *gvfs* prevents the error messages to complete KNOPPIX1 creation options --exclude *cache* is to reduce size of KNOPPIX1
I had already set up by default to move cromium cache, ~/.thumbnails, proxy cache to ramdisk.
option -iso-level 4 is to prevent file/directory name truncation as you can see at the beginning of the process of mkisofs.


cd /
sudo mkisofs --exclude *gvfs* --exclude *cache* -iso-level 4 -R -U KNOPPIX-DATA | create_compressed_fs -B 131072 -m - - > /mnt-system/KNOPPIX/KNOPPIX1

I'm testing KNOPPIX1 without overlay at the moment and everything goes well.
The size of KNOPPIX1 is only 720MB instead of 3GB overlay (2GB really used)
With KNOPPIX1 ramdisk is used by default by knoppix

[utu]

option -iso-level 4 is to prevent file/directory name truncation

Interesting.
I haven't noticed this as a problem, but I may have just been lucky.

FWIW, I often write things back to /tmp first just to see how things are turning out.

[Werner P. Schulz]

Caution! The size of '/tmp' is limited to 2 GB. I have had trouble with my script to remaster Knoppix. When I did CD remaster, all went well. But it takes a while to find the error when I did DVD remaster using '/tmp'-

[utu]

Thanks for the warning, Werner.

I was expecting the limit was 4 Gb, but I seem to remember there IS some
uncertainty as to whether 2 Gb or 4 Gb is the limit. You've apparently
'been there, done that'. If I understand you correctly, if I have a
KNOPPIXn that is compressed to 2 Gb or more it will cause trouble
expanding it on /tmp. Is that any different than expanding it to /?
I'm assuming 2.5:1 compression here.

I would have expected I might be able to
conjure up a KNOPPIX-DATA of about 10 Gb and store that as 4 Gb cloop
compressed on /mnt-system and sometime or other expand it again to
10 Gb on /. All this assuming no file in KNOPPIX-DATA ever exceeds 4 Gb.

Since my overlays are always incremental, small additions, I don't think
this is a problem if I am limited to 5 Gb in reiser terms and compressed
to 2 Gb on Fat32 on /mnt-system, but I can see this would be a problem
for re-mastering.

My understanding is that the limit is not a Fat32 limit per se, but
rather a limit of iso9660 filesize ON fat32 filesystems.

[Werner P. Schulz]

"df -h" shows the size of tmpfs mounted at '/tmp' is 2 GB. I've had the trouble while building new ISO of DVD at the point

genisoimage ... | create_compressed_fs .... -f /tmp ...The uncompressed size of Knoppix CD is about 1.9 GB and compressed about 0.8 GB and it fits within '/tmp'. The uncompressed size of DVD is about 9 GB and compressed it exceeds for me the size of 4 GB and that didn't work. Now I use the option "-f /opt/knx/knoppix_tmp".

The other problem while remaster Knoppix DVD: If I don't purge many big packages within DVD, I also cannot remaster a DVD, because the size limit for the new '/KNOPPIX/KNOPPIX' is 4 GB.

[utu]

If I don't purge many big packages within DVD, I also cannot remaster a DVD, because the size limit for the new '/KNOPPIX/KNOPPIX' is 4 GB.
Can't you break up the re-master into 4 Gb chunks?
Isn't that how the Knoppix DVD is constructed?
Doesn't it have KNOPPIX and KNOPPIX1 in /mnt-system/KNOPPIX/ ?
And can't you add KNOPPIX2, KNOPPIX3, etc ?

Also thanks for the clarification. I thought the KNOPPIX, KNOPPIX1, etc were
limited to 4 Gb not 2 Gb, and didn't know where the 2 Gb limit came in.
For my purposes, I don't need to play games with anything larger than 2 Gb on /tmp.
I only use that to test out the syntax of some of my small backup programs.

[utu]

I had already set up by default to move cromium cache, ~/.thumbnails, proxy cache to ramdisk.

This also interesting.
I had been establishing or re-establishing a reiserfs r/w persistence.
An alternative might be to use the ramdisk as a temporary r/w (in ram) for
some functions like *[Cc]ache* files.
I assume if I use the cheatcode noimage and forget my reiserfs that sets this up.
Not sure how to go about the re-routing of caches, however.

[Blacksimon]

To move cromium cache in ramdisk you have to use

/usr/bin/chromium %U --disk-cache-dir="/ramdisk"
right click on Menu/Internet/Browser web Chromium to change property or modify the Exec line in /home/knoppix/.local/share/application/Browser web Chromium.desktop file

To move ~/.thumbnails in ramdisk you have to add in /etc/init.d/knoppix-autoconfig at line 739 (function start_fs) this code

[ -d /home/knoppix/.thumbnails ] || mkdir /home/knoppix/.thumbnails
mount -t tmpfs -o rw,size=200M,suid,dev tmpfs /home/knoppix/.thumbnails

I think you can use even other directory

[utu]

To move cromium cache in ramdisk you have to use...
The chromium cache is already in ramdisk, operating without persistence. See


updatedb; locate *[Cc]ache* | grep chromiumafter booting with noimage and no prior re-arrangement of the cache.
This cache's current contents are probably lost on re-boot of course.

[Blacksimon]

sure with noimage code and KNOPPIX1 always is in ramdisk
post #16 was intended for normal r/w persistence which motivated the exclusion of cache in mkisofs in post #9

[utu]

@Blacksimon
My concern is that you _may_ have allotted chromium cache space in two places in ramdisk.
If your intention is to end up with one chromium cache, I don't think you need to go through
the extra effort you have provided. I think the cache just moves from r/w to ramdisk with no extra effort
on your part. The exclude is still useful to minimize the what's in KNOPPIX1.

[Capricorny]

'~/.gvfs' isn't a ordinary file; it is the mountpoint for GVFS. Therefore you cannot delete or copy it.

http://en.wikipedia.org/wiki/GVFS
All my self-created mount points can be deleted when not in use. And nothing seems to be mounted on my .gvfs, still there is no way to modify it using the ordinary file system commands. So .gvfs is something else, to me seemingly introducing new system features in userspace.

Maybe no wonder Miguel de Icaza is now a dedicated Mac user...? ;-)

[utu]

Greetings, Capricorny.

AFAICT, PCMan needs fuse to do something with policykit in creating LXDE,
so gvfs comes in with PCManFM. You can unmount and delete .gvfs, but it
will re-appear after a while. It is annoying, but appeals to both PCMan and
Klaus K will have been of no avail up to now.

Also, FYI, Ubuntu has re-located this annoyance. See:
http://askubuntu.com/questions/211414/where-did-gvfs-go-on-ubuntu-12-10

[Capricorny]

Greetings, Capricorny.

AFAICT, PCMan needs fuse to do something with policykit in creating LXDE,
so gvfs comes in with PCManFM. You can unmount and delete .gvfs, but it
will re-appear after a while. It is annoying, but appeals to both PCMan and
Klaus K will have been of no avail up to now.

Also, FYI, Ubuntu has re-located this annoyance. See:
http://askubuntu.com/questions/211414/where-did-gvfs-go-on-ubuntu-12-10


Of course, PCMan doesn't want to break Gnome, which, AFAIK is used several places in LXDE. And Klaus K wants to stay as compatible with the major desktops/APIs as possible. So they should not be expected to be positive. The trouble is with the GIO implementation, I think - KDE's version, KIO, does much of the same, but AFAIK without creating such problems.

[utu]

The trouble is with the GIO implementation, I think - KDE's version, KIO, does much of the same, but AFAIK without creating such problems.
I think you have got it right, but I can only guess that a kde approach
must have some downside like requiring more infrastructure to be brought in
to enable fuse that outweighs the known gnome .gvfs annoyance.
See: https://bbs.archlinux.org/viewtopic.php?id=92957

[Capricorny]

I think you have got it right, but I can only guess that a kde approach
must have some downside like requiring more infrastructure to be brought in to enable fuse that outweighs the known gnome .gvfs annoyance.
......


Maybe, but I'd guess it is more of a cultural thing. Real programmers don't use C++, you know.
FUSE = FileSystem in User Space is actually entirely independent of GNOME, gvfs is just kind of implementation that some think should long have been forked..