Three different kinds of persistent store with Knoppix:Implementation example

[Capricorny]

The default way to run Knoppix on a daily basis is to use the "built-in" persistent store, which is unionfs-mounted together with the compressed KNOPPIX image. To fit on vfat-formatted devices (and standard DVDs), the max size of persistent store has traditionally been 4GB, but there has, actually, been possible to use as big store as the file system allows.
Now, there is also built-in support for larger persistent storage, but I have found the use of such to be a double-edged sword: Not only will the efficiency for my daily use tend to be lower with larger persistent stores, but the upgrade process between Knoppix releases often is much more convoluted when theres is a lot of stuff on the persistent store to take care of.

The efficient solution for me has been
1. Mount a suitable storage partition on the PC for a larger amount of persistent storage, persistent between Knoppix instances.
2. Use virtual machines on the storage partition for "normal" PC use (larger OS-release specific partitions etc).
3. Keep updated copies of the storage partition on external media (NAS, external HDD) for backup and copying.
4. Use a 4GB persistent store, remaster as necessary. (Maybe several times per Knoppix release.)

With the increase in use of Java or similar languages, the remastering process has become less efficient, as Java-based systems to a large extent consists of jar/war packages, which are already compressed. Also, such packages are generally quite platform agnostic, and in contrast with ordinary Debian packages, they don't, generally, need to be updated between Linux OS releases.

Therefore to have a third, movable kind of persistent store, I have returned to a practice from before I started to do remastering routinely: Mounting (not unionfs, just ordinary) extra volumes of persistent store, stored in (loop-mounted) .img files together with knoppix-data.img.
This way, I can avoid most of space problems with ordinary persistent store, while still keeping everything at manageable sizes. Those files are easily copied and moved around together with the basic Knoppix system, and they can be made persistent between releases. I use 32 GB USB3 sticks for backup and copying.

I keep one volume for programs (including program development) and one for data. They are mounted on directories /usr/local/opt and /usr/local/data.

Type of storage                          File                Mounted on     Movable  Release-persistent        Size (ca)  
KNOPPIX compressed                       KNOPPIX             /KNOPPIX            Yes              No                4GB
Standard persistent store                knoppix-data.img    /KNOPPIX-DATA       Yes              No                4GB
PC store  volume                         ..............      /store              No               Yes             450GB
Extra persistent store for programs      knoppix-data2.img   /usr/local/opt      Yes            Possibly            4GB
Extra persistent store for data          knoppix-data3.img   /usr/local/data     Yes              Yes               4GB

The extra volumes are not involved in Knoppix init, they are mounted by /etc/rc.local

Here is an excerpt from df output on a PC with 250GB SSD disk, (8 GB RAM, using 64-bits standard kernel.)

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda1       46057488  31581072  14476416  69% /mnt-system
tmpfs            6463488         0   6463488   0% /ramdisk
/dev/cloop       9646060   9646060         0 100% /KNOPPIX
/dev/loop0       4031680   3402740    424140  89% /KNOPPIX-DATA
unionfs          4031680   3402740    424140  89% /UNIONFS
/dev/loop1       4031680     73596   3753284   2% /usr/local/opt
/dev/loop2       4031680     73596   3753284   2% /usr/local/data
/dev/sda5      141110136 102054728  31887408  77% /store

The necessary modification of /etc/rc.local is straightforward:

function to_exist() {
   [ -d "$1" ] || mkdir -p $1 ;
}

#Knoppix directory names vary with release and remastering version
KNOPPIX_RUN_DIR=KNOPPIX705_2
#Create mount point for PC store -no mounting, actual partitions may vary
mkdir /store
# Mount extra volumes, make sure mount points exist
to_exist /usr/local/opt ;
to_exist /usr/local/data ;
mount -o loop /mnt-system/${KNOPPIX_RUN_DIR}/knoppix-data2.img /usr/local/opt ;
mount -o loop /mnt-system/${KNOPPIX_RUN_DIR}/knoppix-data3.img /usr/local/data ;

I have made a small, simple script for creating the extra volumes. Excerpt:

# Try to avoid stealing loops.. make some outside the ordinary range..
 [ -b /dev/loop15 ]    || sudo mknod -m 755 /dev/loop15 b 7 15
 [ -b /dev/loop14 ]    || sudo mknod -m 755 /dev/loop14 b 7 14
 [ -b /dev/loop13 ]    || sudo mknod -m 755 /dev/loop13 b 7 13

function to_exist() {
   [ -d "$1" ] || sudo mkdir -p $1 ;
}

function purge_or_create() {
    [ -d "$1" ] && sudo rm -rf $1 
    sudo mkdir -p $1 ;
}


function setup_loopfile() {
  loopdev=$1 ; workdir=$2 ; loopfile=$3 ; psize=$4  ;
  to_exist /tmp/knx-mkloopfiles ;
  purge_or_create /tmp/knx-mkloopfiles/setup_mnt ;

  sudo dd if=/dev/zero of=${workdir}/${loopfile} bs=1M count=$psize
  sudo losetup /dev/loop${loopdev} ${workdir}/${loopfile}
  sudo mkfs.ext3 /dev/loop${loopdev} 2>create_workspace.log ; 
  sudo losetup /dev/loop${loopdev}

}

# Pick parameters from command line ;
running_knoppixdir=$1 ; psz=$2 ;

setup_loopfile 15 ${running_knoppixdir} knoppix-data2.img $psz ;
setup_loopfile 14 ${running_knoppixdir} knoppix-data3.img $psz ;

This Knoppix variant is also very well suited for running in virtual machines, for example for program development, testing out services etc.

[utu]

Greetings, Capricorny.

Nice to see someone experimenting with Knoppix.

As an alternative to your cloop arrangement, you might consider
lining all your cloops up alongside your KNOPPIX overlays in /mnt-system/KNOPPIX/.
These would be automatically mounted by Knoppix, without your having
to order your own mounts in /etc/rc.local.

The attached screenshot shows my arrangement for a CD-size LiveUSB.
my Knoppix itself and /mnt-system are on sdb1.
My persistence file is a reiserfs on sdb2.
My KNOPPIX1 and KNOPPIX2 are my occasional cloops representing past KNOPPIX-DATA snapshots.

We can hang six or seven cloops with names KNOPPIX1, KNOPPIX2 ... KNOPPIX7 and they
get mounted automatically by Knoppix.
I suppose one could add some more /dev/cloops if eight total isn't enough.

The KNOPPIX<n> numbering is not entirely arbitrary here. The first part of KNOPPIX
is really always KNOPPIX0; for the CD-size Knoppix, the first additional cloop is cloop1,
and I call that KNOPPIX1. Any additional KNOPPIX-DATA I care to compress has to have a
next higher number.

I'm pretty sure KK uses this same scheme to add other content to his CeBIT and
Linux Magazin offerings.

[Capricorny]

You missed that these extra volumes are NOT cloops - and that I actually don't want to have them automatically mounted. They are not joined into the unionfs mount, as they are, essentially, persistent over Knoppix releases.
Of course KK uses "your" mechanism for temporarily adding stuff to Knoppix releases. I don't need to remove anything, so it's not so useful for me with more overlays. Remastering is, essentially, collapsing overlays.