Could cheatcode noimage be the smartest way to handle persistence?

[Capricorny]

I have just started experimenting with Knoppix on SSD drive, and in many ways, it seems like an ideal match. Using a 240GB drive, I can set up a couple of 16 GB partitions for different Knoppix versions, plus a 50GB Win7 partition, and still have space for a /store partition for virtual machines, development work etc.

Remastering goes very fast this way (ca 5 min for 12GB system copy), but the most important practical benefit was the speed of virtual machines (under VMware workstation 8 ). Suddenly, it became quite fine to surf in a Knoppix VM under Knoppix.

Regardless of all assurances, there are some concerns about SSD: 25 micron NANDs may have as little as 3000 write cycles/cell. And though smart controllers should be able to even out the write load, we may, for axample, have error situations where they are not able to cope fully. Last year's kernel 2.37-bug, for example, locked up one of the cores in certain configurations in a loop (machine was still usable with the other cores..), and it made my externally mounted SSD run rather hot. As long as it's only reading, it should in principle be fine - but it is possible to thrash it.

Durability may also be an issue, not all SSD drives are created equal.

So - in ordinary use, it may be advisable to minimize disk writing also with SSD. (It is of course even more important when running from USB sticks.) One simple scheme is to use knoppix-data.img mostly as a backup-and-restore volume, and use /ramdisk as the ordinary "persistent" store.

Content can be read into /ramdisk on boot, and (eventually) written back on shutdown, or when needed. It could also be scheduled as a cron job, but that would not by itself ensure full updating.

One of the reasons I have never thought much about such a solution, is that my persistent store has often been well-filled, 1-3 GB, and that's not very suitable for ramdisk handling. And I try to stick with just one KNOPPIX compressed image for ordinary use. With a SSD and squashfs, remastering is, however, so fast and simple that there is not much sense in having much data in the persistent store for a long time.

For ordinary backing up etc, it is of course often easier to use something smaller than knoppix-data.img, but switching seamlessly between noimage and persistent use may be important, for example when running same Knoppix on different machines or under different RAM requirements.

Adding a cheatcode like norestore could direct knoppix.sh to not initialize /ramdisk with the contents of knoppix-data.img on boot, and not write it automatically back on shutdown. If knoppix-data.img is written under norestore, it should probably be completely rewritten, as we could otherwise create inconsistencies.

With the emergence of cheap and fast USB3 sticks, running Knoppix off USB media has become a useful option, and this method could make it even simpler and safer. And if persistent store is not updated, the potential loss is limited to the actual session. The most important things to keep are of course program installs and updates, and good practice would be to update persistent store after each such modification.

[utu]

So - in ordinary use, it may be advisable to minimize disk writing also with SSD. (It is of course even more important when running from USB sticks.) One simple scheme is to use knoppix-data.img mostly as a backup-and-restore volume, and use /ramdisk as the ordinary "persistent" store.

You are onto a very interesting thread here.

FWIW, I have used SD cards and SDHCs exclusively and daily for about two years
in LiveUSBs-with-persistence mode. I've yet to see one fail. Knock on wood.
2-Gb LiveCD-size SD Cards for about 18 months; 8-Gb LiveDVD-size SDHCs for the last 6 months.

I don't know how SDHCs compare with SDDs either, of course.

[Capricorny]

SDDs vs flash memory: Actually, SDDs should be in another division entirely wrt durability and reliability. And recent controllers are supposed to do a great job levelling out the write load over cells. Because worst case scenarios are more scary, I think, however, that a defensive approach may be indicated. There are two important reasons, first, the one I mentioned about situations where writing is out of control, second the occurence of short-lived copies wearing out much faster than advertised. Linux does a very good job in minimizing accesses by using RAM as cache.

But sheer performance is also a factor here: Ramdisk is, AFAIK, always faster than persistent image. It's also OK to be able to choose not to save a session. Right now, ca 230MB is used, on the ramdisk, much of it from doing an aptitude update.

Here is a small and simplistic script that could be placed, for example, in /usr/local/bin and called from knoppix.sh or /etc/rc.local, or command line. For general use, some more checks are needed, but it's a starting point.

#!/bin/bash
#   update_image.sh - Script for restoring ramdisk content 
#    from or saving to persistent image  20120701 tay
#
# Create  loop15 if necessary 
[ -b /dev/loop15 ] || sudo mknod -m 755 /dev/loop15 b 7 15

task=$1 ; shift ;

case "${task}" in
  "restore")
       echo -e 'Restore to ramdisk..' ;
       [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
       mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
       rsync -ax /tmp/mntimg0/* /ramdisk ;
       umount /tmp/mntimg0 ;
       echo -e 'Restore to ramdisk done' ;
      ;;

  "save")
        echo -e 'Save to persistent store...' ;
      [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
       mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
       rsync -axu  /ramdisk/* /tmp/mntimg0 ;
       umount /tmp/mntimg0 ;
       echo -e 'Save to persistent store done' ;
   ;;

  *)
     echo -e "Ooops: $task???" ;
     ;;

esac

#
#

The logic could be that it is called automatically when noimage but not norestore is present in /proc/cmdline. It could be called manually anytime.

[utu]

As a side issue, I've always wanted a command to NOT automatically update my persistent
image on shutdown when I know that doing so will irreparably corrupt that image.
For example, having 'run out of memory on the device' comes to mind.

[Capricorny]

As a side issue, I've always wanted a command to NOT automatically update my persistent
image on shutdown when I know that doing so will irreparably corrupt that image.
For example, having 'run out of memory on the device' comes to mind.

I have thought a little bit more about this, and I think noimage should keep its current meaning: No link to persistent store. Then two cheatcodes could be added for a linking: update for restore on boot and save on shutdown, and noupdate for restore without final update. So that you could get what you want with the two cheatcodes noimage noupdate.

If you want to save without prior restore, you are effectively rewriting persistent store, so that should be done manually.

To sum up, noimage update would give "ramdisk-as-persistent-store" action, noimage noupdate would init ramdisk from persistent store, but not save automatically, and noimage alone would start with blank ramdisk, as it does now. In a table:

                           restore
                         Y          N
save        Y      update   --manually--
            N      noupdate   noimage

The script could be expanded with more cases/parameters, for example for creating a cloop/squashfs overlay.

[Capricorny]

I haven't tested the generated overlays, but it shouldn't be far off from something usable

For creating overlays, like KNOPPIX1, KNOPPIX2.sq etc, the overlay number must be given as a parameter. The created overlays are just dumped in /tmp, but that's the way Klaus K seems to do it.

#!/bin/bash
#   update_image.sh - Script for restoring ramdisk content 
#    from or saving to persistent image  20120701 tay
#
# Create  loop15 if necessary 
[ -b /dev/loop15 ] || sudo mknod -m 755 /dev/loop15 b 7 15

task=$1 ; shift ;

case "${task}" in
    "restore")
    echo -e 'Restore to ramdisk..' ;
    [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
    mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
    rsync -ax /tmp/mntimg0/* /ramdisk ;
    umount /tmp/mntimg0 ;
    echo -e 'Restore to ramdisk done' ;
    ;;

    "save")
        echo -e 'Save to persistent store...' ;
    [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
    mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
    rsync -axu  /ramdisk/* /tmp/mntimg0 ;
    umount /tmp/mntimg0 ;
    echo -e 'Save to persistent store done' ;
    ;;

    "cloop")
    overlay_no=$1 ; shift ;
    sudo mkisofs -R -U -m 'root/*' -m 'home/*' -m 'lost+found/*' -m '.wh*'  /ramdisk | create_compressed_fs -B 131072 -m - - > /tmp/KNOPPIX${overlay_no} 2>~/make_comprisofs.log
    ;; 

    "squashfs")
    overlay_no=$1 ; shift ;
    sudo mksquashfs /ramdisk /tmp/KNOPPIX${overlay_no}.sq -e root home lost+found .wh..wh.aufs .wh..wh.orph -b 262144 -noappend 2>~/make_squashfs.log 
    ;; 

    *)
    echo -e "Ooops: $task???" ;
    ;;

esac

[utu]

In my Post #17 on the following thread in a little program I call 'smite' I develop an update image name 'UPDIMG' which figures out what to call the next image to be stacked, based on what's gone on before. Maybe something like that would be useful to you.

http://knoppix.net/forum/threads/298...083#post127083

[Capricorny]

In my Post #17 on the following thread in a little program I call 'smite' I develop an update image name 'UPDIMG' which figures out what to call the next image to be stacked, based on what's gone on before. Maybe something like that would be useful to you.

http://knoppix.net/forum/threads/298...083#post127083

It is useful yes, though I don't think of this overlay business as necessarily sequential right now. Therefore I would prefer that the overlay number is given as an explicit parameter. Why not use KNOPPIX9 for something really far out, for example? They could all be made independent, making sequence number rather irrelevant.

There are a few alternatives for calling the scripts. On boot, /etc/init.d/knoppix-autoconfig, /etc/rc.local or /mnt-system/KNOPPIX/knoppix.sh can be used. On shutdown, /etc/init.d/knoppix-halt is at least one good candidate.

They could be patched with something like this. (Again, I haven't checked this out bash/busybox syntax is a bitch )

# On boot  e.g. /etc/init.d/knoppix-autoconfig
if grep -s -w -i "noimage" /proc/cmdline ; then 
     if grep -s -w -i "update" /proc/cmdline || grep -s -w -i "noupdate" /proc/cmdline ; then
         update_image.sh restore
     fi
fi  
#
# On shutdown  e.g.  /etc/init.d/knoppix-halt
 if grep -s -w -i "noimage" /proc/cmdline ; then 
     if grep -s -w -i "update" /proc/cmdline  ; then
        update_image.sh save
     fi
fi 
#
#

Then this would be implemented without any real internal surgery, like patching minirt init. The only places the new cheatcodes are referred to, are in these scripts.