Results 1 to 8 of 8

Thread: Could cheatcode noimage be the smartest way to handle persistence?

  1. #1
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802

    Could cheatcode noimage be the smartest way to handle persistence?

    I have just started experimenting with Knoppix on SSD drive, and in many ways, it seems like an ideal match. Using a 240GB drive, I can set up a couple of 16 GB partitions for different Knoppix versions, plus a 50GB Win7 partition, and still have space for a /store partition for virtual machines, development work etc.

    Remastering goes very fast this way (ca 5 min for 12GB system copy), but the most important practical benefit was the speed of virtual machines (under VMware workstation 8 ). Suddenly, it became quite fine to surf in a Knoppix VM under Knoppix.

    Regardless of all assurances, there are some concerns about SSD: 25 micron NANDs may have as little as 3000 write cycles/cell. And though smart controllers should be able to even out the write load, we may, for axample, have error situations where they are not able to cope fully. Last year's kernel 2.37-bug, for example, locked up one of the cores in certain configurations in a loop (machine was still usable with the other cores..), and it made my externally mounted SSD run rather hot. As long as it's only reading, it should in principle be fine - but it is possible to thrash it.

    Durability may also be an issue, not all SSD drives are created equal.

    So - in ordinary use, it may be advisable to minimize disk writing also with SSD. (It is of course even more important when running from USB sticks.) One simple scheme is to use knoppix-data.img mostly as a backup-and-restore volume, and use /ramdisk as the ordinary "persistent" store.

    Content can be read into /ramdisk on boot, and (eventually) written back on shutdown, or when needed. It could also be scheduled as a cron job, but that would not by itself ensure full updating.

    One of the reasons I have never thought much about such a solution, is that my persistent store has often been well-filled, 1-3 GB, and that's not very suitable for ramdisk handling. And I try to stick with just one KNOPPIX compressed image for ordinary use. With a SSD and squashfs, remastering is, however, so fast and simple that there is not much sense in having much data in the persistent store for a long time.

    For ordinary backing up etc, it is of course often easier to use something smaller than knoppix-data.img, but switching seamlessly between noimage and persistent use may be important, for example when running same Knoppix on different machines or under different RAM requirements.

    Adding a cheatcode like norestore could direct knoppix.sh to not initialize /ramdisk with the contents of knoppix-data.img on boot, and not write it automatically back on shutdown. If knoppix-data.img is written under norestore, it should probably be completely rewritten, as we could otherwise create inconsistencies.

    With the emergence of cheap and fast USB3 sticks, running Knoppix off USB media has become a useful option, and this method could make it even simpler and safer. And if persistent store is not updated, the potential loss is limited to the actual session. The most important things to keep are of course program installs and updates, and good practice would be to update persistent store after each such modification.

  2. #2
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    Quote Originally Posted by Capricorny View Post
    So - in ordinary use, it may be advisable to minimize disk writing also with SSD. (It is of course even more important when running from USB sticks.) One simple scheme is to use knoppix-data.img mostly as a backup-and-restore volume, and use /ramdisk as the ordinary "persistent" store.
    You are onto a very interesting thread here.

    FWIW, I have used SD cards and SDHCs exclusively and daily for about two years
    in LiveUSBs-with-persistence mode. I've yet to see one fail. Knock on wood.
    2-Gb LiveCD-size SD Cards for about 18 months; 8-Gb LiveDVD-size SDHCs for the last 6 months.

    I don't know how SDHCs compare with SDDs either, of course.

  3. #3
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    SDDs vs flash memory: Actually, SDDs should be in another division entirely wrt durability and reliability. And recent controllers are supposed to do a great job levelling out the write load over cells. Because worst case scenarios are more scary, I think, however, that a defensive approach may be indicated. There are two important reasons, first, the one I mentioned about situations where writing is out of control, second the occurence of short-lived copies wearing out much faster than advertised. Linux does a very good job in minimizing accesses by using RAM as cache.

    But sheer performance is also a factor here: Ramdisk is, AFAIK, always faster than persistent image. It's also OK to be able to choose not to save a session. Right now, ca 230MB is used, on the ramdisk, much of it from doing an aptitude update.

    Here is a small and simplistic script that could be placed, for example, in /usr/local/bin and called from knoppix.sh or /etc/rc.local, or command line. For general use, some more checks are needed, but it's a starting point.

    Code:
    #!/bin/bash
    #   update_image.sh - Script for restoring ramdisk content 
    #    from or saving to persistent image  20120701 tay
    #
    # Create  loop15 if necessary 
    [ -b /dev/loop15 ] || sudo mknod -m 755 /dev/loop15 b 7 15
    
    task=$1 ; shift ;
    
    case "${task}" in
      "restore")
           echo -e 'Restore to ramdisk..' ;
           [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
           mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
           rsync -ax /tmp/mntimg0/* /ramdisk ;
           umount /tmp/mntimg0 ;
           echo -e 'Restore to ramdisk done' ;
          ;;
    
      "save")
            echo -e 'Save to persistent store...' ;
          [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
           mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
           rsync -axu  /ramdisk/* /tmp/mntimg0 ;
           umount /tmp/mntimg0 ;
           echo -e 'Save to persistent store done' ;
       ;;
    
      *)
         echo -e "Ooops: $task???" ;
         ;;
    
    esac
    
    #
    #
    The logic could be that it is called automatically when noimage but not norestore is present in /proc/cmdline. It could be called manually anytime.

  4. #4
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    As a side issue, I've always wanted a command to NOT automatically update my persistent
    image on shutdown when I know that doing so will irreparably corrupt that image.
    For example, having 'run out of memory on the device' comes to mind.

  5. #5
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Quote Originally Posted by utu View Post
    As a side issue, I've always wanted a command to NOT automatically update my persistent
    image on shutdown when I know that doing so will irreparably corrupt that image.
    For example, having 'run out of memory on the device' comes to mind.

    I have thought a little bit more about this, and I think noimage should keep its current meaning: No link to persistent store. Then two cheatcodes could be added for a linking: update for restore on boot and save on shutdown, and noupdate for restore without final update. So that you could get what you want with the two cheatcodes noimage noupdate.

    If you want to save without prior restore, you are effectively rewriting persistent store, so that should be done manually.

    To sum up, noimage update would give "ramdisk-as-persistent-store" action, noimage noupdate would init ramdisk from persistent store, but not save automatically, and noimage alone would start with blank ramdisk, as it does now. In a table:
    Code:
                               restore
                             Y          N
    save        Y      update   --manually--
                N      noupdate   noimage
    The script could be expanded with more cases/parameters, for example for creating a cloop/squashfs overlay.

  6. #6
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802

    Updated script with cloop/squashfs overlay generation

    I haven't tested the generated overlays, but it shouldn't be far off from something usable

    For creating overlays, like KNOPPIX1, KNOPPIX2.sq etc, the overlay number must be given as a parameter. The created overlays are just dumped in /tmp, but that's the way Klaus K seems to do it.

    Code:
    #!/bin/bash
    #   update_image.sh - Script for restoring ramdisk content 
    #    from or saving to persistent image  20120701 tay
    #
    # Create  loop15 if necessary 
    [ -b /dev/loop15 ] || sudo mknod -m 755 /dev/loop15 b 7 15
    
    task=$1 ; shift ;
    
    case "${task}" in
        "restore")
        echo -e 'Restore to ramdisk..' ;
        [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
        mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
        rsync -ax /tmp/mntimg0/* /ramdisk ;
        umount /tmp/mntimg0 ;
        echo -e 'Restore to ramdisk done' ;
        ;;
    
        "save")
            echo -e 'Save to persistent store...' ;
        [ -d /tmp/mntimg0 ] || mkdir /tmp/mntimg0 ;
        mount -t ext3 -rw -o loop=/dev/loop15 /mnt-system/KNOPPIX/knoppix-data.img /tmp/mntimg0 ;   
        rsync -axu  /ramdisk/* /tmp/mntimg0 ;
        umount /tmp/mntimg0 ;
        echo -e 'Save to persistent store done' ;
        ;;
    
        "cloop")
        overlay_no=$1 ; shift ;
        sudo mkisofs -R -U -m 'root/*' -m 'home/*' -m 'lost+found/*' -m '.wh*'  /ramdisk | create_compressed_fs -B 131072 -m - - > /tmp/KNOPPIX${overlay_no} 2>~/make_comprisofs.log
        ;; 
    
        "squashfs")
        overlay_no=$1 ; shift ;
        sudo mksquashfs /ramdisk /tmp/KNOPPIX${overlay_no}.sq -e root home lost+found .wh..wh.aufs .wh..wh.orph -b 262144 -noappend 2>~/make_squashfs.log 
        ;; 
    
        *)
        echo -e "Ooops: $task???" ;
        ;;
    
    esac

  7. #7
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    In my Post #17 on the following thread in a little program I call 'smite' I develop an update image name 'UPDIMG' which figures out what to call the next image to be stacked, based on what's gone on before. Maybe something like that would be useful to you.

    http://knoppix.net/forum/threads/298...083#post127083

  8. #8
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Quote Originally Posted by utu View Post
    In my Post #17 on the following thread in a little program I call 'smite' I develop an update image name 'UPDIMG' which figures out what to call the next image to be stacked, based on what's gone on before. Maybe something like that would be useful to you.

    http://knoppix.net/forum/threads/298...083#post127083
    It is useful yes, though I don't think of this overlay business as necessarily sequential right now. Therefore I would prefer that the overlay number is given as an explicit parameter. Why not use KNOPPIX9 for something really far out, for example? They could all be made independent, making sequence number rather irrelevant.

    There are a few alternatives for calling the scripts. On boot, /etc/init.d/knoppix-autoconfig, /etc/rc.local or /mnt-system/KNOPPIX/knoppix.sh can be used. On shutdown, /etc/init.d/knoppix-halt is at least one good candidate.

    They could be patched with something like this. (Again, I haven't checked this out bash/busybox syntax is a bitch )

    Code:
    # On boot  e.g. /etc/init.d/knoppix-autoconfig
    if grep -s -w -i "noimage" /proc/cmdline ; then 
         if grep -s -w -i "update" /proc/cmdline || grep -s -w -i "noupdate" /proc/cmdline ; then
             update_image.sh restore
         fi
    fi  
    #
    # On shutdown  e.g.  /etc/init.d/knoppix-halt
     if grep -s -w -i "noimage" /proc/cmdline ; then 
         if grep -s -w -i "update" /proc/cmdline  ; then
            update_image.sh save
         fi
    fi 
    #
    #
    Then this would be implemented without any real internal surgery, like patching minirt init. The only places the new cheatcodes are referred to, are in these scripts.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Intel Xeon E5-2689v4 10 Core 3.1G - SR2T7 picture

Intel Xeon E5-2689v4 10 Core 3.1G - SR2T7

$125.00



Intel - Core i9-12900K Desktop Processor 16 (8P+8E) Cores up to 5.2 GHz Unloc... picture

Intel - Core i9-12900K Desktop Processor 16 (8P+8E) Cores up to 5.2 GHz Unloc...

$331.99



Intel - Core i7-12700K Desktop Processor 12 (8P+4E) Cores up to 5.0 GHz Unloc... picture

Intel - Core i7-12700K Desktop Processor 12 (8P+4E) Cores up to 5.0 GHz Unloc...

$242.99



Intel - Core i9-14900K 14th Gen 24-Core 32-Thread - 4.4GHz (6.0GHz Turbo) Soc... picture

Intel - Core i9-14900K 14th Gen 24-Core 32-Thread - 4.4GHz (6.0GHz Turbo) Soc...

$539.99



Intel 6 Core i5-8600 3.1GHZ Desktop Processor SR3X0 picture

Intel 6 Core i5-8600 3.1GHZ Desktop Processor SR3X0

$50.00



Intel - Core i7-14700K 14th Gen 20-Core 28-Thread - 4.3GHz (5.6GHz Turbo) Soc... picture

Intel - Core i7-14700K 14th Gen 20-Core 28-Thread - 4.3GHz (5.6GHz Turbo) Soc...

$399.99



Intel - Core i7-13700K 13th Gen 16 cores 8 P-cores + 8 E-cores 30M Cache, 3.4... picture

Intel - Core i7-13700K 13th Gen 16 cores 8 P-cores + 8 E-cores 30M Cache, 3.4...

$364.99



Intel - Core i9-13900K 13th Gen 24 cores 8 P-cores + 16 E-cores 36M Cache, 3 ... picture

Intel - Core i9-13900K 13th Gen 24 cores 8 P-cores + 16 E-cores 36M Cache, 3 ...

$519.99



AMD EPYC 7282 cpu processor 16 cores 32 threads 2.8GHZ up to 3.2GHZ 120w picture

AMD EPYC 7282 cpu processor 16 cores 32 threads 2.8GHZ up to 3.2GHZ 120w

$78.00



AMD EPYC 7F52 CPU processor 16 cores 32 threads 3.5GHZ up to 3.9GHZ 240w picture

AMD EPYC 7F52 CPU processor 16 cores 32 threads 3.5GHZ up to 3.9GHZ 240w

$299.00