Page 1 of 4 123 ... LastLast
Results 1 to 10 of 41

Thread: Re-mastering Proof-of-Principle Effort

Hybrid View

  1. #1
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631

    Re-mastering Proof-of-Principle Effort

    .
    Re-mastering will require many gigabytes to accomplish.
    Interestingly, ntsf may offer some unique capabilites for re-mastering.
    I chose to use an external ntsf-formatted hard drive for this purpose.
    Mine is a one-terrabyte-USB-2, which matches the USB capability of my laptop.
    This allows me not to hazard any interference with the Windows 7
    capability of the laptop, even though its gigabytes were tempting.

    I've added ntfs-3g via apt to my Knoppix 6.7 LiveUSB.
    I mount my external ntfs drive 'by hand' with the command
    sudo mount -t ntfs-3g /dev/sdc1 /media/sdc1 when it's on sdc1.
    /etc/fstab should be modified to do this automatically, but
    I've not discovered how to do that just yet.

    Knoppix forum contributors Forester and Capricorny have provided us with
    a programming framework on which further development may proceed.
    Capricorny has provided us with a working program script which has the
    following functional sequence of operations:

    Preparing the workspace
    Copying the current Live system
    Combining the Compressed and un-compressed Elements
    and Preparing an isofs representation of their re-combination
    Compressing the re-combination
    Re-establishing an un-compressed persistence file
    Purging or cleaning-up the workspace

    I would like to add the following functions as well:
    Re-establish the group of programs which define
    both LiveCD and LiveUSB end products and
    Provide both production and and testing menu choices as
    part of these end products

    Current status Aug 18, 2011:

    The current executable script is only about 4700 bytes.
    I envision the final script may be less than 10,000 bytes.
    If so, it is quite likely this may be made to fit on a CD version of Knoppix,
    said version having somewhat the same generous selection of other programs
    to which we have become accustomed.

    I think this forum may, as a group, refine this initial script successively
    until it constitutes, at a minimum, a proof of principle adequate for Knoppix's
    author to endorse and consider for incorporating into a later version of
    Knoppix. Likely, it may also provide useful programming that Klaus may
    co-opt, as is, into his final product.

    I have critiqued the original Forester/Capricorny script to have some rough edges,
    in my estimation, along the following lines:

    1. I initially was concerned that half the time in re-mastering
    following the initial script was spent in preparing the workspace. Luckily,
    kl522 proposed using a sparse-file approach to minimize this effort.
    There may yet be pitfalls here, but the time saving is incredible.

    2. The rsynch and subsequent print-out is excessive and should be trimmed
    down to whatever might serve some useful purposes. I was able to quiet-down
    the rsync effort but not the compress_isofs phase. The latter needs work.

    3. The current choices of what and how to keep are way to restrictive and
    too unclear for my uses. I suggest a redo that is, first of all, more
    transparent. Secondly, let's make sure we can allocate what's-in and what's-
    not-in the compressed file.

    4. I suggest we leave it to Knoppix-itself to replace the persistence file,
    and not do its work for it. Unless of course, the sparse-file approach changes
    the name of the game.

    5. In my case /dev/loop7 isn't found and deleted as it should be in the
    purge process.

    6. I suspect that the remastered product may often be larger than where
    it started even if programs are removed. This is not a tolerable end solution.
    Its ok to get things sorted out, but that's got to come under control at some
    point.

    7. One has to make sure the KNOPPIX670 file from a previous
    remaster is indeed 'gone', not just hidden as PCManFM does when
    you think it's deleting files.
    We may need to look for hidden files and delete them.

    8. Construct an architecture which efficiently provides both production
    and testing of both LiveCD and LiveUSB products as menu choices for the
    Live products.

    9. Address a few collateral issues like the fstab problem & whether there are
    some hidden minirt.gz compatibility difficulties.

    10. An overall wire-brush for bloat; e.g. are all those sudos necessary.

    Here's my current rem_10.sh version of the Forester/Capricorny/kl522 script:

    Code:
    #!/bin/bash
    # Based loosely on Foresters script on Knoppix-forum modified by tay 20110511-20110810
    #   with touches by kl522 & utu.
    
    function to_exist() {
        [ -d "$1" ] || sudo mkdir -p $1 ;
    }
    function purge_or_create() {
        [ -d "$1" ] && sudo rm -rf $1 
        sudo mkdir -p $1 ;
    }
    function remaster_knoppix() {
       command=$1; shift;
       operand=$1; shift;
        case "${command} ${operand}" in
    
        "create workspace")         # Setup workspace as loop image 
         #sudo mount -t ntfs /dev/sdd1 /media/sdd1 (mounted prior to using this script)
        workdir=$1; shift;
            psize=$1; shift; 
    #       sudo dd if=/dev/zero of=${workdir}/knoppix-remaster-data.img bs=1M count=$psize # replace with sparse-file approach
            dd if=/dev/zero of=${workdir}/knoppix-remaster-data.img bs=1 count=0 seek=${wrkspc_sz}  
            sudo losetup /dev/loop7 ${workdir}/knoppix-remaster-data.img   
            sudo mkfs.ext3 /dev/loop7
            sudo losetup -d /dev/loop7
            purge_or_create /tmp/knx-remaster-data;
            sudo mount  ${workdir}/knoppix-remaster-data.img /tmp/knx-remaster-data -o loop=/dev/loop7 ; 
            ;;
         "copy live-system")      # This is the simplified copy
            to_exist /tmp/knx-remaster-data/knx_source ;
               # Copy main /UNIONFS
            sudo rsync -ax --exclude=home --exclude=lost+found --exclude=var /UNIONFS/ /tmp/knx-remaster-data/knx_source;
               # Use a couple of directories/files from KNOPPIX as stubs
            sudo rsync -ax /KNOPPIX/home /KNOPPIX/var  /tmp/knx-remaster-data/knx_source;
            sudo rsync -ax /KNOPPIX/etc/fstab /tmp/knx-remaster-data/knx_source/etc;
            ;;
        "make isofs")  # We don't use pipe here
            purge_or_create /tmp/knx-remaster-data/knx_tmpiso ;
            sudo chmod a+rwx /tmp/knx-remaster-data/knx_tmpiso;
            sudo mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" -quiet -hide-rr-moved \
                 -cache-inodes -pad /tmp/knx-remaster-data/knx_source > /tmp/knx-remaster-data/knx_tmpiso/knoppix.iso ;
            ;;
        "compress isofs") # Not optimized cloop compression
            newknoppix_dir=$1; shift;
            to_exist ${newknoppix_dir}
            sudo create_compressed_fs -B 65536 /tmp/knx-remaster-data/knx_tmpiso/knoppix.iso $newknoppix_dir/KNOPPIX;
            ;;
        "make squashfs")
            newknoppix_dir=$1; shift;
            to_exist ${newknoppix_dir}
            sudo mksquashfs /tmp/knx-remaster-data/knx_source  $newknoppix_dir/KNOPPIX.sq -b 262144 -noappend ;
            ;;   
        "loopcreate persistent") # Create new persistent image, size in MB must be given.  
            newknoppix_dir=$1; shift;
            to_exist ${newknoppix_dir}
            psize=$1; shift;  
            sudo dd if=/dev/zero of=${newknoppix_dir}/knoppix-data.img bs=1M count=$psize
            sudo losetup /dev/loop6  ${newknoppix_dir}/knoppix-data.img
            sudo mkfs.ext3 /dev/loop6
            purge_or_create /tmp/knx-data
            sudo mount -t ext3 -rw -o loop /dev/loop6 /tmp/knx-data  ;
            sudo rsync -ax /UNIONFS/home /UNIONFS/var /tmp/knx-data ;
            sudo umount /tmp/knx-data;
            sudo losetup -d /dev/loop6
            ;;
       "purge workspace")
            sudo umount /tmp/knx-remaster-data
            sudo losetup -d /dev/loop7
            workdir=$1; shift;
            sudo rm -f ${workdir}/knoppix-remaster-data.img 
            ;;
        *)
            echo oops;
            ;;
        esac
    }
    
    echo '...Starting the re-mastering process...'
    
    # Calling examples:
    #./rem_03.sh /media/sdc1 15000 /media/sdc1/KNOPPIX670&
    #./rem_04.sh /media/sdc1 15000 /media/sdc1/KNOPPIX670&
    #./rem_05.sh /media/sdc1 15G /media/sdc1/KNOPPIX670&
    #./rem_10.sh /media/sdc1 15G /media/sdc1/KNOPPIX670&
    
    
      wrkspc_dir=$1 ;   wrkspc_sz=$2 ; remaster_dir=$3 ; persist_sz=$4 ;  
      echo -e 'Set-up workspace ******************************** \c'; date         
      remaster_knoppix create workspace ${wrkspc_dir} ${wrkspc_sz}
      echo -e 'Copy live-system ******************************** \c'; date         
      remaster_knoppix copy live-system
      echo -e 'Make isofs ************************************** \c'; date          
      remaster_knoppix make isofs
      echo -e 'Compress isofs ********************************** \c'; date          
      remaster_knoppix compress isofs ${remaster_dir}
    #  echo -e 'Make squashfs ********************************** \c'; date          
    #  remaster_knoppix make squashfs ${remaster_dir}
    #  echo -e 'Create persistent image ************************ \c'; date          
    #  remaster_knoppix loopcreate persistent  ${remaster_dir} ${persist_sz} 
      echo -e 'Purge workspace ********************************* \c'; date          
      remaster_knoppix purge workspace ${wrkspc_dir}
      echo -e 'All done **************************************** \c'; date 
      exit 0
    Sample to follow output on next post.
    .

  2. #2
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    The following is a slightly abridged copy of the terminal output seen
    when running the rem_10.sh command. Take note of the times for the different
    phases.

    Code:
    knoppix@Microknoppix:~$ ./rem_10.sh /media/sdc1 15G /media/sdc1/KNOPPIX670&
    [1] 3311
    knoppix@Microknoppix:~$ ...Starting the re-mastering process...
    Set-up workspace ******************************** Thu Aug 18 14:53:53 UTC 2011
    0+0 records in
    0+0 records out
    0 bytes (0 B) copied, 0.000100152 s, 0.0 kB/s
    losetup: /dev/loop7: device is busy
    mke2fs 1.41.12 (17-May-2010)
    /dev/loop7 is mounted; will not make a filesystem here!
    loop: can't delete device /dev/loop7: Device or resource busy
    rm: cannot remove `/tmp/knx-remaster-data': Device or resource busy
    mount: stolen loop=/dev/loop7
    Copy live-system ******************************** Thu Aug 18 14:53:53 UTC 2011
    Make isofs ************************************** Thu Aug 18 14:56:20 UTC 2011
    Warning: creating filesystem that does not conform to ISO-9660.
    Compress isofs ********************************** Thu Aug 18 14:59:39 UTC 2011
    2 processor core(s) detected
    Block size 65536, expected number of blocks: 32558
    [ 9] Blk#     0, [ratio/avg.  20%/ 20%], avg.speed: 65536 b/s, ETA: 32557s
    [ 9] Blk#   100, [ratio/avg.  11%/ 11%], avg.speed: 3309568 b/s, ETA: 642s
    [ 9] Blk#   200, [ratio/avg.  11%/  9%], avg.speed: 6586368 b/s, ETA: 321s
    [ 9] Blk#   300, [ratio/avg.   8%/  9%], avg.speed: 6575445 b/s, ETA: 321s
    [ 9] Blk#   400, [ratio/avg.  11%/  9%], avg.speed: 8759978 b/s, ETA: 240s
    
    *********************** deleting repetitive stuff ************************
    
    [ 9] Blk# 31800, [ratio/avg.   0%/ 38%], avg.speed: 9100918 b/s, ETA: 5s
    [ 9] Blk# 31900, [ratio/avg.   0%/ 38%], avg.speed: 9129536 b/s, ETA: 4s
    [ 9] Blk# 32000, [ratio/avg.  51%/ 38%], avg.speed: 9118337 b/s, ETA: 4s
    [ 9] Blk# 32100, [ratio/avg.  39%/ 38%], avg.speed: 9107234 b/s, ETA: 3s
    [ 9] Blk# 32200, [ratio/avg.   0%/ 38%], avg.speed: 9096227 b/s, ETA: 2s
    [ 9] Blk# 32300, [ratio/avg.  99%/ 38%], avg.speed: 9124475 b/s, ETA: 1s
    [ 9] Blk# 32400, [ratio/avg. 100%/ 38%], avg.speed: 9113441 b/s, ETA: 1s
    [ 9] Blk# 32500, [ratio/avg.  48%/ 38%], avg.speed: 9141568 b/s, ETA: 0s
    [ 9] Blk# 32557, [ratio/avg.   0%/ 39%], avg.speed: 9118466 b/s, ETA: 0s
    
    Statistics:
    gzip(0):     0 (    0%)
    gzip(1):     0 (    0%)
    gzip(2):     0 (    0%)
    gzip(3):     0 (    0%)
    gzip(4):     0 (    0%)
    gzip(5):     0 (    0%)
    gzip(6):     0 (    0%)
    gzip(7):     0 (    0%)
    gzip(8):     0 (    0%)
    gzip(9): 32558 (1e+02%)
    7zip:     0 (    0%)
    Writing index for 32558 block(s)...
    Writing compressed data...
    Purge workspace ********************************* Thu Aug 18 15:03:32 UTC 2011
    loop: can't delete device /dev/loop7: No such device or address
    All done **************************************** Thu Aug 18 15:03:32 UTC 2011
    So, please jump in here and help solve some of the problems (#'s 1 to 10)
    or any others you've discovered.

  3. #3
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    For the un-initiated, rem_10.sh does the following:

    It establishes a 15 Gb sparse-file on sdc1, works with it a while
    then erases its tracks. When it's about done, it dumps a new
    KNOPPIX compressed image (about 794 Mb right now)
    into the KNOPPIX670 directory on sdc1.

    You can verify the sparse-file by calling up a second terminal
    while the first is humming away doing its job.

    Mine is a 2 cpu 2.1 GHz laptop with 4 Gb ram.
    The computer is 64-bits, but I'm only using 32-bit kernel, etc.
    It does the 794 Mb cd renovation in about 15 minutes;
    kl522 has cut that down from the 25 minutes required before the using sparse-file approach.
    Last edited by utu; 08-19-2011 at 07:13 PM.

  4. #4
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802

    A few comments

    I haven't worked with remastering issues since August, and, at the first glance, had little to add. Seems OK to me.

    Entering this realm again now, I have some comments. Numbering not following the issues mentioned.

    1. Using the seek parameter with dd ("sparse file") is obvious when one knows about it! It is not necessary to use a loop image for remastering on any file system, but I think it is good&reasonably safe practice to use it on NTFS partitions. Then, only one fixed file is modified, no block moving etc.

    2. This method, with all the copying, may be unnecessarily complicated, but I think it is rather robust, and it works with several settings - it is agnostic wrt cloop vs squashfs, for instance. The new compressed image is created from an actual file system, and, therefore, any inconsistencies resulting from UNIONFS bugs/troubles etc, should have the highest chance of being ruled out. It has never failed in giving me the space I have expected. Beware, however, that space freed by purging one package may partly be filled up installing another, if they have common dependencies. And, it moves installed programs from persistent store (freeing up space there) into the compressed image, which of course leads to compressed image growing. To get everything in place, I typically have to repeat this process 2-3 times.

    If creating isofs/squashfs from /UNIONFS, only omitting directories is totally safe, copying can of course be skipped, but for me, it is enough with just one single potential problem in the shortcut to use the longer and proven way.

    Excessive listings to screen may be avoided by redirection?

    Furthermore, with USB3, it's not that slow on external media, also good USB3 sticks may be used.

    3. Making a new persistent image could be left to Knoppix, but I want to have an exact copy of my environment in the remaster, and for that, I need the /home and /var directories. The handling of /var in remastering may not have been optimal, since in my original 6.7.1, /var takes up 431 MB, and <10MB of this is currently in the persistent image. Files that probably will be modified should not go into the compressed image, for after modification both original and modified version will take up space, but possibly, a considerable part of 6.7.1's /var will be left untouched? I don't know.
    A considerable amount of data in /var is the main reason the new persistent store is very far from empty, and in case very little modification of the remaster is intended, the modified /var could be compressed. In my use, there will often be a series of remasterings, so a lot could be updated.
    Maybe space in the compressed image could be saved by only placing a stub in the compressed image, and have everything persistent?

    4. I think it is good practice to make the necessary privilege level explicit, even if this implies script code sprinkled with sudo's. It's safest to work as an ordinary user. If root privileges are not needed for an operation, the sudo may of course be omitted, but I tend to retain it if the operation under some policies _could_ have required root access.

    5. Loop device handling seems somewhat buggy to me, and in addition, the busybox tool versions used in Knoppix init are not fully compatible with standard tools. But as long as the job gets done, I'm not that much concerned with error messages etc. Also, using scripts for file handling allows us to throw in all kinds of functionality, like moving/releting/renaming existing files etc.

    6. The main issue in remastering is not touched upon in these scripts : How to purge? And the second most important issue is not addressed either: How to ensure a complete&working selection of packages? A major problem is that package organization is constantly changing, so the procedures that worked with one Linux release may need modification to work with another. I set up a series of scripts to do the purging, and also try to automate the installations as much as possible.

  5. #5
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802

    ...And a few on media and booting

    For serious daily Knoppix use, a good persistent store is mandatory, and that means >2 GB for me. Just upgrading/book-keeping stuff easily reaches 1 GB, and I normally go for 4 GB at once. The CD is stripped down, to the point that something important for some uses is usually lacking, therefore, I start with the DVD version whenever I can. Total system size therefore becomes ca 8 GB. But that also means I can't place the remastered version on a CD/DVD. And why should I?

    I run Knoppix from internal HDD, USB HDD, (fast) memory sticks, SD cards, smartphone - seldom CD/DVD. These have the space I need, and the KNOPPIX directory (including persistent store(s)) is easily copied.

    Booting? Thus far, I still prefer legacy GRUB (not the "new" thing) installed on its own partition, where I can add boot alternatives just by adding kernels and initrds and editing /boot/grub/menu.lst. Only limitation is with Win7 - I don't need it enough to chase the workarounds needed to Linux boot it, relying instead on virtual machines. I may turn to GRUB4DOS later.

  6. #6
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    Greetings, Capricorny.

    Nice to hear from you. Welcome back.

    As you can see, prior to your post,
    this thread was not a success.

    Werner has made things a little simpler.
    You might want to catch up on his approach to remastering.

    http://www.wp-schulz.de/knoppix/summary.html

    Best Regards.
    Last edited by utu; 01-04-2012 at 06:42 PM.

  7. #7
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802

    Totally different approaches, for different purposes IMHO

    This approach and Werner's are almost diametrically opposite approaches to remastering, and suitable for very different purposes. And if "remastering" means creating a new ISO image for CD/DVD, this is of course not remastering at all.

    But, in my understanding, remastering means essentially recompressing a modified UNIONFS, possibly together with other modifications. That's what I and a few other users need, and provided the necessary boot files are present, copy/backup by the built-in Install Knoppix to flash disk can be used.

    As for virtual machines, I have found qemu very useful for checking the remastering, but personally, I don't really need them when I don't modify minirt.gz. Nor have I ever needed a hard disk install for remastering. That was necessary to build a pure 64 bits Knoppix version in a straightforward way, but that's a different story.

  8. #8
    Moderator Moderator
    Join Date
    Nov 2010
    Location
    Germany/ Dietzenbach
    Posts
    1,124
    But, in my understanding, remastering means essentially recompressing a modified UNIONFS, possibly together with other modifications.
    You'll get all of this while using a remastered CD by my way with "Install KNOPPIX to flash disk"

    The other method is "recombine"; but I don't like it and I tested it not very well. Modifing Knoppix in a real HD-installation is more versatile.

    Greetings Werner * http://www.wp-schulz.de/knoppix/summary.html
    Own Rescue-CD (Knoppix V6.7.1 remaster)

  9. #9
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Fast USB3 sticks tend to also run very well with USB2. With your assumptions, USB remastering of CD seems quite OK, maybe even of DVD. I think I will try out this myself too - the high number of files to copy could in fact reduce the time difference between stick and HD.

  10. #10
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    I've just ordered a Transcend 16 GB SDHC Class 10 Flash Memory Card.

    I have in mind the following experiment:

    Use Werner's latest re-mastering idea 'Recombine flash disk Installation with persistent memory',
    with the following adaptations:.

    1. Use an existing 2 Gb Card with Knoppix 6.7.1 and its own well-developed persistent store
    instead of a live CD to prepare a working 16 Gb Knoppix system;
    2. Add Werner's scripts to the 16 Gb system;
    3. Use the 16 GB System to 'recombine' the KNOPPIX and KNOPPIX-DATA files from the 2 Gb card;
    4. Replace the original KNOPPIX file on the 2 Gb card with the 'recombined'; and
    5. Regenerate an 'empty' KNOPPIX-DATA file on the 2 Gb card.

    Notes:
    1. I expect step (1) will initially clone the 2 Gb system, including its small persistent store.
    2. Use cloop in the initial effort, but experiment with squashfs as well for comparison.

    Any comments on this approach?
    Last edited by utu; 01-13-2012 at 09:46 PM.

Page 1 of 4 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


SanDisk 32GB Ultra Drive Dual Go USB Type-C Flash Drive, Green - SDDDC3-032G-G46 picture

SanDisk 32GB Ultra Drive Dual Go USB Type-C Flash Drive, Green - SDDDC3-032G-G46

$8.99



SanDisk 128GB Ultra Dual Drive USB Type-C, USB 3.1 Flash Drive - SDDDC2-032G-G46 picture

SanDisk 128GB Ultra Dual Drive USB Type-C, USB 3.1 Flash Drive - SDDDC2-032G-G46

$10.99



SanDisk 256GB Ultra Dual Drive Go USB Type-C Flash Drive, Black- SDDDC3-256G-G46 picture

SanDisk 256GB Ultra Dual Drive Go USB Type-C Flash Drive, Black- SDDDC3-256G-G46

$21.99



SanDisk 256GB Ultra USB 3.0 Flash Drive - SDCZ48-256G-U46 picture

SanDisk 256GB Ultra USB 3.0 Flash Drive - SDCZ48-256G-U46

$19.99



64GB USB 3.0 Flash Drive USB Memory Stick High Speed Retractable USB Thumb Drive picture

64GB USB 3.0 Flash Drive USB Memory Stick High Speed Retractable USB Thumb Drive

$5.99



Type C USB 3.0 Flash Drive Thumb Drive Memory Stick for PC Laptop 512GB 1TB 2TB picture

Type C USB 3.0 Flash Drive Thumb Drive Memory Stick for PC Laptop 512GB 1TB 2TB

$10.30



2TB USB 3.0 Flash Drive Memory Photo Stick for iPhone Android iPad Type C 3 IN1 picture

2TB USB 3.0 Flash Drive Memory Photo Stick for iPhone Android iPad Type C 3 IN1

$13.00



Sandisk 16GB 32GB 64GB 128GB Cruzer Blade Flash Drive Memory Stick USB Lot Pack picture

Sandisk 16GB 32GB 64GB 128GB Cruzer Blade Flash Drive Memory Stick USB Lot Pack

$4.99



LOT 32GB 64GB 128GB USB 3.0 Flash Drive Memory Stick Retractable Thumb Drive picture

LOT 32GB 64GB 128GB USB 3.0 Flash Drive Memory Stick Retractable Thumb Drive

$420.99



USB Flash Drive Memory Stick Pendrive Thumb Drive 4GB, 8GB, 32GB, 64GB 128GB LOT picture

USB Flash Drive Memory Stick Pendrive Thumb Drive 4GB, 8GB, 32GB, 64GB 128GB LOT

$259.34