Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 29

Thread: Re-master your flash installation

  1. #11
    Senior Member
    Join Date
    Jan 2011
    Posts
    242

    Re-master your flash installation - DVD version

    Has this thread been renamed ? I like the new title. Lots. It is exactly what I was looking for.

    Are the instructions for the CD or the DVD ? For the CD you may need /lots_of_free_space but for the DVD you need /super_lots_of_free_space. Like about 5 times as much so you've got to plan ahead.

    I tried following the instructions with the DVD (6.4.3 EN). The 'main' command did not scale so well. It failed claiming I was out of virtual memory. The 'main' command (taken I see from http://www.knoppix.net/wiki/Knoppix_Remastering_Howto)

    Code:
    mkiosfs ... | create_compressed_fs ... > KNOPPIX/KNOPPIX
    is a pipe, which is usually good. Usually the right-hand side consumes data as fast as the left-hand side produces it. The pipe means no (large) temporary files are needed and it all runs faster as a result.

    However, a look at the output of create create_compressed_fs -h, particularly the description of the -s flag and the stuff at the end, suggests the pipe is a bad idea: the right-hand side is going to buffer its entire input before it processes anything.

    If you are remastering the CD, you might have enough real memory and/or enough swap (the HowTo describes how to create a 1 Gb swap) but for remastering the DVD you're looking at 8 Gb or so. I have no other use for an 8 Gb swap partition, so I split the main command in two and used a temporary file (a bit like using the -f option of create_compressed_fs).

    The 'main' command examples all prefix the create_compressed_fs with nice -5. This too I find strange. The right hand side runs at a lower priority that the left, so you would expect data to pile up in the pipe even if you use the -s or -f options of create_compressed_fs. The right hand side takes a long time to run and uses a lot of processor. The intention of the nice -5 is run the compression at a low priority so you can do other 'stuff' while you wait. But it uses up a lot of memory too so doing other 'stuff' may not be all that great. Me, I left off the nice -5 and got in some zeds.

    The instructions do cover in detail how to meet the challenge of getting the new KNOPPIX image onto the USB and how to clean up the contents of the old persistent store. However, I had to read it three times before it was clear that you don't have to do it the 'risky' bit booting Knoppix with the debug cheatcode.

    So, for the benefit of those new to Knoppix and remastering, once you have created the new KNOPPIX image:
    - shutdown your PC / laptop;
    - remove the USB;
    - boot your Knoppix Live CD (the one you installed to USB with the first place);
    - insert the USB stick again;
    - now follow the instructions.

    Also for the benefit of those new to Knoppix and remastering:
    - you should run all the commands from a 'root shell'
    - if you don't, you'll have to prefix most commands with sudo
    - even so, the command with the '>' may fail.

    Those new to Knoppix and remastering may also be a little nonplussed by the suggestion that what they put on the new image is an entirely a free choice:
    - update/remove/install software until you are happy/fed up/get a life;
    - you don't want /home in the new image (that's your stuff)
    - you don't want /var in the new image (stuff that changes all the time);
    - take /home and /var from the old image
    - it is a reasonable bet everything else should go in the new image
    - until you know better
    - when you clean up your knoppix-data.img, delete everything except /home and /var.

    I working my way towards describing how much disk space you (might) need to remaster the DVD. To do that I'm going to have to show you how I did it. I do everything a directory created for the purpose and always write to a subdirectory. Fewer disasters that way. The subdirectories may be real directories or symbolic links or mount points (my favourite). That's not important at this stage. I put all the scary commands into a script so that next time I remaster, I won't have to type them in or copy/paste them from a web page. Fewer disasters that way.

    Code:
    #!/bin/bash
    
    command=$1; shift;
    operand=$1; shift;
    
    case "${command} ${operand}" in
      ("copy source")
        excludes="--exclude=/home --exclude=/lost+found --exclude=/var";
        sudo rsync -ax --delete ${excludes} /UNIONFS/ knoppix_source;
        sudo rsync -ax /KNOPPIX/home /KNOPPIX/var knoppix_source;
        sudo cp -p /KNOPPIX/etc/fstab knoppix_source/etc;
        ;;
      ("make isofs")
        sudo chmod 1000:1000 knoppix_tmpiso;
        sudo mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" -hide-rr-moved -cache-inodes -pad knoppix_source > knoppix_tmpiso/knoppix.iso
        ;;
      ("compress isofs")
        mkdir -p knoppix_master/KNOPPIX;
        sudo create_compressed_fs -b -B 65536 knoppix_tmpiso/knoppix.iso knoppix_master/KNOPPIX/KNOPPIX;
        echo "Now reboot to CD";
        ;;
      ("backup usb")
        mkdir -p knoppix_backup/KNOPPIX;
        cp -p knoppix_usb/KNOPPIX/KNOPPIX knoppix_usb/KNOPPIX/knoppix_data knoppix_backup/KNOPPIX;
        ;;
      ("copy master")
        cp -p knoppix_master/KNOPPIX/KNOPPIX knoppix_usb/KNOPPIX;
        ;;
      ("mount data")
        mkdir -p knoppix_data;
        sudo mount -o loop knoppix_usb/KNOPPIX/knoppix-data.img knoppix_data;
        ;;
      ("prune data")
        cd knoppix_data;
        sudo rm -fr $(ls -ignore=home --ignore=var);
        cd ..;
        ;;
      ("umount data")
        sudo umount knoppix_data;
        rmdir knoppix_data;
        ;;
      (*)
        echo oops;
        ;;
    esac
    So, meet the directories:

    knoppix_source - where the filesystem for the new image is assembled. It has to be a Linux filesystem. It must be at least the size of /UNIONFS.
    Budget 10 Gb.

    knoppix_tmpiso - where the uncompressed iso image goes. It can't be a FAT filesystem as it takes a single file bigger than 4 Gb. The iso file is slightly smaller than its source.
    Budget 10 Gb.

    knoppix_master - where the compressed iso image (the new Knoppix image) goes. This can be a FAT filesystem, since the image will be copied to a FAT filesystem on USB. This also means it can't be bigger than 4 Gb (the original 6.4.3 is 3.5 Gb).
    Budget 4 Gb.

    knoppix_backup - back up the old Knoppix image and knoppix-data.img goes here. I'm using a 1 Gb knoppix-data.img. Again, can be any filesystem, even FAT.
    Budget 5 Gb.

    knoppix_usb - the USB installation that is being remastered is under here.
    This already exists on the USB so ...
    Budget 0 Gb.

    That's 29 Gb total but if you use just one large Linux partition and delete stuff as you go along you can do it in 20 Gb. I don't know what you'd need to remaster the CD but I guess 4 Gb would be enough.

    Where does all this disk space come from ?

    I'm lucky in that I when I remaster I don't have ntfs between me and my hard drive. I only have lvm2 so I can whip up a new filesystem in less time that it takes to type this.

    Many (me included) first try Knoppix because they want to try Linux but do not want to touch Windows. If you are one of those, my advice is don't touch Windows (aka "Don't try this at work"). Find another way or find something else to do.

    I don't know how inexpensive hard disks or USB sticks are in $ but my advice would be to buy a USB stick big enough to do job. That's a 32 Gb stick for the DVD and may be only a 4 Gb for the CD.

    You would need to plan ahead. Figure out what goes in which partition, how big each partition must be, partition and format the USB all before you install Knoppix to USB. Me, I'm already on my third iteration.

  2. #12
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by dinosoep View Post
    I hoped that knoppix forums would sent me a mail when someone reacted.
    Go to the top of the page. Click on the caret to the right of Thread Tools and select Subscribe to this thread... from the drop down list.

  3. #13
    Senior Member
    Join Date
    Jan 2011
    Posts
    123
    sudo chmod 1000:1000 knoppix_tmpiso; didn't work, chmod 777 knoppix_tmpiso did however
    thank you for the script, I'm currently trying it out

  4. #14
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by dinosoep View Post
    sudo chmod 1000:1000 knoppix_tmpiso; didn't work, chmod 777 knoppix_tmpiso did however
    thank you for the script, I'm currently trying it out
    Oops, you're quite right. I meant:

    Code:
    sudo chown 1000:1000 knoppix_tmpiso
    I often type chmod when I mean to type chown.

    The idea is to make knoppix_tmpiso writeable without using sudo, so either what I meant to type or what you did type will do.

  5. #15
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by kl522 View Post
    In any case, the steps above involved will be somewhat tricky to the Linux newbies. I wish there
    is a cheatcode called "upgrade=/dev/sda4:/master/KNOPPIX/KNOPPIX" which will automatically
    perform the upgrade. I will leave this as an exercise for those interested to do so by modifying minirt.gz.
    If you organise thing right, you can use cheat codes that are already there (assuming Knoppix 6.4.3 or equivalent).

    Suppose the Knoppix you have just remastered is on filesystem /dev/sdb1 under the directory KNOPPIX. Suppose further that you arranged for the new remastered Knoppix to be on device /dev/sda1 under the directory /home/yourself/remaster. This directory contains both the new KNOPPIX file and the cleaned-up knoppix-data.img (or aes) file. I don't think it has to contain all the other little files normally found in the KNOPPIX directory but it wouldn't hurt.

    You can now reboot to the remastered knoppix for test purposes with the cheat codes:

    Code:
    fromhd=/dev/sda1 knoppix_dir=home/yourself/remaster
    The directory /home/yourself/remaster has to be a real directory (avoid symbolic links and mount points). A nice feature of this approach is that /dev/sda1 doesn't have to be a vfat file system so long as it is a) visible at boot time and b) of a type the init script understands. This means it could (you didn't get this from me) be your ntfs Windows 7 main disk.

    If the new remastered Knoppix seems to be working OK you can use it to copy the new KNOPPIX file over the old one:

    Code:
    mount /dev/sdb1 /media/sdb1
    cp -pf /mnt-system/home/yourself/remaster/KNOPPIX /media/sdb1/KNOPPIX/
    Then clean up the old knoppix-data.img:

    Code:
    mkdir knoppix_tmp
    sudo mount -o loop=/dev/loop1 /media/sdb1/KNOPPIX/knoppix-data.img knoppix_tmp/
    cd knoppix_tmp
    sudo rm -fr $(ls --ignore=home --ignore=var);
    Finally, reboot Knoppix from your USB stick 'as normal'.

  6. #16
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    801

    Isn't ext3 working partitions safer? Updating 6.4.4 packages forces remaster

    Quote Originally Posted by Forester View Post
    If you organise thing right, you can use cheat codes that are already there (assuming Knoppix 6.4.3 or equivalent).
    ....
    Code:
    fromhd=/dev/sda1 knoppix_dir=home/yourself/remaster
    The directory /home/yourself/remaster has to be a real directory (avoid symbolic links and mount points). A nice feature of this approach is that /dev/sda1 doesn't have to be a vfat file system so long as it is a) visible at boot time and b) of a type the init script understands. This means it could (you didn't get this from me) be your ntfs Windows 7 main disk.
    I wouldn't recommend newbies experimenting with remastering on Win7 NTFS partitions. I think it is much safer to shrink the Win partition on a disk, set up an ext3 partition there, and do everything on "native Linux".

    Apart from that, I think the method looks very promising. I will try out something like this myself now, as I feel forced to do it: Upgrading all packages on Knoppix 6.4.4 DVD today, filled up about 2.6 of total 4 GB persistent store. Therefore, people should not attempt it without really in need of it. OR, as a first step in remastering.

    I'll start out using internal hard disk, then I may try with USB-sticks - but I think a better external storage for remastering would be an external SSD drive with USB3 connection.

  7. #17
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    801

    Expanding the persistent store to 8+ GB as a first step

    To follow up, I think it may be an advantage to do as much as possible before starting the remastering process. That implies doing package installs, upgrades etc the "ordinary" way as much as possible. But how to, within 4GB space? No way there, but we don't have to confine ourselves to FAT32 and DVD limitations.

    I just tried out expanding knoppix-data.img to 8.5 GB, and that works well on ext3. I place the Knoppix package on an ext3 partition and run from there, starting from USB with the fromhd= cheatcode. I'm not sure if there will be some kind of performance problems with large loop-mounted images, but I haven't seen any so far. So I will try with an even larger persistent store, allowing me to do more comprehensive installs and compile large programs.

    If we, after cleaning up and remastering, get outside the 4GB limits, there are several ways to still use USB sticks as basis. The simplest is to format the stick to ext2/ext3, install (for example) GRUB on it and just dump the Knoppix files to it. Somewhat more elaborate would be to split the compressed packages on two volumes, or work with several uncompressed volumes - that has worked just fine for me.

    I think using Linux file systems on sticks is very practical. For exchange of data with other platforms, we can set up a smaller FAT (or even NTFS) partition on the stick, and have that partition automatically mounted on boot-up. Typically, a 16 GB stick could be partitioned into 15 GB ext2 and 1 GB FAT32, the ext2 volume: 4 GB compressed, 10 persistent and 1 spare.

  8. #18
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by Capricorny View Post
    I just tried out expanding knoppix-data.img to 8.5 GB, and that works well on ext3. I place the Knoppix package on an ext3 partition and run from there, starting from USB with the fromhd= cheatcode. I'm not sure if there will be some kind of performance problems with large loop-mounted images, but I haven't seen any so far. So I will try with an even larger persistent store, allowing me to do more comprehensive installs and compile large programs.
    I have used ext3 persistent store for quite sometime now. The only performance problem with large ext3 and ext2 file system ( not specifically to loop-mounted ones ) is when you have to do file system check. The file system check time is really slow when compared to ext4.

  9. #19
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    801

    Package purging

    Quote Originally Posted by kl522 View Post
    ... The file system check time is really slow when compared to ext4.
    I use a 16.5 GB ext2 persistent store, for testing now, seems like checking takes some time.. But, it seems to work. Full system upgrade went well. Which is not obvious, as, for example, I found that dpkg halts in updates on one missing newline in one file list - regardless of dependencies. Now is the time for purging/pruning. Is there any available hinting for this? The packages list, like in http://ftp.knoppix.nl/os/Linux/distr...ckages-dvd.txt doesn't give too much clue, it seems. Is there any way to pick out the games here? Here are some deletion candidates among the 40+ MB packages. Are any of these strictly necessary or very useful?

    Code:
    dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -n ...... 
    ....
    42884  context 
    44452  wireshark-common   
    45644  lmodern 
    46912 kdewallpapers  
    49972 gnome-games-data  
    51448 scribus-ng  
    52616 kde-l10n-es  
    59692 evolution-common  
    60760 pinyin-database  
    62844 openvas-plugins-dfsg  
    65260 tuxpaint-stamps-default  
    68852 kdebase-workspace-data  
    85476 mingw32  
    85972 inkscape  
    86044 neverball-data  
    98064 gcompris-data  
    101889 linux-image-2.6.37-64  
    120892 wesnoth-1.8-data  
    136684 wesnoth-1.8-music  
    203404 crossfire-maps
    In particular, identifying game packages in a simple way would be nice.

  10. #20
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by Capricorny View Post
    I use a 16.5 GB ext2 persistent store, for testing now, seems like checking takes some time.. But, it seems to work. Full system upgrade went well.
    16.5 GB persistent store is considered a huge persistent store but it is still probably not near to a point where file system check is unbearably slow. I used to have a 130 G ext3 file system which file system check takes unbearably long time to complete. I have since migrated to ext4. I am glad that I did. 130 GB is already slow, I can't imagine having a 1 TB ext3 file system !

    As I am writing, I did a small mod to my minirt.gz to support ext4. I think it is about time it should do that.

    Cheers.

Page 2 of 3 FirstFirst 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •