Simple remastering procdure for 7.0.2

[Capricorny]

I am writing this from a remastered 7.0.2, using the usual Poor Man's HD install, running from an external USB3 HD (1000GB WD) with 3.5 GB persistent storage, and a 3.8GB cloop image with Oracle XE, Bibble, R, Octave and some other programs added.

The remastering script is virtually unchanged from earlier versions. I haven't tried to make it very complete. It may rather easily be integrated into a larger script, for "real" remastering, i.e. creating an ISO image for DVD.

Times for 7.0.2 on Toshiba 830R (i5-2410 processor, 8GB RAM), running 32-bits kernel and copying 7.0.2 from Corsair Flash Voyager 32GB USB3, using an ext3 partition on the internal HD for creating a new compressed image. About 65 minutes.

I haven't changed the init script, as it works well for my use right now. But this can be done afterwards.


Times from the output of the script:
Set up workspace...............Sat Jun 2 18:27:50 UTC 2012
Copy live system............... Sat Jun 2 18:28:09 UTC 2012
Make isofs.............. Sat Jun 2 19:07:22 UTC 2012
Compress isofs............... Sat Jun 2 19:22:39 UTC 2012
Loopcreate persistent....... Sat Jun 2 19:31:12 UTC 2012
Purge workspace............... Sat Jun 2 19:32:56 UTC 2012

#!/bin/bash
# Based loosely on Foresters script on Knoppix-forum modified by tay 20110511-20110810,20120602


function to_exist() {
    [ -d "$1" ] || sudo mkdir -p $1 ;
}

function purge_or_create() {
    [ -d "$1" ] && sudo rm -rf $1 
    sudo mkdir -p $1 ;
}

function remaster_knoppix() {
   command=$1; shift;
   operand=$1; shift;

    case "${command} ${operand}" in
 
    "create workspace")         # Setup workspace as loop image 
        workdir=$1; shift;
        psize=$1; shift; 
       # sudo dd if=/dev/zero of=${workdir}/knoppix-remaster-data.img bs=1M count=$psize
        sudo dd if=/dev/zero of=${workdir}/knoppix-remaster-data.img bs=1 count=0 seek=${psize}
        sudo losetup /dev/loop7 ${workdir}/knoppix-remaster-data.img
        sudo mkfs.ext3 /dev/loop7 2>create_workspace.log ; 
        sudo losetup -d /dev/loop7
        purge_or_create /tmp/knx-remaster-data;
        sudo mount  ${workdir}/knoppix-remaster-data.img /tmp/knx-remaster-data -o loop=/dev/loop7 ;
        ;;


     "copy live-system")      # This is the simplified copy
        to_exist /tmp/knx-remaster-data/knx_source ;
           # Copy main /UNIONFS
        sudo rsync -ax --exclude=home --exclude=lost+found --exclude=var /UNIONFS/ /tmp/knx-remaster-data/knx_source;
           # Use a couple of directories/files from KNOPPIX as stubs
        sudo rsync -ax /KNOPPIX/home /KNOPPIX/var  /tmp/knx-remaster-data/knx_source;
        sudo rsync -ax /KNOPPIX/etc/fstab /tmp/knx-remaster-data/knx_source/etc;
        ;;


    "make isofs")  # We don't use pipe here
        purge_or_create /tmp/knx-remaster-data/knx_tmpiso ;
        sudo chmod a+rwx /tmp/knx-remaster-data/knx_tmpiso;
        sudo mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" -hide-rr-moved -cache-inodes -pad /tmp/knx-remaster-data/knx_source > /tmp/knx-remaster-data/knx_tmpiso/knoppix.iso 2>make_isofs.log ;
        ;;

    "compress isofs") # Not optimized cloop compression, also copies "surrounding" system files into target directory
        newknoppix_dir=$1; shift;
        to_exist ${newknoppix_dir}
        sudo create_compressed_fs -B 65536 /tmp/knx-remaster-data/knx_tmpiso/knoppix.iso $newknoppix_dir/KNOPPIX 2>compress_isofs.log;
        sudo rsync -ax --exclude=KNOPPIX --exclude=knoppix-data.img /mnt-system/KNOPPIX/ $newknoppix_dir ;
 
        ;;
    
    "make squashfs")
        newknoppix_dir=$1; shift;
        to_exist ${newknoppix_dir}
        sudo mksquashfs /tmp/knx-remaster-data/knx_source  $newknoppix_dir/KNOPPIX.sq -b 262144 -noappend ;
        ;;   

    "loopcreate persistent") # Create new persistent image, size in MB must be given.  
        newknoppix_dir=$1; shift;
        to_exist ${newknoppix_dir}
        psize=$1; shift;  
        sudo dd if=/dev/zero of=${newknoppix_dir}/knoppix-data.img bs=1 count=0 seek=${psize}
        sudo losetup /dev/loop6  ${newknoppix_dir}/knoppix-data.img
        sudo mkfs.ext3 /dev/loop6 2>loopcreate_persistent.log
        purge_or_create /tmp/knx-data
        sudo mount -t ext3 -rw -o loop /dev/loop6 /tmp/knx-data  ;
        sudo rsync -ax /UNIONFS/home /UNIONFS/var /tmp/knx-data ;
        sudo umount /tmp/knx-data;
        # sudo losetup -d /dev/loop6
        ;;

   "purge workspace")
        sudo umount /tmp/knx-remaster-data
        # sudo losetup -d /dev/loop7
        workdir=$1; shift;
        sudo rm -f ${workdir}/knoppix-remaster-data.img 
        ;;

    *)
        echo oops;
        ;;

    esac


}

# Calling examples:

# ./rem_02.sh /store/local 20G /store/local/KNOPPIX671 3.5G
# ./rem_02.sh /store/local 20G /store/local/KNOPPIX702 3.5G

  wrkspc_dir=$1 ;   wrkspc_sz=$2 ; remaster_dir=$3 ; persist_sz=$4 ; 
echo -e 'Set up workspace...............\c' ; date ; 
  remaster_knoppix create workspace ${wrkspc_dir} ${wrkspc_sz}
echo -e 'Copy live system...............\c' ; date ; 
  remaster_knoppix copy live-system
echo -e 'Make isofs...............\c' ; date ; 
  remaster_knoppix make isofs
echo -e 'Compress isofs...............\c' ; date ; 
  remaster_knoppix compress isofs ${remaster_dir}
 # remaster_knoppix make squashfs ${remaster_dir}
echo -e 'Loopcreate persistent...............\c' ; date ; 
  remaster_knoppix loopcreate persistent  ${remaster_dir} ${persist_sz} 
echo -e 'Purge workspace...............\c' ; date ; 
  remaster_knoppix purge workspace ${wrkspc_dir}

A short overview of the whole process

0. Install Knoppix

0.1 Download and burn DVD
0.2 Run from DVD, use flash-knoppix to write to USBdrive (usb3 if possible)
0.3 Boot from USB, set up persistent storage.
0.4 Do some customizations
0.5 Install some packages:
* flashplayer nonfree plugin
* R, basic package selection
* Some octave packages
* bison and flex
* bibble (last vs to get, it has been acquired by Corel)
* oracle xe server&client
* squashfs-tools
* vmware workstation 32-bits

1. Look at packages and do some purging

1.1 First, original package list

  dpkg-query -W --showformat='${Section} ${Package} ${Installed-Size}\n' | sort -n > knx702_DVDpack_sect2.txt

1.2 Purging done here:
Games, ca 975 MB freed
bacula, 6M, lyx, 50M, abiword etc 29M, gnucash 30M, kmail 30M, icedove 40M, amanda 8M, tuxpaint 75M, scribus 62M, gnome-games 11M, kdewallpapers 95M, kde localiz 166M, libreoffice-help-de 25M, mingw 135M, texmacs 34M

Commands used:

#Purging games 
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^games | cut -d' ' -f2`

  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep bacula | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^editors | grep lyx | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^editors | grep abiword | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^gnome | grep gnucash | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^mail | grep kmail | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^mail | grep icedove | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^utils | grep amanda | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^graphics | grep tuxpaint | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep scribus | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^gnome | grep gnome-games | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^kde | grep kdewallpapers | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^kde | grep kde-l10n | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^doc | grep libreoffice-help-de | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^devel | grep mingw | cut -d' ' -f2`
  sudo aptitude purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^editors | grep texmacs | cut -d' ' -f2`

Now, the extra packages will fit comfortably into 4GB cloop - using squashfs we would get even more space.

2. Run first remastering
2.1 Run the script

   ./rem_02.sh /store/local 20G /store/local/KNOPPIX702 3500M

2.2 Copy over the remaining Knoppix files

sudo rsync -ax --exclude=KNOPPIX --exclude=knoppix-data.img /mnt-system/KNOPPIX/* KNOPPIX702

2.3 The /boot directory and ldlinux.sys I copy manually over to the new structure for now.

Personally, I don't feel much need for developing this further right now, but there may be some bugs involved - although I have been running all day long from such remasterings since last spring.

I will see if this release can be used for creating a 64-bits Knoppix 7, so that I can run 64-bits versions of the most important applications. With 16GB RAM, (even with 8 ) I can run several virtual machines simultaneously on a 64-bits system. It worked with 6.4.4, but a bit rough.

[utu]

Greetings, Capricorny; nice to hear from you.

I'm familiar with Werner's kn-recombine approach.
If you are familiar with his approach as well, I'd appreciate a brief word-comparison
of the two for the technically challenged, like myself.

What are the relative strengths and advantages of one vs the other?

Regards.

[Capricorny]

I'm not familiar with it - this approach takes an ordinary running instance of Knoppix, from whatever medium, and creates new cloop/squashfs and persistent images from it in a relatively robust way. It can be run as often as needed - this is actually written from my second in row remastering, having included VMware workstation in the cloop. So I have still around 3GB free on the persistent store now. The cloop image is close to the 4GB limit, and as I want to use FAT32 if I like, I may have to use squashfs - or purge some more. (Actually, the cloop I run from now resides on PC main NTFS partition - never use Windows myself, but keep it because some others may do.)
Having way too many of them behind me, I'm not fond of ordinary HD installs - just got my prejudices confirmed today with Debian 6.0.5. Compared it to the beatuful simplicity of pure 64-bits version of Knoppix 6.4.4 - I am trying to find a better way to proceed - but ordinary 64-bits Debian seems to have made little progress the last year.

[utu]

Greetings, Capricorny

Slight side issue for you:
If you use 'knoppix64' as a cheat code with an unmodified 7.0.2,
what do you get for 'uname -s'?

I can't seem to turn on a 64-bit kernel.

Thanks.

[Capricorny]

I think it can. And this behavior shouldn't be changed by a simple remastering, leaving init and packages versions unchanged.

In an otherwise rather defunct purgatorial remastering, I get "x86_64" as response to the "arch" command, but that's with 16GB RAM.

[Capricorny]

Small update:
1. Installed several Java packages to make it more complete for server use, purging things like gnucash and evolution to make space. Will copy cloop+persistent into existing 7.0.2 VMs afterwards. Groovy, Grails, Tomcat and JBoss installed, pluss Oracle Java EE 6 w/Glassfish3. Glassfish etc turned out to be very bulky, ca 550MB, and I moved it to /var/lib to get it out of the compressed image, to keep that down at ca 4GB. I think things like Java development and server may be good candidates for KNOPPIX2 cloops - also because of possible version changes. The same may apply to other specialized program suits. Will look more into this, as it is very tempting to have everything in a 4GB image..

2. On an i7-2630 with a slow HD, the process (running script listed above) took about 32 mins, with almost half of it spent creating isofs. Will try out with squashfs later, possibly together with SSD.

3. One reason for stuffing all kinds of programs into one compressed image, is the simplicity of setting up development and testing environments: Just copy Knoppix VMs, and run different servers on different VMs. Two Windows XP and two Knoppixes ran very smoothly in testing under VMware workstation 8, and when virtual Knoppix is started in runlevel 3, it doesn't draw much resources when idle.

[Capricorny]

Update:
Times for the above script running, creating squashfs:

Set up workspace...............Thu Jun 21 10:01:06 UTC 2012
Copy live system...............Thu Jun 21 10:01:19 UTC 2012
Create squashfs...............Thu Jun 21 10:10:15 UTC 2012
Loopcreate persistent...............Thu Jun 21 10:15:57 UTC 2012
Purge workspace...............Thu Jun 21 10:17:14 UTC 2012

Ca 16 minutes for full remastering of a Poor Man's Install, on a slow (5400RPM, small cache) harddisk. Note that without a loop-mounted work image, purging temporary files may take quite some time. So the image technique is both file system agnostic and rather efficient.
And sizes for both images:

-rw-r--r-- 1 root root 3799736320 Jun 21 10:15 KNOPPIX.sq
-rw-r--r-- 1 root root 4035784755 Jun 20 13:06 KNOPPIX

It is of course possible to get higher isofs compression, but that will take some more time.
I think this illustrates the handiness of squashfs, and I wish Klaus K would include the small changes needed to support it in the minirt init script. Doesn't need to tell anybody about it, either, in case he is afraid of more support work

If I get around to modifying it, I'll post a patch for the 7.0.2 minirt init, otherwise I think the structure of earlier posted patches still apply. This exercise was just for testing, but I must say I am tempted to use squashfs as the default - my whole current setup, including a lot of Java infrastructure, will then fit on a FAT32-file.