Remaster from HD Install How-To: Comments Please

[CrashedAgain]

This is not really finished but is far enough along to post for comments & maybe will be of assistance to some. I'm not going to post the clean up script yet, I just tried it with 3.4 & there appears to be some problems.

REMASTERING FROM HD SETUP HOW TO

System Requirements & Setup

Basic method: Remaster directly from an operating HD install instaed of from a specially installed system accessed by 'chrooting' into it. This method also includes a second installation of a 'toHD' or 'poor mans install' as there are some operations (such as creation of the actual compressed file system) which must be done from outside the system being remastered. This second installation also serves as a backup system and as a test bed for the final remastered compressed file system.
Applicable to Knoppix version 3.3 and 3.4. Earlier version do not have the provision for the required 'Knoppix style' HD installation.

Any computer capable of running Knoppix can be used for remastering provided it has enough free space on the hard drive.

This method of remastering requires approxiamtely 6 Gig space:
HD installed Knoppix min 2.3 Gig
assembled compressed file system 700 Meg
final iso filesystem ready to burn to CD 700 Meg
combined RAM & swap space totalling 1 Gig 500-700Meg
(optional but desireable) 'toHD' Knoppix operating system 700 Meg
Total 6.1 Gig


The 'toHD' installation is not absolutely necessary, the Knoppix CD could be used instead but the 'toHD' install is faster and provides a convenient means of testing the compressed file system simply by replacing the file 'KNOPPIX/KNOPPIX' with the remastered 'KNOPPIX' compressed file system.

For this example we are going to assume we are starting from a HD with some form of Windows on it which will be set up for dual boot (the only difference having the Windows system makes is to change the partition numbers but this is probably a more 'typical' system than a Linux only system). Partition the HD as follows:
hda 1 Windows O/S size as appropriate
hda5 Knoppix HD install min 2.3 Gig, 3-3.5 preferred
hda6 new remastered isos min 1.5 Gig
hda7 shared Windows/Linux data drive, VFAT format, all remaining HD space
hda8 Linux swap approx 500-700 Meg, as needed to total 1Gig RAM + swap

The HD install being remastered must be on a separate partition (hda5), we will use the whole partition as the source for creating the compressed filesystem. The other limitation is that the create compressed filesystem command must be run from a different operating system than the source for the compressed file system. We are going to use the 'toHD' install or Knoppix running from CD for this operation.

The 'toHD' install should preferably be installed on hda1 with a 'persistent home' and configuration file on hda7. If this is not possible (for reasons such as hda1 being a NTFS filetype which would prevent any Knoppix version earlier than 3.4 with the 2.6 kernel from writing to it), the 'toD' install should go on hda7 with no persistent home. The reason is that Knoppix will not mount the partition on which the 'toHD' system is installed. This does not matter if the 'toHD' install is on hda1 as it contains only the Windows operating system; however as we must have write access to hda6 to create the compressed filesystems, the 'toHD' install must go on hda7 if it cannot go on hda1.

Step 1 is to partition the HD as above and install Knoppix as a 'Knoppix style' HD install. I am not going to go into any more detail with this step, it is well documented elsewhere. With Knoppix version 3.3 the install command is 'sudo knoppix-installer', this may change for later versions of Knoppix. Put the bootloader lilo on the MBR, it will be used to boot the 'toHD' install as well.

Step 2 is to install the 'toHD' install, preferrably on hda1. This is done with the boot code 'knoppix tohd=/dev/hda1' at the boot prompt.

In order to boot the 'toHD' install, a directory /boot containing the required files to boot Knoppix must be installed on hda1.

For Knoppix 3.3:
The minimum files are vmlinuz and miniroot.gz which are not directly available on the system, they are buried in /boot.img. For simplicity we are going to add all the files in the knoppix 'boot.img'. The boot.img must first be mounted on a loop device then on an intermediate mount point (use the existing directory /mnt/floppy) from which they can be copied to hda1.

Running from the HDinstall on hda5, first mount hda1 read/write then as root execute the following:
mkdir /mnt/hda1/boot
/sbin/losetup /dev/loop0 /mnt/cdrom/KNOPPIX/boot.img
mount /dev/loop0 /mnt/floppy/
cp /mnt/floppy/* /mnt/hda1/boot

Then edit /etc/lilo.conf adding the following

image=/mnt/hda1/boot/vmlinuz
label="toHDKnoppix"
append="lang=us fromhd=/dev/hda1 home=scan myconfig=scan apm=power-off"
initrd=/mnt/hda1/boot/miniroot.gz
read-only

For Knoppix 3.4

The required files are linux24, linux26, minirt24.gz & minirt26.gz. These are located in the directory /boot/isolinux on the CD and can be directly to hda1 from there. The easiest way is to just use konqueror and drag-drop the entire directory '/mnt/cdrom/isolinux' to /mnt/hda1/boot

Then add the following to lilo.conf

image=/mnt/hda1/boot/isolinux/linux24
label="Knoppix3.4-24"
append="lang=us fromhd=/dev/hda1 home=scan myconfig=scan apm=power-off"
initrd=/mnt/hda1/boot/isolinux/minirt24.gz
read-only

image=/mnt/hda1/boot/isolinux/linux26
label="Knoppix3.4-26"
append="lang=us fromhd=/dev/hda1 home=scan myconfig=scan apm=power-off"
initrd=/mnt/hda1/boot/isolinux/minirt26.gz
read-only
(Only include the home=scan & myconfig=scan if you are using 'persistent home')


Don't forget to run lilo to update the settings before exiting.

Now you can reboot into the 'toHDKnoppix', change the settings to your preferences, save your configuration on hda7 and set up your persistent home also on hda7. Persistent home doesn't have to be very large as this is a 'temporary' installation. It does make using a lot easier as you don't have to keep replacing your desktop setup etc.

Preparing hda6 to receive the remaster:

Make a directory which will contain the assembeled remastered filesystem prior to creating the final iso.
mkdir /mnt/hda6/knx-remaster
Copy everything from the CD except the big KNOPPIX filesystem to this directory

You are now ready to tweak the remaster into your own personal system. Again there is lots of information on customizing and remastering available, I am not going to go into any detail except to add the following tips.

Tip: Use 'dselect update' instead of 'apt-get update'
'dselect update' will keep the package lists in sync.

Tip: Removing large applications:
List large applications: (fromCharan):
If you are looking for big installed packages

# dpkg-awk "Status: .* installed$" -- Package Installed-Size| awk '{print $2}' | egrep -v '^$' | xargs -n2 echo | perl -pe 's/(\S+)\s(\S+)/$2 $1/' | sort -rg

will list the packages with size in descending order.

Tip: Removing applications you don't want:
Make up a 'PurgeList' of the applications to remove then remove them all at once.
example PurgeList:
3270* ace-of* amanda* anacron artwiz-cursor atitvout bonobo* brltty chromium* emacs* enigma euro* falconseye* fetchmail fluxbox fortunes* freeciv*
Note that this is a space-separated list, all on one line and that wildcards are permitted. If you use a CR separated list and apt-get comes to an item it cannot find, it will not continue. If you use dpkg instead of apt-get you cannot use wildcards. OpenOffice can generate a space separated list from a CR separated by saving as filetype .csv

The command for using the PurgeList is
apt-get remove `cat PurgeList`
The ` is the character under the tilde (~) key.

When making up your PurgeList, TEST BEFORE USING. Interwoven dependencies can create some unexpected results...for instance removing aalib* will remove the Gimp and removing libarts1 will nuke all of KDE. For all but the simplest operations, apt-get will ask for confirmation before it does anything but be safe & use 'apt-get -s remove...' first. Read through the lists and make sure of what is going to happen before actually doing it.

Tip: Language support
Internationalization takes up over 5% of the Knoppix filesystem; removing uneeded language support will free up considerable filespace. Install the package 'localepurge' to remove all the language support you don't need. 'Apt-get remove i18n*' will remove all alternate language support files except english (almost 100 Meg). Of course if you need support for another language you will only be able to remove the ones you don't need.

Tip: Accidentally removing knoppix-installer
The hard drive installer package needs the lib file kdelibs4. Removing KDE by the common method of 'apt-get remove libarts1' (which will remove all of KDE) will also remove the knx-installer, so you must either reinstall it after nuking KDE or use a different method of removing KDE packages. Personally I am a KDE user so I keep it but it may still become an issue if you are upgrading KDE.

Tip: Cleaning Up
There is a lot of 'cleaning up' which should be done before creating the final iso. Most of it is optional but two items are necessary:
1. Remove the file /etc/X11/XFConfig-4. It will be regenerated from the script /etc/X11/xsession.d/45xsession to use the correct settings for the hardware found only if there is no existing XFConfig-4 file so it must be removed before creating the remaster.
2. Save your desktop settings & personalizations as /etc/skel, otherwise your remaster will start up with all default settings.

Tip: Cleanup Script
A 'clean-up' script called 'remaster-clean' will be included with this how-to. It is based on the original script by Klaus Knopper but modifications by mzilkazi, myself and others have been incorporated as well as comments to help the neophyte understand what it is doing.
It must be run as root. Copy the script to the /user/sbin directory.
'Sudo remaster-installer' to run.
Copy by using cut & paste but check the script first using a text editor; the most likely source of copy problems is unwanted spaces at the beginning of the line. This may cause problem, in particular with the line
EOT
found in two places in the script. This 'EOT' MUST be at the beginning of the line; ie <space>EOT will not work.


Tip: Removing Window managers:
Boot 'Knoppix 2' if you are planning to remove your window manager; the system gets pretty confused if you remove KDE while running KDE.


Making the compressed file system

This must be run from our 'alternate' operating system (the 'toHD' install):
mkisofs -R -U -V "KNOPPIX.net filesystem" -P "KNOPPIX www.knoppix.net" -hide-rr-moved -cache-inodes -no-bak -pad /mnt/hda5 | nice -5 /usr/bin/create_compressed_fs - 65536 > /mnt/hda6/knx-remaster/KNOPPIX/KNOPPIX

Testing the compressed filesystem
We are going to do this simply by replaing the compressed file system 'KNOPPIX' on our 'toHD' install with the remaster. Save the original first if you want to but it is almost as easy to just reinstall from the CD if you need to restore it.
After copying, boot into your 'toHD' install. You should now see the results of your remastering efforts.
I also use this method instead of making a full CD to transfer the remaster to my laptop which is currently running a 'toHD' system rather than a full HD install due to lack of HD space.

Make the final iso ready for burning to CD:

There is some information on sorting the files prior to making the final iso but I haven't sorted it out yet.....somebody please help????

1.Update the md5 sums:
cd /hda6/knx-remaster/
rm -f KNOPPIX/md5sums
find -type f -not -name md5sums -not -name boot.cat -exec md5sum {} \; >> KNOPPIX/md5sums

2. make iso file to put on CD:

For knoppix 3.3:
mkisofs -pad -l -r -J -v -V "KNOPPIX" -b KNOPPIX/boot.img -c KNOPPIX/boot.cat -hide-rr-moved -o /mnt/hda6/knx-remaster/knoppix.iso /mnt/hda6/knx-remaster/

For knoppix 3.4:
mkisofs -pad -l -r -J -v -V "KNOPPIX" -no-emul-boot -boot-load-size 4 -boot-info-table -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat -hide-rr-moved -o /mnt/hda6/knx-remaster/knoppix.iso /mnt/hda6/knx-remaster


Burn your iso CD:
I just use k3b to do this.

[mzilikazi]

Indeed it is far easier to remaster if you can boot it. Plus desktop customizations become possible that are much more difficult if you're using chroot.

I've never tried to use the persistent home feature during a remaster however. I simply do

mv /home/knoppix /etc/skel

Some other things I've learned......

You can change the default keymep
install-keymap us
dpkg-reconfigure console-common

Change default background:

cp /your/fav/image.jpg /usr/local/lib/knoppix.jpg
Make it the same as your WM background and you have a nice transition
from text mode to desktop.

Debian menus:
update-menus -v

To remove the original KDE menus:
rm -rf /etc/skel/.kde/applnk
rm -rf /usr/share/applnk
(or just go into those directories
and remove all references to KDE apps).

[CrashedAgain]

Indeed it is far easier to remaster if you can boot it. Plus desktop customizations become possible that are much more difficult if you're using chroot.

I've never tried to use the persistent home feature during a remaster however. I simply do

mv /home/knoppix /etc/skel

Some other things I've learned......

You can change the default keymep
install-keymap us
dpkg-reconfigure console-common

Change default background:

cp /your/fav/image.jpg /usr/local/lib/knoppix.jpg
Make it the same as your WM background and you have a nice transition
from text mode to desktop.

Debian menus:
update-menus -v

To remove the original KDE menus:
rm -rf /etc/skel/.kde/applnk
rm -rf /usr/share/applnk
(or just go into those directories
and remove all references to KDE apps).

I don't actually use the persistent home for the remaster, it just makes running the 'toHD' install easier...and the toHD install makes a much more convenient backup system than using the CD. The persistent home would still be active after you copied the KNOPPIX file to the toHD install to test it however, unless you modified lilo so that it booted without it. I suppose it could possibly cause some problems since it would be set up for the original KNOPPIX filesystem.

I tried your method for saving /home/knoppix, I had actually included the code in my remaster.clean script at one point:

  mv /home/knoppix /etc/skel  
  touch /home/knoppix  
  chown knoppix:knoppix /home/knoppix  
  chown -R root:root /etc/skel

but something went wrong & I was left with an empty /home/knoppix & an empty /etc/skel! I think it was unable to write to /etc/skel so it removed /home/knoppix as step 1 of the mv but couldn't write it so it just reported an error & lost it. Then dumb me fixed whatever reason it couldn't do the write & tried it again....this time it worked but now /home/knoppix was empty so it wiped out /etc/skel as well.
That was my last remaster effort with knx3.3...I then tried apt-get upgrade which pretty much slaughtered the system so I started working on 3.4 instead.
I've just tried the cleanup script & a remaster with 3.4, there were some errors is the cleanup script but making the compressed fs did not go well...it ended up 1.1Gig, overfilled the target partition & crashed with 'fatal: could not load /lib/mopdules/2.6.6/modules.dep-no such file' error when I tried to run it. Not sure why, the remaster was downsized from Knoppix original & should have ened up about 550Meg. I don't think anything has changed in the create compressed file system step from knx3.3 to knx3.4. Or has it?

The background menus is a nice tip, I will include it in the final HOWTO. I don't want to include the kde menus info because I use KDE but should include something about removing uneeded /home/knoppix/.hidden-setup-files. There should be more documentation about .desktop & other setup files & how they work but I'm not knowledgable enough to write it.

Can you enlighten me about sorting prior the making the compressed fs? Is it necessary? or desirable?
I see you were using:

ls -U /build/KNOPPIX 
boot.img KNOPPIX 
cd /build

what is this doing? (probably it is now a moot point as 3.4 does not use boot.img).

Thanks

[j.drake]

Personally, I'm not anywhere near ready for any project this ambitious - I haven't even done a HD install yet - but I might some day. When I do, it would be really nice to have this saved on the WIKI for easy reference, so that people won't have to search and sift to find the comprehensive how-to. Would you be willing to post there when you're done?

just my 2 cents.

jd