HOWTO: Upgrading the kernel

News - Fri, 09 Jun 2006
* Finally updated using the semi-latest sources.

Change Log:
* Version 1.4 - Fri, 09 Jun 2006
Compiled using kernel version 2.6.16.20, additional information provided regarding cloop and unionfs compilation (thanks mcreel & hellospencer1).

* Version 1.3 - Sun, 02 Apr 2006
Summarised the solutions within the thread (thanks guys/girls??).

* Version 1.2 - Thu, 01 Dec 2005
Fixed ommission of making the actual kernel (thanks mcreel).

* Version 1.1 - Wed, 21 Sep 2005
Elaborated compilation of cloop and unionfs

* Version 1.0 - Tue, 20 Sep 2005
Initial write

Acknowledgements:
Snippets are taken from the "Knoppix Custom Kernel" wiki
* Daniel R. Tarsky
* Marc Haber
* Rich Ercolani

Contributing posters
* antoine.boutet
* hellospencer1
* mcreel
* meatwad
* tr

Disclaimer:
Always needed cause you just never know how people will interpret things. Please people, use this at your own risk. Whilst this method has and continues to work for me, I will not be held responsible for anything that might go wrong, due to an ommision or typo in this document. I too, like you, am human and do make mistakes.

Preface:
There are many excellent guides for remastering Knoppix on these forums. The better ones, continue to evolve such as UnderScore's at http://www.knoppix.net/forum/viewtopic.php?t=13867. The following should be used in conjuction with a Remastering Howto as it will not cover the requirements for establishing or finishing a remastering sesssion.

This howto is based on the way I replace kernels. I like nothing better than seeing a new kernel release over at kernel.org and upgrading my remaster to the bleeding edge. The document will assume remastering of Knoppix since the inception of UnionFS and the solitary 2.6.x kernel. In time, support for more historic versions may be incorporated.

So you have downloaded your preferred version of Knoppix and there is a new kernel waiting at kernel.org, itching to be installed on your remaster. Taking Knoppix version 3.8.1/2 for example, it utlises the 2.6.11 kernel that has the command line properties patched to allow 512 characters instead of 256. And that's about it, it's exact configuration can be found at /boot/config-2.6.11 inside the compressed filesystem.

The miniroot
Knoppix utilises a special image file known as an Initial RAM Disk (inirtd) image (ie. minirt.gz) for bootup. The minirt.gz file is in fact a minitaure root environment for Knoppix. It has the essential elements of a minimal root filesystem and performs the folling process:
1. Probe the CD/DVD drive,
2. Find the compressed filesystem (on the CD),
3. Mount it,
4. Switch the boot process from the mini root environment to the now available (de)compressed filesystem.

A look inside the minirt.gz file will reveal a number of directories and symlinks. The directories are /cdrom, /etc, /mnt, /modules, /proc and /static. Everything else is symlinked to /KNOPPIX, which is where the compressed system image will be mounted once minirt has does it's job.

Aside from the symlinks, /modules and /static stick out as non-traditional directories. /static is where the special statically-linked ash shell resides, which also doubles as init and modutils. When miniroot is mounted as a boot/root environment, the kernel executes the linuxrc script on the 'root' of miniroot with the static ash-knoppix. Because ash-knoppix (or 'init' in this context) is statically linked with modutils, you may also need to recompile ash-knoppix to upgrade your modutils for any newer kernel versions that may require them. This is of course in addition to installing the new modutils in your Knoppix image source directory.

The only modules in the /modules folder are the cloop module for the system image, scsi modules in case you happen to boot your CD from a SCSI CD-ROM and unionfs for merging the CDROM with the RAMDisk.

As linuxrc executes, it attempts to find the compressed filesystem image (ie KNOPPIX) from the medium you are booting from (eg. CD-ROM, USB, etc). At this point the whole filesystem is still in ram, so the medium must first be mounted to access the compressed filesystem image. Once mounted, a second RAMdisk is created with a dynamic size (percentage of of available and free memory) which will be used for /home and /var directories. Finally, cloop.o is loaded into the kernel and the compressed system image is mounted on /KNOPPIX, making all the symlinks work as part of the root filesystem. Next, more symlinks are created and important system files are migrated to the RAMdisk so they can be written to if need be.

Once linuxrc is finished, it gives control back to the kernel init process which in turn starts running the rc boot scripts, most notably knoppix-autoconfig.

What are we going to replace/upgrade:
OK, the big issue with upgrading the kernel is that people are not always certain of what has to be replaced. If they encounter an error they first assumption is that not everything essential has been replaced. From my experience, in short, the steps are:

For the compressed filesystem (ie chroot environment, /source)
1. Untar linux source
2. Apply any patches
3. Perform a make, make modules and make modules_install
4. Copy bzImage, System.map and update symlinks in boot

For the miniroot image (ie /master)
1. Copy bzImage
2. Update modules with those created when compiling (eg. cloop, unionfs, etc).

In short, that is it ... now for a little elaboration

Compressed Filesystem (chroot environment)
In your chroot environment you need to extract your linux kernel source (eg. version 2.X.YY) to the /usr/src directory. It's always good to point the linux symlink to the working source.
Code:
tar -xjf /path/to/linux-2.X.YY.tar.bz2 -C /usr/src
cd /usr/src
ln -s linux-2.X.YY linux
cd linux
At this point I will apply any patches I want to use in the kernel, for example squashfs, gensplash, etc. Once the kernel is patched to your liking it is time to configure it. To ensure the new kernel is baselined from the Knoppix kernel I use the Knoppix config as the baseline. Taking Knoppix version 3.8.1/2 as an example, and using the default options that popup:
Code:
cp /boot/config-2.6.11 .config
make oldconfig
make menuconfig
make
make modules
make modules_install
If you are unable to 'make oldconfig' or 'make menuconfig' becuase it borks with errors then it is likely that you just need to set up the asm symlink as follows:
Code:
cd include
ln -s asm-i386 asm
cd ..
I don't do a 'make install' cause I find it simpler to just copy what I need.
Code:
cp arch/i386/boot/bzImage /boot/vmlinuz-2.X.YY
cp System.map /boot/System.map-2.X.YY
cp .config /boot/config-2.X.YY
Update the boot symlinks
Code:
cd /boot
rm config System.map vmlinuz
ln -s config-2.X.YY config
ln -s System.map-2.X.YY System.map
ln -s vmlinuz-2.X.YY vmlinuz
At this point you will need to obtain both the cloop and unionfs sources to ensure compatibilty with the new kernel. For this we will be using cloop-2.02-1 and until a stable unionfs 2.X release we will use the snapshot 20060503-0408, but you may use later (bleeding edge ) You can obtain these sources from:

I extract them to the /usr/src/modules directory for clarity.
Code:
tar -xzf /path/to/cloop-source.tar.gz -C /usr/src/modules
tar -xzf /path/to/unionfs-source.tar.gz -C /usr/src/modules
cd /usr/src/modules
Compiling cloop
When compiling cloop, you no longer require the conf.vars file (resides in /usr/src/linux), instead you can simply edit the cloop Makefile to specify the name of your kernel. Using 'vi' or your favourite equivalent text editor, open the Makefile.
Code:
cd cloop-XX.YY
vi Makefile
Edit the line below to read the version of the kernel you are upgrading to (2.X.YY)
Code:
#!/usr/bin/make

KVERSION=2.6.16.16-pk1                              # Edit this to your new kernel version
KERNEL_DIR=/lib/modules/$(KVERSION)/build
Save and close the file and start the compiling process.
Code:
make
The 2.02-X sources should compile with no worries what so ever, however for historic purposes if you come accross the following compilation issue:
cc -I/usr/src/linux/include -I/usr/src/linux/include/asm/mach-default -D__KERNEL__ -DMODULE -fno-builtin -nostdlib -DKBUILD_MODNAME="cloop" -DKBUILD_BASENAME="cloop" compressed_loop.c -c -o compressed_loop.o
compressed_loop.c: In function `clo_read_actor':
compressed_loop.c:205: error: structure has no member named `buf'
compressed_loop.c: In function `clo_read_from_file':
compressed_loop.c:231: error: structure has no member named `buf'
make: *** [compressed_loop.o] Error 1
Chances are you just need to modify, funnily enough as the previous output says, lines 205 and 231 of compressed_loop.c from:
205: struct clo_read_data *p = (struct clo_read_data*)desc->buf;
231: desc.buf = (char*)&cd;
to
205: struct clo_read_data *p = (struct clo_read_data*)desc->arg.buf;
231: desc.arg.buf = (char*)&cd;
With that done cloop should compile just fine and resulting in a cloop.ko module in the current directory.

Compiling unionfs
Now for unionfs, the compilation requires ctags to be installed, so you'll have to apt-get install it if you haven't got it already. Similarly we will edit the Makefile to point to our new kernel version as well as remove the debugging symbols to reduce the size. Again using your favourite text editor.
Code:
cd ../unionfs-snapshot-YYYYMMDD
vi Makefile
Find and edit the lines as shown below.
Code:
# this should point to where your kernel headers are
KVERS=2.6.16.16-pk1                                   # Change this to your new kernel version
MODDIR= /lib/modules/$(KVERS)
LINUXSRC = /lib/modules/$(KVERS)/build
TOPINC   = -I${LINUXSRC}/include
# This is where the make install target will put stuff
PREFIX   = /usr/local
MANDIR   = ${PREFIX}/man
MODPREFIX=
VERSION=20060503-0408
TAROPTS = czvf
COMPEXT = .gz
UNIONFS_DEBUG_CFLAG =                                # Remove the -g if present to remove debuggin symbols
Save and close this file. To aid the compiling process we will also build the fistdev.mk for some additional options.
Code:
vi fistdev.mk
Add the contents shown below to the fistdev.mk file.
Code:
LINUXSRC=/usr/src/linux-2.X.YY
TOPINC=-I$(LINUXSRC)/include
MODDIR=/lib/modules/2.X.YY/build
EXTRACFLAGS=-DUNIONFS_NDEBUG
Save, close and start the compilation process.
Code:
make unionfs2.6
make install
There has been a number of issues with unionfs sources and the 2.6.14.xx and 2.6.15.xx kernels (haven't experienced it in the 2.6.16.xx as yet) with the following error:
unionfs: Unknown symbol __rcuref_hash
FATAL: Error inserting unionfs (/lib/modules/2.6.15/kernel/fs/unionfs/unionfs.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Essentially unionfs source hasn't kept up with the kernel sources and is using a lower level function that is being phased out. A patch, which essentially makes this "Unknown Symbol" available is shown below.
Code:
--- linux-2.6.14.3/kernel/rcupdate.c	2005-11-29 14:53:02.000000000 +1100
+++ linux-2.6.14.x-unionfs/kernel/rcupdate.c	2005-11-29 14:51:39.000000000 +1100
@@ -84,6 +84,8 @@
 spinlock_t __rcuref_hash[RCUREF_HASH_SIZE] = {
 	[0 ... (RCUREF_HASH_SIZE-1)] = SPIN_LOCK_UNLOCKED
 };
+
+EXPORT_SYMBOL(__rcuref_hash);
 #endif
 
 /**
If you find that your module is too big, you may be able to scrape a few KB's off by stripping the module.

With that done you should now have a unionfs.ko to be used at a later stage.

At this point, it's your choice to leave the kernel source intact when building the compressed image, but be warned that it can take a large chunk of space. Now we've done the compressed filesystem, time to update the miniroot image.

Miniroot image (minirt.gz)
Just before we update the minirt.gz image, we need to update the linux kernel in the master directory. Just copy the new vmlinuz-2.X.YY to the master/boot/isolinux directory

If you don't need SCSI support or you don't need to boot from a SCSI CD-ROM, you can safely remove all the scsi modules. If you do so, be sure to comment out the 'SCSIMODULES=' section of linuxrc and replace it with SCSIMODULES="" so the script will skip SCSI autoprobing.

Now what I do is remove the current cloop.o and unionfs.o module from the modules directory. Then copy the newly compiled cloop.ko and unionfs.ko to the modules directory. Lastly a simply modification to linuxrc will see us load the newly named cloop.ko and unionfs.ko modules. Search for any reference to cloop.o (or cloop.*o) and change to cloop.ko. and similarly, search for unionfs.o (or unionfs.*o) and change to unionfs.ko.

That is it

Closing:
This HOWTO is in its infancy and will continue to evolve from any constructive feedback received. For now ... use it, abuse it, and let me know what you think.