Problems remastering the ISO

[job317]

Hi List,

I'm trying to follow the instrux at http://www.knoppix.net/wiki/Knoppix_Remastering_Howto with the KNOPPIX_V6.2.1CD-2010-01-31-EN.iso image. I'm remastering using QEMU where I have a 8GB partition and boot the Knoppix disc as the cdrom.

My goal is to add 4-5 apps to my custom build but not really change anything else. I was able to mount, chroot and apt-get my apps.

AFAICT, the problems begin after I leave chroot and start the mkisofs part. The HowTo has several flavors of this command yet none were working for me. I would end up with a ~/master/KNOPPIX/KNOPPIX compressed file ~8-16MB depending on which variant of the mkisofs command I followed.

My latest build "worked" which gave me a >831MB compressed image and the Live CD ISO build gave me a ~836MB ISO - which would have been fine if it would have booted :( I got the following errors when I tried to boot the new ISO (again using QEMU on Linux):

Knoppix 6 found at: /dev/sr0
>>> Starting in Live-Mode. <<<
>>> Please do not remove mediom until shutdown! <<<
/init: line 693: /bin/mount: not found
/init: exec: line 742: /sbin/init: not found

Is there an updated Remastering Howto around or can someone please assist. I'm unclear on why the Howto model isn't working when (I think) I'm following the instrux pretty close to the letter.

Help please,

Job

[kl522]

Knoppix 6 found at: /dev/sr0
>>> Starting in Live-Mode. <<<
>>> Please do not remove mediom until shutdown! <<<
/init: line 693: /bin/mount: not found
/init: exec: line 742: /sbin/init: not found


This seems to be a very common problem.

Error shows that 'minirt.gz' has problem, likely due to you messed it up in the remaster process.

If you are not changing 'minirt.gz', I recommend you use the original version of 'minirt.gz' from the distribution.

If you are changing 'minirt.gz', you need to do it on a 'proper' file system, ie if you do the 'cpio' thingie on FAT or VFAT, you will mess up the symbolic links on 'minirt.gz'.

[job317]

I did not modify minirt.gz. I added a couple of packages with apt-get and removed a few which were unneeded to save space and yet the image size grew larger - even with cleaning up with deborphan and apt-get clean.

I tried a slight modification which was rather than using my QEMU VM to build it in, I tried remastering by booting the system with the KNOPPIX CD and following the intrux even more closely but with the same results -- a larger ISO size and an image that won't boot. The system has 2 GB of memory so I didn't bother with making the swap image.

I am curious about a few of things:

1.) When I build the compressed image, I have unmounted /proc and left chroot, right? After leaving chroot, the source "/dev" folder is still populated. Is this expected?
2.) How sensitive is the location where the mkisofs commands are executed?
3.) I'm using the create_compressed_fs command provided with the Knoppix 6.2.1 disk yet I saw some comments regarding a newer version. Shouldn't the one that came with the disk be sufficient?
4.) What kind of troubleshooting should I try next? e.g. Is there a test script to run on the KNOPPIX image or the final ISO to test integrity?

Thanks,

Job

[kl522]

1.) When I build the compressed image, I have unmounted /proc and left chroot, right? After leaving chroot, the source "/dev" folder is still populated. Is this expected?
Yes.


2.) How sensitive is the location where the mkisofs commands are executed?The first mkisofs which makes the compress image is not sensitive to location, as it uses absolute pathnames.

The second mkisofs which makes the boot image ***IS*** sentive to location, as you notice it has the -b & -c, both of them using relative pathnames.


3.) I'm using the create_compressed_fs command provided with the Knoppix 6.2.1 disk yet I saw some comments regarding a newer version. Shouldn't the one that came with the disk be sufficient? Should be adequate.


4.) What kind of troubleshooting should I try next? e.g. Is there a test script to run on the KNOPPIX image or the final ISO to test integrity? I thought using qemu is already a very good tool to test. You can try to add cheatcode 'debug' and see if it will display additional information.

By the way, what kind of file system are you building the remaster image on ? Remember VFAT is bad !

[job317]

FWIW, the host system is a 32-bit P4 with Hyperthreading enabled. I went through everything again by booting the system with the original Knoppix 6.2.1 disk and testing the resulting .iso with QEMU.

One thing to note is that there was no .rr_moved file/folder in the source after remastering. The HowTo says to remove it but I didn't have one.

Here is a portion of my log:


root@Microknoppix:/mnt/sda2/root/KNOPPIX# mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net"
-hide-rr-moved -cache-inodes -no-bak -pad knx/source/KNOPPIX
| nice -5 /usr/bin/create_compressed_fs - 65536 > knx/master/KNOPPIX/KNOPPIX
Warning: creating filesystem that does not conform to ISO-9660.
2 processor core(s) detected
Warning, number as file string found. Assuming old command syntax and
choosing compatible parameters (-m -B 65536). See the usage info (-h)
for better/correct parameters.
Block size 65536, expected number of blocks: 2147483647
Ignoring file sm.bak
Ignoring file trusted.gpg~
Ignoring file mplayer.conf~
Using bt8xx000 for ./.rr_moved/bt8xx (bt8xx)
<snip>
99.87% done, estimate finish Thu Nov 4 11:45:23 2010
[ 9] Blk# 32500, [ratio/avg. 46%/ 38%], avg.speed: 2909816 b/s, ETA: 131416s
Total translation table size: 0
Total rockridge attributes bytes: 9654685
Total directory bytes: 28191440
Path table size(bytes): 141180
Max brk space used 47f4000
1041322 extents written (2033 MB)

Statistics:
gzip(0): 0 ( 0%)
gzip(1): 0 ( 0%)
gzip(2): 0 ( 0%)
gzip(3): 0 ( 0%)
gzip(4): 0 ( 0%)
gzip(5): 0 ( 0%)
gzip(6): 0 ( 0%)
gzip(7): 0 ( 0%)
gzip(8): 0 ( 0%)
gzip(9): 32542 (1e+02%)
7zip: 0 ( 0%)
Writing index for 32542 block(s)...
Writing compressed data...

root@Microknoppix:/mnt/sda2/root/KNOPPIX# ls -l knx/master/KNOPPIX/KNOPPIX
-rw-r--r-- 1 root root 824974599 Nov 4 11:45 knx/master/KNOPPIX/KNOPPIX

root@Microknoppix:/mnt/sda2/root/KNOPPIX# md5sum /mnt-system/boot/isolinux/minirt.gz /mnt/sda2/root/KNOPPIX/knx/master/boot/isolinux/minirt.gz
8dac243b731c1d32036874b31a2776ac /mnt-system/boot/isolinux/minirt.gz
8dac243b731c1d32036874b31a2776ac /mnt/sda2/root/KNOPPIX/knx/master/boot/isolinux/minirt.gz

root@Microknoppix:/mnt/sda2/root/KNOPPIX/knx# cd knx/master

root@Microknoppix:/mnt/sda2/root/KNOPPIX/knx/master# find -type f -not -name sha1sums -not -name boot.cat -not -name isolinux.bin -exec sha1sum '{}' \; > KNOPPIX/sha1sums

root@Microknoppix:/mnt/sda2/root/KNOPPIX/knx/master# mkisofs -pad -l -r -J -v -V "KNOPPIX" -no-emul-boot -boot-load-size 4 -boot-info-table -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat -hide-rr-moved -o /mnt/sda2/root/KNOPPIX/knoppix.iso /mnt/sda2/root/KNOPPIX/knx/master
genisoimage 1.1.10 (Linux)
Scanning /mnt/sda2/root/KNOPPIX/knx/master
Scanning /mnt/sda2/root/KNOPPIX/knx/master/KNOPPIX
Scanning /mnt/sda2/root/KNOPPIX/knx/master/KNOPPIX/modules
Scanning /mnt/sda2/root/KNOPPIX/knx/master/boot
Scanning /mnt/sda2/root/KNOPPIX/knx/master/boot/isolinux
Excluded by match: /mnt/sda2/root/KNOPPIX/knx/master/boot/isolinux/boot.cat
Writing: Initial Padblock Start Block 0
Done with: Initial Padblock Block(s) 16
<snip>
99.68% done, estimate finish Thu Nov 4 12:00:33 2010
Total translation table size: 2048
Total rockridge attributes bytes: 3835
Total directory bytes: 10240
Path table size(bytes): 70
Done with: The File(s) Block(s) 406124
Writing: Ending Padblock Start Block 406165
Done with: Ending Padblock Block(s) 150
Max brk space used 1b000
406315 extents written (793 MB)

root@Microknoppix:/mnt/sda2/root/KNOPPIX/knx/master# ls -l /mnt/sda2/root/KNOPPIX/knoppix.iso
-rw-r--r-- 1 root root 832133120 Nov 4 12:00 /mnt/sda2/root/KNOPPIX/knoppix.isoBooting the new image with QEMU results in a loop of error messages after the
KNOPPIX 6.2 splash screen:


boot:
Could not find kernel image: knoppix

So the image is larger than I started with and I only added two packages and removed over a dozen.

Thanks,

Job

[kl522]

boot:
Could not find kernel image: knoppix
This error says it could not find the linux kernel file, this file as described by boot/isolinux/isolinux.cfg to have a file name called 'linux' resided also in the same directory ( ie boot/isolinux/linux ). Please check if you have as many files as the original boot/isolinux directory ( isolinux.cfg, linux, minirt.gz and so on ).

[job317]

The only files changed in my master build (after mkisofs) are:

/boot/isolinux/boot.cat
/boot/isolinux/boot.msg
/KNOPPIX/KNOPPIX
/KNOPPIX/sha1sums

I know that the KNOPPIX compressed image is the big one but the other files are intact.

I did a diff between my source folder after modifications I made (e.g. apt-get xxx) and my source is ~100MB larger after my modifications. Looks like some of the packages I removed actually added some stuff.

So, is there a size limit for the KNOPPIX compressed image?

Also, to your earlier question, the filesystem I'm working on is a EXT2/3.

Thanks,

Job

[kl522]

You are not answering to what the errors are pointing you to. You have not even come to the part about your compressed image. You have to understand the booting sequence :-

BIOS--> bootloader ( isolinux) ---> Kernel Image ( linux ) --> minirt.gz ---> Compressed Image ---> ( others ).

Based on the last error, the isolinux could not find the kernel image. You could check if the kernel image is contained in your knoppix.iso by doing a mount :-

# mkdir /tmp/tmp
# mount -o loop /(whatever path )/knoppix.iso /tmp/tmp
# ls /tmp/tmp/boot/isolinux
( make sure isolinux.bin, isolinux.cfg, linux, minirt.gz are there )

There are some files missing, or not copied from the original directory.

Cheers.

[kl522]

BTW, Knoppix has a bug with loop mount, if you trying to loop mount and it complains about no more free loop devices, you can do this to fix the problem :-

# rm -rf /dev/loop

( This has been mentioned many times in the forum )

[job317]

This morning I performed everything from scratch minus modifications to my source files. I simply started in a new folder and copied all the source and master files and generated the new compressed image and ISO - no chroot, etc.

The result was an expected CD-sized .iso (754989056 bytes) that has a boot error when booting with QEMU. But first, here is the post-mkisofs info you requested:


[root@test /root]# ls -l /knx/knoppix.iso
-rw-r--r-- 1 root root 754989056 2010-11-05 05:42 /knx/knoppix.iso
[root@test /root]# mount -o ro,loop /knx/knoppix.iso /mnt/image
[root@test /root]# cd /mnt/image

[root@test /mnt/image]# ls -la
total 52
dr-xr-xr-x 4 root root 2048 2010-11-05 04:33 .
drwxr-xr-x 10 root root 4096 2009-01-14 15:47 ..
-r-xr-xr-x 1 root root 54 2001-06-17 04:50 autorun.bat
-r-xr-xr-x 1 root root 45 2003-02-22 22:23 autorun.inf
-r-xr-xr-x 1 root root 967 2004-05-01 16:35 autorun.pif
dr-xr-xr-x 3 root root 2048 2009-07-13 06:46 boot
-r-xr-xr-x 1 root root 32606 2006-02-21 18:33 cdrom.ico
-r-xr-xr-x 1 root root 1744 2010-02-07 22:50 index.html
dr-xr-xr-x 3 root root 4096 2010-11-05 05:40 KNOPPIX

[root@test /mnt/image]# md5sum *
f3086b686d99c6681718d1a49faab5bb autorun.bat
5de9c74859eae949e7ce1d8492a1a3cf autorun.inf
0a5d116ce3058adfceb3407577e42d82 autorun.pif
md5sum: boot: Is a directory
9cd1c5812a3d2758585afd6cf9212855 cdrom.ico
2499c99253f14a4723e1435e14322e3a index.html
md5sum: KNOPPIX: Is a directory

[root@test /mnt/image]# ls -l boot/isolinux/
total 6010
-r--r--r-- 1 root root 1474560 2009-02-07 12:53 balder.img
-r--r--r-- 1 root root 2048 2010-11-05 05:41 boot.cat
-r--r--r-- 1 root root 93 2010-02-08 01:48 boot.msg
-r--r--r-- 1 root root 1255 2010-02-08 03:37 f2
-r--r--r-- 1 root root 1324 2010-02-08 03:37 f3
-r--r--r-- 1 root root 256 2008-07-15 05:50 german.kbd
-r--r--r-- 1 root root 12363 2010-11-05 04:53 isolinux.bin
-r--r--r-- 1 root root 1434 2010-02-08 03:37 isolinux.cfg
-r-xr-xr-x 1 root root 3545232 2010-01-28 11:46 linux
-r--r--r-- 1 root root 18050 2009-08-09 20:58 logo.16
-r--r--r-- 1 root root 20052 2009-02-07 12:54 memdisk
-r--r--r-- 1 root root 124648 2009-07-24 19:48 memtest
-r--r--r-- 1 root root 949265 2010-01-28 16:55 minirt.gz

[root@test /mnt/image]# md5sum boot/isolinux/*
4a67138437af105de83685abcf5e66b5 boot/isolinux/balder.img
981895696f200ab735161d58968a7171 boot/isolinux/boot.cat
e7d159a554abb281ff3697f2a43d0aa7 boot/isolinux/boot.msg
6e7a5ddb5bdc0420b97477f42a6611ff boot/isolinux/f2
9d9becdc141ee80d22cdfedcfa5e8d6d boot/isolinux/f3
9b758201ff56879eacca2dc7857af533 boot/isolinux/german.kbd
07c9b578ad1bffcc32cfcb164e056c9d boot/isolinux/isolinux.bin
bc8e7437818da0e2480937ef850fa916 boot/isolinux/isolinux.cfg
838023c9a7c3cc56be1d187cdbef6e9c boot/isolinux/linux
917d9875d67bfb84ea1f5aac3dbf714a boot/isolinux/logo.16
ad9c162faca33d30418760100fbb605c boot/isolinux/memdisk
70e35fe7cf0f286714755c4785f7935b boot/isolinux/memtest
8dac243b731c1d32036874b31a2776ac boot/isolinux/minirt.gz
Still when I boot the .iso I get the KNOPPIX splashscreen and the message:


boot:
Could not find kernel image: knoppixRepeat same steps + expect different outcome = insanity?

Job

[kl522]

That's weird. At this stage I could only guess that your 'mkisofs' somehow is not encoding things for 'isolinux' to read 'isolinux.cfg' correctly, ie even though isolinux.cfg is there and present, it is not able to read it or it is not reading it correctly. This is likely related to locale issues ( keyboard, language and so on ). For that matter, you are unable to produce a bootable CD using 'mkisofs'.

The thing to note is that now your current error message is actual one step backward from before. Before this, you were having problem error message from 'init', that at least, it means isolinux is able to boot 'linux', and 'linux' is able to load minirt.gz.

[klaus2008]

So the image is larger than I started with and I only added two packages and removed over a dozen.
I think that the size of the compressed image file is larger than the old one although you deleted some packages because you do not use the best compression method. You always use zlib's best compression method but 7zip does a better job in most cases. Therefore I would suggest that you try the option -L -1 to get a smaller compressed image file. There is also the option -b which instructs create_compressed_fs to try all of the compression methods available and keep the best one but it needs much more time. You can get a list of the available options if you issue create_compressed_fs -h.


root@Microknoppix:/home/knoppix# create_compressed_fs -h
1 processor core(s) detected
Usage: advfs [options] INFILE OUTFILE [HOSTS...]
Options:
-b Try all and choose the best compression method, see -L
-B N Set the block size to N
-m Use memory for temporary data storage (NOT recommended)
-r Reuse output file as temporary file (NOT recommended)
-p M Set a default value for port number to M
-l Listening mode (as remote node)
-t T Total number of compressing threads
-s Q Expect data with size Q from the input, see below
-f S Temporary file S (or see -r)
-q Don't print periodic status messages to the console
-v Verbose mode, print extra statistics
-h Help of the program
-S X Experimental option: store volume header in file X, see manpage
Performance tuning options:
-a U Job pool size (default: threadcount+3)
-L V Compression level (-2..9); 9: zlib's best (default setting), 0: none,
-1: 7zip, -2: do all and keep the best one

To use standard input/output - can be used as INFILE/OUTFILE. However, this will
require additional memory (or diskspace) for data size calculation and header
update (after compression). Passing the INPUT data size with -s may help.
The size numbers can be declared with a suffix which acts as multiplier
(K: KiB, k: KB, i: iso9660 block (2KiB); M: MiB; m: MB; G: GiB; g: GB).

So, is there a size limit for the KNOPPIX compressed image?
The size limit for the compressed image is 4 GB because you cannot create a larger file on a device that uses the FAT32 filesystem otherwise you are not able to use the flash installation utility.

[kl522]

boot:
Could not find kernel image: knoppixAnother clue to the problem is that, the isolinux is looking for 'knoppix' as the kernel image; whereas a "conventional" knoppix kernel image is 'linux'. Here it seems to suggest a malformed isolinux.cfg.

[job317]

To fix the insanity aspect (repeating same steps and expecting different outcome), this morning I did a test build on a different system and it worked fine. My resulting ISO booted with no errors.

So, there is something different about the original build system which is functioning fine otherwise.

I will continue with my customized, remastered Knoppix disk on the different system now. It may be helpful to troubleshoot the issue should I find the time but for now I will continue my work and be done.

Thanks for all your attempts to assist.

Regards,

Job