Remastered Knoppix 5.0.1 - config changes only - too big

[sparkyjf]

Hi all,

Last year I remastered Knoppix 4.0.2 to add some apps needed to set up a demo, and it was a great success. This year I have something slightly different I need to do. The long and short of it is that I need to modify Knoppix 5.0.1 (CD edition), but all I need to do initially is make some changes to the config files on the CD.

I've been following the procedure on the wiki page at http://www.knoppix.net/wiki/Knoppix_Remastering_Howto

It all seems fairly straightforward - as I'm happy with the config changes I'm making, I'm not even performing a chroot - basically I've just copied out the contents of /KNOPPIX, changed the files I need, then attempted to rebuild.

However when I rebuild the CD, it's too big :( and won't fit on a CD any more.

The first thing I've noticed is that when I copy /KNOPPIX out to my hard disk (ext3 parition mounted as /mnt/sda1), du reports it is bigger than the original:

root@3[source]# du -sb /mnt/sda1/source/KNOPPIX/
1975775659 /mnt/sda1/source/KNOPPIX/
root@4[KNOPPIX]# du -sb /KNOPPIX/
1967972624 /KNOPPIX/

So straight off my copy of the KNOPPIX file contents is 10MB bigger than the mounted copy at /KNOPPIX. The config file edits add virtually no size to the data. Now when I rebuild the KNOPPIX file it comes out as 723MB. The original file from the CD is reported as 693MB, so I've gained 30MB without adding any files. (sizes reported using ls -lh)

I ran create_compressed_fs overnight with the -b option in case that was responsible, but after 10 hours of processing it came to 722MB.

I've read some posts on the forum saying that this is quite a common problem, and although I think I understand the reasons behind these posts I haven't chroot'ed into my copy of the KNOPPIX data, and haven't mounted anything under it.

Here's my build command for your reference (it's a copy and paste from the Wiki):


mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
-hide-rr-moved -cache-inodes -no-bak -pad /mnt/hda1/knx/source/KNOPPIX \
| nice -5 /usr/bin/create_compressed_fs - 65536 > /mnt/hda1/knx/master/KNOPPIX/KNOPPIX


I must be missing something, or doing something wrong - please can someone help me with this? I'm so close to home now - I've checked the ISO out in VMWare and it works as I want it to - it's just bigger than a CD which is a real problem for the application I have in mind - DVD is not an option.

Thanks in advance for your help,

sparky

[ddawson]

However when I rebuild the CD, it's too big :( and won't fit on a CD any more.

I ran into this problem myself. After looking around for a while, I've concluded that the problem is hard links are being lost somewhere between Klaus' system and the running Knoppix system.[1] This is because there is a problem associated with using an ISO-9660 filesystem if you want hard links: from what I gather, although proper support for hard links is included in the Rock Ridge standard, it's not well implemented.[2]

One solution, which I've tested (though I haven't tried booting the result), is to find sets of identical files[3] before you remaster, then replace all but one in each set by hard links to that one. One should be careful here, since some of these sets may have different owners, permissions, etc. They also probably have different date stamps, though I wouldn't worry too much about those. (Files that differ in any of these properties must not have been hard-linked before mastering.) So basically, if you search for files that are identical and have the same owner, group, permissions, and whatever else is important to preserve, delete the duplicates and replace them with hard links, that would allow mkisofs to assign the same ISO-9660 files to those links. Note that I don't know how wise this is, so try it at your own risk.

Another thing. I tried this with unionfs, but it didn't work that way. There seems to be a bug in its inode number assignment (at least in 1.3+debian-4 -- haven't tried with the one on Knoppix itself): the linking would start off OK, but at some point, files that had already been processed mysteriously had their numbers changed! No idea why that's happening. After puzzling over it, I finally gave up and worked with a real copy.

Anyway, after doing this, I compared the uncompressed result against the cloop decompressed from the official version: mine was about 5 MiB smaller. To be fair, I haven't actually gotten around to ruling out files with different owners, permissions, etc., so the legitimate savings is probably less. Still, I think this proves my point about lost hard links inflating things a bit. In fact, the size should come out about the same if one manages to restore all the hard links and only the hard links.

I suppose I could share the scripts I use, but they're not very polished at this point (not to mention not quite correct). If anyone is interested, though...

Later,
Daniel

Notes:
[1] See, for instance, http://www.knoppix.net/wiki/Trivial_remaster_3.8.2#Notes_3.
[2] See http://www.opensolaris.org/jive/thread.jspa?messageID=57112.
[3] The way I did this (i.e. without comparing everything to everything else in the 2 GiB of data) was to first hash all of the regular files with md5sum, sort the list by the hash values, filter out singletons, then verify what was left.

EDIT: In light of the difficulties with ISO/Rock Ridge, why not use something else? Are there better alternatives? What about, say, squashfs? (Not that I'm familiar with it; but I've seen it mentioned a few times.) Of course, we wouldn't want anything too CPU- or memory-intensive, in order to preserve Knoppix on older systems. cromfs looks nice (superior compression), but would require too much memory.