zisofs instead of cloop on a remaster

[bizarro]

I remastered a knoppix cd to act as a windows rescue cd for our group of about 200 tech specialists. The cd is individualized by each tech. They visit a website, enter in their info (username, password, etc) and submit for an ISO image. The server then takes that info and incorporates it into the image including an expiration date. I have been able to do this fairly quickly via ZISOFS on a debian live-cd based off of Timos RescueCD. Recently the co. changed hardware platforms and now I must rebuild a new rescue cd incorporating the new hardware. I found Knoppix to handle this very easily and I have been able to setup a beta of this but I see that using CLOOP takes entirely too long.

Is there a writeup somewhere on how to setup a Knoppix remaster using zisofs? Using cloop for each and every tech will take too long for 3 servers to do.

thanx!

[bfree]

Could you simply use a config script on the cd's, which is outside of the cloop, to make the customisations? Even better (ok this is a lot to ask but perhaps you would be able to do it) you could default to taking the config from a usb key so that they can use the stock iso and tweak the customisations themselves without having to give you the usernames and password (you could probably even customise it to prompt them for the values for the config to do this if it finds a usb key without a config setup, so they never have to give you any details). You could also setup the remaster to automatically look for a persistent home (on the same usb key) so they/you can klik in extra apps and carry around useful/important/sensitive files (I'm sure you can use encrypted persistent homes).

Perhaps you have requirements that can't be met by this though?

[mzilikazi]

I've been doing remasters of Kantoix (http://kanotix.com) w/ squashfs instead of cloop. Not only does it give about 5% better compression it also compresses faster. MUCH faster. On an XP2200 w/ 1G RAM it takes about 15 minutes to compress down to 700MB. It is possible to remaster Knoppix w/ squashfs (I have no idea how) but the Kanotix kernel already supports squashfs. Something to note. The Kanotix kernel is not smp enabled.

I personally would be very interested to know exactly how you've set this up for your techs to build their own .iso's if you'd not mind posting some details.

[bizarro]

I'll try to give as much detail as I can...but it would be best if you asked questions...

The idea started when our company was about to pay over $150k for an enterpise license of ERD Commander. I asked for the company to give me a few months to develop something similar.

Last summer I sat down and started...I compiled a kernel that had all the drivers that we use on our hardware, then added in the utilities that would be useful in a Windows environment...examples are...

Samba

NTFS Tools (allows for reading a corrupted NTFS drive even better than windows itself!)

CHNTPW (NT/W2K/XP local password reseter)

NTFS Fix (forces a scandisk on next boot)

MemTest and Hard Drive Fitness Tester

Our own in house Universal boot disk to allow for ghosting

And a flurry of other utils all bundled in a GUI of IceWM with a windows theme.

The techs in our co. as knowledgable as they are do not know Linux as well as I do so if they downloaded any other RCD like Knoppix they would be lost :( So I have to customize it to make it easier for them...I setup scripts to backup files to a network share, scripts that helps them reset passwords, scripts that would fix corrupted filesystems...etc...it has been working very well except for the new equipment we recently purchased...so back to the drawing board for me. I tried working with the new Timos but I figured that Knoppix is so much easier in respect to hardware detection that I want to build the next one based on this. Well after a few days I've gotten the ISO image down to 500mb after cloop (still too big) but I have it now with newer features and driver support...newer features include Captive-NTFS and QTParted :)

Now for the distribution end of it. Basically we have a website that requires everyone to authenticate in. Once authenticated the server knows your username and email handle. The tech then enters in a password they would use on the webpage as well as default network maps they use to backup client data or reimage machines using Ghost. This information is entered into a text file that is then sent to a roundrobin DNS name landing on 1 of 3 servers in the country. The server then via a cron job every minute checks a specified directory for this text file...once detected the file is then parsed for the information inside then through bash scripting entered into the source cd before compilation. Then ISO compilation begins and meanwhile an email is sent to the client that the process begun and will recieve another email once completed. This process takes about 10 mins right now and once done the server then emails the client an ftp link to the ISO image, enters in an "at" entry to email the client in 5 months of expiration and another "at" entry to remove the lock and another "at" to delete the ISO image after 3 days.

The last part where the CD expires in 6 months was setup so the techs would always have a new CD and so tech can't maliciously use the CD upon release if it were to happen. The end ISO image is 150mb which is my current goal with the Knoppix image...

I did down the Kanotix CD but its missing some utils that Knoppix already has built in....Can I move the Kanotix kernel over to the Knoppix ISO and it work? I would like to try this Squashfs if its faster...I also have to work on shrinking down the image as well...

Oh! Another feature in the website is the ability to authenticate in and have the server email you your password...in case you forget it :)

Any questions ask away on the process...and can someone please help me in what I can remove safely to shrink it down further? And how to get squashfs working? THANKS!

[bfree]

I'll try to give as much detail as I can...but it would be best if you asked questions...
...once detected the file is then parsed for the information inside then through bash scripting entered into the source cd before compilation.
And here lies my question, is there any reason you cannot simply leave the cloop identical for all your users, and simply make the config script on the cd unique for each user, therefore speeding up the process even further? What is being "entered into the source cd" that needs to be "compiled" (you could simply echo lines onto /etc/passwd and /etc/group from the config script for example)?

[bizarro]

the information being entered into the cd are as follows:

username
password
account expiration of 6 months
md5sum formula to change the root password monthly
tech's name and human resouces ID (HRID) entered into the bootscreen
RCD version number into the bootscreen
Compile date and expiration date into bootscreen
Other boot options lime memtest, hdd fitness, universal bootdisk, console version of RCD (non-GUI) for those machines that have low mem or incompatible vid drivers.
Custom drive mappings for backing up client data

Thats the bulk of it.

I still want to make the cd smaller...1.2gb uncompressed and 500mb compress is still way too big. These ISO images are downloaded across the country through our WAN and so 500mb is way too big for that.

Any ideas on making it smaller? I tried Damn Small Linux, it turns out to be a stable release not having some of the tools needed like QTParted and can't install it even if i changed it to testing release. With Kanotix...that release seems BIGGER than Knoppix! I ran out of disk space copying it to the hdd.

BTW, I'm doing all of this via VMWare which makes it much much easier to remaster Knoppix and experiment with all of this.

Does anyone know of an exact step by step procedure on compiling in squashfs? Kernel config, Kernel source w/ patches of Knoppix etc...etc...

[bfree]

Of those items you mention above, I guess only username, password, bootscreen changes and custom drive mappings are actually unique to each individual iso image? The rest (boot options and root password changer) presumably aren't changes made individually on a per cd basis? Those changes can be made to a Knoppix-alike remaster without having to remake the cloop iso, from the config script ammend passwd/shadow/group/fstab (using perhaps echo "">/etc/fstab, or sed). The boot image is also held outside of the cloop so you can create it by whatever means you want, and drop it into the new image. So all you have to do to create the image is a plain mkiso which takes 100th/1000th the time of making the cloop image.

[bizarro]

thanx for the info bfree...i appreciate your help in this...i've managed to get the cd down to 330mb now and compilation is down to 12 minutes...i'd still like to make it much smaller though...any tips?

also, i'll have to write a whole new bash script on rcd requests...but at least i'm ready for GUI customization and beta testing on our environment...is there a howto on entering that info into the config script?

thanx!

[probono]

Really, you don't want to recompile every copy. You want to have the config data outside of the compressed fs. Or, since your data is under 400mb, you could make the whole CD uncompressed.

Btw, I suggested something like what you are doing in the past in order to make customized versions of Knoppix (with the user being able to select on a webpage which applications his Knoppix CD should contain). So I would be really interested in your server-side scripts. Which languages do you use? A combination of PHP and bash seems a good choice.

Would you be willing to release your server part under GPL? Then probably people would be much more willing to improve it and optimize everything...

[bizarro]

Would you be willing to release your server part under GPL? Then probably people would be much more willing to improve it and optimize everything...

CLICK HERE! (http://www.knoppix.net/forum/viewtopic.php?p=61424#61424) :D [/url]

[bluefoxicy]

zisofs may be a better solution anyway. The compression may range a bit lower (or higher, depending; it's higher than squashfs for binaries IIRC, but not for /etc), but it has its advantages:

- Contained files are individually compressed. Damage to an area containing an app you don't use is ignorable, whereas squashfs and cloop may skip and have major problems in surrounding data
- Some machines appear to (in my experience) have trouble when the CD spins down and they try to access a loopback FS on it; if the drive takes too long to spin back up (as mine does) the cloop can begin to bitch about not finding the media. I don't see why; the only flaws that cause this would be elementary and i'd think nobody could screw this up.
- zisofs based livecds can be updated and customized versions can be distributed using jigdo (http://atterer.net/jigdo) to prevent redownloading entire images.
- Some space can be saved by crafting the CD so that the data outside the knoppix uses the same binaries as inside it (symlinks). This is not possible if those overlapping binaries are inside a squashfs or cloop.

I personally like zisofs. Porting it to the LZMA redundancy eliminating algorithm may be useful (what 7zip uses) and may allow more Knoppix data to be squashed onto a CD.