Success making bootable and persistent USB key with SYSLINUX [Archive] - Download Knoppix Live CD or DVD, Get Documentation and Help

View Full Version : Success making bootable and persistent USB key with SYSLINUX

Krellan2

03-16-2006, 12:24 AM

I've had success installing an entire Knoppix CD to a USB thumbdrive. Also was able to create the Knoppix persistent image, with encryption, on the same USB thumbdrive. All told, it fits perfectly on a 1GB thumbdrive, a fairly common size these days.

750MB for Knoppix's 700MB CD (plus overhead)
100MB for the encrypted persistent image
150MB leftover for unimportant file storage, and use on Windows systems

After checking this page:
http://www.knoppix.net/wiki/USB_Based_FAQ

I got the thumbdrive to boot with GRUB, as it said. All went well... until I went to another computer. It had more SCSI drives, and so the drive numbers had changed! GRUB's awkward (hd0,0) syntax could no longer find the drive, and GRUB failed to boot (didn't even get as far as the GRUB command line).

So, GRUB was out.

KNOPPIX itself uses ISOLINUX, part of the SYSLINUX suite of bootloaers, so why not just use SYSLINUX? Turns out, it worked great! GRUB also doesn't give you the pretty "speedometer" loading screen, which SYSLINUX does (since it's essentially a twin of ISOLINUX, it boots up in the same style). I'm posting this from Knoppix now. :)

Some rough steps (I'll write this up more, if there's interest, perhaps make a new Wiki page or something):

1) Partition and format the ENTIRE USB thumbdrive as one big FAT32 partition. Mark it as active. If you have "cfdisk", use its Maximize feature to squeeze out an extra 0.5MB of space "for free". No need to make more than one partition. Copy SYSLINUX's "mbr.bin" file over your existing MBR, to make it bootable. Don't mount the new partition just yet.

2) Run SYSLINUX on that new partition. The default SYSLINUX options are OK. Now, it's OK to mount the new partition. You should see the file "ldlinux.sys" there.

3) Mount the Knoppix ISO. Copy its entire contents to your USB thumbdrive! Make sure the directories line up, so that all things are in the same place on the USB thumbdrive as they are on the CD.

4) Unmount the Knoppix ISO, you won't need it anymore. On your USB thumbdrive, go to the directory /boot/isolinux. Move ALL files in this directory to the top level of the drive (where the existing autorun files already are). As they are now empty, you may completely remove the /boot/isolinux and /boot directories.

5) Rename the file isolinux.cfg to syslinux.cfg.

6) You're ready to boot! If you have a newer USB 2.0 thumbdrive, though, you've got some more work ahead of you. Now comes the hard part. You'll have to edit the linuxrc file inside of the minirt.gz file, to fix a Knoppix bug where it will fail to load the USB module at startup. Search other forum posts here for how to do this :)

7) Reboot, trying to boot from USB. Did it work? Cool!

8) Make sure Knoppix works. Try to create a persistent Knoppix disk image, using the tool under the penguin menu. When it asks you to choose a partition, choose your USB thumbdrive. If it complains about it being unable to mount because it's already mounted, do this:

mount -o remount,rw /cdrom
mount --bind /cdrom /mnt/sdz1
(replace sdz1 with the true device name of your USB thumbdrive's partition, of course)

Now, try again, and it should let you make the persistent image.

9) As an optimization, try editing the syslinux.cfg file. I like reducing the timeout in the 3rd line (to 30 from 300) so that it boots automatically without having to wait so long. Also, I like adding these options to the 2nd line:

noeject noprompt home=scan dma noswap

No point trying to "eject the CD and close drive door" when it's a USB thumbdrive! The other cheatcodes are for autoloading your persistent image next time, DMA speed increase, and security fix (never use swap, because it's not encrypted).

10) Reboot again. It should ask you for your password, for the encryption, and then load up just fine.

Nice!

If you saw an earlier post of mine, I tried EXTLINUX at first (not SYSLINUX). Unfortunately, that was no good, because Knoppix is more difficult to boot on case-sensitive filesystems (it will have trouble finding certain things as it loads). Also, EXTLINUX can't be easily used on Windows, but FAT32 can. Having one big partition makes a much better use of space on the thumbdrive.

If anyone else has had success doing something like this, feel free to post your tips here as well.

Josh

zoltanthegypsy

03-16-2006, 01:37 AM

Krellan2

03-16-2006, 03:17 AM

Intesting. I just did this too. Remastered KNOPPIX to shrink it a bit and added chntpw and captive-ntfs and my own KDE menus, boot splashes, and artwork and put it on a 1G fob booted w/grub. Pretty slick.

I haven't tried it on a system w/scsi drives, but I'm a little surprised it wouldn't boot for you. Did you still have the device.map file in /boot/grub? If so I wonder if that's what is confusing things.

I didn't use GRUB. For something standalone like a USB key, SYSLINUX seems like a better choice, since it doesn't require a device map at all. SYSLINUX is entirely self-contained and just "sees" the media it's booting off of, and no others. This is bad for a multibooting desktop machine, but very good for removable/rearrangeable things like USB keys, since there's no more problems with devices appearing in different places on various computers.

As far as remastering goes, I'm waiting until the official release of 5.0. Even though I have the DVD image already, I'm hoping they have a CD-sized image, so I can easily fit it into the USB key.

zoltanthegypsy

03-16-2006, 03:38 PM

I didn't use GRUB. For something standalone like a USB key, SYSLINUX seems like a better choice, since it doesn't require a device map at all.

Grub doesn't need device map either, unless there is some sort of confusion between BIOS drive mapping and OS drive mapping, IIRC.

My questions was whether grub would have worked for you if you hadn't included device.map when you installed it /booted from it. W/out device.map, the drive booted from (the flash fob) should just be drive 0 in menu.lst entries regardless of other drives on the box.

I don't mean to argue the merits of SYSLSINUX vs GRUB, just wondering why grub didn't work well for you. It has certainly worked OK for me so far.

Z.[/quote]

Krellan2

03-17-2006, 03:48 AM

Grub doesn't need device map either, unless there is some sort of confusion between BIOS drive mapping and OS drive mapping, IIRC.

My questions was whether grub would have worked for you if you hadn't included device.map when you installed it /booted from it. W/out device.map, the drive booted from (the flash fob) should just be drive 0 in menu.lst entries regardless of other drives on the box.

I don't mean to argue the merits of SYSLSINUX vs GRUB, just wondering why grub didn't work well for you. It has certainly worked OK for me so far.

Z.[/quote]

That's cool, I didn't know that grub did not require the device map. It had failed for me in the past when I tried it. SYSLINUX also is handy for Knoppix since Knoppix uses ISOLINUX already. Maybe I'll give GRUB another chance someday.

zoltanthegypsy

03-17-2006, 04:18 PM

Maybe I'll give GRUB another chance someday.

If/when you do, please let us know what you find. I'm also looking forward to your full SYSLINUX write-up.

My neighbor does a lot of winXX rescue/recovery - now _there's_ a growth industry :wink: - and I'm trying to put together a set of flash drives that boot on as many machines as possible.

Along with the grub/syslinux thing, it also seems to matter (to some boxen) just how the fob gets formatted in the first place. I've put up a page describing my struggles:

http://www.beezmo.com/FloobyDustDir/FDKnoppixUsb.htm

No doubt there are errors/omissions. I welcome feedback. Be gentle :)

Thanks again for your SYSLINUX info.

Z.

Krellan2

03-18-2006, 10:55 AM

If/when you do, please let us know what you find. I'm also looking forward to your full SYSLINUX write-up.

Thanks. I've gotten busier recently, so that writeup might have to wait a while.

However, I checked that link on your page, and what I did is very, very similar to what is here:

http://rz-obrian.rz.uni-karlsruhe.de/knoppix-usb/

It looks like he used an older version of Knoppix. The USB scripting has changed since then. I wonder if the sleep is still needed. Sometimes I get failed boots, where it just hangs after Knoppix has found the USB device. It starts to boot, Knoppix loads up, it loads the USB drivers, and then it hangs. Lots of SCSI errors in the log. Most of the time, though, it works. Maybe my USB key is flaky: I keep it on a keychain (like a real key :) and it's been bumped around a few times.

I also didn't know about the HP utility, and just used Linux cfdisk and mkdosfs tools to format the USB key. What is the "magic" that is done by the HP utility? It might be useful to zero out the entire USB key, and then use the HP tool on it, to see just what changes it makes to the partition table and such. I wonder how it would be different from the generic Linux tools?

I do plan to revisit this project once the Knoppix 5.0 CD (not DVD) comes out! Hopefully they'll have fixed the USB ehci.ko bug by then :)

zoltanthegypsy

03-18-2006, 04:49 PM

I also didn't know about the HP utility, and just used Linux cfdisk and mkdosfs tools to format the USB key. What is the "magic" that is done by the HP utility? It might be useful to zero out the entire USB key, and then use the HP tool on it, to see just what changes it makes to the partition table and such. I wonder how it would be different from the generic Linux tools?

I'm pretty sure it's the (fake) geometry: sectors/cylinder, etc. I have looked at the partition table w/fdisk in expert mode after a partiton/format with the HP tool and it looks pretty strange. I'll dig back into this when I get a minute. It should be possible to maximize "bootability" entirely with Linux tools once this is puzzled out (he said optimisticly).

Z.

Krellan2

03-27-2006, 09:45 PM

Just had to add this to the links in this thread:

http://www.sysresccd.org/Howto_install-usb-stick

Some more good information on making USB thumbdrives bootable under both Linux and Windows.

Krellan2

04-13-2006, 12:04 PM

Finally got around to analyzing what that HP formatter utility does!

I ran it on Windows XP.

Tried it with FAT and FAT32 filesystems. Tried it with bootable DOS system files, and without. So, total of 4 trials. The DOS system files came from the generic DOS boot disk you get when you use Windows XP to format a floppy. The HP formatter also offers the choice of NTFS, but I have no interest in keeping my data locked up inside Microsoft's proprietary silo :)

Here's what we get in the MBR.

Boot sector code:

http://mirror.href.com/thestarman/asm/mbr/Win2kmbr.htm

The HP formatter tool's boot sector compared identical to the standard Microsoft 2000/XP boot sector, but with one exception. There was a 3-byte patch at offset 0xCA: Microsoft's bytes of B8 01 02 were changed to some new bytes of EB 1A 90.

What this does is overwrite the start of the old-fashioned CHS INT 13 commands, with a jump to the newer LBA commands (INT 13 BIOS Extensions), which are located at offset 0xE6. This effectively FORCES the use of LBA, no matter what the size of the disk is. (Without this patch, a very small disk that fit into the CHS limits would still have been eligible to use the old CHS commands.)

Since every BIOS that is even remotely capable of seeing bootable USB thumbdrives, would have to already have supported INT 13 BIOS Extensions for years, this isn't a problem. It's probably a good thing, as it means the bogus CHS values, in the table below, will be completely ignored.

Other things between the boot sector code and the partition table:

Windows 2000/XP disk serial number at 0x1B8: The HP formatter writes a random value here, as it should, but strangely, the 4th byte (at 0x1BB) is always 00! So, there's only 3 bytes of randomness here, not 4. Not sure at all why this was done. A bug?

Error message pointers at 0x1B5: My Microsoft boot sector has the standard values of 2C 44 63 for US English. The HP formatter sets these to 00 00 00, though, so if the MBR were ever to print an error message, it would show up blank or garbage! Another bug?

Partition table:

It uses partition 1, which will be marked active if the bootable DOS system files were installed. CHS size is */255/63.

So, it's definitely not USB-ZIP compatible. (USB-ZIP compatibility would have required the use of partition 4 and CHS size of */64/32.)

Partition types are always 0x0E for FAT16 LBA, or 0x0C for FAT32 LBA. As with the patch above, HP forces LBA to always be used, no matter what the size of the partition is. (I don't have any very small USB keys, so can't fully prove this.)

CHS start is standard 0/1/1. This means the "hidden track" is still intact, and so the remaining 62 sectors of cylinder 0 head 0 are empty wasted space. So, feel free to install your favorite boot managers, or viruses, here. :)

CHS end is messed up, though. It does not match the LBA end! Instead, the CHS end is truncated to the highest possible full cylinder boundary. On my USB key, I have a total of 2001888 sectors. This should give me a CHS end of 124/155/63, but it doesn't. The HP formatter instead writes a CHS end of 123/254/63. It ends at the boundary of the last full cylinder (123) instead of continuing to include the final partial cylinder (124).

LBA start is 63 (0x3F), which is correct.

LBA size is the entire size of the USB key, minus those 63 sectors. Mine works out to 2001825 sectors, which is correct.

So, there you have it. That's pretty much a complete breakdown of what the HP formatter utility does, at least in the boot sector.

I didn't analyze past the boot sector, because once the MBR is running and LBA has been forced on, the BIOS hopefully won't be able to cause the rest of the boot process to fail. Assuming the BIOS returns valid data when reading the disk with the INT 13 BIOS Extensions, things should just work from there, as it's all generic filesystem and kernel loader stuff from there. It shouldn't be BIOS-dependent, from that point forward.

Still, I'd be curious to know if any other failures happen, when trying to boot from a USB thumbdrive that has been correctly recognized by the BIOS. What BIOS versions are in the computers that worked to boot drives formatted with the HP formatter but didn't work to boot drives formatted with other tools?

That, to me, is by far the hardest problem: getting the BIOS to see the drive, in the first place! If the BIOS can see the USB thumbdrive well enough to set it up as the bootable "hard drive" (the 0x80 device), and run the MBR properly, things just work from there, I have found. This is assuming the BIOS can provide valid INT 13 support when loading data, until the operating system gets fully up and running.

Testing with my SYSLINUX MBR and Knoppix distro on USB, that I made earlier, versus the bootable DOS disks created by this HP formatter utility, unfortunately didn't help me boot up from any more computers. I still had the same successes and the same failures. As before, the hard part is getting the BIOS to just see the USB thumbdrive in the first place. Cleverly patching the MBR won't help, if the BIOS isn't willing to load the MBR at all....

Josh

zoltanthegypsy

04-13-2006, 05:28 PM

Well Done!

I'm still stuck on "geometry" as one of the keys to bootability, though. I get maximum bootability by formatting (and creating the partition) with the HP utility even though I then install grub in the mbr - which overwrites the boot code but not the part table. Touching the part table (other than setting the active flag) with Linux fdisk limits bootability on some of my test boxen. (Pretty sure about this - I need to re-test, though with Linux fdisk in expert mode).

I've also tried putting grub in the partition's boot sector and using a box-stock XP mbr w/the HP part table. That seems to boot as well as the HP + grub mbr approach: machines that reject fob formatted w/Linux will still boot this combo.

Still, I'd be curious to know if any other failures happen, when trying to boot from a USB thumbdrive that has been correctly recognized by the BIOS. What BIOS versions are in the computers that worked to boot drives formatted with the HP formatter but didn't work to boot drives formatted with other tools?

That, to me, is by far the hardest problem: getting the BIOS to see the drive, in the first place! If the BIOS can see the USB thumbdrive well enough to set it up as the bootable "hard drive" (the 0x80 device), and run the MBR properly, things just work from there, I have found.

One older Intel MB/BIOS will try to boot my test fob if I use the HP formatter, but hangs solid at "GRUB". :(

I will try to find the time to experiment more w/geometry and see if my theory has any merit. AND note the BIOSes involved.

Z.

Krellan2

04-13-2006, 11:13 PM

Nice, thanks for checking into it. I've looked into it about as much as I can, given that my computers either all boot perfectly or fail to recognize the USB key at all = no in-between behaviour at all, unlike what you have seen.

As for geometry, a thought: since USB keys have no hardware CHS geometry at all (a good thing!), maybe a BIOS of yours is synthesizing the geometry from the CHS ending values in the partition table. It's using the head and sector numbers given, to set the geometry. That may be why the HP formatter limits itself to only including full cylinders in the CHS values, to guarantee that the head and sector numbers will be maxed out. Even though they are technically wrong (the total CHS count is then undersized, and doesn't match the LBA count). If the BIOS is reading these somehow, to establish a geometry, this would at least get the BIOS to establish the proper */255/63 values.

Ironically, including a proper CHS count in the partition table, instead of undersized like this, would cause failures, as the BIOS would then be getting bogus counts from the final, partial, cylinder at the end of the disk.

The BIOS could even be doing this from the extra Microsoft geometry fields in the PBR, at the beginning of FAT and FAT32 partitions. I did not check these at all. You might want to check these numbers and see if they compare with what is in the PBR.

Another experiment would be to try and replicate the HP formatter's patch, to GRUB's boot sector code: just force it outright to use LBA, and completely disable the use of CHS entirely, for loading any more sectors, no matter what the size of the disk is, or what the BIOS claims it supports. Since GRUB is open source, it should be a lot easier to make this patch (although the MBR is so small that it can fairly easily be completely reverse-engineered, as it was on that webpage I mentioned earlier).

Good luck!

Josh

hardawayd

04-24-2006, 03:38 PM

I am having trouble getting knoppix on usb to connect to the internet. Can someone tell me if knoppix has the ipw2200 firmware necessary for my dell laptop centrino wireless chip to work?

Krellan2

04-25-2006, 10:23 AM

I am having trouble getting knoppix on usb to connect to the internet. Can someone tell me if knoppix has the ipw2200 firmware necessary for my dell laptop centrino wireless chip to work?

I don't know about that chip, but I doubt it. The problem with firmware is that it is copyrighted. So, Knoppix can't distribute it. For a CD-based live distro like Knoppix, this really sucks, because you will have to remaster the CD yourself or download the firmware each time you boot up the CD. That's just one of the many problems with binary-only firmware.

http://ipw2200.sourceforge.net/firmware.php?fid=5
(big ugly licence agreement here)

Since you have a USB key, this will be slightly easier, because you can just copy the firmware file to your USB key and load it from there each time.

Unfortunately, it is very hard to find "clean" 802.11g cards: chipsets that don't require firmware or ndiswrapper in order to run. There are some reverse-engineered drivers that don't require firmware, but they tend to be unstable, as the reverser doesn't have enough information to understand the device fully and make a truly reliable driver (ACX111, Broadcom).

The only card I have found so far, that has both a manufacturer-supported Linux driver and no firmware/ndiswrapper requirement, is RaLink. Are there any others?

zoltanthegypsy

04-27-2006, 05:54 PM

zoltanthegypsy

04-27-2006, 06:13 PM

For the record, w/Knoppix 4.0.2, the ipw-2.2-xxx firmware is in /usr/lib/hotplug/firmware.

This is getting quite a way off topic :)

Z.

spalla

01-22-2007, 06:47 AM

hey there i followed all the instruction, but i keep getting the same error message:

could not find kernel image: linux

any idea what i did wrong?

thank you for your help