Page 1 of 2 12 LastLast
Results 1 to 10 of 17

Thread: Success making bootable and persistent USB key with SYSLINUX

  1. #1
    Member registered user
    Join Date
    Mar 2006
    Posts
    50

    Success making bootable and persistent USB key with SYSLINUX

    I've had success installing an entire Knoppix CD to a USB thumbdrive. Also was able to create the Knoppix persistent image, with encryption, on the same USB thumbdrive. All told, it fits perfectly on a 1GB thumbdrive, a fairly common size these days.

    750MB for Knoppix's 700MB CD (plus overhead)
    100MB for the encrypted persistent image
    150MB leftover for unimportant file storage, and use on Windows systems

    After checking this page:
    http://www.knoppix.net/wiki/USB_Based_FAQ

    I got the thumbdrive to boot with GRUB, as it said. All went well... until I went to another computer. It had more SCSI drives, and so the drive numbers had changed! GRUB's awkward (hd0,0) syntax could no longer find the drive, and GRUB failed to boot (didn't even get as far as the GRUB command line).

    So, GRUB was out.

    KNOPPIX itself uses ISOLINUX, part of the SYSLINUX suite of bootloaers, so why not just use SYSLINUX? Turns out, it worked great! GRUB also doesn't give you the pretty "speedometer" loading screen, which SYSLINUX does (since it's essentially a twin of ISOLINUX, it boots up in the same style). I'm posting this from Knoppix now.

    Some rough steps (I'll write this up more, if there's interest, perhaps make a new Wiki page or something):

    1) Partition and format the ENTIRE USB thumbdrive as one big FAT32 partition. Mark it as active. If you have "cfdisk", use its Maximize feature to squeeze out an extra 0.5MB of space "for free". No need to make more than one partition. Copy SYSLINUX's "mbr.bin" file over your existing MBR, to make it bootable. Don't mount the new partition just yet.

    2) Run SYSLINUX on that new partition. The default SYSLINUX options are OK. Now, it's OK to mount the new partition. You should see the file "ldlinux.sys" there.

    3) Mount the Knoppix ISO. Copy its entire contents to your USB thumbdrive! Make sure the directories line up, so that all things are in the same place on the USB thumbdrive as they are on the CD.

    4) Unmount the Knoppix ISO, you won't need it anymore. On your USB thumbdrive, go to the directory /boot/isolinux. Move ALL files in this directory to the top level of the drive (where the existing autorun files already are). As they are now empty, you may completely remove the /boot/isolinux and /boot directories.

    5) Rename the file isolinux.cfg to syslinux.cfg.

    6) You're ready to boot! If you have a newer USB 2.0 thumbdrive, though, you've got some more work ahead of you. Now comes the hard part. You'll have to edit the linuxrc file inside of the minirt.gz file, to fix a Knoppix bug where it will fail to load the USB module at startup. Search other forum posts here for how to do this

    7) Reboot, trying to boot from USB. Did it work? Cool!

    Make sure Knoppix works. Try to create a persistent Knoppix disk image, using the tool under the penguin menu. When it asks you to choose a partition, choose your USB thumbdrive. If it complains about it being unable to mount because it's already mounted, do this:

    mount -o remount,rw /cdrom
    mount --bind /cdrom /mnt/sdz1
    (replace sdz1 with the true device name of your USB thumbdrive's partition, of course)

    Now, try again, and it should let you make the persistent image.

    9) As an optimization, try editing the syslinux.cfg file. I like reducing the timeout in the 3rd line (to 30 from 300) so that it boots automatically without having to wait so long. Also, I like adding these options to the 2nd line:

    noeject noprompt home=scan dma noswap

    No point trying to "eject the CD and close drive door" when it's a USB thumbdrive! The other cheatcodes are for autoloading your persistent image next time, DMA speed increase, and security fix (never use swap, because it's not encrypted).

    10) Reboot again. It should ask you for your password, for the encryption, and then load up just fine.

    Nice!

    If you saw an earlier post of mine, I tried EXTLINUX at first (not SYSLINUX). Unfortunately, that was no good, because Knoppix is more difficult to boot on case-sensitive filesystems (it will have trouble finding certain things as it loads). Also, EXTLINUX can't be easily used on Windows, but FAT32 can. Having one big partition makes a much better use of space on the thumbdrive.

    If anyone else has had success doing something like this, feel free to post your tips here as well.

    Josh

  2. #2
    Junior Member registered user
    Join Date
    Jan 2006
    Posts
    11
    Intesting. I just did this too. Remastered KNOPPIX to shrink it a bit and added chntpw and captive-ntfs and my own KDE menus, boot splashes, and artwork and put it on a 1G fob booted w/grub. Pretty slick.

    I haven't tried it on a system w/scsi drives, but I'm a little surprised it wouldn't boot for you. Did you still have the device.map file in /boot/grub? If so I wonder if that's what is confusing things.

    I didn't use the wiki procedure to put mine together. I just copied my remastered CD to the fob, copied the grub files over, and used grub -> setup to write the mbr. No device.map file.

    This is just a total WAG, but I wonder how it would have worked w/out device.map.

    Later,
    Z

  3. #3
    Member registered user
    Join Date
    Mar 2006
    Posts
    50
    Quote Originally Posted by zoltanthegypsy
    Intesting. I just did this too. Remastered KNOPPIX to shrink it a bit and added chntpw and captive-ntfs and my own KDE menus, boot splashes, and artwork and put it on a 1G fob booted w/grub. Pretty slick.

    I haven't tried it on a system w/scsi drives, but I'm a little surprised it wouldn't boot for you. Did you still have the device.map file in /boot/grub? If so I wonder if that's what is confusing things.
    I didn't use GRUB. For something standalone like a USB key, SYSLINUX seems like a better choice, since it doesn't require a device map at all. SYSLINUX is entirely self-contained and just "sees" the media it's booting off of, and no others. This is bad for a multibooting desktop machine, but very good for removable/rearrangeable things like USB keys, since there's no more problems with devices appearing in different places on various computers.

    As far as remastering goes, I'm waiting until the official release of 5.0. Even though I have the DVD image already, I'm hoping they have a CD-sized image, so I can easily fit it into the USB key.

  4. #4
    Junior Member registered user
    Join Date
    Jan 2006
    Posts
    11
    I didn't use GRUB. For something standalone like a USB key, SYSLINUX seems like a better choice, since it doesn't require a device map at all.
    Grub doesn't need device map either, unless there is some sort of confusion between BIOS drive mapping and OS drive mapping, IIRC.

    My questions was whether grub would have worked for you if you hadn't included device.map when you installed it /booted from it. W/out device.map, the drive booted from (the flash fob) should just be drive 0 in menu.lst entries regardless of other drives on the box.

    I don't mean to argue the merits of SYSLSINUX vs GRUB, just wondering why grub didn't work well for you. It has certainly worked OK for me so far.

    Z.[/quote]

  5. #5
    Member registered user
    Join Date
    Mar 2006
    Posts
    50
    Quote Originally Posted by zoltanthegypsy
    Grub doesn't need device map either, unless there is some sort of confusion between BIOS drive mapping and OS drive mapping, IIRC.

    My questions was whether grub would have worked for you if you hadn't included device.map when you installed it /booted from it. W/out device.map, the drive booted from (the flash fob) should just be drive 0 in menu.lst entries regardless of other drives on the box.

    I don't mean to argue the merits of SYSLSINUX vs GRUB, just wondering why grub didn't work well for you. It has certainly worked OK for me so far.

    Z.
    [/quote]

    That's cool, I didn't know that grub did not require the device map. It had failed for me in the past when I tried it. SYSLINUX also is handy for Knoppix since Knoppix uses ISOLINUX already. Maybe I'll give GRUB another chance someday.

  6. #6
    Junior Member registered user
    Join Date
    Jan 2006
    Posts
    11
    Maybe I'll give GRUB another chance someday.
    If/when you do, please let us know what you find. I'm also looking forward to your full SYSLINUX write-up.

    My neighbor does a lot of winXX rescue/recovery - now _there's_ a growth industry - and I'm trying to put together a set of flash drives that boot on as many machines as possible.

    Along with the grub/syslinux thing, it also seems to matter (to some boxen) just how the fob gets formatted in the first place. I've put up a page describing my struggles:

    http://www.beezmo.com/FloobyDustDir/FDKnoppixUsb.htm

    No doubt there are errors/omissions. I welcome feedback. Be gentle

    Thanks again for your SYSLINUX info.

    Z.

  7. #7
    Member registered user
    Join Date
    Mar 2006
    Posts
    50
    Quote Originally Posted by zoltanthegypsy
    If/when you do, please let us know what you find. I'm also looking forward to your full SYSLINUX write-up.
    Thanks. I've gotten busier recently, so that writeup might have to wait a while.

    However, I checked that link on your page, and what I did is very, very similar to what is here:

    http://rz-obrian.rz.uni-karlsruhe.de/knoppix-usb/

    It looks like he used an older version of Knoppix. The USB scripting has changed since then. I wonder if the sleep is still needed. Sometimes I get failed boots, where it just hangs after Knoppix has found the USB device. It starts to boot, Knoppix loads up, it loads the USB drivers, and then it hangs. Lots of SCSI errors in the log. Most of the time, though, it works. Maybe my USB key is flaky: I keep it on a keychain (like a real key and it's been bumped around a few times.

    I also didn't know about the HP utility, and just used Linux cfdisk and mkdosfs tools to format the USB key. What is the "magic" that is done by the HP utility? It might be useful to zero out the entire USB key, and then use the HP tool on it, to see just what changes it makes to the partition table and such. I wonder how it would be different from the generic Linux tools?

    I do plan to revisit this project once the Knoppix 5.0 CD (not DVD) comes out! Hopefully they'll have fixed the USB ehci.ko bug by then

  8. #8
    Junior Member registered user
    Join Date
    Jan 2006
    Posts
    11
    I also didn't know about the HP utility, and just used Linux cfdisk and mkdosfs tools to format the USB key. What is the "magic" that is done by the HP utility? It might be useful to zero out the entire USB key, and then use the HP tool on it, to see just what changes it makes to the partition table and such. I wonder how it would be different from the generic Linux tools?
    I'm pretty sure it's the (fake) geometry: sectors/cylinder, etc. I have looked at the partition table w/fdisk in expert mode after a partiton/format with the HP tool and it looks pretty strange. I'll dig back into this when I get a minute. It should be possible to maximize "bootability" entirely with Linux tools once this is puzzled out (he said optimisticly).

    Z.

  9. #9
    Member registered user
    Join Date
    Mar 2006
    Posts
    50
    Just had to add this to the links in this thread:

    http://www.sysresccd.org/Howto_install-usb-stick

    Some more good information on making USB thumbdrives bootable under both Linux and Windows.

  10. #10
    Member registered user
    Join Date
    Mar 2006
    Posts
    50
    Finally got around to analyzing what that HP formatter utility does!

    I ran it on Windows XP.

    Tried it with FAT and FAT32 filesystems. Tried it with bootable DOS system files, and without. So, total of 4 trials. The DOS system files came from the generic DOS boot disk you get when you use Windows XP to format a floppy. The HP formatter also offers the choice of NTFS, but I have no interest in keeping my data locked up inside Microsoft's proprietary silo

    Here's what we get in the MBR.

    Boot sector code:

    http://mirror.href.com/thestarman/asm/mbr/Win2kmbr.htm

    The HP formatter tool's boot sector compared identical to the standard Microsoft 2000/XP boot sector, but with one exception. There was a 3-byte patch at offset 0xCA: Microsoft's bytes of B8 01 02 were changed to some new bytes of EB 1A 90.

    What this does is overwrite the start of the old-fashioned CHS INT 13 commands, with a jump to the newer LBA commands (INT 13 BIOS Extensions), which are located at offset 0xE6. This effectively FORCES the use of LBA, no matter what the size of the disk is. (Without this patch, a very small disk that fit into the CHS limits would still have been eligible to use the old CHS commands.)

    Since every BIOS that is even remotely capable of seeing bootable USB thumbdrives, would have to already have supported INT 13 BIOS Extensions for years, this isn't a problem. It's probably a good thing, as it means the bogus CHS values, in the table below, will be completely ignored.

    Other things between the boot sector code and the partition table:

    Windows 2000/XP disk serial number at 0x1B8: The HP formatter writes a random value here, as it should, but strangely, the 4th byte (at 0x1BB) is always 00! So, there's only 3 bytes of randomness here, not 4. Not sure at all why this was done. A bug?

    Error message pointers at 0x1B5: My Microsoft boot sector has the standard values of 2C 44 63 for US English. The HP formatter sets these to 00 00 00, though, so if the MBR were ever to print an error message, it would show up blank or garbage! Another bug?

    Partition table:

    It uses partition 1, which will be marked active if the bootable DOS system files were installed. CHS size is */255/63.

    So, it's definitely not USB-ZIP compatible. (USB-ZIP compatibility would have required the use of partition 4 and CHS size of */64/32.)

    Partition types are always 0x0E for FAT16 LBA, or 0x0C for FAT32 LBA. As with the patch above, HP forces LBA to always be used, no matter what the size of the partition is. (I don't have any very small USB keys, so can't fully prove this.)

    CHS start is standard 0/1/1. This means the "hidden track" is still intact, and so the remaining 62 sectors of cylinder 0 head 0 are empty wasted space. So, feel free to install your favorite boot managers, or viruses, here.

    CHS end is messed up, though. It does not match the LBA end! Instead, the CHS end is truncated to the highest possible full cylinder boundary. On my USB key, I have a total of 2001888 sectors. This should give me a CHS end of 124/155/63, but it doesn't. The HP formatter instead writes a CHS end of 123/254/63. It ends at the boundary of the last full cylinder (123) instead of continuing to include the final partial cylinder (124).

    LBA start is 63 (0x3F), which is correct.

    LBA size is the entire size of the USB key, minus those 63 sectors. Mine works out to 2001825 sectors, which is correct.

    So, there you have it. That's pretty much a complete breakdown of what the HP formatter utility does, at least in the boot sector.

    I didn't analyze past the boot sector, because once the MBR is running and LBA has been forced on, the BIOS hopefully won't be able to cause the rest of the boot process to fail. Assuming the BIOS returns valid data when reading the disk with the INT 13 BIOS Extensions, things should just work from there, as it's all generic filesystem and kernel loader stuff from there. It shouldn't be BIOS-dependent, from that point forward.

    Still, I'd be curious to know if any other failures happen, when trying to boot from a USB thumbdrive that has been correctly recognized by the BIOS. What BIOS versions are in the computers that worked to boot drives formatted with the HP formatter but didn't work to boot drives formatted with other tools?

    That, to me, is by far the hardest problem: getting the BIOS to see the drive, in the first place! If the BIOS can see the USB thumbdrive well enough to set it up as the bootable "hard drive" (the 0x80 device), and run the MBR properly, things just work from there, I have found. This is assuming the BIOS can provide valid INT 13 support when loading data, until the operating system gets fully up and running.

    Testing with my SYSLINUX MBR and Knoppix distro on USB, that I made earlier, versus the bootable DOS disks created by this HP formatter utility, unfortunately didn't help me boot up from any more computers. I still had the same successes and the same failures. As before, the hard part is getting the BIOS to just see the USB thumbdrive in the first place. Cleverly patching the MBR won't help, if the BIOS isn't willing to load the MBR at all....

    Josh

Page 1 of 2 12 LastLast

Similar Threads

  1. making the damn thing bootable?
    By jengiscan in forum MS Windows & New to Linux
    Replies: 3
    Last Post: 08-25-2008, 11:15 AM
  2. making bootable CD from Knoppix_v3.6-2004-08-16-EN.iso
    By Joseph Schiller in forum Customising & Remastering
    Replies: 2
    Last Post: 11-17-2004, 11:44 PM
  3. HELP making/burning 3.4 bootable AGAIN in WIN
    By hw.stadl0r in forum Customising & Remastering
    Replies: 2
    Last Post: 05-14-2004, 04:01 PM
  4. making knoppix bootable
    By matt_w_lambert in forum Customising & Remastering
    Replies: 6
    Last Post: 01-05-2004, 10:20 PM
  5. Making the HDD install bootable with Grub...
    By ironstorm in forum Hdd Install / Debian / Apt
    Replies: 5
    Last Post: 11-23-2002, 08:26 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Dell R730xd 12LFF 2.6Ghz 20-C 128GB H730 2x10G+2x1G NIC 2x1100W 12x Trays Rails picture

Dell R730xd 12LFF 2.6Ghz 20-C 128GB H730 2x10G+2x1G NIC 2x1100W 12x Trays Rails

$721.05



Dell PowerEdge R720XD Xeon E5-2680 V2 2.8GHz 20 Cores 256GB RAM 12x4TB picture

Dell PowerEdge R720XD Xeon E5-2680 V2 2.8GHz 20 Cores 256GB RAM 12x4TB

$510.00



DELL PowerEdge R730 Server 2x E5-2690v3 2.6GHz =24 Cores 32GB H730 4xRJ45 picture

DELL PowerEdge R730 Server 2x E5-2690v3 2.6GHz =24 Cores 32GB H730 4xRJ45

$275.00



Dell PowerEdge R630 2x Xeon E5-2630 V4 2.20GHZ 128GB DDR4-2133MHZ 2x 750W PSU picture

Dell PowerEdge R630 2x Xeon E5-2630 V4 2.20GHZ 128GB DDR4-2133MHZ 2x 750W PSU

$170.95



Dell PowerEdge R730XD 28 Core Server 2X Xeon E5-2680 V4 H730 128GB RAM No HDD picture

Dell PowerEdge R730XD 28 Core Server 2X Xeon E5-2680 V4 H730 128GB RAM No HDD

$389.99



Dell PowerEdge R730, 2 sinks, SystemBoard, 8 trays,H330,Idrac 8 exp, 2x750w Psu picture

Dell PowerEdge R730, 2 sinks, SystemBoard, 8 trays,H330,Idrac 8 exp, 2x750w Psu

$135.00



Dell PowerEdge R620 Server 2x Xeon  E5-2620 @ 2.0GHz 64GB RAM NO HDDs picture

Dell PowerEdge R620 Server 2x Xeon E5-2620 @ 2.0GHz 64GB RAM NO HDDs

$108.96



Dell PowerEdge R620 Server - 256GB RAM, 2x8c CPU, 128Gb SSD/3x600Gb SAS, Proxmox picture

Dell PowerEdge R620 Server - 256GB RAM, 2x8c CPU, 128Gb SSD/3x600Gb SAS, Proxmox

$320.00



Dell PowerEdge R430 3.5 1U 2x E5-2690 v3 2.6ghz 24-Cores 256gb 4x Trays 2x 550w picture

Dell PowerEdge R430 3.5 1U 2x E5-2690 v3 2.6ghz 24-Cores 256gb 4x Trays 2x 550w

$289.99



DELL PowerEdge R630 8SFF Server 2x E5-2690v4 2.6GHz =28 Cores 128GB H730 4xRJ45 picture

DELL PowerEdge R630 8SFF Server 2x E5-2690v4 2.6GHz =28 Cores 128GB H730 4xRJ45

$444.00