Page 1 of 2 12 LastLast
Results 1 to 10 of 18

Thread: USB Remasterig Using Minimum Real Estate

  1. #1
    Senior Member
    Join Date
    Jan 2011
    Posts
    242

    USB Remasterig Using Minimum Real Estate

    USB remastering traditionally requires lots of temporary file space. More than most people have spare on their USB sticks leading to the temptation to use nfts file space.

    I've refined the process and managed to cut the temporary file space down to zero (and speed up the process considerably as a result). The assumption is that you start with a Knoppix image file and a persistent store file and finish with a new pair of files. You need to test the new pair before deleting the old so you need at least enough flash file storage for both but that is all the file storage you need.

    The first USB remaster I did comprised four stages:

    1. Prepare a new master file system. This is largely a copy of /UNIONFS.
    2. Generate an ISO image from the master file sytem
    3. Generate a cloop image by compression of the ISO image
    4. Copy the cloop image to the USB

    The obvious first refinement was to generate the cloop image directly on the USB filesystem.

    Most people already combine the second and third stages by using a pipe but I couldn't because I was remastering the DVD MAXI edition and just ran out of (virtual) memory. I suspect somehow you could have enough virtual memory if you ran a 64-bit kernel but even swap space is temporary file space and I didn't want that. Instead I used squashfs that generates a compressed file system in one pass.

    Note: you do not need to use squashfs if you remaster the CD edition. You just need a machine with 1 or 2 Gb of RAM.

    The final refinement is to avoid creating a complete new master file system. I did this by feeding squashfs a list of directories thus:
    Code:
    sudo mksquashfs ${dirs} $DST/KNOPPIX/KNOPPIX.sq -b 65536 -noappend
    The equivalent for the CD edition might be:

    Code:
    sudo genisoimage ${options} ${dirs} | sudo create_compressed_fs -B 65536 - $DST/KNOPPIX/KNOPPIX
    Repeat: you do not need to use squashfs if you remaster the CD edition.



    Which directories are fed to the remastering and which are not ?

    I deliberately kept that simple: all directories under /UNIONFS except (possibly) etc, var, home, dev and lost+found.

    lost+found should be empty. It isn't wanted anyway so I deleted anything in /UNIONFS/lost+found and then ignored it.

    dev isn't really needed except as a back-up in case knoppix-autoconfig goes wrong. It is very small, so I included it in the remastering.

    home is my stuff so it is not in the remastering. It stays in knoppix-data.img but needs a tidy up.

    var is stuff that changes anyway so there seems little point including it in a remastering of a USB install (CD remaster yes). Unfortunately, it's rather big because I've updated the package lists. I could delete all those but they'd just come back next time I want to install anything. The best I could do was sudo rm /var/cache/apt/archives/*.deb to recover 50 Mb of space in knoppix-data.img.

    etc is the difficult one. It contains a lot of files that are put there when packages are installed that I've never been anywhere near. These should go into the remaster. On the other hand there are files that have been changed since installation, many of which contain information about my machines and network, including passwords, that should not go into the remaster.

    Really a second level sort is needed. Since etc isn't very big, this could be done in memory using /tmp. However, I decided since etc isn't very big and comprises a lot of little files that probably don't compress and decompress efficiently, I'd keep it out of the remaster and copy all of it into the new knoppix-data.img.

    For etc, var and home, I included in the remaster the directory contents from /KNOPPIX so what was on the original DVD is carried forward.

    Script ? Of course I have a script. I've remastered twice this way. The whole process, from start to rebooting with a new KNOPPIX.sq and knoppix-data.img takes less than 20 minutes. Not bad.
    Last edited by Forester; 03-15-2011 at 12:18 AM.

  2. #2
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    Just like Fermat's last theorem.

  3. #3
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by utu View Post
    Just like Fermat's last theorem.
    I've had a word with my mate Fermat and he apologises. He latest theoretical script is only proven for one of nine Lagrangian points and for the least general point at that. He has ordered more UFDs so that he can test the hypothesis in Cartesian space.

  4. #4
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    I think you may safely ignore the Noetherian domain
    in cases where only wireless use is anticipated.

  5. #5
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Attached is Fermat's Last Script that proves the hypothesis that it is possible to remaster a Knoppix USB installation without the need for temporary storage. It would be nice to just say QED and leave it at that but would not be terribly helpful.

    The script assumes two USB installations, one of which is in use. It will determine automatically which is in use and overwrite the Knoppix image and persistent data of the other. The user has to declare where to find the two installations by editing the script. This should be a one time edit of the script.

    There are three usage scenarios:

    1. The good, where the USB installations are on different UFDs (aka pendrives, USB sticks ...). It is easy to lose (temporarily) one USB stick but two at once takes genius. Good also because this scenario does not require cheat codes.

    2. The bad, where the cheat code fromhd= is used to boot USB installations on different file systems on the same USB stick. The redeeming feature of this arrangement is you're free to use proper file systems but otherwise the complexity is more that most people can be bothered with.

    3. The ugly, where the cheat code knoppix_dir= is used to boot USB installations in different directories on the same file system on the same USB stick. This may be the scenario most would try first. Trials using a 'generic' USB were a disaster. Trials using a branded USB were successful. Both sticks were brand new. You have been warned.

    The script contains instructions on how to declare your installations and requires an edit to run for the first time. Cheat codes not included.

    After first time edit, the script will perform one of five actions when invoked correctly. Two are concerned with remastering the knoppix image and three with the persistent store. If you use still use knoppix.sh you will have to copy it yourself. It is suggested you consider using /etc/rc.local instead.

    The Knoppix image can be remastered with either of the actions:

    Code:
    cloop image
    squashfs image
    Creation of the squashfs image is entirely at your discretion but does require the squashfs-tools package (not included). It is easy to convert between the two images and to compare execution times so you can pick the faster if you dare.

    Note to boot a squashfs image requires prior modification of the /init script as discussed in the thread http://www.knoppix.net/forum/threads...hfs-ed-knoppix. Patches you can apply to /init are attached to post #29.

    The actions:

    Code:
    create knoppix-data.img <size>
    create knoppix-data-aes <size>
    may (again at your discretion) be used to create a new persistent store for the remastered installation. These options allow the size or encryption of the persistent store to be modified during remastering (and yes, it is faster than using the backup / restore cheatcodes).

    Finally the action:

    Code:
    prune knoppix-data
    is used to copy /home etc. to the persistent store of the remastered installation.

    The persistent store actions requires the loop device fix (not included), which is best done as a one off before remastering for the first time:

    Code:
    sudo rm -fr /dev/loop
    sudo rm -fr /lib/udev/devices/loop
    Since the upload facility of the forum limits file extensions, the attachment is named doremaster.txt. After download it is suggested you rename it to doremaster and make it executable.
    Attached Files Attached Files
    Last edited by Forester; 03-21-2011 at 09:36 AM.

  6. #6
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    @ Forester

    I'm getting ready for the plunge, but I'd like to clarify a few points.
    I'm assuming the following:

    1. Both USB's are assumed to be knoppix LiveUSB's of the same version,
    in good working order. Presumably the working knoppix will have a persistent
    store already, the target may or may not.

    2. The two LiveUSBs while of the same version, may or may not be equally
    well-developed from the same initial state.

    3. The physicality of the two liveUSBs is immaterial. If one is a 2 Gb
    SD Card and the other is a 4 Gb 'regular' USB, that's ok. And, noone cares
    about the underlying format details for either, either.


    I'm puzzled over the FLIP and FLOP
    4. I would feel more comfortable saying that FLIP IS the working LiveUSB
    and FLOP IS the target, with my own specifications. How do I know this
    won't work out the other way?

    Are we ok so far?

  7. #7
    Senior Member
    Join Date
    Jan 2011
    Posts
    123
    and what if I want to remaster but put the new knoppix image in /mnt ?
    I know knoppix is on /dev/sdb1/knoppix but I do not know what to put in flop if I want it in mnt considering there is no knoppix install over there.
    would be great if you clarify it a bit more

  8. #8
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    And another thing, Forester:

    I am further presuming that

    1....The objective is to re-manufacture the KNOPPIX compressed image of
    the target incorporating all previous knoppix-data.img improvements of
    the source LiveUSB into it. And, that a new persistent store contains
    only its minimum essential content, that is, nothing added by the user.
    .....As a test, if one deferred to add persistent store to the target,
    presumably the target would be as fully developed as the source, with
    all its persistent store baggage, and not requiring any time to mount
    a knoppix-data.img.


    Since no good deed goes unpunished here, I offer this

    2. If you've done all this, it would seem a relatively small matter
    conceptually speaking, NOT to require a working LiveUSB to start with
    for the target. Might there be some way to tap into a process already
    in place on the master to prepare a carcass on the target on which to
    hang the new files you have prepared. If not, you've left me with the
    tedious task of going back to my LiveCD to prepare such carcasses.
    I'm guessing I can omit the persistent store part of this tedium.

  9. #9
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Gentlemen,

    Think of the glass as half full, not half empty. The beer tastes better that way, though I admit many beers benefit from a slice of lemon →

    Are you any good at drawing lines ? There are two kinds of lines: those you draw and those drawn for you. The second are usually called deadlines. Yes, you tick all the feature boxes but you know most only work downhill with a following wind.

    Work is work. This is fun. I drew the line where I knew I could fill in all the details up to the line. That's what make me an amateur.

    I started with a perfectly good script that did what I needed: squashfs image and prune knoppix-data for the bad scenario. Not a script that many would find useful. I've added the two other scenarios I can think of, the two potentially useful create actions, though I for one will probably never encrypt my data, and add the obvious cloop image I have no use for. I believe you and many others can now get the utility from script that I got out of the original.

    Yes, there is stuff that can be added and I hope it will be. Each time I add a feature, I more there is to test (and retest) and the chances I miss something that hurts someone else goes up. Describing it clearly becomes harder and the chances that folks who never used the simple version are bewildered and confused into self-harm goes up. Why add features I don't need and that I don't know anyone else wants ?

    This is an Open Source environment. That means someone has an idea, implements it, tests it, publishes it and then others try it, test it, discuss it, criticise it, amend it, extend it and you get a result that is better than one man can do (even with the supernatural help of a dead Frenchman).

    utu and dinosoep, you have already raised some interesting points. I'll give feedback presently but it's time for my nightly TV fix.

  10. #10
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by dinosoep View Post
    and what if I want to remaster but put the new knoppix image in /mnt ?
    I know knoppix is on /dev/sdb1/knoppix but I do not know what to put in flop if I want it in mnt considering there is no knoppix install over there.
    would be great if you clarify it a bit more
    I am not sure I understand your problem. Here's my understanding of how things work ...

    Linux knows about 'block devices' and partitions on them (e.g sdxn). You can find out which it knows about by typing in cat /proc/partitions. For each of these Linux creates a device file in /dev (e.g. /dev/sdxn). For each of these (with a file system), Knoppix adds a entry in /etc/fstab and creates directory under /media so you can mount the file system with just mount /media/sdxn.

    The doemaster script uses this to mount and umount file systems so you don't have to. You are not expected to use /mnt so you need to explain a bit more what you think you want so I can see what you mean.

    For most people, the (only) hard drive is sda and their USB stick is sdb. Stick in a second USB and it will be sdc. So declare FLIP and FLOP to be /dev/sdb1/KNOPPIX and /dev/sdc1/KNOPPIX as the example in the script shows.

    If you have two USB sticks plugged in when you boot and use the fromhd= cheat code to boot from /dev/sdc1/KNOPPIX, the script will spot this and remaster onto /dev/sdb1/KNOPPIX.

    Does that help ?

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Dell PowerEdge R720 Server - 2x8c CPU,256Gb RAM, 128Gb SSD/3x600Gb SAS, Proxmox picture

Dell PowerEdge R720 Server - 2x8c CPU,256Gb RAM, 128Gb SSD/3x600Gb SAS, Proxmox

$340.00



DELL PowerEdge R730 Server 2x E5-2690v3 2.6GHz =24 Cores 32GB H730 4xRJ45 picture

DELL PowerEdge R730 Server 2x E5-2690v3 2.6GHz =24 Cores 32GB H730 4xRJ45

$275.00



Dell PowerEdge R720XD Xeon E5-2680 V2 2.8GHz 20 Cores 256GB RAM 12x4TB picture

Dell PowerEdge R720XD Xeon E5-2680 V2 2.8GHz 20 Cores 256GB RAM 12x4TB

$510.00



Dell PowerEdge R730XD 28 Core Server 2X Xeon E5-2680 V4 H730 128GB RAM No HDD picture

Dell PowerEdge R730XD 28 Core Server 2X Xeon E5-2680 V4 H730 128GB RAM No HDD

$389.99



Dell PowerEdge R620 Server 2x Xeon  E5-2620 @ 2.0GHz 64GB RAM NO HDDs picture

Dell PowerEdge R620 Server 2x Xeon E5-2620 @ 2.0GHz 64GB RAM NO HDDs

$108.96



Dell PowerEdge R730, 2 sinks, SystemBoard, 8 trays,H330,Idrac 8 exp, 2x750w Psu picture

Dell PowerEdge R730, 2 sinks, SystemBoard, 8 trays,H330,Idrac 8 exp, 2x750w Psu

$135.00



Dell PowerEdge R630 2x Xeon E5-2630 V4 2.20GHZ 128GB DDR4-2133MHZ 2x 750W PSU picture

Dell PowerEdge R630 2x Xeon E5-2630 V4 2.20GHZ 128GB DDR4-2133MHZ 2x 750W PSU

$170.95



Dell PowerEdge R620 Server - 256GB RAM, 2x8c CPU, 128Gb SSD/3x600Gb SAS, Proxmox picture

Dell PowerEdge R620 Server - 256GB RAM, 2x8c CPU, 128Gb SSD/3x600Gb SAS, Proxmox

$320.00



DELL PowerEdge R630 8SFF Server 2x E5-2690v4 2.6GHz =28 Cores 128GB H730 4xRJ45 picture

DELL PowerEdge R630 8SFF Server 2x E5-2690v4 2.6GHz =28 Cores 128GB H730 4xRJ45

$444.00



DELL PowerEdge R730 Server 2x E5-2643v3 3.4GHz =12 Cores 64GB H730 4xRJ45 picture

DELL PowerEdge R730 Server 2x E5-2643v3 3.4GHz =12 Cores 64GB H730 4xRJ45

$306.00