Page 1 of 3 123 LastLast
Results 1 to 10 of 29

Thread: Re-master your flash installation

  1. #1
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423

    Re-master your flash installation

    The title is kind of confusing. If you have already installed knoppix on flash, and you
    have enabled a persistent store, meaning you could go ahead with adding, upgrading
    and customizing packages, why do you still need to re-master knoppix ?

    Reasons as follows :-

    1. When you upgrade your files/packages, the original copy resided on the
    read-only compressed image is still there. The system exposes you to the ones on
    the persistent store, and hides away the ones on the compressed image.

    This is a waste of space, if you have upgraded many files/packages, for
    every file/package in the persistant store, there is a counterpart in the
    compressed image. Re-master will delete the hidden copies.

    Even when there is no compressed image counterpart, due to you added some
    new interesting packages, by moving your added packages to the compressed
    image, you also free up the (uncompressed) space on the persistent store.

    2. Another reason to re-master is that you want to maintain a clean separation between
    experimental things verses the "stable" portion of your customization, ie you want to
    move the stable and proven things to the compressed image, so that you can later
    proceed to add your experimental things to the persistent store. In case you
    are not happy with the new experimental things you added, you can always erase it
    from your persistent store, or even you can wipe out the entire persistence store.
    If you have previously re-mastered your flash installation, deleting things from the
    persistent store is an easy, clear-cut and safe operation; because even if you delete
    everything, you will still have the last snapshot of a fully working, stable and proven,
    and yet already fully customized version of knoppix to start with. There is no need
    to falling back to original uncustomized knoppix.

    The end result of a flash re-master is not a new copy of CD or DVD, as you have already
    booted off from flash, CD or DVD is irrelevant. The end result of flash re-master is just a
    new copy of compressed image KNOPPIX.

    This is how I re-master my flash installation, in brief. Note there is no "chroot" needed :-

    Find a directory or a partition where you have a few GB of space :-
    # mkdir -p /lots_of_free_space/source/KNOPPIX
    # mkdir -p /lots_of_free_space/master/KNOPPIX
    # cd /lots_of_free_space/source/KNOPPIX
    # cp -a /dev .
    # cp -a /UNIONFS/bin .
    # cp -a /UNIONFS/etc .
    # cp -a /UNIONFS/lib .
    # cp -a /UNIONFS/sbin .
    # cp -a /UNIONFS/usr .
    # cp -a /UNIONFS/var .
    # cp -a /UNIONFS/opt .
    # cp -a /UNIONFS/home .
    # cp -a /UNIONFS/root .
    ( Whatever you want it to appear in the new compressed image, you copy it there )
    .....

    # cd ..
    # mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
    -hide-rr-moved -cache-inodes -pad /lots_of_free_space/source/KNOPPIX \
    | nice -5 /usr/bin/create_compressed_fs - 262144 > /lots_of_free_space/master/KNOPPIX/KNOPPIX

    After a long time, you will get a compressed image at /lots_of_free_space/master/KNOPPIX/KNOPPIX,
    which you can use it to replace your /mnt-system/KNOPPIX/KNOPPIX.

    You can see that the steps above are quite simple. This last step of replacing existing KNOPPIX
    compressed image must be perform "offline", ie you should perform it when you are not
    using /mnt-system/KNOPPIX/KNOPPIX.

    There is a challenge, because once you have booted in knoppix, you are already using
    /mnt-system/KNOPPIX/KNOPPIX. So either you boot from a CD with your flash detached
    and then put the flash back after you have booted so that you can overwrite it, OR you
    could configure /mnt-system/boot/syslinux/syslinux.cfg to add the cheatcode "debug"
    in it. At the first shell prompt, the /mnt-system/KNOPPIX/KNOPPIX is not mounted yet,
    this is when you could manually perform this somewhat "risky" task.

    Basically the steps will be roughly like this :-

    If your free space which contains the compressed image resides on a separate partition :-
    # mount /dev/sdaX /lot_of_free_space
    If you have enough space, you can rename KNOPPIX for backup purposes :-
    # mv /mnt-system/KNOPPIX/KNOPPIX /mnt-system/KNOPPIX/KNOPPIX.bak
    Now overwrite it with the re-mastered copy :-
    # cp /lots_of_free_space/master/KNOPPIX/KNOPPIX /mnt-system/KNOPPIX
    You can skip mounting knoppix-data.img by renaming it,
    # mv /mnt-system/KNOPPIX/knoppix-data.img /mnt-system/KNOPPI/knoppix-data.img.bak
    If using the cheatcode "debug", once done, you type "exit" to proceed with the rest of booting
    to verify that you new image is working. You will have to exit 6 times before it will be booted
    completely.

    If you are happy with it, you can shutdown and boot another time into "debug" shell, this time
    you can "restore" your knoppix-data.img and free up the space in there :-

    # mv /mnt-system/KNOPPIX/knoppix-data.img.bak /mnt-syste/KNOPPIXknoppix-data.img
    # mkdir /tmp/tmp
    # mount -o loop /mnt-systsem/KNOPPIX/knoppix-data.img /tmp/tmp
    ( Now you can delete whichever files you want to delete to free up the space, as
    they have already been transferred to the compressed image ).
    # cd /tmp/tmp
    # rm -rf bin sbin lib etc var usr .......

    Once you have finished :-
    # cd /
    # umount -d /tmp/tmp
    Again, if you are using "debug" cheatcode, you have to type 'exit' 6 times before you
    will be booted completely with the freed-up persistant store.

    In any case, the steps above involved will be somewhat tricky to the Linux newbies. I wish there
    is a cheatcode called "upgrade=/dev/sda4:/master/KNOPPIX/KNOPPIX" which will automatically
    perform the upgrade. I will leave this as an exercise for those interested to do so by modifying minirt.gz.

    Last note: the "debug" cheatcode is a very useful thing. As you can use it to rename/move the
    knoppix-data.img ( or knoppix-data.aes ) for the purpose of resizing.

  2. #2
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by kl522 View Post

    # cd ..
    # mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
    -hide-rr-moved -cache-inodes -pad /lots_of_free_space/source/KNOPPIX \
    | nice -5 /usr/bin/create_compressed_fs - 262144 > /lots_of_free_space/master/KNOPPIX/KNOPPIX

    After a long time, you will get a compressed image at /lots_of_free_space/master/KNOPPIX/KNOPPIX,
    which you can use it to replace your /mnt-system/KNOPPIX/KNOPPIX.
    Maybe I was too agressive with the blocksize 262144 ? Most web site only recommends 65536.
    I suspect blocksize 262144 might be related to some intermittent problem which I am facing.

  3. #3
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    I really appreciate that you share this!
    The first rule about remastering is, don't do it. But it's not the last rule...

    I would like to postulate a few more good (IMHO) reasons for remastering, I fully agree with yours.
    3. Bloat. The DVD contains gigabytes (uncompressed) of programs we will probably never run. While some essentials (every man his own!) are missing.
    4. Bugfixes made permanent, moved from "user space".
    5. Extra package install/updating made more systematic.
    6. Smaller total system footprint. Will always be relevant, if not so important economically as storage gets cheaper.
    7. Possible to make several cloop images. Could be useful for things like heavy development tools.
    8. Could even try other approaches than through cloop.

    I imagine that a good way of doing the full KNOPPIX upgrade, would be to copy the newly made compressed image to, for example, an external USB HD, and do the "field testing" on that. Speaking for myself, I'm pretty sure I would want to make quite a few adjustments.

  4. #4
    Senior Member
    Join Date
    Jan 2011
    Posts
    123
    I created an account just for this tutorial.
    I tried your method of remastering as I like the way it works.
    But did however made the mistake of trying to do the source/master directory on a ntfs filesystem. this caused all persmissions to be lost while copying and after the remaster nearly everything was broken.
    Of course I was going to rage on this tutorial when I realized it maybe couldve been because it was ntfs and took another memorystick and formatted it to an ext2 filesystem. this did the job

    is there a way to not break anything while still copying to a ntfs filesystem?

  5. #5
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    dinosoep wonders ...

    is there a way to not break anything while still copying to a ntfs filesystem?
    Well, yes there is. Have you noticed how Knoppix uses (compressed) loop devices a lot ? You can think of these as filesystems-in-a-file instead of filesystems-on-a-partition. You can use the same trick to create an Linux file system on an ntfs partition.

    There are four steps:

    1. Mount your ntfs partition under Knoppix (if not already mounted). That will probably be something like:

    Code:
    mount /media/sda1
    2. Create a large file on it. How big depends but for remastering the Knoppix CD let's guess 4 Gb is enough. You need something like:

    Code:
    dd bs=4K count=1M if=/dev/zero of=/media/sda1/big.one
    3. Create a file system in the big file:

    Code:
    mkfs -t ext3 -F /media/sda1/temp/big.one
    4. Mount the file as a loop back device:

    Code:
    mount -o loop /media/sda1/temp/big.one /lots_of_free_space
    You may need to sudo one or more of the commands and, of course, adapt the parameters to your particular situation. The mkfs command takes many parameters and some may be more useful in this situation than others.

  6. #6
    Administrator Site Admin-
    Join Date
    Apr 2003
    Location
    USA
    Posts
    5,441
    Quote Originally Posted by Forester View Post
    Well, yes there is. Have you noticed how Knoppix uses (compressed) loop devices a lot ? You can think of these as filesystems-in-a-file instead of filesystems-on-a-partition....
    As a moderator, I passed the above post, as it is not spam and on topic.

    However, as a user and someone who has been on this site for nearly 8 years and seen over and over again the reports of disaster after writing to NTFS, I couldn't just pass it and not warn others that it is not a universal opinion. Yea, you can create a file system within another file system under Linux. But to do so you still need to write to the physical file system. Not just to create it but during all of the writing processes. I know of no reason to think that this is safer than just writing to NTFS (which is unfortunately still not safe).

    If you want another file system on your hard disk consider shrinking the main partition, or use that "back up" partition that the vendor wasted disk space to create (after making a backup disc), and create a second Linux or FAT partition there, or even just add another inexpensive hard disk. An extra partition can be a Linux partition or a FAT partition (A FAT partition will be subjected to the 4GB-1 file size but lets you transfer files easily between Windows and Linux). But if you write to a NTFS partition, have very good backups of everything and keep making backups, the effects of corruption are often not noticed for some time after the damage is done.
    ---
    Verifying of md5 checksum and burning a CD at slow speed are important.

  7. #7
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by Harry Kuhman View Post
    However, as a user and someone who has been on this site for nearly 8 years and seen over and over again the reports of disaster after writing to NTFS, I couldn't just pass it and not warn others that it is not a universal opinion. Yea, you can create a file system within another file system under Linux. But to do so you still need to write to the physical file system. Not just to create it but during all of the writing processes. I know of no reason to think that this is safer than just writing to NTFS (which is unfortunately still not safe).
    This subject has been mentioned quite many times. 8 years of observation is quite a long time, especially in the world of IT. I am not sure how extensive your observation is substantiated with facts, as software version changes, some bad experience in the past with bugged software may not be applicable to newer versions of software.

    The fact is over the recent years, the use of NTFS (for read and write access ) in Linux has got wide spread usage and penetration. If you look at embedded devices, quite many of them are Linux based, they read/write usb flashes and SATA harddisk. I came to know quite many of the media players are Linux based. I also know that Samsung TV is Linux-based too. These devices read/write on NTFS file system (and FAT32) !

    I cannot help but to think that read/write of NTFS on Linux has gotten to a usable degree of safety - if one does not use those fancy features of NTFS. But if indeed your observation - substantiated with facts - does indicate that NTFS writing is still unsafe, perhaps it is a problem unique to Knoppix ? I am wondering ......

  8. #8
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    I think the creation of an image file, organizing a file system on it and loop-mounting that file system, may be relatively safe on NTFS. But, as NTFS is a rather swiftly and erratically moving object, I can't really understand how anyone can be quite sure everything will always go well. Even if everything works perfectly today, there is a chance that the next service pack, or even bugfix patch, may introduce problems. Therefore, I like to make a distinction between simple tasks like creating, updating and reading an image file, and more general file system use. In particular, if all reading/writing to the image file is to be done under Linux, one may mostly avoid inconsistencies etc between Windows and Linux NTFS implementations.

  9. #9
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    I agree with you that creation of an image file, organizing a file system on it and loop-mounting that file system is considered relatively safe - with a catch I shall explain below.

    I was reading the ntfs-3g write up, they mentioned this in the Wiki :-

    NTFS-3G supports all operations for writing files: files of any size can be created, modified, renamed, moved, or deleted on NTFS partitions. Transparent compression is supported, but there is no support for encryption. Support to modify access control lists and permissions is available.
    From what I read, if you want to use NTFS-3g to fix access control list and permission on a Windows system, you might not succeed because of the catchy support of ACL and permission. However, if you want to implement what Knoppix needed, ie compressed loop mount a file on NTFS, it would appear to be safe.

    On the other hand ......

    I did mention before for Knoppix 6.2.1 ( which I did not check if it is still valid after that ), it is in my humble opinion that knoppix does not shutdown the persistent file system properly, more so for FUSE. If you look at the shutdown script, basically it did not manage to unmount the UNIONFS/aufs2. The FUSE user space driver is terminated before the persistent file system got unmounted. If FUSE keeps buffer, the buffer content is it flushed ? I am not sure. Specifically I noticed when using vanilla Knoppix 6.2.1 with a ext2 persistent store loop mounted on NTFS, I noticed the system had to fsck the persistent store, and each time there is inconsistencies and it requires manual intervention. That was Knoppix 6.2.1 anyway .....

  10. #10
    Senior Member
    Join Date
    Jan 2011
    Posts
    123
    sorry for replying this late, I hoped that knoppix forums would sent me a mail when someone reacted.
    this was exactly what I was looking for
    I have been writing from linux mint to a ntfs partition a lot and I got no problems whatsoever so I think writing is getting more and more stable to ntfs

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


~OVERSTOCK~ 14

~OVERSTOCK~ 14" Lenovo ThinkPad Laptop PC: Intel i5 Dual Core Built in Webcam

$174.99



~OVERSTOCK~ 15.6

~OVERSTOCK~ 15.6" Lenovo ThinkPad Laptop: Intel i5 8GB RAM 256GB SSD Win 10

$169.99



For Lenovo ideapad Flex 5 15IIL05 15ITL05 LCD Touch Screen Bezel 5D10S39643 New picture

For Lenovo ideapad Flex 5 15IIL05 15ITL05 LCD Touch Screen Bezel 5D10S39643 New

$135.00



Lenovo Legion 5 15.6

Lenovo Legion 5 15.6" Gaming Laptop AMD R7 7735HS RTX 4060 16GB RAM 512GB SSD

$849.99



Lenovo IdeaPad Pro 5i, 16″, i5-13500H, 16 GB, 1 TB SSD, RTX 3050, 120Hz, Laptop picture

Lenovo IdeaPad Pro 5i, 16″, i5-13500H, 16 GB, 1 TB SSD, RTX 3050, 120Hz, Laptop

$709.99



Lenovo ThinkPad Yoga 11e 11.6

Lenovo ThinkPad Yoga 11e 11.6" 2in1 Touch Intel Core i3 4GB RAM 128GB SSD Win10

$58.00



Lenovo ThinkPad E560 Intel Core i5-6200U 2.3GHz 8GB RAM 500GB HDD W10P w/Charger picture

Lenovo ThinkPad E560 Intel Core i5-6200U 2.3GHz 8GB RAM 500GB HDD W10P w/Charger

$74.99



Lenovo 300e 11.6

Lenovo 300e 11.6" 2in1 Touchscreen Laptop Computer 4GB RAM 64GB SSD Windows 10

$83.99



Lenovo Notebook ThinkPad L14 AMD Gen 3 Laptop, 14

Lenovo Notebook ThinkPad L14 AMD Gen 3 Laptop, 14" FHD IPS 60Hz

$409.99



Lenovo Notebook ThinkPad T14s Gen 4 Laptop, 14

Lenovo Notebook ThinkPad T14s Gen 4 Laptop, 14" IPS 60Hz, i7-1355U,

$704.63