Results 1 to 3 of 3

Thread: Using overlayfs to re-master 64-bit Linux Mint 17.2 Xfce

  1. #1
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631

    Using overlayfs to re-master 64-bit Linux Mint 17.2 Xfce

    .
    I use a Linux LiveUSB as my everyday computing medium by choice. I prefer to maintain
    my Windows hardware unaltered, except for OEM boot media selections. In other words,
    I don't care to even dual-boot Linux and Windows on the same hardware.

    A LiveUSB with self-contained read-write persistence has served me well for years as
    a proxy for an 'installed' Linux system architecture. The down-side of this architecture
    is the volatile nature of the persistence mechanism which may easily be catastrophically
    corrupted by inadvertent user commands. Knoppix allows making iso9660/cloop-compacted
    overlays, which combine compaction and protection in that these compacted overlays are
    read-only and virtually immune to being inadvertently written-over and corrupted.
    Up until now, combining overlays with Knoppix either is either restrictive on whether base
    and overlay material 'overlaps' in use of address space, or prohibitively demanding in
    workspace required to carry out the combining of overlapped material.

    In its simplest form overlays are assumed not to refer to any storage addresses in common
    and may therefor be combined in a simple additive manner. If there is some overlap
    of common address material, simple addition is not appropriate. The best example is
    where a program in an earlier overlay is 'deleted' in a later overlay. Simple addition
    will not recover the unused space the 'deleted' program occupies.

    A more involved treatment known as re-mastering is often used to more properly join the
    net material represented as a result of the final overlay with redundant use of
    storage eliminated. This is usually achieved by first expanding any compacted overlay,
    then dealing with redundancies, and finally joining and re-compacting the resultant
    material. The workspace required to do this is often larger than twice the size
    of the original, un-compacted live system. The final result may be either larger
    or smaller than the initial configuration to be re-mastered, depending on the net
    additions and subtractions that are involved.

    Mint 17.2 uses overlayfs as a replacement for aufs to perform the necessary LiveUSB
    start-up capability of combining compact read-only base OS material with read-write
    persistence file contents. But, as an enormous bonus, with the advent of overlayfs,
    there now exists, in principle, a way to combine COMPACTED material rather than only
    EXPANDED material to achieve the effect of re-mastering. The resultant workspace
    required is much less in this instance. For example, fully within the capability
    of an 8 Gb LiveUSB to contain both persistence and the ability to re-master an identical
    8 Gb clone.

    Dav Stott has provided a blog to describe-, and a bash program using overlayfs on
    a Ubuntu LiveUSB- to carry-out an actual proof-of-principle of 'merging' compacted
    program material to achieve the same end as 're-mastering' un-compacted program
    material and in much less workspace. Dav's September 14, 2013 blog is here:
    http://davstott.me.uk/index.php/2013...stent-storage/
    Dav's March 23, 2015 updated bash program is here:
    https://github.com/davstott/ubuntu-u...aster/mergeusb

    I used Dav's program to 'merge' the basic squashed 1.5 Gb of my Mint Linux OS with
    ~400 MB net of my unique ext3 persistence additions, and one subtraction. The
    result being a fully functional new squashed 1.4 Gb tailor-made-adaptation of
    Mint 17.2 to suit my own specific needs. My one 'subtraction' was Ubiquity,
    Mint's borrowed Ubuntu installed-program installer, which I don't need.
    I provided for ~4.0 Gb of Fat32 persistence (ext3fs) in order to have enough
    workspace for this effort. Even so, there is still 2.2 Gb yet-unused Fat32
    filespace on my 8 Gb LiveUSB. More-than 4.0 Gb persistence would have exceeded
    Fat32 address space limitations, and 2.0 Gb isn't quite enough to do the job
    in this instance.

    To use Dav's program, one's Mint 17.2 LiveUSB also needs its own squashfs-tools,
    which Mint 17.2 currently doesn't otherwise need. Mint 17.2 does use overlayfs, but
    only for the usual start-up combining of the squashed Mint OS and the ext3 persistence material.
    It took 24 minutes using an exact clone of the original un-merged 64-bit Mint 17.2
    Xfce to carry-out the merge on an 8 Gb Class 10 SDHC.
    The clone used a 7.51 Gb partition of a 16 Gb USB3 SanDisk Cruzer; 7.51 Gb is
    the same size as the '8 Gb' SDHC to facilitate dd cloning.

    I've been composing all this on my first mergeusb/re-mastered Mint 17.2 LiveUSB,
    while enjoying all my favorite tweaks and additions. I have a whopping 3.6 Gb of
    unused volatile persistence to use, and a good feeling that most of my own tweaks
    and additions are now protected in read-only form, alongside a slimmed-down
    Mint 17.2 sans Ubiquity.

  2. #2
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Nice to hear that things are moving a bit in the field of union file systems. But I really don't see the big point in doing filesystem-remodeling in small spaces on flash media. Computers usually have disks, and if they don't you can always use an external USB disk. My cloop/squashfs images are usually 3.3-4.1GB, and together with persistent store, they expand to 8-12 GB when the union file system is copied to an ordinary structure. From there, a new image is created by compression, so the total extra space requirement for remastering a 4+4 GB system in this case is about 16 GB. Doesn't sound like av awful lot, and it isn't.

    As for backing up, I do remasterings so often that it is less of a point to have a good backup regime for persistent store. I just create a backup copy of that every now and then, as needed.

  3. #3
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802

    I think Knoppix could be modified to use overlayFS too

    Update: I have done quite a few remasterings lately, and the filesystem copy takes about 3-5 minutes for ca 16GB on my SSDs. So it is in no way a showstopper, but the main reason I think I will keep it even if I switch from aufs to overlayFS, is (perceived) robustness. When the persistent store is damaged, some files get lost during the copy process, but this may be fixed by reinstalling the packages the lost files belong to, or restoring them from backup. It may well happen the overlayFS merging procedure is as robust (or even better) in this respect, but that remains to be seen. I have also become very convinced that using FAT32 for system administration is a very bad idea whenever it is possible to avoid it. But for daily use, it is the opposite for me: I keep the volumes at

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


A-Tech 8GB DDR3 1600 PC3-12800 Laptop SODIMM 204-Pin Memory RAM PC3L DDR3L 1x 8G picture

A-Tech 8GB DDR3 1600 PC3-12800 Laptop SODIMM 204-Pin Memory RAM PC3L DDR3L 1x 8G

$13.99



HyperX FURY RAM DDR4 16GB 8GB 32GB 4GB 3200 2666 2400 2133 Desktop Memory DIMM picture

HyperX FURY RAM DDR4 16GB 8GB 32GB 4GB 3200 2666 2400 2133 Desktop Memory DIMM

$9.64



A-Tech 8GB PC3-12800 Desktop DDR3 1600 MHz Non ECC 240-Pin DIMM Memory RAM 1x 8G picture

A-Tech 8GB PC3-12800 Desktop DDR3 1600 MHz Non ECC 240-Pin DIMM Memory RAM 1x 8G

$13.99



CRUCIAL DDR3L 8GB 16GB 32GB 1600 MHz PC3-12800 Laptop Memory RAM SODIMM 204-Pin picture

CRUCIAL DDR3L 8GB 16GB 32GB 1600 MHz PC3-12800 Laptop Memory RAM SODIMM 204-Pin

$14.35



Team T-FORCE VULCAN Z 16GB (2 x 8GB) 288-Pin PC RAM DDR4 3200 (PC4 25600) Intel picture

Team T-FORCE VULCAN Z 16GB (2 x 8GB) 288-Pin PC RAM DDR4 3200 (PC4 25600) Intel

$35.99



A-Tech 16GB 2 x 8GB PC3-12800 Laptop SODIMM DDR3 1600 Memory RAM PC3L 16G DDR3L picture

A-Tech 16GB 2 x 8GB PC3-12800 Laptop SODIMM DDR3 1600 Memory RAM PC3L 16G DDR3L

$27.98



HMT84GL7AMR4C-RD 32GB DDR3 Server Memory RAM 14900L ECC REG 4Rx4 SK Hynix Cisco picture

HMT84GL7AMR4C-RD 32GB DDR3 Server Memory RAM 14900L ECC REG 4Rx4 SK Hynix Cisco

$13.99



A-Tech 16GB 2x 8GB PC3-12800 Desktop DDR3 1600 MHz 240pin DIMM Memory RAM 16G 8G picture

A-Tech 16GB 2x 8GB PC3-12800 Desktop DDR3 1600 MHz 240pin DIMM Memory RAM 16G 8G

$27.98



Hynix 64GB 4Rx4 PC4-2133P-L LRDIMM DDR4-17000 ECC Load Reduced Server Memory RAM picture

Hynix 64GB 4Rx4 PC4-2133P-L LRDIMM DDR4-17000 ECC Load Reduced Server Memory RAM

$64.99



A-Tech 32GB 2x 16GB PC4-25600 Laptop SODIMM DDR4 3200 MHz Non-ECC Memory RAM 32G picture

A-Tech 32GB 2x 16GB PC4-25600 Laptop SODIMM DDR4 3200 MHz Non-ECC Memory RAM 32G

$59.99