Results 1 to 3 of 3

Thread: Using overlayfs to re-master 64-bit Linux Mint 17.2 Xfce

  1. #1
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631

    Using overlayfs to re-master 64-bit Linux Mint 17.2 Xfce

    .
    I use a Linux LiveUSB as my everyday computing medium by choice. I prefer to maintain
    my Windows hardware unaltered, except for OEM boot media selections. In other words,
    I don't care to even dual-boot Linux and Windows on the same hardware.

    A LiveUSB with self-contained read-write persistence has served me well for years as
    a proxy for an 'installed' Linux system architecture. The down-side of this architecture
    is the volatile nature of the persistence mechanism which may easily be catastrophically
    corrupted by inadvertent user commands. Knoppix allows making iso9660/cloop-compacted
    overlays, which combine compaction and protection in that these compacted overlays are
    read-only and virtually immune to being inadvertently written-over and corrupted.
    Up until now, combining overlays with Knoppix either is either restrictive on whether base
    and overlay material 'overlaps' in use of address space, or prohibitively demanding in
    workspace required to carry out the combining of overlapped material.

    In its simplest form overlays are assumed not to refer to any storage addresses in common
    and may therefor be combined in a simple additive manner. If there is some overlap
    of common address material, simple addition is not appropriate. The best example is
    where a program in an earlier overlay is 'deleted' in a later overlay. Simple addition
    will not recover the unused space the 'deleted' program occupies.

    A more involved treatment known as re-mastering is often used to more properly join the
    net material represented as a result of the final overlay with redundant use of
    storage eliminated. This is usually achieved by first expanding any compacted overlay,
    then dealing with redundancies, and finally joining and re-compacting the resultant
    material. The workspace required to do this is often larger than twice the size
    of the original, un-compacted live system. The final result may be either larger
    or smaller than the initial configuration to be re-mastered, depending on the net
    additions and subtractions that are involved.

    Mint 17.2 uses overlayfs as a replacement for aufs to perform the necessary LiveUSB
    start-up capability of combining compact read-only base OS material with read-write
    persistence file contents. But, as an enormous bonus, with the advent of overlayfs,
    there now exists, in principle, a way to combine COMPACTED material rather than only
    EXPANDED material to achieve the effect of re-mastering. The resultant workspace
    required is much less in this instance. For example, fully within the capability
    of an 8 Gb LiveUSB to contain both persistence and the ability to re-master an identical
    8 Gb clone.

    Dav Stott has provided a blog to describe-, and a bash program using overlayfs on
    a Ubuntu LiveUSB- to carry-out an actual proof-of-principle of 'merging' compacted
    program material to achieve the same end as 're-mastering' un-compacted program
    material and in much less workspace. Dav's September 14, 2013 blog is here:
    http://davstott.me.uk/index.php/2013...stent-storage/
    Dav's March 23, 2015 updated bash program is here:
    https://github.com/davstott/ubuntu-u...aster/mergeusb

    I used Dav's program to 'merge' the basic squashed 1.5 Gb of my Mint Linux OS with
    ~400 MB net of my unique ext3 persistence additions, and one subtraction. The
    result being a fully functional new squashed 1.4 Gb tailor-made-adaptation of
    Mint 17.2 to suit my own specific needs. My one 'subtraction' was Ubiquity,
    Mint's borrowed Ubuntu installed-program installer, which I don't need.
    I provided for ~4.0 Gb of Fat32 persistence (ext3fs) in order to have enough
    workspace for this effort. Even so, there is still 2.2 Gb yet-unused Fat32
    filespace on my 8 Gb LiveUSB. More-than 4.0 Gb persistence would have exceeded
    Fat32 address space limitations, and 2.0 Gb isn't quite enough to do the job
    in this instance.

    To use Dav's program, one's Mint 17.2 LiveUSB also needs its own squashfs-tools,
    which Mint 17.2 currently doesn't otherwise need. Mint 17.2 does use overlayfs, but
    only for the usual start-up combining of the squashed Mint OS and the ext3 persistence material.
    It took 24 minutes using an exact clone of the original un-merged 64-bit Mint 17.2
    Xfce to carry-out the merge on an 8 Gb Class 10 SDHC.
    The clone used a 7.51 Gb partition of a 16 Gb USB3 SanDisk Cruzer; 7.51 Gb is
    the same size as the '8 Gb' SDHC to facilitate dd cloning.

    I've been composing all this on my first mergeusb/re-mastered Mint 17.2 LiveUSB,
    while enjoying all my favorite tweaks and additions. I have a whopping 3.6 Gb of
    unused volatile persistence to use, and a good feeling that most of my own tweaks
    and additions are now protected in read-only form, alongside a slimmed-down
    Mint 17.2 sans Ubiquity.

  2. #2
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Nice to hear that things are moving a bit in the field of union file systems. But I really don't see the big point in doing filesystem-remodeling in small spaces on flash media. Computers usually have disks, and if they don't you can always use an external USB disk. My cloop/squashfs images are usually 3.3-4.1GB, and together with persistent store, they expand to 8-12 GB when the union file system is copied to an ordinary structure. From there, a new image is created by compression, so the total extra space requirement for remastering a 4+4 GB system in this case is about 16 GB. Doesn't sound like av awful lot, and it isn't.

    As for backing up, I do remasterings so often that it is less of a point to have a good backup regime for persistent store. I just create a backup copy of that every now and then, as needed.

  3. #3
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802

    I think Knoppix could be modified to use overlayFS too

    Update: I have done quite a few remasterings lately, and the filesystem copy takes about 3-5 minutes for ca 16GB on my SSDs. So it is in no way a showstopper, but the main reason I think I will keep it even if I switch from aufs to overlayFS, is (perceived) robustness. When the persistent store is damaged, some files get lost during the copy process, but this may be fixed by reinstalling the packages the lost files belong to, or restoring them from backup. It may well happen the overlayFS merging procedure is as robust (or even better) in this respect, but that remains to be seen. I have also become very convinced that using FAT32 for system administration is a very bad idea whenever it is possible to avoid it. But for daily use, it is the opposite for me: I keep the volumes at

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Dell R640 8x 2.5

Dell R640 8x 2.5" SFF Server iDRAC ENT - Wholesale Custom Build Your Server

$328.99



Dell Poweredge R630 Server 2x E5-2620 V4 =16 Cores | S130 | 32GB RAM | 2x trays picture

Dell Poweredge R630 Server 2x E5-2620 V4 =16 Cores | S130 | 32GB RAM | 2x trays

$165.99



Dell PowerEdge R630 Server 2x E5-2640v3 2.60Ghz 16-Core 64GB H330 picture

Dell PowerEdge R630 Server 2x E5-2640v3 2.60Ghz 16-Core 64GB H330

$195.00



Dell PowerEdge R630 Server | 2x Xeon  E5-2667v4, 128GB, RAM, H730, Dual 750w PSU picture

Dell PowerEdge R630 Server | 2x Xeon E5-2667v4, 128GB, RAM, H730, Dual 750w PSU

$229.00



New Gigabyte 10 bay SFF Server 2x Xeon Gold 6130 Total 32 Core 64GB DDR4 3x PCIe picture

New Gigabyte 10 bay SFF Server 2x Xeon Gold 6130 Total 32 Core 64GB DDR4 3x PCIe

$319.20



DELL POWEREDGE C6100 with 4x Nodes 8x Xeon X5660 CPU 96GB RAM no HDD WORKING picture

DELL POWEREDGE C6100 with 4x Nodes 8x Xeon X5660 CPU 96GB RAM no HDD WORKING

$118.99



Dell PowerEdge R630 Server 2x E5-2690v3 2.60Ghz 24-Core 128GB H330 Rails picture

Dell PowerEdge R630 Server 2x E5-2690v3 2.60Ghz 24-Core 128GB H330 Rails

$288.85



Dell Poweredge R730xd LFF 14-Bay 2U Server | Choose Your CPU & RAM Config picture

Dell Poweredge R730xd LFF 14-Bay 2U Server | Choose Your CPU & RAM Config

$444.99



Dell PowerEdge R430 w/ 2x E5-2640v4, 64GB, NO RAID picture

Dell PowerEdge R430 w/ 2x E5-2640v4, 64GB, NO RAID

$149.99



New 48Core 256G 2x 960GB SSD Gigabyte Server Intel Platinum 8160 3 year Warranty picture

New 48Core 256G 2x 960GB SSD Gigabyte Server Intel Platinum 8160 3 year Warranty

$699.00