Re-mastering Proof-of-Principle Effort
.
Re-mastering will require many gigabytes to accomplish.
Interestingly, ntsf may offer some unique capabilites for re-mastering.
I chose to use an external ntsf-formatted hard drive for this purpose.
Mine is a one-terrabyte-USB-2, which matches the USB capability of my laptop.
This allows me not to hazard any interference with the Windows 7
capability of the laptop, even though its gigabytes were tempting.
I've added ntfs-3g via apt to my Knoppix 6.7 LiveUSB.
I mount my external ntfs drive 'by hand' with the command
sudo mount -t ntfs-3g /dev/sdc1 /media/sdc1 when it's on sdc1.
/etc/fstab should be modified to do this automatically, but
I've not discovered how to do that just yet.
Knoppix forum contributors Forester and Capricorny have provided us with
a programming framework on which further development may proceed.
Capricorny has provided us with a working program script which has the
following functional sequence of operations:
Preparing the workspace
Copying the current Live system
Combining the Compressed and un-compressed Elements
and Preparing an isofs representation of their re-combination
Compressing the re-combination
Re-establishing an un-compressed persistence file
Purging or cleaning-up the workspace
I would like to add the following functions as well:
Re-establish the group of programs which define
both LiveCD and LiveUSB end products and
Provide both production and and testing menu choices as
part of these end products
Current status Aug 18, 2011:
The current executable script is only about 4700 bytes.
I envision the final script may be less than 10,000 bytes.
If so, it is quite likely this may be made to fit on a CD version of Knoppix,
said version having somewhat the same generous selection of other programs
to which we have become accustomed.
I think this forum may, as a group, refine this initial script successively
until it constitutes, at a minimum, a proof of principle adequate for Knoppix's
author to endorse and consider for incorporating into a later version of
Knoppix. Likely, it may also provide useful programming that Klaus may
co-opt, as is, into his final product.
I have critiqued the original Forester/Capricorny script to have some rough edges,
in my estimation, along the following lines:
1. I initially was concerned that half the time in re-mastering
following the initial script was spent in preparing the workspace. Luckily,
kl522 proposed using a sparse-file approach to minimize this effort.
There may yet be pitfalls here, but the time saving is incredible.
2. The rsynch and subsequent print-out is excessive and should be trimmed
down to whatever might serve some useful purposes. I was able to quiet-down
the rsync effort but not the compress_isofs phase. The latter needs work.
3. The current choices of what and how to keep are way to restrictive and
too unclear for my uses. I suggest a redo that is, first of all, more
transparent. Secondly, let's make sure we can allocate what's-in and what's-
not-in the compressed file.
4. I suggest we leave it to Knoppix-itself to replace the persistence file,
and not do its work for it. Unless of course, the sparse-file approach changes
the name of the game.
5. In my case /dev/loop7 isn't found and deleted as it should be in the
purge process.
6. I suspect that the remastered product may often be larger than where
it started even if programs are removed. This is not a tolerable end solution.
Its ok to get things sorted out, but that's got to come under control at some
point.
7. One has to make sure the KNOPPIX670 file from a previous
remaster is indeed 'gone', not just hidden as PCManFM does when
you think it's deleting files.
We may need to look for hidden files and delete them.
8. Construct an architecture which efficiently provides both production
and testing of both LiveCD and LiveUSB products as menu choices for the
Live products.
9. Address a few collateral issues like the fstab problem & whether there are
some hidden minirt.gz compatibility difficulties.
10. An overall wire-brush for bloat; e.g. are all those sudos necessary.
Here's my current rem_10.sh version of the Forester/Capricorny/kl522 script:
Code:
#!/bin/bash
# Based loosely on Foresters script on Knoppix-forum modified by tay 20110511-20110810
# with touches by kl522 & utu.
function to_exist() {
[ -d "$1" ] || sudo mkdir -p $1 ;
}
function purge_or_create() {
[ -d "$1" ] && sudo rm -rf $1
sudo mkdir -p $1 ;
}
function remaster_knoppix() {
command=$1; shift;
operand=$1; shift;
case "${command} ${operand}" in
"create workspace") # Setup workspace as loop image
#sudo mount -t ntfs /dev/sdd1 /media/sdd1 (mounted prior to using this script)
workdir=$1; shift;
psize=$1; shift;
# sudo dd if=/dev/zero of=${workdir}/knoppix-remaster-data.img bs=1M count=$psize # replace with sparse-file approach
dd if=/dev/zero of=${workdir}/knoppix-remaster-data.img bs=1 count=0 seek=${wrkspc_sz}
sudo losetup /dev/loop7 ${workdir}/knoppix-remaster-data.img
sudo mkfs.ext3 /dev/loop7
sudo losetup -d /dev/loop7
purge_or_create /tmp/knx-remaster-data;
sudo mount ${workdir}/knoppix-remaster-data.img /tmp/knx-remaster-data -o loop=/dev/loop7 ;
;;
"copy live-system") # This is the simplified copy
to_exist /tmp/knx-remaster-data/knx_source ;
# Copy main /UNIONFS
sudo rsync -ax --exclude=home --exclude=lost+found --exclude=var /UNIONFS/ /tmp/knx-remaster-data/knx_source;
# Use a couple of directories/files from KNOPPIX as stubs
sudo rsync -ax /KNOPPIX/home /KNOPPIX/var /tmp/knx-remaster-data/knx_source;
sudo rsync -ax /KNOPPIX/etc/fstab /tmp/knx-remaster-data/knx_source/etc;
;;
"make isofs") # We don't use pipe here
purge_or_create /tmp/knx-remaster-data/knx_tmpiso ;
sudo chmod a+rwx /tmp/knx-remaster-data/knx_tmpiso;
sudo mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" -quiet -hide-rr-moved \
-cache-inodes -pad /tmp/knx-remaster-data/knx_source > /tmp/knx-remaster-data/knx_tmpiso/knoppix.iso ;
;;
"compress isofs") # Not optimized cloop compression
newknoppix_dir=$1; shift;
to_exist ${newknoppix_dir}
sudo create_compressed_fs -B 65536 /tmp/knx-remaster-data/knx_tmpiso/knoppix.iso $newknoppix_dir/KNOPPIX;
;;
"make squashfs")
newknoppix_dir=$1; shift;
to_exist ${newknoppix_dir}
sudo mksquashfs /tmp/knx-remaster-data/knx_source $newknoppix_dir/KNOPPIX.sq -b 262144 -noappend ;
;;
"loopcreate persistent") # Create new persistent image, size in MB must be given.
newknoppix_dir=$1; shift;
to_exist ${newknoppix_dir}
psize=$1; shift;
sudo dd if=/dev/zero of=${newknoppix_dir}/knoppix-data.img bs=1M count=$psize
sudo losetup /dev/loop6 ${newknoppix_dir}/knoppix-data.img
sudo mkfs.ext3 /dev/loop6
purge_or_create /tmp/knx-data
sudo mount -t ext3 -rw -o loop /dev/loop6 /tmp/knx-data ;
sudo rsync -ax /UNIONFS/home /UNIONFS/var /tmp/knx-data ;
sudo umount /tmp/knx-data;
sudo losetup -d /dev/loop6
;;
"purge workspace")
sudo umount /tmp/knx-remaster-data
sudo losetup -d /dev/loop7
workdir=$1; shift;
sudo rm -f ${workdir}/knoppix-remaster-data.img
;;
*)
echo oops;
;;
esac
}
echo '...Starting the re-mastering process...'
# Calling examples:
#./rem_03.sh /media/sdc1 15000 /media/sdc1/KNOPPIX670&
#./rem_04.sh /media/sdc1 15000 /media/sdc1/KNOPPIX670&
#./rem_05.sh /media/sdc1 15G /media/sdc1/KNOPPIX670&
#./rem_10.sh /media/sdc1 15G /media/sdc1/KNOPPIX670&
wrkspc_dir=$1 ; wrkspc_sz=$2 ; remaster_dir=$3 ; persist_sz=$4 ;
echo -e 'Set-up workspace ******************************** \c'; date
remaster_knoppix create workspace ${wrkspc_dir} ${wrkspc_sz}
echo -e 'Copy live-system ******************************** \c'; date
remaster_knoppix copy live-system
echo -e 'Make isofs ************************************** \c'; date
remaster_knoppix make isofs
echo -e 'Compress isofs ********************************** \c'; date
remaster_knoppix compress isofs ${remaster_dir}
# echo -e 'Make squashfs ********************************** \c'; date
# remaster_knoppix make squashfs ${remaster_dir}
# echo -e 'Create persistent image ************************ \c'; date
# remaster_knoppix loopcreate persistent ${remaster_dir} ${persist_sz}
echo -e 'Purge workspace ********************************* \c'; date
remaster_knoppix purge workspace ${wrkspc_dir}
echo -e 'All done **************************************** \c'; date
exit 0
Sample to follow output on next post.
.
...And a few on media and booting
For serious daily Knoppix use, a good persistent store is mandatory, and that means >2 GB for me. Just upgrading/book-keeping stuff easily reaches 1 GB, and I normally go for 4 GB at once. The CD is stripped down, to the point that something important for some uses is usually lacking, therefore, I start with the DVD version whenever I can. Total system size therefore becomes ca 8 GB. But that also means I can't place the remastered version on a CD/DVD. And why should I?
I run Knoppix from internal HDD, USB HDD, (fast) memory sticks, SD cards, smartphone - seldom CD/DVD. These have the space I need, and the KNOPPIX directory (including persistent store(s)) is easily copied.
Booting? Thus far, I still prefer legacy GRUB (not the "new" thing) installed on its own partition, where I can add boot alternatives just by adding kernels and initrds and editing /boot/grub/menu.lst. Only limitation is with Win7 - I don't need it enough to chase the workarounds needed to Linux boot it, relying instead on virtual machines. I may turn to GRUB4DOS later.
Totally different approaches, for different purposes IMHO
This approach and Werner's are almost diametrically opposite approaches to remastering, and suitable for very different purposes. And if "remastering" means creating a new ISO image for CD/DVD, this is of course not remastering at all.
But, in my understanding, remastering means essentially recompressing a modified UNIONFS, possibly together with other modifications. That's what I and a few other users need, and provided the necessary boot files are present, copy/backup by the built-in Install Knoppix to flash disk can be used.
As for virtual machines, I have found qemu very useful for checking the remastering, but personally, I don't really need them when I don't modify minirt.gz. Nor have I ever needed a hard disk install for remastering. That was necessary to build a pure 64 bits Knoppix version in a straightforward way, but that's a different story.
May be a very useful idea
Yes, with the present DVD image size, in order to stick to the 4GB limit for cloop KNOPPIX, we have to remove programs to make room for substantial additions. (In my case, R, VMware Workstation and Oracle XE are the most important ones.) Instead of doing this, which is getting harder and harder to achieve while still keeping functionality intact, using the ability to chain cloop files can be very useful. I think that the main reason it has not been exploited more, is that it is kind of "third level" extension: First, the persistent store caters for most needs, second, creating a new primary cloop image suffices for most of the rest. I'm rather doubtful as to the usefulness of implementing version upgrades this way. Even within one release, it quickly becomes an hassle to keep it upgraded, and I guess this will be even worse across versions.
My current two cents worth:
.
I use a 2 Gb Knoppix 6.7.1 LiveUSB as my mainstay OS.
I am not a programmer, but a frequent user.
My uses include internet browsing, e-mail and maintenance of some
personally useful spreadsheets with on-line information.
The CD size of Knoppix with its overlay process suits me very well,
with one exception: that is, when it comes to upgrading some its
larger elements like LibreOfffice or IceWeasel. Re-mastering is the
only really effective way I see to maintain a small end-product. However,
for my particular end uses, re-mastering requires too much additional
resource outlay, and frankly, getting it done exceeds
my short attention span.
I've tried several virtual system aproaches, and am not really
comfortable with any I've tried.
I expect I might be able to pull together an 'in situ' menu-item re-
mastering from available on-line material, but it would require most
of a 16 Gb usb. That would seem to be an absurdly wasteful
use of resources IMO.
Some other distros are going to minimalist base distributions as one
way of getting around my sort of problem. More frequent Knoppix upgrades
would also suffice, but KK has a day job and a family. I suppose he
might delegate some of his distro maintenance tasks if someone
competent were to propose something appropriate.
Remastering: Can it be made easy?
1. I think remastering can be made easy "enough", I will post an updated version of the remastering script used here as soon as I have tested and honed it a little more, and checked it for CD version use. I made a complete remastering of 6.7.1 DVD in about 70 minutes with it (running that cloop now), but slow file copying took a lot of the time.
2. Remastering may be the simplest way to contribute to Knoppix development, and creating and posting ISOs may be useful for that purpose. Wrt bandwidth use, posting scripts may, however, be vastly more effective ;-) But I would definitely download a pure 64-bits remastering of 6.7.1 instead of creating one myself. (My 64-bits version of Knoppix is 6.4.4-based and therefore "obsolete"..) Creating further cloops might also be a solution, a cloop with ca 2GB programs would be about CD image size.
3.
Quote:
I expect I might be able to pull together an 'in situ' menu-item re-
mastering from available on-line material, but it would require most
of a 16 Gb usb. That would seem to be an absurdly wasteful
use of resources IMO.
In principle of course yes - but a 16GB USB3 stick costs me about $40, so these resources can hardly be called very expensive. I also see no reason for using an USB stick for remastering if HD real estate is available, USB or otherwise. The space requirement is only for temporary use, and I use a 20 GB loopfile for full DVD remastering. With 64-bits systems and more RAM, ramdisks may also be used for this - I have 16 GB system RAM in the machine I'm posting from, more than enough for CD remastering to ramdisk.
A $19 re-mastering 'appliance'
.
This is a status report on my experience with an adaptation of Werner's kn-recombine.
kn-recombine is an excellent and generalized algorithm for re-mastering Knoppix LiveUSBs.
My adaptation seeks to further simplify the use of kn-recombine specific to my particular
use of the algorithm. My end use is re-mastering 2 Gb LiveCD-size LiveUSbs with persistence.
I choose not to utilize the ntfs fields of the Windows 7 capability but rather to perform
all the necessary manipulations via external USBs attached to my laptop. Recent cost
reductions and speed increases of USB chips has now made this choice inexpensive & viable.
For $19 one can now create a re-mastering 'appliance' for CD-size Knoppix Flash Drives.
I use a 16 Gb USB card with a Class 10 rating. This seems to be 15 Gb formatted Fat32
initially. I use Gparted to re-partition this to one 1.84 Gb Fat32 partition, and then
create a second partition with the remainder and format that to ext3. I clone my current
2 Gb LiveUSB and its persistence on the Fat32 partition and install Werner's kn-recombine
on the ext3 partition. 1.84 Gb is the net size of my 2 Gb LiveUSBs.
Working with the 16 Gb unit's LiveUSB I find Werner's algorithm will re-master my LiveUSB
in about 41 minutes. In this 'appliance' use of Werner's algorithm, when it comes time to
write to a new flash-disk, the first prompt requests the location of the re-mastered
KNOPPIX file. Don't anticipate (as I did at first) the prompt to ask rather for the
location of the flash-disk to be modified. With a minor re-do of Werner's algorithm,
I have hopes that perhaps the partition location dialog may be further simplified, and
even possibly omitted in an 'appliance' adaptation of the algorithm.
As a bonus, the appliance serves as a backup, it may be noted. Also, my 2 Gb LiveUSBs
are on Class 2 USB cards; I would expect Classes less than 10 may be much too slow
for the remastering use of such a 16 Gb 'appliance'.
Some notes on kn-recombine
.
My UNIONFS is about 2.3 Gb. The new stuff in KNOPPIX-DATA is probably
about about 500 Mb. Compressed KNOPPIX grows from 722 to 953 Mb in
39 minutes during re-mastering with kn-recombine.
The rsync phase of remastering with kn-recombine takes about 22 minutes
with my class 10 LiveUSB, 2.10 Ghz dual cpus & 32-bit kernel.
This is about the same as the result of cp -r UNIONFS /media/sdc2/knx.
Perhaps the rsync phase can be speeded up by first pre-loading recombine*
by expanding the un-re-mastered compressed KNOPPIX into that space before
finishing it off with an rsync againts UNIONFS. Otherwise, rsync is never
getting any benefit out of working on small differences.
I am also wondering, as another alternative to kn-recombine, if it might
be possible to just compress UNIONFS directly to compressed KNOPPIX at
/media/sdc2/knx without mirroring it first to recombine*.
* recombine is a folder (or directory) in /media/sdc2/knx in the
statements above.
A note about the 'lame duck':
'
The 'lame duck' may have the potential of a self-contained
re-mastering capability.
It doesn't need external storage nor another OS to complete
the re-mastering. After the kn-recombine rsync phase,
dismount the 'duck and reboot IT with the option 'noimage'
and proceed with the kn-recombine write operation.
This may require a slight tweak of kn-recombine to allow this,
but it should work since none of the files being re-arranged
are in use on a noimage reboot.
Here's where I think we are:
.
I began this thread some months ago, bemoaning the high cost of
Gbs necessary to the needs of re-mastering. I had hoped for some
algorithm that would somehow magically perform this neat operation
on my meager 2 Gb LiveUSB.
Aside from the obvious, of acquiring the necessary Gbs outright,
over time, a number of our senior users have come to a consensus
that there are several ways to tap into the otherwise unused NTFS
real estate most of us have to get around 'the high cost of Gbs'.
At least three such approaches have been used and advocated:
1. Re-partitioning a Windows drive to create a new partition;
2. Using VirtualBox or equivalent virtual installs on either
...Windows or another Linux system; and.
3. Reading & writing to an out-of-service NTFS partition.
Surprisingly to me, over this time, the cost of LiveUSBs has
come down so much that an 8 Gb LiveUSB Class 10 can now be
obtained at a cost LESS than the cost of a 2 Gb Class 2 LiveUSB
when this effort began. Class10 LiveUSBs are now a viable and
inexpensive alternative for re-mastering CD-size Knoppix LiveUSBs.
Also, over this time, Werner Schulz has given us at least two
useful methods to carry out practical schemes of re-mastering
CD-size Knoppix LiveUSBs:
1. Using Virtual box (or equivalent) and Knoppix's built-in Own
outine to create a virtual system within which to develop and
finalize a custom LiveUSB; and
2. A generalized algorithm, kn-recombine that can be utilized in
several ways to recombine an initial compressed Knoppix image with
un-compressed Knoppix-data image to create a final compressed
Knoppix imgage containing both the contents of both the original
and that of the un-compressed data changes.
I believe Werner has nailed the re-mastering Proof-of-Principle
for CD-size Knoppix LiveUSBs with his algoritm, kn-recombine.
kn-recombine leaves it up to the user to specify where and in
what form to locate the necessary Gbs. Find kn-recombine at:
http://www.wp-schulz.de/knoppix/recombine.html
And thanks, Werner; well done.
