fusecompress on knoppix-data.img?

[dinosoep]

I was looking around and found fusecompress, does anyone has experience with this?
Later this evening or tomorrow I'll try to mnt my knoppix-data.img with fusecompress and see how it works out.

are there better alternatives out there?

[dinosoep]

and another question (as I'm worried about my usb lifespan) can I mount knoppix-data.img read only at startup with /ramdisk mount on top of unionfs and when I feel the need "flush" the changes by remounting knoppix-data.img read-write and syncing the changes?
and do this at shutdown?
if so, could one help me a bit with this? I'm not very smart :p

[Capricorny]

I don't know anything about fusecompress, but I think the USB wear issue is significant. Entries in the /home directory are updated all the time. It seems that we may mount /home separately with the present standard setup, but normally, /var is also frequently updated. So at least those two, plus eventually parts of /etc should go somewhere wear-proof. For instance on a ramdisk that is initiatied at boot and written to persistent store at shutdown.

It should not be too hard to achieve this, but as I have no recent experience with ramdisks, I will only point to the possibility.

From 6.4.4 cheatcodes, we have the following choices on boot:


### Configuration / Persistent image ###
knoppix nonetworkmanager Don't start network manager
knoppix home=/dev/sda1/knoppix.img Mount loopback file for overlay
knoppix toram Copy to RAM and run from there
knoppix tohd=/dev/sda1 Copy to Harddisk and run from there
knoppix fromhd=/dev/sda1 Boot from previously copied CD-Image
knoppix bootfrom=/dev/sda1/KNX.iso Access image, boot from ISO-Image. ***)
knoppix knoppix_dir=KNOPPIX Directory to search for on the CD.
knoppix knoppix_name=KNOPPIX Cloop-File to search for on the CD.
knoppix noswap Don't use existing swap partitions
knoppix forensic Don't use swap and mount read-only
knoppix secure Disable root access
knoppix noimage Do NOT use persistent image



Maybe we could have a new cheatcode "wearproof", that copies frequently updated directories like /home and /var to ramdisk and runs from there?

[dinosoep]

Exactly what I'm going to try.
I think that could be achieved by mounting knoppix-data.img read only, create a new ramdisk about the same size as knoppix-data.img and mount it read-write over /KNOPPIX-DATA. Then everytime you run a certain script or shutdown knoppix-data.img get's remounted read-write and all changes are copied over to knoppix-data.img...

To bad I'm not good enough :p
First I'm going to try and make fusecompress work, then I'll play around with the ramdisk stuff

[Capricorny]

Exactly what I'm going to try.
I think that could be achieved by mounting knoppix-data.img read only, create a new ramdisk about the same size as knoppix-data.img and mount it read-write over /KNOPPIX-DATA.

You surely describe the easy way, but my knoppix-data.img is 4GB... ;-)
So in my case, there must be some extracting and updating of selected directories involved.

[dinosoep]

with unionfs only the changes you try to make to the read-only filesystem will be stored on the ramdisk, So there will be no extraction of files on from knoppix-data.img to the ramdisk unless you edit them.

[Capricorny]

with unionfs only the changes you try to make to the read-only filesystem will be stored on the ramdisk, So there will be no extraction of files on from knoppix-data.img to the ramdisk unless you edit them.
In that case, it would be enough to setup a, say, 200MB ramdisk to accomodate the changes made during an ordinary session? Not knowing UNIONFS, I thought the union process could not be iterated beyond the cloop and persistent image, but it can? As for knoppix-data.img, a simple scheme could then be to have two images mounted alternatingly, so at shutdown, the actual UNIONFS directories are written to the backup image, which is then used for booting the next time? Then we would stay entirely within live UNIONFS during the whole process, like we are in remastering.

[dinosoep]

It's going great, at the moment I have a fully functional system forgetting everything at shutdown :p
No really, I managed to make it create a ramdisk (currently of 4 gigs max, I know it's ridiculous) and make knoppix-data.img read-only
There is a script out there (I forgot the name) that can sync that ramdisk to knoppix-data.img, I just need to put it in a cron job.

Having started up chromium, a couple of tabs, conky,... I'm using 219 mb of ram with only 26 mb used by the ramdisk :D
i'll keep everyone informed

[Capricorny]

...I managed to make it create a ramdisk (currently of 4 gigs max, I know it's ridiculous) and make knoppix-data.img read-only
There is a script out there (I forgot the name) that can sync that ramdisk to knoppix-data.img, I just need to put it in a cron job.


I have had a quick look at some of the unionfs documentation.
http://www.fsl.cs.sunysb.edu/project-unionfs.html. In particular the presentation from the 2006 Ottawa symposium.
It seems clear that mounting knoppix-data.img read-only is quite OK, Only the highest priority branch must be mounted rw to create a writable system, and it only has to be big enough to accomodate the updates/additions to the file system. But I wonder what kind of mounting gymnastics must be performed if the persistent, read-only-mounted image is to be updated in a cronjob on a running system?

[dinosoep]

Knoppix uses aufs2, this is like a more improved version of unionfs (according to kl522 I don't know enough to compare both of them).
You can specify branches and mount them over eachother and also specify the operations that can be performed on them WITHIN the unionfs space.
So say I have a writeable directory A and another B, then I can mount A over B and say that A is read-only while in fact it is read-write.

This way I can make it save all unionfs changes to the ramdisk and also run another script to bypass the unionfs read-only limitation and that script exists, it's called "aubrsync".

This all sounded too good to be true and in fact it is :( there is one catch and that is that aufs-tools need to be installed and be the same version as the one that was used to patch the kernel. Too bad I have no idea what that is...

I am currently trying to compile the newest version without succes, I'll continue my quest tomorrow :D

[kl522]

I am currently trying to compile the newest version without succes, I'll continue my quest tomorrow :D

Instead of compiling, have you tried just install aufs-tools from synaptics ?

What is the error when you try to compile it ?
Do you have /usr/include/linux/aufs_type.h on your system ?

[Capricorny]

I just wonder why special tools should be needed if the semantics in aufs2 is as advertised? If knoppix-data.img and ramdisk are both mounted read-write, shouldn't it be possible just to rsync ramdisk onto knoppix-data.img?

BTW, I have another use for aufs: When using databases and web servers, we may use a lot of space, not easily accomodated on the persistent image, but standard package installation will place them there nonetheless. Instead of changing installation data, or introduce symlinks, we could simply move data to persistent image versions, and union mount those with larger store versions. We could then union merge the versions, and, eventually, create dummy stores for use when larger stores are not mounted. I think of something like:



mount -t aufs -o dirs=/var/lib/mysql_ps:/store/var/lib/mysql none /var/lib/mysql
mount -t aufs -o dirs=/var/www_ps:/store/var/www none /var/www

[Capricorny]

In the sourceforge mailing list for aufs, the main developer recently posted the following:
http://sourceforge.net/mailarchive/forum.php?thread_name=12978.1303711403%40jrobl&forum_name=aufs-users


And more importantly, the aufs module in debian is totally broken. Last
time I saw aufs in debian, they use TOO OLD version which is obsoleted
and not maintained. Additionally they made their original changes. It
means I cannot support aufs in debian.

Maybe the safest thing is to get the source off sourceforge and proceed?

[utu]

@ Capricorny

I think you should post this on the debian-knoppix list,
with attn to Klaus Knopper. Sounds serious.

http://lists.debian.org/debian-knoppix/2011/06/

[kl522]

Maybe the safest thing is to get the source off sourceforge and proceed?
If you are talking about aufs kernel module for KNOPPIX, there is nothing to worry, Klaus K would have compiled the right version for it. If it had been the wrong version, it wouldn't compile with newer kernel and it won't run. It would be noticed !

By contrast the debian version might be old and faulty, because a typical debian user won't use aufs, so even if it is old and faulty, it would go unnoticed.

If you are talking about aufs-utils or aufs-tools, then I can't say anything about the debian version, because in a typical use, the aufs-utils are not used at all. It would not be surprising if the package faulty.

[kl522]

I just wonder why special tools should be needed if the semantics in aufs2 is as advertised? If knoppix-data.img and ramdisk are both mounted read-write, shouldn't it be possible just to rsync ramdisk onto knoppix-data.img?
Aufs is more than just adding new files and modifying existing files. It is also able to "delete" or "hide" an existing files or directory. There is certain encoding or protocol to fully represent the modified file system structure. And therefore it is better to use the tools provided than to re-invent the wheel.

[Capricorny]

Aufs is more than just adding new files and modifying existing files. It is also able to "delete" or "hide" an existing files or directory. There is certain encoding or protocol to fully represent the modified file system structure. And therefore it is better to use the tools provided than to re-invent the wheel.

If you have to modify the whole structure, definitely yes. But is that really necessary for this particular use?

If we do something like


mount -t aufs -o dir=ramdisk=rw:persistent_image=rw:KNOPPIX=ro none /unionfs


What would go wrong if we did (evt. as a cron job)


rsync -ax ramdisk/ persistent_image


I just tested this with a couple of directories with a few files. Seemed to work in that case, at least.

[dinosoep]

@kl522 the script I use needs to have aufs-utils the same version as the aufs patch-version, when I tried with the one from synaptic it gave me:


"aubrsync move_with_wh /UNIONFS /KNOPPIX-DATA-RAMDISK /KNOPPIX-DATA"
and that gives me the following errors:
+ mount -o remount,ro,udba=reval,noshwh /UNIONFS
/sbin/mount.aufs:plink.c:223: AUFS_CTL_PLINK_MAINT: Inappropriate ioctl for device
+ mount -o remount,udba=reval,noshwh,rw,relatime,si=aa3aa37c, noplink /UNIONFS
/sbin/mount.aufs:plink.c:223: AUFS_CTL_PLINK_MAINT: Inappropriate ioctl for device


when I tried compiling from source it gave me:


install -d /usr/share/man/man5
install -o root -g root -p -m 644 aufs.5 /usr/share/man/man5
install -d /sbin
install -o root -g root -p -m 755 mount.aufs umount.aufs auplink /sbin
install: cannot stat `umount.aufs': No such file or directory
make: *** [install_sbin] Error 1

The maintainer suggested updating the kernel module which I'm going to try this evening :D


@Capricorny
The problem with rsync here is that when you delete a file, it should create a whiteout on the writeable upper filesystem and rsync doesn't sync those, only the real files get synced. (thats at least what I think, why else would they've created aubrsync?)

[Capricorny]

The problem with rsync here is that when you delete a file, it should create a whiteout on the writeable upper filesystem and rsync doesn't sync those, only the real files get synced. (thats at least what I think, why else would they've created aubrsync?)

Exactly. And I would ask: Why would they have to use aubrsync? My (implicit) point was that file creatons and updates are taken care of through rsync, while deletions must be done directly on the persistent image, preferably upon shutdown. Of course, it would be nice to have a "commit" function doing the actual deletions, determined from the ramdisk-recorded changes (whiteouts), but I will rather have something simple, stupid and robust than something that nobody seems to be able to tell how and why works or not.

[dinosoep]

I have no idea how I could make that and aubrsync sounds like a working solution.
I have not the smallest idea on how I can update the aufs module in my kernel to the latest version without recompiling everything. or compile anything for that mather from aufs git without getting errors I'm hoping someone who does understand a bit about that would help me out :D

for the intersted persons, here is the "nowear" initrd.gz
Everyone can try their own scripts and see if they get the sync working :p
http://dl.dropbox.com/u/15024434/initrd_nowear.gz

[Capricorny]

If you are talking about aufs kernel module for KNOPPIX, there is nothing to worry, Klaus K would have compiled the right version for it. If it had been the wrong version, it wouldn't compile with newer kernel and it won't run. It would be noticed !

By contrast the debian version might be old and faulty, because a typical debian user won't use aufs, so even if it is old and faulty, it would go unnoticed.

If you are talking about aufs-utils or aufs-tools, then I can't say anything about the debian version, because in a typical use, the aufs-utils are not used at all. It would not be surprising if the package faulty.

May I suggest that you seem very optimistic about the situation?

First, aufs is a general tool, and it is used "everywhere", not only in Knoppix. The message I linked to was in response to a Debian user. A "typical" Debian user may not need it, true. But many essential libraries are never required by "typical" users, still they are essential.

Second, IF Klaus K has compiled the right version for aufs, then he has done a (rare) departure from his (recent?) general Debian compliance. Which would probably mostly be regarded as a hack, since it is not in sync with utils and tools, which we frequently need when we will modify or check something.

Third, I tend to believe the creator and maintainer of aufs. And the aufs relation to the mainline kernel is - well - "undecided" at best, see for instance the main sourceforge page (http://aufs.sourceforge.net/ ) :


Note: it becomes clear that "Aufs was rejected. Let's give it up." According to Christoph Hellwig, linux rejects all union-type filesystems but UnionMount.

So, as I interpret this, the last thing we should expect, is any kind of automagical updating and integration of aufs.

[kl522]

[/code]when I tried compiling from source it gave me:


install -d /usr/share/man/man5
install -o root -g root -p -m 644 aufs.5 /usr/share/man/man5
install -d /sbin
install -o root -g root -p -m 755 mount.aufs umount.aufs auplink /sbin
install: cannot stat `umount.aufs': No such file or directory
make: *** [install_sbin] Error 1
The maintainer suggested updating the kernel module which I'm going to try this evening :D


I have more confidence on the installed aufs kernel module than your compilation. :)

Your compilation output seems to suggest that umount.aufs is not present. But 'umount.aufs' is to be provided by YOUR compilation, so it is still more like your compilation has error. Furthermore the output is coming from 'make install', you should have an error free 'make' prior to 'sudo make install'. Perhaps you have a wrong version of /usr/include/linux/aufs_type.h installed on knoppix.

[kl522]

May I suggest that you seem very optimistic about the situation?
Yes I am. Because I update my kernel very often, each time I have to match it against the correct version of aufs. I have compiled aufs kernel module too many times already.



First, aufs is a general tool, and it is used "everywhere", not only in Knoppix. The message I linked to was in response to a Debian user. A "typical" Debian user may not need it, true. But many essential libraries are never required by "typical" users, still they are essential.I don't know what "essential" libraries you are talking about. A typical aufs user does not even use the aufs tools. Even knoppix does not use the aufs tools ! Only in rare occasions that you need to use the aufs utils or tools.

[dinosoep]

Yeah, Make was succesfull ...
Are you able to compile it succesfully?

[kl522]

Yeah, Make was succesfull ...
Are you able to compile it succesfully?

You may have to re-read your 'make' output more carefully. Here is my output :-


cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" ver.c -o ver
./ver
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" c2tmac.c -o c2tmac
rm -f aufs.5
./c2tmac > aufs.5
awk '{ \
gsub(/\140[^\047]*\047/, "\\[oq]&\\[cq]"); \
gsub(/\\\[oq\]\140/, "\\[oq]"); \
gsub(/\047\\\[cq\]/, "\\[cq]"); \
gsub(/\047/, "\\[aq]"); \
print; \
}' aufs.in.5 >> aufs.5
chmod a-w aufs.5
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o proc_mnt.o proc_mnt.c
ar rv libautil.a proc_mnt.o
ar: creating libautil.a
a - proc_mnt.o
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o br.o br.c
ar rv libautil.a br.o
a - br.o
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o plink.o plink.c
ar rv libautil.a plink.o
a - plink.o
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o mtab.o mtab.c
ar rv libautil.a mtab.o
a - mtab.o
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o auibusy.o auibusy.c
cc -static -s auibusy.o -L. -lautil -o auibusy
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o auplink.o auplink.c
cc -static -s auplink.o -L. -lautil -o auplink
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o mount.aufs.o mount.aufs.c
cc -static -s mount.aufs.o -L. -lautil -o mount.aufs
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" -c -o umount.aufs.o umount.aufs.c
cc -static -s umount.aufs.o -L. -lautil -o umount.aufs
cc -I./libau -O -Wall -DMOUNT_CMD_PATH=\"\" c2sh.c -o c2sh
rm -f etc_default_aufs
echo '# aufs variables for shell scripts' > etc_default_aufs
./c2sh >> etc_default_aufs
echo >> etc_default_aufs
sed -e '0,/^$/d' aufs.shlib >> etc_default_aufs
make -C libau all
make[1]: Entering directory `/mnt/part3/tmp/linux-2.6.38.7/aufs2-util.git/libau'
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -DMOUNT_CMD_PATH=\"\" -c -o libau.o libau.c
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -DMOUNT_CMD_PATH=\"\" -c -o rdu_lib.o rdu_lib.c
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -DMOUNT_CMD_PATH=\"\" -c -o rdu.o rdu.c
ln -sf rdu.c rdu64.c
cc -I./libau -O -Wall -DRdu64 -fPIC -DNDEBUG -D_REENTRANT -I. -DMOUNT_CMD_PATH=\"\" -c -o rdu64.o rdu64.c
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -DMOUNT_CMD_PATH=\"\" -c -o pathconf.o pathconf.c
cc --shared -Wl,-soname,libau.so.2 \
-o libau.so.2.5 libau.o rdu_lib.o rdu.o rdu64.o pathconf.o -ldl -lpthread
ln -sf libau.so.2.5 libau.so.2
ln -sf libau.so.2 libau.so
make[1]: Leaving directory `/mnt/part3/tmp/linux-2.6.38.7/aufs2-util.git/libau'
ln -sf ./libau/libau*.so .
rm ver c2tmac c2sh

[Capricorny]

Yes I am. Because I update my kernel very often, each time I have to match it against the correct version of aufs. I have compiled aufs kernel module too many times already.

If you compile from latest sourceforge version from the developer, it is surely OK. My concern was about the "correct" version of aufs.



I don't know what "essential" libraries you are talking about. A typical aufs user does not even use the aufs tools. Even knoppix does not use the aufs tools ! Only in rare occasions that you need to use the aufs utils or tools.

"Essential" is of course a relative concept. As a new example, on Friday an error message appeared on R-help: http://www.mail-archive.com/r-help@r-project.org/msg137086.html
A possible explanation, mentioned by Brian Ripley, is Debian's versions of the lapack and blas libraries. When Debian is used on a scientific/engineering workstation, those libraries are surely essential, even though the vast majority of users don't use them. They could, however, crop up anywhere, embedded in image processing routines, games etc.

A more actual example is provided by this thread: Use of aufs-tools to make no-wear Knoppix versions for flash memory. With widespread use, those tools could migrate into the "essential" category, I think.

[dinosoep]

this is my make output:



cc -I./libau -O -Wall c2tmac.c -o c2tmac
rm -f aufs.5
./c2tmac > aufs.5
awk '{ \
gsub(/\140[^\047]*\047/, "\\[oq]&\\[cq]"); \
gsub(/\\\[oq\]\140/, "\\[oq]"); \
gsub(/\047\\\[cq\]/, "\\[cq]"); \
gsub(/\047/, "\\[aq]"); \
print; \
}' aufs.in.5 >> aufs.5
chmod a-w aufs.5
cc -I./libau -O -Wall -c -o proc_mnt.o proc_mnt.c
ar rv libautil.a proc_mnt.o
ar: creating libautil.a
a - proc_mnt.o
cc -I./libau -O -Wall -c -o br.o br.c
ar rv libautil.a br.o
a - br.o
cc -I./libau -O -Wall -c -o plink.o plink.c
ar rv libautil.a plink.o
a - plink.o
cc -I./libau -O -Wall -c -o mtab.o mtab.c
ar rv libautil.a mtab.o
a - mtab.o
cc -I./libau -O -Wall -c -o auplink.o auplink.c
cc -static -s auplink.o -L. -lautil -o auplink
cc -I./libau -O -Wall -c -o mount.aufs.o mount.aufs.c
cc -static -s mount.aufs.o -L. -lautil -o mount.aufs
cc -I./libau -O -Wall c2sh.c -o c2sh
rm -f etc_default_aufs
echo '# aufs variables for shell scripts' > etc_default_aufs
./c2sh >> etc_default_aufs
echo >> etc_default_aufs
sed -e '0,/^$/d' aufs.shlib >> etc_default_aufs
make -C libau all
make[1]: Entering directory `/home/knoppix/aufs2.1-tools/libau'
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -c -o libau.o libau.c
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -c -o rdu_lib.o rdu_lib.c
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -c -o rdu.o rdu.c
ln -sf rdu.c rdu64.c
cc -I./libau -O -Wall -DRdu64 -fPIC -DNDEBUG -D_REENTRANT -I. -c -o rdu64.o rdu64.c
cc -I./libau -O -Wall -fPIC -DNDEBUG -D_REENTRANT -I. -c -o pathconf.o pathconf.c
cc --shared -Wl,-soname,libau.so.2 \
-o libau.so.2.3 libau.o rdu_lib.o rdu.o rdu64.o pathconf.o -ldl -lpthread
ln -sf libau.so.2.3 libau.so.2
ln -sf libau.so.2 libau.so
make[1]: Leaving directory `/home/knoppix/aufs2.1-tools/libau'
ln -sf ./libau/libau*.so .
rm c2tmac c2sh

as you can see, no errors at all.
only mount.aufs gets made, umount.aufs doesn't...

[dinosoep]

hehe, whiteout files seem to simply be .wh..original_filename So rsync should do the job as capricony suggested, going to try it out now

[kl522]

as you can see, no errors at all.
only mount.aufs gets made, umount.aufs doesn't...

Then it is obvious your version of source code is faulty.

If you check careful, my source compilation also compiles another program called 'ver'. This is a program which verifies version matching. If there is a version mismatch, compilation of 'ver' will not pass.

[kl522]

If you compile from latest sourceforge version from the developer, it is surely OK. My concern was about the "correct" version of aufs.

Whatever it is, the point I want to make is that, I am pretty confident that Klaus K have compiled aufs correctly to match his version of kernel. The last thing to suspect is the version of kernel aufs module is wrong. Knoppix will not run if the kernel aufs module is wrong.

The aufs toolset is a totally different story. Knoppix does not use it. Most debian users also do not use it. So I am not surprised that it might be faulty.

[dinosoep]

rsync story didn't work out, cause after an rsync + delete you get nfs stale locks

[dinosoep]

Nfs stale locks can be overcome by remounting /UNIONFS. Here is the problem, with a (kinda) clean knoppix I can't unmount an aufs mount point, just mounting.
Is this normal?
Can I fix this easily?

[kl522]

Here is the problem, with a (kinda) clean knoppix I can't unmount an aufs mount point, just mounting.
Is this normal?

If you mount your own aufs mount point, you should be able to umount it.
However if you want to umount KNOPPIX mount point, you can't do it.
I mentioned this sometime ago, but probably no one notices it and no one bothers about it.

As result, when Knoppix shutdowns down, knoppix does not manage to umount everything before it halts. PROBABLY it does not matter if you mount the persistent file over file system such as VFAT and ext2/ext3. But if you mount Knoppix persistent file over NTFS, Knoppix kills the fuse process before shutdown and and it can corrupt the persistent file system.

If you ask is this normal, then I would say it is not normal but it is a knoppix behaviour.



Can I fix this easily? Yes this can be fixed but perhaps not so easily. In my own customization I umount EVERYTHING before shutdown down.

[dinosoep]

Well kl522, I want to unmount the /UNIONFS mount point and then remount that. For some crazy reason I am not able to do this.
And it would be great if you could share your fixes with us as your solution seems way cleaner compared to the default knoppix behaviour

[kl522]

Well kl522, I want to unmount the /UNIONFS mount point and then remount that. For some crazy reason I am not able to do this.
And it would be great if you could share your fixes with us as your solution seems way cleaner compared to the default knoppix behaviour

Before I share with you how I accomplish "clean unmounting' of the entire KNOPPIX system, I would like to understand a bit more. Can you explain to us why do you want to unmount /UNIONFS and then remount it ?

If you are trying to do all this in the middle of system running, I think the chances of it succeeding will be slim. If you are trying to do this while the system is shutting down, chances of it succeeding will be higher. Because to umount /UNIONFS you will have to kill the processes which are using the file system.

I succeeded in umount /UNIONFS, /KNOPPIX-DATA and /mnt-system, but all these are done during system shutdown.

[dinosoep]

when I use rsync to copy everything from ramdisk to knoppix-data there is no problem but as soon as I remove everything from the knoppix-data-ramdisk (after a copy) I get nfs stale handle. This is because the filesystem still thinks to know where the files are while in fact they are moved to somewhere else.
Every site I came up with suggested remounting the whole package and I don't see a better solution :(

[kl522]

A quick check on the script aubrsync also contains rsync however it does not contain umount, but it does include 'remount'. So I suggest you still use aubrsync. What you need to do is to manually copy 'mount.aufs' to /sbin, aubrsync to /usr/bin, since your 'make install' has failed. There is no need to 'umount' before you 'remount'.

If you want I can post you my compiled aufs-utils. But it was done for kernel 2.6.38.7 which I am not sure it will be compatible with your kernel.

[Capricorny]

A quick check on the script aubrsync also contains rsync however it does not contain umount, but it does include 'remount'. So I suggest you still use aubrsync. What you need to do is to manually copy 'mount.aufs' to /sbin, aubrsync to /usr/bin, since your 'make install' has failed. There is no need to 'umount' before you 'remount'.

If you want I can post you my compiled aufs-utils. But it was done for kernel 2.6.38.7 which I am not sure it will be compatible with your kernel.

I think it would be nice to have the complete aufs2-package compiled with the kernel version of the actual release - as a starting point. Speaking for myself, I'm pretty sure that I will be able to break things during kernel updates, so I think I need a safe and stable place to start. :) From sourceforge, it seems that basically the same aufs2 version goes with a series of kernel releases?

In addition, could there be compatibility problems with 32/64-bits compiles?



As result, when Knoppix shutdowns down, knoppix does not manage to umount everything before it halts. PROBABLY it does not matter if you mount the persistent file over file system such as VFAT and ext2/ext3. But if you mount Knoppix persistent file over NTFS, Knoppix kills the fuse process before shutdown and and it can corrupt the persistent file system.

If you ask is this normal, then I would say it is not normal but it is a knoppix behaviour.


Can I fix this easily?

Yes this can be fixed but perhaps not so easily. In my own customization I umount EVERYTHING before shutdown down.


I think your way is the right way here. And I really can't see why this should not be fixed, as it may also provide keys to better and safer system administration. I see very little reason for not having two alternatives on exiting X: The ordinary command-line, and the basic start position where /UNIONFS is unmounted. The ordinary shutdown/reboot then proceeds through the basic stage, with several possibilities for adding in system administration procedures.

BTW, maybe we should ask the aufs developer to include a restricted varaiant of aufs-mounting branches from aufs-mounted volumes? It is mentioned among future possibilities, and one major reason why aufs is invisible and not too useful for our day-to-day tasks with Knoppix, is that we can't perform anything unless we have volumes outside of /UNIONFS mounted.

Typical uses would be database storage directories, where a cloop startup is unified with extra storage (outside the persistent image), or web content, like directories with large downloadable files.

[dinosoep]

Well, that was the problem kl522, when you remount "sudo mount -o remount" it throws those
/sbin/mount.aufs:plink.c:223: AUFS_CTL_PLINK_MAINT: Inappropriate ioctl for device
errors.
It would be great to have a starting point with an actually working mount - umount - remount for aufs :(

[kl522]

Well, that was the problem kl522, when you remount "sudo mount -o remount" it throws those
/sbin/mount.aufs:plink.c:223: AUFS_CTL_PLINK_MAINT: Inappropriate ioctl for device
errors.

This is because the /usr/include/linux/aufs_type.h included in Knoppix is a very very old version. Either you fix it yourself or you have to get Klaus Knopper to fix it.

Also your downloaded aufs-utils source code is questionable too, as we discussed earlier, it does not seem to compile a few other programs.



It would be great to have a starting point with an actually working mount - umount - remount for aufs :(There is also a design problem with this approach. As I mentioned before, because /UNIONFS is kind of the root file system for Knoppix. If you insist on unmounting it, though it can be done, but it is only suitable during system shutdown. Meaning, your 'no-wear' thingie is only able to flush the changes during system shutdown. So if there is a power failure in between, you lose all the changes.

Anyway I have included my compiled aufs2-tools for kernel 2.6.38. If you test the tools against a non-root file system, you can convince yourself that it is working.

http://dl.free.fr/pFytkL0aw

I believe you will still have other errors even with using the said tools, for reason that /UNIONFS is kind of the root file system for Knoppix.

[dinosoep]

Heheheh, thanks that actually works :D
aubrsync works succesfully but I haven no idea how to clear the ramdisk and safe persistend on a running system, gonna look at puppy linux and see how they do it :p.
It took so long to reply as some creepy update/removal made my keyboard useless after knoppix-autoconfig got run, I hope I'm past that missery now

[kl522]

gonna look at puppy linux and see how they do it :p.

Great, loot at it and let us know.

[Capricorny]

Heheheh, thanks that actually works :D
It took so long to reply as some creepy update/removal made my keyboard useless after knoppix-autoconfig got run, I hope I'm past that missery now

One way to always have a working version while experimenting is creating several small harddisk partitions and place one testing Knoppix version on each. Normally, I work with at least two or three images, and I also store select versions on external media.

[kl522]

Heheheh, thanks that actually works :D
aubrsync works succesfully but I haven no idea how to clear the ramdisk and safe persistend on a running system, gonna look at puppy linux and see how they do it :p.

Actually I found out how Puppy does it :-

1. 'aubrsync' does it the safe way, it locks the file system by putting it in 'readonly' mode before any attempt of copy or move to the lower layer aufs. So 'aubrsync' fails active root file system.

2. Puppy does not do it the safe way, so it does not lock the file system. Puppy uses a self-developed script call 'snapmergepuppy'. If you are backing up a huge file on the top layer of aufs which is the ram file system, while it is being copied to the lower layer of the aufs which is the persistent store, it is possible that the file is deleted during the copy, so you will end up having the deleted file reappearing in the aufs. Puppy developers are fully aware of this. Between the two devils, Puppy developers have chosen this.

3. Puppy does not 'flush' the ram file system after making a copy to the persistent store. Using 'aubrsync' terminology, it is sort of a 'aubrsync copy' without 'lock'. That being the case, does puppy do incremental between the copy or it does full copy each time ? This is something I don't know. If it does incremented, the script must be very sophisticated then. Otherwise if it does full copy each time, if you have written a lot of things into the RAM file system, you will end up repeatedly make a huge amount of data transfer between the layers of aufs.

In any case anyone who is interested to use this Puppy behaviour ( called PUPMODE 13 in Puppy terminology ) can study the script 'snapmergepuppy'. But the sophistication of this has far gone beyond the reach of a typical non-programmer.

[Capricorny]

Some comments:

1. I think the "nowear" cheatcode should not always be used. When doing heavy work involving the persistent store, like package upgrading, I think it may be better to skip the ramdisk layer. The ramdisk layer is for "everyday", not "administrative" use.

2. If we often find ourselves doing lots of heavy work with the persistent store - why are we doing it there? Much better to mount extra partitions and use them directly. Myself, I now find that I need about 4GB persistent store for comfort, but not more. Rather remaster, if programs fill it up, or move data out.

3. With modest modifications, how much does it matter if the ramdisk crashes without having been written to persistent store now and then? And with laptops, power outage is normally not a reason for crash.

4. I wonder what is so terribly impractical with an aubrsync-based scheme, where the ramdisk is normally copied at shutdown, but there is also an option to close X, unmount&update persistent store, and remount/restart X. That is safe, and I wonder where the pressing performance/use issues lie that motivate giving up on safety?

[kl522]

Some comments:

1. I think the "nowear" cheatcode should not always be used. When doing heavy work involving the persistent store, like package upgrading, I think it may be better to skip the ramdisk layer. The ramdisk layer is for "everyday", not "administrative" use.
.....

But there is actually an easier way to accomplish the "nowear" thing, is to ask operating system to be lazy on writing to flash. Here is one write up of such thing :-

http://www.cyrius.com/debian/nslu2/linux-on-flash.html

I thought this is a more elegant solution than to mimic the operating system behaviour in userspace.

[Capricorny]

But there is actually an easier way to accomplish the "nowear" thing, is to ask operating system to be lazy on writing to flash. Here is one write up of such thing :-

http://www.cyrius.com/debian/nslu2/linux-on-flash.html

I thought this is a more elegant solution than to mimic the operating system behaviour in userspace.
As far as this mostly is about OS behavior, I fully agree. But I don't think that is the whole story. We have a lot of proper user space tasks resulting in many disk writes, furthermore, use of ramdisk can speed up running from flash sticks a lot. I also don't quite see that a nowear options has to be that much of an ugly hack, and in the actual use situations, we will mostly have more free memory to use in the time to come. (Because of games requirements and cheap RAM.)

At the very least, I think it is worthwile having a few users testing a clean implementation of it. And using a similar solution for things like databases could also be effective: We could, for example, aufs-mount a compressed ro part, a rw part from a disk partition and a ramdisk. Today, aufs won't let us use the compressed part, when that comes from something already aufs-mounted.

[kl522]

We have a lot of proper user space tasks resulting in many disk writes, furthermore, use of ramdisk can speed up running from flash sticks a lot.

The OS caches all disk writes using memory, so in a way, it is like a layer of ramdisk. It becomes slow only when the Ram is flushed to disk or flash. And here we have a choice to ask the OS not to flush it using the kernel parameters. This is the OS behaviour which I am talking about.



At the very least, I think it is worthwile having a few users testing a clean implementation of it. And using a similar solution for things like databases could also be effective: We could, for example, aufs-mount a compressed ro part, a rw part from a disk partition and a ramdisk. Today, aufs won't let us use the compressed part, when that comes from something already aufs-mounted.Again, you are duplicating what database servers are doing. The database server caches as much thing as memory permits in the memory.

In any case, I shall not stop anyone from doing anything they so prefer. Feel free to implement these. It's useful learning exercise anyway.

[Capricorny]

The OS caches all disk writes using memory, so in a way, it is like a layer of ramdisk. It becomes slow only when the Ram is flushed to disk or flash. And here we have a choice to ask the OS not to flush it using the kernel parameters. This is the OS behaviour which I am talking about.

Again, you are duplicating what database servers are doing. The database server caches as much thing as memory permits in the memory.

In any case, I shall not stop anyone from doing anything they so prefer. Feel free to implement these. It's useful learning exercise anyway.

Of course, I know this.
There are two issues that don't get addressed by your approach:
1. Asking the OS to be as lazy as possible is not necessarily beneficial in all respects.
2. Even with max caching and I/O laziness everywhere, there are in many cases more than enough read/write operations left both to slow down operation and wear out flash.

So, while adding yet another layer of ramdisk may not be optimal in several respects, the issue for many, including me, is whether it severely degrades performance, and whether it goes with a defensive approach to system administration. The issue of performance degradation I think is probably a rather moot one, as all the caching going on will of course reduce the ramdisk involvment too. As for defensiveness, only experience can tell, but there are good reasons to expect at least some benefits.

[dinosoep]

kl522, I have been studying that specific script a little bit back and as you predicted, I don't understand it :p. I'm used to object oriënted java but I should consider learning bash xd.
What I wanted to do was check for every file in the ramdisk if a process was using it and if not copy it to the knoppix-data, remove from ramdisk and sync.
This should happen when the ramdisk is over 100 mb ever 2 minutes a check.

That link you posted is indeed a way more elegant solution and I'm going to aply it too :D

[kl522]

What I wanted to do was check for every file in the ramdisk if a process was using it and if not copy it to the knoppix-data, remove from ramdisk and sync.
This should happen when the ramdisk is over 100 mb ever 2 minutes a check.


You can use 'lsof' to do it and probably that's the same scheme which is used by Puppy.

But you have to know that it is still not 100% safe. Because a file system allows concurrent access, at one moment your check says the file is not opened, but as soon as you start the copying and deleting the copy on the ramdisk, another process starts opening it for modification or deletion. You can imagine it will result in various confusion to your file system.