improper shutdown - root unionfs and power off

[krishna.murphy]

For the purpose of testing and getting down to the bottom of things, yes I tested using ext2 on partition 2, and yes, now it is unmounted cleanly.

Testing aside, in my actual setup, because my flash is a slower device compared to hard disk, that's why I chose to 'host' knoppix-data.img on an existing NTFS harddisk/partition ( and I want to retain by original NTFS ). It's a lot faster than doing it on a flash. But this testing also revealed that there is indeed a problem with hosting knoppix-data.img on NTFS.

Thanks for sharing the results of all your testing and glad it's working for you! I use NTFS on my hd, too, and it always unmounts cleanly unless the system crashes. In fact it's the same setup as my flash: ~4GB for the KNOPPIX OS files, 2GB for my knoppix-data.aes, and the rest for Windows/shared files. FWIW, I looked around a lot before I got the flash, and I'm pretty happy with it - a very fast and reliable Verbatim TUFF-'N'-TINY 8GB, though it's not quite as fast as a hard drive.

Cheers!
Krishna

[kl522]

Your setup is similar but no exactly the same, the key difference is that you are hosting the knoppix-data.xxx on VFAT, whereas my knoppix-data.xxx is hosted on the NTFS. It would be interesting if you could make a copy of your /mnt-system/KNOPPIX directory onto the NTFS and rename your existing KNOPPIX directory on the VFAT to KNOPPIX.bak ( you would have to do it offline ) - just to see if it is repeatable.

Anyway I would look into /etc/init.d/knoppix-halt in greater detail to see if I could umount the rootfs which is aufs2-ed onto the NTFS.

Regards.

[kl522]

I found the problem but not a fix. Somewhere in /etc/init.d/knoppix-halt, after the modules are freed, the process ntfs-3g is still around. Everything is still ok here.

But right after /bin/umount -t ....., the ntfs-3g is gone and /mnt-system is no more mounted, but /KNOPPIX-DATA is still mounted, ie the NTFS is umount properly but not knoppix-data.img. It will cause corruption to knoppix-data.img.

Looks like having knoppix-data.img hosted on NTFS is not a supported configuration.

[krishna.murphy]

I found the problem but not a fix. Somewhere in /etc/init.d/knoppix-halt, after the modules are freed, the process ntfs-3g is still around. Everything is still ok here.

But right after /bin/umount -t ....., the ntfs-3g is gone and /mnt-system is no more mounted, but /KNOPPIX-DATA is still mounted, ie the NTFS is umount properly but not knoppix-data.img. It will cause corruption to knoppix-data.img.

Looks like having knoppix-data.img hosted on NTFS is not a supported configuration.

I think that's some pretty good detective work! For some unknown reason, with your hardware, the shutdown script isn't working quite properly when dealing with NTFS. I wonder if it might have something to do with sound, as playing the shutdown sound gives running programs more time to halt; do you have sound? I do, and as I stated previously, my NTFS (used on the HD, not on the flash) unmounts cleanly.

Cheers!
Krishna

[kl522]

and as I stated previously, my NTFS (used on the HD, not on the flash) unmounts cleanly.
Krishna

I keep telling you but not sure if you got my point, I am not saying NTFS is not unmounted cleanly. I am saying knoppix-data.xxx which is a loop mounted ext2/ext3 file system sitted on NTFS which is not unmounted cleanly.

And you won't even notice it !!! Unless you have some means to check it offline, otherwise the only exhibits it gives is slower boot time, because the knoppix minirt.gz contains code which automatically perform e2fsck on the knoppix-data.xxx !

( As a side-track, even if NTFS is not cleanly removed from the mounts, you also will not know it. I have seen many cases where I poweroff NTFS in Linux, the system will still come back without needing a file system check. )

[kl522]

Sorry to bore you with more information. As I did more and more testings, I found that this problem is not limited to hosting knoppix-data.img on NTFS. As I perform more tests, I found that this problem happens to almost all kinds of file systems which hosting the loop mounted knoppix-data.img, ie it also happens to ext2/ext3 and VFAT. The tendency of this to happen are :-

1. The knoppix-data.xxx must be big ( in my case it is 2.5 G ).
2. There were some file system activities prior to shutdown ( such as you have saved some files to knoppix home directory ).
3. Maybe the machine need to have huge memory ( > 1 G ).

You notice that after you shutdown/poweroff the system, the knoppix-data.img ( ie the /KNOPPIX-DATA directory ) was not unmounted cleanly.

Most of you do not notice this, as I mentioned before, it is because knoppix minirt.gz has e2fsck inside it. But as with any uncleanly detached file systems, there might be corruption or unsaved data.

[krishna.murphy]

I keep telling you but not sure if you got my point, I am not saying NTFS is not unmounted cleanly. I am saying knoppix-data.xxx which is a loop mounted ext2/ext3 file system sitted on NTFS which is not unmounted cleanly.

Yes, I get it - what I was saying is, I have the same filesystems, and it doesn't appear to have difficulty unmounting cleanly on shutdown. If I have to kill it, like when Hulu, or any Flash video is maxing out the processor to do the NTFS filesystem stuff (complicated to obfuscate, thanks so much Mr. Gates!) then I notice it takes a little longer to boot up afterwards.

And you won't even notice it !!! Unless you have some means to check it offline, otherwise the only exhibits it gives is slower boot time, because the knoppix minirt.gz contains code which automatically perform e2fsck on the knoppix-data.xxx !

( As a side-track, even if NTFS is not cleanly removed from the mounts, you also will not know it. I have seen many cases where I poweroff NTFS in Linux, the system will still come back without needing a file system check. )

Sorry to bore you with more information. As I did more and more testings, I found that this problem is not limited to hosting knoppix-data.img on NTFS. As I perform more tests, I found that this problem happens to almost all kinds of file systems which hosting the loop mounted knoppix-data.img, ie it also happens to ext2/ext3 and VFAT. The tendency of this to happen are :-

1. The knoppix-data.xxx must be big ( in my case it is 2.5 G ).
2. There were some file system activities prior to shutdown ( such as you have saved some files to knoppix home directory ).
3. Maybe the machine need to have huge memory ( > 1 G ).

You notice that after you shutdown/poweroff the system, the knoppix-data.img ( ie the /KNOPPIX-DATA directory ) was not unmounted cleanly.

What's the best way to see that? I imagine re-booting without mounting knoppix-data might be necessary, but what else do you do?

Most of you do not notice this, as I mentioned before, it is because knoppix minirt.gz has e2fsck inside it. But as with any uncleanly detached file systems, there might be corruption or unsaved data.

Thanks for doing all that testing!

Cheers!

[kl522]

First of all I would like to thank you for taking effort to follow this thread.

Yes, I get it - what I was saying is, I have the same filesystems, and it doesn't appear to have difficulty unmounting cleanly on shutdown. If I have to kill it, like when Hulu, or any Flash video is maxing out the processor to do the NTFS filesystem stuff (complicated to obfuscate, thanks so much Mr. Gates!) then I notice it takes a little longer to boot up afterwards.

This is not how knoppix works, according to /etc/init.d/knoppix-halt, it will perform a few things to try to unmount the file systems cleaningly, but failing which, eventually, it will just go ahead to power off the system.

What's the best way to see that? I imagine re-booting without mounting knoppix-data might be necessary, but what else do you do?

That's one way to do it. Another way is boot it off the CD-ROM/DVD and examine it. But since I have to do this so often, eventually what I did is to run knoppix 6.2 in qemu ( its almost as fast as native if you use 'kvm' module ). That offers me a very convenient and fast way to example the qemu file system, as they will appear as normal files on the host operating system.

[kl522]

I would like to post an update to this issue.

Finally I managed to umount /KNOPPIX-DATA cleanly everytime I shutdown the system. But unfortunately, the fix needed is elaborate, and it seems no one using knoppix seems to be at all bothered with this. Maybe I am just too paranoid about it, the /etc/init.d/knoppix-halt script already perform sync before pulling the system to total halt/poweroff. Nevertheless the file system were not totally umounted cleanly.

The fix is to carefully craft out the programs so that during the halting stage, nothing is occupying the unionfs, and /KNOPPIX-DATA. I just broadly outline the changes needed :-

1. Change the initramfs minirt.gz :-
#mv bin bin.static
#mkdir bin
# cd bin
# ln -s /bin.static/busybox busybox
# ln -s /bin.static/sh sh

2. compile a static version of '/sbin/init', and keep it in directory 'bin.static'.

3. Edit the shell script 'init' inside minirt.gz :-
a) change ntfs-3g to /bin.static/ntfs-3g
b) change exec /sbin/init to exec /bin.static/init

4. Various changes to /etc/init.d/knoppix-halt so that it will run /bin.static/busybox and umount /home, umount /UNIONFS and umount /KNOPPIX-DATA.

Have fun.

[krishna.murphy]

Thanks for being persistent! Please post a copy of the changed scripts; I, for one, wouldn't mind faster startup, and it might even have something to do with other issues (like the inability to hibernate, for instance.)

Yours Truly!
Krishna