A sanity check for 'persistent store' on Knoppix from Flash

[utu]

I occasionally crash my 2Gb SD Chip Knoppix 6.2.1 installation doing things
beyond my competence. Knoppix gives one root privileges that one may not
always deserve.

Unfortunately, if a crippled 'Knoppix from Flash' system survives enough to
re-write a 'persistent store' on shut-down, then it's highly likely the next
boot-up won't be successful, since the crippled system will have replaced an
earlier, presumably healthy boot image. At which point you may not only have
lost all the tweaks you have added, you may not even have a bootable system.

Sometimes you may know when the system has been crippled, just by the way it
is behaving. In such cases, perhaps you can preserve what you have by not
even attempting a normal shut-down. Best in this case that rather than try
a normal shut-down, to instead yank your SD chip or USB, and do a power-down
and hope your earlier copy of Knoppix is still ok.

It would be nice if, in Knoppix's process of preparing a 'persistent store'
on normal shut-down, that some sanity checks would first be applied to
determine whether to revise an originally-booted image or not. If, for
example, the new image would be larger than the SD or USB device, then it
won't be useful to replace the earlier image. There may be other useful,
less crude checks, but I'd settle even for just this one, which I don't
think is there now.

[kl522]

I occasionally crash my 2Gb SD Chip Knoppix 6.2.1 installation doing things
beyond my competence. Knoppix gives one root privileges that one may not
always deserve.

I can share your frustration from a user perspective.

You have presented your 'hypothesis' of the knoppix persistent image extended beyond it's provided space. Untill now, that remains as a hypothsis, as I mentioned earlier, both knoppix compressed image and knoppix persistent image are "hard sized", should not grow beyond their provided space, in theory. If they at all do that, it is a very serious lower level bug.

What you may not quite realize is that, in the other discussion thread, it has been discussed and established, it's possible to crash knoppix by growing the knoppix persistent image just to it's maximum size, leaving it un-fit for the next boot. I offer you an easy way to repeat this yourself :-

# cd /KNOPPIX-DATA
# dd if=/dev/zero of=x

Upon shutdown and reboot, you will notice that knoppix will hit a total failure to boot. And I did offer a possible way that one can recover from this, ie boot by adding 'debug' cheatcode to syslinux.cfg ( since the system already cannot boot, you will have to take the flash to another computer to do this ). And I did mention also that, it would be nice if knoppix during the booting, perform a small little sanity check, it would be able to avoid such a disastrous total failure to boot. And it's a simple modification, to the author of KNOPPIX.

If however, you are remain convinced that the crash you encounter is the first type, please post a consistent way where you could crash it that way. Then it will be useful to establish the root cause of the problem.

Cheers.

[utu]

@kl522

Thank you for you comments.

I know the following, that somehow on occasion, I can crash my Knoppix 6.2.1
operating with 4Gb of ram. The system becomes crippled, but is not dead.
I know that du -h thinks UNIONFS is 4Gb or so, and is complaining at this point.
From previous similar experience, I find that if I do a normal shut-down under
these conditions that I cannot subsequently reboot from my 2Gb SD Chip.

I presume this means the system wasn't so crippled that it couldn't write over
the previous image with unuseable data of some sort. UNIONFS starts out in my
setup at 2Gb. I've never seen it otherwise except when I suspect the system has
become impaired. I had thought perhaps some commands like updatdb or
apt-get update might have been more memory intensive and related to running out
of memory, but have confirmed that was not the case of my two crashes over two
months. But I don't have a handle on just what did cause these crashes.

I am more vigilant now, but not looking to force a crash that I don't under-
stand. I would be interested to know what you get as a result of your proposed
test, assuming your setup is similar to mine.

I would imagine my (or our) suggestion could easily be fixed by the Knoppix
folks, but for all I know, that may already be programmed in. I'm looking
for any normal program which might be the cause of my occasional upset.
If I find one, "I'll report back".

Cheers

[kl522]

And I did mention also that, it would be nice if knoppix during the booting, perform a small little sanity check, it would be able to avoid such a disastrous total failure to boot. And it's a simple modification, to the author of KNOPPIX.

I think there is a very simple mechansim can be incorporated into the booting scheme to avoid knoppix-data.img completedly filled up and making it unfit for reboot. The mechanism is as follows :-

1. When the system is completedly booted up, reserve a small space on /KNOPPIX-DATA, this can be added to the last line of /etc/init.d/knoppix-autoconfig :-

              dd if=/dev/zero of=/KNOPPIX-DATA/.sanity_reserve bs=500k count=1

2. When the system is booting up, in the 'init' script ( inside minirt.gz ), after mounting knoppix-image, delete this file to free up the space :-

              [ -f /KNOPPIX-DATA/.sanity_reserve ] && rm /KNOPPIX-DATA/.sanity_reserve

With this two mods, there will not be worries that knoppix-image will be completedly filled up by users and making it unfit for next reboot.

[utu]

@kl522

You may be on the right track here, but I would worry that whatever the
crippled system does with KNOPPIX-DATA may be corrupted. I'd rather keep
the original than take a chance on replacing it with something worse.

The next time I crash, I'll do a better job of identifying just what
the SD Chip files look like after a normal shutdown. I've been avoiding
this, but there'll come a time, I'm sure.

[utu]

An update on trying to trackdown the reason for some prior crashes:

I found a hidden file .xsession-errors in /home/knoppix that grows owing to a number of problems that
don't seem to be affecting any performance I'm aware of, nor any crashes.
Googling '.xsession-errors' there are a lot of bug reports and complaints by Debian and Ubuntu users
going back over the years relating to this file and its growth.
Some reports are in the many-Gigabytes. Some ops are bragging about the size of theirs.

My Flash disk has behaved marvelously ever since I started looking for the cause of crashes.
Even a few hundred megabytes would explain my problem due to this cause.
So far, it's only grown to 2.6 Meg in a week or so.

[utu]

I finally clobbered my SD Chip again, third time. .xsession-errors had nothing to do with it. I was attempting to bring in Chromium with Synaptic. Apparently over-wrote some of the system. Synaptic complained there's not enough room and quit. System in ram was obviously crippled, not responsive to command line commands. I yanked the SD Chip to avoid over-writing, but on next boot the system was obviously flawed, however minimally so. Lost some items on menu at lower left, otherwise able to post and surf, etc. Compiz still going strong. I had three ideas that have _all_ turned out not to be so. 1. I thought perhaps .xsession-errors might be eating up precious memory and causing an occasional crash; 2. I thought a crude way to avoid having to reconstitute knoppix would be to avoid an over-write on normal knoppix shut-down by yanking the SD chip; and 3. I thought some clever test might suffice to automate the intentions of thought 2. Back to the drawing board. Working within 2Gb with 250M set aside for persistent store is hazardous, given the root privilege that is ever-present in Knoppix. I'm sure Synaptic is not the only way to crash, just the latest one I've found. I still don't have an idea what I might have saved and how I might have reconstituted the system short of a full rewrite from the CD, and whether this might be preferable.

[kl522]

I'm sure Synaptic is not the only way to crash, just the latest one I've found.

From what you have described, it's obvious that this is just a disk full problem. It's not a nasty low lever driver bug, corruption or something like that. As with all disk full, it's hard to guarantee any write or saving will be at a consistent or atomic state. But if you use synaptic, it has certain degree of protection against package inconsistencies. There is not too much you can expect out of this, except just to allow more space for it.

What could constitute an issue to KNOPPIX is that, when KNOPPIX is designed to work with low capacity flash, disk full happens a lot more frequently; and when it happens, it should not put itself in a state where you can't boot anymore. A total failure to boot to me is kind of unacceptable, for something which is designed to work on flash.

Nevertheless what I have written prior to this ( #2, #4 ) are still applicable, if only one is interested to have protection over such behaviour. Recovery without wiping out everything is certainly possible, and that has been covered in some other threads.

Cheers

[Capricorny]

The only really foolproof solution, I think, is to keep backups of knoppix-data.img, for instance, on a USB harddisk. Should something go wrong, one may always go back to a working version. I think that reservation trick is a smart safety measure, but lots of things may go wrong. As for sanity checks, what kinds of checks are run on the fs in knoppix-data.img? And what about fragmentation etc - does that have any practical significance there?