Thread: Knoppix 6.7.0: A View from the Forest

Originally Posted by utu

Greetings again, Forester

Hi utu,

I quite liked your description (post #7, this thread) of what you want the first time I read it. It is a fair reflection of what I think we discussed way back when.

I don't know that others will understand it quite the same way. Some might not recognise it as 'remastering', some might wonder what is so special about this set of requirements. I seem to remember KK himself wondering why bother.

I still think you have a valid point to make but it needs to be able to made convincingly.

When I think of what is involved in remastering I see a 'selection' of what to include, 'assembling' what is selected, 'generating' a new image from the assembly and, of course, 'testing, testing, testing'. Others might see it differently - they might include publishing the new image, for example. I think the process is iterative and I'd see testing as continual, rather than something that is done at the end.

In the middle of all this, the 'generation' bit may not seem all that important but that was all we were really talking about. Perhaps our approach is a bit to simplistic. I don't think so but you would need to convince the skeptics.

You start with a working LiveUSB installation. You use synaptic to add a few packages and may be remove a few others. That covers 'selection'. You've also covered 'assembly': everything is in your persistent store. You've been playing with the new packages as you've installed them. That covers 'testing' too (for all practical purposes). So all you need is the 'generation' bit and that isn't something you can do with a click or two of the mouse.

This may not be 'remastering' as KK knows it but it seems to be a reasonable scenario (at least) for the 'ordinary' user's first attempts at remastering. I think there is a case of providing a simple (to use) tool for this.

Do this make sense ? Is there any point in my continuing ? It only gets harder from here on.

The next questions you might be asked as about the scope the method covers (like is this LiveUSB only ?). There is no problem with there being limitations but it helps if you know what they are. Open discussion may help widen the scope at little cost and bring others on board.

P.S. Still interested in the special properties of ntfs for remastering.

.. and Werner use to re-generate
a new KNOPPIX file from an old one plus the accumulated 'additions' in persistence

... no, no! I don't work with accumulated stuff in any persistence memory. I do all my changes in a real Harddrive Installation and test it within this. That's the way, all people work.

You need a backup concept or a version control system to go back to former state of testing in case of error.

And only then, when all went well, I repeat the changes in the chroot jail using some scripts and build a new ISO.

And why do you insist on working within Windows? Windows is the worst case for handling Linux development. My suggestion, to use VirtualBox is only a poor man solution, it isn't "essential"; the better way is using a separate partition for Knoppix-Installation and boot it.


Greetings Werner * http://www.wp-schulz.de/knoppix/summary.html
Own Rescue-CD with Knoppix (Knoppix V6.7.0 remaster)

@ Werner

I hope I haven't offended you with my simplistic representation of your re-mastering approach.

In my table, I am trying to contrast my suggested modest addition to Knoppix with what you
and others are using for much more complex and thorough manipulations. What I'm proposing
is a menu-driven back-up utility of sorts, but capitalizing, I hope, on being able
to occasionally move back-up content into the compressed image, once any changes have
'panned-out'. Plus a 'hard-copy', LiveCD product as well.

My unhappy experience with VirtualBox and VMWare on Windows was a learning experience for me.
At the time, I had no source of workspace storage adequate for re-mastering except the ntfs
partitions of my laptop, nor other large Linux partitions. Since one of our forum members
uses Windows partions for workspace, I thought I could probably at least do some virtual stuff there.
Well, after several occasions that I needed to 'restore' my Win7 image, I decided that's
not for me.

Suffice it to say, if what I'm proposing requires any significant external workspace, then it
probably won't be very popular. I do think Knoppix should have some built-in back-up choices
of some kind.

Originally Posted by Werner P. Schulz

... no, no!

Hi Werner,

I am glad to see that someone else is at least reading this thread. I have read your pages on remastering several times and I do have questions but I've never tried it so why should you answer ? No, I don't use Knoppix installed to hard drive but that is pretty feeble excuse.

I've read a fair way through the thread you started on your laudable attempt to 'update' the Remastering How To but that rambled off onto a long discussion of virtual machines. That discussion took place after I stopped using Knoppix and stopped attempting to contribute to this forum. The irony was that all the time I was attempting to contribute I was using Knoppix in Virtual Box under Windows 7.

My first experience was negative. I got a job to write some PHP scripts for Linux with some people who thought cross-development from Windoze was cool. So I reached for the latest Knoppix only find Plan A blocked by M$ fixing something they called a security loophole and Plan B blocked by the 'withdrawal' of the home= cheatcode from Knoppix. So I had be be inventive and spent a lot of time tinkering with the init script. Virtual box never gave me any trouble. I think it is a really good piece of software even if the 'user interface' is designed to be so easy to use it's insulting.

At home I am privileged. I haven't run Windoze seriously in years: just occasionally inside a virtual machine, of course. I find virtual machines handy: I have Knoppix 6.7.0 LiveCD, LiveUSB and even installed to hard drive all running in virtual machines. I don't have disk space problems either. I've several machines with gigs of LVM space I can carve up any way I need for testing purposes. This means I struggle to appreciate all this nonsense about the safety or otherwise of nfts.

Back on topic (I hope), I understood utu was trying to put together some kind of proposal for a "poor man's" mastering to float past KK. Something as dumb as the current install the USB and install to hard drive scripts. Something the average Knoppix user could use. Bearing in mind that, by definition, half of users are below average.

I don't know that this is possible but I'd be happy to discuss the possibilities if only to understand remastering better. After all, this is only a hobby. I've tried to start with something simple that might be extended rather than start with something ambitious that needs simplifying. The starting point is an method of shuffling files from read/write persistent store to the read-only compressed iso image. As for progression, there is naturally Plan A, Plan B and Plan C.

Plan A considers how far the simple approach could be made to go. What are its real limitations ? How generally useful could we make it ? The greater its potential utility, the more reason KK might have to say OK.

Plan B gives up. Let's call the approach "reflashing" or something catchy and see if KK thinks the simple idea is worthy of distribution.

Plan C is about finding out what utu's real problem is. I suspect here 'backup and recovery' are probably key. We will know a whole lot more when utu explains the special properties he has discovered in the ntfs file system organisation.

My simple approach certainly has limitations.

One is the use of persistent store. I understand that with a hard drive installation there is no persistent store. The simple view is the approach is limited to (real or virtual) Live USB installations.

Another is the output. The approach specifically is about remastering the compressed iso file only. It does not include generating the final iso file that can be burnt to disk. That could be added easily but requires 'additional' disk space and that is a problem.

Another is scale. We begin with adding and removing a few packages. This can easily be extended to include some individual changes if one wants. However, what if you try replace every single package ? I saw somewhere someone tried to replace all 32-bit packages with their 64-bit equivalents. That would require a very big persistent store. Bigger than the 4 Gb limit implied by the current install to USB script. Rewriting that script is off-topic.

Of course you are not going to need a persistent store anywhere near that big unless you are remastering the DVD. The DVD hits another limit. The original remastering uses the original cloop and is a two stage pipe. That should work for most people. The old wiki notes say you need 1 Gb of memory (which, when those notes were written, certainly wasn't the norm). There are complaints from that era from folks whose attempts a remastering failed mysteriously. I think that is because they ran out of memory.

For a the DVD you generally have to scale up by a factor of 7 or 8. Who has 8 Gb of memory on their personal workstation or laptop ? Even with swap space ('additional' disk space by another name) you can't use 8 Gb of memory without using a 64-bit kernel (and possibly some applications compiled for 64-bits).

My way round this was to use squashfs because that's a one stage process with no pipe and so doesn't require lots and lots of memory. But squashfs isn't even an option in the Knoppix distribution.

Hmm ... so the simple approach either has to limit itself to CD sized installations (at which point several people, myself included, lose interest) or use 'additional' disk space. Even then we have a bit of a problem. If you want to remaster a LiveCD, your iso has to fit and a tool, however simple, should check for this. If you just want to remaster a LiveUSB there is no such limitation. The limit is actually 4 Gb unless we rewrite the install to USB script. The DVD is only 3.5 Gb. Oops.

Stopping people running out of memory or at least giving them a sensible error message when they do could be the hardest part of producing a tool that the average Knoppix user could use.

What about the vexed question of 'additional storage' ? The first time I tried remastering I was horrified by how much disk space it required just to add java and eclipse to the Knoppix DVD. I have plenty of disk space spare (none of it ntfs) so I wasn't to worried.

However, I realised that many people might not. They might be tempted to borrow a few gigs from their Windoze partition. I can't recommend that but I recognise that those who need to will rationalise until they are convinced it will be OK. That's for the individual conscience but I don't think it would be OK in a script for inclusion in a distribution. Many people still come to Knoppix because Windoze has trashed its file system. I'm told that is what Windoze does. Would KK accept a script that might be even suspected of complicity ? It would like a fireman setting fires.

If, fortunately this is not likely, I was forced at gun point to remaster Knoppix using a machine with Windows 7 as its hard drive operating system, I would use a virtual machine. That would use big ntfs files inside of which were Linux file systems. I would see this as as safe as it gets since Windows 7, not Knoppix, would be read and writing these files. It seems from the long thread that discussed virtual machines that utu found his Windows trashed his file system more often when he used virtual machines,

Hmm ... somewhere I wrote that I think all you can prove from statistics about file system trashing is that inexperienced people make more mistakes. That's the point: any tool included in the Knoppix distribution has to be safe in the hands of the inexperienced.

Originally Posted by Forester

Something the average Knoppix user could use. Bearing in mind that, by definition, half of users are below average.

Enough, already, of this kind of spam.
This is carrying the MEAN-value theorem to a new level of MEANness.
Give us some good TECHNICAL ideas here.

@ Forester

I found your post #15 about 50% offensively trite with its Bilious excesS; and
about 25% recounting things as they are, but not news.
In the remainder, you glossed over two things important to what I am interested-in.
I would be interested in delving into these areas in a more, but not excruciatingly so,
level of detail:

The approach specifically is about remastering the compressed iso file only.
It does not include generating the final iso file that can be burnt to disk.
That could be added easily but requires 'additional' disk space and that is a problem.

A final USB product can be fabricated via cut-and-paste without 'additional' disk space,
if the required iso file and a replacement persistence file are available.

My way round this was to use squashfs because that's a one stage process with no pipe
and so doesn't require lots and lots of memory.
But squashfs isn't even an option in the Knoppix distribution.

Are you sure squashfs is not 'in' 6.7?
If not, how big a change would that be?
What are the elements of this change?

... it is hard, to classify Foresters excellent analysis as spam. And I can understand him, if he isn't willing to help you in this Principle thread - a huge building site, only some problems solved and far away to become in the future a nice little script for unexperienced users.

@Forester

I have read your pages on remastering several times and I do have questions but I've never tried it so why should you answer ?

I'm glad to hear you've visited my page. But if you have questions, why don't you ask me; there is a link how to contact me.

Originally Posted by Werner P. Schulz

... it is hard, to classify Foresters excellent analysis as spam.

Thank you Werner for your encouragement. I do rather seem to have upset utu.

The point I failed to make was that KK might not want something that only half of Knoppix users could use. Of course any solution offered to KK would have to satisfy utu. He is the sponsor and possibly the only test bed. Do the math.

utu, the points glossed over.

On the first, I think the answer is yes. I used a USB installation because I needed a portable solution. Yes, I backed it up to hard drive every day. I know many people think back ups are an imposition and don't/won't/can't do them. I had a second USB stick in case I lost the first. I know many people don't. My script 'remastered' from the first to the second without requiring any 'additional' (aka temporary) file space. It seemed to me that that modest investment in a second USB stick might be easier for inexperienced users than repartitioning their hard drive or learning to use virtual machines. That is, of course, just a personal opinion.

On the second, I should have written that squashfs is not supported as an alternative to cloop. The Linux kernel squashfs module is in 6.4.3 and 6.4.4. I have every reason to believe that it is also in 6.7.0 (but haven't checked). However, you can't boot Knoppix from squashfs because the init script does not support it. There is a long thread that discusses this. I posted a patch for init that supported this (and I think dinosoep tried it). The patch works for 6.4.4 but it might not for 6.7.0.

I write scripts for two purposes: for things I don't do often and things I do do often. This is to be less dependent on my poor memory and my poor typing. Scripts are meant to be read and changed. I have little experience of writing scripts for others to use, especially those who can't read and change them. I made a mistake when I published the one utu can't use. I won't do that again.

If I ever do such a thing again, I will provide a GUI front-end but I have no experience of writing such things and have no idea when or if I could start such a project. It is easy to say I want simple GUI script but it is quite something else to write one. After all, when a command line script goes wrong the user can see what has happened and is supposed to figure out what to do whereas with a GUI, the script hides the error and tries to figure out what to do by itself.

My analysis, as far as it has gone, is that the first step is to replace the install to USB script. As it stands, it uses vfat, which has the distinction of being the only hard drive file system I know of that does not support files with holes (or sparse files or whatever you call them). That's why the backup / restore functions I did for 6.4.4 were dismissed as 'too slow'. It imposes other limitations that make doing something nice and simple like utu wants very much more difficult that it should be.