Page 2 of 2 FirstFirst 12
Results 11 to 20 of 20

Thread: Knoppix 6.7.0: A View from the Forest

  1. #11
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by utu View Post
    Greetings again, Forester
    Hi utu,

    I quite liked your description (post #7, this thread) of what you want the first time I read it. It is a fair reflection of what I think we discussed way back when.

    I don't know that others will understand it quite the same way. Some might not recognise it as 'remastering', some might wonder what is so special about this set of requirements. I seem to remember KK himself wondering why bother.

    I still think you have a valid point to make but it needs to be able to made convincingly.

    When I think of what is involved in remastering I see a 'selection' of what to include, 'assembling' what is selected, 'generating' a new image from the assembly and, of course, 'testing, testing, testing'. Others might see it differently - they might include publishing the new image, for example. I think the process is iterative and I'd see testing as continual, rather than something that is done at the end.

    In the middle of all this, the 'generation' bit may not seem all that important but that was all we were really talking about. Perhaps our approach is a bit to simplistic. I don't think so but you would need to convince the skeptics.

    You start with a working LiveUSB installation. You use synaptic to add a few packages and may be remove a few others. That covers 'selection'. You've also covered 'assembly': everything is in your persistent store. You've been playing with the new packages as you've installed them. That covers 'testing' too (for all practical purposes). So all you need is the 'generation' bit and that isn't something you can do with a click or two of the mouse.

    This may not be 'remastering' as KK knows it but it seems to be a reasonable scenario (at least) for the 'ordinary' user's first attempts at remastering. I think there is a case of providing a simple (to use) tool for this.

    Do this make sense ? Is there any point in my continuing ? It only gets harder from here on.

    The next questions you might be asked as about the scope the method covers (like is this LiveUSB only ?). There is no problem with there being limitations but it helps if you know what they are. Open discussion may help widen the scope at little cost and bring others on board.

    P.S. Still interested in the special properties of ntfs for remastering.

  2. #12
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    Greetings, again Forester.

    Are we on the same wavelength again? I think so.
    The word 're-mastering' is itself a problem.

    I surely don't imply the process KK must go through to make a new version of Knoppix.
    Surely, he includes some new source compilation; I can forego that.
    He brings in new kernels and makes all the drivers that go with that.
    Lots of stuff I'm sure I don't even know about.
    I'm talking about things a non-pro user might want to tide himself over between
    official Knoppix versions. Something just little more elegant than persistence plus
    backup.

    What I would settle for in 'selection' might simply include a few Synaptic additions,
    some minor home-grown script additions, plus the sundry tweaks one puts into tailoring
    one's desktop. Nail these down and pack them into the compressed image. Have the
    option to store that as a snapshot on a liveCD as well as a LiveUSB. I don't find it
    useful to move all my stuff into VirtualBox or VMWare to try out new things, especially
    on a Windows workspace. Been there; done that.

    I have the feeling that handling Synaptic removal of things in KKs original compressed
    file may be off-limits for any simple solution. I can give that up. Or, I can try it and
    throw out the result and go back to a previous snapshot of everything that worked before.

    The 'testing' phase I use is just everyday use of my LiveUSB. Whatever works, I keep
    in persistence. If it doesn't work, I don't keep it. If it doesn't work and ruins my
    current LiveUSB, I back up to an earlier version for which I have a (compressed) copy
    of KNOPPIX-DATA that works and bring that in as update.tar.gz. Moving KNOPPIX-DATA into
    KNOPPIX occasionally would solve the back-up problem automatically, and more elegantly.

    I see 'Generating' as the neat scripts you and Capricorny and Werner use to re-generate
    a new KNOPPIX file from an old one plus the accumulated 'additions' in persistence.
    And re-plenishing the persistence file, of course.
    Creating such a script is beyond both my capabilities and interest, but I'd like to
    have the capability of such a script in my working LiveUSB. Here's my take on the
    possibilities I know about:

    The following table outlines my current estimate of the state of progress along these lines.
    It would have been a little prettier if odf formats were allowed on the forum.
    I will be glad to correct any errors in this representation.

    Code:
                                           
    iso Product  Workspace   Virtual     Infra-     Principle     LiveCD  LiveUSB  KK        KK
    Concept      Req'd?      Req'd?      Structure  Status        Product Product  Aware     Interest
    
    PCLinuxOS    No          Immaterial  Mandriva   In Practice   Yes     No       Probably  Prob Not
    Forester     No          Optional    Knoppix    --?--         No      No       Prob Not  Likely
    Capricorny   Yes         Optional    Knoppix    Proven        No      No       Prob Not  --?--
    Schulz       Yes         Essential   Knoppix    Proven        No      No       Yes       Some
    
    Desired      Pref No     Optional    Knoppix    Proven        Yes     Yes      Yes       Yes
    My intention is to advocate a progression of one or more of these, or similar, concepts toward
    a state of visibility sufficent to attract KK's awareness and interest such that that these new
    functions might actually appear in some future incarnation of Knoppix.

    We probably need a different name than re-mastering for this modest upgrade of Knoppix.
    Something like it is called mylivecd in PCLinuxOS.
    Last edited by utu; 08-28-2011 at 12:35 AM.

  3. #13
    Moderator Moderator
    Join Date
    Nov 2010
    Location
    Germany/ Dietzenbach
    Posts
    1,124
    .. and Werner use to re-generate
    a new KNOPPIX file from an old one plus the accumulated 'additions' in persistence
    ... no, no! I don't work with accumulated stuff in any persistence memory. I do all my changes in a real Harddrive Installation and test it within this. That's the way, all people work.

    You need a backup concept or a version control system to go back to former state of testing in case of error.

    And only then, when all went well, I repeat the changes in the chroot jail using some scripts and build a new ISO.

    And why do you insist on working within Windows? Windows is the worst case for handling Linux development. My suggestion, to use VirtualBox is only a poor man solution, it isn't "essential"; the better way is using a separate partition for Knoppix-Installation and boot it.


    Greetings Werner * http://www.wp-schulz.de/knoppix/summary.html
    Own Rescue-CD with Knoppix (Knoppix V6.7.0 remaster)

  4. #14
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    @ Werner

    I hope I haven't offended you with my simplistic representation of your re-mastering approach.

    In my table, I am trying to contrast my suggested modest addition to Knoppix with what you
    and others are using for much more complex and thorough manipulations. What I'm proposing
    is a menu-driven back-up utility of sorts, but capitalizing, I hope, on being able
    to occasionally move back-up content into the compressed image, once any changes have
    'panned-out'. Plus a 'hard-copy', LiveCD product as well.

    My unhappy experience with VirtualBox and VMWare on Windows was a learning experience for me.
    At the time, I had no source of workspace storage adequate for re-mastering except the ntfs
    partitions of my laptop, nor other large Linux partitions. Since one of our forum members
    uses Windows partions for workspace, I thought I could probably at least do some virtual stuff there.
    Well, after several occasions that I needed to 'restore' my Win7 image, I decided that's
    not for me.

    Suffice it to say, if what I'm proposing requires any significant external workspace, then it
    probably won't be very popular. I do think Knoppix should have some built-in back-up choices
    of some kind.

  5. #15
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by Werner P. Schulz View Post
    ... no, no!
    Hi Werner,

    I am glad to see that someone else is at least reading this thread. I have read your pages on remastering several times and I do have questions but I've never tried it so why should you answer ? No, I don't use Knoppix installed to hard drive but that is pretty feeble excuse.

    I've read a fair way through the thread you started on your laudable attempt to 'update' the Remastering How To but that rambled off onto a long discussion of virtual machines. That discussion took place after I stopped using Knoppix and stopped attempting to contribute to this forum. The irony was that all the time I was attempting to contribute I was using Knoppix in Virtual Box under Windows 7.

    My first experience was negative. I got a job to write some PHP scripts for Linux with some people who thought cross-development from Windoze was cool. So I reached for the latest Knoppix only find Plan A blocked by M$ fixing something they called a security loophole and Plan B blocked by the 'withdrawal' of the home= cheatcode from Knoppix. So I had be be inventive and spent a lot of time tinkering with the init script. Virtual box never gave me any trouble. I think it is a really good piece of software even if the 'user interface' is designed to be so easy to use it's insulting.

    At home I am privileged. I haven't run Windoze seriously in years: just occasionally inside a virtual machine, of course. I find virtual machines handy: I have Knoppix 6.7.0 LiveCD, LiveUSB and even installed to hard drive all running in virtual machines. I don't have disk space problems either. I've several machines with gigs of LVM space I can carve up any way I need for testing purposes. This means I struggle to appreciate all this nonsense about the safety or otherwise of nfts.

    Back on topic (I hope), I understood utu was trying to put together some kind of proposal for a "poor man's" mastering to float past KK. Something as dumb as the current install the USB and install to hard drive scripts. Something the average Knoppix user could use. Bearing in mind that, by definition, half of users are below average.

    I don't know that this is possible but I'd be happy to discuss the possibilities if only to understand remastering better. After all, this is only a hobby. I've tried to start with something simple that might be extended rather than start with something ambitious that needs simplifying. The starting point is an method of shuffling files from read/write persistent store to the read-only compressed iso image. As for progression, there is naturally Plan A, Plan B and Plan C.

    Plan A considers how far the simple approach could be made to go. What are its real limitations ? How generally useful could we make it ? The greater its potential utility, the more reason KK might have to say OK.

    Plan B gives up. Let's call the approach "reflashing" or something catchy and see if KK thinks the simple idea is worthy of distribution.

    Plan C is about finding out what utu's real problem is. I suspect here 'backup and recovery' are probably key. We will know a whole lot more when utu explains the special properties he has discovered in the ntfs file system organisation.

    My simple approach certainly has limitations.

    One is the use of persistent store. I understand that with a hard drive installation there is no persistent store. The simple view is the approach is limited to (real or virtual) Live USB installations.

    Another is the output. The approach specifically is about remastering the compressed iso file only. It does not include generating the final iso file that can be burnt to disk. That could be added easily but requires 'additional' disk space and that is a problem.

    Another is scale. We begin with adding and removing a few packages. This can easily be extended to include some individual changes if one wants. However, what if you try replace every single package ? I saw somewhere someone tried to replace all 32-bit packages with their 64-bit equivalents. That would require a very big persistent store. Bigger than the 4 Gb limit implied by the current install to USB script. Rewriting that script is off-topic.

    Of course you are not going to need a persistent store anywhere near that big unless you are remastering the DVD. The DVD hits another limit. The original remastering uses the original cloop and is a two stage pipe. That should work for most people. The old wiki notes say you need 1 Gb of memory (which, when those notes were written, certainly wasn't the norm). There are complaints from that era from folks whose attempts a remastering failed mysteriously. I think that is because they ran out of memory.

    For a the DVD you generally have to scale up by a factor of 7 or 8. Who has 8 Gb of memory on their personal workstation or laptop ? Even with swap space ('additional' disk space by another name) you can't use 8 Gb of memory without using a 64-bit kernel (and possibly some applications compiled for 64-bits).

    My way round this was to use squashfs because that's a one stage process with no pipe and so doesn't require lots and lots of memory. But squashfs isn't even an option in the Knoppix distribution.

    Hmm ... so the simple approach either has to limit itself to CD sized installations (at which point several people, myself included, lose interest) or use 'additional' disk space. Even then we have a bit of a problem. If you want to remaster a LiveCD, your iso has to fit and a tool, however simple, should check for this. If you just want to remaster a LiveUSB there is no such limitation. The limit is actually 4 Gb unless we rewrite the install to USB script. The DVD is only 3.5 Gb. Oops.

    Stopping people running out of memory or at least giving them a sensible error message when they do could be the hardest part of producing a tool that the average Knoppix user could use.

    What about the vexed question of 'additional storage' ? The first time I tried remastering I was horrified by how much disk space it required just to add java and eclipse to the Knoppix DVD. I have plenty of disk space spare (none of it ntfs) so I wasn't to worried.

    However, I realised that many people might not. They might be tempted to borrow a few gigs from their Windoze partition. I can't recommend that but I recognise that those who need to will rationalise until they are convinced it will be OK. That's for the individual conscience but I don't think it would be OK in a script for inclusion in a distribution. Many people still come to Knoppix because Windoze has trashed its file system. I'm told that is what Windoze does. Would KK accept a script that might be even suspected of complicity ? It would like a fireman setting fires.

    If, fortunately this is not likely, I was forced at gun point to remaster Knoppix using a machine with Windows 7 as its hard drive operating system, I would use a virtual machine. That would use big ntfs files inside of which were Linux file systems. I would see this as as safe as it gets since Windows 7, not Knoppix, would be read and writing these files. It seems from the long thread that discussed virtual machines that utu found his Windows trashed his file system more often when he used virtual machines,

    Hmm ... somewhere I wrote that I think all you can prove from statistics about file system trashing is that inexperienced people make more mistakes. That's the point: any tool included in the Knoppix distribution has to be safe in the hands of the inexperienced.

  6. #16
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    Quote Originally Posted by Forester View Post
    Something the average Knoppix user could use. Bearing in mind that, by definition, half of users are below average.
    Enough, already, of this kind of spam.
    This is carrying the MEAN-value theorem to a new level of MEANness.
    Give us some good TECHNICAL ideas here.

  7. #17
    Senior Member registered user
    Join Date
    May 2006
    Location
    Columbia, Maryland USA
    Posts
    1,631
    @ Forester

    I found your post #15 about 50% offensively trite with its Bilious excesS; and
    about 25% recounting things as they are, but not news.
    In the remainder, you glossed over two things important to what I am interested-in.
    I would be interested in delving into these areas in a more, but not excruciatingly so,
    level of detail:


    The approach specifically is about remastering the compressed iso file only.
    It does not include generating the final iso file that can be burnt to disk.
    That could be added easily but requires 'additional' disk space and that is a problem.
    A final USB product can be fabricated via cut-and-paste without 'additional' disk space,
    if the required iso file and a replacement persistence file are available.



    My way round this was to use squashfs because that's a one stage process with no pipe
    and so doesn't require lots and lots of memory.
    But squashfs isn't even an option in the Knoppix distribution.
    Are you sure squashfs is not 'in' 6.7?
    If not, how big a change would that be?
    What are the elements of this change?

  8. #18
    Moderator Moderator
    Join Date
    Nov 2010
    Location
    Germany/ Dietzenbach
    Posts
    1,124
    ... it is hard, to classify Foresters excellent analysis as spam. And I can understand him, if he isn't willing to help you in this Principle thread - a huge building site, only some problems solved and far away to become in the future a nice little script for unexperienced users.

  9. #19
    Moderator Moderator
    Join Date
    Nov 2010
    Location
    Germany/ Dietzenbach
    Posts
    1,124
    @Forester
    I have read your pages on remastering several times and I do have questions but I've never tried it so why should you answer ?
    I'm glad to hear you've visited my page. But if you have questions, why don't you ask me; there is a link how to contact me.

  10. #20
    Senior Member
    Join Date
    Jan 2011
    Posts
    242
    Quote Originally Posted by Werner P. Schulz View Post
    ... it is hard, to classify Foresters excellent analysis as spam.
    Thank you Werner for your encouragement. I do rather seem to have upset utu.

    The point I failed to make was that KK might not want something that only half of Knoppix users could use. Of course any solution offered to KK would have to satisfy utu. He is the sponsor and possibly the only test bed. Do the math.

    utu, the points glossed over.

    On the first, I think the answer is yes. I used a USB installation because I needed a portable solution. Yes, I backed it up to hard drive every day. I know many people think back ups are an imposition and don't/won't/can't do them. I had a second USB stick in case I lost the first. I know many people don't. My script 'remastered' from the first to the second without requiring any 'additional' (aka temporary) file space. It seemed to me that that modest investment in a second USB stick might be easier for inexperienced users than repartitioning their hard drive or learning to use virtual machines. That is, of course, just a personal opinion.

    On the second, I should have written that squashfs is not supported as an alternative to cloop. The Linux kernel squashfs module is in 6.4.3 and 6.4.4. I have every reason to believe that it is also in 6.7.0 (but haven't checked). However, you can't boot Knoppix from squashfs because the init script does not support it. There is a long thread that discusses this. I posted a patch for init that supported this (and I think dinosoep tried it). The patch works for 6.4.4 but it might not for 6.7.0.

    I write scripts for two purposes: for things I don't do often and things I do do often. This is to be less dependent on my poor memory and my poor typing. Scripts are meant to be read and changed. I have little experience of writing scripts for others to use, especially those who can't read and change them. I made a mistake when I published the one utu can't use. I won't do that again.

    If I ever do such a thing again, I will provide a GUI front-end but I have no experience of writing such things and have no idea when or if I could start such a project. It is easy to say I want simple GUI script but it is quite something else to write one. After all, when a command line script goes wrong the user can see what has happened and is supposed to figure out what to do whereas with a GUI, the script hides the error and tries to figure out what to do by itself.

    My analysis, as far as it has gone, is that the first step is to replace the install to USB script. As it stands, it uses vfat, which has the distinction of being the only hard drive file system I know of that does not support files with holes (or sparse files or whatever you call them). That's why the backup / restore functions I did for 6.4.4 were dismissed as 'too slow'. It imposes other limitations that make doing something nice and simple like utu wants very much more difficult that it should be.

Page 2 of 2 FirstFirst 12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •