Page 5 of 6 FirstFirst ... 3456 LastLast
Results 41 to 50 of 51

Thread: fusecompress on knoppix-data.img?

  1. #41
    Senior Member
    Join Date
    Jan 2011
    Posts
    123
    Heheheh, thanks that actually works
    aubrsync works succesfully but I haven no idea how to clear the ramdisk and safe persistend on a running system, gonna look at puppy linux and see how they do it .
    It took so long to reply as some creepy update/removal made my keyboard useless after knoppix-autoconfig got run, I hope I'm past that missery now

  2. #42
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by dinosoep View Post
    gonna look at puppy linux and see how they do it .
    Great, loot at it and let us know.

  3. #43
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Quote Originally Posted by dinosoep View Post
    Heheheh, thanks that actually works
    It took so long to reply as some creepy update/removal made my keyboard useless after knoppix-autoconfig got run, I hope I'm past that missery now
    One way to always have a working version while experimenting is creating several small harddisk partitions and place one testing Knoppix version on each. Normally, I work with at least two or three images, and I also store select versions on external media.

  4. #44
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by dinosoep View Post
    Heheheh, thanks that actually works
    aubrsync works succesfully but I haven no idea how to clear the ramdisk and safe persistend on a running system, gonna look at puppy linux and see how they do it .
    Actually I found out how Puppy does it :-

    1. 'aubrsync' does it the safe way, it locks the file system by putting it in 'readonly' mode before any attempt of copy or move to the lower layer aufs. So 'aubrsync' fails active root file system.

    2. Puppy does not do it the safe way, so it does not lock the file system. Puppy uses a self-developed script call 'snapmergepuppy'. If you are backing up a huge file on the top layer of aufs which is the ram file system, while it is being copied to the lower layer of the aufs which is the persistent store, it is possible that the file is deleted during the copy, so you will end up having the deleted file reappearing in the aufs. Puppy developers are fully aware of this. Between the two devils, Puppy developers have chosen this.

    3. Puppy does not 'flush' the ram file system after making a copy to the persistent store. Using 'aubrsync' terminology, it is sort of a 'aubrsync copy' without 'lock'. That being the case, does puppy do incremental between the copy or it does full copy each time ? This is something I don't know. If it does incremented, the script must be very sophisticated then. Otherwise if it does full copy each time, if you have written a lot of things into the RAM file system, you will end up repeatedly make a huge amount of data transfer between the layers of aufs.

    In any case anyone who is interested to use this Puppy behaviour ( called PUPMODE 13 in Puppy terminology ) can study the script 'snapmergepuppy'. But the sophistication of this has far gone beyond the reach of a typical non-programmer.

  5. #45
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Some comments:

    1. I think the "nowear" cheatcode should not always be used. When doing heavy work involving the persistent store, like package upgrading, I think it may be better to skip the ramdisk layer. The ramdisk layer is for "everyday", not "administrative" use.

    2. If we often find ourselves doing lots of heavy work with the persistent store - why are we doing it there? Much better to mount extra partitions and use them directly. Myself, I now find that I need about 4GB persistent store for comfort, but not more. Rather remaster, if programs fill it up, or move data out.

    3. With modest modifications, how much does it matter if the ramdisk crashes without having been written to persistent store now and then? And with laptops, power outage is normally not a reason for crash.

    4. I wonder what is so terribly impractical with an aubrsync-based scheme, where the ramdisk is normally copied at shutdown, but there is also an option to close X, unmount&update persistent store, and remount/restart X. That is safe, and I wonder where the pressing performance/use issues lie that motivate giving up on safety?

  6. #46
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by Capricorny View Post
    Some comments:

    1. I think the "nowear" cheatcode should not always be used. When doing heavy work involving the persistent store, like package upgrading, I think it may be better to skip the ramdisk layer. The ramdisk layer is for "everyday", not "administrative" use.
    .....
    But there is actually an easier way to accomplish the "nowear" thing, is to ask operating system to be lazy on writing to flash. Here is one write up of such thing :-

    http://www.cyrius.com/debian/nslu2/linux-on-flash.html

    I thought this is a more elegant solution than to mimic the operating system behaviour in userspace.

  7. #47
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Quote Originally Posted by kl522 View Post
    But there is actually an easier way to accomplish the "nowear" thing, is to ask operating system to be lazy on writing to flash. Here is one write up of such thing :-

    http://www.cyrius.com/debian/nslu2/linux-on-flash.html

    I thought this is a more elegant solution than to mimic the operating system behaviour in userspace.
    As far as this mostly is about OS behavior, I fully agree. But I don't think that is the whole story. We have a lot of proper user space tasks resulting in many disk writes, furthermore, use of ramdisk can speed up running from flash sticks a lot. I also don't quite see that a nowear options has to be that much of an ugly hack, and in the actual use situations, we will mostly have more free memory to use in the time to come. (Because of games requirements and cheap RAM.)

    At the very least, I think it is worthwile having a few users testing a clean implementation of it. And using a similar solution for things like databases could also be effective: We could, for example, aufs-mount a compressed ro part, a rw part from a disk partition and a ramdisk. Today, aufs won't let us use the compressed part, when that comes from something already aufs-mounted.

  8. #48
    Senior Member registered user
    Join Date
    Dec 2009
    Posts
    423
    Quote Originally Posted by Capricorny View Post
    We have a lot of proper user space tasks resulting in many disk writes, furthermore, use of ramdisk can speed up running from flash sticks a lot.
    The OS caches all disk writes using memory, so in a way, it is like a layer of ramdisk. It becomes slow only when the Ram is flushed to disk or flash. And here we have a choice to ask the OS not to flush it using the kernel parameters. This is the OS behaviour which I am talking about.

    At the very least, I think it is worthwile having a few users testing a clean implementation of it. And using a similar solution for things like databases could also be effective: We could, for example, aufs-mount a compressed ro part, a rw part from a disk partition and a ramdisk. Today, aufs won't let us use the compressed part, when that comes from something already aufs-mounted.
    Again, you are duplicating what database servers are doing. The database server caches as much thing as memory permits in the memory.

    In any case, I shall not stop anyone from doing anything they so prefer. Feel free to implement these. It's useful learning exercise anyway.

  9. #49
    Senior Member registered user
    Join Date
    Sep 2006
    Posts
    802
    Quote Originally Posted by kl522 View Post
    The OS caches all disk writes using memory, so in a way, it is like a layer of ramdisk. It becomes slow only when the Ram is flushed to disk or flash. And here we have a choice to ask the OS not to flush it using the kernel parameters. This is the OS behaviour which I am talking about.

    Again, you are duplicating what database servers are doing. The database server caches as much thing as memory permits in the memory.

    In any case, I shall not stop anyone from doing anything they so prefer. Feel free to implement these. It's useful learning exercise anyway.
    Of course, I know this.
    There are two issues that don't get addressed by your approach:
    1. Asking the OS to be as lazy as possible is not necessarily beneficial in all respects.
    2. Even with max caching and I/O laziness everywhere, there are in many cases more than enough read/write operations left both to slow down operation and wear out flash.

    So, while adding yet another layer of ramdisk may not be optimal in several respects, the issue for many, including me, is whether it severely degrades performance, and whether it goes with a defensive approach to system administration. The issue of performance degradation I think is probably a rather moot one, as all the caching going on will of course reduce the ramdisk involvment too. As for defensiveness, only experience can tell, but there are good reasons to expect at least some benefits.

  10. #50
    Senior Member
    Join Date
    Jan 2011
    Posts
    123
    kl522, I have been studying that specific script a little bit back and as you predicted, I don't understand it . I'm used to object oriënted java but I should consider learning bash xd.
    What I wanted to do was check for every file in the ramdisk if a process was using it and if not copy it to the knoppix-data, remove from ramdisk and sync.
    This should happen when the ramdisk is over 100 mb ever 2 minutes a check.

    That link you posted is indeed a way more elegant solution and I'm going to aply it too

Page 5 of 6 FirstFirst ... 3456 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Intel Xeon E5-2689v4 10 Core 3.1G - SR2T7 picture

Intel Xeon E5-2689v4 10 Core 3.1G - SR2T7

$125.00



Intel - Core i7-12700K Desktop Processor 12 (8P+4E) Cores up to 5.0 GHz Unloc... picture

Intel - Core i7-12700K Desktop Processor 12 (8P+4E) Cores up to 5.0 GHz Unloc...

$242.99



Intel - Core i9-14900K 14th Gen 24-Core 32-Thread - 4.4GHz (6.0GHz Turbo) Soc... picture

Intel - Core i9-14900K 14th Gen 24-Core 32-Thread - 4.4GHz (6.0GHz Turbo) Soc...

$539.99



Intel 6 Core i5-8600 3.1GHZ Desktop Processor SR3X0 picture

Intel 6 Core i5-8600 3.1GHZ Desktop Processor SR3X0

$50.00



INTEL CORE i5-9500 6-CORE 3.00GHz SRF4B CPU PROCESSOR LGA 1151 picture

INTEL CORE i5-9500 6-CORE 3.00GHz SRF4B CPU PROCESSOR LGA 1151

$64.99



Intel - Core i7-13700K 13th Gen 16 cores 8 P-cores + 8 E-cores 30M Cache, 3.4... picture

Intel - Core i7-13700K 13th Gen 16 cores 8 P-cores + 8 E-cores 30M Cache, 3.4...

$364.99



Intel - Core i7-14700K 14th Gen 20-Core 28-Thread - 4.3GHz (5.6GHz Turbo) Soc... picture

Intel - Core i7-14700K 14th Gen 20-Core 28-Thread - 4.3GHz (5.6GHz Turbo) Soc...

$399.99



Intel - Core i9-13900K 13th Gen 24 cores 8 P-cores + 16 E-cores 36M Cache, 3 ... picture

Intel - Core i9-13900K 13th Gen 24 cores 8 P-cores + 16 E-cores 36M Cache, 3 ...

$519.99



AMD Ryzen 7 2700X CPU Processor 3.7GHz AM4 picture

AMD Ryzen 7 2700X CPU Processor 3.7GHz AM4

$74.99



AMD EPYC 7282 cpu processor 16 cores 32 threads 2.8GHZ up to 3.2GHZ 120w picture

AMD EPYC 7282 cpu processor 16 cores 32 threads 2.8GHZ up to 3.2GHZ 120w

$78.00