Poor man's install, with extra storage, on NTFS/Win7

Here is a short account of my (rather trivial) procedure for running Knoppix off an unmodified NTFS partition, with extra storage.

In case partitions can't be shrinked or deleted, this is a non-invasive way of adapting the PC to Linux. I have just used it on a new Toshiba Satellite R830, w/640GB disk.

1. Boot from USB. (In this case, external SSD-disk on USB3, very fast)
2. Copy KNOPPIX over to main NTFS partition.
3. Setup a new, large image file on the NTFS partition with ext4, mount it as /store.
4. Copy over /store from backup to the new "partition".

Here's what I did run:

Code:

---

# fdisk -l

  Device Boot      Start        End      Blocks  Id  System
/dev/sda1  *          1        192    1536000  27  Unknown
/dev/sda2            192      76169  610284544    7  HPFS/NTFS
/dev/sda3          76169      77826    13309952  17  Hidden HPFS/NTFS

# mount /media/sda2
# rsync -ax /mnt-system/KNOPPIX /media/sda2 &
# dd if=/dev/zero of=/media/sda2/store.img bs=1M count=60000 &
60000+0 records in
60000+0 records out
62914560000 bytes (63 GB) copied, 682.558 s, 92.2 MB/s
# losetup /dev/loop6 /media/sda2/store.img
# mkfs.ext4 /dev/loop6
# losetup -d /dev/loop6
# mkdir /store
# mount -o loop=/dev/loop6 /media/sda2/store.img /store
# mount /media/sdb2
# rsync -ax /media/sdb2/local/ /store &

---

All the indirection is of course not optimal, but with a Sandy Bridge I5-2410, I think I can live with it. Here's a snapshot of top while copying the files

Code:

---

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
  5071 root      20  0  2668 1184  664 S  22  0.0  14:26.44 mount.ntfs       
21056 root      20  0  7668 4704  840 R  19  0.1  2:20.18 rsync             
21058 root      20  0 19144 1916  620 R  19  0.0  2:21.09 rsync

---

After starting from HD, I can manually mount this new image file, or I can add the commands to rc.local.

This /store can be used also for remastering, but to be as failsafe as possible, I would recommend repeating the procedure for another similar image for that purpose. Maybe I'll come back with a detailed exposition of remastering on "foreign land" like NTFS.

Normally, it should be enough just to enter knoppix{64} and the usual boot options to run from NTFS - that image should be found first and used. But, of course, using the fromhd=/dev/sda2 makes it even more foolproof.

Quick update: I'm writing this from Firefox under WinXP Pro, running under VMware workstation in a virtual machine located in that second persistent image I created, mounted on /store. It is of course not for really heavy work but it works quite well, though., So, everything works right out of the box in this procedure.

To reduce the services overhead, I only start VMware server when I need it, so the init is done manually.

Code:

---

# su
# mkdir /store
# mount -o loop=/dev/loop6 /mnt-system/store.img /store
# /etc/init.d/vmware start

---

The two lines creating and mounting /store can be added to /etc.rc.local to have this automatically carried out. Beware that this system feature is special for this install, so, ulike basic installs, the setup cant be simply copied using the ordinary Knoppix tools.

Also, there was a Windows Boot directory, so I chose not to create the Knoppix /boot directory on NTFS.

@ Capricorny

Having Windows Professional and VMWare as part of your arsenal,
you don't represent a lot of us poor souls with mere basic Windows
setups.

Windows 7 Home premium will make nice Virtual Hard Drives
but it wont boot them; you need Win 7 Professional or Ultimate for
that. VMWare and Win7 Professional are both in the $200 upgrade
category. I'd like to see what's possible with more meager software
tools to begin with.

My impression of VirtualBox is that it offers some rudimentary
Virtual capability, but is not a very satisfactory environment
to work around in for very long. The price is right, however.

After temporary removal of the Knoppix-related files, and defragmenting twice, the main NTFS volume's minimal size is 308 GB - that is, 31 GB used and 277 GB free.

Impossible to shrink it more, according to Windows.

I was able to free up 282 GB, so there is not that much hassle to run Linux several ways on this computer now. BUT: Imagine I had wanted to use the disk for, say, NTFS databases, or video footage, but keep them apart from the program installs. In that case, my dear friends, Windows 7 wastes away half the whole disk, by insisting on the principle that a 10% filled disk volume cannot be shrunk.

So, effectively, I would have had to wipe it all clean and reinstall Win7, drivers and all..

Think I'm going to create that store.img image file 2-3 times as big, so that I can at least use that real estate for something useful... :)

Quote:

---

Originally Posted by utu

@ Capricorny

Having Windows Professional and VMWare as part of your arsenal,
you don't represent a lot of us poor souls with mere basic Windows
setups.

---

This is no professional stuff, it's Win7 Home Premium right out of the box on a Toshiba R830 - middle class no-nonsense 13.3" I5-2410/4GB laptop. VMware workstation is ca $200 full price, but you don't need it to run virtual machines, you can download that running software from VMware for free. And, as you must have noted: I used it merely for illustrative purposes here: To show that running off vm in a loop-mounted ext4 image residing in a NTFS file works quite well. You can do just the same with other virtualization programs. And virtualization support is getting real strong in Linux now. Only reason I'm running VMware, is that I'm a long-time user, and it's hassle-free for my needs.

My main point in this exercise is to provide a recipe for you and others who feel stuck with Windows-occupied hardware: That is no reason for not running full versions of Linux, and you can even do virtualization on top of it. With no system changes to the M$ infestations.

Quote:

---

Originally Posted by utu

My impression of VirtualBox is that it offers some rudimentary
Virtual capability, but is not a very satisfactory environment
to work around in for very long. The price is right, however.

---

May I suggest that your impression of VirtualBox may be somewhat influenced by you "insisting" on running it off a Windows host, with all M$ antics in effect, rather than on a host built on a more empowering philosphy? Perform the described poor man's install, add the storage, boot Knoppix natively, and then you can experiment with Knoppix Virtualbox.
I may come back with some notes on that - but virtualization in itself isn't that interesting to me. And I have never understood the people running Linux guests on Wondows hosts instead of the other way around, when they have a choice. And as I just have shown, with Knoppix you always have a choice :).

Quote:

---

Originally Posted by Capricorny

Perform the described poor man's install, add the storage, boot Knoppix natively, and then you can experiment with Knoppix Virtualbox.

---

Sad state of affairs on my machine- Pre-installed Virtualbox won't run, and apt-get:

Code:

---

# apt-get install virtualbox
......
The following packages have unmet dependencies:
 virtualbox : Depends: libqtcore4 (>= 4:4.7.0~beta1) but 4:4.6.3-4+squeeze1 is to be installed
              Depends: libstdc++6 (>= 4.6) but 4.4.5-8 is to be installed
              Recommends: virtualbox-qt (= 4.0.10-dfsg-1) but it is not going to be installed
E: Broken packages

---

So, it seems virtualbox is going to be extremely virtual for a while.

@ Capricorny

I can report that VMWare Player solves all my complaints about VirtualBox.
Thanks for that.

I don't understand your comments regarding my 'insistence', since I'm not aware of any choice;
don't I have to choose the host as the object with all the resources I want to use?

Quote:

---

Originally Posted by utu

@ Capricorny
I don't understand your comments regarding my 'insistence', since I'm not aware of any choice;
don't I have to choose the host as the object with all the resources I want to use?

---

Well - why just don't you try out a poor man's install, then? I can't really see any aspect where that would not be optimal in your use situation. But I may have gotten it terribly wrong.

Quote:

---

Originally Posted by Capricorny

Code:

---

# apt-get install virtualbox
......
The following packages have unmet dependencies:
 virtualbox : Depends: libqtcore4 (>= 4:4.7.0~beta1) but 4:4.6.3-4+squeeze1 is to be installed
              Depends: libstdc++6 (>= 4.6) but 4.4.5-8 is to be installed
              Recommends: virtualbox-qt (= 4.0.10-dfsg-1) but it is not going to be installed
E: Broken packages

---

So, it seems virtualbox is going to be extremely virtual for a while.

---

As you can see from here, virtualbox has many packages built with many different dependencies :-

http://www.virtualbox.org/wiki/Linux_Downloads

I always use the last entry, ie All distribution i386. Using this one, it is able to avoid the file dependency problem.

By the way virtualbox need to rebuild it's kernel module driver for a target kernel, and this can be done by doing once after installation :-

# /etc/init.d/vboxdrv setup

This will compile the kernel driver needed to run vb if you have the proper kernel header.

Quote:

---

Originally Posted by utu

My impression of VirtualBox is that it offers some rudimentary
Virtual capability, but is not a very satisfactory environment
to work around in for very long.

---

I think you are referring to OSE version - Knoppix is preinstalled with OSE version if I am not mistaken. But if you are using PUEL version, it should be at par with VMplayer.

Quote:

---

Originally Posted by kl522

I think you are referring to OSE version - Knoppix is preinstalled with OSE version if I am not mistaken. But if you are using PUEL version, it should be at par with VMplayer.

---

I think utu has used it under Windows? But it may seem that the OSE version isn't that much to brag about, plus there is a dependency maze sometimes. Still, it may be a strategical tool to care about for us?

@ Capricorny

Rather than put all your stuff in with Win7 as 'just another file'
why not at least first segregate the section you use into a Virtual Hard Drive?
That is a built-in capability of Win7 Home Premium that would seem
to offer some protection from lousing up Win7 when using its resources.

Quote:

---

Originally Posted by utu

@ Capricorny

Rather than put all your stuff in with Win7 as 'just another file'
why not at least first segregate the section you use into a Virtual Hard Drive?
That is a built-in capability of Win7 Home Premium that would seem
to offer some protection from lousing up Win7 when using its resources.

---

I still don't get why you want to do this so terribly complicated!
Please tell me the chance of corrupting the whole Win7 file system by using my method. It's close to nil. We are not deleting, relocating, using metadata, not anything except for creating a few files at one point of time, and doing random access on them, mostly ro, later. If this had been very risky, I think we should have had reports about it.
Using virtual hard drives might well offer some protection, but I simply don't trust Windows concepts before I have seen them thoroughly tested, and the extra level of complexity must be well motivated by some actual problems.

BTW, I wonder if these VHD's are not simply loop-mounting coming to Windows. In which case we would get an extra, unnecessary level of indirection using them. But I may be wrong.

Quote:

---

Originally Posted by Capricorny

I simply don't trust ......concepts before I have seen them thoroughly tested, .....But I may be wrong.

---

Your idea is attractive, it's new to me, I'd like to see it succeed.
Glad you are willing to use your own Win system to see if it
works out.

I won't risk mine so willingly until I see more folks doing it with
no ill effects. I'm no Windows hater, I like Windows, I just like
linux better.

Lots of luck to you.

Quote:

---

Originally Posted by utu

Rather than put all your stuff in with Win7 as 'just another file'
why not at least first segregate the section you use into a Virtual Hard Drive?
That is a built-in capability of Win7 Home Premium that would seem
to offer some protection from lousing up Win7 when using its resources.

---

There is certainly some confusion here.

case 1:
When Knoppix poor-man install is made to a NTFS file system, and when Knoppix is booted, it is accessing the NTFS via fuse/ntfs-3g. The risk is about fuse/ntfs-3g.

If you created a VHD on NTFS using Win7 capability, now assuming Knoppix is somehow about to boot, what is the driver you are going to use to access the (Win7) VHD on NTFS on ***LINUX*** ? Assuming that you could find such a driver, that will have to be a Linux driver. Certainly the risk of higher than using fuse/ntfs-3g !!!

case 2:
Then assuming that you are not running Knoppix natively, ie you are running Knoppix in Virtual Machine under Win7, then when Knoppix accessing it's harddisk, it goes through Linux file system drivers in the guess OS then it passes on to drivers on the virtual machine on the Host OS. The virtual machine may access it via it's proprietary driver ( eg Virtualbox .vdi ) or if you are lucky, it might understand Win7 virtual HD format. Whatever it is, assuming Linux file system driver is safe to use ext3/cloop/etc, then the risk is entirely on the virtual machine. Now it is matter of you want to trust the proprietary file system driver of the virtual machine, or you want it to trust the virtual machine able to also understand win7 VHD.

Whatever is the configuration, I would say in conclusion by having Win7 VHD, you are adding complexity and RISKs, especially case 1 above.

Quote:

---

Originally Posted by utu

I won't risk mine so willingly until I see more folks doing it with
no ill effects. I'm no Windows hater, I like Windows, I just like
linux better.

---

Sure, we all have to choose the level of risk taking, and there IS some risk involved. I would not do this on mission-critical machines, BUT:

1. The correct thing to do is keeping a sharp as possible distinction between program and data re disk use. Now, Windows makes this hard in a completely meaningless way by not letting you shrink NTFS partitions by more than ca 50%. But if you are serious about your system, you have a system partition of a few GB, which is easily restored from backup, plus one (or several) larger data partitions which you backup to external media, typically USB harddisks. Corrupting either of these, you get back by a few simple commands.
Therefore, if using Knoppix poses a great risk for you, that's just an indication of wrong system design. I would in fact say, not really serious. But yes, that's what Windows arranges for.

2. To me, your worry is a bit similar to walking in the fast lane on a highway and being very concerned about being hit by an helicopter. Harddisks start developing failures after some time, inevitably. And with a running laptop, it is not that difficult to inflict damage. It's not for fun that my new Toshiba R830 has a movement sensor and a utility (under Windows) that kicks in and parks the harddisk head whenever movement is detected. Furthermore, NTFS is a somewhat errantly moving target, and guess who has the most problems with that. Yes, of course M$ themselves. A good portion of all disk corruptions seem to stem from software bugs. But the most important, is the malware aspect. Check the percentage of Win computers being infected to some degree.

3. If you do the really relevant comparison here: What kind of use is assocated with the highest risk of corruption per hour of use, I think poor man's Knoppix may turn out the safest way to use your Windows computer. In particular if you run it off the SD card or the smartphone, but also from hard disk. So, when you insist on running Knoppix in a VM, that seems much more of a safety hazard to me, because of Windows, than running it natively. I do it the other way round, and it is much, much simpler. And as I demonstrated, itcan easily be done off a NTFS partition. (Now I have made that store.img file 120 GB.)

Code:

---

# mount /media/sda2
# rsync -ax /mnt-system/KNOPPIX /media/sda2 &
# dd if=/dev/zero of=/media/sda2/store.img bs=1M count=60000 &
60000+0 records in
60000+0 records out
62914560000 bytes (63 GB) copied, 682.558 s, 92.2 MB/s
# losetup /dev/loop6 /media/sda2/store.img
# mkfs.ext4 /dev/loop6
# losetup -d /dev/loop6
# mkdir /store
# mount -o loop=/dev/loop6 /media/sda2/store.img /store
# mount /media/sdb2
# rsync -ax /media/sdb2/local/ /store &

---

You have no 'sdb2' partition; and where does the directory '/local' comes from?

And then you wrote:

Quote:

---

After starting from HD, I can manually mount this new image file, or I can add the commands to rc.local.

---

After starting from HD whith my Windows 7 installed on it, I don't know, how to mount with the tools from Windows7-tools this new image file. And where is the "rc.local" of Windows 7?

Or do you need changes of Windows 7 bootloader or are fiddling with VMServer? If so, I can not see the big advantage of your method in comparison with remastering in a virtual machine on a virtual drive.

Quote:

---

Originally Posted by Werner P. Schulz

Code:

---

# mount /media/sda2
# rsync -ax /mnt-system/KNOPPIX /media/sda2 &
# dd if=/dev/zero of=/media/sda2/store.img bs=1M count=60000 &
60000+0 records in
60000+0 records out
62914560000 bytes (63 GB) copied, 682.558 s, 92.2 MB/s
# losetup /dev/loop6 /media/sda2/store.img
# mkfs.ext4 /dev/loop6
# losetup -d /dev/loop6
# mkdir /store
# mount -o loop=/dev/loop6 /media/sda2/store.img /store
# mount /media/sdb2
# rsync -ax /media/sdb2/local/ /store &

---

You have no 'sdb2' partition; and where does the directory '/local' comes from?

And then you wrote:After starting from HD whith my Windows 7 installed on it, I don't know, how to mount with the tools from Windows7-tools this new image file. And where is the "rc.local" of Windows 7?

Or do you need changes of Windows 7 bootloader or are fiddling with VMServer? If so, I can not see the big advantage of your method in comparison with remastering in a virtual machine on a virtual drive.

---

I should have given more details! Thank you for calling attention to it! :)
The situation was, that I booted from an external USB SSD harddisk, with a (partial) copy of the /store partition on /local. That is one of the ways I keep things in semi-sync between computers, and also quickly test out things on new hardware.

My install is based on booting into Knoppix from external media - optical drive, USB etc. When booting without fromhd= cheat code from a USB stick with this KNOPPIX directory in place on the NTFS partition, Knoppix will find that first, and run from it, "auto-mounting" NTFS. So, just copying the KNOPPIX files and choosing an external boot option with Knoppix, this whole thing will get started. When up and running, we can modify the /etc/rc.local of running Knoppix to automount the /store image.

In fact, Windows doesn't need to know about this at all! No changes to bootloader, nothing in the system setup.

Later on, we may create a /boot partition and install grub - still no change to Windoze bootloader. Setting Win7 as default in grub menu, it will come up as usual - been doing that for years, several computers mostly in WIndows use. I don't want to go into booting Linux from Win bootloader - it can be quite easily done also with Win7, but I don't think it's the best way to proceed.

Remastering is the next step, think I hinted at that. For that purpose, I create another loop-mounted store, 10-35GB according to project, and mount it on a directory /tmp/knx_remaster. Then running a version of Forester's script. But I'll get back to that, it's basically the procedure described by Forester, but run with loop-mounted images off NTFS. It's simple and relatively quick.

I have started researching 'ntfs-3g' on this forum and find that
using ntfs for remastering was a hot topic in early May of this
year. Use the forum search function to see where Forester
suggested it in so many words.

Some of the current cast of characters on this topic were then
just as uncertain about its advisability as I have expressed.

What was the Defining Experience that made you (anyone) become
less concerned with whether this was a risky business or not?

Quote:

---

Originally Posted by utu

I have started researching 'ntfs-3g' on this forum and find that
using ntfs for remastering was a hot topic in early May of this
year. Use the forum search function to see where Forester
suggested it in so many words.

Some of the current cast of characters on this topic were then
just as uncertain about its advisability as I have expressed.

What was the Defining Experience that made you (anyone) become
less concerned with whether this was a risky business or not?

---

First, you have to distingush between full use of NTFS, e.g. for file system administration, and the very restricted use with this method, For instance, I would not recommend expanding the store files, rather create them anew, as I did with store.img yesterday.
Second, there is an issue of relative risk. And the relative risk using ntfs-3g relative to using NTFS natively, is way lower than I had realized.
Third, the use of NTFS for such purposes is exploding. Are the error reports also exploding, from competent users? Go figure.

Quote:

---

Originally Posted by utu

What was the Defining Experience that made you (anyone) become
less concerned with whether this was a risky business or not?

---

I wouldn't say poor-man install on NTFS and running knoppix fuse/ntfs-3g is totally safe, considering that in my opinion, the original Knoppix does not shutdown properly, especially on fuse/NTFS. However, it is definitely safer than trying to use Win7 VHD on Linux.

Quote:

---

Originally Posted by kl522

I wouldn't say poor-man install on NTFS and running knoppix fuse/ntfs-3g is totally safe, considering that in my opinion, the original Knoppix does not shutdown properly, especially on fuse/NTFS. However, it is definitely safer than trying to use Win7 VHD on Linux.

---

Sure, but I perceive this mostly to be a problem for Knoppix. Persistent image is inherently unsafe storage because of non-proper shutdown. If lack of proper file closure causes a file system corruption problem after random access to an established, allocation-wise unchanged file, then NTFS is worse than I am able to believe it is. Any program, at any time, might get stuck and die, leaving open files behind, and Windows can't rely on any special OS monitor being alive and kicking at the time to fix it.

Do you think this bug may corrupt NTFS?

In any case, please register this as a bug, as you know so much about it, and have a fix: http://www.knoppix.net/wiki/Bugs/6.4.4DVD. Very nice if you could include your patch(es) too. Of course, Klaus K is completely free to ignore all bug reports/suggestions, but the bug pages on the wiki here have been actively used by Klaus&co in earlier releases.

Quote:

---

Originally Posted by Capricorny

Of course, Klaus K is completely free to ignore all bug reports/suggestions, but the bug pages on the wiki here have been actively used by Klaus&co in earlier releases.

---

I think the first bug which needs to be logged is the extra /dev/loop/0 thingie. For decades, this unwanted "feature" has been around to bother all knoppix users, and Klaus&co are still totally unaware of it.

Quote:

---

Originally Posted by kl522

I think the first bug which needs to be logged is the extra /dev/loop/0 thingie. For decades, this unwanted "feature" has been around to bother all knoppix users, and Klaus&co are still totally unaware of it.

---

Then go ahead and log it! I don't want to "spam" the knoppix-devel list with lots of detailed bug reports, but presenting a list of bugs/misfeatures, we may expect them to be corrected.
And that shutdown thing is important, I think, so please file it.
BTW, I think Knoppix just turned 10 ;)