Working with ntfs

[utu]

@kl522 (and others)

I think you could help us all if you would define some practical examples of do's and don'ts
working with ntfs for linux purposes.

I avoid almost all writes to ntfs from within linux, I avoid automatic mounts of ntfs on linux,
but I have one experience I think I survived which is not quite so conservative.

I think it is correct to say the material on my 6.2 and 6.4 USB installs, including the persistent
store image, resides on a vfat medium. vfat is a linux format deemed two-way compatible with Windows FAT16 and FAT 32, including Windows XP an 7, which operate basically on ntfs. Please pardon any picayune distinctions for the moment.

I occasionally store copies of the vfat contents of the USBs on the Windows 7 partition, manipulate the contents among the vfat files with a Win 7 file manger, and subsequently return the USBs to some activity as mounted and active Knoppix installs.

I have not observed this type of activity to have any ill effect on Win 7 or XP media or systems.

So, I'd be interested in any opinions on whether I'm doing anything scary here, and moreover, would hope this might prompt the exposiion of other examples of working with ntfs files we might deem safe.

[Forester]

Working with ntfs

Is this not a different topic ? Start a thread in the Lounge and I might venture an opinion.

[krishna.murphy]

Is this not a different topic ? Start a thread in the Lounge and I might venture an opinion.
Actually, it's better to split this off into the MS Windows forum, methinks (NTFS being an M$oft creation.)

Cheers!
Krishna :mrgreen:

[Forester]

Hi utu,

I will get to your question but first I have to respond to Harry Kuhman's post (# 6) under
http://www.knoppix.net/forum/threads/28447-Re-master-your-flash-installation.

Harry is a Sys Admin and I take what Sys Admin's say very seriously. I didn't mean to imply that (I think) accessing Windows partitions from Linux is perfectly safe.

Now ... this is relevant though at first it may not appear so.

My Knoppix 6.4.3 USB installation is less that 4 weeks old. The second bit of software I installed was a plug-in for Eclipse. The plug-in didn't work so I installed it again. This time the plug-in worked fine. Eclipse is like Windows (heck it will even run under Windows): you've a GUI, all you do is point and click and press return when the OK button appears. What could possibly go wrong ?

When I came to remaster I discovered that /UNIONFS/lost+found was not empty. It contained, I believe, files missing from the first installation attempt. So, the file system screwed up.

This was a timely reminder that even accessing Linux file systems under Linux is not perfectly safe.

Anything we do carries some risk. Risk has two halves: the probability of shit happening and how deep the shit is when it does happen.

I am happy using Linux file systems because my experience (going back over many years) is that the probability is low and the consequences not serious. All those years I used Windows 98 I was not happy because experience taught me the probability the file system was damaged after a power cut was quite high and I could loose important files.

I don't know ntfs all that well. When someone whose authority I tend to respect says what Harry said about serious consequences, I think "I'd better not do that again." I don't know and I don't know how Harry knows but I do know that if I start repeating that kind of thing I'm effectively spreading rumour. Just because a rumour gets repeated doesn't make it any more or less true.

Harry wrote that often corruption is not noticed until long after the damage is done. That makes proving cause and effect very difficult. My very first PC gave me lots of file system problems. Until I installed a UPS. I had circumstantial evidence, but no proof, that power brown-outs were the root cause of my problems.

Someone might try statistical analysis to correlate using Linux to access ntfs with later file system problems under Windows but I suspect all they would infer is that inexperienced people tend to make more mistakes than experience people and that reckless people are more likely to wreck things than cautious people.

I can and do approve of Harry reminding people to back up their file systems.

I also approve of this thread. Many people new to Knoppix may see ntfs as their first option when they need some extra disk space. They need to understand a little the risk they may be taking.

[utu]

@Forester

Thanks for this excellent response.
It is a good beginning of a longer discussion I hope will continue.

@KL522, Harry Kuhman, and Capricorny

When I stepped on KL522's thread, posts #'s 5.Forester, 6.Kuhman,
7.KL522, 8.Capricorny, and 9.KL522 seemed to be appropriate to comment on.
Each (seemed to me) was concerned with some ntfs-related fear or uncertainty.

I hope KL522, Kuhman and Capricorny will also participate here.

@krishna

Groucho Marx once remarked, "I've been thrown out of better places than this."
I've got that feeling. Keep up the great work of the Thread Police.

[krishna.murphy]

There are several problems with NTFS, and IMHO it should be avoided where possible. That said, M$oft being as ubiquitous as it is, it is frequently necessary to deal with it even if one chooses not to operate on/under/within their OS-s on a daily basis. Thus I am grateful that ntfs-3g is available, and included in Knoppix. My usage/reasoning is detailed below, but I am not a member of the "don't ever write to NTFS in Linux" camp - it's not SO bad, these days, and sometimes necessary (or at least convenient.)

I am using Knoppix version 6.2.0-DVD from the FAT32-formatted first partition of my hard drive, and there are no doubts that this imposes limitations. In addition to the 4GB of KNOPPIX itself, I have a persistent store of 2 GB, wherein all my system additons (e.g. Google Chrome) and some files kept in home, etc. (all under ext3) reside, leaving about 2GB of FAT32 storage for certain downloads, Windows-readable. Since this is the system my wife and I use to watch downloaded TV and movies (mostly), it follows that more storage for media is required. I fill this need by using slack space on TWO other NTFS partitions with large amounts of relatively dead storage. I often (daily) have several torrents running simultaneously (sometimes more), which can definitely ratchet up the processor load noticeably, especially as those partitions fill up and the files get more fragmented.

This brings us to the most serious drawback of the use of NTFS - it requires defragging, in some form or the other, relatively often. The Linux ext2/3/4 filesystems are relatively immune to fragmentation, and what does exist therein (always some, usually not much) is not a serious handicap and NEVER requires user (or admin) intervention, being automatic.

I have also used Knoppix copied to an NTFS (Windows) partition, and thereunder running, causing auto-mounting when booted. While this is an anecdotal report, and certainly NOT an endorsement of such a practice, I can report that the most serious trouble I had using that system on a daily basis was also processor load, perhaps due to allowing the automatic use of the Windows swap file. Experientially, I found that ntfs3g was quite capable, and robustly handled the access Knoppix required of it to fulfill the role of providing underlying filesystem support, and shares files with Windows with no trouble (unless I tried to hibernate Windows and then access files it thought were locked while it was asleep.)

So, on balance, while I am not thrilled with being forced, as it were, to use a substandard filesystem, and in the near future I expect to eliminate my daily NTFS-access needs, I am content for the time being with using ntfs3g. I don't lose ANY data, and if I keep an eye on the processor load (via the cute little graphic available on the taskbar called "CPU usage monitor") to avoid making the system unresponsive by overloading it, it works fine (sort of, anyway.)

Cheers!
Krishna :mrgreen:

[Forester]

@ Kishna

Thanks for telling us a little of how you use Knoppix. I see posts by Junior Members that don't say how they are trying to use Knoppix with replies from Senior Members saying you can do this and that and the other. I read these replies and I think "that's all very well but ...".

I am using a USB install of the Knoppix 6.4.3 EN DVD at work because I've been tasked to write some scripts for Linux servers and I don't agree that 'cross-development' from Windows is the better option. I dare not repartition the hard drive or anything: it isn't mine. I cannot boot Knoppix: my boss requires I use corporate e-mail and messaging services (so he can yell at me without embarrassment). I want a portable solution because next week I may be using a different PC and next month I may be working for a different client. My compromise was to install VirtualBox on Windows 7 and run Knoppix in a virtual machine. There are advantages as well as disadvantages.

At work I don't use ntfs-3g even though I use the ntfs file system. I have virtual disks, which are large ntfs files on the outside but Linux file systems on the inside. When Knoppix accesses these file systems, the (VirtualBox) driver calls Windows and Windows accesses the nfts file system. That's about as safe as it gets.

One nice thing about this arrangement is that I can keep all the work stuff that probably should not leave the building at work but still take Knoppix and any personal stuff home. I do that because I don't have the time at work to learn as much as I want to. I certainly can't be messing around remastering Knoppix on company time.

At home, you may have guessed, I run Linux, not Windows and I have no real use for ntfs-3g. It was not always that way. At one time I had a home network with several PC and a laptop running Windows 2000. I used Windows shares to make one large network file system. When I first installed Linux at home I naturally used a dual-boot arrangement. In order to 'share' files between Linux and Windows I learnt how to set up a samba network. When a PC was running Linux, I used ntfs-3g to mount the old Windows shares so that other PCs could access data over the samba network just as if the PC were still running Windows. Since I didn't really expect to use this all that often, I used the auto mounter to mount the ntfs file systems only when they were actually in use.

The solution is 'technically' sound but, I suppose, how 'safe' it is depends entirely on how safe ntfs-3g is. I was using this several years ago when it was first included in an Ubuntu distributed it (I don't use Ubuntu any more but that's a different story). I never heard of ntfs-3g being unreliable. I never had a problem but I never used it all that much. The auto mounter and the samba network are still in place but they only ever get used when I do a 'fire drill' to make sure they are still working.

I cannot say whether ntfs-3g is safe or not but I have messed around with different file systems and operating systems and machines and stuff to have a fair idea of what the options are and their pros and cons.

[dinosoep]

after writing a lot to ntfs from knoppix/linux mint I didn't experience any issues while booting to windows.
But chkdsk found some errors on my filesystem and corrected them, so it wasn't that bad.
the antivirus I use in windows 7 has corrupted my filesystem more then knoppix and it was knoppix who repaired the antivirus mistakes :p

[Forester]

Hi utu,



I occasionally store copies of the vfat contents of the USBs on the Windows 7 partition, manipulate the contents among the vfat files with a Win 7 file manger, and subsequently return the USBs to some activity as mounted and active Knoppix installs.


I have read this many times and am still not sure what you are doing so I can't tell if it is safe or scary.

I'll try to sort out the confusion another way.

When you access a file under Linux you are accessing a file on a mounted filesystem. This is true for files on your hard drive, files on a CD or DVD, files on an NFS or CIFS network file system or files on a USB drive or an old style floppy/diskette. It is also true under Windows. The concept of 'disk drive' with a letter corresponds roughly to the mounted filesystem.

On a hard drive you can have more that one partition. Each partition can contain a file system and they don't all have to be the same type of file system.

Suppose my friend lends me a USB hard drive that he says has four partitions on it: a FAT partition, an NTFS partition, a Linux ext3 partition and a Linux swap partition. If I attach the disk to a Linux machine, Linux will see all four partitions and allow me to mount the first three. It won't let me mount the swap partition because there is no file system on it. If I attach the disk to a Windows machine, Windows will see all four partitions but it does not understand the Linux partitions so it will only let me see the VAT and NTFS partitions. I will see two new drives with letters.

The point is the file system is a property of the medium (the disk partition, the USB stick, the DVD) and not the operating system. Where the operating system comes in is that it can only mount file systems it understands. So a FAT file system on a USB stick is always a FAT file system, it doesn't become a vfat file system just because it is plugged into a Linux machine and it certainly does not become an ntfs file system when plugged into a Windows machine.

The file system is inert. One file system cannot corrupt another. Suppose my friend sends me some files on a USB stick but the file system is corrupt. My machine may read garbage where I expect it to read a file. If I copy from the corrupt file system to another file system I won't get a second corrupt file system: I will get a good file containing garbage.

I suppose that if a file system in a partition were damaged so that it looked to be bigger than the partition then writing to it might eventually corrupt the next partition. The only other way I can see a 'corrupt' file system corrupting another is if the corruption is a 'virus' hidden in a file.

So, utu, if you are using a VFAT partition on a USB stick to transfer files from a Linux machine to a Windows machine (or the one machine running both operating systems at different times), then what you are doing is quite safe.

Once upon a time (before the days of the CD), every computer, big and small, had a floppy/diskette drive and every operating system understood the VFAT file system so it was the way to transfer files (I used to send and receive diskette through 'snail-mail' before I had dial-up internet access). The floppy/diskette is gone and we broadband and firewalls and virus scanners so now we use USB sticks to transfer files discretely but VFAT is still the transfer King.

However, VFAT is not a very robust file system. It is easily corrupted. Before you remove a USB stick from a Linux machine, you must unmount it. On a Windows machine you have to 'eject' the USB stick before removing it. If you don't, you risk corrupting the file system.

If you are transferring files, this is not serious as you can just reformat and start again: all you loose is time. However, if you do this to a USB stick with Knoppix on it, you could loose all your persistent data in knoppix-data.img. So back up your USB installation of Knoppix often. (I back mine up daily).

Under Linux your knoppix-data.img. is another file system within a file but if you reboot to Windows, it is just a file and it is quite safe to copy it onto your NTFS hard drive.

[utu]

@ Forester

Greetings.

My scenario is very simple, and I can probably refine my question to be simpler.

I have a 2G SD Card which acts as a LiveUSB, made from a Knoppix 6.X LiveCD.
I assume its format is vfat, readable by either Linux or Windows.

I have a nice Windows 7 set-up, which has three partitions: one is fat32, with OEM stuff on it for booting, service and diagnostics; two are ntfs: one of these is for Windows 7 and file space; the other is ntfs which stores Win7 system images for system backups (or restores, if you prefer). For the time being, I am the more concerned only with the Win 7 OS and some files I may have there.

If I boot Win7, I can mount the vfat Knoppix USB, save, modify, add-to and delete with no Win7 complaints. My usual instance is for saving and resurrecting knoppix-data.img, thereby shortening the time it takes to re-create a few months of accumulated tweaks. I occasionally save a current copy of the whole of the 2Gb USB files contents on Win7, call this File A. On rare occasions, I bring the similar contents of another 2Gb USB file over to Win7, call this
File B. Working within Win7, using its file manager, I delete knoppix-data.img from File B and restore this with a copy of the knoppix-data.img from File A.

I have done this a few times only, but apparently with no ill effects, at least so far. The ill effect I most fear is screwing up either the OS or the files on Win7.

The file manager operations on my vfat files, I assume, are translated to an ntfs environment, operated upon there using ntfs rules, and the results then re-translated back to the vfat medium.

I am aware of remastering as an alternative, an am not at this time interested in that avenue. I am most interested in getting a sound professional appraisal of the simple process I have outlined.

If it turns out there is a hazard that the vfat product may have some difficulties I haven't anticipated, I can live with that. That might be interesting.

[Forester]

@ Forester

My scenario is very simple, and I can probably refine my question to be simpler.



Thanks utu for your description. I can't see anything dangerous in what you are doing.

I'm a bit surprised you say you copy the entire contents of the USB onto the hard drive and all you get is a file. I'd expect a folder with a few files and a couple of sub folders.

Anyway, the important thing is you are writing to the ntfs file system using Windows so you aren't running any (more) risks (than the next guy).

If you not remastering then I think you will find the only file that ever changes on the USB's FAT file system is knoppix-data.img so you could probably simplify your procedure further and even save yourself a little time and disk space.

[Forester]

NTFS

So, on balance, while I am not thrilled with being forced, as it were, to use a substandard filesystem, ... I am content for the time being with using ntfs3g.


Hmm .... I fear that you may be being a tad libellous in branding (Windows) NTFS a sub-standard file system for I suspect much of what you complain about is a result of (the way you are) using (the Linux) ntfs-3g driver. Tut, tut. Shame on you. ;)



This brings us to the most serious drawback of the use of NTFS - it requires defragging, in some form or the other, relatively often.


From the one time I read up on Windows NTFS, I remember something along the lines of "A whole generation grew up defragging FAT file systems under Windows 98 and continue religiously defragging NFTS file systems when there really is no need to." I've defragged NTFS file systems but only as a prelude to cuttings the b******ds down to size so I could use the disk space for something useful.

It is not that fragmentation does not occur on NTFS: it is more that fragmentation has no adverse affect on performance. If you do really have performance problems due to fragmentation, it may be an ntfs-3g issue.



I often (daily) have several torrents running simultaneously (sometimes more).


Think about what you're doing. You are down loading a large file in small chunks in random order and, you admit, sometimes you are down loading more than several at once. I think fragmentation is inevitable.


debian@aspen:~$ sudo filefrag /media/iso/KNOPPIX_V6.2.1DVD-2010-01-31-EN/KNOPPIX_V6.2.1DVD-2010-01-31-EN.iso
/media/iso/KNOPPIX_V6.2.1DVD-2010-01-31-EN/KNOPPIX_V6.2.1DVD-2010-01-31-EN.iso: 106113 extents found
That's a torrent download I did recently under Linux to an XFS file system. I quote Wikipedia: "XFS is particularly proficient at handling large files and at offering smooth data transfers." 106 133 extents: looks pretty fragmented to me. My bitTorrent client claims the download had less that 4 500 chunks !



I keep an eye on the processor load ... to avoid making the system unresponsive by overloading it, it works fine (sort of, anyway.)


The processor load surprised me at first.

For me, just one torrent download can max out my broad band. Not only does my browser become unresponsive, but so do the browsers anyone else might be using elsewhere in the house. So I don't: I limit the bandwidth of individual torrents but I can also limit the grand total.

If I had fast enough broadband, I'd expect the disk access to max out before the processor load (same effect - system goes treacle). Remember you're using ntfs-3g. That's a File System In User Space (aka FUSE) driver. No way is it going to be as efficient as a lean, mean Linux Kernel driver or the genuine Windows kernel driver from M$soft. Furthermore, since NTFS is a complex file system, I expect the design goals for ntfs-3g were correct operation rather than efficient use of processor or memory. Now I'm not so surprised.


... in the near future I expect to eliminate my daily NTFS-access needs ...Sounds like you've a serious habit to kick. :)

If it were me, I'd free up space on the NTFS partitions, defrag 'em and shrink 'em and use the space free for a new partition with a file system better suited to the job. I'd do some research starting with jfs and xfs.

I suspect that even using the space freed for another NTFS partition dedicated to torrent downloads would give better performance. Also, if Harry's nightmare comes true, all you lose is a few TV recordings, not them and your OS, personal correspondence, bank records and the e-mail address of that cute little ....

Cheers!