Copy File Transfer Speeds & Stalling [Recovery to USB HD

[KnoppixNubster]

Hi,

Windows has corrupted and im trying to use Knoppix to copy files to my external hard drive before I format the internal drive.
I have managed to find my files using knoppix and am able to copy/paste these to my external hard drive.

I get transfer rates of up to 1000 KB/s for a few seconds then the rate drops to about 160KB/s then stalls for up to a minute before restarting at 1000KB/s and stalling again. It spends more time stalled than actually transferring data. I'm trying to copy files totalling 90GB and Knoppix is estimating that this will take 47 hours (i'm assuming that this excludes time spent stalled?).

Both the drive and the USB port are USB2.0 and I was expecting a faster transfer rate than what i'm getting, nevermind the frequent stalling.

Are these normal transfer rates or have I omitted some settings?

Or is there a better way to back-up my files?

Thanks for any help, as I'm a complete newb to all this.

[Harry Kuhman]

Or is there a better way to back-up my files?....
You're likely not going to be happy about hearing this and may not even accept it, but years of experience reading these forums and following the experiences of many people has me convinced that it is never safe to write to a NTFS partitioned hard drive with Knoppix. You didn't say how your USB2 drive is partitioned, so I'll assume that it is NTFS. If that is that case I would strongly suggest that you do not try to back up files to it with Knoppix, and further I would suggest that you consider the drive potentially corrupted even if you don't yet see any problems. If it were my drive I would get all of the files that are currently on it off of the drive and onto another system, and then completely reformat the drive before I had any faith in it at all.

So is there a better way to back up your files (recover your files actually, since there is obviously no backup)? This depends on the number and size of the files. For a small number of files, or small files, you can use a FAT formatted usb device, such as a flash memory "stick". For large numbers of files, and files 4 gig or larger that will not fit on a FAT partition, I like to connect two system together on a network and transfer the files across a network to another Windows machine. Since it is the second windows machine that is doing the actual file writing it is relatively safe to write to NTFS that way. (I say relatively safe because you have already demonstrated that writing to NTFS is not always safe even with Windows, let alone with Linux, and personal experience and many posts here lead me to believe that external drives under Windows are particularly vulnerable). I do this by installing a FTP server on the target Windows system and using Knoquror from Knoppix on the system to recover files from as the FTP client. There are other ways to transfer the files, but this works well and it turns out that a FTP transfer is much much faster than using Microsoft file sharing (when you can get it to work at all).

There are many completely free FTP servers available for Windows on the web.

There are other ways to do this also. You might even just want to write the files to CD or DVD with Knoppix. That would not only get them off the failing system, but would give you the real physical backup of these apparently important files that you have failed to make so far.

[KnoppixNubster]

Thanks for your detailed response Harry,

Luckily during my research in how to recover my files I had read about the problems writing to NTFS drives, so I partitioned the drive (that I bought today) and made a 160GB and a 90GB FAT32 and left 250GB unallocated. I am attempting to move files from the windows HD to the 90GB FAT32 Partition on the External USB Drive.

The files I am attempting to recover are RAW High Def video's from my camcorder of my daughter. Unfortunately less than 10% of them have been edited processed converted to DVD's (I cant believe how stupid i've been not having them all backed up, my fault for not putting the hours in to edit and process them more frequently). There are 1501 files totalling 89GB to transfer and i'm pretty sure (99.9%) that no individual file is over 4GB. There are other files that I will try to recover also, but these videos are the ones that I am desperate to save.

I have access to a second laptop with Vista installed and would be prepared to try the ftp route. Is it difficult to set up an FTP server? or to get Konqueror to act as a FTP client? So far I haven't even been able to get the internet working with Knoppix.

As you didn't comment on the transfer speeds that I'm achieving or the frequent periods where the transfer is stalled, am I to assume that this is normal?

Thanks

[Harry Kuhman]

It really should only take a few minutes to set up an FTP server, although the first time one does it one may wish to go a little slower and be extra careful.

I deliberately didn't recommend a server because the one that I last used was "upgraded" and I never did like the new "improved" version. So I stuck with an older version that I can no longer find offered on the web. All that was really involved was running the install program and then adding a user account from the server configuration screen so that you would have an account that had the privileged to write files to the server. If you are going to try to make the FTP server available to others on the Internet (not required here) then you also need to know about forwarding ports in your firewall and may need to learn about active and passive FTP transfers.

I did see some new (to me) FTP servers offered recently at FreewareFiles.com and was going to try installing some, but had not got to that on my to-do list yet. You may get a chance to do that before I do. One that caught my eye was called Home FTP Server 1.7.x.xxx , but as I have not yet tried it I'm certainly not recommending it over any other FTP server that you find.

I didn't comment on the transfer speed mostly because I don't use an external hard drive with USB. I have used one with firewire, and even that is slow. USB is much slower (in spite of the supposedly slightly higher data rate that USB2 claims over firewire, since USB wastes so much of it's capacity on overhead). USB is just a bad technology, and it may just be that you are not going to get good performance out of it, although I don't know that you don't have some other issue involved here.

[KnoppixNubster]

Hi Harry,

Is it possible for me to transfer files via FTP and USB simulateously? As i'm hesitant to stop the USB transfer which has been going for 18 hours now and has so far transferred 8.5GB.

Previously when I have used my USB pen stick to transfer files from my Windows laptop I would achieve transfer rates of around 6MB/s. I'd assumed that similar rates would be achievable with Knoppix. But the fastest that I have seen the transfer achieve is 1.2MB/s, that is only for 1 or 2 seconds before the speed starts dropping to zero for the next stall.
To be honest though i'd be happy if the Knoppix transfer would just continue without the regular stalling. I've been timing it and on average I get 10 seconds of transfer followed by 20 seconds of stall.

Do these periods of stall also occur is transferring via FTP?

Thanks for your help

[Harry Kuhman]

I don't know what this USB transfer problem is. I was thinking it was just the slow nature of USB, but if Windows had done better than Knoppix is doing then it apparently is some other issue.

I should mention that high speed burns can cause some strange performance problems, see here for one example (http://www.knoppix.net/forum/viewtopic.php?p=78582#78582), but I really don't suspect that in this case, although I can't completely rule it out from the little that I know.

As to can you do both at once, under normal conditions the answer would be yes, but these are not normal conditions. If the delays are being caused by something that Knoppix needs to do to keep trying to get data off the hard drive that it is trying to recover files from then two operations at once would likely make things much worse. Or if the slow down is indicative of some sort of low memory problem then doing other things at the same time would also be problematic.

I expect that if you got a FTP transfer going that you would only do an FTP transfer, and that all the files could be transfered over to another system in a reasonable time (a lot faster than you can transfer files from Linux to Windows Xp when using Microsoft file sharing protocols). But that is only from my own past experience, if there is some issue reading the recovery drive or some issue running Knoppix on that system (such as reading the disc) that is slowing things down then you could well see similar problems doing an FTP transfer.

This is extremely speculative, but you might even have some sort of hardware problem that is slowing down Knoppix and could be the reason that Windows can't run at all, such as an unexpected interrupt that is constantly firing and will not stop.

One technique that might help, if you are using an CD version of Knoppix rather than a DVD version, and you have plenty of memory (as many modern systems do), would be to use the toram cheat code and force all of Knoppix to load into ram at boot time. This does make the boot a lot slower, but gives a faster running Knoppix after that and would completely eliminate any issues of needing re-read anything off of CD. Unfortunately, to test this you would have to stop your USB transfer that you are reluctant to stop, but if the problem persisted after using the toram cheat code it would rule out burn speed being the cause of this particular problem (and if it did correct the problem would be a reason to suspect that burn speed was indeed an issue). This would be a worthwhile test to do if you can later, after your USB transfer finally finishes.

[chip.ling]

Is it possible for me to transfer files via FTP and USB simulateously? As i'm hesitant to stop the USB transfer which has been going for 18 hours now and has so far transferred 8.5GB.


Yes, you can do it at the same time. But the bottleneck will be the I/O on the hard disk since both process request files from the same physical hard disk.

To setup the ftp, you can check out the FAQ below:
http://www.knoppix.net/wiki/Network_FTP_FAQ

Another alternative is to set up the a NFS (network file system) using Samba on knoppix machine and share the harddisk with whatever other windows machines you have on your network. See below FAQ for the setup:
http://www.knoppix.net/wiki/Network_Shared_Drive_FAQ

However, with what has described in your post, it is very likely your harddisk is almost dead or some blocks are corrupted. The excessive time the system is using to copy the files can be due to a lot of read errors encountered. Usually the system will keep on retry until it successfully read the block before it moves on to the next one.

You can try all 3 methods above and hopefully you can save your data.

Good luck.

Rgds,
Chip

[KnoppixNubster]

An update:
The periods of transfer somehow increased in length so during the second half of the transfer I was getting 10-15 seconds of stall followed by 15-20 seconds of transfer. So the transfer completed today ahead of my original estimates.

I ran into problems setting up ftp, I think because I was using a very old knoppix 'MiniKNOPPIX 2005' Based upon Knoppix 3.7.
I tried using newer versions but none of these could read my HD. The only other live CD that could read my HD was SysRescueCD but I couldn't figure out how to ftp using this. So I have gone back using MiniKnoppix '05 to recover the rest of my files via USB transfer.

I am very impressed with Knoppix for handling a 3.5 day continuous file transfer without any hiccups. Im sure Windows would not have completed this task! Also my laptop has not got hot at all, in fact I can never before recall it running so quietly and cooly.

I think that the frequent stall periods are most likely due to bad sectors on my hard drive and am grateful that they could be recovered at any speed at all, even if I had had to wait a month for these files to transfer then I would have done.

Thanks to both Harry and Chip for your assistance in this thread.

[Harry Kuhman]

If you used the approach that I take, installing a FTP server on the target windows system and using Konquror as the FTP client then the version or age of Knoppix should not matter.

The speed up may well indicate that Chip is correct (which I also alluded to as a potential problem but not as clearly stated) that the hard drive is going bad and the the system was retrying reads to recover the data. It is not unusual for such drives to be worse in some sections of the drive than in others. You should use great care in checking this drive before reinstalling anything to it, as it could well fail again. There is software call SpinRite that can analyze and perhaps even repair the drive if it is only a formatting issue and not a hardware issue, but unfortunately SpinRite is overly expensive (more than the cost of a new drive) and in spite of the author's claims it really only recovers data from this type of problem, not general Windows corruption issues (in my experience). So if you don't already have access to it I would not advise buying it in this case, better to spend the cash on a big new drive.

Using Knoppix 3.7 could be a problem in trying to recover data from SATA drives; I don't believe that 3.7 had SATA support, although it's been quite a while and I could be remembering incorrectly. I would suggest grabbing one or more 5.x versions while they are still available (the 6.x versions are disappointing to a number of people, including myself).

Thanks for posting the update and I'm glad that you were able to recover your files.

[Man in the Moon]

I have a similar problem, but there is nothing wrong with my hard drive except that windows has become unbootable and unrepairable. The drive and file system is intact. So I'm trying to transfer the files to my USB hard drive. The problem is that every time I start the transfer, Knoppix automatically and for no reason changes the device file for my USB drive and remounts it ro. There is apparently no way to stop this from happening. So I basically cannot use an external USB drive with Knoppix (6.0.1). I really like the idea of burning the files directly to a CD as Harry suggested, but how do I do this when the CD drive has to have the Knoppix CD in it to do anything and can't even be unmounted while Knoppix is running?

[Harry Kuhman]

... but how do I do this when the CD drive has to have the Knoppix CD in it to do anything and can't even be unmounted while Knoppix is running?Multiple drives, or, as long as you have 1 gig or more memory, use the toram cheat code and a CD version of Knoppix (not DVD), which will load all of Knoppix to memory at boot time, freeing up the drive.