Live CD and network file transfer speeds

[camel_456]

I'm completely new to Linux and Knoppix and am using the latest Live CD
I'm using Knoppix to transfer a large amount of data (about 190GB) from a Linux formatted hard drive over to my new NAS unit which is connected to the computed via an Ethernet cable. The unit is 10/100MBps (not a gigabit unit)
So I have started the transfer but the speed it at max 2.7MBps, and will take 18 hours to complete (and that’s just 165GB's worth)
I don't really fancy leaving my computer on over night since it's quite noisy and so is the NAS unit.
My question is basically, is the fact that I am using the Live CD hindering my transfer rate, or is this what I would be experiencing if I had Knoppix installed on the hard drive?

Thanks for your help

[Harry Kuhman]

You didn't give any details about your NAS or the system that your are running Knoppix on, but there is absolutely no reason to expect installing Knoppix will make things any faster. If anything, "installing" knoppix generally makes things worse.

If you have a gig of memory or more and you really really think that this is a cd speed issue, you could boot a cd version of Knoppix with the toram cheat code and run completely from memory. I would expect no improvement.

I expect that the speeds that you are experiencing are due to the NAS system itself. But you might be looking at some speed issues in Knoppix if running on very old slow and memory starved hardware. There is also a very very small chance that you are looking at some network speed issue, but I really don't think that is likely.


Since you have a NAS it seems likely that you have more than one computer, Some things to look at:

How fast can the other systems write to the NAS? (use a mix of file sizes similar to your Knoppix transfers for this test, a huge number of small files may transfer much slower that a single large file of the same size).

How fast can you transfer files between Knoppix and the other systems?

how fast can you transfer files between the other systems?

What software is the NAS based on? There seems to be a deliberate and significant slow down in XP when it is transferring files to non-XP systems as contrasted to transfering with other XP systems. This is true not just for transfers to Linux but also transfers with Win98.

[eadz]

With a 100Mbs link, the maximum theoretical speed is in the ballpark of 10 megabytes a second. I usually see 7-8MB/s, so you're not toooo far off.
There is overhead of TCP/IP, and whatever file sharing protocol you are using, so you never see full throughput. If you have the cable connected to a hub that could slow things down, or a switch under heavy load.

[johnrw]

There must be usb in there somewhere. Is the drive attached to your nas through a usb port? That is the same write speed I got on my Linksys Wrtsl54gs... that had a fast external drive attached through usb. Without more of the details about your 'nas' device... I'll just guess it has a usb bottleneck in it. I dragged an old PIII machine out of the heap, and stuck a pci sata card and a 1tb drive in it, which runs Knoppix and serves files through Samba.

[Harry Kuhman]

....I dragged an old PIII machine out of the heap, and stuck a pci sata card and a 1tb drive in it, which runs Knoppix and serves files through Samba.
What speeds are you seeing? And are you actually using Knoppix in this task, or did you take to more reasonable (IMHO) approach of using a more target distro such as Debian?

Also, I'm curious, were there any disk space BIOS problems here? I generally had not thought of doing this because my older systems all had serious IDE BIOS disk size limits. I may have to track down an inexpensive SATA interface and try this, would give some new life to an old system and could have some nice utility.

[johnrw]

Hi Harry,
Actually I am at the end of my need for speed campaign... brought about by the routers with a usb craze I fell into. Time is money... and I was losing too much time messing with openwrt's x86 port (which once I got it running was a complete disaster... in networking throughput terms, ie there wasn't any to speak of) and also of buying a couple different "nas" routers... which were much better as routers... than they were with making drives available. SLOW.
My wife begged for her Linksys back!

Other Nas like distro's... I tried them all... ebox(Ubuntu only now), devil-linux(kernel 2.4 anyone?), openwrt-x86(x86 was toooo slow network wise), freenas(No workie with Silicon Image chips, ie most of those cheap sata chips out there)... and some names I can't remember. Nothing compares to Knoppix. It is the World Class Expertise that Klaus has with building a complete distro... I just can't seem to find a replacement for. While I was wasting all that time... my gut kept telling me... I was going to do what I needed to do with Knoppix, again. But I checked out other solutions 'out there.'
None were for me.

So here is what I did. I bought a SIIG SCUNS0-12. Here at Compuvest for $30 US (http://www.compuvest.us/ProductDetails.aspx?ProductID=14542) It has 4 USB 2 ports, 3 Firewire ports, and a Silicon Image 3112 sata chip with 2 sata connectors. It would not work with my 1 tb Seagate drive... UNTIL I upgraded the bios. Trung Pham, at Siig Tech Support... assured me it would fix that issue so I took the plunge. He said I had to use a floppy disk, and I said i'll make a bootable cd cause I don't have a floppy on that machine. So I made a bootable cd with Dos and their bios upgrade on it. Worked like a charm. It gets around my old bios problem with their own bios. Just know in advance this sata chip... is so problematic on BSD... you cannot use it with things like www.freenas.org. I didn't want to touch BSD... so I am okay with that. On linux... it is fine. Here is the bios I used. http://www.siliconimage.com/docs/BIO-003112-xxx-4284.zip I only flashed the Raid bios portion into the card. It works. Now... this is a Sata 150... and the drive is a Sata 300... but the drive can deal with it just fine. Since I am providing urls... http://www.siliconimage.com/docs/UpdFlash_v336.zip is the Dos based firmware upgrade utility I used.

Okay so the new Raid bios upgraded my card. Card and drive and Knoppix workie nicely. :)
Now about Knoppix, Samba, NFS, and other choices one could make. Knoppix has a nice Share Everything option when you use it's Start Samba server item in the Knoppix menu. I used that to generate a starting point for the /etc/samba/smb.conf. As long as I don't use it everytime I start Samba, because it creates a new /etc/samba/smb.conf I am okay. I messed with NFS... until it kept some open files on shutdown... and caused fsck to run a couple of times. Chucked NFS... which actually got a higher throughput speed. 8 or 9 MB/sec. But since Samba plays nicer on my persistent home's file system. I get around 7 or 8 MB/sec with Samba. Read or write... if I recall my numbers correctly. I know I was playing file transfer games and transferred my
10,450,000,000 bytes knoppix.img file to the KnoppixNas box... in 18 or 19 minutes. I could not bare to think how long that would have taken on one of those usb routers.

As to which distro I used... I installed Knoppix to disk using the New 0wn (Zero Work Needed) installer. Ripped a bunch off stuff out of it... and added some stuff so I could build Openwrt's x86 port, which uses a cross compiler, to build x86 stuff! So that partition is quite ragged... dpkg warns about cloop info files missing... assuming it is not installed and what not... but I could have done this on a fresh 0wn install just the same. I did grab a recent Debian Lenny/Sid weekly snapshot, installed that to a different partition, but I am too used to Knoppix... and am too fond of his quick and dirty fixes like remountrw to need a full debian to just serve up some Samba managed partitions as a real Nas device. It runs without a monitor or keyboard now, and I telnet into it to do stuff. I also rigged a null modem cable to my real Knoppix machine... so I can manage grub4dos on bootup if I want to use a different boot option for some special case things. For some reason... I can't seem to get Knoppix to use a serial console. Maybe it wasn't built into the kernel or something. Telnet is fine for my purposes.

The main reason I picked this old PIII machine... is it is almost silent. The power supply fan, the cpu fan, barely audible compared to some 'modern' pc's. And if it dies next month... I'll just replace it with another from the heap. But a PIII 833 Mhz, has all the horsepower needed for a 100Mbit network. Could do OpenVPN too... which I can't say for most of those router appliances out there. When I transfer a large file, and look at the cpu usage inside the KnoppixNas box... it sits at 16%. So it could handle a few of those transfers simultaneously.

Since I am using it this way... I use the Samba "force user = knoppix" and "force group = knoppix" option in my /etc/samba/smb.conf otherwise I wind up with files owned by uid/gid's that I wind up not having permissions to delete or move around. Syncronizing numeric uid/guid is too much of a pain.

Okay Harry, for me... it is 3AM... so I hope this answers what questions you had...

[Harry Kuhman]

Thanks.