Practical compression advice needed (and a samba mystery)

[WedDa]

Dear forum,

You guys have no idea how much you've helped me in the past when I have stumbled into Knoppix problems and scanned the forum. Thanx! This time though, I think I need to actually post my prob because I can not find a similar problem/request here or elsewhere. Its a general "usage" problem so I guess you don't have to be a hacker to give me some advice. So, here I go.

I am in charge of managing a couple of clustered Quantian 0.5.9.3 (http://dirk.eddelbuettel.com/quantian.html) boxes that we use for biological computation at my department, and to make regular backups of important files. The backups are meant to be transferred to a HP Windows NTFS-RAID server. The server is reachable with (tada!) samba. I have tried mirroring the important folders with rdiff-backup or rsync. But the first doesn't seem to work with a mounted samba drive and the second fails to transfer all files (I think due to odd characters in the file names - like : and ?) and permissions and ownership screws up anyways.

A bz2 compressed tar archive of these files weighs in at about 4 Gigs and can be produced in about two hours or so, which is ok if I do the backup weekly. However, copying such a file to the server is not possible because the copying-process stops after 2,0Gigs or 2 147 483 647 bytes! I have no idea where this limitation appears, but it's really a pain. Therefor I can use arj instead to produce multiple archives, each of which is smaller than 2,0 gigs. Which can then be transferred. However, I have only managed to run arj on a file or folder and not directly on a pipe from tar, which whould be much nicer and faster and requiring less free diskspace. Thus my horde of questions:

Do you think there is a way around the 2 Gig-transfer problem?
Can arj be used in a "tar <files> | arj <archive>" fashion?
Can any of the other free compression utilities like gzip and bzip2 create multiple archives? And of course then also be "piped to".
Am I missing a very different (and simple) solution to this problem?

Regards,
WedDa

[Markus]

You could split the tar file like so: tar -cvzf - source_file | split -b 2000m - tarpiece-
and later glue them together with: cat tarpiece-* > filename.tar.gz

[WedDa]

Aha! I suspected that there was a simple solution to this. Thanx a M Markus!

[firebyrd10]

You could split the tar file like so: tar -cvzf - source_file | split -b 2000m - tarpiece-
and later glue them together with: cat tarpiece-* > filename.tar.gz

Though that techinicly won't make 2 gigs you need 2048m for 2 gigs.

Anway, if your running a windows machine you can peiece that back together with a batch file.

just start a new txt doc and rename it (whatever).bat

then rightclick -> edit and type

Copy /b cut.001 + cut.002 + .. + cut.nnn file.out

changing cut.xxx to the file name (split makes files with the aa. ab, ac, ad,...) and then changing file.out to what you want the final file name to be.

I haven't figured out the cut.nnn part, but it seems to be needed just type the filename -aa ,ab, ac... with .nnn on it.

If that doesn't work you might need to take the aa,ab,ac... parts and make them the extension.

I had to do that when making a back up of my linux partiton because my destination drive is FAT, so I made the copy then peiced it back togther on windows.

[tearinghairout]

Maybe this might fix your problem without needing to cut the file up...

The smb file system driver in Linux 2.4 has a 2GB size limit.

Actually, according to the samba team, the 2Gb limit is actually if you use
mount -t smbfs command which is not within the scope of samba. they
recommend using smbclient.

found here http://www.redhat.com/archives/fedor.../msg00173.html

[Markus]

You could split the tar file like so: tar -cvzf - source_file | split -b 2000m - tarpiece-
and later glue them together with: cat tarpiece-* > filename.tar.gz

Though that techinicly won't make 2 gigs you need 2048m for 2 gigs.

Umm, yes. But I understood that he wanted <2GB splits.
Even though, if tearinghearout's solution to overcome the limitation works, all is well without needing to split.

[OErjan]

A DVDburner in any onld linuxbox on that network should be good backup of those 4G tar files. and should be an extra layer of backup (if the files are then restored in said server usng the DVD's).

[WedDa]

Wow, guys thanx for your help! I have to look into smbclient and see if it can do the trick.

[firebyrd10]

You could split the tar file like so: tar -cvzf - source_file | split -b 2000m - tarpiece-
and later glue them together with: cat tarpiece-* > filename.tar.gz

Though that techinicly won't make 2 gigs you need 2048m for 2 gigs.

Umm, yes. But I understood that he wanted <2GB splits.
Even though, if tearinghearout's solution to overcome the limitation works, all is well without needing to split.

I know, I just had to say that for some reason.

[Markus]

You could split the tar file like so: tar -cvzf - source_file | split -b 2000m - tarpiece-
and later glue them together with: cat tarpiece-* > filename.tar.gz

Though that techinicly won't make 2 gigs you need 2048m for 2 gigs.

Umm, yes. But I understood that he wanted <2GB splits.
Even though, if tearinghearout's solution to overcome the limitation works, all is well without needing to split.

I know, I just had to say that for some reason.

I'm not going to reason with that