How to copy an entire drive more efficiently than using dd

[fergus]

I have a 250G drive dual-booting Windows and Linux and beautifully partitioned to allow swap partitions, FAT32 drives for two versions of Cygwin, and others (from /dev/sda1 to /dev/sda10). The architecture has been polished and refined over time and I would hate to have to reproduce it after a fault or loss.

The back up procedure is not to back up new/ changed files drive by drive, or even to copy drive to drive, but to copy an image of the entire device /dev/sda to a remote drive, using dd. Of course, the entire 250059350016 byte device is copied to a 250059350016 copy, this way.

It's worth it to do it this way and not to get into the vocabulary of chasing new/ changed files, mimicking the same gymnastics from drive to drive, etc. AND the boot partition is also backed up should there be a requirement to use it. The down side is that it takes 2h20m to achieve and if I wanted to look for just one file within the backup, this is of course impossible, the back up just consisting as it does of a single 250G binary file (with lots of zeroes).

Needless to say, all the initial partitioning and subsequent routine backup is achieved with Knoppix.

Question 1. Is there something like partimage (but not actually partimage) that will back up _an_entire_device_ /dev/sda rather than individual partitions /dev/sda[1-10] but that would work faster and more efficiently using some kind of internal zipping protocol (as I assume partimage does, reducing say a 16G partition to a 5G image). Possibly reduce the 250G image to dunno? 80G or something and all in an hour or less.

Question 2. Could dd_rescue be implemented at all usefully? (I have no bad sectors, but maybe dd_rescue skates efficiently over swathes of 00s or something, I dunno?)

Fergus

[klaus2008]

Do you use a USB hdd? Then I don't think that it is only an issue with the program dd because you have a transfer speed of 28 MB/s. I think this is a good value if it is a USB hdd. If you want to reach higher speeds then you must use an eSATA hdd.

if I wanted to look for just one file within the backup, this is of course impossible, the back up just consisting as it does of a single 250G binary file

There exists a program called kpartx in the debian repository (part of the linux multipath-tools) which helps to use disk images created by dd as loop back devices. Maybe this program can be installed and used inside Knoppix.

[OErjan]

actually you can mount that image just like it was a hdd with addition of -o loop at end and using image as a hdd.
like this,
first "I" created an image with random caracters and formated it EXT3 so I could show you how, ignore this part.
dd if=/dev/urandom of=/home/knoppix/a.img bs=1M count=1024
sudo mkfs.ext3 /home//knoppix/a.img
i get a message that it is not a block unit and am asked if i want to continue, i answer Y.

now to part that YOU want. I simply mount it with option -o loop.

sudo mount -t ext3 /home/knoppix/a.img /media/testing -o loop

you have to use separate image for each partition but i think it is still fairly easy.
do not forget to umount it before deleting or some such (to prevent common mistake, umount without n) when you are finished with it.

sudo umount  /media/testing

also another thing, you can use "gzip -9" to compress the image as you go, this will "remove" most of the unwritten areas and some of the others from the final image size.
first how to clone with gzip.

dd if=/dev/disk_to_be_cloned |gzip -9 >/mnt/disk_where_clone_is_saved/image.dd.gz

now to restoring the image.
gzip -cd image.dd.gz > /dev/cloned_disk

was this helpful?
I am now on knoppix5.1.1 (on the road and it was in my computer bag) but it should be same on most Linux versions with gzip, dd and kernel loop abilities.

keep at it and keep asking.

[ckamin]

Do a search for "CloneZilla". It is a disk clone/imaging tool that seems to work well. There are other solutions, but I have used this one extensively with excellent results. It has networking capabilities and can backup to remote shares as well.