Please help me reassemble my RAID5! (long)
Hello, all. "New to Linux" - yep. I've been knee-deep in Linux for about a week now. And I've got a RAID5 array that I'd love to mount and get my approx. 700GB of irreplaceable final cut projects, Logic audio projects, DVD Studio Pro projects, screenplays, photos, and other such intellectual property off of in one piece. I'm hoping that the kind and wise among you can give me a few hints - or at least suggest next steps.
In 2005 I built a Rebyte NAS precisely because I didn't want to learn Linux. I think it's JFS; I wish I'd written that down. It has functioned without issue for two years as a file server. It has *NOT* been backed up because, like most idiots, I assumed that a RAID5 array *was* a backup (yes, lesson learned, thankyouverymuch). Last saturday it corrupted some photos while I was browsing them, threw me a whole bunch of file system errors, and then when I rebooted, it wanted a floppy.
Bad news. Long story short, the BIOS on the mobo blew up. So: new motherboard, new CPU, new memory, new power supply, same Promise ATA133 IDE card, same 4 Maxtor 300GB IDE drives.
The Rebyte runs on a Delkin flash card; little IDE dood. It lives on the mobo; the drives live on the Promise. So we hook everything back up, and power on.
I got:
--- rd:4 wd:2 fd:2
disk 0, s:0, o:0, n:0 rd:0 us:1 dev:[dev 00:00]
disk 1, s:0, o:1, n:1 rd:1 us:1 dev:hdf1
disk 2, s:0, o:1, n:2 rd:2 us:1 dev:hdg1
disk 3, s:0, o:0, n:3 rd:3 us:1 dev:[dev 00:00]
raid5: failed to run raid set md0
md: pers->run() failed...
md :do_md_run() returned -22
md: md0 stopped.
md: unbind<hdg1,1>
md: export_rdev(hdg1)
md: unbind<hdf1,0)
md: ... autorun DONE
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 8192 buckets, 64Kbytes
TCP: Hash tables configured (established 262144 bind 65536)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
ds: no socket drivers loaded!
kmod: failed to exec /sbin/modprobe -s -k block-major-3, errno = 2
VFS: Cannot open root device "303" or 03:03
Please append a correct "root=" boot option
Kernel panic: VFS: Unable to mount root fs on 03:03
...not cool. Not cool at all. Now would be a good time to mention that Rebyte hasn't been supported since, oh, two months after I bought it. They've packed up for parts unknown.
So some theorizing - since it's a LILO linux install, the array (which I think was JFS - wish I'd written that down!) should be addressable by some Linux flavor or other. The thinking was: mount Ubuntu or the like off a LiveCD and try and get things hoppin'.
Well, the Promise Card doesn't like four hard drives and a CD-ROM. No way, no how. So that cooks off any notion of Ubuntu. Fortunately we got Knoppix (KDE 3.5.5, release 2.6.19) up and running off a USB flash drive. So: I've got four drives on my desktop, they won't mount, but they're called hde1, hdf1, hdg1, and hdh1.
Awright. Progress.
This is about where I start gettin' jiggy with the commands that I have only the vaguest understanding of (remember: I went with an embedded Linux appliance so I wouldn't have to make sense of this stuff... that plan turned out well).
~$ sudo mdadm --examine /dev/hde1
/dev/hde1:
Magic : a92b4efc
Version : 00.90.00
UUID : e632930d6:7bfeb139:d5864e82:7e7b2ad7
Creation Time : Thu May 19 08:05:12 2005
Raid Level : raid5
Device Size : 293049600 (279.47 GiB 300.08 GB)
Array Size : 879148800 (838.42 GiB 900.25 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor 0
Update Time : Sat May 21 10:19:09 2005
State : Clean
Active Devices: 4
Working Devices: 4
Failed Devices: 0
Spare Devices : 0
Checksum : f5f885fc - correct
Events : 0.28
Layout : left-symmetric
Chunk Size : 32k
Number Major Minor RaidDevice State
this 0 33 1 0 active sync /dev/hde1
0 0 33 1 0 active sync /dev/hde1
1 1 33 65 1 active sync /dev/hdf1
2 2 34 1 2 active sync /dev/hdg1
3 3 34 65 3 active sync /dev/hdh1
Hokay. There's a RAID array there; I just can't talk to it. (Anybody see anything else that I, with my severely untrained eye, do not see?)
~$ sudo fdisk -l /dev/hde1 (or hdf1, or hdg1, or hdh1)
Disk /dev/hde1: 300.0 GB, 300082889728 bytes
255 heads, 63 sectors/track, 36482 cylinders
Units = cylinders of 16065 *512 = 8225280 bytes
Disk /dev/hde1 doesn't contain a valid partition table
"doesn't contain a valid partition table" worries me. I like the sound of it much better than "Unable to mount root fs" but it still isn't reassuring.
~$ sudo fsck /dev/hde1
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
Group descriptors look bad... trying backup blocks...
Superblock has an invalid ext3 journal (inode 8).
Clear<y>? no
fsck.ext3: Illegal inode number while checking ext3 journal for /dev/hde1
I don't know what an invalid ext3 journal is. I *DO* know that I'm not about to clear something what I don't know what it is. But it seems like a problem. How much trouble am I likely to get into by letting it clear it?
~$ sudo fsck /dev/hdf1
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/hdf1
The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else) then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
I'm thinking that not being able to be read is because it's not the first drive in the RAID. Am I correct in this thinking?
Didn't want to e2fsck nuthin' without knowing what I was doing. Which I don't. Which is what I'm hoping you wonderful people can help me with.
~$ sudo mount /dev/hde1
mount: wrong fs type, bad option, bad superblock on /dev/hde1,
missing codepage or other error
In some cases useful info is found in syslong - try
dmesg | tail or so
~$ dmesg | tail
[drm] Initialized i915 1.5.0 20060119 on minor 0
ADDRCONF(NETDEV_UP): eth1: link is not ready
EXT3-fs error (device hde1): ext3_check_descriptors: Block Bitmap for group 896 not in group (block 130055736)!
EXT3-fs: group descriptors corrupted!
EXT3-fs error (device hde1): ext3_check_descriptors: Block Bitmap for group 896 not in group (block 130055736)!
EXT3-fs: group descriptors corrupted!
EXT3-fs error (device hde1): ext3_check_descriptors: Block Bitmap for group 896 not in group (block 130055736)!
EXT3-fs: group descriptors corrupted!
EXT3-fs error (device hde1): ext3_check_descriptors: Block Bitmap for group 896 not in group (block 130055736)!
EXT3-fs: group descriptors corrupted!
~$ sudo tune2fs -l /dev/hde1
tune2fs 1.40-WIP (14-Nov-2006)
Filesystem Volume Name: None
Filesystem UUID: 9b0cdb0b-b18c-45b9-8ba8-10a0c6acf4a5
Filesystem magic number: 0xEF53
Filesystem Revision #: 1 (dynamic_
Filesystem features: has_journal filetype sparse_super large_file
Default mount options: (none)
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode Count: 109903872
Block Count: 219787200
Reserved block count: 10989360
Free blocks: 45568125
Free inodes: 31402013
First block: 0
Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Last mount time: Sat May 21 08:07:54 2005
Last write time: Tue Jun 19 09:22:58 2007
Max count: 14
Maximum mount count: -1
Last checked: Thu May 19 08:05:28 2005
Check interval: 0 (<none>)
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
~$ sudo tune2fs -l /dev/hdf1 ## (or g1, or h1)
tune2fs 1.40-WIP (14-Nov-2006)
tune2fs: Bad magic number in super-block while trying to open /dev/hdf1
Couldn't find valid filesystem superblock.
...so that's what I know about my array.
QUESTIONS:
1) How much trouble will I get into by clearing the invalid ext3 on hde1? What should I do next if I do that?
2) What will e2fsck -b 8193 <hdf1> or g or h do to my array? Is this a very, very bad idea?
3) What other commands can I run on this sucker to find out more about what's going on?
4) What do I need to do so I can mount the array, suck my data CLEAN OFF of those drives and onto something else, and rebuild it as something else (currently thinking Serverelements Naslite 2 USB, but open to suggestions)?
Any help anybody can offer would be deeply, deeply appreciated. I'm in way over my head; unfortunately I've discovered I"m in way over my friends' heads, too (and a good half dozen of them are sysadmins). I just want my data back. Once I have it I'll go run Ubuntu like a good little boy; please keep me from hurting myself or my data!
Oh, and take your time. I've got to be out out town for a week, so I'm not doing anything to it until next Wednesday. I would *love* to be able to put the silly thing back together again once I get back and get on with my life...
Deepest, sincerest thanks,
Seth
Re: Please help me reassemble my RAID5! (long)
I'm probably not the right guy for this job since I don't have a software RAID array at any level, but after a quick google search for "linux software raid repair" it appears that the answer to question #2 in this link might be of some use:
http://www.linux.com/howtos/Software...-HOWTO-4.shtml
Hopefully the ckraid utility can help you out.
In the interests of stepping lightly:
So I tried this:
$ sudo modprobe md
...and it just sorta did it. If I understand the man pages correctly, I'm basically telling the kernel "Prepare for a Software RAID." Yes?
$ sudo mdadm --assemble --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdg1 /dev/hdh1
Did this, and it says:
mdadm: device 3 in /dev/md0 has wrong state in superbock, but /dev/hdh1 seems ok
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
...and, in the interests of not getting in deeper than I can handle, I calmly, humbly and appreciatively await further advice.
Thanks, greenfly!
Re: In the interests of stepping lightly:
Quote:
Originally Posted by setht
So I tried this:
$ sudo modprobe md
...and it just sorta did it. If I understand the man pages correctly, I'm basically telling the kernel "Prepare for a Software RAID." Yes?
$ sudo mdadm --assemble --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdg1 /dev/hdh1
Did this, and it says:
mdadm: device 3 in /dev/md0 has wrong state in superbock, but /dev/hdh1 seems ok
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
...and, in the interests of not getting in deeper than I can handle, I calmly, humbly and appreciatively await further advice.
Thanks, greenfly!
If I'm reading the error correctly, it seems to be complaining about /dev/hdg1. Since a RAID5 can handle losing a single drive, you can actually rebuild this array in degraded mode. Just replace the partition name with "missing" so if I am correct and /dev/hdg1 is in fact the one with problems, you would type:
Code:
$ sudo mdadm --assemble --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdh1 missing
Then see if the device comes up. If so you should be able to mount it. If not come back here with the error.
Once more into the breach, dear friends...
Quote:
$ sudo mdadm --assemble --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdg1 /dev/hdh1
Did this, and it says:
mdadm: device 3 in /dev/md0 has wrong state in superbock, but /dev/hdh1 seems ok
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
$ sudo mdadm --assemble --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdh1 missing
mdadm: device /dev/md0 already active - cannot assemble it
$ sudo mkdir /mnt/md0
$sudo mount /dev/md0 /mnt/md0
mount: you must specify the filesystem type
$ sudo mount /dev/md0 /mnt/md0 -a
mount: you must specify the filesystem type
$ sudo mount /dev/md0 /mnt/md0 -a -t jfs
mount: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or other error (could this be the IDE device where you in fact use ide-scsi so that sr0 or sda or so is needed?) In some cases useful info is found in syslog - try dmesg | tail or so
$ dmesg | tail
--- rd:4 wd:3
disk 1, o:1, dev:hdf1
disk 2, o:1, dev:hdg1
disk 3, o:1, dev:hdh1
raid5: failed to run raid set md0
md: pers->run() failed...
EFS: 1.0a - http://aeschi.ch.eu.org/efs
EFS: cannot read volume header
EFS: cannot read volume header
JFS: nTxBlock = 7945, nTxLock - 63562
...so then I tried restarting it and doing the "missing" without doing the other stuff first; no dice. Then I tried assembling the array calling other disks "missing" and it still doesn't like "device 3" regardless of the order I type it in.
When I tried to build it from F, G, H and E (in that order) it said "device 3 has wrong state but /hdh1 seems ok" as usual; next:
$ dmesg | tail
raid5: device hdh1 operational as raid disk 3
raid5: device hdg1 operational as raid disk 2
raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
--- rd:4 wd:3
disk 1, o:1, dev:hdf1
disk 2, o:1, dev:hdg1
disk 3, o:1, dev:hdh1
raid5: failed to run raid set md0
md: pers->run() failed...
just for giggles, I tried mounting it as jfs, hfs, and ntfs. At least I got a different error out of ntfs ("the device /dev/md0 doesn't have a valid NTFS. Maybe you selected the wrong device? Or the whole disk instead of a partition? Or the other way around?")
And that's about as far as I can get. Unfortunately I suddenly got a job down in LA so I'm getting on a plane in ten hours. I'll have this lovely little disaster to play with again in about a week... and yet again, I welcome any and all advice.
Re: Once more into the breach, dear friends...
Quote:
...so then I tried restarting it and doing the "missing" without doing the other stuff first; no dice. Then I tried assembling the array calling other disks "missing" and it still doesn't like "device 3" regardless of the order I type it in.
When I tried to build it from F, G, H and E (in that order) it said "device 3 has wrong state but /hdh1 seems ok" as usual; next:
Okay, it's possible that the disk it is complaining about is okay, it's just that some of the superblocks in the array are out of date. Normally I try to avoid "--force" options, but in this case we've tried everything else, and there's a chance that if you tell mdadm to ignore the out-of-sync superblock it might just re-create the array. So, reboot just so that Knoppix is in a fresh state and follow my original instructions, except when you run mdadm this time, add --force to it:
Code:
$ sudo mdadm --assemble --force --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdg1 /dev/hdh1
As always, try that and come back with your results.
Back in the saddle again...
Quote:
Okay, it's possible that the disk it is complaining about is okay, it's just that some of the superblocks in the array are out of date. Normally I try to avoid "--force" options, but in this case we've tried everything else, and there's a chance that if you tell mdadm to ignore the out-of-sync superblock it might just re-create the array. So, reboot just so that Knoppix is in a fresh state and follow my original instructions, except when you run mdadm this time, add --force to it:
Code:
$ sudo mdadm --assemble --force --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdg1 /dev/hdh1
As always, try that and come back with your results.
Well, a move to another state, a new job, a new laptop, an ISP blowing up on me (anyone else hosed by the cPanel update last week? Yeah, didn't think so... avoid Sitelutions like the plague) and I can actually get to work trying to get the server working again. It's only been, what? A month and a half?
So. Feeling sassy. Roll the dice, plug it in, type up sudo modprobe md and let 'er rip:
Code:
$ sudo mdadm --assemble --force --auto=yes /dev/md0 /dev/hde1 /dev/hdf1 /dev/hdg1 /dev/hdh1
mdadm: /dev/md0 has been started with 3 drives (out of 4).
Code:
sudo mkdir /mnt /md0
Code:
sudo mount /dev/md0 /mnt/md0
AND THERE IT WAS!!!!! :shock:
So. futzing around in Konquerer, looking at the somewhat shattered file structure. I had created five mounts in there; it appears the NAS (rebyte) had created a few more defaults. The defaults are still there.
One of my shares is up and happy.
The rest of my shares are shoved into a USERS file.
And they're locked.
SO:
1) How do I unlock the permissions on my shattered files so that I can get them the hell off? The stuff that isn't locked up is busily transferring to a spare LaCie Porsche 80GB right now... and I can't tell you guys how overjoyed I am to have them! But the media files (photos, videos, music files) are still locked up. How do I get to 'em?
2) When I fired the computer up the first time (had to shut it down because in transport a SATA cable had managed to rub up against the CPU fan and when the fan came on, it made a horrible racket) it said something about bad blocks and how it was running with only three drives; Does what I've been able to tell you guys tell you enough to determine if one of the drives up and failed or if it just has a bad block? If it up and failed, should I limit the amount of time I have the other drives running in case they decide to fail too?
My data is so close! I feel like I'm on the ragged edge of seeing it again! I'd hate to have it snatched away irrevocably now that I can see the light at the end of the tunnel...
I owe Greenfly, Harry and this forum in general a tremendous debt. Thanks so much for you help so far - please help me take it over the finish line...
Seth