I'm starting to do more server admin stuff lately, and we had what appeared to be a bad disk in a RAID 1 array, which houses the OS (Server 2003) for our web server. After looking at the logs it turns out it was some sort of timing issue which caused the disk to go offline. I just rebuilt the array on that disk and all was good.

After doing some research it sounds like doing regular consistency checks can prevent these timing issues and other problems from developing with RAID arrays.

1) Do you do regular consistency checks on your RAID arrays?
2) How often do you do them with RAID 1 arrays, and RAID 5 arrays? It sounds like you're supposed to do them more often with RAID 5 arrays...
3) How do you manage the checks? Scheduled tasks? What software do you use to run the checks?

3/7/2008 9:53:08 AM

are you using a RAID card, or having Windows do the RAID?

I haven't heard of losing a disk due to timing issues. With Adaptec cards I've worked with, you can enable "Background Consistency Check" which will constantly check consistency at a performance loss somewhere between 1% to 10%.

3/7/2008 12:38:52 PM

I don't, the hosting company we pay hundreds of dollars a month should though. I said this in another thread earlier, but we had a raid controller fail and write data to the wrong place which of course corrupted our database. They were also supposed to be doing backups, but did not have one available for us and then called me and asked "do you guys have a backup?".

Not to put all the blame on them - I should have had a more recent backup than what I had - but they were supposed to be doing it daily.

[/rant].

3/7/2008 12:43:49 PM

Quote :

"are you using a RAID card, or having Windows do the RAID?"

They're all Dell Poweredge 2850 and 2950 servers with PERC RAID controller cards

Quote :

"I haven't heard of losing a disk due to timing issues. "

It didn't die, just went offline.

Quote :

"With Adaptec cards I've worked with, you can enable "Background Consistency Check" which will constantly check consistency at a performance loss somewhere between 1% to 10%."

that would be cool

Quote :

"They were also supposed to be doing backups, but did not have one available for us and then called me and asked "do you guys have a backup?"."

ooh that sucks. were you paying extra for certain types of backups, or just whatever was included with the hosting?

3/7/2008 12:57:38 PM

I just read somewhere that someone recommends running consistancy checks every 8 weeks on RAID 5 arrays. That sound reasonable?

3/7/2008 3:26:18 PM

bttt

3/8/2008 1:44:47 PM

i do it every week on my raid5 arrays at home for my media server. i do it at work sunday at 2am too.

3/8/2008 6:50:51 PM

After having a 1.2TB RAID-5 fail to rebuild due to bad sectors and spending several days coercing it back online by dd_rescue-ing the affected sectors, I did a full volume read of a few of my other volumes and found that they would have failed to rebuild due to the same reason, so all of my RAIDs are now parity checked daily during off-hours -- daily is probably overkill but I've yet to run into an issue.

[Edited on March 8, 2008 at 7:08 PM. Reason : .]

3/8/2008 7:03:55 PM

what software do you guys use to manage the consistency checks? I have open manage installed on my servers but it doesn't have the option to schedule the checks, just to run them once. i need to figure out a way to script it or run some sort of utility that will do the scheduling...

3/8/2008 7:09:49 PM

totally inapplicable to Server 2003 but I use a cron job to tell Linux to parity check it via /proc

3/8/2008 10:12:42 PM

bttt

3/18/2008 9:49:47 AM

bump per request

7/9/2008 1:04:36 PM