When using Linux's "MD" RAID, especially in RAID-1 (mirrored) mode, it is sometimes helpful to check that the disks in the mirror are consistent with each other (eg, they haven't missed any updates, or both are accessible).

It is possible to do a read-only check that the disks in the mirror are consistent with:

echo check > /sys/block/md${NN}/md/sync_action

(where "${NN}" is the number of the raid array, eg, as shown in /proc/mdstat.)

Somewhat ironically, this command will fail if the array is in a read-only state, with an error like:

linux:~# echo check >/sys/block/md15/md/sync_action
-su: echo: write error: Device or resource busy
linux:~#  

This includes "read-auto" -- which is an automatic read-only state when nothing on the array has been mounted read/write and written to (it's not sufficient to start, eg, LVM volume groups on the RAID set). You can see the array state with:

cat /sys/block/md${NN}/md/array_state

or by looking in /proc/mdstat. If you find that it is read only and you want to be able to do a check, then you need to mark the array as writable, which is mostly easily done with:

mdadm --readwrite /dev/md${NN}

and then you should see "clean" as the array state, and you can start the check again. (The array of course will no longer be in "auto-read" state until it is removed and re-added, because as far as MD is concerned it's now been moved to read/write usage.)

Progress of the check can be monitored by, eg, watching /proc/mdstat, which will show a progress bar, time estimate, and check speed for the relevant array. Issues found during the check will be sent to the kernel log.

On Debian systems (and perhaps others) there is a monthly cron job (/etc/cron.d/mdadm) which will (if you agree during package installation) arrange to run a "check" on all the RAID arrays on the first Sunday of the month. It does this by running /usr/share/mdadm/checkarray --all. That used to have the same problem with read-only arrays, but switched to automatically skipping past them (presumably on the basis that they shouldn't be out of sync). Should you decide you want to manually check them occassionally the process above could be used.

Possibly also of note, it's apparently also possible to pause the check with:

echo idle > /sys/block/md${NN}/md/sync_action

(although it's unclear if it will start checking from the beginning if you resume, or remember where it got to and continue on.)

ETA, 2012-05-27: Also worth noting, since it happened during the same hardware test, repeated lines of:

set_rtc_mmss: can't update from 56 to 9
set_rtc_mmss: can't update from 57 to 9
set_rtc_mmss: can't update from 57 to 10

(where the first and second numbers could be anything 0-59), indicate that the hardware clock is well out of sync with the software clock (eg, as set by NTP). The hardware clock sync can only make small adjustments (eg, minutes and seconds), and not large adjustments (days/hours). You can check with:

date
sudo hwclock

and if they're dramatically different then you have this problem. The best work around is to ensure that the software clock in Linux is accurate (eg, ntpdate), and then sync this into the hardware clock. (At least in my case this was caused by, I think, a dying battery in the hardware clock which was sufficient to retain time, but not sufficient to advance time when the machine was off for hardware changes; the same hardware changes that prompted the desire to check the raid arrays, in fact.)