I was getting I/O errors on one of the physical volumes in an LVM. I tried to run pvmove to migrate the data off the bad drive, and that failed and locked the system hard, leaving the system in an unbootable state. The system in question is purely a backup system with mostly redundant data, so I was less concerned about trying to run the pvmove off the live system.
CAUTION: You should always backup your system before attempting a pvmove operation.
dm-mapper: unrecognized sync argument to mirror log 2 dm-mirror: Error creating mirror dirty log dm-mapper: error adding target to table dm-mapper: device 254:9 too small for target
Since the system was not bootable, I booted the system off the Ubuntu Dapper 6.06 Desktop CD. To be on the safe side, let’s do the pvmove from the Live CD with the logical volumes inactive. I can afford to have this backup system offline for some time, so that should be fine.
CAUTION: pvmove is slow: For a production system, you should probably schedule this maintenance for off hours if possible.
sudo sh modprobe dm-mirror pvmove --abort vgchange -an pvmove /dev/hdb1 /dev/sdb1 vgreduce VolGroup00 /dev/hdb1 pvremove /dev/hdb1
After an hour or so, I could reboot and bask in the system happy goodness. Then I could also run some diagnostics on the failed disk with the system up.
CAUTION: Be sure to run filesystem consistency checks (fsck) after this operation!