Two guests on a RAID 5 running a member short since spring
It came from an engineering firm working into the dockyard: a five-bay chassis that had been running with only four members since about March, with two virtual servers on the datastore. Someone had added a spare to the array and begun a rebuild, which stopped on a second disk. All five members were copied, some of them needing several passes, and the RAID 5 was assembled above those images. The datastore then mounted read-only. The first VM was whole. The other had lost blocks where the failed member's stripes fell, and a delta was missing from its snapshot chain. The base disk and the two deltas that remained were merged in sequence, and the guest's file system was repaired on the copy. Both machines were booted and proved before anything was sent back.