Got to love Linux software raid. It worked very well, but combined with SATA, the scsi subsystem, and
The Problem
Something happens to the SATA bus, and the drive, sda, disconnects. Hotplug takes over and removes the device.
The software raid kicks the drive out as faulty. It however, keeps a hold of the sda device.
The SATA bus resets and the the drive is readded, however, because the raid is still holding on to sda, the drive picks the next device. sdd for example.
So now the drive is in the wrong spot. A reboot should bring it back to sda, but that is hardly a fix in a production system