IO Errors in KVM?
Last sunday we were migrating our server from Ubuntu Hardy to Lucid, because KVM is better supported and more up-to-date (we were using a backport PPA on Hardy). I had done the installation on a seperate machine and after configuring it to our wishes, I had created a tarball of it. So, the upgrade was just a matter of unpacking the tarball after booting with a livecd.
After a reboot, everything seemed to work fine, except for the fact that our two virtual machines were giving IO errors. I had seen this before when KVM had a bug with qcow images, but we were now using raw images. On the host system, we created a new raw image, copied the contents of the old one and tried again. Same problem.
What was causing this? We tried to boot the machines without virtio drivers and again, had the same problem. Eventually, Michael came up with the idea to check permissions on the disk images. Because of the upgrade, the owning group of the files was changed (well, the name was changed, not the id), meaning that my user could not write to the images. Apparently KVM doesn’t check whether the images are writeable when it is started. This cost us almost two hours of debugging…
So, remember to check you disk image permissions when you have IO Errors in KVM!