Inexplicably slow hard disks? Try disabling PIO mode
A baffling problem I came across recently was a server where copying between internal disks ran painfully slowly, getting around 3.3MB/s between internal SATA drives. Of the machine's three disks, two were failing with huge numbers of ATA errors and bad sectors, so I pulled those out an ran test copies to and from the remaining system disk, with no improvement. Incidentally, to find out that the drives were failing I used a brilliant little tool called HDD Guardian which reads SMART monitoring values the same way smartctl does, just with a nice graphical interface.
It turns out the server had defaulted to IDE mode on the disk controller as Server 2003 doesn't really contain any SATA AHCI drivers and you'd need a floppy disk to install them. In of itself this wouldn't cause an issue, but when Windows detects multiple CRC errors on a disk transfer, it will reduce the transfer rate. Eventually the disk will become stuck in Programmed Input/Output or PIO Mode, which essentially means all data transfers must pass through the narrow CPU IO bus rather than going via system memory which chews up a lot of CPU time. There's a lot more info on this topic at http://wiki.osdev.org/ATA_PIO_Mode.
To fix this, there are some instructions in KB817472 in the Microsoft Knowledgebase and some other instructions in this TechLogon article, neither of which fixed the problem. Since the server in question is a VM host and all the VMs were copied onto other systems anyway when the drives failed, I just reinstalled it with Server 2008 and the disks set in AHCI mode, which is cheating, but it made the problem go away. Now my transfer speeds are back to normal, so I just have to replace the two 500GB drives with 4000 reallocated sectors. Drive magnets anyone?