Because IDE drivers are PIO they are greatly affected by available CPU power. Having a USB host HW and it's software stack enabled does not come for free.
Probably interrupts, possibly polling for hotplug. What USB devices do you have connected?
Worth trying Sirion the original Thylacine USB stack instead of Poseidon?
|