In my tests I noticed following (A500-mode):
ASIO: not much better
WDM-KS: freezes (either it does not work in Windows 7 or it is buggy, it was commented out in portaudio sources..)
WASAPI: works PERFECT, even buffer 1 works without glitches.
Only WASAPI didn't do that weird slowdown, it looks like portaudio isn't working 100% properly