Does clicking "Detect 16-bit pixel format"-button help? (WinUAE GUI display-page) Does your problem happen with every available resolution/bit depth? Do you have correct monitor "driver" installed? (wrong "driver" might cause strange refresh rate problems)
WinUAE is confirmed working on XP because my development environment is Windows XP + MSVC7