The current recording capabilities of WinUAE are in early development stage and very buggy atm (looses settings, limited codec selection, audio offset). If you ask me better take Camtasia with TSCC or CRAM Codec on 320x200 window at 640x480 desktop resolution and pointresize*2 afterwards with avisynth/VirtualDub to TSCC. Downside is you will never have exact vsync with desktop/window capturing but it should be ok at corresponding refresh rates (50*X PAL, 60*X NTSC).
Try to drop that mpeg4 codecs. They rely on YV12 colorspace and are unsuitable for pixelexact RGB reproduction. Best codec so far seems TSCC that is as well in ffmpeg library (and thus in ffdshow [decoding only i think] that is available for many os'es) and on Techsmith's homepage for free. Capable of encoding/decoding.
CRAM/MSVC mentioned above isn't bad for RGB as well but ends up in highly uncompressed files (up to 50:1 compression ratio with WinRAR so you will waste a lot of space if you don't store the videos in compressed archives) and if i'm not wrong only captures 8bit color depth that could lead to color distortions.
Last videos i made some weeks ago look like this (cram/msvc - not yet compressed with TSCC):
(watch the balls - they have slight vsync issues i mentioned above)
Another solution to capture could be 'Fraps' but i had several conversations with the devs about limited AGP bandwidth and possible interlaced capturing when it comes to 50fps. In their opinion its not possible to capture 640x480x50 via AGP bus and i don't have a PCIe board so i cannot test it...