You should use TSCC codec or similar for video and uncompressed PCM audio when recording.
Also make sure to use default window setting of 720x568 doubled linemode and untick horizontal & vertical centering.
It's only when you have finished recording what you want you should use a lossy codec for video or audio.
I recommend H.264 (X264) and AAC (Nero Digital Audio) codecs over Xvid.
I used
Avisynth and
MeGUI for all my submissions to RAG, some of my videos are still available too btw (eg. Deliverance and Deep Core).
http://www.recordedamigagames.org/backup.php
MeGUI is also a lot more user friendly now than it was back then, when you load your captured .avi in the fileindexer it automatically opens a video preview which lets you crop & resize.
No more manual editing of Avisynth scripts needed.