That's strange, both the AGA and RTG versions should work really well on your setup???
It might be some kind of hack/patch that you are running, it's very hard for me to know what the problem is.
Are you using the latest versions off AmiNet?
Check your AHI settings (run the AHI Prefs), make sure your AHI has reasonable settings.
Also see if ScummVM has produced any error logs.