Dont let an 040 stop you from enjoying rtg. Get a mac emu installed and try out some of their games. as for doom, You probaly just need to edit its configuration. 640x480 is gonna be crazy slow.
Too much stuff to list really. Alot of the later amiga games wil support rtg and quite a few of the more recent bigger scene demos.
edit: Heres a hol search for rtg, hope it helps.