So clicking "OK" to close that missing ptplay.library requester just does nothing? Mine immediately pops up the TBImages requester and opens up the GUI for HD-Rec.
I would run SnoopDos and see if catches what it's calling and failing on.
Other than that, it could be it's looking for a defined screen bound to the RTG card. You can try binding the VESA in Picasso96Screenmode and trying that. Make sure to test the screenmodes in Picasso96Screenmode.
Raise the stack, try reducing commodities in wbstartup, reduce patching in the startup-sequence, etc...
Don't use default settings in AHI, up the bit rate, etc..
I run a 1280x720 @ 32bit workbench screen, opening HD-Rec opened it's own 1280x720 (based on grabbing the screen with SGrab and looking at the properties of it on my PC).
While my guide is pretty complete, it's nowhere close to how my machine is setup. It's a basic install of Amithlon and getting all the devices working (if compatible). I've added numerous libraries and utilities and what not specific to my needs and or hardware.
I'm also running the x86 AHI install with the needed run_elf in my user-startup.