If you can keep your finger on the chip for more than 5 secs, it's not too hot.
Go run that memtest util on both chipmem and fastmem to see if there's a consistent problem in either, or if it seems random. If the ACA620 runs fine with your other chipmem expansion, it doesn't seem likely that the accelerator should be the culprit.
Edit: And you're sure that it's rev. 2 that you have?