I use the following recording settings on Shadowplay: 480p, 30fps and 30mbps. On Handbrake I just used the "Normal" audio output: AAC audio, 160kbps and Dolby Pro Logic II.
With these, I have recorded Atari 8-Bit demos as well, and the Ataris sound fine as well.
However, I also wanted to record ZX Spectrum demos, but Shadowplay refused to work with Spectaculator, so I was forced to use the emulator's internal AVI RLE recorder. BUT, using the same Handbrake settings yielded fuzzy video (which I cannot improve) and the AY audio was horrible! In the end, I had to switch audio encoding to MP3 192kbps for much-improved sound.
I used Shadowplay to record Atari ST video (from Steem SSE) and again used the Handbrake AAC settings to convert, but again the AY sound was HORRIBLE. In fact, it seems that AAC has a problem with Yamaha AY audio.
Finally, recorded Amstrad CPC video with Shadowplay (after having poor results with its internal recorder) and just converted the audio to MP3 directly.
As of now, I have 500 videos from seven computer platforms taking up about 20Gb of memory on my phone, and the quality varies enormously. The Youtube downloads of PC demos are particularly poor in the visual side of things, but that cannot be helped. I just wanted a portable collection of Demoscene prods.