This is likely due do FS-UAE not running in high-dpi aware mode. That will definitely work in FS-UAE 4.0 (already implemented changes to account for that).
Are you using Windows? You can try to set UI scaling in Windows control panel to 100% and see if the shader renders correctly then.
|