look for mame.ini. i have mine in root mame folder, not "ini" folder.
here you can read:
# DIRECT3D POST-PROCESSING OPTIONS
for specific game hlsl setting look into the HLSL folder (hlslpath). there you have a config file for each game - once you enabled it for the specific game.