It's worse than you think.
For AGA graphics SetRGB4 converts the color values from 4 bit to 32 bit and then calls SetRGB32. However in KS3.0 (and 3.1, others?) the higher bits are not masked off, so some may 'leak' into the 32 bit value.
The code in the ROM looks like this:-
Code:
swap d1
lsl.l #8,d1 ; 4 bit value now in d1 bits 27-24
move.l d1,d4 ; copy to d4
lsl.l #4,d4 ; shift the copy into d4 bits 31-28
or.l d4,d1 ; combine to get 8 bit value in d1 bits 31-24
The idea is that the 4 bit color channel value gets converted to the 'closest' 8 bit value in the upper 8 bits, eg. $00000001 -> $01000000 | $10000000 = $11000000. However the higher bits are not masked off and remain in d1 and d4, so if the '4 bit' value is eg. $80000081 you get $81000000 | $18000000 = $9900000, a vastly different color!
To prevent this you must ensure that bits 31-28 and 7-5 are cleared when calling SetRGB4. So to be safe you should do eg.
Code:
moveq #15,d4
and.l d4,d1
and.l d4,d2
and.l d4,d3