Audio Mixing for Games example + source

[roondar]

Recently I've been inspired by stuff like Octamed and the many threads on here about Paula and it's abilities. So I decided to try my hand at making a real-time audio mixer that would be fast enough for use in A500 games.

I've managed to get a (IMHO anyway) pretty good result, with low CPU use and decent quality. Obviously I had to compromise and so err, I did - the mixer runs at a fixed sample rate. However, it does manage to mix three channels into a single one at less than 6% of a PAL frame (as measured on an A500).
Edit: this perhaps wasn't clear in my original post, but the end result is six channel playback - three channels for music, three mixed ones for sound effects.

There’s also a ‘high quality’ mixer included that is much slower, but results in much better output quality. This one is probably too slow for the A500, but will work nicely on an A1200 (where it uses about 7% CPU time per frame).

Lastly, there are two executables: one optimised for the 68000 and one for the 68020 and up. Both should run on any Amiga with 512KB of memory.

Personally, I'm rather happy with the result and so I wrote an article on my website and made a YouTube video showing the result.

I've also included the full source code and wrote the mixer so that it should be useable in other projects with minimal changes. If you like it, use it as you see fit. Note that the module and samples I've used in the example are not by me and the startup code is by Photon of Scoopex. The ProTracker player used is the one by Frank Wille.

Hope you guys find it useful or neat.

The article/source can be found here:
http://powerprograms.nl/amiga/audio-mixing.html

Here's the YouTube Video:
[ Show youtube player ]

[bloodline]

My first game/demo work was writing audio code for the Amiga. I’ll have to have a look through my archives, but my playback routine would playback a protracker module on the first three audio channels, using the interrupts as normal to keep Paula fed with data. The fourth channel was my sfx channel, which would allow two samples to be played back, using a very simple “add and right shift” technique, for this channel the interrupt contounious played a relatively large buffer (~8khz IIRC), which was filled every VBL.

The sound designers (usually me), would then make sure all music mods were only 3 channels sfx were sampled at the correct frequency. When an sfx needed to be played it would be added to a simple event queue, that would then update the sfx playback buffer every VBL, the oldest sample would be removed if a newer one needed to played, as only 2 sfx were allowed.
-edit- if I did it now, I would probably remove the quietest sample, but almost all sfx got quieter the longer they played, so the effect was the same.

It worked really well on a mono TV, but sounded horrible if you listened to it in stereo.

It would be a fun experiment to have done a proper stereo version of this.

[roondar]

Interesting, that is similar to what I'm doing here - playing back 3 channel ProTracker and mixing SFX to the 4th channel using audio interrupts.

However, my version doesn't shift in real time - it divides* or limits ahead of time (well, for the standard mixer that is - the HQ one does real time limiting) so it's might be a bit faster. The example as is runs at a fixed rate of 11KHz (though this is configurable) and I feel it sounds well, good enough.

Certainly a lot better than I expected

*) Using signed divide. I do this because then a three channel mix only needs to be divided by three rather than four as is the case with shifts.

[ross]

Hi roondar, when I find some time (I am totally overwhelmed by real work these days) I will take a look at your article.
But I'm sure it's written in a simple and clear way as usual
I have several mixing routines with different features in my sources mess (I also remember that in the 80's I wrote a fast routine with linear interpolation in real time of which I was proud ).

So I can compare with the job you did here.

Cheers!

[roondar]

Originally Posted by ross

Hi roondar, when I find some time (I am totally overwhelmed by real work these days) I will take a look at your article.
But I'm sure it's written in a simple and clear way as usual
I have several mixing routines with different features in my sources mess (I also remember that in the 80's I wrote a fast routine with linear interpolation in real time of which I was proud ).

So I can compare with the job you did here.

Cheers!

Thanks!

Comparing notes is always useful.

Do note that the one overarching goal here was to minimise overhead, so the mixing routines themselves limit the amount of tricks they use (though the High Quality mixer does try to keep samples as close to their 8-bit original values as much as possible even when mixed)

To be honest I'm kind of hoping to help people 'see the light' here, mixing some extra channels at a reasonable quality isn't as hard or as processor intensive as it may seem (though you do have to accept a few compromises) and it can really change the way games sound.

More so as I've recently noted (been playing some Mega Drive stuff ) myself that more channels but lower quality SFX generally seems to sound more impressive than fewer channels but higher quality SFX

[bloodline]

Originally Posted by roondar

Interesting, that is similar to what I'm doing here - playing back 3 channel ProTracker and mixing SFX to the 4th channel using audio interrupts.

However, my version doesn't shift in real time - it divides* or limits ahead of time (well, for the standard mixer that is - the HQ one does real time limiting) so it's might be a bit faster. The example as is runs at a fixed rate of 11KHz (though this is configurable) and I feel it sounds well, good enough.

Certainly a lot better than I expected

*) Using signed divide. I do this because then a three channel mix only needs to be divided by three rather than four as is the case with shifts.

11KHz will give fine sound effects, I had originally planned to support 4 sfx samples on the fourth channel, but testing showed that it was too quiet to be particularly good... I decided to do the volume shift in real time so that I could be more flexible in both volume and number of samples.

So if I needed two samples (for gunfire, or foot steps, or object interaction), then I could use two channel mode, and then switch it off for a loud sample (like a bomb going off, which might just be the same gun shot sample played loud to save on chip RAM ), that was actually quite effective.

It was all about saving RAM and making it as flexible as possible

[zero]

Have you tried temporal mixing? As in if you have three samples you just do one sample from each after the other, like ABCABCABC?

As well as avoiding any arithmetic you might be able to use the blitter to do the interleaving.

[bloodline]

Originally Posted by zero

Have you tried temporal mixing? As in if you have three samples you just do one sample from each after the other, like ABCABCABC?

As well as avoiding any arithmetic you might be able to use the blitter to do the interleaving.

I don’t think the Amiga would be able to get the sample rate high enough to make that work effectively.

[roondar]

Originally Posted by bloodline

11KHz will give fine sound effects, I had originally planned to support 4 sfx samples on the fourth channel, but testing showed that it was too quiet to be particularly good... I decided to do the volume shift in real time so that I could be more flexible in both volume and number of samples.

So if I needed two samples (for gunfire, or foot steps, or object interaction), then I could use two channel mode, and then switch it off for a loud sample (like a bomb going off, which might just be the same gun shot sample played loud to save on chip RAM ), that was actually quite effective.

It was all about saving RAM and making it as flexible as possible

Yeah, volume can be an issue if you simply shift/divide each sample by the number of extra channels. Which is why I've also added in a compressor/limiter as an alternative sample pre-processor and that is a lot less quiet (though obviously distorts the samples somewhat). The compressor works surprisingly well for how simple it is.

The idea of masking quiet sounds with loud ones does sound interesting.

My current mixer simply has an AddSampleToMix routine that just adds in an extra sample to the list of samples to play at the current time (unless no free slot exists, in which case it does nothing). I considered adding priorities, but these examples of mine are mostly about showing the effect and so I try to not make them too complicated.

Originally Posted by zero

Have you tried temporal mixing? As in if you have three samples you just do one sample from each after the other, like ABCABCABC?

As well as avoiding any arithmetic you might be able to use the blitter to do the interleaving.

I did consider it. It would be slightly faster (though not by as much as you might think - doing this means you need to write 2x/3x/4x as much data into the sample buffer and that more or less compensates for not having to do any additions). Also, it does require 2x the playback frequency for two channels, 3x the playback frequency for 3 channels, etc.

This would mean even a fairly low frequency sample would need a very high sample rate. Plus, as I understand it, such mixing leads to a high-pitched whine being added to the sound (which can be fixed, but as I understand it only by increasing the sample rate even further).

So I ultimately decided against it.

[bloodline]

Originally Posted by roondar

The idea of masking quiet sounds with loud ones does sound interesting.

Two rules exist with sound; 1. Loud sounds totally dominate, don't bother with quiet parts, when there is a loud sound.

My current mixer simply has an AddSampleToMix routine that just adds in an extra sample to the list of samples to play at the current time (unless no free slot exists, in which case it does nothing). I considered adding priorities, but these examples of mine are mostly about showing the effect and so I try to not make them too complicated.

2. the attack of a sounds playing now is more important than the decay of a sound that has just played, especially for percussive sounds which most game sfx are. This is why 4 track mods sound awesome, most instruments only really need to have their attack phase heard to sound correct, as long as the decay is covered by other sounds... A sustain obviously needs to sustain.

I did consider it. It would be slightly faster (though not by as much as you might think - doing this means you need to write 2x/3x/4x as much data into the sample buffer and that more or less compensates for not having to do any additions). Also, it does require 2x the playback frequency for two channels, 3x the playback frequency for 3 channels, etc.

This would mean even a fairly low frequency sample would need a very high sample rate. Plus, as I understand it, such mixing leads to a high-pitched whine being added to the sound (which can be fixed, but as I understand it only by increasing the sample rate even further).

So I ultimately decided against it.

Zero's idea is in essence Pulse Density Modulation (sort of...), which I use on microcontrollers which don't have audio DACs, in order for it to work you need a reconstruction filter... Now if you can get the sample rate high enough, then the Amiga's audio filters (and the TV speaker itself) will do that job... Actually I think Paula actually generates its audio using a 3Mhz PDM signal and a couple of LPF filters.

[phx]

That's a quite interesting piece of software! Thanks a lot for sharing it. I would have never considered audio mixing in games, but 6% might make me change my mind. It's still a lot of time, but all of my recent games could easily afford that.

You said that mixing the sound effects results in a lower volume, so you reduced the volume of the music channels. As sound effects are always much more important than the background music an alternative solution could be to mix the music instead, and play the sound effects on the hardware channels. Although I understand that this is much more difficult for you, because then the mixer must be integrated into the player routine.

BTW, I just saw a comment in your Makefile that you have to do "make clean" whenever an include file (*.i) changes. There is a vasm-option to automatically emit all dependencies in Makefile-format: -depend=make (no code will be generated).

[roondar]

Originally Posted by phx

That's a quite interesting piece of software! Thanks a lot for sharing it. I would have never considered audio mixing in games, but 6% might make me change my mind. It's still a lot of time, but all of my recent games could easily afford that.

Indeed, it isn't free. But I was certainly pleasantly surprised by the numbers - When I started I had predicted something closer to 10%.

On the other hand, the 6% I mention is all CPU cycles. But as we know, the 68000 only uses about half the cycles of the system, so roughly half of these cycles can be used by other DMA such as the Blitter or bitplane DMA. I've not calculated how much of a difference that makes, but maybe the actual overhead can be slightly lower if (on average) some interleaving between DMA and CPU occurs.

You said that mixing the sound effects results in a lower volume, so you reduced the volume of the music channels. As sound effects are always much more important than the background music an alternative solution could be to mix the music instead, and play the sound effects on the hardware channels. Although I understand that this is much more difficult for you, because then the mixer must be integrated into the player routine.

To be honest I didn't even think about it like that. Obviously you could also mix music instead. I still wouldn't have done it in my example though - I try to keep the effects I do as easy to understand as I can. But it's certainly possible.

The volume issue is improved by using the compressor/limiter option for pre-processing, but that does impact audio quality more than the divide/shift approach.

One thing about using it for music would be having to deal with many different pitches (=period values). The main reason it's only 6% is that I don't do any resampling - the program just assumes all samples are (roughly) 11025Hz and never changes period values.

BTW, I just saw a comment in your Makefile that you have to do "make clean" whenever an include file (*.i) changes. There is a vasm-option to automatically emit all dependencies in Makefile-format: -depend=make (no code will be generated).

That is quite useful, I'll be sure to try that out!

To be honest, the makefile is the thing I'm least proud of. I've used a rather lazy approach to keep it as short as possible. It mostly works, but indeed doesn't care much about dependencies.

[LeCaravage]

Holy shit ! Roondar your work is stunning. 3 samples playing at the same time than music is a dream. It adds so much atmosphere to a game.
Do the sfx have to be played in the main program or does it need to be played under irq ?
BTW, your article about fast bobs triggered an idea to make my bobs routine much much faster (or to be more precise : less slower).

[roondar]

Originally Posted by LeCaravage

Holy shit ! Roondar your work is stunning. 3 samples playing at the same time than music is a dream. It adds so much atmosphere to a game.
Do the sfx have to be played in the main program or does it need to be played under irq ?
BTW, your article about fast bobs triggered an idea to make my bobs routine much much faster (or to be more precise : less slower).

Thanks for the compliment! I'm happy my work has helped you, helping the Amiga community is kind of why I do these things

As for your question:
The code as I wrote it uses the Amiga's audio interrupt (so it runs from an IRQ). However, you could, in theory, run the mixing routines without interrupts. That said, it's much easier to use audio interrupts.

This is because Paula's period values (which are what govern the sample playback rate) only really support sample rates that do not neatly divide up into 50 or 60. This means that the buffer to play back & fill would vary in size from frame to frame when not using interrupts*. Using an audio interrupt means you don't have to consider this, which makes the code easier to follow and a bit faster to boot.

Note that the code as I provide it can be used as is and has a routine that you can just call from inside of your main loop to add a sample to be played back. Note you might want to remove any pre-processor and mixing mode you don't use to save space.

*) you could also wait the exact amount of time needed, but that would desynchronise your main loop from the display, which is probably not what you had in mind.

[phx]

Originally Posted by roondar

On the other hand, the 6% I mention is all CPU cycles. But as we know, the 68000 only uses about half the cycles of the system, so roughly half of these cycles can be used by other DMA such as the Blitter or bitplane DMA.

You're right. Although I'm not very good in using these cycles. In the best case I have started a 16x16 tile-blit before calling the mixer.

One thing about using it for music would be having to deal with many different pitches (=period values). The main reason it's only 6% is that I don't do any resampling - the program just assumes all samples are (roughly) 11025Hz and never changes period values.

Ok. Forgot about that. So music would be no option.

BTW, I only had a quick glance over the source, so I assume that you always allocate one fixed channel for mixing, also when no effects are played? Playing the mixing-buffer as a sound-effect only when there is something in it would be a great enhancement!

[roondar]

Originally Posted by phx

BTW, I only had a quick glance over the source, so I assume that you always allocate one fixed channel for mixing, also when no effects are played? Playing the mixing-buffer as a sound-effect only when there is something in it would be a great enhancement!

Correct, the channel is permanently allocated and the mixer always plays something on the channel. Either silence directly through Paula so no actual mixing is done in that case (which means that for silence the overhead drops to handling the audio interrupt and nothing else), or one or more mixed samples.

Your idea would be a interesting improvement to consider. You could even combine that with your ProTracker with sound effects system - play 4 channels of music and interrupt that as needed for a sound channel from the mixer. There would be a mechanism needed to make sure that the sound effects don't skip from channel to channel mid-mix though, but I'm sure that could be made.

So as always, there's some room for improvement here

The primary reason for it being like this is to keep the example code as simple as possible (well that and the less choices the audio interrupt handler has to make, the smaller the overhead). The second reason is more of a personal preference, not allowing music on the mixer channel means it won't ever cut out music playback and that has been a standard critique of Amiga audio for years so I decided to simply play back 3 channels for music instead.

That said, your ideas are certainly interesting and could potentially make this idea work even better

[ross]

Originally Posted by roondar

This is because Paula's period values (which are what govern the sample playback rate) only really support sample rates that do not neatly divide up into 50 or 60. This means that the buffer to play back & fill would vary in size from frame to frame when not using interrupts*.

Hi roondar, a tech solution for this problem.
You must always remember that in Amiga there are no values like exactly 50 (or exactly 60) but everything is a multiple of the C2 clock (3546895Hz on PAL Amiga).
So in frame you have 227 C2/line * 313/line (on default long frame) for a grand total of 71051 'slot'.
Yes, ugly values (are even prime numbers..) but anyway both value are possible audio periods to 'sync'/subdivide with 71051.
Ok, ok, Paula accept only values in word granularity but not a problem at all.
PTR and LEN are buffered so in one frame you can setup (x+1)/2 in LEN, then (x-1)/2 and alternate.
Every source that sync with video (VBI, Copper, even a polling to a fixed line) can be used.
What values for x? Well, the reciprocals
If you setup PER = 227 (15625.09 Hz) then LEN = (313+1)/2 = 157 and (313-1)/2 = 156.
If you setup PER = 313 (11331.93Hz) then LEN = (227+1)/2 = 114 and (313-1)/2 = 113.
It is therefore necessary to define the double buffer, in which to insert the mixed samples, of the right dimensions (2*LEN).
Among other things, the value PER = 313 gives you a frequency that is practically indistinguishable from the 11025 you chose.
So this can be a viable alternative: insert the mixing routine in VBI, together with the module player.

Cheers!

[roondar]

Originally Posted by ross

Hi roondar, a tech solution for this problem.
You must always remember that in Amiga there are no values like exactly 50 (or exactly 60) but everything is a multiple of the C2 clock (3546895Hz on PAL Amiga).
So in frame you have 227 C2/line * 313/line (on default long frame) for a grand total of 71051 'slot'.
Yes, ugly values (are even prime numbers..) but anyway both value are possible audio periods to 'sync'/subdivide with 71051.
Ok, ok, Paula accept only values in word granularity but not a problem at all.
PTR and LEN are buffered so in one frame you can setup (x+1)/2 in LEN, then (x-1)/2 and alternate.
Every source that sync with video (VBI, Copper, even a polling to a fixed line) can be used.
What values for x? Well, the reciprocals
If you setup PER = 227 (15625.09 Hz) then LEN = (313+1)/2 = 157 and (313-1)/2 = 156.
If you setup PER = 313 (11331.93Hz) then LEN = (227+1)/2 = 114 and (313-1)/2 = 113.
It is therefore necessary to define the double buffer, in which to insert the mixed samples, of the right dimensions (2*LEN).
Among other things, the value PER = 313 gives you a frequency that is practically indistinguishable from the 11025 you chose.
So this can be a viable alternative: insert the mixing routine in VBI, together with the module player.

Cheers!

I currently use a period value of 320 and process 222 bytes of samples per 'channel' per audio interrupt. That is a period between audio interrupts of almost* one frame (hence the very slow progression of the raster bars across the frame in my example program).

Your method would obviously also work and thus is fully valid, but to be honest I personally don't like the idea of switching between buffers of multiple sizes that much. Not because it doesn't work, but because it means a little bit of extra code to select between buffer sizes

The Amiga (IMHO anyway) offers a much nicer solution to this - the Audio interrupts allow you to pick any buffer size and simply fill it as needed. No change in size ever needed. On the other hand, keeping it in the VBI does mean the VBI never gets delayed and my code does sometimes delay the VBI (because the audio interrupt has a higher priority). Personally, I still like this more than the alternative, but I do understand that's merely my opinion and yours might differ.

So, if you really do not want to use audio interrupts, your solution is definitely an option

*) the audio interrupt effectively runs slightly slower than 1/50th of a second.

[Kalms]

Originally Posted by ross

Hi roondar, a tech solution for this problem.
You must always remember that in Amiga there are no values like exactly 50 (or exactly 60) but everything is a multiple of the C2 clock (3546895Hz on PAL Amiga).

<Snip maths>

So this can be a viable alternative: insert the mixing routine in VBI, together with the module player.

Cheers!

If you know the display mode then you can do that. Some time ago I wrote an implementation that counts bus cycles and then triggers mixing as needed from the VBI, and mixes incrementally into a single ring buffer: https://github.com/Kalmalyzer/adpcm-.../PaulaOutput.s

(It assumes 227 buscycles per line, 312 lines during a short frame, 313 lines during a long frame. It counts samples using 32.32 fixed point, and looks at the LOF flag when determining how many bus cycles will pass during the upcoming frame.)

Perhaps a tad overcomplicated though Switching between two buffers is probably more straightforward.


Another approach that I have used in the past is to do mixing in the VBI, always mixing a few more samples than necessary, and use the end-of-audio interrupt to inform the VBI of when the audio hardware has reached the end of the buffer - and include throttling in the VBI, so it skips mixing if the end-of-audio interrupt is too delayed compared to the mixing.

[ross]

Originally Posted by roondar

I currently use a period value of 320 and process 222 bytes of samples per 'channel' per audio interrupt. That is a period between audio interrupts of almost* one frame (hence the very slow progression of the raster bars across the frame in my example program).

With the 'alternative method' you have 226 (/228) as a buffer (a little bigger because of the greater frequency).

Your method would obviously also work and thus is fully valid, but to be honest I personally don't like the idea of switching between buffers of multiple sizes that much. Not because it doesn't work, but because it means a little bit of extra code to select between buffer sizes

Lazy!

The Amiga (IMHO anyway) offers a much nicer solution to this - the Audio interrupts allow you to pick any buffer size and simply fill it as needed. No change in size ever needed. On the other hand, keeping it in the VBI does mean the VBI never gets delayed and my code does sometimes delay the VBI (because the audio interrupt has a higher priority). Personally, I still like this more than the alternative, but I do understand that's merely my opinion and yours might differ.

Well, you force an open door
I've a full working mixer for a 4channels->1channel module player that use Audio IRQs.
When I find some time I also tell you how the mixing is done.

Originally Posted by Kalms

If you know the display mode then you can do that. Some time ago I wrote an implementation that counts bus cycles and then triggers mixing as needed from the VBI, and mixes incrementally into a single ring buffer: https://github.com/Kalmalyzer/adpcm-.../PaulaOutput.s

(It assumes 227 buscycles per line, 312 lines during a short frame, 313 lines during a long frame. It counts samples using 32.32 fixed point, and looks at the LOF flag when determining how many bus cycles will pass during the upcoming frame.)

Perhaps a tad overcomplicated though Switching between two buffers is probably more straightforward.


Another approach that I have used in the past is to do mixing in the VBI, always mixing a few more samples than necessary, and use the end-of-audio interrupt to inform the VBI of when the audio hardware has reached the end of the buffer - and include throttling in the VBI, so it skips mixing if the end-of-audio interrupt is too delayed compared to the mixing.

Hi Kalms, I've to absolutely check your code deposit