More .iff sound compression than delta

[Steam Ranger]

As the title says, I want to be able to compress .iff sound in a way that can do more than the 50% reduction delta compression offers.

I'm going to implement this so it fits within the 8SVX structure, and with decoding can be played back the same as any other 8SVX sound. I'm thinking it would be best to apply some kind of compression on top of delta compression, rather than an entirely different thing.

For performance, it must be able to be decoded in real time at 20KHz sample rate on an Amiga with a 68000.

Does anyone know a suitable compression algorithm I could use to make this possible? Thanks.

[Steam Ranger]

Side note: Forgot to mention this but it might be important, it would be best if the compression works on short and long pieces of audio alike.

I want to use this primarily so long PCM audio can be fit onto a floppy at lower sample rates (2 minutes at 10KHz with room to spare for instance) as well as being able to work on high sample rate sound effects, hence the requirement for being able to decode at 20KHz.

[paraj]

I experimented with fitting as much audio onto a floppy last year, and got 5min 8seconds @ 7813Hz by pulling all stops (routine is just fast enough to handle ~30KHz).
Getting more than 50% compression ratio is probably not worth it on plain A500. At 20KHz you only have around 350 CPU cycles/sample, and if you're streaming from disk you also need to factor in the time to read and decode each track.

I used (slightly modified for speed) ADPCM (50%) with Huffman coding for each nibble (~92% so ~46% total), and an alternate MFM encoding scheme @ 164 tracks increasing disk capacity to 1.06MB.

Challenge is that audio data (even delta) has a lot of entropy. To really improve on the ratio you probably need to use some advanced coding scheme, but the realtime constraint really puts a limit on how much you can do, and I'm not aware of any that will work.

[Steam Ranger]

I suppose that's why PCM floppy music disks never took off, only trackers.

I've been curious about the idea of automatically detecting repetetive chunks of audio and only saving them once. Like how .zip files can encode variably sized blocks of data rather than single characters. If you have instrumental music, even when two instruments are playing at the same time, usually they have synchronised timing such that chunks would repeat.

Is there any kind of precedent for that as audio compression?

[Karlos]

Basic delta encoding only gives you so much, to improve on that you want linear predictive coding on top so that you typically have smaller differences to encode. However, that just adds more complexity.

[Steam Ranger]

Originally Posted by Karlos

Basic delta encoding only gives you so much, to improve on that you want linear predictive coding on top so that you typically have smaller differences to encode. However, that just adds more complexity.

Complexity doesn't matter in terms of implementation, this algorithm is going to be part of a series of libraries that I've been working on for over a year now. Complexity may matter in terms of performance, but as stated before if it runs on a 68000 in real time that's fine, although preferably a max CPU usage of 50%.

Is there a list anywhere of the compression codes for .iff? Surely someone would have implemented codes other than just delta compression. I'd rather not have it where I try and use an audio file with the custom encoding and some software thinks it's in it's own proprietary format.

[pandy71]

As mentioned by paraj ADPCM is probably most advanced and at the same time computationally friendly with plain A500 sound compression - you can also try to play around A-law/u-Law codecs - i.e. non-linear quantization step - for example 4 bit then you could achieve close to 50% compression and use simple table to decode 4 bit to 8 bit value (something like Fibonacci Delta Compression but with different quantization steps). Something more complex will be probably beyond capability of plain 68000.


---

Started to dig on 8 bit audio compression and some discussion already negate some of my ideas but anyway

https://www.bitsnbites.eu/hiqh-quality-dpcm-attempts/
https://www.drdobbs.com/database/aud...sion/184408798
https://stackoverflow.com/questions/...its-per-sample
https://comp.compression.narkive.com...ression-scheme

[meynaf]

Originally Posted by Steam Ranger

As the title says, I want to be able to compress .iff sound in a way that can do more than the 50% reduction delta compression offers.

I'm going to implement this so it fits within the 8SVX structure, and with decoding can be played back the same as any other 8SVX sound. I'm thinking it would be best to apply some kind of compression on top of delta compression, rather than an entirely different thing.

For performance, it must be able to be decoded in real time at 20KHz sample rate on an Amiga with a 68000.

Does anyone know a suitable compression algorithm I could use to make this possible? Thanks.

You didn't specify the quality you want to achieve. If lossless, then delta compression you mention is already out of league. If lossy, by how much this is acceptable.

But as you prefer something on top of delta compression, you may try to use data that has been delta crunched as source, but in the decrunched form (i.e. normal data but with the losses). It may sound strange but this removes entropy. You can then use smpl mode of xpk library, or do it yourself and convert it to sample mode (values will be not direct but difference with previous), then recrunch with some huffman encoder.

[Steam Ranger]

Originally Posted by meynaf

You didn't specify the quality you want to achieve. If lossless, then delta compression you mention is already out of league. If lossy, by how much this is acceptable.

Lossy is completely acceptable. The quality of fibonacci delta compression is fine, maybe even a bit more compressed than that. The main this is keeping the samples rate at or above 10KHz.

[Docent]

Originally Posted by Steam Ranger

Is there a list anywhere of the compression codes for .iff? Surely someone would have implemented codes other than just delta compression. I'd rather not have it where I try and use an audio file with the custom encoding and some software thinks it's in it's own proprietary format.

I believe there is no official compression except delta. Some software implement additional, unofficial compression ids. There are:
wavepak http://aminet.net/package/mus/misc/wavepak - implements another compression scheme, called expotential delta - its basically Fibonacci delta with different quantisation table.

http://aminet.net/package/mus/misc/8svx_comp implements adpcm compressions in 8svx.
These packages implement 8svx compression using compression id 1-4.
Perhaps there are more, but havent seen any so far.

[pandy71]

Btw 'algorithm' experiments with vector quantization are truly impressing - check https://www.pouet.net/prod.php?which=65976 ; https://www.pouet.net/prod.php?which=64861 and https://www.pouet.net/prod.php?which=65459 - others https://www.pouet.net/user.php?who=66775&show=credits

[Steam Ranger]

Originally Posted by pandy71

Btw 'algorithm' experiments with vector quantization are truly impressing - check https://www.pouet.net/prod.php?which=65976 ; https://www.pouet.net/prod.php?which=64861 and https://www.pouet.net/prod.php?which=65459 - others https://www.pouet.net/user.php?who=66775&show=credits

I watched that first video and I have to agree that it is quite impressive. They say they get a 5:1 ratio, and the audio quality is on par with the best I've ever heard in practical use on the Amiga. Sure the music isn't full PCM, but with that ratio you could definitely have full-length PCM music on a disk without it having to be horribly stripped back.

Thanks for the references, I'll try and implement a test algorithm using the techniques.

[Steam Ranger]

Originally Posted by Docent

I believe there is no official compression except delta. Some software implement additional, unofficial compression ids. There are:
wavepak http://aminet.net/package/mus/misc/wavepak - implements another compression scheme, called expotential delta - its basically Fibonacci delta with different quantisation table.

http://aminet.net/package/mus/misc/8svx_comp implements adpcm compressions in 8svx.
These packages implement 8svx compression using compression id 1-4.
Perhaps there are more, but havent seen any so far.

So essentially any id over 15 would be a safe bet not to be unintentionally recognized by software?

[pandy71]

Originally Posted by Steam Ranger

So essentially any id over 15 would be a safe bet not to be unintentionally recognized by software?

If i can say something - yes, it is fine, ideally id should be documented somewhere, if open source this will be perfect.

[Steam Ranger]

Originally Posted by pandy71

If i can say something - yes, it is fine, ideally id should be documented somewhere, if open source this will be perfect.

While the code I'm writing won't be open source (or at least I plan for it not to be), I will make open source the structure of the compressed data such that someone else could write their own program to compress or decompress it. Along with documentation of the compression, IDs for it will be documented (IDs plural, as there will likely be several levels of compression trading off size for CPU time).

[pandy71]

Originally Posted by Steam Ranger

While the code I'm writing won't be open source (or at least I plan for it not to be), I will make open source the structure of the compressed data such that someone else could write their own program to compress or decompress it. Along with documentation of the compression, IDs for it will be documented (IDs plural, as there will likely be several levels of compression trading off size for CPU time).

Fair. It is up to you as author.

[nocash]

Originally Posted by Steam Ranger

So essentially any id over 15 would be a safe bet not to be unintentionally recognized by software?

Just curious, why do you want to go above 15?
As far as I understand, method 0-4 are used for official 8svx, wavepak, and 8svx_comp, so 5-15 should be also free for use... or are they used for something else?

[Minuous]

They are free for use.

[Steam Ranger]

Originally Posted by nocash

Just curious, why do you want to go above 15?
As far as I understand, method 0-4 are used for official 8svx, wavepak, and 8svx_comp, so 5-15 should be also free for use... or are they used for something else?

It's incase anyone figures that 5-15 are free to use, so they don't make software that will misinterpret my IDs as their propietry format.

[Minuous]

But they would not think they are free to use, if you register your compression format at the official registry at https://wiki.amigaos.net/wiki/IFF_FO...Chunk_Registry, as has been done for eg. various new IFF ANIM compression types.