Compressing Samples

[phx]

Does a good, portable C source for compressing Amiga 8-bit audio samples exist? Probably some delta-compression algorithm has the best effect?

[meynaf]

Depends how fast you want it to be. Sources for common formats (e.g. Flac) are available.
You may as well get good results with a general purpose compressor applied after converting samples to delta values (aka sample mode, write a,b-a,c-b instead of a,b,c).
Also depends if you accept it to be lossy or not.

[hitchhikr]

http://ww1.microchip.com/downloads/e...tes/00643c.pdf

[ross]

Originally Posted by hitchhikr

http://ww1.microchip.com/downloads/e...tes/00643c.pdf

Good reference but this is an ADPCM application.
There is a (asm) player for Amiga but is better suited for 16bit(->14 bit) samples.

Maybe phx is looking for something simpler and faster.
I normally use the 8bit delta from P61 and after some LZ pack (that is basically what maynaf has already recommended).
No quality loss and good compression.

[hitchhikr]

That's the one i'm using in PtPack:

static int pack_stepsizeTable[] = {
    7, 8, 9, 10, 11, 12, 13, 14, 16, 17,
    19, 21, 23, 25, 28, 31, 34, 37, 41, 45,
    50, 55, 60, 66, 73, 80, 88, 97, 107, 118,
    130, 143, 157, 173, 190, 209, 230, 253, 279, 307,
    337, 371, 408, 449, 494, 544, 598, 658, 724, 796,
    876, 963, 1060, 1166, 1282, 1411, 1552, 1707, 1878, 2066,
    2272, 2499, 2749, 3024, 3327, 3660, 4026, 4428, 4871, 5358,
    5894, 6484, 7132, 7845, 8630, 9493, 10442, 11487, 12635, 13899,
    15289, 16818, 18500, 20350, 22385, 24623, 27086, 29794, 32767
};

/* Intel ADPCM step variation table */
int pack_indextable[16] = {
    -1, -1, -1, -1, 2, 4, 6, 8,
    -1, -1, -1, -1, 2, 4, 6, 8,
};

int adpcm_coder(char *indata, char *outdata, int len) {
    char *inp;			/* Input buffer pointer */
    signed char *outp;			/* output buffer pointer */
    int val;			/* Current input sample value */
    int sign;			/* Current adpcm sign bit */
    int delta;			/* Current adpcm output value */
    int diff;			/* Difference between val and valprev */
    int step;			/* Stepsize */
    int valpred;		/* Predicted output value */
    int vpdiff;			/* Current change to valpred */
    int index;			/* Current step change index */
    int outputbuffer;		/* place to keep previous 4-bit value */
    int bufferstep;		/* toggle between outputbuffer/output */
	 int ret_len = 0;
    outp = (signed char *) outdata;
    inp = indata;

    valpred = 0;
    index = 0;
    step = pack_stepsizeTable[index];

    bufferstep = 1;

    for(; len > 0; len--) {
		val = (*inp++) << 8;
		//if(val & 0x8000) val |= 0xffff0000;
		/* Step 1 - compute difference with previous value */
		/* Step 4 - Clamp previous value to 16 bits */
		if(valpred > 32767) valpred = 32767;
		else if(valpred < -32768) valpred = -32768;

		diff = val - valpred;
		sign = (diff < 0) ? 8 : 0;
		if(sign) diff = (-diff);

		/* Step 2 - Divide and clamp */
		/* Note:
		** This code *approximately* computes:
		**    delta = diff*4/step;
		**    vpdiff = (delta+0.5)*step/4;
		** but in shift step bits are dropped. The net result of this is
		** that even if you have fast mul/div hardware you cannot put it to
		** good use since the fixup would be too expensive.
		*/
		delta = 0;
		vpdiff = (step >> 3);

		if(diff >= step) {
			delta = 4;
			diff -= step;
			vpdiff += step;
		}
		step >>= 1;
		if(diff >= step) {
			delta |= 2;
			diff -= step;
			vpdiff += step;
		}
		step >>= 1;
		if(diff >= step) {
			delta |= 1;
			vpdiff += step;
		}

		/* Step 3 - Update previous value */
		if(sign) valpred -= vpdiff;
		else valpred += vpdiff;

		/* Step 5 - Assemble value, update index and step values */
		delta |= sign;

		index += pack_indextable[delta];
		if(index < 0) index = 0;
		if(index > 88) index = 88;
		step = pack_stepsizeTable[index];

		if(bufferstep) {
             outputbuffer = (delta << 4) & 0xf0;
         } else {
             *outp++ = (delta & 0x0f) | outputbuffer;
			   ret_len++;
		 }
       bufferstep = !bufferstep;
    }

    /* Output last step, if needed */
    if(!bufferstep) {
		*outp++ = outputbuffer;
		ret_len++;
	}
	return(ret_len);
}

And the decoder i converted to asm:

pta_smp_depacker:           move.l  d0,d7
                            lea     pta_stepsizeTable(pc),a4
                            move.w  (a4),d1
                            lea     pta_indexTable(pc),a3
                            lea     pta_bufferstep(pc),a2
                            sf.b    (a2)
                            moveq   #0,d0
                            moveq   #0,d6
                            moveq   #0,d3
                            moveq   #0,d4
                            move.l  d4,a5
pta_depack_sample:          tst.b   (a2)
                            beq.b   pta_load_new_byte
                            move.b  d2,d3
                            and.b   #$f,d3
                            bra.b   pta_no_smp_new_byte
pta_load_new_byte:          move.b  (a0)+,d2
                            move.b  d2,d3
                            lsr.b   #4,d3
pta_no_smp_new_byte:        not.b   (a2)
                            move.w  d3,d6
                            add.w   d6,d6
                            add.w   (a3,d6.w),d4
                            bge.b   pta_min_index
                            clr.w   d4
pta_min_index:              cmp.w   #88,d4
                            ble.b   pta_max_index
                            moveq   #88,d4
pta_max_index:              move.b  d3,d5
                            and.b   #8,d5
                            and.b   #7,d3
                            move.w  d1,d0
                            asr.w   #3,d0
                            move.b  d3,d6
                            and.b   #4,d6
                            beq.b   pta_delta_1
                            add.w   d1,d0
pta_delta_1:                move.b  d3,d6
                            and.b   #2,d6
                            beq.b   pta_delta_2
                            move.w  d1,d6
                            asr.w   #1,d6
                            add.w   d6,d0
pta_delta_2:                move.b  d3,d6
                            and.b   #1,d6
                            beq.b   pta_delta_3
                            move.w  d1,d6
                            asr.w   #2,d6
                            add.w   d6,d0
pta_delta_3:                tst.b   d5
                            beq.b   pta_set_sign
                            sub.l   d0,a5
                            bra.b   pta_no_sign
pta_set_sign:               add.l   d0,a5
pta_no_sign:                move.w  d4,d6
                            add.w   d6,d6
                            move.w  (a4,d6.w),d1
                            move.l  a5,d6
                            cmp.l   #32767,d6
                            ble.b   pta_max_clamp
                            move.l  #32767,d6
pta_max_clamp:              cmp.l   #-32767,d6
                            bge.b   pta_min_clamp
                            move.l  #-32767,d6
pta_min_clamp:              move.l  d6,a5
                            asr.w   #8,d6
                            move.b  d6,(a1)+
                            subq.l  #1,d7
                            bne.w   pta_depack_sample
                            rts

pta_stepsizeTable:          dc.w    7,8,9,10,11,12,13,14,16,17
                            dc.w    19,21,23,25,28,31,34,37,41,45
                            dc.w    50,55,60,66,73,80,88,97,107,118
                            dc.w    130,143,157,173,190,209,230,253,279,307
                            dc.w    337,371,408,449,494,544,598,658,724,796
                            dc.w    876,963,1060,1166,1282,1411,1552,1707,1878,2066
                            dc.w    2272,2499,2749,3024,3327,3660,4026,4428,4871,5358
                            dc.w    5894,6484,7132,7845,8630,9493,10442,11487,12635,13899
                            dc.w    15289,16818,18500,20350,22385,24623,27086,29794,32767

pta_indexTable:             dc.w    -1,-1,-1,-1,2,4,6,8
                            dc.w    -1,-1,-1,-1,2,4,6,8
pta_bufferstep:             dc.l    0

[hitchhikr]

afair this gives smaller and better quality than p61 samples compression.

[ross]

Originally Posted by hitchhikr

afair this gives smaller and better quality than p61 samples compression.

You're certainly right about the 4-bit P61 compression, but I was talking about delta plus a later LZ (so without loss of quality).

In your case a successive 16-bit->8-bit pass is due.
Ok ok, you can simply grab the hi-byte, but w/o some audio noise shaping filter you loss even more quality!
So for 8-bit I stick to my choice

Thanks for you code!

[phx]

Sorry, I should have been more precise.

Indeed, I need a lossless compression, and at least the decompression shouldn't be too complex, because I have to implement it in assembler as part of the game.

I'm already using LZ for all files, so the approach of doing a delta-transformation before compressing it sounds good. I didn't imagine that such a simple trick is so effective.

ptpack looks interesting. But I really only need a simple lossless 8-bit compression.

[meynaf]

You may try using xpkmaster.library if your game can run under OS.
The delta transform is done by xpk SMPL directly. SQSH is also very good at crunching samples (and pretty fast).

But what exactly is best for you depends on the prerequisites for the game.

Methods such as ADPCM are good for real time, but worthless if you decrunch only while loading.
If you have enough cpu power available then Flac becomes a good option (i have an assembler decoder for this if needed).
Else it's indeed the LZ applied on delta transformed data that fits the best.

[phx]

Originally Posted by meynaf

You may try using xpkmaster.library if your game can run under OS.

No OS available.

SQSH is also very good at crunching samples (and pretty fast).

Crunching performance does not matter. It is done during the build process and must be portable.

Methods such as ADPCM are good for real time, but worthless if you decrunch only while loading.

Exactly. Samples are only loaded once into memory. Then used for effects in the game.

If you have enough cpu power available then Flac becomes a good option

I doubt that. The target is 68000/7MHz.

[roondar]

Now, I know next to nothing about audio algorithms, but I do recall reading about a standard compression format being available for IFF audio samples.

It wasn't very good, but did work on an A500.

And that's my total knowledge on the subject, maybe it's useful

[meynaf]

Originally Posted by phx

Crunching performance does not matter. It is done during the build process and must be portable.

I was more speaking about decrunching performance but if it must be portable then you have little choice.

Originally Posted by phx

I doubt that. The target is 68000/7MHz.

Then you know what to do...

[phx]

Ok, I have some results with real data. As I feared, the gain by such a delta-transformation is not impressive. Three test cases:

1. All instrument samples from a Protracker MOD (53476 bytes)
lz: 42940, delta-lz: 41527

2. Scream sample (15418 bytes)
lz: 15100, delta-lz: 14994

3. Get-Ready speach sample (5926 bytes)
lz: 3825, delta-lz: 3021

I guess that's better than nothing...

[Don_Adan]

Originally Posted by phx

Ok, I have some results with real data. As I feared, the gain by such a delta-transformation is not impressive. Three test cases:

1. All instrument samples from a Protracker MOD (53476 bytes)
lz: 42940, delta-lz: 41527

2. Scream sample (15418 bytes)
lz: 15100, delta-lz: 14994

3. Get-Ready speach sample (5926 bytes)
lz: 3825, delta-lz: 3021

I guess that's better than nothing...

If you need very good 68000 decompression and good compression for samples use ARJ mode 7. This packer was used for Wanted Team's 1-disked versions of BC Kid and Turrican 2. If you need very good (best?) compression use hitchhikr packer (LZMA). But this packer has slow decompression. Anyway can be used for packing main program file.

[meynaf]

Originally Posted by phx

Ok, I have some results with real data. As I feared, the gain by such a delta-transformation is not impressive. Three test cases:

1. All instrument samples from a Protracker MOD (53476 bytes)
lz: 42940, delta-lz: 41527

2. Scream sample (15418 bytes)
lz: 15100, delta-lz: 14994

3. Get-Ready speach sample (5926 bytes)
lz: 3825, delta-lz: 3021

I guess that's better than nothing...

LZ alone does not appear to be a good method in this case because it only compresses sequences and pays no attention to the relative weight of individual values. In delta samples you should find a strong statistical bias toward small values and thus you would gain quite a lot of benefit by using some kind of Huffman encoding.

[phx]

Originally Posted by meynaf

In delta samples you should find a strong statistical bias toward small values and thus you would gain quite a lot of benefit by using some kind of Huffman encoding.

Excellent idea!

I will certainly find a portable Huffman encoder in C somewhere. The 68k decoder is another topic. But the encoder is sufficient for some tests.

[ross]

Originally Posted by phx

I will certainly find a portable Huffman encoder in C somewhere. The 68k decoder is another topic. But the encoder is sufficient for some tests.

You can try Crunch-Mania LZ-H and if I'm not mistaken also RNC Propack have a LZH variant.
But sure Arj m7 is much more effective (depack speed like ZIP Deflate).

[NorthWay]

Originally Posted by phx

Ok, I have some results with real data.

Your numbers do not sound(!) right. IIRC you typically see 40-50% reduction in size.

Just to check, why not try with xpk and some of the sample specific crunchers for comparison?

[Don_Adan]

Originally Posted by NorthWay

Your numbers do not sound(!) right. IIRC you typically see 40-50% reduction in size.

Just to check, why not try with xpk and some of the sample specific crunchers for comparison?

Are Ok. Samples are very hard to good compression. Only a few packers can do good enough results. From my memory, only Arj m7, PackFire (LZMA), xpkSHRI are the best for sample compression. All other packers are poor or average only.

[SKOLMAN_MWS]

Originally Posted by phx

Indeed, I need a lossless compression, and at least the decompression shouldn't be too complex, because I have to implement it in assembler as part of the game.

https://encode.ru/threads/780-TTA-ve...ll=1#post14513