Fast ZIP (Deflate) unpacking

[Photon]

Been a while, was just wondering if your header stripper file-prepping code will make it to a shell command?

[Keir]

Originally Posted by Photon

Been a while, was just wondering if your header stripper file-prepping code will make it to a shell command?

I use a Linux cross-dev environment, so I have no setup for making Amiga CLI utils. This is one reason why deflate was interesting to me, as I can create compressed streams easily on the Linux host.

[Photon]

So how would anyone without your setup prepare a compressed file for use with your decompression source?

[Keir]

Originally Posted by Photon

So how would anyone without your setup prepare a compressed file for use with your decompression source?

Produce a gzip file and strip the gzip header (trivial job, format described in RFC 1952, also I provide POSIX C source to do the job). Produce a zlib file and strip the zlib header (trivial job, format described in RFC 1950). Extract deflate stream from a PNG file. And so on; many uses, possibilities, but not a direct replacement for an Amiga-dedicated packer/unpacker toolkit. But there are very many of those, some very good, whereas easy creation of compressed files on a foreign host is a rarer feature.

[phx]

Originally Posted by Photon

Been a while, was just wondering if your header stripper file-prepping code will make it to a shell command?

Ok, I spent one or two hours today to convert degzip.c into a portable ISO-C99 source. Then used vbcc to make an AmigaOS 68000 executable from it. You find it together with the source in the attached archive.

Originally Posted by kaffer

also I provide POSIX C source to do the job).

Unfortunately POSIX only guarantees portability under Unix-like systems, which AmigaOS is not. Also the endian-conversion functions (htobe32, le32toh, etc.) are not POSIX. And you should not depend on 'int' being 32 bits, so I changed insz and outsz to uint32_t.

[Keir]

Originally Posted by phx

Ok, I spent one or two hours today to convert degzip.c into a portable ISO-C99 source. Then used vbcc to make an AmigaOS 68000 executable from it. You find it together with the source in the attached archive.

That's awesome thank you. I hope you don't mind me adding this into my github repo, I'll cite you at the top of the file.

Unfortunately POSIX only guarantees portability under Unix-like systems, which AmigaOS is not. Also the endian-conversion functions (htobe32, le32toh, etc.) are not POSIX. And you should not depend on 'int' being 32 bits, so I changed insz and outsz to uint32_t.

You're right, I forget how much I deviate from POSIX and use GNU/Linux-isms, such as getopt_long() also, so your job was bigger than I'd've expected.

[phx]

Originally Posted by kaffer

I hope you don't mind me adding this into my github repo

Not at all. Please do!
I have quickly tested all options and everything seemed ok, but no guarantee.

[arti]

Hi,

I would like to use this optimised inflate function in NetSurf for PNG decoding but I have no knowledge of asm.
Can sb complie an object file so I could link it?

What arguments does it take?

int __wrap_inflate(mz_streamp pStream, int flush)
{
	inflate_asm(pStream, flush);
}

[arti]

Will it work for PNG's at anyway?

[phx]

It seems this is the wrong thread, as you don't want to use kaffer's degzip to inflate your data.

Concerning your questions: Yes, as far as I know PNG uses zlib's inflate-algorithm. For everything else you posted too little information.

[arti]

I want to use inflate.asm mentioned in first post, so it should be on topic.

I am sorting out how link vasm a.out with c gcc compiled program.

[phx]

Originally Posted by arti

I want to use inflate.asm mentioned in first post, so it should be on topic.

Oh, indeed. That's a long time ago. Now I remember. Sorry.

I didn't write the code, only ported it to vasm. From looking at the source I would say that you call the "inflate" routine with register A5 pointing to the input stream and A4 pointing to the output buffer for the inflated (uncompressed) data.

I am sorting out how link vasm a.out with c gcc compiled program.

Just use the gcc-linker GNU-ld. I assume it expects a.out object file format, which is also generated by your gcc-version for Amiga/68k? Then call vasm with the option "-Faout".

You may also want to rename the inflate function in the assembler source from "inflate" to "_inflate" and export the symbol with "xdef _inflate". So you can call it from C as inflate().

[meynaf]

Originally Posted by arti

Will it work for PNG's at anyway?

Normal zlib's inflate should work for PNG (so this one will too if it's compliant with the spec) but isn't necessarily optimal as it has to support features that PNG doesn't use (i don't remember the details).

[arti]

With xdef _inflate and -Fhunk I have linked this.
Now , how do I call inflate.
Simply inflate(stream)?

[arti]

This how I have implemented inflate.asm in png lib:

register z_streamp instream __asm("a5");
register int flush __asm("a6");

int inflate_asm(register z_streamp instream __asm("a5") ,register int flush __asm("a6"));

int
png_zlib_inflate(png_structrp png_ptr, int flush)
{
...
   return  inflate_asm(&png_ptr->zstream, flush);
}

or

register int outdata __asm("a4");
register z_streamp instream __asm("a5");
register int flush __asm("a6");

int inflate_asm(register z_streamp instream __asm("a5"), register int outdata __asm("a4") ,register int flush __asm("a6"));

int
png_zlib_inflate(png_structrp png_ptr, int flush)
{

...

    inflate_asm(&png_ptr->zstream, outdata, flush);
	
   return  outdata;

}

None of this works, have I missed something ?

[phx]

I don't know which gcc version you use and how it behaves, but I doubt this will work.

What are you trying to achieve with these global register definitions?

register int outdata __asm("a4");
register z_streamp instream __asm("a5");
register int flush __asm("a6");

I'm not sure what will happen here. Probably gcc will allocate these three registers for all functions with these variables, degrading code quality.

And there is no "flush" argument in the assembler implementation of inflate(). Do you know what "flush" is doing? A6 is used for an (optional) alternative stack area, AFAIK, but only when you define the appropriate symbol (it is not used by default).

[arti]

My bad, it is first time I work with asm you know.
I do not understand it fully yet.

int inflate_asm(register z_streamp instream __asm("a5"), register int outdata __asm("a4"));

int
png_zlib_inflate(png_structrp png_ptr, int flush)
{
int outdata;
...

inflate_asm(&png_ptr->zstream, outdata);
	
return  outdata;
}

Still no go.

[arti]

Here is example code:
http://netsurf.baderman.net/inflate.zip

[Keir]

Hi, Sorry I did not check this thread in a long while. Plumbing into libpng is not as straightforward as you think. It seems the library has a zlib-style streaming interface, and wishes to call an inflate handler which will decompress until the input buffer is empty or the output buffer is full. My inflate asm has a simpler interface where you pass a pointer to the (complete) input in memory, and a pointer to sufficient output space in memory. There is no facility for returning to the caller to collect more input stream or to empty an output buffer. This could be added, it's a question of doing the work and also no doubt slowing down the inflate routine as it will need to check for buffer empty/full conditions within some inner loops.

Alternatively if it can be established or arranged that libpng always fully reads the input into memory and provide sufficient output buffer space then you should be able to pull the input/output buffer pointers out of the z_streamp structure and pass those to inflate_asm() in a5/a6. The 'flush' parameter to png_zlib_inflate can be disregarded.

[arti]

Thanks for reply kaffer.
I was hoping it to be as simple as replacing only inflate function.
To achieve what you wrote I need to educate myself a bit more first.