Loading WHDLoad games by decompressing on the fly

[temisu]

Originally Posted by Wepl

Which zip implementation does support protection bits and file comments?
I did not know about.

I think they all (most?) do support protection bits, although they do not agree about the format they are stored in.
in zip-format the attributes are stored as 32-bit integer of which first 16 bits are used for dos and next 16 bits for amiga. Some of the compressors invert the bits, while some others store the attributes as is. (Does not make situation any better)

For the filenotes, I would need to check again. I forgot which one I used earlier for my testing

Originally Posted by Wepl

I really would like to add this. C would be fine, there is already a small part compiled using vbcc. I could probably also define a small ABI which would allow to have this as a separate BLOB.
Basically needed are:
- function which iterates over all file names (PreLoad)
- function which returns size of a file (GetFileSize)
- function which reads part of a file (or complete file)
- function which iterates over all file names in a sub directory (ListFiles)
- function which iterates over all files/dirs and delivers all filesystem meta data (Examine)

Only if the Slave has Examine flag set WHDLoad collects all filesystem meta data at startup. This data is needed for the Examine/ExNext calls and collected before to avoid many os-switches. So this happens for most kickemu games.

Ok, few questions to help get started.

• Are dynamic memory allowed (or desired)?
• Does the code have to be re-entrant. Assuming the slave run in f.e. kickemu environment? Does WHDLoad support it or handle it somehow?
• Is it better to leave caching for example for the directory structure out of code?

The API/ABI could be something like this

typedef int (*read_func)(void *dest, unsigned length, unsigned offset);

/* assuming we do not have dynamic memory allocations the state would
be allocated outside of the library, otherwise we can return it as void* */
int container_state_initialize(struct container *container_info,read_func reader,unsigned file_length);

int container_get_file_size(struct container *container,const char *name);
/* examine struct allocated by callee, null name for recursive read of everything */
int container_examine(FIB *dest,struct container *container,struct examine *examine,const char *name);
int container_exnext(FIB *dest,struct examine *examine)

/* file info allocated by callee */
int container_file_open(struct container *container,struct file *file,const char *name);
/* length>file size to read a whole file i.e. 0xffffffff */
int container_file_read(struct file *file,void *dest, unsigned length, unsigned offset);

Well, I need to work on my naming skills but maybe this gives an idea?

I'll start looking the code I have. It is mostly in C++ so it needs bit work to translate into C.

I can also do it in steps so we can see whether we are going to right directly i.e. I wont include compression first, but a plain container loading, then we can hopefully test. I can try to see if I can fit both lha/zip to the implementation since I have both at hand...

[Wepl]

Originally Posted by temisu

[*]Are dynamic memory allowed (or desired)?

At the first implementation step this will be only called when the OS lives. So all system functions (V37+) can be used. This implies that this way files can be loaded and uncompressed only at startup via PreLoad or later via an OSSwitch. It maybe later desired to enable the possibility to PreLoad compressed data and to decompress it on demand when OS is disabled. This would then require a decompress routine when cannot use any OS resources. But I would leave that for later.
All memory needed to handle the archive should be allocated at initial opening the archive. Reading a file should not allocate memory. The PreLoad operation may consume nearly all free memory (a small amount is kept free).

Originally Posted by temisu

[*]Does the code have to be re-entrant. Assuming the slave run in f.e. kickemu environment? Does WHDLoad support it or handle it somehow?

Strictly there is no requirement for re-entrant I think. But i would prefer re-entrant code as it is good coding style. Keep in mind that WHDLoad supports multiple data sources and should also support multiple archives. So there maybe more than one archive open at a time.

Originally Posted by temisu

[*]Is it better to leave caching for example for the directory structure out of code?

Cannot say. For the iterating functions at startup it make sense I think. These are anyway called only once. For the function to load a single file later its the question how expensive the search for it is. I tend to say no cache here to save memory.

There should be one initial function which opens the archive and returns a allocated handle which contains all data for further calls. Close will free all resources.

fs* fsOpen(const char * name, char ** errorstring);
void fsClose(*fs);

read/get size

int error fsRead(*fs, const char * name, void *dest, unsigned length, unsigned offset);
int error fsGetSize(*fs, const char * name, unsigned **length);

for PreLoad, 1 function pointer

/* this will check the name and allocate memory,
    returns 0=stop preload, -1=skip this file, memory to load file */
typedef int (*whdAllocFC)(const char * name, unsigned length);

int error fsFileCache(*fs, whdAllocFC alloc);

for Examine/ExNext, 1 function pointer

/* this will copy relevant data from the FIB to own memory,
    path is the relative path to FIB-object is located
    returns 0=stop function, -1=continue */
typedef int (*whdRegDC)(const char * path, FIB *fib);

int error fsDirCache(*fs, whdRegDC reg);

Originally Posted by temisu

I can also do it in steps so we can see whether we are going to right directly i.e. I wont include compression first, but a plain container loading, then we can hopefully test. I can try to see if I can fit both lha/zip to the implementation since I have both at hand...

Yes. Also, in the first step only FileCache will maybe sufficient.
It's also unclear when I can find time to make the necessary adaptions in WHDLoad.

[temisu]

Ok, I think I got it. I'll try to prototype something and I'll share the code when I have something presentable.

Originally Posted by Wepl

It's also unclear when I can find time to make the necessary adaptions in WHDLoad.

Well, it is the same for everyone I guess. Work/Life/Hobbies balance and all that.

I don't mind writing code that will be eventually useful, even though it takes time.

[temisu]

ok, here is my humble proposal (It is still very much work in progress)

https://github.com/temisu/WHDLoadContainers

Basically what it has is a simple API that only requires external memory allocation functions + defines for proper int type. Other than that it has zero dependencies (not even to standard libs). I also provided a simple test.c file to test it.

Caveats:

• It probably has a huge amount of bugs (work in progress)
• Only works with uncompressed LHA level1 archives so far
• I did not try it in vbcc, I might have accidentally used "too modern" C-syntax

In any case, at this point not a much of code is written so it is easier to see if I'm going to correct direction or not. Feedback is welcome.

[temisu]

One more thing. For those wanting to test you need to archive with

lha -arz a archive.lha directory_to_arhive

[temisu]

Hi,

Did you have a chance to look at it?

Is the main structure ok? How is the API/ABI? Is it worth finishing in its current form?

[Wepl]

Thanks for your work!
I have created two PRs to show my ideas regarding the API.

[temisu]

Originally Posted by Wepl

I have created two PRs to show my ideas regarding the API.

Good comments.

Maybe I was overthinking a bit for the modularization Your comments make sense.

I'm away for a few days but then I work the code according to the PRs

[temisu]

Changed the api according to your ideas (hopefully)

I did not add any other features yet, not address the testing. I'm hoping we can agree on the API first.

Also, currently the file reading and memory allocations are put into a single file (container_integration.c). It is very posix like and obviously needs more work.

But if you think we are closer to truth here I can work it further

[temisu]

I now updated the codebase with the support for zip, a better documentation and a test suite.

(The test suite is so far very unix-style coding but we have to start somewhere)

Now, my plans next are:
1. Add decompression support. This will be useful mostly for the fileCache mode since seeking performance will be bad.
2. Create the enhancements for seeking plus the tooling required to go with it. This will be obviously a bit bigger bit of work and before I go there. I would like to have some feedback whether the code base is good enough for its intended usage

[Wepl]

I have not deeply checked to all the source.
As long as the API is valid it will be ok I think.
At some point it should be tested to build this for 68k. I would suggest vbcc.

I will create a plugin interface for this in WHDLoad so you can then test this on your own.

For container_getFileSize we still need a possibility to distinguish between a non-existing file and a file of size=0. Please see my patch.

We maybe also need a support function which returns texts for the error codes of the API functions.

[temisu]

Originally Posted by Wepl

At some point it should be tested to build this for 68k. I would suggest vbcc.

Yes, I have that on my todo list

Originally Posted by Wepl

I will create a plugin interface for this in WHDLoad so you can then test this on your own.

Sounds great

Originally Posted by Wepl

For container_getFileSize we still need a possibility to distinguish between a non-existing file and a file of size=0. Please see my patch.

I believe here we had a mis-communication. The current API return non-negative number for existing file. for errors (missing file, name points to directory) negative number for error code is returned.

Is this ok, or would you prefer some other approach? (Obviously this make the file size limit of 2G, but I don't see it as a big minus)

Originally Posted by Wepl

We maybe also need a support function which returns texts for the error codes of the API functions.

Do you have a preference or should it be a simple

const char *container_getErrorString(int error_code)

[Wepl]

Originally Posted by temisu

I believe here we had a mis-communication. The current API return non-negative number for existing file. for errors (missing file, name points to directory) negative number for error code is returned.

Is this ok, or would you prefer some other approach? (Obviously this make the file size limit of 2G, but I don't see it as a big minus)

This will be fine.

Originally Posted by temisu

Do you have a preference or should it be a simple

const char *container_getErrorString(int error_code)

This is also fine

[temisu]

After few gurus I got it also working in a classic. I did not test it though on 1.3, but I did not use anything that would prevent it for doing so.

Also added the missing bits and fixed few things.

When you have idea what the plugin interface might look like I can try to build it f.e. .library or something

[Wepl]

Kick 1.3 compatibility is not necessary because WHDLoad requires 2.0.

I created a assembler include file describing the file format and interface (see attachment).
Maybe you add the relevant stuff to your container_api.h or add a new file.
Please check and tell me if something is wrong or unclear.

Deviations:

I have used wvfs instead your 'container' as prefix because I'm not so lucky with container. I think it is more like a virtual filesystem. Just leave your container naming as it is. It doesn't make a difference.

I have renamed 'examine' to dirCache. Maybe we need later a function to examine a single entry. So I think this is better.

[temisu]

Hi,

Looks good. And naming change does not sound bad either (I have always been bad at naming things) - I can do a simple search and replace and use the new name everywhere. This way it does not get confusing in the future

I'll see when I have something working that conform to the spec. It is almost like WHDLoad slave I see...

[temisu]

Probably it is going to explode horribly. But I made a build...

https://github.com/temisu/WHDLoadArc...eases/tag/v0.2

Also, the makefiles are currently a mess. I need to clean them a bit so that others can build it if they want. Can I include the header file you sent to my repository as well, and if so, what is the license for it?

[Wepl]

Originally Posted by temisu

Probably it is going to explode horribly. But I made a build...

Thanks a lot
I will now make the changes in WHDLoad and do initial tests. Than will provide official beta.

Originally Posted by temisu

Can I include the header file you sent to my repository as well, and if so, what is the license for it?

You may include it but it's anyway not used by your code. It will be part of the next WHDLoad release. Don't know about the licence..

[temisu]

Instead of opening a new thread I'm using this one instead.

We have a first pre-release of archive-loading for WHDLoad.

Wepl made it as a modular feature so you need both a new WHDLoad and the extension in order to use it. You can get them here:

http://whdload.de/whdload/whd187dev.lha

https://github.com/temisu/WHDLoadArc...02/WHDLoad.VFS

How to install?

Copy the beta WHDLoad executable on top of the original WHDLoad, usually C:

Download (or compile) the WHDLoad.VFS and copy it into the installation directory of WHDLoad, which is usually C:

What the first release does?

It allows preload of WHDLoad data-directory. Instead of pointing "Data" into a directory, it can point to an archive file. For slaves which saves savegames / high scores you also need to set "SavePath" since archive files are read only

What are supported formats?

• Lha archives with Level0, Level1 and Level2 headers
• Lha compression methods LH4, LH5, LH6
• Zip archives that are RFC-compliant and not zip64
• Zip compression method deflate
• Amiga specific extensions to both Lha and Zip (for example file attributes, file notes)

Please note that by default only archives that are created by Amiga are supported!

What are the practical formats?

The most efficient format is uncompressed zip. Created with "-r0N" flags. Lha files although supported will incur a slow initialization delay since the file needs to be scanned from beginning to the end. Compression is supported but currently it is not optimized performance wise.

What are the benefits?

By using uncompressed zip-files the game startup is faster for those games using a lot of small files.

What are the planned features?

There is a lot of possibilities here. We can do improved progress display for loading. We can do on-demand decompression for cached files reducing the amount of memory needed for preload. Among other things. In short, we have not taken advantage of all the things we can for now.

In any case I hope that people interested about the feature are happy to test it and give us feedback

[jotd]

superb work. will improve speed even from winuae. and allows to store one archive per game. great