English Amiga Board


Go Back   English Amiga Board > Other Projects > project.WHDLoad

 
 
Thread Tools
Old 08 September 2020, 22:46   #21
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Quote:
Originally Posted by Wepl View Post
Which zip implementation does support protection bits and file comments?
I did not know about.
I think they all (most?) do support protection bits, although they do not agree about the format they are stored in.
in zip-format the attributes are stored as 32-bit integer of which first 16 bits are used for dos and next 16 bits for amiga. Some of the compressors invert the bits, while some others store the attributes as is. (Does not make situation any better)

For the filenotes, I would need to check again. I forgot which one I used earlier for my testing

Quote:
Originally Posted by Wepl View Post
I really would like to add this. C would be fine, there is already a small part compiled using vbcc. I could probably also define a small ABI which would allow to have this as a separate BLOB.
Basically needed are:
- function which iterates over all file names (PreLoad)
- function which returns size of a file (GetFileSize)
- function which reads part of a file (or complete file)
- function which iterates over all file names in a sub directory (ListFiles)
- function which iterates over all files/dirs and delivers all filesystem meta data (Examine)

Only if the Slave has Examine flag set WHDLoad collects all filesystem meta data at startup. This data is needed for the Examine/ExNext calls and collected before to avoid many os-switches. So this happens for most kickemu games.
Ok, few questions to help get started.
  • Are dynamic memory allowed (or desired)?
  • Does the code have to be re-entrant. Assuming the slave run in f.e. kickemu environment? Does WHDLoad support it or handle it somehow?
  • Is it better to leave caching for example for the directory structure out of code?

The API/ABI could be something like this
Code:
typedef int (*read_func)(void *dest, unsigned length, unsigned offset);

/* assuming we do not have dynamic memory allocations the state would
be allocated outside of the library, otherwise we can return it as void* */
int container_state_initialize(struct container *container_info,read_func reader,unsigned file_length);

int container_get_file_size(struct container *container,const char *name);
/* examine struct allocated by callee, null name for recursive read of everything */
int container_examine(FIB *dest,struct container *container,struct examine *examine,const char *name);
int container_exnext(FIB *dest,struct examine *examine)

/* file info allocated by callee */
int container_file_open(struct container *container,struct file *file,const char *name);
/* length>file size to read a whole file i.e. 0xffffffff */
int container_file_read(struct file *file,void *dest, unsigned length, unsigned offset);
Well, I need to work on my naming skills but maybe this gives an idea?

I'll start looking the code I have. It is mostly in C++ so it needs bit work to translate into C.

I can also do it in steps so we can see whether we are going to right directly i.e. I wont include compression first, but a plain container loading, then we can hopefully test. I can try to see if I can fit both lha/zip to the implementation since I have both at hand...
temisu is offline  
Old 10 September 2020, 02:20   #22
Wepl
Moderator
Wepl's Avatar
 
Join Date: Nov 2001
Location: Germany
Posts: 759
Quote:
Originally Posted by temisu View Post
[*]Are dynamic memory allowed (or desired)?
At the first implementation step this will be only called when the OS lives. So all system functions (V37+) can be used. This implies that this way files can be loaded and uncompressed only at startup via PreLoad or later via an OSSwitch. It maybe later desired to enable the possibility to PreLoad compressed data and to decompress it on demand when OS is disabled. This would then require a decompress routine when cannot use any OS resources. But I would leave that for later.
All memory needed to handle the archive should be allocated at initial opening the archive. Reading a file should not allocate memory. The PreLoad operation may consume nearly all free memory (a small amount is kept free).
Quote:
Originally Posted by temisu View Post
[*]Does the code have to be re-entrant. Assuming the slave run in f.e. kickemu environment? Does WHDLoad support it or handle it somehow?
Strictly there is no requirement for re-entrant I think. But i would prefer re-entrant code as it is good coding style. Keep in mind that WHDLoad supports multiple data sources and should also support multiple archives. So there maybe more than one archive open at a time.
Quote:
Originally Posted by temisu View Post
[*]Is it better to leave caching for example for the directory structure out of code?
Cannot say. For the iterating functions at startup it make sense I think. These are anyway called only once. For the function to load a single file later its the question how expensive the search for it is. I tend to say no cache here to save memory.

There should be one initial function which opens the archive and returns a allocated handle which contains all data for further calls. Close will free all resources.
Code:
fs* fsOpen(const char * name, char ** errorstring);
void fsClose(*fs);
read/get size
Code:
int error fsRead(*fs, const char * name, void *dest, unsigned length, unsigned offset);
int error fsGetSize(*fs, const char * name, unsigned **length);
for PreLoad, 1 function pointer
Code:
/* this will check the name and allocate memory,
    returns 0=stop preload, -1=skip this file, memory to load file */
typedef int (*whdAllocFC)(const char * name, unsigned length);

int error fsFileCache(*fs, whdAllocFC alloc);
for Examine/ExNext, 1 function pointer
Code:
/* this will copy relevant data from the FIB to own memory,
    path is the relative path to FIB-object is located
    returns 0=stop function, -1=continue */
typedef int (*whdRegDC)(const char * path, FIB *fib);

int error fsDirCache(*fs, whdRegDC reg);
Quote:
Originally Posted by temisu View Post
I can also do it in steps so we can see whether we are going to right directly i.e. I wont include compression first, but a plain container loading, then we can hopefully test. I can try to see if I can fit both lha/zip to the implementation since I have both at hand...
Yes. Also, in the first step only FileCache will maybe sufficient.
It's also unclear when I can find time to make the necessary adaptions in WHDLoad.
Wepl is offline  
Old 10 September 2020, 22:54   #23
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Ok, I think I got it. I'll try to prototype something and I'll share the code when I have something presentable.

Quote:
Originally Posted by Wepl View Post
It's also unclear when I can find time to make the necessary adaptions in WHDLoad.
Well, it is the same for everyone I guess. Work/Life/Hobbies balance and all that.

I don't mind writing code that will be eventually useful, even though it takes time.
temisu is offline  
Old 23 September 2020, 00:19   #24
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
ok, here is my humble proposal (It is still very much work in progress)

https://github.com/temisu/WHDLoadContainers

Basically what it has is a simple API that only requires external memory allocation functions + defines for proper int type. Other than that it has zero dependencies (not even to standard libs). I also provided a simple test.c file to test it.

Caveats:
  • It probably has a huge amount of bugs (work in progress)
  • Only works with uncompressed LHA level1 archives so far
  • I did not try it in vbcc, I might have accidentally used "too modern" C-syntax

In any case, at this point not a much of code is written so it is easier to see if I'm going to correct direction or not. Feedback is welcome.
temisu is offline  
Old 23 September 2020, 00:20   #25
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
One more thing. For those wanting to test you need to archive with

lha -arz a archive.lha directory_to_arhive
temisu is offline  
Old 30 September 2020, 23:02   #26
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Hi,

Did you have a chance to look at it?

Is the main structure ok? How is the API/ABI? Is it worth finishing in its current form?
temisu is offline  
Old 02 October 2020, 22:56   #27
Wepl
Moderator
Wepl's Avatar
 
Join Date: Nov 2001
Location: Germany
Posts: 759
Thanks for your work!
I have created two PRs to show my ideas regarding the API.
Wepl is offline  
Old 04 October 2020, 17:22   #28
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Quote:
Originally Posted by Wepl View Post
I have created two PRs to show my ideas regarding the API.
Good comments.

Maybe I was overthinking a bit for the modularization Your comments make sense.

I'm away for a few days but then I work the code according to the PRs
temisu is offline  
Old 10 October 2020, 00:04   #29
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Changed the api according to your ideas (hopefully)

I did not add any other features yet, not address the testing. I'm hoping we can agree on the API first.

Also, currently the file reading and memory allocations are put into a single file (container_integration.c). It is very posix like and obviously needs more work.

But if you think we are closer to truth here I can work it further
temisu is offline  
Old 25 October 2020, 23:43   #30
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
I now updated the codebase with the support for zip, a better documentation and a test suite.

(The test suite is so far very unix-style coding but we have to start somewhere)

Now, my plans next are:
1. Add decompression support. This will be useful mostly for the fileCache mode since seeking performance will be bad.
2. Create the enhancements for seeking plus the tooling required to go with it. This will be obviously a bit bigger bit of work and before I go there. I would like to have some feedback whether the code base is good enough for its intended usage
temisu is offline  
Old 29 October 2020, 21:21   #31
Wepl
Moderator
Wepl's Avatar
 
Join Date: Nov 2001
Location: Germany
Posts: 759
I have not deeply checked to all the source.
As long as the API is valid it will be ok I think.
At some point it should be tested to build this for 68k. I would suggest vbcc.

I will create a plugin interface for this in WHDLoad so you can then test this on your own.

For container_getFileSize we still need a possibility to distinguish between a non-existing file and a file of size=0. Please see my patch.

We maybe also need a support function which returns texts for the error codes of the API functions.
Wepl is offline  
Old 30 October 2020, 14:54   #32
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Quote:
Originally Posted by Wepl View Post
At some point it should be tested to build this for 68k. I would suggest vbcc.
Yes, I have that on my todo list

Quote:
Originally Posted by Wepl View Post
I will create a plugin interface for this in WHDLoad so you can then test this on your own.
Sounds great

Quote:
Originally Posted by Wepl View Post
For container_getFileSize we still need a possibility to distinguish between a non-existing file and a file of size=0. Please see my patch.
I believe here we had a mis-communication. The current API return non-negative number for existing file. for errors (missing file, name points to directory) negative number for error code is returned.

Is this ok, or would you prefer some other approach? (Obviously this make the file size limit of 2G, but I don't see it as a big minus)

Quote:
Originally Posted by Wepl View Post
We maybe also need a support function which returns texts for the error codes of the API functions.
Do you have a preference or should it be a simple
Code:
const char *container_getErrorString(int error_code)
temisu is offline  
Old 02 November 2020, 15:06   #33
Wepl
Moderator
Wepl's Avatar
 
Join Date: Nov 2001
Location: Germany
Posts: 759
Quote:
Originally Posted by temisu View Post
I believe here we had a mis-communication. The current API return non-negative number for existing file. for errors (missing file, name points to directory) negative number for error code is returned.

Is this ok, or would you prefer some other approach? (Obviously this make the file size limit of 2G, but I don't see it as a big minus)
This will be fine.
Quote:
Originally Posted by temisu View Post
Do you have a preference or should it be a simple
Code:
const char *container_getErrorString(int error_code)
This is also fine
Wepl is offline  
Old 12 November 2020, 23:39   #34
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
After few gurus I got it also working in a classic. I did not test it though on 1.3, but I did not use anything that would prevent it for doing so.

Also added the missing bits and fixed few things.

When you have idea what the plugin interface might look like I can try to build it f.e. .library or something
temisu is offline  
Old 16 November 2020, 00:41   #35
Wepl
Moderator
Wepl's Avatar
 
Join Date: Nov 2001
Location: Germany
Posts: 759
Kick 1.3 compatibility is not necessary because WHDLoad requires 2.0.

I created a assembler include file describing the file format and interface (see attachment).
Maybe you add the relevant stuff to your container_api.h or add a new file.
Please check and tell me if something is wrong or unclear.

Deviations:

I have used wvfs instead your 'container' as prefix because I'm not so lucky with container. I think it is more like a virtual filesystem. Just leave your container naming as it is. It doesn't make a difference.

I have renamed 'examine' to dirCache. Maybe we need later a function to examine a single entry. So I think this is better.
Attached Files
File Type: s whdvfs.i.s (3.2 KB, 34 views)
Wepl is offline  
Old 16 November 2020, 01:32   #36
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Hi,

Looks good. And naming change does not sound bad either (I have always been bad at naming things) - I can do a simple search and replace and use the new name everywhere. This way it does not get confusing in the future

I'll see when I have something working that conform to the spec. It is almost like WHDLoad slave I see...
temisu is offline  
Old 17 November 2020, 23:33   #37
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Probably it is going to explode horribly. But I made a build...

https://github.com/temisu/WHDLoadArc...eases/tag/v0.2

Also, the makefiles are currently a mess. I need to clean them a bit so that others can build it if they want. Can I include the header file you sent to my repository as well, and if so, what is the license for it?
temisu is offline  
Old 18 November 2020, 22:07   #38
Wepl
Moderator
Wepl's Avatar
 
Join Date: Nov 2001
Location: Germany
Posts: 759
Quote:
Originally Posted by temisu View Post
Probably it is going to explode horribly. But I made a build...
Thanks a lot
I will now make the changes in WHDLoad and do initial tests. Than will provide official beta.

Quote:
Originally Posted by temisu View Post
Can I include the header file you sent to my repository as well, and if so, what is the license for it?
You may include it but it's anyway not used by your code. It will be part of the next WHDLoad release. Don't know about the licence..
Wepl is offline  
Old 03 January 2021, 14:57   #39
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 51
Instead of opening a new thread I'm using this one instead.

We have a first pre-release of archive-loading for WHDLoad.

Wepl made it as a modular feature so you need both a new WHDLoad and the extension in order to use it. You can get them here:

http://whdload.de/whdload/whd187dev.lha

https://github.com/temisu/WHDLoadArc...02/WHDLoad.VFS

How to install?

Copy the beta WHDLoad executable on top of the original WHDLoad, usually C:

Download (or compile) the WHDLoad.VFS and copy it into the installation directory of WHDLoad, which is usually C:

What the first release does?

It allows preload of WHDLoad data-directory. Instead of pointing "Data" into a directory, it can point to an archive file. For slaves which saves savegames / high scores you also need to set "SavePath" since archive files are read only

What are supported formats?
  • Lha archives with Level0, Level1 and Level2 headers
  • Lha compression methods LH4, LH5, LH6
  • Zip archives that are RFC-compliant and not zip64
  • Zip compression method deflate
  • Amiga specific extensions to both Lha and Zip (for example file attributes, file notes)
Please note that by default only archives that are created by Amiga are supported!

What are the practical formats?

The most efficient format is uncompressed zip. Created with "-r0N" flags. Lha files although supported will incur a slow initialization delay since the file needs to be scanned from beginning to the end. Compression is supported but currently it is not optimized performance wise.

What are the benefits?

By using uncompressed zip-files the game startup is faster for those games using a lot of small files.

What are the planned features?

There is a lot of possibilities here. We can do improved progress display for loading. We can do on-demand decompression for cached files reducing the amount of memory needed for preload. Among other things. In short, we have not taken advantage of all the things we can for now.

In any case I hope that people interested about the feature are happy to test it and give us feedback
temisu is offline  
Old 03 January 2021, 16:35   #40
jotd
This cat is no more
jotd's Avatar
 
Join Date: Dec 2004
Location: FRANCE
Age: 49
Posts: 5,096
superb work. will improve speed even from winuae. and allows to store one archive per game. great
jotd is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
Decompressing Shrinkler executables? oRBIT Coders. General 3 09 February 2019 00:45
WHDLOAD not loading games Taff64 project.WHDLoad 60 06 September 2011 09:22
Crash when loading anything with WHDload KONEY project.WHDLoad 30 27 May 2010 21:58
Tower of Babel (demo) and Magic Fly - 3D games Angus Retrogaming General Discussion 0 05 December 2007 12:22
Damn Mac games fly in Fusion!! DDNI Amiga scene 1 04 June 2007 02:44

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 18:51.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.
Page generated in 0.16855 seconds with 14 queries