[FS-UAE] Amiga Game Database for FS-UAE Launcher

[nexusle]

Hey,

as you can read here FS-UAE now have support for an Amiga Game Database. To discuss about this feature I've created this thread.

@Frode

Is it planned to host this database in the internet?

[FrodeSolheim]

Originally Posted by nexusle

Is it planned to host this database in the internet?

If people are interested in using it and contributing fixes to it, I will make it available / host it.

I don't yet know what form would be most useful to let people contribute to it. One way could be to set up a git repository (but it would become quite large with screens and covers).

[nexusle]

Yes! I'm very interessted.

I'm also interrested in an web interface for this database. I mean, if you own the disk(s), FS-UAE would scan the floppy folder and detect this disks (by checksum) and download the database xml(s).

What do you think about this?

[nexusle]

I've created a little JPG to show, what I mean:

[FrodeSolheim]

Hi, sorry for the late reply. Your idea to download on demand is (good drawing, btw) relatively easy to implement . Whether it is efficient depends on how many disks you have (If you have a lot, it will be more efficient to download a complete / compact database of checksums at once and compare locally while larger images (etc) will be generally more efficient to download on demand).

But the first step is to have good-quality information for download in the first place. Distributing database information to end-users is the easy bit, the real work lies in creating the database in the first place (*) and make it easy to maintain. If more than one person is contributing, guidelines and tools are necessary to manage the database.

I can provide a starting point for such a database by automatically collecting and organizing information from multiple sources, but entries will need to be manually verified / altered as needed. Also, in order to achieve good quality, screenshots and covers would have to be added manually.

(* There exists at least one similar database already (GameBase Amiga), but the licensing terms are much too strict -it does not allow use and/or re-distribution except for with GameBase Amiga)

[FrodeSolheim]

If anyone is interested in using the game database feature of FS-UAE Launcher, it would be nice if you would post a notice about it in this thread or otherwise let me know. I will look at improving the feature set and make it possible for people to contribute to a shared Amiga game configuration database if there is sufficient interest

[bcripon]

I would like to see support for a game database. It can eliminate some frustration when first setting up a game configuration.

Bob C

[nexusle]

+1

I'm definitely for a game Database!!

@Frode: can you please privide a little tool to create the cryptic numbers? (uuid, md5, crc32 and sha1)

So we could help to create database entry.

[FrodeSolheim]

Originally Posted by nexusle

+1

I'm definitely for a game Database!!

@Frode: can you please privide a little tool to create the cryptic numbers? (uuid, md5, crc32 and sha1)

So we could help to create database entry.

I will provide such a tool if necessary The reason it may not be necessary (initially, at least) is that I can create best-effort configurations automatically (where these values are included) based on published DATs (etc), and the work lies in testing the configuration and registering options / adding screenshots / correcting the entry if necessary and marking the it as verified. I think this should be less work overall than starting from scratch. Also, the nice thing is that very many configurations will work out of the box anyway, so even the non-verified entries in the database will be useful.

I have created this configuration db, but the launcher needs an update to be able to use it. The main thing that has changed is using UUIDs for game identity / resource lookup instead of names, so that eventual re-naming will be less painful.

[FrodeSolheim]

Here is a new version which supports an alternative database layout (hopefully without breaking existing setups) - Version 1.3.19:

• Line endings / white space cleanup for public git repo.
• Improved .spec file for openSUSE builds [RedDwarf].
• Added docdir option to Makefile [RedDwarf].
• Compilation fixes and buffer underflow fix [RedDwarf].
• (Launcher) Support modified database layout format.
• (Launcher) Added missing save disk image from distributions.
• (Launcher) Updated translations: it [Speedvicio].

http://fengestad.no/wp/fs-uae/download-beta

[FrodeSolheim]

The database can be downloaded here:
http://fengestad.no/wp/fs-uae/database

The database contains a *lot* of generated configurations. One of the reasons that there is one file per configuration is so they can be easily put into version control systems, such as git.

So what's needed (eventually) is to verify the configurations, and add/correct information where necessary (e.g if some game or configuration only works with one floppy drive, or needs more than the default memory) and of course add high-quality screens and covers. I'll have to document what options should be used (and create launcher editor GUI for the most common ones).

There is a (disabled) button in the launcher which in the future can be used to display only configurations which have been verified. Also, the star button will be used to only display favorites (when you later can mark configurations as such).

While possible, it is not convenient to manipulate these files directly. You can start the launcher with the "--editor" argument to open a (basic) editor view within the launcher which makes it a bit easier to manage. It should get a simpler interface + features such as drag and drop of files to the image boxes to copy images in place (etc).

I suggest you don't start seriously modifying the database yet until documentation, guidelines (and better tools) are available.

But you can play around with it (and ask questions if you have any so far).

[nexusle]

Absolutely cool!!! Very, very good!!

Just a idea: should we ask Tony Wiles as he would implement the same database feature in WinUAE?

[FrodeSolheim]

Originally Posted by nexusle

Absolutely cool!!! Very, very good!!

Just a idea: should we ask Toni Wilen [ed.] as he would implement the same database feature in WinUAE?

I'm quite sure he wouldn't want to do that -It really does not need to be supported directly in WinUAE -and not in FS-UAE either. FS-UAE itself does not know anything about this feature.

(It would make more sense to extend FS-UAE Launcher to be able to set up and start WinUAE as an alternative to FS-UAE ;-) -Which is rather trivial to do)

[FrodeSolheim]

Originally Posted by nexusle

Absolutely cool!!! Very, very good!!

I've noticed you using Traps 'n' Treasures as example quite often, and since you are so enthusiastic, here are db modifications for Traps 'n' Treasures :
http://fengestad.no/fs-uae/files/traps_n_treasures.zip -Just extract over the existing files (and re-scan)

The changes are:
* Traps 'n' Treaures config entries are linked to the Traps 'n' Treaures game entry.
* The game only supports one floppy drive, so this is registered in the database in the game entry -options registered in game entries applies to all linked configurations (unless overridden).
* Viewport options are included to correct some cases where autoscaling gets confused.
* Configurations are renamed to the name from the game entry [done automatically by a tool] so all linked configurations use the same spelling of the game name.
* Covers and screens included (cover is a based on a scan from Exotica, screens are cropped versions of screens from LemonAmiga).
* Link to LemonAmiga entry added to the game entry (not displayed in the UI yet).

[nexusle]

Many thanks!

Yes, TnT is one of the best platform jump 'n run after Turrican in my eyes! There are many other JnR out there, i.e. Yo! Joe! or P.P. Hammer, but only TnT I've played to the end

[nexusle]

Hi Frode,

I, and I think many other of us too, have some CD32 images. Are they supported by the database at this moment?

[FrodeSolheim]

Originally Posted by nexusle

Hi Frode, I, and I think many other of us too, have some CD32 images. Are they supported by the database at this moment?

Yes, the database also supports "CD32" and "CDTV" platforms in addition to "Amiga", I just didn't include configurations for them. The updated database can be downloaded here:
http://fengestad.no/wp/fs-uae/database

[nexusle]

Originally Posted by FrodeSolheim

Yes, the database also supports "CD32" and "CDTV" platforms in addition to "Amiga", I just didn't include configurations for them. Here are configurations created from information from TOSEC:
CD32: http://fengestad.no/fs-uae/files/cd32-data.7z
CDTV: http://fengestad.no/fs-uae/files/cdtv-data.7z

Thanks!

Works well!

Is it possible, to match the area of the cover to the cover size of CD32 games, if they are CD32 games?

Example:

1. Amiga Floppy Version


2. Amiga CD32 Version

[FrodeSolheim]

Originally Posted by nexusle

Thanks! Works well!

Is it possible, to match the area of the cover to the cover size of CD32 games, if they are CD32 games?

Yes, I'll improve the display of square covers -if not right now, at least eventually (I'll fix it along with other image-related improvements -cache resized thumbnails etc for faster loading, load images in a background thread, etc).

[nexusle]

Originally Posted by FrodeSolheim

Game configuration database:
http://fengestad.no/fs-uae/files/amiga-data.7z

A selection of covers and screenshots
http://fengestad.no/fs-uae/files/amiga-images.7z

Hey Frode,

files doesn't work to download. Have you delete them?!



Edit:

Just found the problem. Files are renamed. Here the right links:

http://fengestad.no/fs-uae/files/data.7z
http://fengestad.no/fs-uae/files/images.7z

@Frode: Could you put these files on your Ressources page?

[FrodeSolheim]

Originally Posted by nexusle

@Frode: Could you put these files on your Ressources page?

I have updated the database, and it is available from: http://fengestad.no/wp/fs-uae/database. You also need FS-UAE Launcher 1.3.22. This database update contains more screenshots/covers, and many configurations are linked to the corresponding games.

[nexusle]

Hey Frode,

we know, many of the Game Database entries doesn't have pictures. It's hard to fill these pictures by finding the right config id for creating the right folder.

Please can you implement a possibility to drag and drop pictures in the fields for covers and screenshots so FS-UAE Launcher creates the folder for the game config by it self. This would help to complete the database by us users much easier.

[Dic_Ray]

Thank you sooooo much for that.

Originally Posted by FrodeSolheim

I have updated the database, and it is available from: http://fengestad.no/wp/fs-uae/database. You also need FS-UAE Launcher 1.3.22. This database update contains more screenshots/covers, and many configurations are linked to the corresponding games.

[FrodeSolheim]

Originally Posted by nexusle

Hey Frode,

we know, many of the Game Database entries doesn't have pictures. It's hard to fill these pictures by finding the right config id for creating the right folder.

Please can you implement a possibility to drag and drop pictures in the fields for covers and screenshots so FS-UAE Launcher creates the folder for the game config by it self. This would help to complete the database by us users much easier.

As an interim solution, I have created a python script (attached: create_game_links.py.txt) -If you download and run it with:

python create_game_links.py.txt

it will scan the game database files, and create symbolic links to the game-specific image folders (this will not work on Windows!):

Example:
~/Documents/FS-UAE/Amiga/Links/L/Lotus Turbo Challenge 2 (930) - Images ->
~/Documents/FS-UAE/Amiga/Images/df/df05121d-e093-5e11-be88-b1cbbb36f03d

The number in paranthesis is the game ID in hol.abime.net -added to the link name so you can easily doublecheck what game it is if there is ambiguity in the game.

Inside these folders, you can put front.png, title.png, screen1.png, screen2.png, screen3.png, screen4.png and screen5.png. When you put resources here (since this is a symbol link), the images are really stored in the correct UUID-name-based folder and the images will appear in the launcher immediately when reselecting the game.

We must also have some guidelines for contributors to follow (for both images and configuration data) to get a coherent database. There are some short points regarding images:

• front.png should be of good quality, around 768x1024 is OK, but don't resize to this size if it is smaller (or a bit larger) -resizing it will unecessarily reduce image quality (better to leave resizing (if desired) to an automated process when exporting the database for distribution.
• front.png should ideally be rotated / cropped and retouched a bit if needed. When the thumbnail is shown in the launcher, it does not matter, but it should also work for a full-screen game browser...
• Covers should not have watermarks unless it is impossible to obtain one without.
• Screenshots (title.png and screenX.png) should have borders cropped away, see existing screenshots in the database for examples.
• There is no requirement for the screenshots to have any specific size, but the screenshots should be 1:1 or 2:1 (e.g. 320x200 or 640x400 -or 288x192 -just crop to the natural size for the given game, don't force the screenshots to 640x400 if it is really 640x410...), - and if the screenshots are "double size", they must not be processed with any smoothing filters. I have utilities that will shrink the screenshots to 1:1 later if the screenshots do not contain high-res data, and convert to indexed color if possible.

Another important point: Images are associated with *game* entries, not configuration entries, and configurations are in turn associated with games:

configuration -> game -> images

This means that if a configuration is not linked with a game, no images will be shown for the configuration. When running the launcher with the "--editor" argument, there is a button "Pick Game" which can be used to browse and select a game if the configuration is not linked to a game (remember to also use the Save button).

The last point: If you are to contribute changes, you need to send new and updates files to me of course. Unfortunately, there is no easy way determine what files you have updated, unless you
1. Run some kind of program to detect the changes between your Data/Images directories and the original Data/Images from the zip file, or
2. Carefully copy new/updated files to a separate directory structure.
So you need to think about how you can send only modified/new files back to me.

Another way is to use tools to help with this, for instance git, which I like very much, but it has a little bit of a learning curve. (I could set up git repositories for the database and write some short tutorials, I suppose..). If you have other good suggestions which can be implemented without a lot of work, let me know

Originally Posted by Dic_Ray

Thank you sooooo much for that.

Good luck playing with it -And remember that most entries are automatically generated and some will need tweaking to work optimally (or at all). I will announce updates here, of course

[FrodeSolheim]

Here is an alternative link-creating script (create_game_links_2.py.txt). Instead of just symlinking to the image-directory for each game, it recreates the entire database directory structure (relevant for a game) for each game and includes symlinks to the images directory, the game xml and all configuration xml's for that game.

You'll understand what I mean if you run the script and see what's created in Documents/FS-UAE/Amiga/Links... (I forgot to mention in the last post that the links are created in "Documents/FS-UAE/Amiga/Links"...)

This means that when you have modified something for a game (either XML or images), you can zip down the directory for that game and this zip file will contain all XML's and images for the game -with the directory structure intact.

Remember that if you associate a (previously unassociated) configuration with a game, you must run the create_game_links_2.py.txt again to ensure that the configuration XML is symlinked into the "game link directory".

[nexusle]

Originally Posted by FrodeSolheim

Here is an alternative link-creating script (create_game_links_2.py.txt). Instead of just symlinking to the image-directory for each game, it recreates the entire database directory structure (relevant for a game) for each game and includes symlinks to the images directory, the game xml and all configuration xml's for that game.

You'll understand what I mean if you run the script and see what's created in Documents/FS-UAE/Amiga/Links... (I forgot to mention in the last post that the links are created in "Documents/FS-UAE/Amiga/Links"...)

This means that when you have modified something for a game (either XML or images), you can zip down the directory for that game and this zip file will contain all XML's and images for the game -with the directory structure intact.

Remember that if you associate a (previously unassociated) configuration with a game, you must run the create_game_links_2.py.txt again to ensure that the configuration XML is symlinked into the "game link directory".

If this is used, the "Links" folder should be copied outside the FS-UAE scanning folders. In other case all configs are shown twice in FS-UAE Launcher.

[nexusle]

Hi Frode,

I know, it's a GAME database, but what must/can I specifiy for the <file> entry, if I want to create a database file for a plain A1200 config with a hard disk (folder) and no floppies?

At this time, I used a workaround and created a simple text file with a description in it and calculate the checksums, but this text file is inserted as a floppy in DF0:

What can I do?

[FrodeSolheim]

Originally Posted by nexusle

If this is used, the "Links" folder should be copied outside the FS-UAE scanning folders. In other case all configs are shown twice in FS-UAE Launcher.

Good point, bad default placement of the Links folder

Originally Posted by nexusle

Hi Frode,

I know, it's a GAME database, but what must/can I specifiy for the <file> entry, if I want to create a database file for a plain A1200 config with a hard disk (folder) and no floppies?

At this time, I used a workaround and created a simple text file with a description in it and calculate the checksums, but this text file is inserted as a floppy in DF0:

It's not so much it is a game database, that I don't see how it is very useful to use the database for this purpose? I would think using plain .fs-uae files in Configurations would be both easier and more appropriate? Though if there is a use case not properly supported yet, or a bug in the launcher preventing the use of .fs-uae in this case, I would rather fix that problem

[nexusle]

I like the database structure! With a plain A1200 it works very well (if I use a floppy system). Does it work with hard files?!

[nexusle]

Hey Frode,

how can I specifiy, that my machine have only one floppy drive, if the game have more than one disk, but doesn't support more then one drive?

In this case, I write a database entry for "Amiga Workbench 3.1". The complete disk set has six floppies. But if I scan it in FS-UAE Launcher the firt 4 entries are mapped to 4 drives. But I want to have a plain A1200 machine with only one drive.

"<num_floppy_drives>1</num_floppy_drives>" will be set, but doesn't work!

<config uuid="amiga-workbench-31-a1200">
  <name>Amiga Workbench Version 3.1, Amiga A1200</name>
  <game uuid="amiga-workbench-31">
    <name>Amiga Workbench 3.1</name>
    <platform>Amiga</platform>
  </game>
  <options>
    <amiga-model>A1200</amiga-model>
    <num_floppy_drives>1</num_floppy_drives>
  </options>
  <file>
    <name>Workbench 3.1 - Workbench Disk.adf</name>
    <size>901120</size>
    <crc32>52f8f733</crc32>
    <md5>590c42a69675d6970df350e200fe25dc</md5>
    <sha1>c784edf5899da2fa01c0315761e5c10a77fdc845</sha1>
  </file>
  <file>
    <name>Workbench 3.1 - Extras Disk.adf</name>
    <size>901120</size>
    <crc32>32f55078</crc32>
    <md5>c1c673eba985e9ab0888c5762cfa3d8f</md5>
    <sha1>bb1ab9acfc90b8911c78cba3b07a305306391a6c</sha1>
  </file>
  <file>
    <name>Workbench 3.1 - Locale Disk.adf</name>
    <size>901120</size>
    <crc32>e0e5de81</crc32>
    <md5>b53c9ff336e168643b10c4a9cfff4276</md5>
    <sha1>51c754634343eacb69f9f2c114acbfa2f14ca6f6</sha1>
  </file>
  <file>
    <name>Workbench 3.1 - Fonts Disk.adf</name>
    <size>901120</size>
    <crc32>63470de4</crc32>
    <md5>6fae8b94bde75497021a044bdbf51abc</md5>
    <sha1>df890f6d92002a332f26f35004c9446651d4f5fc</sha1>
  </file>
  <file>
    <name>Workbench 3.1 - Storage Disk.adf</name>
    <size>901120</size>
    <crc32>4bd38ef4</crc32>
    <md5>4fa1401aeb814d3ed138f93c54a5caef</md5>
    <sha1>0ff271946f3071a84572dd8e3b9651f8fce783e8</sha1>
  </file>
  <file>
    <name>Workbench 3.1 - Install Disk.adf</name>
    <size>901120</size>
    <crc32>961d28fa</crc32>
    <md5>f4d1e95711f2b80ffc3f8cde95b69078</md5>
    <sha1>0a455a52e0145bfa099aa66ddfeaa6d4e3714f85</sha1>
  </file>
</config>

[FrodeSolheim]

I should clarify one thing about the database: It is structured the way it is to make it possible for people to cooperate in creating and maintaining a set of known good configurations (though future tools could make this much easier and more accessible, but that is a bit beside the point). So ideally, I would hope in time that people would fix entries, add images etc and send these back to be merged and re-distributed for all to enjoy.

So if "personal" configurations are mixed in with the database, it would be more difficult to merge fixes (etc), though it is not really a problem if these personal files were kept strictly separate from the other.

Another point is that I don't really consider the database format final. If I have a good reason to consider another way of organizing the information for better collaboration, I will probably change it. If I get database fixes contributed back, I will then convert these to a possible future format, but private and withheld database entries may not work in a future launcher version...

And in your case, I still don't see the advantage of putting the Amiga/Workbench configuration into the XML database when a .fs-uae file would be more fitting Also, with the recent support for portable FS-UAE folders in the launchers, .fs-uae files are even more convenient to use for a small set of personal setups, even across computers and operating systems.

But to answer your specific questions:
* The floppy drive issue you mention is a practical issue also for games. One can specify the option "floppy_drive_count" to set the max number of drives to be automatically!
* Another option which is needed, but not implemented yet, is a way to specify the floppy insert order, for instance -for Monkey Island 2, you will not want to initially start with disks 1, 2, 3 and 4 as the game needs another set of disks to start the game.
* Hard drive folders and .hdf files are not supported in the database, because there haven't been a use case for this. Zip files will be assumed are hard drives though (part of the WHDLoad support).

[nexusle]

Ok,

I understand this.

If I want use hardfiles etc. I'll create special .fs-uae files.

I now have build a first config XML for "Amiga OS 3.1". This file is based on the ADF files from Cloantos Amiga Forever. Please check them and import the files in database, if you want!

I'll create this also for "Workbench 1.3" and "Amiga OS 2.0".

One point for my question above:

The config entry "<num_floppy_drives>1</num_floppy_drives>" works, but only if I clear DF1 to DF3. The config XML still fills the fields for DF1 to DF3. Is there a way to fix this, if "<num_floppy_drives>1</num_floppy_drives>" is set?