WHDLoad Game Name Lookup

[Enverex]

Is there anything (site or even FS-UAE itself) that I can query to turn a WHDLoad game name, e.g. "3001OConnorsFight" into the game's real name?

I noticed the game's real name is included in the games ".Slave" file, but unfortunately directly before the game's name there are random characters before the game's name each time (different for each game) meaning I can't automatically read the data from there.

How does FS-UAE track down which WHDLoad games you own and can I tap into that data somehow?

[jbl007]

Originally Posted by Enverex

Is there anything (site or even FS-UAE itself) that I can query to turn a WHDLoad game name, e.g. "3001OConnorsFight" into the game's real name?

I don't think it can be done. At least not without too much guessing. Fsuae-launcher recognizes games by checksum, not by name. You can view the data online, for example: http://oagd.net/amiga/pinball-dreams...7-e97d7b12394f -> No name for the .zip

How does FS-UAE track down which WHDLoad games you own and can I tap into that data somehow?

Are you trying to solve your problem?
You may need a launcher for the launcher which does the following:

1. Search your .zip for the .slave file
2. Search the database for the .slave file and get the game uuid
3. Run "fs-uae-launcher $uuid"

Are you using Linux? If so, maybe I could quickly hack something together (already played around with the database some time ago).

[Enverex]

Originally Posted by jbl007

I don't think it can be done. At least not without too much guessing. Fsuae-launcher recognizes games by checksum, not by name. You can view the data online, for example: http://oagd.net/amiga/pinball-dreams...7-e97d7b12394f -> No name for the .zip

If there's an API for looking up by hash then I could use that, I just can't find it documented anywhere though.

Originally Posted by jbl007

Are you trying to solve your problem?
You may need a launcher for the launcher which does the following:

1. Search your .zip for the .slave file
2. Search the database for the .slave file and get the game uuid
3. Run "fs-uae-launcher $uuid"

Are you using Linux? If so, maybe I could quickly hack something together (already played around with the database some time ago).

This is a separate issue but tied into the same end goal. I need the game's proper names so that they are listed nicely in the launcher I'm using and then in turn I need to be able to launch those WHDLoad games from the launcher with FS-UAE.

The key "how do I" part here is "Search the database for the .slave file and get the game uuid". Is there an API or something public that can be queried?

EDIT: I can see the key information on the page you've linked to, but only in Edit mode. None of that shows up in normal browsing (or search) mode...

[jbl007]

Originally Posted by Enverex

Is there an API or something public that can be queried?

You can't query oagd.net online. Data is also stored in "FS-UAE/Data/Game Database.sqlite" on your hard disk. Unfortunately you can't access it using simple sql statements because it is compressed/json-encoded. You need to loop through all the entries to find stuff. There is no "official" documentation. I asked Frode about it some time ago, he posted a code snippet (which was helpful). The rest I needed to figure out by myself.
http://eab.abime.net/showthread.php?t=71258

[Enverex]

Well I've managed to scrape the real game names from the WHDLoad page (http://www.whdload.de/games/all.html) which gives me the proper names.

The FS-UAE database is confusing me though. It stores the ID of a game (I assume this isn't of relevance, it's just the internal ID), the UUID and then the compressed "misc" data field. That's all fine, but what does FS-UAE use to find the game in the database in the first place? Is it hashing the Slave file to create the UUID and using that, or is it doing something else?

[Enverex]

I'm wondering if something's changed since you did all that. There is no data or value field in the "game" database anymore...

sqlite> .schema game
CREATE TABLE game (
                id INTEGER PRIMARY KEY,
                uuid TEXT,
                name TEXT,
                platform TEXT,
                year INTEGER,
                publisher TEXT,
                front_image TEXT,
                title_image TEXT,
                screen1_image TEXT,
                screen2_image TEXT,
                screen3_image TEXT,
                screen4_image TEXT,
                screen5_image TEXT,
                sort_key TEXT,
                have INTEGER,
                path TEXT,
                update_stamp INTEGER
        , adult INT);

EDIT: Found it, it's in the Games.sqlite file in the Cache folder now. It's like JSON inception in this database!

[jbl007]

Originally Posted by Enverex

I'm wondering if something's changed since you did all that. There is no data or value field in the "game" database anymore...

Looks like you're right. Last modified timestamp for "Data/Game Database.sqlite" is 2014-01-05. Means: I used an outdated database without even noticing.

EDIT: Found it, it's in the Games.sqlite file in the Cache folder now. It's like JSON inception in this database!

I don't have Games.sqlite in the Cache dir.
I have Files.sqlite Locker.sqlite and oagd.net.sqlite.
I think I need to wipe all and start from scratch.

[FrodeSolheim]

Originally Posted by Enverex

Is there anything (site or even FS-UAE itself) that I can query to turn a WHDLoad game name, e.g. "3001OConnorsFight" into the game's real name?

The WHDLoad game/variant entries contains a couple of fields which may be of interest:
- file_list (contains the list of files for the slave, including .slave files)
- whdload_args (contains the name of the slave FS-UAE will execute)
- whdload_url (link to the installer on the WHDLoad page, URL includes some kind of "WHDLoad name").

(As an example, you can see the key-value pairs for Lotus 2 WHDLoad here: http://oagd.net/amiga/lotus-turbo-ch...f-4133db25f62e).

Originally Posted by Enverex

How does FS-UAE track down which WHDLoad games you own and can I tap into that data somehow?

FS-UAE Launcher checks the file entries in file_list for the WHDLoad variant against the local file database. If all files are found, the variant is marked as found.

Originally Posted by jbl007

Looks like you're right. Last modified timestamp for "Data/Game Database.sqlite" is 2014-01-05. Means: I used an outdated database without even noticing.

I don't have Games.sqlite in the Cache dir.
I have Files.sqlite Locker.sqlite and oagd.net.sqlite.

The database layout have changed somewhat since the start. Currently, the relevant files for the development version are:
- Cache/oagd.net.sqlite (contains detailed game/variant information from oagd.net)
- Cache/Files.sqlite (database of scanned files)
- Data/Database.sqlite (high-level information about games - what games you have, etc)

These databases are not an "API", so I will reserve the right to change database layout whenever there's a reason for it

[Enverex]

Looks like I'm on the (older) 2.4 version which is why I'm not seeing what JBL is seeing. I'll update to the dev version and check again, then we'll be on the same page :P

[jbl007]

Originally Posted by FrodeSolheim

These databases are not an "API", so I will reserve the right to change database layout whenever there's a reason for it

Got it!


I pushed whdload-launch.py for Linux to github:
https://github.com/sonnenscheinchen/...load-launch.py

Usage: whdload-launch.py /path/to/whdloadgame.zip

But there's a bug, who can find it?
If a game has more than 1 variant with the same slave-name (eg. Alien Breed 3D) the launcher might pick the wrong one. I think I'll add sha1-checking for the slaves too (and hope there are no variants with the (byte-)exact same slave.

[Enverex]

Mine's working fine now, I thought there was something wrong with it for ages but it turns out the OAGD is actually just missing a -lot- of games so of course they don't have SHA sums for FS-UAE.

Would a list be useful to anyone?

[jbl007]

I spotted an error in oagd.net.sqlite. In one row the data cell is empty and the uuid is invalid

SELECT * FROM game WHERE (SELECT typeof(data) IS NOT 'blob');
or just
SELECT * FROM game WHERE data="";

Anyone else have this, or is it just me?

[FrodeSolheim]

I have an entry with empty data, but there's a valid UUID there... -what's the content of the uuid field?

[jbl007]

Originally Posted by FrodeSolheim

what's the content of the uuid field?

'\x8b\xbb\x00Y\x8bqM\x15\x972\xa8-t_\xb2\xfd'

Quotes '' are included. Perhaps it's not really an invalid uuid, but only stored with wrong type (encoded byte-)string instead of binary.

[FrodeSolheim]

Originally Posted by jbl007

'\x8b\xbb\x00Y\x8bqM\x15\x972\xa8-t_\xb2\xfd'

This is a valid UUID (8bbb0059-8b71-4d15-9732-a82d745fb2fd) -It's just converted to a 128-bit number and stored binary in the database, and the sqlite3 interactive shell (or Python) will print the binary data it like that.

You can test the following in a Python interpreter:

import binascii
binascii.hexlify(b'\x8b\xbb\x00Y\x8bqM\x15\x972\xa8-t_\xb2\xfd')

Anyway, an empty data field might be a feature, I cannot remember right now. I suggest you just interpret this game/UUID has having no key-value pairs.

[FrodeSolheim]

Btw, I checked the source code, and I'm doing:

SELECT id, uuid FROM game WHERE data != '';

(so allowing empty data fields is/was intentional, and is supposed to be skipped when looking for valid variants )

[staropram]

A dream I have, a datfile extracted from oagd, I dont care of names, just want to know
what it wants

[Enverex]

Originally Posted by staropram

A dream I have, a datfile extracted from oagd, I dont care of names, just want to know
what it wants

That's what oagd.net.sqlite is, isn't it?

[staropram]

And with Locker.sqlite gives us what ?

[staropram]

Simple question as first post say

Is there a dat for all whdload games that fs swallow ?

All crc32 and size in one dat, don't care of names, it could be whatever
Thats later.