Local access to OAGD.net configurations

[jbl007]

Originally Posted by FrodeSolheim

[*](Launcher) More compact (OAGD) game database, faster synchronization

Faster indeed, but not so open anymore.
How is the data encoded inside the local db now? I'm asking, because I'm about to write a command line launcher for linux which also has (or had ) access to the local db.

BTW:
I would be really awesome to query the online db from "outside". For example ask the server: Hey, I have a disk image with SHA1 yxz, gimme a config for this game and the SHA1s of the corresponding disk images. That's I was trying to do with the local db.

[FrodeSolheim]

Originally Posted by jbl007

Faster indeed, but not so open anymore.

Not less open, just more compact encoding in the launcher database

Originally Posted by jbl007

How is the data encoded inside the local db now? I'm asking, because I'm about to write a command line launcher for linux which also has (or had ) access to the local db.

Now the game table has three columns (id, uuid and data). uuid is a 128-bit UUID represented as 16 bytes. The data field is a JSON document which is compressed with zlib. Here is python code to extract a dictionary ("doc") with key-value pairs given a game id (follows the parent chain and inherits values):

        cursor.execute("SELECT data FROM game WHERE id = ?", (game_id,))
        data = zlib.decompress(cursor.fetchone()[0])
        doc = json.loads(data)
        next_parent_uuid = doc.get("parent_uuid", "")
        while next_parent_uuid:
            cursor.execute(
                "SELECT data FROM game WHERE uuid = ?",
                (sqlite3.Binary(
                    unhexlify(next_parent_uuid.replace("-", ""))),))
            data = zlib.decompress(cursor.fetchone()[0])
            next_doc = json.loads(data)
            next_parent_uuid = next_doc.get("parent_uuid", "")
            # let child doc overwrite and append values to parent doc
            next_doc.update(doc)
            doc = next_doc

Originally Posted by jbl007

BTW:
I would be really awesome to query the online db from "outside". For example ask the server: Hey, I have a disk image with SHA1 yxz, gimme a config for this game and the SHA1s of the corresponding disk images. That's I was trying to do with the local db.

It is something I have though about, but if used frequently like that, it would be much more efficient in clients downloaded all configurations in one big operation and just queried the local data. So it is a performance issue to consider here, but I won't discount the possibility though.

[jbl007]

Thank you very much for your quick reply, Frode. Your help is greatly appreciated.

Ok, gzip + json. I thought I also tried to decompress using gzip but it did not work for some reason. I'll give it another try...

But the basic problem still exists: you can't search by sha1 easily. You need to decompress/decode at runtime > 12000 entries. I think this will be very slow. I will do some tests.

I did something like this before, btw.:

search = dohash(filename)
searchname = ('%' + search + '%', 'file_list')
conn = sqlite3.connect(db)
c = conn.cursor()
c.execute('SELECT game,value FROM value WHERE value LIKE ? AND name IS ?', searchname)
searchgameno, namefetched = c.fetchone()

[FrodeSolheim]

No, zlib, not gzip. It is basically the same compression algorithm but not the same file format.

Regarding performance, you will probably want to scan through the entries once and create a cache/database with a sha1 -> game_uuid mapping

[jbl007]

Originally Posted by FrodeSolheim

No, zlib, not gzip. It is basically the same compression algorithm but not the same file format.

Aaargh! Of course! That's why "zcat /tmp/extracted" did not work. lol

Regarding performance, you will probably want to scan through the entries once and create a cache/database with a sha1 -> game_uuid mapping

Phew, that sounds like a lot of work.
I just made a single file python3 launcher script, which shortens some --really-long-command-line=options, makes a little config file handling and automates .zip/.lha extraction.
I didn't want to rewrite the fs-uae-launcher with databases, config files and so on.
With my limited python skills it would take ages.

[jbl007]

Actually is was quite easy to get a sha1->variant-list dictionary out of the db based on your example code. And it only takes 0.5s on my i5, so caching isn't really needed. But I did cache the sha1's of the users disk images, but that's another story...
I hope the db-format will not be changed in the near future.

I almost got my one-click-launcher at a usable state. I'll push it on github and post the link here.