English Amiga Board

English Amiga Board (https://eab.abime.net/index.php)
-   project.TOSEC (amiga only) (https://eab.abime.net/forumdisplay.php?f=33)
-   -   new online TOSEC tool (https://eab.abime.net/showthread.php?t=90589)

Steffest 29 January 2018 22:57

new online TOSEC tool
 
Hey,
I've been busy the past week helping with the imaging of the Bamiga Sector One Collection from physical disks to ADF.
Doing that I had to dig around to find the correct TOSEC tools and .dat files to check if the disks where already known or not.
Installing all this is fine if you really are into collecting, but I missed a quick and easy way to quickly check a few disks.
So I made one - all in the bowser, no installing needed:

http://www.amigaclub.be/tosec/

I have a question though: Currently only the md5 hash of the disk is checked.
In the BS1 collection, this results in HEAPS of disks not already in TOSEC, while in most cases it's just a slight variation of a disk that is already present.
Is there a way to quickly identify these "alt" versions?
Maybe by comparing file hashes or track/sector hashes?
Are there any existing databases for this or should I just create one myself?

Ideally I would like the tool to return stuff like "This disk matches xxxx.yyyy.adf for 95% so it's probably this disk" instead of only the black&white match/no match.
That way you can actually see if you have found something rare or not.

DamienD 29 January 2018 23:04

Quote:

Originally Posted by Steffest (Post 1215950)
Hey,
I've been busy the past week helping with the imaging of the Bamiga Sector One Collection from physical disks to ADF.
Doing that I had to dig around to find the correct TOSEC tools and .dat files to check if the disks where already known or not.
Installing all this is fine if you really are into collecting, but I missed a quick and easy way to quickly check a few disks.
So I made one - all in the bowser, no installing needed:

http://www.amigaclub.be/tosec/

Wow!!! :great

In the past I've provide "combined" TOSEC .DATs to assist people but I'm no coder and alas these still need to be run the old fashioned way; via ClrMAMEPro:

Amiga - Latest Public DATS Combined (TOSEC-v2007-03-14) & (TOSEC-v2017-04-23)

emufan 29 January 2018 23:30

nice tool :cool

#1) ADF workshop is able to identify similiar disks,
it lists the correct tosec diskname, or lists disks close to it sorted by percentages.

BippyM 30 January 2018 00:31

1 Attachment(s)
Quote:

Originally Posted by Steffest (Post 1215950)
I have a question though: Currently only the md5 hash of the disk is checked.
In the BS1 collection, this results in HEAPS of disks not already in TOSEC, while in most cases it's just a slight variation of a disk that is already present.
Is there a way to quickly identify these "alt" versions?
Maybe by comparing file hashes or track/sector hashes?
Are there any existing databases for this or should I just create one myself?

I seem to remember there was a tool that did this, however I am unsure of it right now.

I guess it wouldn't be that difficult to code something that does a byte comparison on the adf and let's you know the difference %, num bytes different etc... Worth pondering I suppose

Edit in fact yes I do remember ;) see the screenshot

chip 30 January 2018 06:47

How TOSEC team choose amongst the various slight variations of a disk ?

In other words, when a disk tagged with [a] is inserted in the archive ?

Steffest 30 January 2018 10:53

Quote:

Originally Posted by emufan (Post 1215970)

#1) ADF workshop is able to identify similiar disks,
it lists the correct tosec diskname, or lists disks close to it sorted by percentages.

Hmm, Can't seem to find that particular option.
Do you mean there are fingerprints other then the usual CRC/MD5/SH1 in those .db files? Or does it do a binary compare on 2 adfs ?
I guess I should ask Crashdisk about this.

Steffest 30 January 2018 10:58

Quote:

Originally Posted by BippyM (Post 1215991)

I guess it wouldn't be that difficult to code something that does a byte comparison on the adf and let's you know the difference %, num bytes different etc... Worth pondering I suppose

Edit in fact yes I do remember ;) see the screenshot

Ah, thanks for the screenshot.
I would like to avoid a full binary compare though, as then the entire ADF would have to be transfered to/from the server.
And even then: I you do a disksalv or disk optimisation on a DOS disk, it still would be the same disk but it would be binary totally different.

A "simple" file list compare or a track hash compare would be much faster, but setting up these kinds of fingerprint databases for 80.000 disks would be a little out of scope for this project for now.
I guess I'll have to bug the guys on the TosecDev forums for this kind of data.

mai 30 January 2018 16:04

Quote:

Originally Posted by Steffest (Post 1216055)
Hmm, Can't seem to find that particular option.
Do you mean there are fingerprints other then the usual CRC/MD5/SH1 in those .db files? Or does it do a binary compare on 2 adfs ?

Alcatraz demo.adf within part 64
Quote:

File selected : Alcatraz demo.adf
File size : 901.120 bytes
CRC32 : $BD7ED8A3
File loaded as an ADF : AmigaDOS OFS
Quote:

CRC32 based identification :
Special disk____*[BD7ED8A3] I've Got the Power (1991)(Alcatraz)

TOP 5 based on track identification :
1) 99.4% [FFB491CB] I've Got the Power (1991)(Alcatraz)[b dump]
2) 76.3% [0E2842B5] alcatraz.power.music.
Its a TOSEC disk and there are a few known similar disks according to the database!
Isnt this, what you are looking for?

BTW: To be up to date you have to download database of current TOSEC release!

-----
Anyway, nice online tool for quick and easy checking ADFs.:great

Steffest 30 January 2018 16:33

Quote:

Originally Posted by mai (Post 1216115)

Its a TOSEC disk and there are a few known similar disks according to the database!
Isnt this, what you are looking for?

Ah ... yes indeed.
So that database DOES contain track based disk fingerprints!
Thanks, I'll dig into that.

mai 30 January 2018 16:58

Database of current TOSEC release is here:
http://eab.abime.net/showthread.php?t=64668
TOSEC-v2017-11-01.db

chip 31 January 2018 19:42

Your TOSEC tool is nice, but can i suggest you one thing ?

At the moment your tool miss the support for .dms files

This kind of disks are so common that i guess they should be supported by your tool

Anyway, no idea of the effort needed for such kind of improvement

DamienD 31 January 2018 19:46

BS1 Collection - awesome looking haul of disks is the wrong thread for this discussion chip.

Moved accordingly ;)

Steffest 01 February 2018 04:55

Quote:

Originally Posted by chip (Post 1216425)
At the moment your tool miss the support for .dms files

This kind of disks are so common that i guess they should be supported by your tool

Sure, and plain .zip, .adz and .7z files too ...
One step at a time :-)
It will never be a batch converter though, there are other/better options for that.
Dropping 200 .dms files in one go would probably kill your browser.

chip 01 February 2018 05:48

Sure Steffest, one step at a time ;)

And since we are talking of this, do you have intention to create also a standalone application for such kind of TOSEC check ? :blased

DamienD 01 February 2018 11:09

Quote:

Originally Posted by chip (Post 1216537)
do you have intention to create also a standalone application for such kind of TOSEC check ? :blased

If you are going to check thousands and thousands of files, I don't know why you don't just use ClrMAMEPro and the latest combined dats?

Again, see these threads:

... ADFs - renaming / comparing / finding out if you have anything rare!!!
... Amiga - Latest Public DATS Combined (TOSEC-v2007-03-14) & (TOSEC-v2017-04-23)

chip 01 February 2018 11:24

No, i'm not going to check thousands and thousands of files, it's not that my intention

I only guessed that such kind of operation (checking few disks against TOSEC) could be done also with a standalone application

I'm still not able to use ADF-Workshop to do that, even if i know there's a way to do it ..... my fault :banghead

mai 07 February 2018 17:18

Quote:

Originally Posted by chip (Post 1216571)

I'm still not able to use ADF-Workshop to do that, even if i know there's a way to do it

You have disqualified yourself to get further Explanations how to use this tool, because of some statements!

ascp 27 June 2018 14:11

Quote:

Originally Posted by Steffest (Post 1215950)
Hey,
I've been busy the past week helping with the imaging of the Bamiga Sector One Collection from physical disks to ADF.
Doing that I had to dig around to find the correct TOSEC tools and .dat files to check if the disks where already known or not.
Installing all this is fine if you really are into collecting, but I missed a quick and easy way to quickly check a few disks.
So I made one - all in the bowser, no installing needed:

http://www.amigaclub.be/tosec/

I have a question though: Currently only the md5 hash of the disk is checked.
In the BS1 collection, this results in HEAPS of disks not already in TOSEC, while in most cases it's just a slight variation of a disk that is already present.
Is there a way to quickly identify these "alt" versions?
Maybe by comparing file hashes or track/sector hashes?
Are there any existing databases for this or should I just create one myself?

Ideally I would like the tool to return stuff like "This disk matches xxxx.yyyy.adf for 95% so it's probably this disk" instead of only the black&white match/no match.
That way you can actually see if you have found something rare or not.


Handy tool, but at the moment it just stops at "checking tosec". I tried latest chrome and firefox and with a few different adf-files.

Steffest 28 June 2018 08:18

Quote:

Originally Posted by ascp (Post 1250752)
Handy tool, but at the moment it just stops at "checking tosec". I tried latest chrome and firefox and with a few different adf-files.

Fixed!

demolition 29 November 2018 04:56

Quote:

Originally Posted by Steffest (Post 1215950)
So I made one - all in the bowser, no installing needed:
http://www.amigaclub.be/tosec/

Thanks a lot for the lovely tool - this was just what I was looking for after having dumped a bunch of floppies. :)


One problem for me is that many of these disks that I dumped had been copied using something like Bamcopy so unused sectors were left untouched thus invalidating the checksum.. I know that this has nothing to do with your tool, but it does mean a lot of extra work to verify the disks against the versions in TOSEC.


All times are GMT +2. The time now is 10:52.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.

Page generated in 0.05608 seconds with 11 queries