It's time for a change!

[Adderly]

We now have around ~21000 adf game images in Amiga-Tosec. On a Windows PC you need a NTFS partition for the whole games/adf directory coz Fat32 only allows 10k files per dir. And it takes more than just some seconds of loading time if you just want to browse the games/adf dir with a filemanager.

It's time for a change!

Why not have multiple adf's in a single zip? IPF/SPS already have multiple disks collected in one zip so why not do this for adf's as well.
With this we could shrink the whole set a lot too: All the [a]'s with only a few bytes difference would be very well compressed if put together into one archive.

Since i already bring this (old) discussion up again i also suggest a radical change to fix the too long filenames issue. The new created zips could have shorter names. As extreme we could just use the gamename for the zip and put all according adf's (with long filenames) into it. That would mean ~3500 zips in the whole games/adf dir. Still enough if you ask me...


Constructive comments are welcome!

[Codetapper]

Personally I think TOSEC lost the plot years ago. You used to be able to search for games, see the CRC's, suggest changes etc. Now you can't do a bloody thing on their site. The grandiose 5 year plans with no details are also pretty stupid, why not say what the plan is so others can make suggestions before it goes live?

I also hate their massively long names and the fact nobody seems to check for basic errors - half of the [a] games can be deleted straight away as bad files by a simple file copy off the disks shows you read errors all through them. But with nowhere to report the problems TOSEC will gain more and more useless files meaning file listings go slower and slower.

TOSEC really have lost the plot imho - a perfect example of how to destroy a useful site and replace it with nothing...

[Marcuz]

i agree in part with Codetapper, but i don't think to blame people at Tosec: there have been defections and rehauls, and some decision i don't agree with. the problem is that is a big project whose goals interest relatively few people or at least are not much clear. the most of the people' -here and elsewhere- interest is to play with them games.
but i prefer not to discuss that, nor it's my place.
a rationalization of amiga stuff is surely a good idea.
my point is that for the sake of renaming only, probably the archive is not bad as it is. while for the gaming needs, probably it would be useful to concertate an unique set of best versions of games for to use for instance in gamebase / mage without it to need to do yet another set. that would be just one option, but why don't to do something together with Lemonade and gamebase about that? i mean if the sake is the usability of the archives

[mr_0rga5m]

Why not have multiple adf's in a single zip? IPF/SPS already have multiple disks collected in one zip so why not do this for adf's as well.
With this we could shrink the whole set a lot too: All the [a]'s with only a few bytes difference would be very well compressed if put together into one archive.

This is what i did to my 'horde' quite some time (years) back .. I have collated most of them and then rezip'd them to 7z.
I had to write a coupla QUAD utils to assist me, and i did plan to one day tidy them up and release them.
I even (partially) built a romcenter .dat too.

Example:


¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 1 of 5)(Boot)[cr CSL].adf¬59fc7bbe¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 1 of 5)(Boot)[cr CSL][a].adf¬c153e372¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 1 of 5)(Boot)[cr CSL][t +10 LGD].adf¬86a2cf92¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 1 of 5)(Boot)[cr CSL][t +10 LGD][a2].adf¬1e0d575e¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 1 of 5)(Boot)[cr CSL][t +10 LGD][a].adf¬099d55f5¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 1 of 5)(Boot)[cr FLT].adf¬efff1927¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 1 of 5)(Boot)[cr QTX].adf¬a0502813¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 2 of 5)(1)[cr CSL][t +10 LGD].adf¬7e6d3803¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 2 of 5)(1)[cr FLT].adf¬a2caac9d¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 3 of 5)(2)[cr CSL][t +10 LGD].adf¬6083d14e¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 3 of 5)(2)[cr FLT].adf¬50af1388¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 4 of 5)(3).adf¬913772c2¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 4 of 5)(3)[cr CSL][t +10 LGD].adf¬aed4d2f0¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 5 of 5)(4).adf¬274b7d72¬901120¬¬¬
¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)¬Heimdall (1991)(Core)(Disk 5 of 5)(4)[cr CSL][t +10 LGD].adf¬14aab1ea¬901120¬¬¬


This gave me a file called "Heimdall (1991)(Core).7z" which contains the 15 TOSEC named .adf's which is 3.62 MB (3,804,052 bytes) as opposed to the TOSEC made 15 .zips which are 11.8 MB (12,411,303 bytes).

Basically I think its a good idea. :D

.

[Adderly]

@Codetapper
Tosec is not a collection of verified good images like gamebase/mage/lemonade etc. is. It's a database of all available images. Especially the bad [b] and alternate [a] images are needed if somebody is going to scan his collection! If we would remove all bad images and useless alt's users wouldn't got them identified among their stuff. Personally i think of Tosec as a collection of trash with some pearls. All files are needed, sorry.

@mr_0rga5m
I did exactly what you did, with the help of a JAVA tool coded by poohbear! (see http://eab.abime.net/showthread.php?t=20620)
The whole Tosec collection (without CD's) is around 5GB smaller if 7zip is used and much better to handle this way!
The problem: With the next Dat update you need to depack all the stuff, rescan and then sort/repack it again. :banghead

[mr_0rga5m]

Yea like i mentioned i wrote some QUAD (Quick And Dirty) utils to assist in taking a tosec dat and making my own for easy re-scanning with romcanter. Perhaps i need to dig this out and bring it upto date. Now you got me interested. :)

[Unknown_K]

The last full tosec I collected (for the Amiga) was V.24, and thats a bit old. I don't see what having a newer version buys me except for more alternates or some rare games I would never bother playing.

I don't mind the long names, all my stuff is on an NTFS Win2k partition and I burn batches to DVD with ISO 9999 which allows for real long filenames.

People who want to burn a cd readable to the classic OS 3.0 will have a problem with long filenames.

[Codetapper]

@Codetapper
Tosec is not a collection of verified good images like gamebase/mage/lemonade etc. is. It's a database of all available images. Especially the bad [b] and alternate [a] images are needed if somebody is going to scan his collection! If we would remove all bad images and useless alt's users wouldn't got them identified among their stuff. Personally i think of Tosec as a collection of trash with some pearls. All files are needed, sorry.

You miss my point though. The old website was useful, you could suggest fixes yourself. You can't seem to do anything on the current one.

For example I downloaded a coverdisk the other day. There were 2 versions, I grabbed them both and turned out the [a] one is simply unusable, broken files all through it. It should be marked as [b]ad but I can't even report the problem to them. Any reasonably competant cracker could go through and dismiss most of the alt files straight away when the files don't depack or have read errors etc.

[Kodoichi]

TOSEC really have lost the plot imho - a perfect example of how to destroy a useful site and replace it with nothing...

I blame it on everybody writing into the offtopic forum here on EAB and not helping out Tosec.

[Marcuz]

i think that if the TOSEC site/community are or not right on the spot with their decision, it is not the discussion here. i personally find very useful what Adderly and [idoru] do about it.
also Kodoichi you cannot blame everybody who's not interested in Tosec or in anything for the shortcomings of it, if ever.

[Anubis]

One of the reason I'm not using the tosec is that contains way too much crap. Another reason is that there is no dat file that will keep all files for the game in one zip (or prefferably 7zip) file. This would help with organization and dat file should be able to search trough zips and see difference (same it does with mame roms).

Why whole amiga tosec is 1 file per zip I have no idea.

[NOB]

And it takes more than just some seconds of loading time if you just want to browse the games/adf dir with a filemanager.


same here.
It is nearly impossible to browse the dir.

IMHO the best way to solve this problem is to collect
multiple adf´s in a single zip/dir.

[[idoru]]

I suggested multi-file archives several times down the years & it was always vetoed, don't forget TOSEC is a general project & doesn't revolve around Amiga. It's worth bearing in mind that it's also only a hobby project with few people involved currently. Having said that, I'd still like to see it happen so if any other life is shown outside Amiga renaming then I'll make the case again.

Also, it's quite easy to split your files into multiple directories - there's nothing telling you you HAVE to have all 21k games in 1 directory...

As for Codetapper's comments about the website, yes the website sucks & there is no maintainer. The old "useful" site wasn't replaced with something crap on purpose - the host's server fried & they couldn't recreate it from their backups for whatever reason. I was under the impression that the then admin had his own backups but who knows where they are (both admin & backups). There have been requests for site admins, but as usual the 1 or 2 offers never resulted in anything.

I also agree that the dat options in TIM suck, all the stuff from the old website (ie. create dats without bad dumps, without adf's of games that also appear in SPS dat etc etc.) are all still missing from TIM. Guess what - yeah, the coder & site admin are the same person :) Don't expect me to defend TIM either, I use it just like a lot of people - purely to generate ClrMamePro dats, but there are people who use it in preference. Again, a few people have corrected the odd bug in the code but nothing more substantial has been offered helpwise.

I disagree that there isn't anywhere you can report corrections though - there has always been a forum on the TOSEC site with sections for that very thing. There has been the corrections thread here at EAB for years also.

Various renamers down the years have contributed, some better than others. An idea was to eventually have a "crap" dat containing all the useless adf's (bad dumps, bad cracks etc) but I personally don't have the time or the want to go over all the previously included files. If anyone wants to help then great - you know where we are.

I do this as and when I want to, which is the same I'm sure as Bippym. If I don't want to do anything, I won't - as it's a hobby not a job.

To be honest this is also not the place to be starting this type of discussion, the TOSEC site has a perfectly usable forum of it's own for suggestions & discussion.

[Kodoichi]

also Kodoichi you cannot blame everybody who's not interested in Tosec or in anything for the shortcomings of it, if ever.
I blame it on everybody that downloads ADFs and ROMs named after Tosec convention but doesn't contribute to it.

[Anubis]

As I'm the one who (mostly) posts in off-topic area I wonder how do you like me to contribute.

Just finished reading idoru's post which tell sme that they (they as TOSEC) people are not much open for change and that my input hardly would make any changes.

And yes, I'm using random files from tosec, but it's been more then 3 years since last time I scaned my folders.

Should TOSEC adjust to Amiga and allow mutlipe disks in one archive or stay the same unusable as it is today.

Just for comparison Gametronik has most of the stuff in 7zip archives and one archive has all roms for the same game (also different language, bad dumbs, overdumps...) Their SNES, NES, Gen, SMS, GG collection all together is half of what is TOSEC for SNES by it's self and yet they have more files for SNES?!

[[idoru]]

As I'm the one who (mostly) posts in off-topic area I wonder how do you like me to contribute.

I'm not asking you to contribute. You contribute if YOU want to. My post said "If anyone wants to help", not specifically calling on individuals that SHOULD help.


Just finished reading idoru's post which tell sme that they (they as TOSEC) people are not much open for change and that my input hardly would make any changes.

I don't see where you get that from. The only thing I mentioned was that multi-file zips containing all images for a single title was vetoed on the occassions I raised it. If it was a closed shop then surely it wouldn't even have been discussed. In my experience, people only tend to call things as not open for change if they disagree with the way something is done. Everyone is entitled to their own opinions of course, but just because something isn't done your way doesn't mean it's any less valid.


Should TOSEC adjust to Amiga and allow mutlipe disks in one archive or stay the same unusable as it is today.

Many, many people would disagree that they are unusable. Some would agree it's not ideal. Some would also agree with you I'm sure. If you find it unusable then don't use it - I won't get offense :)


Just for comparison Gametronik has most of the stuff in 7zip archives and one archive has all roms for the same game (also different language, bad dumbs, overdumps...) Their SNES, NES, Gen, SMS, GG collection all together is half of what is TOSEC for SNES by it's self and yet they have more files for SNES?!

Never looked at Gametronik, but there's nothing stopping people 7zip-ing stuff after they've scanned it. None of the utilities (ClrMamePro, RomCenter, TIM) work with 7zip archives which is partly why the various renaming projects don't, of course there are a few addon utils that will 7-zip stuff for you based on the setnames etc... The TOSEC SNES dat is only very early so doesn't contain everything yet.

I've said it before, if people have genuine suggestions then go ahead and make them in the correct place, but if people have got nothing constructive to say then please ignore the threads.

[Adderly]

']The only thing I mentioned was that multi-file zips containing all images for a single title was vetoed on the occassions I raised it.Do you mean vetoed by EAB users or by the other Tosec staff? :confused

[[idoru]]

Do you mean vetoed by EAB users or by the other Tosec staff? :confused

Vetoed by those involved with the project.

[Anubis]

Idoru,
my comment about being called was meant for Kodoichi, not you.

Who else is involved with Amiga TOSEC?

Why they don't try to see what users would like to see and benefit from?

WinUAE supports multiple adfs inside archive, wich is another reason to get them all grouped.

And when did you suggest this?

[Marcuz]

i understand the point of TOSEC being the catalogue and preservation of all the adf from the original data, and not the gaming, but on the other hand why should people to catalogue the adfs if not to play with them?
believe me when i say i treasure what you renamers do and do not complain about it being not a job but a free time occupation.
i would like just to see a nice, ordered, lighter, consultable and most of all usable TOSEC archive and a more opened TOSEC community as their site simply don't do the job, imho, neither it does TIM.

[nnever2000]

I think that an application that read the tosec dat and output a new dat with games merged (comparing it's radice name) can be easly coded. So that no need that tosec changes anything.

I thing the Adderly idea is a very good idea.

UPDATE: I coded an utility the convert cm dat in cm dat merged. I updated in the zone the dat

Commodore Amiga - Games - [ADF][MERGED] (TOSEC-v2006-04-29_CM).dat

seems good to my, but is untested because im at work and dont have rc o clrpro installed.

[Adderly]

You asked [idoru], but maybe i can help:
Who else is involved with Amiga TOSEC?
http://eab.abime.net/showthread.php?t=25291
And when did you suggest this?The Tosec thread got a cleanup some weeks ago. All the old posts are gone...
Edit: "thread" in the last sentence is wrong, the whole Tosec forum got wiped out!

[Anubis]

That's not very helpfull, is it. :)

Thank you for info. :)

[Adderly]

UPDATE: I coded an utility the convert cm dat in cm dat merged. I updated in the zone the dat

Commodore Amiga - Games - [ADF][MERGED] (TOSEC-v2006-04-29_CM).dat

seems good to my, but is untested because im at work and dont have rc o clrpro installed.I did a test with it, had some double filenames warning as i imported the DAT into CLMAMEPro but it worked pretty good so far. :great
Now we only need a ZIP to 7zip converter tool.... :)

Maybe you guys (nnever2000 and mr_0rga5m) want to release your tools into the public?

[nnever2000]

I did a test with it, had some double filenames warning as i imported the DAT into CLMAMEPro but it worked pretty good so far. :great


Nice!! Im going to investigate about the warning.


Now we only need a ZIP to 7zip converter tool.... :)

Hard task (the tool have to decompress the zip format then recompress in 7zip format. Anyway, WinUae support the 7z files??


Maybe you guys (nnever2000 and mr_0rga5m) want to release your tools into the public?

In the zone: Tosec merger

It need the vb run time and is in very beta stage so use with caution and do a backup of the tosec rom set first.

For the future Im going to add four optional check:

1) remove alternate adf, that remove the [a] flag adf
2) remove bad adf, that remove the [b]flag adf
3) include release year in the base name (zip name)
4) include developer in the base name (zip name)

[Adderly]

... Anyway, WinUae support the 7z files??Winuae has support for 7zip but you need a plugin (downloadable from the official winuae website)


For the future Im going to add four optional check:

1) remove alternate adf, that remove the [a] flag adf Oh no, please don't add this. An alternate adf means just that it was added later into Tosec. Often an alt image is better working than the non-alt.

The other options would be great things! Nice work nnever2000! :bowdown

[bippym]

I am in agreement with multie file zips :)

As for contributing, I'm pretty sure me and Idoru are the most active renamers (I do perhaps 1000 unks everty couple months) and I only do it when i'm in the mood.

I try and be thorough when renaming, I try and check file integrity (where possible) and for a few adf's I diskcheck them with xcopy to ensure there are no disk errors!

It's slow work, but someone needs to do it ;)

[nnever2000]

Ok, I tested my dat in the week and worked very well!!! I converted my adf dir in the new merged format.

The merged dat load faster in romcenter and is much easy to select the game in winuae because the number of file in the directory are less.

My dat 'TOSEC-v2006-04-29_CM' seem to be very old, someone has a newer version?

I will improve the TosecMerger program.

UPDATE:
Tosec merger version 0.1.4 in the zone!

[andreas]

Well I was just going to ask in another thread why [idoru] hasn't spit out any DATs since more than half a year, but you've beaten me to it :D

Yes the DATs are horribly old. But [idoru] sits on the new versions (see Overdoc's request thread, lots of new files in those txt files :D) so we can only ask him *kindly* to release them :cheese

[Methanoid]

Argh I am totally confused :banghead

I thought what CLRMamePro called Merged sets are designed to deal with the multiple versions of the same title i.e Gods, Gods (a), Gods (b), Gods cracked by QTX, Gods cracked by FLT and put the whole lot into one Zip file.:spin

Is there a DAT or can someone release a DAT for that? I also thought that if you have made files yourself in such a way you can create a DAT from the directory (function inbuilt into CLR)? :spin

Finally I thought that Zip allows the merged sets but 7Zip doesn't. :spin

OK lots of thoughts but surely someone knows the answer. :confused

I've lost all faith in TOSEC since the tools become so crap. I actually found TUGID easier to use that TIM but I find OfflineList even easier!!

Maybe some fixed DATs could also be combined with the MAGE project. So we get an Amiga database (and other formats) that is useable, smaller and works with the emulators we have. :cheese

??

[thevoice]

omg please no crap 7zip support for tosec. its crap slow, not pretty compatible on all plattforms and e-uae in linux dont support it at all.

imho before merging , using crap compressors i would prefer the chaos in tosec as it is as long you can use it on ALL plattforms that have uae :)

[mr_0rga5m]

omg please no crap 7zip support for tosec. its crap slow, not pretty compatible on all plattforms and e-uae in linux dont support it at all.

imho before merging , using crap compressors i would prefer the chaos in tosec as it is as long you can use it on ALL plattforms that have uae :)


Rubbish, 7zip isnt 'crap slow', granted it may be a little sluggish whilst compressing but that rather depends on the size of the files your compressing (we are talking Amiga files here which typically are not greater than 15mb merged) and the speed of the system used for compressing. (I take it your not compressing them on a 486 pc or something) Also you only compress once, the decompression is quite speedy. :agree

As for e-uae in linux doesnt support it at all, well perhaps somebody needs to pull their finger out and make support for it. All it takes is a little enthusiasm. :)

<imho> Linux is a Windows wannabe anyhow, i dont know why ppl bother with it. :lol </imho>

[Methanoid]

Complaining about 7-zip being slow is like complaining that ZIP is slower than LZH.

FFS no-one uses LZH now, and PCs are much faster these days anyway.

And NO_ONE IS FORCING YOU TO USE 7-ZIP!!!!! :banghead :banghead

[Adderly]

Using 7zip and merged romes i was able to store the whole Amiga Tosec non-iso stuff on 3 DVD's (as zip's and not merged it was 5 DVD's). Winuae has now build in 7zip support. I love 7Zip! :)

[Methanoid]

Using 7zip and merged romes i was able to store the whole Amiga Tosec non-iso stuff on 3 DVD's (as zip's and not merged it was 5 DVD's). Winuae has now build in 7zip support. I love 7Zip! :)

I guess then that all you need to do is copy all the files into one dir on your PC. Load CLRMamePro and using the "make dat from dir" option and create the dat file.

Then the rest of us can use it?!??!

[Toni Wilen]

Implementing EUAE 7zip support takes max 5 minutes. (copy new files, merge zfile.c changes)

[Adderly]

I guess then that all you need to do is copy all the files into one dir on your PC. Load CLRMamePro and using the "make dat from dir" option and create the dat file.

Then the rest of us can use it?!??!I can try this when i rebuild my collection after the next Tosec update.:agree
Too bad CLRMamePro does not support 7zip compressed images...

[Methanoid]

I thought 7-zip support was added in last update?

[DamienD]

You are correct Methanoid (and rar also) :great

http://www.clrmame.com/download.htm

[Methanoid]

So Mr Adderly CAN make that tidier DAT file and provide it for the rest of us.

[Adderly]

It does not work. :(
CRLMAMEPRO is not able to "dir2dat" the content of an 7zipped archive. It just lists the xxx.7z file but not the adf ones IN this archive.
clrmamepro 3.95b (13-Nov-2006)
# added: 7z/rar support for About's drag'n drop feature
# added: 7z/rar support for About's Search CRC32 in Folder feature
It's a lack of real 7zip support! :scream

[[idoru]]

Well, bogy did say in the readme when he added REBUILD support for 7z/rar in v3.94 that he had no plans to implement that.


Before you ask:
===============

No, currently I don't plan to add any compress-related support.

[Methanoid]

I wonder how much a Zip compressed Merged set like this saves? IS it possible to produce CLR dats for ZIP merged set (and I could batch convert them to 7zip later?!)

[Adderly]

I wonder how much a Zip compressed Merged set like this saves? IS it possible to produce CLR dats for ZIP merged set (and I could batch convert them to 7zip later?!)This should be possible.
I am going to merge the whole Amiga Tosec stuff after the next update and then i can provide the Dats.

[Methanoid]

I wonder if the TOSEC db can be output in some other format I can use... like OLL which uses XML....

I have to confess I really DONT like TIM, I liked TUGID as I could understand it and CLRMame and RomCenter have TOO many switches and settings for my liking.

I want a SIMPLE file management tool like OLL which tells me what I have and what I am missing and not a lot else!

[Methanoid]

Update: Well I found TIM is supposed to be able to export XML but everytime I do it the damn thing crashes... I wonder if they fix the bugs it has... TUGID was stable for me!!

[dlfrsilver]

oh well, to me the baddest point is that we have up to 5 entries for only one game when we only need a standard cracked version + a trained one. Others are optional to me....

[Methanoid]

Ah... but if the DAT was in XML and used in Offlinelist there is a duplicate filter function. Hence you can set the thing to ignore the extra copies.

All I want is the XML TOSEC dats and I'll be doing the rest of conversion myself. There are some TOSEC OLL dats but they are 2004 dated

[sut]

Is this not all simlar to what the Goodmerge program does for the Goodtools sets ?

Is there no way that program could be adapted ? it does show a tosec option but i've not explored it.

http://goodmerge.sourceforge.net/

[Methanoid]

IIRC Goodmerge uses GoodTools type DATS and not TOSEC dats (ie diff files are collected) ???

[Methanoid]

If anyone else feels that TOSEC has "lost the plot" on the Alts and Bad dumps issue then get on over to TOSEC.info and make yourself heard, otherwise they will continue with this mess (IMHO)

http://www.tosec.info/modules/newbb/viewtopic.php?topic_id=472&post_id=2676#forumpost2676

[DamienD]

Methanoid, you may say that they've lost the plot when it comes to "alts" but how do you know which versions are 100% working? I know that in my personal WinUAE configs I've had to sometimes use [a1], [a2] or others as the .ADFs without the alt flag haven't worked fully :agree

Yes it would be good to remove alts but only once a 100% verified / tested version has been found. This will take a lot of time.

Ok, I can see your point about about "bad dumps" but even these are sometimes known to work...

[Methanoid]

Damien

I think we have "almost" had an offer from Codetapper to identify the good ones.

But TBH the problem just grows until eventually it will be impossible to fix. Unless and until the TOSEC team take the bull by the horns and decide to stop taking new Alts in and to look to eliminate them in a way (perhaps not the way I suggested) that works... well until that time I think TOSEC is (fatally) flawed.

Look at projects like Gamebase and Lemonade that only include working files.

I mean, whats to stop me taking a workbench disk and changing one file (and doing this 40 times) and guess what... next TOSEC dat... 40 new Workbench 1.3 Alt files... Really.... I mean how can they say No when they have accepted all that crap so far? Oooops I left write protect off the disk and saved my Highscore... new Alt file...

:banghead :banghead :banghead

[bippym]

There will never be the perfect solution to the alts/bad dump pickle!

Personally I think alts and bad dumps should be moved to their own dat and only verified 100% working games should be in the main dat. As DamienD pointed out though, how can one bew certain that an adf or set of adf's work for a particular game without the renamer playing through the whole game?

Also take into account the number of un-named adf's that are floating about. Eventually these will make it into tosec, now if the tosec team decide to simply delete and not include these alts in dats they WILL sooner or later find their way back as users keep sending in unidentified adf's

A line needs to be drawn and for the average user all the alts and bad dumps are not needed but for comleteness and to ensure that all possible adf's are identified so no alts are left they need to be included.

So again my suggestion is for someone (do you want to offer) sits and plays through every game, checks every disk structure for i/o errors and then clean dats for each set are created. This way the user has the option to download the slimline dats or the complete set.

[andreas]

FFS no-one uses LZH now, and PCs are much faster these days anyway.
No one - really? :D

Well, then do an experiment: Take a LHA archive from the Amiga, (optionally RENAME it to LZH first) and open it with WinRAR!
Then press Alt-I ... :D

[mr_0rga5m]

No one - really? :D

Well, then do an experiment: Take a LHA archive from the Amiga, (optionally RENAME it to LZH first) and open it with WinRAR!
Then press Alt-I ... :D


And then what ... Stare in amazement at the =

Version to Extract : Unknown
Host OS : Unknown

:confused :confused :confused I saw the point fly past the window but didnt manage to get it. :lol

[Marcuz]

Personally I think alts and bad dumps should be moved to their own dat and only verified 100% working games should be in the main dat. oh boy, how i would like it me too...

And then what ... Stare in amazement at the =

Version to Extract : Unknown
Host OS : Unknown

:confused :confused :confused I saw the point fly past the window but didnt manage to get it. :lol heheheheheeeeee... cute :)

[Methanoid]

There will never be the perfect solution to the alts/bad dump pickle!

Personally I think alts and bad dumps should be moved to their own dat and only verified 100% working games should be in the main dat. As DamienD pointed out though, how can one bew certain that an adf or set of adf's work for a particular game without the renamer playing through the whole game?


Sounds at least something of a solution....

I was rather horrified as having left the machine on all day scanning my older TOSEC set (chunks of which are now not recognised at all like 600Mb of X68000 games!!!! wtf???) and returning to find that I have approx 20Gb of files that aren't recognised.

TOSEC which in my naivety thought was almost comprehensive has large gaps on all the "console" type formats. I had a full set of PCEngine stuff that Offlinelist assembled. Only 2 of the 200 files were spotted by TIM. Similar with the N64 collection if I recall.

If I thought there was life in the TOSEC project I'd send them to a renamer but.. can I be arsed?:sad

[bippym]

If you have amiga adf's to rename then send them to me.. I'm a renamer and when I get the spur I'll rename 1000-2000 adf's (Ask Ido if you don't believe me) before I get bored again!

If you want to do this tell me and i'll send you dats of my non-named stuff so you don't send dupes :)

Gimme a holler :D

[Marcuz]

i give you a holler! :)

[Methanoid]

If you have amiga adf's to rename then send them to me.. I'm a renamer and when I get the spur I'll rename 1000-2000 adf's (Ask Ido if you don't believe me) before I get bored again!

Gimme a holler :D

Who does the CD32 stuff?

[bippym]

Not too sure tbh. Ask idoru

[killergorilla]

idoru and Duke handle cd32 stuff if I'm not mistaken

[Marcuz]

Bibbym, i meant for real, can you send me the dats and i'll send you the not renamed after, if you want to do it :)

[bippym]

marco I'm a bit busy with some stuff right now. I'll compile the dats later tonight/tomorrow :D

[Marcuz]

sure, thank you mate :)

[bippym]

You could compile dats of your stuff if you prefer and i'll scan my stuff.. I'll remove anything you have and send you a dat back (either a fix dat or a removal dat) to separate what I have

[DamienD]

...and here's some quick instructions I wrote when I did this with bippym ;)


... Load ClrMAMEPro.
... On the Profiler page click the button called "DIR2DAT".
... On the right you have a few string gadgets, on the left you have tick boxes.
... Keep the default ticks and also tick MD5.
... Fill in the information on right hand side.
... Name is datname rest is self exp.
... Input source folder is folder where adf's/zip files are.
... Datfile is path/name of datfile.
... Click force zipping if you want it to zip them up then just press create and wait.
... Then put dat in The Zone! and I'll do a scan and then send you a fixdat.
... You then run the fixdat over your files and it'll create a collection of adf's I don't have.

[Marcuz]

you two guys are just great, i'll do that right away :)

[Marcuz]

mmm force zipping did not zipped anything... ?

however, i've putted in the zone the 1st datfile this one should contain almost only games. however, 80% of it, more than 1GB of files MAY BE dupes of what already had renamed, but Clrmame did not fixed them in the destination directorues during rebuilds, so for now i've kept them where they were. about 10 % of it all should be not tosec

also a little OT: i've 30 something magazine italian disk grabbed from a site. if i post one can you guys suggest a nomenclature (let's say is the first one) so that i do a quick dat?
in 5 mins that too is in the zone
[edit] upped that too.

[andreas]

And then what ... Stare in amazement at the =

Version to Extract : Unknown
Host OS : Unknown


:confused
??!

Not my fault if you're blind! :lol

So much to "LZH? No one uses that!!11"

[mr_0rga5m]

:confused
??!

Not my fault if you're blind! :lol

So much to "LZH? No one uses that!!11"


Well fuck me (!) .. I skimmed over that .. i thought it just showed what the .ext was :lol

Anyway i still dont see the point of your statement .. lzh is just an lha archive descriptor, be it amiga or pc or nix or whatever. And its not a mainstream compressor these days.

[andreas]

Agree on the last sentence, but still ... not quite. Old PC games were also archived in lzh.

My statement was: that everyone who uses LHA archives on Amiga with WinRAR uses nothing else but LZH, just a different extension. But the algorithm behind - almost or completely the same!
On (very?) old versions of WinRAR, one even had to rename LHA to LZH to unpack the files...
Still WinRAR has no "official LHA support". You have to make the assignment yourself, e. g. install WinRAR, double-click a .lha, then say "Open With..." etc.

LHAs can only be opened because WinRAR has LZH support. Simple as that. :cool

[AmigaHope]

Anyway i still dont see the point of your statement .. lzh is just an lha archive descriptor, be it amiga or pc or nix or whatever. And its not a mainstream compressor these days.

Actually it *is* a mainstream compressor -- in Japan. Until very recently it was the default archive format for just about everything, and to a large degree it still is.

The Japanese generally didn't use .zip much at all until WinXP supported it natively. Newer versions of XP in Japan support .lzh natively too, though, so .lzh still has widespread use there.

LHarc/LHA are Japanese programs, and were thus better documented in Japan, leading to their wider initial acceptance. This is also why you see people in Japan using .gca, which isn't used anywhere else. (The GCA algorithm is actually quite decent but there's zero English documentation for it, and I don't think there's an open-source implementation.)

.7z pwns all but getting it accepted by Joe Sixpack is next-to-impossible. =(

My statement was: that everyone who uses LHA archives on Amiga with WinRAR uses nothing else but LZH, just a different extension. But the algorithm behind - almost or completely the same!

LHAs can only be opened because WinRAR has LZH support. Simple as that. :cool
It isn't just almost the same algorithm, it *is* LHA support. The original PC-98xx/MS-DOS versions were called "LHA", just like on the Amiga.

LHarc was the name of the original archiver that created the .lzh format. Later an enhanced program called LHA was created that added more compression options. This is the funny thing, LHA was just an advanced reimplementation of LZH, so it still called its files .lzh. It was only when the *Amiga* port was made that it was changed to call its files .lha by default.

So the Amiga .lha extension is really just an aberration put in place by the person (was it Stefan Boberg? I don't remember) who made the Amiga port of LHA, the original LHA never did that.

[sut]

Although I've never tried it, there is a tosec XMDB file that you can use with Goodmerge

here's the download page (read the notes on the latest zip file 2006.11.23 the file in question has been renamed from tosec to ~other~, but should work the same.

Download page
http://sourceforge.net/project/showf...kage_id=134854 (http://sourceforge.net/project/showfiles.php?group_id=122744&package_id=134854)


Notes
http://sourceforge.net/project/shown...roup_id=122744 (http://sourceforge.net/project/shownotes.php?release_id=466021&group_id=122744)


I would make sure you have a backup of your original sets, until you can verify that it works http://forums.edgeemu.com/images/smilies/smile.gif

[DamienD]

This is also why you see people in Japan using .gca, which isn't used anywhere else. (The GCA algorithm is actually quite decent but there's zero English documentation for it, and I don't think there's an open-source implementation.)
I've actually had to use DCGA (Digital G Codec Archiver) for some archives I've downloaded:

http://www.emit.jp/

There's an English version available on that site as well ;)