Denise C64 mode problems!

[amilo3438]

There are also examples where it doesn't seem to work but in fact it is something else...

Not all games will load and start with: LOAD"*",8,1 ... because the required file is not always in the first place on the disk, which can give the wrong impression that the game is not working.

Here are some examples:

forbidden_forest[cosmi_1983](later) ... LOAD"*",8,1 (works because FF is the 1st file on the disk)
forbidden_forest[cosmi_1983](earlier)(!) ... LOAD"FF",8,1 (works because is not the 1st file on the disk)

top_gun[ocean_1986](pal) ... load"*",8,1 (works because LOADER is the 1st file on the disk)
top_gun[thunder_mtn_1987](pal)(!) ... LOAD"LOADER",8,1 (works because is not the 1st file on the disk)

repton[sirius_1983](!) ... LOAD"SIRIUS",8,1 and RUN (works because is not the 1st file on the disk)

platos_cave[krell_1983] ... load"MENU",8,1 and RUN

so sometimes it is necessary to look into the directory structure to find an explanation if it doesn't work!
(Fortunately, it is not necessary to type in the emulator, but to look in the directory structure and click on one of the files.)

[amilo3438]

Originally Posted by PiCiJi

nightly:
- switch D64 to low level emulation when written
- rework wobble, remove randomness, a wobble setting should lead to same result (with warp too)
- add option to disable motor decelaration

Great, something there fixed previously problematic games to work now always:

andy_capp[mirrorsoft_1987)(pal) ... no need to disable wobble or other!
vixen[martech_1988](pal)(paraprotect_v2) ... no need to reduce speed now!

and probably some others too!

PS. I haven't found anything so far that requires disabling decelaration! Almost everything that has been tested today simply works!

EDIT: Found it: "falcon_patrol[advantage_1983]" ... now need decelaration to be disabled to work! (I guess now it's like vice(sc) which doesn't emulate it anyway.)
PS. It doesn't work on Micro64 emulator too if mechanic emulation is enabled! EDIT: No need to disable deceleration if selected drive is 1541-C in denise!

[PiCiJi]

Originally Posted by amilo3438

EDIT: Found it: "falcon_patrol[advantage_1983]" ... now need decelaration to be disabled to work! (I guess now it's like vice(sc) which doesn't emulate it anyway.)

basically it is a hack that a "broken" G64 does not deviate so far from the norm that it can no longer be loaded because of emulation of the motor deceleration.

Correctly, an old G64 should no longer run. For the user, this is confusing and not understandable why some still work, and he gets the impression that it is a bug.

[amilo3438]

15) championship_lode_runner[br0derbund_1984](!) ... when the fire button is pressed on the title screen, it won't go into the game, it goes back to the title screen!? (works in hoxs64, micro64 and vice but not in denise!?)

EDIT: It works in denise if disk is unprotected!

[PiCiJi]

Originally Posted by amilo3438

EDIT: It works in denise if disk is unprotected!

same in VICE

[amilo3438]

16) turrican_s1[rainbow_arts_1990](pal) ... Works only in denise and hoxs64!!! (weak bits/illegal GCR on many sectors) (vice and micro64 fail to load it)

Note: But the other side of the disc needs a new conversion from nib to work!

PS. Maybe you should check it on a real c64 & ultimate-II+ to see how accurate it is! (i.e. can it be trusted when testing)

EDIT: Some info from c64preservation.com...
It's Rainbow Arts earlier protection that is used on the original release of Turrican, and was also used on other Rainbow Arts, Magic Bytes, and Time Warp disks released 1987-88. (all PAL)
It contains an aggressive sync length check on track 36 that, if the correct length isn't found, formats the disk and resets the C64. On Turrican it is on track 37 and locks instead of formatting the disk.

PS.
So, many disks in existence that use this protection don't work in any emu, but... if they are re-converted from nib, using the latest nibconv with the -r option, there is a good chance that they will work.
Some examples: katakis/denaris and great giana sisters which worked only after the conversion, but there are still also those (albeit in the minority) that did not work even then.

Also, re-converted version of "16) turrican_s1[rainbow_arts_1990](pal)" will load in vice(sc) emu, but locks on track 37! (while works in denise and hoxs64)

[amilo3438]

17) jump_jet[anirog_1984] ... Another one that needs 1541 drive to work!
18) wild_west[ariolasoft_1985] ... same

Is there any reason why 1541-II is set default in the emu!? Is there something special that works on the 1541-II and not the 1541 drive!?

PS. Yes, I remember something was broken on 1541 kernal that was fixed in 1541-II! (maybe that's why 1541-II is default, but compatibility seems lower)

[amilo3438]

I found an interesting topic on Lemon64 forum (from 2011) which deals with the ways of archiving C64 discs.

There seem to be two ways to do this:

1) With the use of Nibtools and 1541 disk drives.
2) Kyroflux from the PC drive.

The first archiving method is the one used by C64PP (i.e. C64PreservationProject)

Another way is the one used by SPS (i.e. Software Preservation Society).

You can find out more about it on the Lemon64 forum: https://www.lemon64.com/forum/viewtopic.php?t=39740

Some quotes...

1) Nibtools

Originally Posted by r.cade@lemon64 forum

The best way to explain it is that a G64 (created from nibtools, anyway) contains what the 1541 sees on the disk, filtered through the drive head circuitry. Each byte is copied as it appears when the byte-ready signal is triggered.

Is that exactly what is "on the disk". No, but it's good enough because we only care about how the 1541 saw the data in most cases.

The disk only really contains magnetic flux transitions (or lack thereof) at certain rates. There are lower-level archival devices like the Kryoflux that capture this information directly. There are drawbacks to it, though, because it uses a PC disk drive, which cannot read the flip-side of disks without hardware hacks.

2) Kyroflux

Originally Posted by mr.vince@kyroflux forum

The problem of the G64 format is that it is sufficient for many, but not all titles. Second, all implementations so far are flawed. For some games I have three different G64 files to a) rewrite to disk b) use in Vice c) use in CCS64. This is not acceptable for a preservation effort.

As support for G64 could give the impression something was preserved, we initially decided against this. It would be more useful to introduce our format (IPF) to Vice and CCS64 as needed. We know that at least Vice team has sorrows because of the closed source nature of our library, so we have to work on a suitable license for source code release.

I might happen as an interim solution that G64 gets supported, but again this might send the completely wrong signal. G64 is not suitable for original bitstream preservation.

So I guess, around that time, they introduced their own new Extended G64 SPS file format, firstly introduced into VICE 2.4a SPS!

PS. Two disk images with a new file format can be find on their site: https://kryoflux.com/?page=download (find on the bottom: G64 Images)

What is interesting is that nibtools/nibscan.exe (using last version) can recognize this new format and reports it as: "Extended SPS G64 detected"! (so any emu can recognize this also)

What is also interesting is, when you re-convert a .nbz file into .g64 by nibtools/nibconv.exe (using last version) it doesn't convert it into this new extended format, or I don't see in option how to do that. So, after converting with nibtools, a g64 file stays in an old (not extended) format! (or this is not necessary for existing nib/nzb files)

Originally Posted by Retro-Nerd

Add Nib Support. Maybe somewhat useful for testing original preserved games but not converted to other formats. Micro64 has it.

It would be a good idea if an emu would work with a nib/nbz file as the disk drive head reads it (ie. sees it as byte), rather than first internally converting it to g64 and then reading it as such.


Does a NIB file contain more raw information than a G64 file?

Originally Posted by r.cade@lemon64 forum

NIB files contain very little more information than G64. They contain about 1.25 revolutions of each track, captured raw from the drive during rotation. This is done so I can find the overlap (track cycle) and create properly "aligned" G64 files. It also contains a flag for each track containing the scanned density information during the read.

There is no reason for emulators to support NIB files directly, as it is only a raw file processed by nibconv to prepare a working G64 or D64. If I later find the track was not processed properly when converted to G64/D64, I can modify nibconv/nibscan to "fix" it, and not have to re-read a new NIB file.

Is Kyroflux better than NIBtools?

Originally Posted by r.cade@lemon64 forum

You can consider it the same data at a "higher resolution". A stream file is a high-res scan of the disk, while NIB is low-res. Most of the time, low-res is enough, but it's always nice to go back to the original high-res source, if needed.

However, as you mentioned, currently there is no conversion for stream files to G64, IPF doesn't apply to anything except Amiga (or possibly Atari ST) disks since is based on a PC floppy chip core, and is closed-source. Technically the capture is better, but currently you can't do anything with it.

Unless that new extended sps format converts the stream file to g64!?

EDIT: Yes, more info about here... KryoFlux: enhanced G64 generation & new VICE patch --> https://www.lemon64.com/forum/viewtopic.php?p=517763

[amilo3438]

Originally Posted by PiCiJi

Correctly, an old G64 should no longer run. For the user, this is confusing and not understandable why some still work, and he gets the impression that it is a bug.

True, but can we also consider g64 files made with the new version of nibtools/nibconv as old g64 files too!? (because they don't use the new extended g64 SPS format)

(Apart from the two g64 files offered on the kyroflux pages, there is actually nothing else available in a new extended g64 format!)

[PiCiJi]

Originally Posted by amilo3438

Also, re-converted version of "16) turrican_s1[rainbow_arts_1990](pal)" will load in vice(sc) emu, but locks on track 37! (while works in denise and hoxs64)

I believe this is a bug in 1541 VIA emulation in VICE. evolution: PIA -> VIA -> CIA

Originally Posted by amilo3438

Is there any reason why 1541-II is set default in the emu!? Is there something special that works on the 1541-II and not the 1541 drive!?

1541-II is most common drive

---------------------
The NIB files don't show flux changes (what is really on disc) and are not sufficient for preservation.
only IPF, P64 have these kind of informations.

the NIB files contain GCR encoded data (=from disc controller decoded raw data)

[r.cade]

Originally Posted by amilo3438

True, but can we also consider g64 files made with the new version of nibtools/nibconv as old g64 files too!? (because they don't use the new extended g64 SPS format)

(Apart from the two g64 files offered on the kyroflux pages, there is actually nothing else available in a new extended g64 format!)

The differences in how nibconv generates G64 files now vs. the old way:

1) Tracks used to be limited to 7928 bytes, which while never in the specification, was a limitation in VICE. This is removed now since a long time ago.

2) Tracks are now shortened/truncated to a length that matches closely to the density of the track (also you can override this with -C). In the past, you could have (for example) a much longer track than could fit on a physical disk track. Because VICE now adjusts it's bitrate to the length of the track (maybe that's the best way they could think to improve it) these "old" G64's may not work properly.

3) It will automatically render G64's with fat tracks properly. VICE used to ignore half tracks and just present existing data from the previous track, and it longer does this.

As mentioned, you can re-convert to the new "format", even just using the existing G64 and convert in-place.

Still, there are limitations using a 1541 to read sync-lengths, so they will never be perfect. Protections that expect an accurate sync length may fail unless you adjust the image.

As far as I know, the only thing that SPS added was a extra block of data in the header that describes some internal information used by DTC to write the image to disk. VICE ignores it.

[amilo3438]

@r.cade

Thanks for the detailed feedback!

Now we learned something new, which is that we have 3 versions of G64 files:

1) old g64 format ... generated with older versions of nibconv
2) new g64 format ... generated with newer versions of nibconv
3) extended sps format ... generated by sps team tool

However, I would have some additional questions, so if it wouldn't be a problem to answer at least some...

What remains unknown is since when was that change introduced for the nibconv new g64 format!?
In fact, I'm currently testing some 2015 dated files so I'm not sure if it's the old format or the new format!
(i.e. would it be necessary to convert them, although most of the ones tested so far did not have any problems)

I'm also wondering, is there a way for say a user or an emulator to tell the difference between the old and the new format!?
(except maybe by the date of creation or testing whether it works on an old or a new version of vice)

Browsing the c64pp database, I come across files that are marked green (ie verified), so I wonder does this mean that such files have been tested/verified that they work on the original 1541 disk drive after writing to the diskette,
and can we consider such files valid enough for the verification of emulator testing!? (e.g. in the case that it does not work on the emulator and it is checked that it works on the 1541 disk)

What I notice that is missing in the database is the information about whether any of those verified files still needed to be "additionally processed" before it was written to the disk!?
(or for some reason it is left to the user to experiment on his own)

Is there any information about which files these are, and that they have been successfully converted to a diskette, that the emulators are still not able to run them!?
(this would be of interest when checking the accuracy of the emulator)

Last question...
Can nibwrite and nibconv be considered similar in function, except that the first one writes to a diskette and the second one creates a g64 file!?
(i.e. if a successful write to the diskette was created, then with the same procedure we get a successful g64 file ... and vice versa)

Thanks in advance for the answers!

[r.cade]

What remains unknown is since when was that change introduced for the nibconv new g64 format!?
In fact, I'm currently testing some 2015 dated files so I'm not sure if it's the old format or the new format!
(i.e. would it be necessary to convert them, although most of the ones tested so far did not have any problems)

nibtools changed the default when Vice 2.4 SPS version came out, roughly. I think that was 2012. The only "distribution" of any G64's before that was when "No-Intro" got a hold of the collection and renamed and released it, as well as GB64 releases from before then. GB64 also often ran "parameters" on the images when they wouldn't work on VICE at the time.

I'm also wondering, is there a way for say a user or an emulator to tell the difference between the old and the new format!?
(except maybe by the date of creation or testing whether it works on an old or a new version of vice)

No, because it's not a new format. It just rendered G64's within VICE's limitations at the time.

Browsing the c64pp database, I come across files that are marked green (ie verified), so I wonder does this mean that such files have been tested/verified that they work on the original 1541 disk drive after writing to the diskette,
and can we consider such files valid enough for the verification of emulator testing!? (e.g. in the case that it does not work on the emulator and it is checked that it works on the 1541 disk)

Verified only means it matched all sector data to another physical copy of the disk.

What I notice that is missing in the database is the information about whether any of those verified files still needed to be "additionally processed" before it was written to the disk!?
(or for some reason it is left to the user to experiment on his own)

Is there any information about which files these are, and that they have been successfully converted to a diskette, that the emulators are still not able to run them!?
(this would be of interest when checking the accuracy of the emulator)

There are notes on many of the entries that may mention this, but it's not 100% and things changed over 20 years.

Last question...
Can nibwrite and nibconv be considered similar in function, except that the first one writes to a diskette and the second one creates a g64 file!?
(i.e. if a successful write to the diskette was created, then with the same procedure we get a successful g64 file)

No, because the process of writing successfully to disk through the "window" of a 1541 is different than generating a G64. That plus all the nuances with emulators and disk emulation devices, there is no way to know this 100%.

[amilo3438]

Thanks for the quick reply!

This further clarified whether the g64 format could be suitable for reliable emulator testing.

After testing quite a few files and comparing them on different emulators, it can be concluded that it depends on the particular emulator.

On, for example, hoxs64, the tested falcon works without problems, while on others, denise and micro64, it is necessary to turn off deceleration emulation or mechanical emulation in order to work.
Start trekking ii (from other source) also works on hoxs64, but for the same on denise and micro64 it is necessary to turn on deceleration or mechanical emulation for it to work.

So it either works or it doesn't... the only thing that remains is the comparison that if something doesn't work on the denise (emu which I'm testing) but works on others, determine the reason why it's like that.
Although it remains incomprehensible why it sometimes works when the RPM is changed on say 305, while on another emulator eg hoxs64 it has no effect at all. (it's like hoxs64 is somehow adapting to the file it's reading)


EDIT:
So I wondered again why falcon works in hoxs64 and found the reason... hoxs64 uses 1541-C drive!

falcon_patrol[advantage_1983] ... in denise no need to turn off deceleration emulation if selected drive is 1541-C!!!

@PiCiJi - After this knowledge I'm not sure if Emulate Motor Deceleration (hack) is needed anymore!?

@r.cade - I wonder if it depends on the version of the disk drive from which a nib file was created! (ie does it have any effect)

-----------------

19) vampires_empire[digitek_19xx](weak) ... c64pp database says "protection fails in emulation", but it works in all 4 tested emus! (denise, vice(sc), hoxs64, micro64) (Note: To load use: LOAD":*",8,1)

[PiCiJi]

@r.cade thanks for infos

Originally Posted by @r.cade

No, because it's not a new format. It just rendered G64's within VICE's limitations at the time.

At that time, the GCR stream was simply transferred to the VIA using standard bitcell timing. In Vice 2.4 SPS, the drive mechanics (UF4, UE7, ..) were emulated.

Originally Posted by amilo3438

@PiCiJi - After this knowledge I'm not sure if Emulate Motor Deceleration (hack) is needed anymore!?

I assume you have it permanently disabled/unchecked ?

[amilo3438]

Originally Posted by PiCiJi

I assume you have it permanently disabled/unchecked ?

No, the entire test time it was enabled! It was only necessary to turn it off for one game, but since that game works even if it is enabled but with 1541-C drive, then nothing left for which it would need to be disabled.

So, everthing that work, work with the enabled motor deceleration!
And for other that doesn't work, disabled motor deceleration didn't help.
( and some of might work after re-converting it with nibconv tool ... what will test after )
So far I found very low % of those that doesn't work in comparison to those that work.