UAE CPU core based 680x0 tester/validator

[Toni Wilen]

UAE CPU Tester

http://www.winuae.net/files/b/cputester.7z (Windows generator executable + Amiga m68k tester executable. Sources in WinUAE github)

I finally wrote utility (This was my "Summer 2019" project) that can be used to verify operation of for example software emulated or FPGA 680x0 CPUs.
It is based on UAE CPU core (gencpu generated special test core). All the CPU logic comes from UAE CPU core.

[More detailed information in WinUAE forum: http://eab.abime.net/showthread.php?t=98746]

Verifies:

- All CPU registers (D0-D7/A0-A7, PC and SR/CCR)
- All FPU registers (FP0-FP7, FPIAR, FPCR, FPSR)
- Generated exception and stack frame contents (if any)
- Memory writes, including stack modifications (if any)
- Loop mode for JIT testing. (generates <test instruction>, dbf dn,loop)
- Supports 68000, 68020, 68030 (only difference between 020 and 030 seems to be data cache and MMU), 68040 and 68060. (I don't currently have real 68010)

Tests executed for each tested instruction:

- Every CCR combination (32 tests)
- Every FPU condition combination (4 bits) + 2 precision bits + 2 rounding bits (256 tests)
- Every addressing mode, including optionally 68020+ addressing modes.
- If instruction generated privilege violation exception, extra test round is run in supervisor mode (Total 64 tests).
- Optionally can do any combination of T0, T1, S and M -bit SR register extra test rounds.
- Every opcode value is tested. Total number of tests per opcode depends on available addressing modes etc. It can be hundreds of thousands or even millions..

Test generation details:

Instruction's effective address is randomized. It is accepted if it points to any of 3 test memory regions. If it points outside of test memory, it will be re-randomized few times. Test will be skipped if current EA makes it impossible to point to any of 3 test regions.
If 68000/68010 and address error testing is enabled: 2 extra test rounds are generated, one with even and another with odd EAs to test and verify address errors.

Notes and limitations:

- Test generator is very brute force based, it should be more intelligent..
- Address error testing is optional, if disabled, generated tests never cause address errors.
- RTE test only tests stack frame types 0 and 2 (if 68020+)
- All tests that would halt or reset the CPU are skipped (RESET in supervisor mode, STOP parameter that would stop the CPU etc)
- Single instruction test set will take long time to run on real 68000. Few minutes to much longer...
- Undefined flags (for example DIV and CHK) are also verified. It probably would be good idea to optionally filter them out.
- Instruction cycle order or timing is ignored. It is not possible without extra hardware.
- 68000 bus errors are not tested but there are some plans for future version with custom hardware. (Hatari also uses UAE CPU core)
- FPU testing is not yet fully implemented.
- Sometimes reported old and new condition code state does not match error report..

Tester compatibility (integer instructions only):

68000: Complete. (Bus errors are not yet verified)
68010: Not supported yet (Don't have real 68010, at least not yet).
68020: Almost complete (DIVS.W/DIVS.L V-flag weirdness).
68030: Same as 68020.
68040: Almost complete (Weird unaligned MOVE16 behavior which may be board specific).
68060: Same as 68040.

Not implemented or only partially implemented:

68010+:

- MOVEC: Only MOVEC to An/Dn is tested and read value is ignored. Basically only verifies if correct and only correct CPU model specific control registers exist.
- RTE: long frames with undefined fields are skipped. Basic frame types 0 and 2 are verified, also unsupported frame types are tested, error is reported if CPU does not generate frame exception.
- 68020+ undefined addressing mode bit combinations are not tested.

All models:

- Interrupts (stack frames and STOP)
- MMU instructions (Not going to happen)
- 68020+ cache related instructions.
- FPU FSAVE/FRESTORE, FPU support also isn't fully implemented yet.

Build instructions:

- buildm68k first (already built if UAE core was previously compiled)
- gencpu with CPU_TEST=1 define. This creates cpuemu_x_test.cpp files (x=90-94), cpustbl_test.cpp and cputbl_test.h
- build cputestgen project.
- build native Amiga project (cputest directory). Assembly files probably only compiles with Bebbo's GCC.


Test generator quick instructions:

Update cputestgen.ini to match your CPU model, memory settings etc...

"Low memory" = memory accessible using absolute word addressing mode, positive value (0x0000 to 0x7fff). Can be larger.
"High memory" = memory accessible using absolute word addressing mode, negative value (0xFFF8000 to 0xFFFFFFFF)

If high memory is ROM space (like on 24-bit address space Amigas), memory region is used for read only tests, use "high_rom=<path to rom image>" to select ROM image. Last 32k of image is loaded.

Use "test_low_memory_start"/"test_high_memory_start" and "test_low_memory_end"/"test_high_memory_end" to restrict range of memory region used for tests, for example if part of region is normally inaccessible.

"test_memory_start"/"test_memory_size" is the main test memory, tested instruction and stack is located here. Must be at least 128k but larger the size, the easier it is for the generator to find effective addresses that hit test memory. This memory space must be free on test target m68k hardware.

All 3 memory regions (if RAM) are filled with pseudo-random pattern and saved as "lmem.dat", "hmem.dat" and "tmem.dat"

Usage of Amiga m68k native test program:

Copy all memory dat files, test executable compiled for target platform (currently only Amiga is supported) and data/<cpu model> contents to target system, keeping original directory structure.

cputest all = run all tests, in alphabetical order. Stops when mismatch is detected.
cputest tst.b = run tst.b tests only
cputest all tst.b = run tst.b, then tst.w and so on in alphabetical order until end or mismatch is detected.

If mismatch is detected, opcode word(s), instruction disassembly, registers before and after and reason message is shown on screen. If difference is in exception stack frame, both expected and returned stack frame is shown in hexadecimal.

Example output:

CHK2.L:
data/68020/chk2.l/0000.dat. 0...

0: 04d0 0800    chk2.l   (a0),d0
SR: expected 0000 -> 0009 but got 0001 (ffff)
Registers before:
D0: 00000010 D1: 00000000 D2: ffffffff D3: ffffff00
D4: ffff0000 D5: 80008080 D6: 7fff7fff D7: aaaaaaaa
A0: 00000000 A1: 00000080 A2: 00008000 A3: 00007fff
A4: fffffffe A5: ffffff00 A6: 0883ff00 A7: 08877ff8
SR:*0000   PC: 08840000 ISP: 0887ff80 MSP: 08880000
T1=0 T0=0 M=0 S=0 X=0 N!0 Z=0 V=0 C*0
Registers after:
D0: 00000010 D1: 00000000 D2: ffffffff D3: ffffff00
D4: ffff0000 D5: 80008080 D6: 7fff7fff D7: aaaaaaaa
A0: 00000000 A1: 00000080 A2: 00008000 A3: 00007fff
A4: fffffffe A5: ffffff00 A6: 0883ff00 A7: 08877ff8
SR:*0001   PC: 08840000 ISP: 0887ff80 MSP: 08880000
T1=0 T0=0 M=0 S=0 X=0 N!0 Z=0 V=0 C*1
OK: Generated exception 6

[supaduper]

Fantastic work Toni

And I just donated to you as well, for I would be lost without Winuae, and I hope other Amigans throw some funds your way too,as its well deserved

Supaduper

[DamienD]

Originally Posted by supaduper

And I just donated to you as well, for I would be lost without Winuae, and I hope other Amigans throw some funds your way too,as its well deserved

Yup, I too would be absolutely lost without WinUAE so try to donate to Toni as often as I can

[ceztko]

Originally Posted by Toni Wilen

UAE CPU Tester

I finally wrote utility (This was my "Summer 2019" project) that can be used to verify operation of for example software emulated or FPGA 680x0 CPUs.

Cool! Can you put link of this thread on the currently locked WinUAE beta changelog thread?

[Toni Wilen]

Done. And thanks.

I know this has very limited use so I don't really expect anything but perhaps at least someone could test if first post "weird stuff" can be confirmed

[gururise]

Would be interesting to see the results from the Amiga emulation in Mister FPGA

[rare_j]

68010: Not supported yet (Don't have real 68010, at least not yet).

I have a 68010 to donate if you want it. Let me know if interested.

[Toni Wilen]

Originally Posted by gururise

Would be interesting to see the results from the Amiga emulation in Mister FPGA

Run test generator and run the generated tests on that hardware. cputestgen.ini defaults should work, if ROM is A500 3.1 and system has 8M of Z2 Fast RAM. Generated test files are not included because they take almost few hundred megabytes for basic full 68000 test set.

It is almost guaranteed address error tests are not correct because it is totally undefined behavior in 68000 and probably no program cares.. (Some old copy protections do but they mainly care about stacked PC, not flags)

Originally Posted by rare_j

I have a 68010 to donate if you want it. Let me know if interested.

I am already getting 68010 soon.

[apolkosnik]

Originally Posted by Toni Wilen

Run test generator and run the generated tests on that hardware. cputestgen.ini defaults should work, if ROM is A500 3.1 and system has 8M of Z2 Fast RAM. Generated test files are not included because they take almost few hundred megabytes for basic full 68000 test set.

It is almost guaranteed address error tests are not correct because it is totally undefined behavior in 68000 and probably no program cares.. (Some old copy protections do but they mainly care about stacked PC, not flags)



I am already getting 68010 soon.

This is really good stuff, thank you for creating it. I've started using it to test my attempts at implementing a couple of missing instructions on MiSTer FPGA.

[turrican3]

it's hard stuff !!!
winuae 4.3.0 won't be a piece of cake too !!!
thanks toni !!!

[Toni Wilen]

Important changes since last release:

- Fast test mode (CCR=0, CCR=31 only)
- Improved MOVEC test (read/write all ones, all zeros, verify, restore.
- SR/flag state was sometimes incorrect in error output (Visual bug only)
- More detailed error output. Depending on instruction type, source address, destination address and/or stack memory contents are listed. Contents before and after if instruction modified data. Only output "after" registers if they changed.
- Atari ST support (thanks Nicolas)

[asymetrix]

This project is very useful. Great work ! If I may, maybe add an MD5 check to a specific group of ROMS, to keep consistency in testing.

Its exciting - does this mean we can have a open source PCIE i7 CPU card emulating a 68060 in 64bit ASM ?

[Toni Wilen]

Originally Posted by asymetrix

This project is very useful. Great work ! If I may, maybe add an MD5 check to a specific group of ROMS, to keep consistency in testing.

I can CRC32 of the ROM image used during generating which gets printed when tester runs but I am not going to add any checksum check code to tester because it should not be too slow even on slow 7MHz 68000 hardware.

Its exciting - does this mean we can have a open source PCIE i7 CPU card emulating a 68060 in 64bit ASM ?

What does this have to do with tester?

It is an automatic tester, it can only make compatibility testing faster and less boring than manual testing. Nothing more.

It won't make developing actual emulator any easier.

[apolkosnik]

Toni, thank you again for this great tool! I think that its brute-force approach is an important feature, perhaps leaving it as an option would make sense, at least for people that might want to try to do some fuzzing on their new implementations.

Your cputest tool helped me to get some bugs related to unsupported addressing modes (for instructions using register bits in the second word) out of the TG68K core on MiSTer. Now, the mystery behind the texture corruption in the Gouraud Pulse in Nexus 7 demo has been solved! :-)

[Toni Wilen]

Originally Posted by apolkosnik

Toni, thank you again for this great tool! I think that its brute-force approach is an important feature, perhaps leaving it as an option would make sense, at least for people that might want to try to do some fuzzing on their new implementations.

I am not going to remove it, as you said, it can reveal weird undocumented cases by creating bit combinations that no one is supposed to use.. (Note that currently some invalid combinations are never generated because some of them confuse the CPU, some parts of calculations use "hidden" address register, some ignore the bit and use matching data register..)

"target ea mode" is the alternative, non-random one. It only generates EAs listed in ini (one or more, comma separated). Mainly meant for testing address and bus errors.

Your cputest tool helped me to get some bugs related to unsupported addressing modes (for instructions using register bits in the second word) out of the TG68K core on MiSTer. Now, the mystery behind the texture corruption in the Gouraud Pulse in Nexus 7 demo has been solved! :-)

Which instruction? I am quite sure UAE didn't support those undocumented A/D field instructions and the demo did work fine.

[apolkosnik]

Originally Posted by Toni Wilen

I am not going to remove it, as you said, it can reveal weird undocumented cases by creating bit combinations that no one is supposed to use.. (Note that currently some invalid combinations are never generated because some of them confuse the CPU, some parts of calculations use "hidden" address register, some ignore the bit and use matching data register..)

"target ea mode" is the alternative, non-random one. It only generates EAs listed in ini (one or more, comma separated). Mainly meant for testing address and bus errors.



Which instruction? I am quite sure UAE didn't support those undocumented A/D field instructions and the demo did work fine.

The patch link is below, I'll list them when I get home.
https://github.com/MiSTer-devel/Minimig-AGA_MiSTer/pull/23/files

[apolkosnik]

For the Nexus7 demo, I think that some loop was getting bfins instruction tripped up to generate bad output, I think. I've even found it to be the case on the Alpha release of Vampire AGA core. (http://www.apollo-core.com/knowledge...21853&z=1RQMVi). lol.

Some concrete samples:

Bitfield instructions:

bfchg eac0 31fe ... should be 01fe since bits 15-12 of the second word should be 0. No dest in this opcode.
bfclr ecc2 ec68 ... should be 0c68 since bits 15-12 of the second word should be 0. No dest in this opcode.

bfset eec0 8fc4 ... should be 0fc4 since bits 15-12 of the second word should be 0. No dest in this opcode.
bftst e8c3 3a9a ... should be 0a9a since bits 15-12 of the second word should be 0. No dest in this opcode.

bfexts ebc0 f094 ... should be 7094 since bit 15 of the second word should be 0 - makes it write into An.
bfextu e9c0 116b ... bfextu d0{#5:d3},d1 ... if Dw(bit 5)=1 bits 4,3 should be 0. Also, if Do(bit 11)=1 bits 10,9 should be 0.
bfffo edc0 d771 ... should be 5771 since bit 15 of the second word should be 0 - makes it write into An.

bfins efc0 7ee1 ... if Do(bit 11)=1 bits 10,9 should be 0. Also, if Dw(bit 5)=1 bits 4,3 should be 0.

Division and multiplication:
divul.l d0,d6:d0 - 4c40 000e ... Result written into A6 rather than D6.
mulu.l d0,d1:d0 - 4c00 05e9 ... A1: modified 00000080 -> 00000000 but expected no modifications

eorsr/orsr/etc:
8194: 0a7c 0800 eori.w #$800,sr -- In this case the bit12 of SR is supposed to be 0 and is not supposed to be used for anything at least on 68000-68030.


I'm not sure about callm/cas/cas2/chk/chk2 as these are not really implemented on TG68K

[grond]

Originally Posted by apolkosnik

For the Nexus7 demo, I think that some loop was getting bfins instruction tripped up to generate bad output, I think. I've even found it to be the case on the Alpha release of Vampire AGA core.

The Vampire AGA core you are mentioning is from 2017. The 080 bitfield instructions were fixed long ago. If you look at the V4, you'll see that there is no problem.

[apolkosnik]

Originally Posted by grond

The Vampire AGA core you are mentioning is from 2017. The 080 bitfield instructions were fixed long ago. If you look at the V4, you'll see that there is no problem.

I could technically check the 2.11 or 2.12 core on my Vampire 500.

[apolkosnik]

I must be doing something wrong, but I get from cputestgen.exe the following:
...
data/68020/MOVE.L
Loop mode didn't end!?

and it just stops right there.