Emulation Benchmarking

[AmigaBoy]

I was just thinking about a feature for WinUAE which might prove to be useful - benchmarking.

For example:

winuae.exe -benchmark -whatever=a500.uae

This would launch the configuration a500.uae and emulate everything, except there will be no display, no sound, no interaction, etc. A suitable ADF would need to be selected (entirely up to the user), such as one that jumps straight into a game or demo, otherwise it would sit around waiting for some type of user interaction and skew results.
After a certain period (perhaps definable as another command line, such as -runseconds) or until the user explicitly tells it to stop, it stops and displays something like:

Execution Time: 371 seconds
Average FPS: 312

This type of benchmark would give insight into things like the speed of JIT, 32-bit vs 64-bit, the best compiler to use, fastest version of WinUAE (since benchmarking was implemented), etc.

Obviously over time, this feature could be expanded to cater for other things, such as:

Number of ILLEGAL instructions found: 13
Number of TRAP #0: 3
Number of TRAP #1: 1
...
Number of tracks loaded: 38
...

Anyone think this might be useful?

[Minuous]

No.

Why would you want to disable input and output? These are part of the emulation and the performance of those components is important.

>would sit around waiting for some type of user interaction and skew results.

It's still emulating while "sitting around" in any case.

This would launch the configuration a500.uae and emulate everything, except there will be no display, no sound, no interaction, etc. A suitable ADF would need to be selected (entirely up to the user), such as one that jumps straight into a game or demo, otherwise it would sit around waiting for some type of user interaction and skew results.
After a certain period (perhaps definable as another command line, such as -runseconds) or until the user explicitly tells it to stop, it stops and displays something like:

>Execution Time: 371 seconds
>Average FPS: 312

It might be useful for WinUAE to be able to optionally display this info at exit, if it doesn't already.

>this feature could be expanded to cater for other things

Generating this kind of profiling data that you suggest would add overhead and affect the results.

[Maren]

Originally Posted by Minuous

Why would you want to disable input and output? These are part of the emulation and the performance of those components is important

Perhaps to benchmark CPU emulation alone.

[AmigaBoy]

Originally Posted by Minuous

Why would you want to disable input and output? These are part of the emulation and the performance of those components is important.

I don't want the emulation of these components to be disabled, but I would like the ability for the user to interact with emulation to be disabled. Joysticks, mice and keyboard should all still be polled by the emulation, but the user won't be able to use any of them. As Maren said, benchmarking the CPU or chipset is the primarily concern since that's where everything related to emulation speed happens. If the emulation with no display or sound is giving you full speed results, but normal emulation doesn't, you know the overhead exists elsewhere, perhaps in the user's display (who might be using an intensive filter).

Imagine trying to benchmark a game that uses a really simple cracktro that does nothing more than display text on screen and wait for a left mouse click. The emulation gets to that point and sits there. It may as well be sitting at the Kickstart screen. At this point, the emulation isn't doing anything intensive because it's at an almost minimum. Compare that situation to one where it boots into an intensive megademo. The results are going to be completely different.

Originally Posted by Minuous

Generating this kind of profiling data that you suggest would add overhead and affect the results.

If you're a developer and want profiling data, you're not going to care about speed. Perhaps these results would be put into a seperate command line option.
Imagine a WHDLoad slave writer who wants to see if their slave works properly and disables all floppy access. This sort of data would be invaluable as they can quickly see that no tracks were read from any of the drives.
If the base benchmarking system is already in place, adding this sort of data is trivial, hence why I mentioned it.

[AmigaBoy]

Thinking about things, perhaps everything can be definable. You can say whether or not the display should be there, or whether interaction should be available, etc. The situation I was thinking of involved start to end with no user interaction for a definable period of time to give fairly consistent results on each run, but if someone wants to try playing a game to see those results, then yeah, why not.

Anyway, this is all a void conversation since it doesn't seem like there's much interest from anyone

[gilgamesh]

No, the idea is cool. How about a display of instructions per second for a start?

[Toni Wilen]

It isn't that simple. Pointless statistics are worse than no statistics at all.

For example: just counting CPU speed is pointless in Amiga emulation. I don't see it being "better and faster" if CPU runs too fast when it should have been locked out of bus due to blitter etc..

Instruction length varies, different memory can have different speed, CPU maybe in STOPped state -> Number of instructions per frame does not mean anything..

Benchmarking using "headless" emulation (display emulated but final step missing, sound emulated but no sound output etc..) + warp mode enabled would be less pointless but building standard environment isn't so easy..

and the biggest problem: too easy way to benchmark without standard evinronment and some clueless users start posting <censored> and comparing apples and oranges.

I guess few built-in tests (for example 3 different A500 configuration tests, plain bitplanes, cpu does most of the stuff, medium=some effects and high=very heavy dma activity or something like that) could be useful. It still does not happen because nobody is interested in writing this kind of code.

[AmigaBoy]

You're right that statistics will vary across similar specced machines. I know that most people don't really pay attention to stuff like memory speed, bus speed, etc. and just assume that the CPU is the only factor. My intention was to allow a user to benchmark their own machines, with a nice side effect being the ability to see how much things like filters are affecting performance. Simple stuff like that.

In terms of a standard environment, as you said, some standard configs are all that's needed. They could be embedded into UAE itself or downloadable somewhere. Just like the Quickstart configs, you make it clear that anyone that doesn't use the standard benchmarks will be ignored.

Originally Posted by Toni Wilen

It still does not happen because nobody is interested in writing this kind of code.

Exactly. This thread was created to pretty much sit here until one day someone thinks it might truly be useful. It's pretty much a novelty feature right now