English Amiga Board


Go Back   English Amiga Board > Requests > request.UAE Wishlist

 
 
Thread Tools
Old 01 February 2010, 10:42   #1
AmigaBoy
Registered User

 
Join Date: Aug 2004
Location: 19 Jump Street
Posts: 232
Emulation Benchmarking

I was just thinking about a feature for WinUAE which might prove to be useful - benchmarking.

For example:

Quote:
winuae.exe -benchmark -whatever=a500.uae
This would launch the configuration a500.uae and emulate everything, except there will be no display, no sound, no interaction, etc. A suitable ADF would need to be selected (entirely up to the user), such as one that jumps straight into a game or demo, otherwise it would sit around waiting for some type of user interaction and skew results.
After a certain period (perhaps definable as another command line, such as -runseconds) or until the user explicitly tells it to stop, it stops and displays something like:

Quote:
Execution Time: 371 seconds
Average FPS: 312
This type of benchmark would give insight into things like the speed of JIT, 32-bit vs 64-bit, the best compiler to use, fastest version of WinUAE (since benchmarking was implemented), etc.

Obviously over time, this feature could be expanded to cater for other things, such as:

Quote:
Number of ILLEGAL instructions found: 13
Number of TRAP #0: 3
Number of TRAP #1: 1
...
Number of tracks loaded: 38
...
Anyone think this might be useful?
AmigaBoy is offline  
Old 01 February 2010, 20:41   #2
Minuous
Coder/webmaster/gamer
Minuous's Avatar
 
Join Date: Oct 2001
Location: Canberra/Australia
Posts: 2,156
No.

Why would you want to disable input and output? These are part of the emulation and the performance of those components is important.

>would sit around waiting for some type of user interaction and skew results.

It's still emulating while "sitting around" in any case.

This would launch the configuration a500.uae and emulate everything, except there will be no display, no sound, no interaction, etc. A suitable ADF would need to be selected (entirely up to the user), such as one that jumps straight into a game or demo, otherwise it would sit around waiting for some type of user interaction and skew results.
After a certain period (perhaps definable as another command line, such as -runseconds) or until the user explicitly tells it to stop, it stops and displays something like:

>Execution Time: 371 seconds
>Average FPS: 312

It might be useful for WinUAE to be able to optionally display this info at exit, if it doesn't already.

>this feature could be expanded to cater for other things

Generating this kind of profiling data that you suggest would add overhead and affect the results.
Minuous is offline  
Old 01 February 2010, 22:59   #3
Maren
Banned
 
Join Date: Jul 2009
Location: *
Posts: 567
Quote:
Originally Posted by Minuous View Post
Why would you want to disable input and output? These are part of the emulation and the performance of those components is important
Perhaps to benchmark CPU emulation alone.
Maren is offline  
Old 02 February 2010, 00:12   #4
AmigaBoy
Registered User

 
Join Date: Aug 2004
Location: 19 Jump Street
Posts: 232
Quote:
Originally Posted by Minuous
Why would you want to disable input and output? These are part of the emulation and the performance of those components is important.
I don't want the emulation of these components to be disabled, but I would like the ability for the user to interact with emulation to be disabled. Joysticks, mice and keyboard should all still be polled by the emulation, but the user won't be able to use any of them. As Maren said, benchmarking the CPU or chipset is the primarily concern since that's where everything related to emulation speed happens. If the emulation with no display or sound is giving you full speed results, but normal emulation doesn't, you know the overhead exists elsewhere, perhaps in the user's display (who might be using an intensive filter).

Imagine trying to benchmark a game that uses a really simple cracktro that does nothing more than display text on screen and wait for a left mouse click. The emulation gets to that point and sits there. It may as well be sitting at the Kickstart screen. At this point, the emulation isn't doing anything intensive because it's at an almost minimum. Compare that situation to one where it boots into an intensive megademo. The results are going to be completely different.

Quote:
Originally Posted by Minuous
Generating this kind of profiling data that you suggest would add overhead and affect the results.
If you're a developer and want profiling data, you're not going to care about speed. Perhaps these results would be put into a seperate command line option.
Imagine a WHDLoad slave writer who wants to see if their slave works properly and disables all floppy access. This sort of data would be invaluable as they can quickly see that no tracks were read from any of the drives.
If the base benchmarking system is already in place, adding this sort of data is trivial, hence why I mentioned it.
AmigaBoy is offline  
Old 02 February 2010, 00:55   #5
AmigaBoy
Registered User

 
Join Date: Aug 2004
Location: 19 Jump Street
Posts: 232
Thinking about things, perhaps everything can be definable. You can say whether or not the display should be there, or whether interaction should be available, etc. The situation I was thinking of involved start to end with no user interaction for a definable period of time to give fairly consistent results on each run, but if someone wants to try playing a game to see those results, then yeah, why not.

Anyway, this is all a void conversation since it doesn't seem like there's much interest from anyone
AmigaBoy is offline  
Old 02 February 2010, 01:12   #6
gilgamesh
Linux snob

gilgamesh's Avatar
 
Join Date: Sep 2008
Location: Monkey Island
Posts: 997
No, the idea is cool. How about a display of instructions per second for a start?
gilgamesh is offline  
Old 02 February 2010, 09:14   #7
Toni Wilen
WinUAE developer
 
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 45
Posts: 24,057
It isn't that simple. Pointless statistics are worse than no statistics at all.

For example: just counting CPU speed is pointless in Amiga emulation. I don't see it being "better and faster" if CPU runs too fast when it should have been locked out of bus due to blitter etc..

Instruction length varies, different memory can have different speed, CPU maybe in STOPped state -> Number of instructions per frame does not mean anything..

Benchmarking using "headless" emulation (display emulated but final step missing, sound emulated but no sound output etc..) + warp mode enabled would be less pointless but building standard environment isn't so easy..

and the biggest problem: too easy way to benchmark without standard evinronment and some clueless users start posting <censored> and comparing apples and oranges.

I guess few built-in tests (for example 3 different A500 configuration tests, plain bitplanes, cpu does most of the stuff, medium=some effects and high=very heavy dma activity or something like that) could be useful. It still does not happen because nobody is interested in writing this kind of code.
Toni Wilen is online now  
Old 03 February 2010, 23:09   #8
AmigaBoy
Registered User

 
Join Date: Aug 2004
Location: 19 Jump Street
Posts: 232
You're right that statistics will vary across similar specced machines. I know that most people don't really pay attention to stuff like memory speed, bus speed, etc. and just assume that the CPU is the only factor. My intention was to allow a user to benchmark their own machines, with a nice side effect being the ability to see how much things like filters are affecting performance. Simple stuff like that.

In terms of a standard environment, as you said, some standard configs are all that's needed. They could be embedded into UAE itself or downloadable somewhere. Just like the Quickstart configs, you make it clear that anyone that doesn't use the standard benchmarks will be ignored.

Quote:
Originally Posted by Toni Wilen
It still does not happen because nobody is interested in writing this kind of code.
Exactly. This thread was created to pretty much sit here until one day someone thinks it might truly be useful. It's pretty much a novelty feature right now
AmigaBoy is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
PC Emulation DyLucke support.Apps 6 15 November 2009 05:39
Emulation....why do we do it? Jim Nostalgia & memories 35 02 April 2004 08:28
PS emulation Djay Retrogaming General Discussion 26 17 December 2003 20:34
Emulation punkie New to Emulation or Amiga scene 4 11 May 2002 01:16
New to emulation, need some help a500 support.WinUAE 1 26 April 2002 01:30

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 16:59.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.
Page generated in 0.11285 seconds with 15 queries