Is P96Speed a reliable benchmark?

[codepoet]

I was playing with some changes to the P96 code in WinUAE and trying to evaluate performance differences. I *think* I managed to speed up BitBlt by about 3-5% for the optimal case (srccpy).

However, the deltas from run-to-run, both before and after, are pretty sizable. 750-780K ops at 1280x1024 32bit, and then 800-830K ops with the change.

I'm not sure I trust the tool, or perhaps it is because I'm using JIT and introducing variance that way? Although I'd expect the JIT to be fairly consistent as well, since I'm not running a ton of other apps and JIT cache shouldn't be getting swamped.

Toni, if you're lurking, would appreciate your thoughts.

[rutra80]

Wild speed swings disappear in exclusive fullscreen mode on my system.

[Toni Wilen]

Originally Posted by codepoet

I was playing with some changes to the P96 code in WinUAE and trying to evaluate performance differences. I *think* I managed to speed up BitBlt by about 3-5% for the optimal case (srccpy).

SRCPY is a plain memory copy, there is not much left for optimizations.

However, the deltas from run-to-run, both before and after, are pretty sizable. 750-780K ops at 1280x1024 32bit, and then 800-830K ops with the change.

I'm not sure I trust the tool, or perhaps it is because I'm using JIT and introducing variance that way? Although I'd expect the JIT to be fairly consistent as well, since I'm not running a ton of other apps and JIT cache shouldn't be getting swamped.

Toni, if you're lurking, would appreciate your thoughts.

No benchmark can be fully reliable due to way fastest possible CPU mode works (approximately sync CPU with raster time and if there is still time left when frame has been completed, keep executing CPU instructions until remaining real time frame time has been used).

I don't recommend JIT if benchmarking something else than pure CPU.

Originally Posted by rutra80

Wild speed swings disappear in exclusive fullscreen mode on my system.

This could also be side-effect of some other speed limit (like vsync)

[codepoet]

Originally Posted by Toni Wilen

SRCPY is a plain memory copy, there is not much left for optimizations.

Yup, I'm aware of that, but there are faster memcpy around than what the compiler will inline and/or what the C-runtime provides, including memcpy that take better advantage of AVX etc.

Originally Posted by Toni Wilen

I don't recommend JIT if benchmarking something else than pure CPU.

Noted, i'll disable JIT and see if i get more consistent results, and can better measure. Alternatively, i'll just author some local benchmarks that run and can produce results, and use that to justify my approach

[Toni Wilen]

Originally Posted by codepoet

Yup, I'm aware of that, but there are faster memcpy around than what the compiler will inline and/or what the C-runtime provides, including memcpy that take better advantage of AVX etc.

Ok. Makes sense, as long it is not slower when doing relatively small copies. Optimization for the worst case is not always the best idea

[codepoet]

Originally Posted by Toni Wilen

Ok. Makes sense, as long it is not slower when doing relatively small copies. Optimization for the worst case is not always the best idea

Yup, needs to be data driven, not just for the fun of it. Although the P96 bench is doing a variety of blits, and a variety of rect-fills, so hopefully it kinda averages it out

Still, doesn't tell you any heuristics what the typical operations are, so i may need to build in some basic "benchmarking" to the P96 code to provide data on what's happening in normal typical usage scenarios, and not synthetic benchmarks

[Mad-Matt]

SysSpeed has very similar tests to P96Speed and may or may not offer more comparative results.

[pixie]

Is there a common test that could be used on AROS/AmigaOS?

[rutra80]

SysSpeed has nice graphics & intuition tests each running for 5 secs, but other tests run a given number of operations and in non-CE JIT mode finish almost immediately.

P96Speed works in both AROS & AmigaOS, I have problems running SysSpeed in AROS.

AROS seems to have some memory leaks in intuition so these tests often fail...