Better Living Through Benchmarks
If there’s one thing that computer geeks and enthusiasts care about, it’s computer performance. The easiest way to measure computer performance? Benchmarks! Benchmarks provide a quick and (hopefully!) standard way of comparing performance between two (or more) computers.
One problem with existing benchmarks, though, is that while there are a lot of benchmarks available for Windows, Xbench is really the only benchmark available for Mac OS X. While Xbench is an interesting benchmark, we’ve noticed some limitations with it (such as SMP support).
Another problem is that there aren’t any general cross-platform benchmarks, making it difficult for the average user to compare performance between Windows and Mac OS X, and between different processor architectures.
We’ve decided to address these problems by writing our own benchmark: Geekbench.
Geekbench is a cross-platform benchmark that’s currently under development. It’s going to run on both Mac OS X and Windows (with the possibility of running on other operating systems, such as Linux, at a later date). Geekbench is also going to be SMP-aware (where possible), so the advantages of an SMP system are more clearly highlighted.
What we’d like to know, though, is what sort of tests you (our readers and users) would like to see in Geekbench? Is there any particular test (or kind of test) you’d like to see included? If there is, drop us a note at geekbench at geekpatrol dot ca.
Needless to say we’ll keep you informed as Geekbench progresses. Keep your eyes peeled for a preview of Geekbench in November!
bench mark applications should be simple and yet informative .
most apps are ineffective in the same.
it would be nice to have a app with different sections.
Section A: Processing power (includes all tests to evaluate the architecture but one single reading with a neat aqua graph representing base line comparison.
Section B: Graphics Power: give approx. values as to how many fps one would get if he/she were to run a specific game. i.e. bench values should bee converted to fps values which most people are more familiar with.
Section c: Application power. showing graphical representation of approximate performance expected when the system s subjected to photoshop or office suite tests.
The critical thing is that you show results with and without special extensions being used, such as SSE, SSE2, SSE3, and especially AltiVec (G4 and G5 uses it). This will probably require assembly programing to do correctly. I know that Macs see 50-100% performance gains using AltiVec for AAC encoding, for example.
A crash test computer dummy!!!
A test mesuring the difficulty (or not) to crash a operating system… Like how manys apps or docs or process can run before the sys crash… for example!
I’ve always found raw benchmarks to be rather uninformative. Ok, so if you only ever run one app at a time and turn off all your normal networking and other services, fine, you might get performance close to what you expect, but this is unrealistic. I’m most interested in how a system performs under a proper, multitasking load. How do the processor, system bus and hard drive cope with the system’s virtual memory needs? How well do the graphics sub-systems cope with multiple apps with multiple windows moving around (e.g. how fast can you pull a window to the front of a stack). How many apps can you run together without system performance degrading – e.g. before video playback is effected or there are problems with usb-based audio outputs (system can’t keep up with streaming audio on the usb channels). How much memory is sensible in a system – when does more stop giving a noticeable performance gain. How does software raid effect the system.
These are just some of the real, practical and useful bits of info that will change as both an OS is upgraded and as you run it on different hardware.
Much more useful than peak drive throughput or max fps on a game. These days, most desktops, at least, can throw around enough power to do what most people want. Move above raw numbers and get stats on what using it actually feels like when doing normal stuff
I think you need the option to use SSE/Altivec (if available) and while a “graphical” view showing the “total” performance of one section would be good, it also needs to be able to show actual numbers… and what about something like linpack that shows terraflops?
I’m curious how you are going to get it to run on different platforms. Will each version be optimized for each type of processor and each type of operating system? A Mac app “ported” to Windows does not perform as well on Windows and vise versa. Also a Mac app written for a G3 or G4 may run a on G5, but it could run even better if it was written specifically for the G5. The same is true for the AMD 64 processors. How are you going to get around this without giving one operating system or processor an unfair advantage?
there isn’t really any way to make sure each platform is equal, because they’re not. the best way is probably to have many different tests run, then have a final value based of an average of all those tests?
but it’s tricky stuff… glad that i’m not a dev making benchmarking software.
There is already a cross-platform CPU/memory benchmark. The source was originally published by BYTE magazine I think. Perhaps this could be incoporated with permission? If it were also updated to use smp and the extended instruction sets, it could demonstrate the improvements possible.
http://www.tux.org/~mayer/linux/bmark.html
Just make sure it’s dual processor aware, please. Also, actual times, megabytes, kilobytes and flops, versus some numeric scale.
Make sure the source is open so people may critize the way the tests are done and if the comparisions are fair.
I would like to see networking tests run on similar machines using industry standard switches etc. How well does the networking system perform under certain constraints?
Make a benchmark that gives the option of running ALL the tests simultaneously. So there is video load while processor load while memory load while disk load… That way, you can see overall system performance – including things like bus speed, thruput, system saturation, multitasking, all that sort of stuff…
Benchmark results should contain at least te opportunity to visualise resuls as bar graph or plain numbers.
Provide support to run on Linux, Solaris, and *BSD as well as MS Windows and Mac OS X.
And run on PPC, x86, Sparc architectures.