CODEWITHSUNDEEP
cachingx86-64pipelinesimdbranch-prediction
user45891
user45891

Reputation: 840

Writing small benchmark tests

I'm buying a new PC. That's great, but I'd like to know how much faster it is.
Now I could use an already established benchmark - BUT I want to learn WHY it is faster.

So my idea was to:
1. write small benchmarks to test very specific stuff
2. compile twice with -march=native
3. profile and compare

Now I just need to come up with benchmarks:
- memcpy(), memset() & memchr() in the GB range to test RAM speed
- vector (or matrix) operations to test SIMD instructions
- bruteforcing a square root to test general instruction speed and pipeline
- simple arithmetic (x[i]++ or something) to test cache - fill an array with a pattern of 2,3 and 4 different values.
For each different value (one test with if/else, one with switch) do something small.
That should test the branch predictor.

Have I missed a feature? Can any one those benchmarks be simplified? Are there other/better tests for a CPU feature I thought to have covered?

Upvotes: 0

Views: 133

Answers (1)

dune.rocks
dune.rocks

Reputation: 517

Now I just need to come up with benchmarks:

"just" :-). Writing benchmarks for people to compare computers is actually a profitable industry. It's FAR from trivial. Even the professionals write benchmarks that are silly. For some examples of commerical providers, search for SPEC, Geekbench, EEMBC... They cost a lot of money.

Even these benchmarks, which have been written by supposed experts, are frequently ridiculous at giving sensible results you can communicate as an engineer (marketers do better). Modern machines (I avoid the term "CPU", because that's just such small part of this, you have memories, interconnects, disks, drivers, operating systems, ...) are really really hard to reason about on a micro-optimization level like you're suggesting. You might get some intuituion about your two computers with those simple tests, but it will be far from good data.

The best I can recommend is to run a compiler benchmark suite. LLVM has a test-suite project, in which there are various benchmarks you can run to get some idea of how much better your new computer is. I'm sure GCC will have something similar, I'm just not familiar with it. Instructions for running test-suite are here.

Even that test-suite, which is much larger than what you'll be able to create yourself, needs to be taken with large chunk of statistical salt. Performance comparisons are full of surprises, and it's so easy to get it wrong.

Now, answering why computer A is faster (or slower) than B on these particular benchmarks can range from trivial to impossible. Trivial cases can be things like more registers or more cache (and a program that uses cache sensibly), and impossible can be some change to the foobaddybar that a manufacturer won't publish the details of, and you'll just be left guessing.

So, sorry, but the days are gone where such "obvious" tests will tell you much of anything interesting about modern machines. You need to employ pretty serious stats and infrastructure to get semi-reasonable results.

Upvotes: 3

Related Questions