determining a better architectural improvement, given spec ratios

Question

I am working through a problem and I just want to make sure my work is correct, so if someone could critique me, it would be appreciated.

The question:

There are two architectural improvements which are being considered (A1 & A2). You have been handed a table which shows the speedups relative to a base system for each of 5 benchmarks (B1-B5); these speedups are the spec ratios to the reference computer.

A1: B1 = 1.5, B2 = 4, B3 = 3, B4 = 2, B5 = 5

A2: B1= 3, B2 = 1.5, B3 = 4, B4 = 4, B5 = 3

Given the above information, which improvement should be chosen?

My answer is as follows:

Using the baseline as a reference, the geometric mean of the spec ratio of the improvements (A1/A2) can be used to determine which is better.

The spec ratios are as follows:

S1 = 1.5/3

S2 = 4/1.5

S3 = 3/4

S4 = 2/4

S5 = 5/3

The geometric mean would be the following: (S1*S2*S3*S4*S5)^(1/5) which is roughly equivalent to 0.96.

Thus, A1 is roughly only 96% as good as A2 and A2 should be chosen.

Once again, I am just looking to see if my thinking is correct here. Any help is appreciated.

EDIT1: Using Mackie's methodology

Taking the base speed for the bench marks, it will be the following for A0:

B1=B2=B3=B4=B5=1

Thus, the geometric mean of the spec ratios of the Base to A1 (A0/A1) will be:

((1/1.5) * (1/4) * (1/3) * (1/2) * (1/5)) ^ (1/5) = 0.35

And the geometric mean of the spec ratios of the Base to A2 (A0/A2) will be:

((1/3) * (1/1.5) * (1/4) * (1/4) * (1/3)) ^ (1/5) = 0.34

However, comparing here (Base/A1) to (Base/A2), you get 0.35/0.34 = 1.03 meaning that A1 is roughly 3% slower than A2? Is my math off here or did I choose the wrong Benchmarks for A0? My initial method seems simpler and what the initial question was asking...

Answer 1

Given the above information, which improvement should be chosen?

This is an interesting question, however I feel that the information given are not sufficient. For one thing, what is more important for the user of the system - does he want to minimize the total time it takes to perform a batch of tasks? or maybe he cares for how much time the longest task takes in the batch and it can't be too high (realtime systems?)? or he cares about the shortest task time in the batch - because he's competing with others to solve it first, and only the first solution is rewarded (competitive Mersenne primes search)?

Additionally, if I supplement the information above with the following new information:

Benchmarks B1,B3-B5 represent tasks that happen just 0.01% of the time in production
Benchmark B2 represents the 99.99% of tasks

Then A1 is over twice as good as A2 when optimising the average case. It's not immediately clear which improvement is better!

EDIT: brainstorming - if B1-B5 represent average tasks and the differences in run times are to be treated as random, maybe it is the improvement with the least standard deviation the best to choose?

Answer 2

I would suggest to calculate the geometrical mean for each of the two architectural improvements, and then compare the resulting two numbers. This way you can compare each of them also to the base system (which would have a score of 1.0). Of course, when you divide the two numbers, you will get the same result as above (A1 is 0.96 times slower than A2).

Doing it this way allows you to characterize the performance of each machine with a single number. Which is the point of the exercise. If you had a lot of machines to compare, your method would be much harder...