CODEWITHSUNDEEP
iosswiftxcodeperformance-testingxctest
Declan McKenna
Declan McKenna

Reputation: 4870

How can I run my performance tests more than ten times?

By default Xcodes performance tests are run ten times and my result is the average of those ten tests. The problem is the averaged result varies considerably each time I run it so I have to run the test at least five times to get a converged result. This is both tedious and time consuming; is there a way to configure either XCode or the unit test itself to run more than ten times?

enter image description here

Upvotes: 27

Views: 3277

Answers (4)

kelin
kelin

Reputation: 11974

In the latest Xcode (11.0+) you don't need swizzling to change iterations count. Use the following function:

func measure(options: XCTMeasureOptions, block: () -> Void)

This will allow you to specify XCTMeasureOptions which has iterationCount property.

Interesting note from docs:

A performance test runs its block iterationCount+1 times, ignoring the first iteration and recording metrics for the remaining iterations. The test ignores the first iteration to reduce measurement variance associated with “warming up” caches and other first-run behavior.

Upvotes: 6

bshirley
bshirley

Reputation: 8357

I typically expand the Standard Deviation value to include the range I'd accept.

One obvious workaround for you is to perform the operation 5 times within the measure block you are executing. (You'll want to change the expected time.)

    measure {
        for _ in [1...5] {
            // ...
        }
    }

Upvotes: 1

JAL
JAL

Reputation: 42449

The short answer: No, there is no current interface exposed to allow a measure block more than ten times.

The longer answer: No, but there is an interface exposed to modify certain metrics of the measure block. The default metrics are in a String array returned from defaultPerformanceMetrics. There appears to only be one metric supported right now: XCTPerformanceMetric_WallClockTime. This only specifies a performance metric that records the time in seconds between calls to a performance test's startMeasuring() and stopMeasuring() methods, and not the number of times the block is run.

Upvotes: 6

Casey
Casey

Reputation: 6691

a class dump of XCTestCase exposes this method:

- (void)_recordValues:(id)arg1 forPerformanceMetricID:(id)arg2 name:(id)arg3 unitsOfMeasurement:(id)arg4 baselineName:(id)arg5 baselineAverage:(id)arg6 maxPercentRegression:(id)arg7 maxPercentRelativeStandardDeviation:(id)arg8 maxRegression:(id)arg9 maxStandardDeviation:(id)arg10 file:(id)arg11 line:(unsigned long long)arg12;

when this method is swizzled the first parameter (arg1) has the 10 durations:

["0.003544568",
"0.003456569",
"0.003198263",
"0.003257955",
"0.003508724",
"0.003454298",
"0.003461192",
"0.00423787",
"0.003359195",
"0.003335757"]

i added 4 new values (1.0, 2.0, 3.0, 4.0) to the end of this list before passing it back to the original implementation, but unfortunately a different class that observes, XCTestLog, has an internal sanity check that gets tripped:

Assertion failure in +[XCTestLog _messageForTest:didMeasureValues:forPerformanceMetricID:name:unitsOfMeasurement:baselineName:baselineAverage:maxPercentRegression:maxPercentRelativeStandardDeviation:maxRegression:maxStandardDeviation:file:line:]
caught "NSInternalInconsistencyException", "Performance Metrics must provide 10 measurements."

once the XCTestLog method is also overridden so it doesn't assert, the additional 4 values can be added without any complaints. unfortunately the view still only shows the 10 results.

it does however update the total time + standard deviation values in the mini view.

Before Swizzling

Before Swizzle

After Swizzling and adding 4 values After Swizzle

in order to view more than 10 results one would probably have to tweak the XCode runtime to tell the table to show more items.

Upvotes: 8

Related Questions