CODEWITHSUNDEEP
objective-cxcodeunit-testingperformance-testingxctest
Hlung
Hlung

Reputation: 14338

Why does the first run of "XCTestCase -measureBlock:" takes so much time?

I'm using Xcode 7. iPhone 5s simulator (iOS 9.0).

I'm trying to use XCTestCase -measureBlock: to measure time spent on some of my functions. This runs the code inside the block 10 times and report the results how long each run takes, the average, STDEV, etc.

It turns out that the time of first run is always very high, like 400-500% higher. Same happens for a very simple method, or even nothing in the measure block. I don't have anything in my - (void)setUp or - (void)tearDown.

Result of NSString -stringWithFormat:. See the spike at the first bar of the graph. enter image description here values: [0.000031, 0.000005, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003]

Result of a blank block. Similar results. enter image description here values: [0.000007, 0.000001, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000001]

Is this a bug or something? What is going on?

Upvotes: 7

Views: 1343

Answers (3)

MrMage
MrMage

Reputation: 7496

It is possible that the actual reason is that Xcode needs to symbolicate your binary upon the first call of measure, presumably in order to detect which test you are in.

At least that's what this post on the Swift forum suggests; the underlying slowdown would probably also apply to ObjC projects.

Upvotes: 0

top.dev
top.dev

Reputation: 457

The measured block is executed ten times and the test output shows the average execution time as well as individual run times and standard deviation (http://nshipster.com/xctestcase/) So first time it can populate cache and prepare data if your code does it, on rest next steps it might work with cached data and execution time can be shorter as well.

Upvotes: 2

Nikolai Ruhe
Nikolai Ruhe

Reputation: 81878

It's probably an artifact of some caches warming up.

The Objective-C runtime maintains a selector cache for each class, for example. This means that the first call to each method is somewhat slower than the following calls.

You could just put the contents of the measured block right before the call to measureBlock: to test if this is true. So the warm up run would just not be measured, then.

Upvotes: 3

Related Questions