Add performance measurement test with instructions count

[kimdv]

As we talked about, it would be nice to have performance tests as part of the CI.

I did dig a bit into XCTCPUMetric.
But I didn't find a nice way to add baselines.

So I tried to make my own measure method so we easily could start adding them in the code

• Run CI 3 times

[ahoppen]

But I didn't find a nice way to add baselines.

AFAIK that’s a known issue that we can’t define baselines when running XCTest using swift test, so we need to define our own, like you did. I have one suggestion though: Define the baselines in a JSON file. By default that JSON file could be Tests/PerformanceTest/baselines.json and we can add the baseline to .gitignore so that everybody can define their own baselines locally. You could then be able to override the baseline using an environment variable and CI would have Tests/PerformanceTest/ci-baselines.json. If the key doesn’t exist in the baseline, the test would fail to make sure we don’t add new performance tests without adding a baseline to ci-baselines.json.

The baseline file could then look as follows:

{
  "testNativeParsingPerformance": 5850081534,
  ...
}

and measureInstructions could infer the test case name from the test function as follows:

func measureInstructions(_ baselineName: StaticString = #function)

[kimdv]

@ahoppen did you think of something like this?

I'm not sure I understand this

You could then be able to override the baseline using an environment variable ...

[kimdv]

@swift-ci please test

[ahoppen]

Very nice.

One thought that just crossed my mind: I think it’s a little bit frustrating if all performance tests fail if no baseline file is provided because this means that if you check out the repo and run tests initially, you’ll see test failures. Because of this, what do you think about changing it so that if PerformanceTest/baselines.json doesn’t exist, all performance tests pass. Once you do care about performance, you’ll create a baselines.json and all performance tests for which you don’t have baselines will start failing, which might make sense.

[kimdv]

@swift-ci please test

[kimdv]

@swift-ci please test

[kimdv]

@swift-ci please test

[kimdv]

@swift-ci please test

[kimdv]

@swift-ci please test

[ahoppen]

Nice 🤩

[kimdv]

@swift-ci please test

Baseline

{
   "testNativeParsingPerformance": 311011168,
   "testClassifierPerformance": 2624475912,
   "testEmptyVisitorPerformance": 48502831,
   "testEmptyRewriterPerformance": 54438776,
   "testEmptyAnyVisitorPerformance": 49123765
}

Failed
XCTAssertTrue failed - Number of instructions '307024328'
is not within range 309456112-312566223

1.29017% difference

[kimdv]

@swift-ci please test windows

[kimdv]

@swift-ci please test

[ahoppen]

Looks like we need to increase the allowed deviation to 1%.

[kimdv]

@swift-ci please test

[ahoppen]

Oh, we should only specify the baseline if we are running on macOS. getInstructionCount doesn’t work on Linux and also, even if it did, we would have different performance counters. You can check if we’re running on macOS using

if platform.system() == "Darwin":

[kimdv]

@swift-ci please test

[ahoppen]

2/3

@swift-ci please test

[kimdv]

3/3

@swift-ci please test