You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Summary:
Given the way iOS benchmark app measures model load time, inference time, and memory usage using `measureWithMetrics` with `XCTClockMetric` and `XCTMemoryMetric`. I think the easiest way to gather the benchmark metric is to do it after the test finishes and parse the output.
In the same spirit as #5332, this PR adds more information about the device so that it can be parsed later. I add the information into the test name, shoumikhin plz let me know if you know a better way to pass this information around.
The output looks like this with the information in the test case name, i.e. `test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1`
```
Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started.
/Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 125171.731, relative standard deviation: 0.010%, values: [125158.624000, 125175.008000, 125175.008000, 125158.624000, 125191.392000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
/Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: -29.491, relative standard deviation: -228.792%, values: [-49.152000, -16.384000, 16.384000, 49.152000, -147.456000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
/Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.160, relative standard deviation: 3.460%, values: [0.163377, 0.165837, 0.164974, 0.152334, 0.154970], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (1.322 seconds).
Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started.
/Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 127403.280, relative standard deviation: 0.000%, values: [127403.280000, 127403.280000, 127403.280000, 127403.280000, 127403.280000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
/Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: 0.000, relative standard deviation: 0.000%, values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
/Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.000, relative standard deviation: 41.029%, values: [0.000001, 0.000001, 0.000001, 0.000001, 0.000001], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (0.132 seconds).
```
Pull Request resolved: #5410
Reviewed By: guangy10
Differential Revision: D62902327
Pulled By: huydhn
fbshipit-source-id: 9b3de5601ae8625ee6db1d63e43accb5c60ec33b
0 commit comments