Skip to content

Commit c9ebb2b

Browse files
huydhnfacebook-github-bot
authored andcommitted
Add DeviceInfo in iOS benchmark run (#5410)
Summary: Given the way iOS benchmark app measures model load time, inference time, and memory usage using `measureWithMetrics` with `XCTClockMetric` and `XCTMemoryMetric`. I think the easiest way to gather the benchmark metric is to do it after the test finishes and parse the output. In the same spirit as #5332, this PR adds more information about the device so that it can be parsed later. I add the information into the test name, shoumikhin plz let me know if you know a better way to pass this information around. The output looks like this with the information in the test case name, i.e. `test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1` ``` Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started. /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 125171.731, relative standard deviation: 0.010%, values: [125158.624000, 125175.008000, 125175.008000, 125158.624000, 125191.392000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: -29.491, relative standard deviation: -228.792%, values: [-49.152000, -16.384000, 16.384000, 49.152000, -147.456000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.160, relative standard deviation: 3.460%, values: [0.163377, 0.165837, 0.164974, 0.152334, 0.154970], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (1.322 seconds). Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started. /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 127403.280, relative standard deviation: 0.000%, values: [127403.280000, 127403.280000, 127403.280000, 127403.280000, 127403.280000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: 0.000, relative standard deviation: 0.000%, values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.000, relative standard deviation: 41.029%, values: [0.000001, 0.000001, 0.000001, 0.000001, 0.000001], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (0.132 seconds). ``` Reviewed By: guangy10 Differential Revision: D62902327 Pulled By: huydhn
1 parent fdc7e45 commit c9ebb2b

File tree

2 files changed

+34
-4
lines changed

2 files changed

+34
-4
lines changed

extension/apple/Benchmark/Benchmark.xcodeproj/project.pbxproj

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
03ED6D152C8AAFFF00F2D6EE /* Metal.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = 03ED6D142C8AAFFF00F2D6EE /* Metal.framework */; };
2525
03ED6D172C8AB00500F2D6EE /* CoreML.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = 03ED6D162C8AB00500F2D6EE /* CoreML.framework */; };
2626
03ED6D192C8AB00A00F2D6EE /* Accelerate.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = 03ED6D182C8AB00A00F2D6EE /* Accelerate.framework */; };
27+
8493389C2C9918950071ABAD /* UIKit.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = 8493389B2C9918950071ABAD /* UIKit.framework */; };
2728
/* End PBXBuildFile section */
2829

2930
/* Begin PBXContainerItemProxy section */
@@ -59,6 +60,7 @@
5960
03ED6D142C8AAFFF00F2D6EE /* Metal.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = Metal.framework; path = Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.5.sdk/System/Library/Frameworks/Metal.framework; sourceTree = DEVELOPER_DIR; };
6061
03ED6D162C8AB00500F2D6EE /* CoreML.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = CoreML.framework; path = Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.5.sdk/System/Library/Frameworks/CoreML.framework; sourceTree = DEVELOPER_DIR; };
6162
03ED6D182C8AB00A00F2D6EE /* Accelerate.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = Accelerate.framework; path = Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.5.sdk/System/Library/Frameworks/Accelerate.framework; sourceTree = DEVELOPER_DIR; };
63+
8493389B2C9918950071ABAD /* UIKit.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = UIKit.framework; path = Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.5.sdk/System/Library/Frameworks/UIKit.framework; sourceTree = DEVELOPER_DIR; };
6264
/* End PBXFileReference section */
6365

6466
/* Begin PBXFrameworksBuildPhase section */
@@ -77,6 +79,7 @@
7779
03ED6D172C8AB00500F2D6EE /* CoreML.framework in Frameworks */,
7880
03ED6D152C8AAFFF00F2D6EE /* Metal.framework in Frameworks */,
7981
03ED6D132C8AAFF700F2D6EE /* MetalPerformanceShaders.framework in Frameworks */,
82+
8493389C2C9918950071ABAD /* UIKit.framework in Frameworks */,
8083
03ED6D112C8AAFF200F2D6EE /* MetalPerformanceShadersGraph.framework in Frameworks */,
8184
03ED6D0F2C8AAFE900F2D6EE /* libsqlite3.0.tbd in Frameworks */,
8285
03DD00A92C8FE44600FE4619 /* backend_coreml.xcframework in Frameworks */,
@@ -135,6 +138,7 @@
135138
03ED6CEB2C8AAF5300F2D6EE /* Frameworks */ = {
136139
isa = PBXGroup;
137140
children = (
141+
8493389B2C9918950071ABAD /* UIKit.framework */,
138142
03ED6D182C8AB00A00F2D6EE /* Accelerate.framework */,
139143
03ED6D162C8AB00500F2D6EE /* CoreML.framework */,
140144
03ED6D142C8AAFFF00F2D6EE /* Metal.framework */,

extension/apple/Benchmark/Tests/Tests.mm

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,9 @@
88

99
#import <XCTest/XCTest.h>
1010

11+
#import <UIKit/UIDevice.h>
1112
#import <objc/runtime.h>
13+
#import <sys/utsname.h>
1214

1315
#import <executorch/extension/module/module.h>
1416
#import <executorch/extension/tensor/tensor.h>
@@ -17,14 +19,32 @@
1719
using namespace ::executorch::runtime;
1820

1921
@interface Tests : XCTestCase
22+
2023
@end
2124

2225
@implementation Tests
2326

27+
+ (NSString *)getDeviceInfo {
28+
NSString *device = [[UIDevice currentDevice] model];
29+
30+
// i.e. iPhone15
31+
struct utsname systemInfo;
32+
uname(&systemInfo);
33+
NSString *name = [NSString stringWithCString:systemInfo.machine
34+
encoding:NSUTF8StringEncoding];
35+
36+
NSString *systemName = [[UIDevice currentDevice] systemName];
37+
NSString *systemVersion = [[UIDevice currentDevice] systemVersion];
38+
return [NSString
39+
stringWithFormat:@"%@_%@_%@_%@", device, name, systemName, systemVersion];
40+
}
41+
2442
+ (void)initialize {
2543
if (self != [self class]) {
2644
return;
2745
}
46+
NSString *deviceInfo = [Tests getDeviceInfo];
47+
2848
for (NSBundle *bundle in @[
2949
[NSBundle mainBundle],
3050
[NSBundle bundleForClass:[self class]],
@@ -51,8 +71,11 @@ + (void)initialize {
5171
NSString *modelName =
5272
modelPath.lastPathComponent.stringByDeletingPathExtension;
5373

54-
SEL testLoadSelector = NSSelectorFromString([NSString
55-
stringWithFormat:@"test_load_%@_%@", directoryName, modelName]);
74+
SEL testLoadSelector = NSSelectorFromString(
75+
[NSString stringWithFormat:@"test_load_%@_%@_%@",
76+
directoryName,
77+
modelName,
78+
deviceInfo]);
5679
IMP testLoadImplementation = imp_implementationWithBlock(^(id _self) {
5780
auto __block module = std::make_unique<Module>(modelPath.UTF8String);
5881
[_self measureWithMetrics:@[
@@ -68,8 +91,11 @@ + (void)initialize {
6891
class_addMethod(
6992
[self class], testLoadSelector, testLoadImplementation, "v@:");
7093

71-
SEL testForwardSelector = NSSelectorFromString([NSString
72-
stringWithFormat:@"test_forward_%@_%@", directoryName, modelName]);
94+
SEL testForwardSelector = NSSelectorFromString(
95+
[NSString stringWithFormat:@"test_forward_%@_%@_%@",
96+
directoryName,
97+
modelName,
98+
deviceInfo]);
7399
IMP testForwardImplementation = imp_implementationWithBlock(^(
74100
id _self) {
75101
auto __block module = std::make_unique<Module>(modelPath.UTF8String);

0 commit comments

Comments
 (0)