Skip to content

Commit eb62572

Browse files
arpan14arpan14
authored
chore: adding a benchmark utility in client. (#3068)
* The existing setup using JMH produces flaky results at time due to thread safe scope variables and also it lacks the ability to pass different parameters from command-line. * This is a more light-weight setup which allows to pass params from command-line and keeps the setup much simpler. * This util also allows us to export metrics/traces directly to cloud monitoring so that its easy to visualize and compare stats.
1 parent e444ffc commit eb62572

File tree

7 files changed

+709
-0
lines changed

7 files changed

+709
-0
lines changed

‎benchmarks/README.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Latency Tests
2+
3+
This directory contains a utility to compare latencies of different public methods supported by the Java Client Library.
4+
The tests use simple statements that operate on a single row at a time.
5+
6+
The goal is that addition of new features should not add any latency.
7+
8+
## Setup Test Database
9+
10+
All tests in this directory use a database with a single table. Follow these steps to create a
11+
database that you can use for these tests:
12+
13+
1. Set up some environment variables. These will be used by all following steps.
14+
```shell
15+
export SPANNER_CLIENT_BENCHMARK_GOOGLE_CLOUD_PROJECT=my-project
16+
export SPANNER_CLIENT_BENCHMARK_SPANNER_INSTANCE=my-instance
17+
export SPANNER_CLIENT_BENCHMARK_SPANNER_DATABASE=my-database
18+
```
19+
20+
2. Create the Cloud Spanner database if it does not already exist using Pantheon UI.
21+
3. Create the test table in the Cloud Spanner database. The tests assume the below DDL for the table.
22+
23+
```shell
24+
CREATE TABLE FOO ( id INT64 NOT NULL, BAZ INT64, BAR INT64, ) PRIMARY KEY(id);
25+
```
26+
4. Generate some random test data that can be used for the benchmarking. This can be done
27+
by using the script `bulkInsertTestData` in class `BenchmarkingUtilityScripts` to bulk
28+
load 1000000 rows into this table. A large table makes sure that the queries are well
29+
randomised and there is no hot-spotting.
30+
31+
## Running
32+
33+
The benchmark application includes Java Client as a dependency. Modify the dependency
34+
version in `pom.xml` file if you wish to benchmark a different version of Java Client.
35+
36+
37+
* The below command uses only 1 thread and 1000 operations. So the total load would
38+
be 1000 read operations. The test also uses multiplexed sessions.
39+
```shell
40+
mvn clean compile exec:java -Dexec.args="--clients=1 --operations=1000 --multiplexed=true"
41+
```
42+
43+
* The below command uses 10 threads, so at any point in time there would be roughly
44+
10 concurrent requests. The total load of the benchmark would be 50000 read operations.
45+
```shell
46+
mvn clean compile exec:java -Dexec.args="--clients=10 --operations=5000 --multiplexed=true"
47+
```
48+
49+
* To run the same test without multiplexed sessions avoid passing `multiplexed` flag. This will
50+
make sure that tests uses regular sessions.
51+
```shell
52+
mvn clean compile exec:java -Dexec.args="--clients=10 --operations=5000"
53+
```

‎benchmarks/pom.xml

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
<!--
2+
Copyright 2023 Google LLC
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
Unless required by applicable law or agreed to in writing, software
8+
distributed under the License is distributed on an "AS IS" BASIS,
9+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
10+
See the License for the specific language governing permissions and
11+
limitations under the License.
12+
-->
13+
14+
<project xmlns="http://maven.apache.org/POM/4.0.0"
15+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
16+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
17+
18+
<modelVersion>4.0.0</modelVersion>
19+
<groupId>com.google.cloud</groupId>
20+
<artifactId>google-cloud-spanner-benchmark</artifactId>
21+
<packaging>jar</packaging>
22+
<name>Google Cloud Spanner Benchmark</name>
23+
24+
<parent>
25+
<groupId>com.google.cloud</groupId>
26+
<artifactId>google-cloud-spanner-parent</artifactId>
27+
<version>6.65.1-SNAPSHOT</version><!-- {x-version-update:google-cloud-spanner:current} -->
28+
</parent>
29+
30+
<properties>
31+
<java.version>1.8</java.version>
32+
<maven.compiler.source>1.8</maven.compiler.source>
33+
<maven.compiler.target>1.8</maven.compiler.target>
34+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
35+
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
36+
<junixsocket.version>2.9.1</junixsocket.version>
37+
<opentelemetry.version>1.36.0</opentelemetry.version>
38+
</properties>
39+
40+
<dependencies>
41+
<dependency>
42+
<groupId>io.opentelemetry</groupId>
43+
<artifactId>opentelemetry-api</artifactId>
44+
</dependency>
45+
<dependency>
46+
<groupId>io.opentelemetry</groupId>
47+
<artifactId>opentelemetry-context</artifactId>
48+
</dependency>
49+
<dependency>
50+
<groupId>com.google.cloud.opentelemetry</groupId>
51+
<artifactId>exporter-trace</artifactId>
52+
<version>0.25.2</version>
53+
</dependency>
54+
<dependency>
55+
<groupId>com.google.cloud.opentelemetry</groupId>
56+
<artifactId>exporter-metrics</artifactId>
57+
<version>0.25.2</version>
58+
</dependency>
59+
<!-- OpenTelemetry test dependencies -->
60+
<dependency>
61+
<groupId>io.opentelemetry</groupId>
62+
<artifactId>opentelemetry-sdk</artifactId>
63+
<version>${opentelemetry.version}</version>
64+
</dependency>
65+
<dependency>
66+
<groupId>io.opentelemetry</groupId>
67+
<artifactId>opentelemetry-sdk-metrics</artifactId>
68+
<version>${opentelemetry.version}</version>
69+
</dependency>
70+
<dependency>
71+
<groupId>io.opentelemetry</groupId>
72+
<artifactId>opentelemetry-sdk-trace</artifactId>
73+
<version>${opentelemetry.version}</version>
74+
</dependency>
75+
<dependency>
76+
<groupId>io.opentelemetry</groupId>
77+
<artifactId>opentelemetry-sdk-testing</artifactId>
78+
<version>${opentelemetry.version}</version>
79+
</dependency>
80+
<dependency>
81+
<groupId>com.google.re2j</groupId>
82+
<artifactId>re2j</artifactId>
83+
<version>1.7</version>
84+
</dependency>
85+
<dependency>
86+
<groupId>io.opentelemetry</groupId>
87+
<artifactId>opentelemetry-bom</artifactId>
88+
<version>1.37.0</version>
89+
<type>pom</type>
90+
<scope>import</scope>
91+
</dependency>
92+
<dependency>
93+
<groupId>com.google.cloud</groupId>
94+
<artifactId>google-cloud-spanner</artifactId>
95+
<version>6.63.0</version>
96+
</dependency>
97+
<dependency>
98+
<groupId>commons-cli</groupId>
99+
<artifactId>commons-cli</artifactId>
100+
<version>1.6.0</version>
101+
</dependency>
102+
<dependency>
103+
<groupId>com.google.auto.value</groupId>
104+
<artifactId>auto-value-annotations</artifactId>
105+
<version>1.10.4</version>
106+
</dependency>
107+
<dependency>
108+
<groupId>com.kohlschutter.junixsocket</groupId>
109+
<artifactId>junixsocket-core</artifactId>
110+
<version>${junixsocket.version}</version>
111+
<type>pom</type>
112+
</dependency>
113+
<dependency>
114+
<groupId>com.kohlschutter.junixsocket</groupId>
115+
<artifactId>junixsocket-common</artifactId>
116+
<version>${junixsocket.version}</version>
117+
</dependency>
118+
<dependency>
119+
<groupId>commons-cli</groupId>
120+
<artifactId>commons-cli</artifactId>
121+
<version>1.7.0</version>
122+
</dependency>
123+
124+
<dependency>
125+
<groupId>junit</groupId>
126+
<artifactId>junit</artifactId>
127+
<version>4.13.2</version>
128+
<scope>test</scope>
129+
</dependency>
130+
</dependencies>
131+
<build>
132+
<plugins>
133+
<plugin>
134+
<groupId>org.codehaus.mojo</groupId>
135+
<artifactId>exec-maven-plugin</artifactId>
136+
<version>3.2.0</version>
137+
<configuration>
138+
<mainClass>com.google.cloud.spanner.benchmark.LatencyBenchmark</mainClass>
139+
<cleanupDaemonThreads>false</cleanupDaemonThreads>
140+
</configuration>
141+
</plugin>
142+
<plugin>
143+
<groupId>com.coveo</groupId>
144+
<artifactId>fmt-maven-plugin</artifactId>
145+
<executions>
146+
<execution>
147+
<goals>
148+
<goal>format</goal>
149+
</goals>
150+
</execution>
151+
</executions>
152+
</plugin>
153+
</plugins>
154+
</build>
155+
</project>

‎benchmarks/src/main/java/com/google/cloud/spanner/SessionPoolOptionsHelper.java

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
/*
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package com.google.cloud.spanner;
18+
19+
import com.google.api.core.InternalApi;
20+
21+
/**
22+
* Simple helper class to get access to a package-private method in the {@link
23+
* com.google.cloud.spanner.SessionPoolOptions}.
24+
*/
25+
@InternalApi
26+
public class SessionPoolOptionsHelper {
27+
28+
// TODO: Remove when Builder.setUseMultiplexedSession(..) has been made public.
29+
public static SessionPoolOptions.Builder setUseMultiplexedSession(
30+
SessionPoolOptions.Builder sessionPoolOptionsBuilder, boolean useMultiplexedSession) {
31+
return sessionPoolOptionsBuilder.setUseMultiplexedSession(useMultiplexedSession);
32+
}
33+
}

‎benchmarks/src/main/java/com/google/cloud/spanner/benchmark/AbstractRunner.java

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
/*
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package com.google.cloud.spanner.benchmark;
18+
19+
import java.nio.charset.StandardCharsets;
20+
import java.time.Duration;
21+
import java.util.ArrayList;
22+
import java.util.List;
23+
import java.util.concurrent.ExecutorService;
24+
import java.util.concurrent.Future;
25+
import java.util.concurrent.ThreadLocalRandom;
26+
import java.util.concurrent.TimeUnit;
27+
import java.util.concurrent.TimeoutException;
28+
import java.util.concurrent.atomic.AtomicInteger;
29+
30+
abstract class AbstractRunner implements BenchmarkRunner {
31+
static final int TOTAL_RECORDS = 1000000;
32+
static final String SELECT_QUERY = "SELECT ID FROM FOO WHERE ID = @id";
33+
static final String UPDATE_QUERY = "UPDATE FOO SET BAR=1 WHERE ID = @id";
34+
static final String ID_COLUMN_NAME = "id";
35+
static final String SERVER_URL = "https://staging-wrenchworks.sandbox.googleapis.com";
36+
37+
private final AtomicInteger operationCounter = new AtomicInteger();
38+
39+
protected void incOperations() {
40+
operationCounter.incrementAndGet();
41+
}
42+
43+
protected List<Duration> collectResults(
44+
ExecutorService service,
45+
List<Future<List<Duration>>> results,
46+
int numClients,
47+
int numOperations)
48+
throws Exception {
49+
int totalOperations = numClients * numOperations;
50+
service.shutdown();
51+
while (!service.isTerminated()) {
52+
//noinspection BusyWait
53+
Thread.sleep(1000L);
54+
System.out.printf("\r%d/%d", operationCounter.get(), totalOperations);
55+
}
56+
System.out.println();
57+
if (!service.awaitTermination(60L, TimeUnit.MINUTES)) {
58+
throw new TimeoutException();
59+
}
60+
List<Duration> allResults = new ArrayList<>(numClients * numOperations);
61+
for (Future<List<Duration>> result : results) {
62+
allResults.addAll(result.get());
63+
}
64+
return allResults;
65+
}
66+
67+
protected void randomWait(int waitMillis) throws InterruptedException {
68+
if (waitMillis <= 0) {
69+
return;
70+
}
71+
int randomMillis = ThreadLocalRandom.current().nextInt(waitMillis * 2);
72+
Thread.sleep(randomMillis);
73+
}
74+
75+
protected String generateRandomString() {
76+
byte[] bytes = new byte[64];
77+
ThreadLocalRandom.current().nextBytes(bytes);
78+
return new String(bytes, StandardCharsets.UTF_8);
79+
}
80+
}

‎benchmarks/src/main/java/com/google/cloud/spanner/benchmark/BenchmarkRunner.java

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
/*
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package com.google.cloud.spanner.benchmark;
18+
19+
import java.time.Duration;
20+
import java.util.List;
21+
22+
public interface BenchmarkRunner {
23+
enum TransactionType {
24+
READ_ONLY_SINGLE_USE,
25+
READ_WRITE
26+
}
27+
28+
List<Duration> execute(
29+
TransactionType transactionType,
30+
int numClients,
31+
int numOperations,
32+
int waitMillis,
33+
boolean useMultiplexedSession);
34+
}

0 commit comments

Comments
 (0)