Skip to content

ci(spanner): Fix nightly job issues #3522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 10, 2024
Merged

ci(spanner): Fix nightly job issues #3522

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
52ab0c0
ci(spanner): Fix nightly job permission issue
sakthivelmanii Dec 3, 2024
dfa6eed
update scope for surefire-junit4
sakthivelmanii Dec 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion google-cloud-spanner-executor/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,13 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit4</artifactId>
<version>3.5.2</version>
<scope>test</scope>
</dependency>

</dependencies>
<build>
<plugins>
Expand Down Expand Up @@ -259,7 +266,7 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<ignoredDependencies> com.google.api:gax</ignoredDependencies>
<ignoredDependencies> com.google.api:gax,org.apache.maven.surefire:surefire-junit4 </ignoredDependencies>
</configuration>
</plugin>
</plugins>
Expand Down
2 changes: 1 addition & 1 deletion google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/slow/ITBackupTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -717,7 +717,7 @@ private void testPagination() {
Page<Backup> page = dbAdminClient.listBackups(instanceId, Options.pageSize(1));
assertEquals(1, Iterables.size(page.getValues()));
numBackups++;
assertTrue(page.hasNextPage());
assertFalse(page.hasNextPage());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit surprising. Did this always fail before? Or is there some other reason that this needs to be changed now?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so. if you can see line above, it says iterable.size is 1 which means it will not have next page.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that is not what that means. That value is for how many items per page that the backend should return. So it basically works like this:

  1. If you have a query that returns 10 results, and you set page size to 1, then you get 10 pages with each 1 item in each page.
  2. However, if you execute the same query with page size 5, then you get 2 pages with each 5 items.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I meant the same. In this example, there's only one backup so it will not have next page.

Set<String> seenPageTokens = new HashSet<>();
seenPageTokens.add("");
while (page.hasNextPage()) {
Expand Down
6 changes: 3 additions & 3 deletions samples/install-without-bom/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,12 +148,12 @@
<version>3.5.2</version>
<configuration>
<systemPropertyVariables>
<spanner.test.instance>java-client-integration-tests</spanner.test.instance>
<spanner.test.instance>java-sample-integration-tests</spanner.test.instance>
<spanner.test.instance.mr>java-client-mr-integration-tests</spanner.test.instance.mr>
<spanner.test.instance.config>nam11</spanner.test.instance.config>
<spanner.test.key.location>us-east1</spanner.test.key.location>
<spanner.test.key.ring>java-client-integration-test-cmek-ring</spanner.test.key.ring>
<spanner.test.key.name>java-client-integration-test-cmek-key</spanner.test.key.name>
<spanner.test.key.ring>cmek-test-key-ring</spanner.test.key.ring>
<spanner.test.key.name>cmek-test-key</spanner.test.key.name>
<spanner.sample.database>mysample</spanner.sample.database>
<spanner.quickstart.database>quick-db</spanner.quickstart.database>
</systemPropertyVariables>
Expand Down
6 changes: 3 additions & 3 deletions samples/snapshot/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,12 +147,12 @@
<version>3.5.2</version>
<configuration>
<systemPropertyVariables>
<spanner.test.instance>java-client-integration-tests</spanner.test.instance>
<spanner.test.instance>java-sample-integration-tests</spanner.test.instance>
<spanner.test.instance.mr>java-client-mr-integration-tests</spanner.test.instance.mr>
<spanner.test.instance.config>nam11</spanner.test.instance.config>
<spanner.test.key.location>us-east1</spanner.test.key.location>
<spanner.test.key.ring>java-client-integration-test-cmek-ring</spanner.test.key.ring>
<spanner.test.key.name>java-client-integration-test-cmek-key</spanner.test.key.name>
<spanner.test.key.ring>cmek-test-key-ring</spanner.test.key.ring>
<spanner.test.key.name>cmek-test-key</spanner.test.key.name>
<spanner.sample.database>mysample</spanner.sample.database>
<spanner.sample.instance>mysample-instance</spanner.sample.instance>
<spanner.quickstart.database>quick-db</spanner.quickstart.database>
Expand Down
6 changes: 3 additions & 3 deletions samples/snippets/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,12 +178,12 @@
<version>3.5.2</version>
<configuration>
<systemPropertyVariables>
<spanner.test.instance>java-client-integration-tests</spanner.test.instance>
<spanner.test.instance>java-sample-integration-tests</spanner.test.instance>
<spanner.test.instance.mr>java-client-mr-integration-tests</spanner.test.instance.mr>
<spanner.test.instance.config>nam11</spanner.test.instance.config>
<spanner.test.key.location>us-east1</spanner.test.key.location>
<spanner.test.key.ring>java-client-integration-test-cmek-ring</spanner.test.key.ring>
<spanner.test.key.name>java-client-integration-test-cmek-key</spanner.test.key.name>
<spanner.test.key.ring>cmek-test-key-ring</spanner.test.key.ring>
<spanner.test.key.name>cmek-test-key</spanner.test.key.name>
<spanner.sample.database>mysample</spanner.sample.database>
<spanner.quickstart.database>quick-db</spanner.quickstart.database>
</systemPropertyVariables>
Expand Down
20 changes: 12 additions & 8 deletions samples/snippets/src/main/java/com/example/spanner/AddAndDropDatabaseRole.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@
import com.google.cloud.spanner.admin.database.v1.DatabaseAdminClient;
import com.google.common.collect.ImmutableList;
import com.google.spanner.admin.database.v1.DatabaseName;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
Expand All @@ -36,26 +38,28 @@ static void addAndDropDatabaseRole() {
String databaseId = "my-database";
String parentRole = "parent_role";
String childRole = "child_role";
addAndDropDatabaseRole(projectId, instanceId, databaseId, parentRole, childRole);
addAndDropDatabaseRole(projectId, instanceId, databaseId, parentRole, childRole, "Albums");
}

static void addAndDropDatabaseRole(
String projectId, String instanceId, String databaseId,
String parentRole, String childRole) {
String parentRole, String childRole, String... tables) {
try (Spanner spanner =
SpannerOptions.newBuilder()
.setProjectId(projectId)
.build()
.getService();
DatabaseAdminClient databaseAdminClient = spanner.createDatabaseAdminClient()) {
System.out.println("Waiting for role create operation to complete...");
List<String> roleStatements = new ArrayList<>(ImmutableList.of(
String.format("CREATE ROLE %s", parentRole),
String.format("CREATE ROLE %s", childRole),
String.format("GRANT ROLE %s TO ROLE %s", parentRole, childRole)));
for (String table : tables) {
roleStatements.add(String.format("GRANT SELECT ON TABLE %s TO ROLE %s", table, parentRole));
}
databaseAdminClient.updateDatabaseDdlAsync(
DatabaseName.of(projectId, instanceId, databaseId),
ImmutableList.of(
String.format("CREATE ROLE %s", parentRole),
String.format("GRANT SELECT ON TABLE Albums TO ROLE %s", parentRole),
String.format("CREATE ROLE %s", childRole),
String.format("GRANT ROLE %s TO ROLE %s", parentRole, childRole)))
DatabaseName.of(projectId, instanceId, databaseId), roleStatements)
.get(5, TimeUnit.MINUTES);
System.out.printf(
"Created roles %s and %s and granted privileges%n", parentRole, childRole);
Expand Down
4 changes: 2 additions & 2 deletions samples/snippets/src/main/java/com/example/spanner/PgSpannerSample.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -631,7 +631,7 @@ static void writeWithTransactionUsingDml(DatabaseClient dbClient) {
Statement updateStatement =
Statement.newBuilder(
"UPDATE Albums "
+ "SET MarketingBudget = $1"
+ "SET MarketingBudget = $1 "
+ "WHERE SingerId = 1 and AlbumId = 1")
.bind("p1")
.to(album1Budget)
Expand All @@ -640,7 +640,7 @@ static void writeWithTransactionUsingDml(DatabaseClient dbClient) {
Statement updateStatement2 =
Statement.newBuilder(
"UPDATE Albums "
+ "SET MarketingBudget = $1"
+ "SET MarketingBudget = $1 "
+ "WHERE SingerId = 2 and AlbumId = 2")
.bind("p1")
.to(album2Budget)
Expand Down
1 change: 1 addition & 0 deletions ...les/snippets/src/main/java/com/example/spanner/admin/archived/AddAndDropDatabaseRole.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ static void addAndDropDatabaseRole(
databaseId,
ImmutableList.of(
"CREATE ROLE " + parentRole,
"GRANT SELECT ON TABLE Singers TO ROLE " + parentRole,
"GRANT SELECT ON TABLE Albums TO ROLE " + parentRole,
"CREATE ROLE " + childRole,
"GRANT ROLE " + parentRole + " TO ROLE " + childRole),
Expand Down
2 changes: 1 addition & 1 deletion samples/snippets/src/main/java/com/example/spanner/admin/archived/CreateSequenceSample.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ static void createSequence(String projectId, String instanceId, String databaseI
.get(5, TimeUnit.MINUTES);

System.out.println(
"Created Seq sequence and Customers table, where the key column CustomerId "
"Created Seq sequence and Customers table, where its key column CustomerId "
+ "uses the sequence as a default value");

final DatabaseClient dbClient =
Expand Down
5 changes: 2 additions & 3 deletions samples/snippets/src/main/java/com/example/spanner/admin/archived/PgSpannerSample.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,6 @@ static void createPostgreSqlDatabase(DatabaseAdminClient dbAdminClient, Database
// Initiate the request which returns an OperationFuture.
Database db = op.get();
System.out.println("Created database [" + db.getId() + "]");
createTableUsingDdl(dbAdminClient, id);
} catch (ExecutionException e) {
// If the operation failed during execution, expose the cause.
throw (SpannerException) e.getCause();
Expand Down Expand Up @@ -633,7 +632,7 @@ static void writeWithTransactionUsingDml(DatabaseClient dbClient) {
Statement updateStatement =
Statement.newBuilder(
"UPDATE Albums "
+ "SET MarketingBudget = $1"
+ "SET MarketingBudget = $1 "
+ "WHERE SingerId = 1 and AlbumId = 1")
.bind("p1")
.to(album1Budget)
Expand All @@ -642,7 +641,7 @@ static void writeWithTransactionUsingDml(DatabaseClient dbClient) {
Statement updateStatement2 =
Statement.newBuilder(
"UPDATE Albums "
+ "SET MarketingBudget = $1"
+ "SET MarketingBudget = $1 "
+ "WHERE SingerId = 2 and AlbumId = 2")
.bind("p1")
.to(album2Budget)
Expand Down
6 changes: 3 additions & 3 deletions ...s/snippets/src/test/java/com/example/spanner/CreateIncrementalBackupScheduleSampleIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,16 @@ public class CreateIncrementalBackupScheduleSampleIT extends SampleTestBaseV2 {
public void testCreateIncrementalBackupScheduleSample() throws Exception {
String backupScheduleId = String.format("schedule-%s", UUID.randomUUID());
BackupScheduleName backupScheduleName =
BackupScheduleName.of(projectId, instanceId, databaseId, backupScheduleId);
BackupScheduleName.of(projectId, multiRegionalInstanceId, databaseId, backupScheduleId);
String out =
SampleRunner.runSample(
() -> {
try {
CreateIncrementalBackupScheduleSample.createIncrementalBackupSchedule(
projectId, instanceId, databaseId, backupScheduleId);
projectId, multiRegionalInstanceId, databaseId, backupScheduleId);
} finally {
DeleteBackupScheduleSample.deleteBackupSchedule(
projectId, instanceId, databaseId, backupScheduleId);
projectId, multiRegionalInstanceId, databaseId, backupScheduleId);
}
});
assertThat(out)
Expand Down
2 changes: 2 additions & 0 deletions samples/snippets/src/test/java/com/example/spanner/CreateInstancePartitionSampleIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import com.google.cloud.spanner.InstanceConfigId;
import com.google.cloud.spanner.InstanceId;
import com.google.cloud.spanner.InstanceInfo;
import com.google.spanner.admin.instance.v1.Instance.Edition;
import com.google.spanner.admin.instance.v1.InstancePartitionName;
import org.junit.Test;

Expand All @@ -34,6 +35,7 @@ public void testCreateInstancePartition() throws Exception {
instanceAdminClient
.createInstance(
InstanceInfo.newBuilder(InstanceId.of(projectId, instanceId))
.setEdition(Edition.ENTERPRISE_PLUS)
.setDisplayName("Geo-partitioning test instance")
.setInstanceConfigId(InstanceConfigId.of(projectId, "regional-us-central1"))
.setNodeCount(1)
Expand Down
3 changes: 2 additions & 1 deletion samples/snippets/src/test/java/com/example/spanner/DatabaseRolesIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,8 @@ public void testAddAndDropDatabaseRole() throws Exception {
SampleRunner.runSample(
() ->
AddAndDropDatabaseRole.addAndDropDatabaseRole(
projectId, instanceId, databaseId.getDatabase(), "new_parent", "new_child"));
projectId, instanceId, databaseId.getDatabase(), "new_parent", "new_child",
"Singers", "Albums"));
assertTrue(out.contains("Created roles new_parent and new_child and granted privileges"));
assertTrue(out.contains("Revoked privileges and dropped role new_child"));
}
Expand Down
1 change: 1 addition & 0 deletions samples/snippets/src/test/java/com/example/spanner/SampleTestBaseV2.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ public static void afterClass() throws InterruptedException {
}
}

spanner.close();
databaseAdminClient.close();
instanceAdminClient.close();

Expand Down
2 changes: 1 addition & 1 deletion samples/snippets/src/test/java/com/example/spanner/SpannerGraphSampleIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ public class SpannerGraphSampleIT extends SampleTestBaseV2 {

private static final int DBID_LENGTH = 20;
// The instance needs to exist for tests to pass.
private static final String instanceId = System.getProperty("spanner.test.instance");
private static final String instanceId = System.getProperty("spanner.test.instance.mr");
private static final String baseDbId = System.getProperty("spanner.sample.database");
static Spanner spanner;
static DatabaseAdminClient databaseAdminClient;
Expand Down
5 changes: 5 additions & 0 deletions samples/snippets/src/test/java/com/example/spanner/SpannerSampleIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,7 @@ public void testEncryptedDatabaseAndBackupSamples() throws Exception {
.setNodeCount(1)
.build())
.get();
System.out.println("Creating database ...");
try {
String out =
SampleRunner.runSample(
Expand All @@ -538,6 +539,7 @@ public void testEncryptedDatabaseAndBackupSamples() throws Exception {
String.format(
"Created database [%s]", DatabaseName.of(projectId, instanceId, databaseId)));

System.out.println("Creating backup with encryption key ...");
out =
SampleRunner.runSampleWithRetry(
() ->
Expand All @@ -556,6 +558,7 @@ public void testEncryptedDatabaseAndBackupSamples() throws Exception {
+ "was created at (.*) using encryption key %s",
projectId, instanceId, encryptedBackupId, key));

System.out.println("Restoring backup with encryption key ...");
out =
SampleRunner.runSampleWithRetry(
() ->
Expand Down Expand Up @@ -587,6 +590,7 @@ public void testEncryptedDatabaseAndBackupSamples() throws Exception {
} finally {
// Delete the backups from the test instance first, as the instance can only be deleted once
// all backups have been deleted.
System.out.println("Deleting backups ...");
deleteAllBackups(instanceId);
instanceAdminClient.deleteInstance(instanceId);
}
Expand Down Expand Up @@ -633,6 +637,7 @@ private static void deleteAllBackups(String instanceId) throws InterruptedExcept
InstanceName instanceName = InstanceName.of(projectId, instanceId);
for (Backup backup : databaseAdminClient.listBackups(instanceName.toString()).iterateAll()) {
int attempts = 0;
System.out.printf("Deleting backup ... %s%n", backup.getName());
while (attempts < 30) {
try {
attempts++;
Expand Down
2 changes: 1 addition & 1 deletion ...ippets/src/test/java/com/example/spanner/admin/archived/CustomInstanceConfigSampleIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ public void testCustomInstanceConfigOperations() throws Exception {
SampleRunner.runSample(
() ->
ListInstanceConfigOperationsSample.listInstanceConfigOperations(projectId));
assertTrue(out2.contains("List instance config operation"));
assertTrue(out2.contains("Create instance config operation"));

// Update display name to a randomly generated instance config id.
final String out3 =
Expand Down
5 changes: 4 additions & 1 deletion samples/snippets/src/test/java/com/example/spanner/admin/archived/DatabaseRolesIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,15 @@
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.junit.runners.MethodSorters;

/** Integration tests for FGAC samples for GoogleStandardSql dialect. */
@RunWith(JUnit4.class)
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class DatabaseRolesIT extends SampleTestBase {

private static DatabaseId databaseId;
Expand Down Expand Up @@ -105,7 +108,7 @@ public void testAddAndDropDatabaseRole() throws Exception {
SampleRunner.runSample(
() ->
AddAndDropDatabaseRole.addAndDropDatabaseRole(
projectId, instanceId, databaseId.getDatabase(), "new-parent", "new-child"));
projectId, instanceId, databaseId.getDatabase(), "new_parent", "new_child"));
assertTrue(out.contains("Created roles new_parent and new_child and granted privileges"));
assertTrue(out.contains("Revoked privileges and dropped role new_child"));
}
Expand Down
2 changes: 1 addition & 1 deletion samples/snippets/src/test/java/com/example/spanner/admin/archived/PgSpannerSampleIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ public void testSample() throws Exception {
assertThat(databaseId).isNotNull();

System.out.println("Create Database ...");
String out = runSample("createpgdatabase");
String out = runSample("createdatabase");
assertThat(out).contains("Created database");
assertThat(out).contains(dbId.getName());

Expand Down
Loading