[server] Fix spicedb retry on DEADLINE_EXCEEDED & UNAVAILABLE

[geropl]

Description

This bug got introduced with #20637.
While that PR fixed two other problems, in the end we went with re-using the manual re-try implementation to work around the Waiting for LB pick issue with @grpc/grpc-js. That logic got copied over without the relevant await, so we never did any of the retries.

Related Issue(s)

Fixes CLC-1402

How to test

• I used this script for testing, but - as always - could not reliably reproduce the root cause in our previews
  • best testing results I got was when going with replicas: 1, maxUnavailable: 1, maxSurge: 0 and an initContainer sleeping for 30s
  • 💡 this indicates that the root cause is indeed some kind of (persistent?) connection issue - hopefully mitigated by the client re-creation now ✔️
• You can do a basic smoke test at https://gpl-1402-grpcjs.preview.gitpod-dev.com

Documentation

Preview status

gitpod:summary

Build Options

Build

• /werft with-werft
  Run the build with werft instead of GHA
• leeway-no-cache
• /werft no-test
  Run Leeway with --dont-test

Publish

• /werft publish-to-npm
• /werft publish-to-jb-marketplace

Installer

• analytics=segment
• with-dedicated-emulation
• workspace-feature-flags
  Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-gce-vm
  If enabled this will create the environment on GCE infra
• /werft preemptible
  Saves cost. Untick this only if you're really sure you need a non-preemtible machine.
• with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
• with-monitoring

/hold

[corneliusludmann]

Code looks good. Cannot access the preview (anymore). Trust your tests. 👍

[geropl]

/unhold