[fga] retry write relationships

[svenefftinge]

Description

Occasionally we see errors when trying to write relkation ships in Gitpod Cloud (logs).

Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
    at callErrorFromStatus (/app/node_modules/@grpc/grpc-js/src/call.ts:82:17)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client.ts:360:55)
    at /app/node_modules/@grpc/grpc-js/src/call-interface.ts:149:27
    at Object.onReceiveStatus (/app/node_modules/@gitpod/gitpod-protocol/src/util/grpc.ts:80:29)
    at InterceptingListenerImpl.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/call-interface.ts:145:19)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:458:34)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48)
    at /app/node_modules/@grpc/grpc-js/src/resolving-call.ts:132:24
    at processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at Proxy.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/src/client.ts:325:42)
    at Proxy.writeRelationships (/app/node_modules/@authzed/authzed-node/src/authzedapi/authzed/api/v1/permission_service.grpc-client.ts:129:21)
    at node:internal/util:375:7
    at new Promise (<anonymous>)
    at Proxy.writeRelationships (node:internal/util:361:12)
    at SpiceDBAuthorizer.writeRelationships (/app/node_modules/@gitpod/server/src/authorization/spicedb-authorizer.ts:61:48)
    at Authorizer.addOrganization (/app/node_modules/@gitpod/server/src/authorization/authorizer.ts:362:31)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at RelationshipUpdater.updateOrganization (/app/node_modules/@gitpod/server/src/authorization/relationship-updater.ts:161:9)
    at /app/node_modules/@gitpod/server/src/authorization/relationship-updater.ts:76:21
    at Redlock.using (/app/node_modules/redlock/src/index.ts:742:14)
    at RelationshipUpdater.migrate (/app/node_modules/@gitpod/server/src/authorization/relationship-updater.ts:56:20)
    at UserService.findUserById (/app/node_modules/@gitpod/server/src/user/user-service.ts:88:20)
    at GithubApp.findInstallationOwner (/app/node_modules/@gitpod/server/src/prebuilds/github-app.ts:672:22)
    at GithubApp.findOwnerAndProject (/app/node_modules/@gitpod/server/src/prebuilds/github-app.ts:230:52)
    at GithubApp.handlePushEvent (/app/node_modules/@gitpod/server/src/prebuilds/github-app.ts:264:37)

This seems to. be a GRPC connection error and has been reported here as well without a solution:

grpc/grpc-node#1747

Since it happens only rarely, I'll add retry logic, so at least we don't miss any relationships until we have understood and fixed the underlying issue.

Summary generated by Copilot

🤖 Generated by Copilot at e8d82e9

Added logging for organization usage limits and improved retry logic for writing permissions to SpiceDB. These changes aim to enhance the visibility and reliability of the billing and authorization features of Gitpod.

Related Issue(s)

Fixes EXP-585

How to test

Documentation

Preview status

Gitpod was successfully deployed to your preview environment.

• 🏷️ Name - se-fga-retry
• 🔗 URL - se-fga-retry.preview.gitpod-dev.com/workspaces.
• 📚 Documentation - See our internal documentation for information on how to interact with your preview environment.
• 📦 Version - se-fga-retry-gha.16660
• 🗒️ Logs - GCP Logs Explorer

Build Options

Build

• /werft with-werft
  Run the build with werft instead of GHA
• leeway-no-cache
• /werft no-test
  Run Leeway with --dont-test

Publish

• /werft publish-to-npm
• /werft publish-to-jb-marketplace

Installer

• analytics=segment
• with-dedicated-emulation
• workspace-feature-flags
  Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-gce-vm
  If enabled this will create the environment on GCE infra
• with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
• with-monitoring

/hold

[geropl]

Code LGTM, tested and still can start a workspace ✔️

[geropl]

I wonder if the grpc-retry mechanism also works for INTERNAL errors. 🤔

@svenefftinge Did you happen to try it?

[svenefftinge]

I did not check that, but I'd love to get some logs for when this happens and have only seen this on write relationships

[svenefftinge]

/unhold