Skip to content

[dashboard] proactively reconnect grpc streams #19185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 4, 2023

Conversation

akosyakov
Copy link
Member

@akosyakov akosyakov commented Dec 3, 2023

Description

This PR addresses the issue where GCP LB is set to terminate connections every 10 minutes. This results in unpredicted behavior across different browsers when using Connect WEB, leading to the reporting of an 'unknown error' instead of a specific failure. Particularly impacted are gRPC server-side streams due to their long-running nature. To mitigate this, the PR introduces a proactive approach by reconnecting gRPC streams 3 minutes before the timeout. This ensures errors are correctly reported as 'deadline exceeded' (actually 'cancelled' because of connectrpc/connect-es#954) rather than as an unknown error.

Summary generated by Copilot

🤖[deprecated] Generated by Copilot at 37667e4

This pull request enhances the public API service and its client by adding a timeout option for streaming calls, improving the error handling and testing logic, and supporting a new error code for deadline exceeded errors. It affects the files public-api.ts, service.tsx, public-api-converter.ts, error.ts, and public-api-converter.spec.ts.

Related Issue(s)

Fixes #

How to test

  • Use yarn telepresence in dashboard to update timeout to the lower value like 1 min.
  • Open dashboard, check LotsOfReplies request, and check out that it gets reconnected within one minute on deadline exceeded error.
  • Kill the server, checks out that LotsOfReplies is reconnected again.

Documentation

Preview status

https://ak-dashboa61b82253e9.preview.gitpod-dev.com/workspaces

Run time leeway run dev:preview --dont-test from Gitpod workspace if it is down.

Build Options

Build
  • /werft with-werft
    Run the build with werft instead of GHA
  • leeway-no-cache
  • /werft no-test
    Run Leeway with --dont-test
Publish
  • /werft publish-to-npm
  • /werft publish-to-jb-marketplace
Installer
  • analytics=segment
  • with-dedicated-emulation
  • workspace-feature-flags
    Add desired feature flags to the end of the line above, space separated
Preview Environment / Integration Tests
  • /werft with-local-preview
    If enabled this will build install/preview
  • /werft with-large-vm
  • /werft with-gce-vm
    If enabled this will create the environment on GCE infra
  • /werft preemptible
    Saves cost. Untick this only if you're really sure you need a non-preemtible machine.
  • with-integration-tests=all
    Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
  • with-monitoring

/hold

@akosyakov akosyakov force-pushed the ak/dashboard_grpc_stream_timeout branch from 76ba165 to a1fc542 Compare December 4, 2023 12:34
@akosyakov akosyakov marked this pull request as ready for review December 4, 2023 12:39
@akosyakov akosyakov requested a review from a team as a code owner December 4, 2023 12:39
@akosyakov akosyakov requested a review from geropl December 4, 2023 12:40
@geropl
Copy link
Member

geropl commented Dec 4, 2023

Code LGTM, will test once I find time...

Copy link
Member

@geropl geropl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and works as expected! 🎉

@akosyakov
Copy link
Member Author

/unhold

@roboquat roboquat merged commit 5a55a7a into main Dec 4, 2023
@roboquat roboquat deleted the ak/dashboard_grpc_stream_timeout branch December 4, 2023 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants