Support pagination to fetch _all_ Bitbucket branches #18563

jankeromnes · 2023-08-21T14:25:10Z

Description

Summary generated by Copilot

`🤖 Generated by Copilot at f818630`

Add pagination support to listBranches method of BitbucketRepositoryProvider. This allows fetching all the branches for a Bitbucket repository when starting a workspace.

Related Issue(s)

Fixes https://linear.app/gitpod/issue/EXP-3/bitbucketorg-support-api-pagination-when-listing-branches

How to test

?

Documentation

Preview status

Gitpod was successfully deployed to your preview environment.

🏷️ Name - jx-paginatedfeca2734
🔗 URL - jx-paginatedfeca2734.preview.gitpod-dev.com/workspaces.
📚 Documentation - See our internal documentation for information on how to interact with your preview environment.
📦 Version - jx-paginate-bitbucket-branches-gha.15806

Build Options

Build

/werft with-werft
Run the build with werft instead of GHA
leeway-no-cache
/werft no-test
Run Leeway with --dont-test

Publish

/werft publish-to-npm
/werft publish-to-jb-marketplace

Installer

analytics=segment
with-dedicated-emulation
workspace-feature-flags
Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

/werft with-local-preview
If enabled this will build install/preview
/werft with-preview
/werft with-large-vm
/werft with-gce-vm
If enabled this will create the environment on GCE infra
with-integration-tests=all
Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
with-monitoring

/hold

jankeromnes · 2023-08-21T14:27:06Z

FYI, reviewing while ignoring whitespace changes is much easier here:

https://github.com/gitpod-io/gitpod/pull/18563/files?w=1

loujaybee · 2023-08-22T09:23:23Z

Thanks, @jankeromnes for raising a fix ! 🙏 - I'm happy for this to be reviewed + tested, however I would suggest we hold off a little longer on the merge + release (internal context).

jankeromnes · 2023-08-22T09:46:56Z

I'm happy for this to be reviewed + tested

Thanks @loujaybee! 🙌

however I would suggest we hold off a little longer on the merge + release (internal context).

Sorry if I'm missing something, but what is the benefit of leaving a feature broken for a longer time? (I.e. what is the downside of fixing this small bug as soon as possible?)

To me, the slightly broken feature and the larger customer UX discovery are two completely separate topics.

filiptronicek

Code changes LGTM. Did not test because preview environment does not seem to have been triggered 👀

components/server/src/bitbucket/bitbucket-repository-provider.ts

jankeromnes · 2023-08-22T12:21:56Z

Many thanks @filiptronicek for the review! And for pointing out that there is no preview env. I've attempted to get GitHub Actions to produce one. Let's see if it does. 👀

filiptronicek · 2023-08-22T13:22:40Z

It took 2 minutes, but I successfully got all of the branches of my gitpod-io/gitpod mirror back!

jankeromnes · 2023-08-22T13:25:24Z

Whoa, many thanks for testing @filiptronicek! 🎉 (2 minutes 🤯)

jankeromnes · 2023-08-22T13:47:25Z

I've removed the unnecessary type annotations and added a pagelen: 100 parameter, in the hopes of optimizing the number of API round-trips necessary to fetch long lists of branches (if the Bitbucket API behaves like the GitLab and GitHub ones, I believe the default page length might be 30).

jankeromnes · 2023-08-22T14:01:56Z

@filiptronicek Could you please briefly test with your fork again? 😇 🙏 I believe it should be faster now (and still correct!)

akosyakov · 2023-08-22T14:57:33Z

components/server/src/bitbucket/bitbucket-repository-provider.ts

+        let nextPage = 1;
+        let isMoreDataAvailable = true;
+
+        while (isMoreDataAvailable) {


Are there risk that hit some rate limitting here? Can we guess how much actual data would be given similar configuration of bbs and test against it?

Asking not blocking.

Good question! 🎯 Probably yes (just like all other Bitbucket API requests), but I think using a pagelen of 100 mitigates this somewhat (i.e. you'd have to have an incredibly large number of branches in order to get into rate limit territory, and we also do all these requests in sequence and not in parallel).

My feeling here is that it's safe to let any (supposedly very rare / unlikely) rate limit error just bubble up as is if it ever happens, i.e. the Gitpod user would see something like "Unable to fetch branches: Bitbucket API rate limit exceeded".

For repositories we have pagination as well but it broke with 10000 as far as I understand and was very slow to fetch everything breaking ability to create a project.

@akosyakov aha -- was that with pagelen: 100 or the default page length? (Optimizing the page length can divide the number of queries required by ~3x)

I think it was 10K repos with 1000 page size, but we were doing for 2 different properties. @AlexTugarev would be more helpful here, but he is out sick today

filiptronicek

Tested with my mirror again (https://bitbucket.org/biased-configuration/gitpod/src/main/) and it seems to work much faster now. Thanks!

jankeromnes · 2023-08-22T15:40:59Z

@loujaybee good to remove the hold? 🚀

loujaybee · 2023-08-22T17:06:23Z

Sorry if I'm missing something, but what is the benefit of leaving a feature broken for a longer time? (I.e. what is the downside of fixing this small bug as soon as possible?)

The context is that this page is planned to be removed, so I'm conscious that we don't set expectations that this page will be invested in more if possible, and I'm also hesitant in introducing more complexity into the page.
Yeah, and that was really also my question, e.g. "what is the downside"? If this won't introduce any subsequent other issues, such as performance related issues, exhausting token rate limits, etc then let's go ahead and merge.

I'm happy to trust your judgement on it, @jankeromnes if you think it's a net benefit with minimal or no downside, let's merge 🙏

jankeromnes · 2023-08-23T07:57:31Z

Many thanks @loujaybee! In this case, I'd like to go ahead with merging this fix:

It fixes our getBranches feature, which is currently broken for Bitbucket when there are more than ~30 branches in a repo
I believe there aren't any downsides or significant risks to this fix (we talked about a possible rate-limit issue, but that can only happen if there are more than 10K or 30K branches in a repo -- quite unlikely)

In any case, if there turns out to be a downside to this fix, I'm happy to revert it and improve my fix. 👍

jankeromnes · 2023-08-23T07:57:35Z

/unblock

jankeromnes · 2023-08-23T07:58:16Z

/unhold

roboquat added do-not-merge/hold do-not-merge/work-in-progress size/M labels Aug 21, 2023

filiptronicek approved these changes Aug 22, 2023

View reviewed changes

components/server/src/bitbucket/bitbucket-repository-provider.ts Outdated Show resolved Hide resolved

Support pagination to fetch _all_ Bitbucket branches

75b87d5

jankeromnes force-pushed the jx/paginate-bitbucket-branches branch from f818630 to 75b87d5 Compare August 22, 2023 13:45

jankeromnes marked this pull request as ready for review August 22, 2023 14:56

jankeromnes requested a review from a team as a code owner August 22, 2023 14:56

roboquat removed the do-not-merge/work-in-progress label Aug 22, 2023

github-actions bot added the team: team-experience label Aug 22, 2023

akosyakov reviewed Aug 22, 2023

View reviewed changes

Merge branch 'main' into jx/paginate-bitbucket-branches

cc5da82

filiptronicek approved these changes Aug 22, 2023

View reviewed changes

filiptronicek added the git provider: bitbucket label Aug 22, 2023

roboquat removed the do-not-merge/hold label Aug 23, 2023

roboquat merged commit c85581c into main Aug 23, 2023

roboquat deleted the jx/paginate-bitbucket-branches branch August 23, 2023 07:58

Support pagination to fetch _all_ Bitbucket branches #18563

Support pagination to fetch _all_ Bitbucket branches #18563

Uh oh!

Conversation

jankeromnes commented Aug 21, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

🤖 Generated by Copilot at f818630

Related Issue(s)

How to test

Documentation

Preview status

Build Options

Uh oh!

jankeromnes commented Aug 21, 2023

Uh oh!

loujaybee commented Aug 22, 2023

Uh oh!

jankeromnes commented Aug 22, 2023

Uh oh!

filiptronicek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jankeromnes commented Aug 22, 2023

Uh oh!

filiptronicek commented Aug 22, 2023

Uh oh!

jankeromnes commented Aug 22, 2023

Uh oh!

jankeromnes commented Aug 22, 2023

Uh oh!

jankeromnes commented Aug 22, 2023

Uh oh!

akosyakov Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jankeromnes Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akosyakov Aug 22, 2023

Choose a reason for hiding this comment

Uh oh!

jankeromnes Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akosyakov Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filiptronicek left a comment

Choose a reason for hiding this comment

Uh oh!

jankeromnes commented Aug 22, 2023

Uh oh!

loujaybee commented Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jankeromnes commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jankeromnes commented Aug 23, 2023

Uh oh!

jankeromnes commented Aug 23, 2023

Uh oh!

Uh oh!

jankeromnes commented Aug 21, 2023 •

edited by github-actions bot

Loading

`🤖 Generated by Copilot at f818630`

akosyakov Aug 22, 2023 •

edited

Loading

jankeromnes Aug 22, 2023 •

edited

Loading

jankeromnes Aug 22, 2023 •

edited

Loading

akosyakov Aug 23, 2023 •

edited

Loading

loujaybee commented Aug 22, 2023 •

edited

Loading

jankeromnes commented Aug 23, 2023 •

edited

Loading