Make db waiter to wait latest migration

[mustard-mh]

Description

Summary generated by Copilot

### 🤖 Generated by Copilot at cdc4ee6

This pull request adds a new feature to the installer and the service-waiter to check and apply database migrations. It embeds a JSON file with the latest migration info into the installer binary and passes it to the service-waiter container. It also improves the database connection and migration check logic in the service-waiter. It adds a new component to the gitpod-db package to produce the JSON file from the migration folder. It updates the installer and the service-waiter packages and adds a new test case for the installer.

Related Issue(s)

Fixes EXP-377

How to test

With preview env

• Check if components that required db ready (like server) log contains checking if database is migrated and log waiting for database (has migration field that should contains latest migration name / timestamp)

Smoke test
See #18455 (comment) #18455 (comment)

Open PR with Gitpod

Script/Dir	Detail
CheckScript	go run main.go database -P test -u root -p 23306 --migration-check true --timeout 5m
WriteLatestMigration	./scripts/generate-latest-migration.sh > ../service-waiter/cmd/resources/latest-migration.txt
CreateNewMigration	yarn typeorm migration:create -n test
DirA	cd components/service-waiter
DirB	cd components/gitpod-db

• Init test db: leeway run components/gitpod-db:init-testdb
• Open two terminal with DirA and DirB
• B: exec CheckScript, it should print no checking if database is migrated 👉 only ping db if no migration required
• A: exec WriteLatestMigration
• B: exec CheckScript, it should print checking if database is migrated and succeed 👉 works if migrated
• A: exec CreateNewMigration
• A: exec WriteLatestMigration
• B: exec CheckScript, it should keep retry and failed. KEEP process running 👉 before migrate should wait
• A: migrate test db leeway run components/gitpod-db:init-testdb
• That running process in dirB should succeed after db migrated 👉 wait until migrated

Documentation

Preview status

Gitpod was successfully deployed to your preview environment.

• 🏷️ Name - hw-exp-377
• 🔗 URL - hw-exp-377.preview.gitpod-dev.com/workspaces.
• 📚 Documentation - See our internal documentation for information on how to interact with your preview environment.
• 📦 Version - hw-EXP-377-gha.14952

Build Options

Build

• /werft with-werft
  Run the build with werft instead of GHA
• leeway-no-cache
• /werft no-test
  Run Leeway with --dont-test

Publish

• /werft publish-to-npm
• /werft publish-to-jb-marketplace

Installer

• analytics=segment
• with-dedicated-emulation
• workspace-feature-flags
  Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-gce-vm
  If enabled this will create the environment on GCE infra
• with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
• with-monitoring

/hold

[svenefftinge]

Generating a go constant directly next to database.go as part of the yarn build in gitpod-db seems to be less error prone than piping it through installer. Also keeps the interface for service-waiter simpler.

[akosyakov]

I wonder can we smoke test it in preview envs to see behaviour actual pods on crash? It should be possible to change DB, i.e. rename latest migration and then delete the server to observer? and then undo and check again?

[mustard-mh]

I wonder can we smoke test it in preview envs to see behaviour actual pods on crash? It should be possible to change DB, i.e. rename latest migration and then delete the server to observer? and then undo and check again?

@akosyakov I tried to do it yesterday with leeway dev:preview

• ✅ it can check if latest migration migrated in preview
• ✅ add another migration server will restart and service-waiter will check with that migration
• (not yet) add a migration that will stuck for 10 minutes, -> 1️⃣ new server should crash after 5m (default service-waiter timeout) and 2️⃣ there should be a metric changed in promethues, 3️⃣ another new server pod (created by deployment) should work after 10m. (not yet since prometheus port-forwarding with dev preview is super unstable and build took too much time)

rename latest migration and then delete the server to observer?

it should be the same with what we're doing in how to test section (open PR with gitpod topic), will try

[mustard-mh]

As part of #18455 (comment)

Smoke tested with leeway dev:preview, with a migration hangs for 10 minutes, we can see:

• k8s terminated message
• prometheus (server unavailable for more than 10 minutes) 👇
• Server pods will restart but not up

Terminated Message	Prometheus

Going to push remind commits. cc @svenefftinge @akosyakov to view test results

Missed a unit test to mock db query and migration check, but it could be a follow up

[svenefftinge]

What happened to the plan to generate it as part of yarn build and check in the latest-migration.txt?
Also, the go feature to bind files during compilation is nice, but I wonder if generating a go file with a constant would not be more direct and therefore simpler in this case.
What does introducing the flag in the installer which is then hard coded buys us vs just making it non optional in the database-waiter?

[mustard-mh]

What happened to the plan to generate it as part of yarn build and check in the latest-migration.txt?

If we consider to use .sh to get the name, we don't need any nodejs related thing, just list migration folder and parse it. So it can reduce the build process (service-waiter don't need to depend on test db). For testing, it does run test in .sh file that it should be able to generate a name

but I wonder if generating a go file with a constant would not be more direct and therefore simpler in this case.

• It works the same but needs more steps (format. don't edit file mark on header of file)
• hard to extend like if you want to introduce another var? otherwise you need to write the generator inside serviec-waiter

What does introducing the flag in the installer which is then hard coded buys us vs just making it non optional in the database-waiter?

Do you mean the flag --migration-check in service-waiter/database.go? It's because service like job/migration should not wait until migrated () otherwise it won't work

@svenefftinge

[svenefftinge]

If we consider to use .sh to get the name, we don't need any nodejs related thing, just list migration folder and parse it.

I meant running the bash script before tsx, because the input to the bash.script is a *.ts file that is processed by the same command (yarn build). That would require to check-in the latest-migration.txt file but I believe that is fine (or even good).
I'm mostly asking for this because getting the dependencies right using leeway in this case seems more complicated than needed and also does leak internals of leeway (i.e. using the path of how leeway internally links packages _deps/components-gitpod-db--latest-migration/) that make this fragile.

From the perspective of the service-waiter it would just use whatever is in the file and doesn't care how it got there. The bash script is just a convenient way to keep things in sync and by adding it to yarn bash we will make sure we don't forget to run it.

[svenefftinge]

I still think the bash script should sit in service-waiter and be called from its BUILD.yaml rather than the indirection of having the gitpod-db generating the file and that it gets copied over. But I'm fine either way and have already blocked you too much with this.

Edit: I have only read through the code but not tested it.