Skip to content

database/versions: Add new semver_no_prerelease column #4022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

Turbo87
Copy link
Member

@Turbo87 Turbo87 commented Oct 17, 2021

This PR is an alternative to:

Instead of using our custom semver_triple database type (which caused issues with diesel) we will use an array (NUMERIC[3]). This provides the same sorting and filtering functionality, but is easier to deal with in diesel.

This PR only adds the new column and adjusts the code to fill it for new crate versions, but it does not backfill it for old versions. Neither does it adjust the reverse dependencies SQL query to use the new column, as that requires the old versions to be backfilled first. This will be changed in a future PR.

A SQL script to backfill the column for old versions is:

-- add temporary function that returns an array instead of `semver_triple`

CREATE FUNCTION to_semver_no_prerelease_array(TEXT) RETURNS NUMERIC[3]
    IMMUTABLE
    LANGUAGE sql
AS
$$
SELECT ARRAY [
           split_part($1, '.', 1)::numeric,
           split_part($1, '.', 2)::numeric,
           split_part(split_part($1, '+', 1), '.', 3)::numeric
           ]
WHERE strpos($1, '-') = 0
$$;

-- fill the new column with data incrementally to avoid table locking for
-- extended time periods.

UPDATE versions
SET semver_no_prerelease = to_semver_no_prerelease_array(num)
WHERE semver_no_prerelease IS NULL
  AND created_at < '2016-01-01';

UPDATE versions
SET semver_no_prerelease = to_semver_no_prerelease_array(num)
WHERE semver_no_prerelease IS NULL
  AND created_at < '2016-02-01';

UPDATE versions
SET semver_no_prerelease = to_semver_no_prerelease_array(num)
WHERE semver_no_prerelease IS NULL
  AND created_at < '2016-03-01';

-- ...

UPDATE versions
SET semver_no_prerelease = to_semver_no_prerelease_array(num)
WHERE semver_no_prerelease IS NULL
  AND created_at < '2021-12-01';

-- drop temporary function again

DROP FUNCTION to_semver_no_prerelease_array(TEXT);

@Turbo87 Turbo87 added A-backend ⚙️ C-internal 🔧 Category: Nonessential work that would make the codebase more consistent or clear labels Oct 17, 2021
@Turbo87 Turbo87 requested a review from pietroalbini October 17, 2021 12:55
@Turbo87 Turbo87 force-pushed the add-semver-array-column branch from 79db897 to 8973443 Compare October 18, 2021 19:51
@bors
Copy link
Contributor

bors commented Oct 19, 2021

☔ The latest upstream changes (presumably #4034) made this pull request unmergeable. Please resolve the merge conflicts.

@Turbo87 Turbo87 force-pushed the add-semver-array-column branch 2 times, most recently from 94c9b70 to 8e8bb01 Compare October 21, 2021 12:32
@Turbo87 Turbo87 requested a review from jtgeibel October 22, 2021 13:16
@Turbo87 Turbo87 force-pushed the add-semver-array-column branch 2 times, most recently from 9689763 to cc66bfa Compare October 26, 2021 06:46
This is needed to support the `Numeric` type in Postgres, which is used by our custom `to_semver_no_prerelease` function
@Turbo87 Turbo87 force-pushed the add-semver-array-column branch from cc66bfa to e49c704 Compare October 29, 2021 11:40
@jtgeibel
Copy link
Member

The PR itself looks good to me. For the SQL script, my understanding is that UPDATE queries will run for every version WHERE semver_no_prerelease IS NULL and will simply reassign NULL to semver_no_prerelease if the version is a pre-release. I have just 2 comments on the update script:

  • Any prerelease versions meeting created_at < '...' will be processed again for all subsequent queries because semver_no_prerelease will still be NULL. So later queries will lock more rows and for longer than expected.
  • I was initially worried that this would cause the updated_at column to be updated for all versions, but it looks like the trigger_versions_set_updated_at trigger has been updated to only run on updates to the yanked column.

@Turbo87
Copy link
Member Author

Turbo87 commented Oct 29, 2021

setting this PR to draft mode for now, until we've decided wether or not #4106 is a better way

@Turbo87 Turbo87 marked this pull request as draft October 29, 2021 15:58
@bors
Copy link
Contributor

bors commented Oct 30, 2021

☔ The latest upstream changes (presumably #4108) made this pull request unmergeable. Please resolve the merge conflicts.

@Turbo87
Copy link
Member Author

Turbo87 commented Dec 10, 2021

closing in favor of #4106

@Turbo87 Turbo87 closed this Dec 10, 2021
@Turbo87 Turbo87 deleted the add-semver-array-column branch October 17, 2022 17:35
@Turbo87 Turbo87 restored the add-semver-array-column branch January 8, 2023 20:01
@Turbo87 Turbo87 deleted the add-semver-array-column branch October 21, 2024 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-backend ⚙️ C-internal 🔧 Category: Nonessential work that would make the codebase more consistent or clear
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants