Don't lock more than 1 row at a time in update-downloads
#1345
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prior to this commit,
bin/update-downloads
will hold a lock on everyrow it updates for the entire batch of 1000. Each row gets locked when
it is
updated
, but the lock isn't released until the whole transactionis comitted. This means that the first row in the batch is locked for
the whole transaction, but the last row in the batch isn't locked longer
than it should be.
Processing 1000 rows is routinely taking more than 30 seconds to
complete. Whenever it does, this causes requests to time out. Even when
it doesn't, this locking behavior is the reason that our 99th percentile
response time is getting to be so bad.
The endpoint affected by this is
/crates/{name}/{version}/download
. Inother words, this bug is causing
cargo install
to fail. Even when itdoesn't, this bug is causing it to take seconds when it should take
milliseconds.
By establishing a transaction per row instead of per batch, we retain
atomicity, without holding a lock for the whole batch. To keep
everything atomic, we need to update the global counter per row instead
of per batch.
This relies on the invariant that no rows in
version_downloads
gettouched once
date != CURRENT_DATE
other than by this script.I believe this is also the root cause of #1304, and that we can maybe
revert #1312