Skip to content

Better index version downloads for the update script #1346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 13, 2018

Conversation

sgrif
Copy link
Contributor

@sgrif sgrif commented Apr 13, 2018

The query used to grab the next batch of unprocessed version downloads
is now taking on average 9 seconds to run. It can take as much as 30
seconds. I suspect this is because the size of the processed index, or
some other measurement PG is using changed so it decided that index
wasn't the right one to use.

That index wasn't very helpful anyway (it was just the processed
column). We can do better by indexing the id with the where clause.

EXPLAIN ANALYZE results:

Before:

                                                                             QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.09..16204.73 rows=1000 width=21) (actual time=39279.902..39298.408 rows=1000 loops=1)
   ->  Index Scan using version_downloads_pkey on version_downloads  (cost=0.09..199171.32 rows=12291 width=21) (actual time=39279.900..39298.335 rows=1000 loops=1)
         Index Cond: (id > 0)
         Filter: (NOT processed)
         Rows Removed by Filter: 8544073
 Planning time: 0.369 ms
 Execution time: 39298.459 ms
(7 rows)

After:

                                                                                QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.06..95.89 rows=1000 width=21) (actual time=0.016..0.796 rows=1000 loops=1)
   ->  Index Scan using index_version_downloads_not_processed on version_downloads  (cost=0.06..1175.30 rows=12264 width=21) (actual time=0.016..0.732 rows=1000 loops=1)
         Index Cond: (id > 0)
 Planning time: 0.123 ms
 Execution time: 0.854 ms

The query used to grab the next batch of unprocessed version downloads
is now taking on average 9 seconds to run. It can take as much as 30
seconds. I suspect this is because the size of the `processed` index, or
some other measurement PG is using changed so it decided that index
wasn't the right one to use.

That index wasn't very helpful anyway (it was just the `processed`
column). We can do better by indexing the id with the where clause.

`EXPLAIN ANALYZE` results:

Before:

```
                                                                             QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.09..16204.73 rows=1000 width=21) (actual time=39279.902..39298.408 rows=1000 loops=1)
   ->  Index Scan using version_downloads_pkey on version_downloads  (cost=0.09..199171.32 rows=12291 width=21) (actual time=39279.900..39298.335 rows=1000 loops=1)
         Index Cond: (id > 0)
         Filter: (NOT processed)
         Rows Removed by Filter: 8544073
 Planning time: 0.369 ms
 Execution time: 39298.459 ms
(7 rows)
```

After:

```
                                                                                QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.06..95.89 rows=1000 width=21) (actual time=0.016..0.796 rows=1000 loops=1)
   ->  Index Scan using index_version_downloads_not_processed on version_downloads  (cost=0.06..1175.30 rows=12264 width=21) (actual time=0.016..0.732 rows=1000 loops=1)
         Index Cond: (id > 0)
 Planning time: 0.123 ms
 Execution time: 0.854 ms
```
@sgrif sgrif requested a review from ashleygwilliams April 13, 2018 17:10
@ashleygwilliams
Copy link
Member

bors: r+

bors-voyager bot added a commit that referenced this pull request Apr 13, 2018
1346: Better index version downloads for the update script r=ashleygwilliams

The query used to grab the next batch of unprocessed version downloads
is now taking on average 9 seconds to run. It can take as much as 30
seconds. I suspect this is because the size of the `processed` index, or
some other measurement PG is using changed so it decided that index
wasn't the right one to use.

That index wasn't very helpful anyway (it was just the `processed`
column). We can do better by indexing the id with the where clause.

`EXPLAIN ANALYZE` results:

Before:

```
                                                                             QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.09..16204.73 rows=1000 width=21) (actual time=39279.902..39298.408 rows=1000 loops=1)
   ->  Index Scan using version_downloads_pkey on version_downloads  (cost=0.09..199171.32 rows=12291 width=21) (actual time=39279.900..39298.335 rows=1000 loops=1)
         Index Cond: (id > 0)
         Filter: (NOT processed)
         Rows Removed by Filter: 8544073
 Planning time: 0.369 ms
 Execution time: 39298.459 ms
(7 rows)
```

After:

```
                                                                                QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.06..95.89 rows=1000 width=21) (actual time=0.016..0.796 rows=1000 loops=1)
   ->  Index Scan using index_version_downloads_not_processed on version_downloads  (cost=0.06..1175.30 rows=12264 width=21) (actual time=0.016..0.732 rows=1000 loops=1)
         Index Cond: (id > 0)
 Planning time: 0.123 ms
 Execution time: 0.854 ms
```
@bors-voyager
Copy link
Contributor

bors-voyager bot commented Apr 13, 2018

Build succeeded

@bors-voyager bors-voyager bot merged commit 9cf06d1 into rust-lang:master Apr 13, 2018
@sgrif sgrif deleted the sg-index-version-downloads branch April 14, 2018 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants