Skip to content

Do not do SELECT count(*) FROM ... if pagination is not requested #6098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

dmugtasimov
Copy link
Contributor

Even if I do not request pagination in HTTP-query LimitOffsetPagination still calls self.get_count(queryset) which results in an extra SELECT count(*) FROM .... This becomes important if the query itself is comparatively heavy. Let's fix it by checking if pagination is requested first.

The pull request was not tested. Please, consider it as an issue report with proposed solution.

P.S.:

Here is my log before

2018-07-27 10:53:09,741 DEBUG django.db.backends (0.350) SELECT COUNT(*) FROM (SELECT "core_agent"."id" AS Col1, (COALESCE(EXTRACT(EPOCH FROM now() - last_online_dt) < 30, False)) AS "is_online_db" FROM "core_agent" WHERE ("core_agent"."user_id" = 14 AND "core_agent"."is_hidden" = false AND "core_agent"."id" IN (SELECT DISTINCT U0."source_id" AS Col1 FROM "core_ping" U0 INNER JOIN "core_target" U1 ON (U0."target_id" = U1."id") WHERE (U0."ts" >= 1468962000 AND U0."ts" <= 1532638799 AND U1."name" = 'mikrotik.com'))) GROUP BY "core_agent"."id", (COALESCE(EXTRACT(EPOCH FROM now() - last_online_dt) < 30, False))) subquery; args=(30, 14, False, 1468962000, 1532638799, u'mikrotik.com', 30)
2018-07-27 10:53:10,063 DEBUG django.db.backends (0.320) SELECT "core_agent"."id", "core_agent"."user_id", "core_agent"."name", "core_agent"."key", "core_agent"."location", "core_agent"."model", "core_agent"."ip_address", "core_agent"."public_ip_address", "core_agent"."version", "core_agent"."last_heard_from", "core_agent"."last_online_dt", "core_agent"."notified_online_dt", "core_agent"."notified_offline_dt", "core_agent"."gps_longitude", "core_agent"."gps_latitude", "core_agent"."module_settings", "core_agent"."tags", "core_agent"."is_active", "core_agent"."is_hidden", "core_agent"."bytes_sent", "core_agent"."bytes_received", (COALESCE(EXTRACT(EPOCH FROM now() - last_online_dt) < 30, False)) AS "is_online_db" FROM "core_agent" WHERE ("core_agent"."user_id" = 14 AND "core_agent"."is_hidden" = false AND "core_agent"."id" IN (SELECT DISTINCT U0."source_id" AS Col1 FROM "core_ping" U0 INNER JOIN "core_target" U1 ON (U0."target_id" = U1."id") WHERE (U0."ts" >= 1468962000 AND U0."ts" <= 1532638799 AND U1."name" = 'mikrotik.com'))) ORDER BY "core_agent"."id" ASC; args=(30, 14, False, 1468962000, 1532638799, u'mikrotik.com')

And after the change:

2018-07-27 11:08:45,964 DEBUG django.db.backends (0.477) SELECT "core_agent"."id", "core_agent"."user_id", "core_agent"."name", "core_agent"."key", "core_agent"."location", "core_agent"."model", "core_agent"."ip_address", "core_agent"."public_ip_address", "core_agent"."version", "core_agent"."last_heard_from", "core_agent"."last_online_dt", "core_agent"."notified_online_dt", "core_agent"."notified_offline_dt", "core_agent"."gps_longitude", "core_agent"."gps_latitude", "core_agent"."module_settings", "core_agent"."tags", "core_agent"."is_active", "core_agent"."is_hidden", "core_agent"."bytes_sent", "core_agent"."bytes_received", (COALESCE(EXTRACT(EPOCH FROM now() - last_online_dt) < 30, False)) AS "is_online_db" FROM "core_agent" WHERE ("core_agent"."user_id" = 14 AND "core_agent"."is_hidden" = false AND "core_agent"."id" IN (SELECT DISTINCT U0."source_id" AS Col1 FROM "core_ping" U0 INNER JOIN "core_target" U1 ON (U0."target_id" = U1."id") WHERE (U0."ts" >= 1468962000 AND U0."ts" <= 1532638799 AND U1."name" = 'mikrotik.com'))) ORDER BY "core_agent"."id" ASC; args=(30, 14, False, 1468962000, 1532638799, u'mikrotik.com')

@codecov-io
Copy link

codecov-io commented Jul 27, 2018

Codecov Report

Merging #6098 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master    #6098   +/-   ##
=======================================
  Coverage   96.17%   96.17%           
=======================================
  Files         128      128           
  Lines       17599    17599           
  Branches     1459     1459           
=======================================
  Hits        16926    16926           
  Misses        465      465           
  Partials      208      208

@tomchristie
Copy link
Member

This one looks sufficiently self-evident that I'm happy with it, yup.

@tomchristie tomchristie merged commit a3ae8ea into encode:master Mar 9, 2021
stefanacin pushed a commit to stefanacin/django-rest-framework that referenced this pull request Mar 22, 2021
…ncode#6098)

* Do not do `SELECT count(*) FROM ...` if pagination is not requested

* Update pagination.py

Co-authored-by: Tom Christie <[email protected]>
@tomchristie tomchristie mentioned this pull request Mar 25, 2021
sigvef pushed a commit to sigvef/django-rest-framework that referenced this pull request Dec 3, 2022
…ncode#6098)

* Do not do `SELECT count(*) FROM ...` if pagination is not requested

* Update pagination.py

Co-authored-by: Tom Christie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants