CODEWITHSUNDEEP
javamysqlsql
Imran Shafqat
Imran Shafqat

Reputation: 518

Optimize the sql query, too slow even on small data

Basically I am trying to get total sum of count from words matched for each url. I have this sql query:

select w.url, w.word, w.count, (
select sum(w2.count)
from wordcounts w2 where w2.url = w.url and w2.word in ('search', 'more')
) as totalcount
from wordcounts w
where w.word in ('search', 'more')

I am using this query to get this kind of result:

URL                              |  word  | count | Total Count

http://haacked.com/              | more   | 61    | 62
http://haacked.com/              | search | 1     | 62
http://feeds.haacked.com/haacked | more   | 58    | 59
http://feeds.haacked.com/haacked | search | 1     | 59
http://www.asp.net/privacy       | more   | 7     | 13
http://www.asp.net/privacy       | search | 6     | 13

My original table structure is 

ID | URL  |  word  | count

But the problem is, this small query is taking too much time. 7+ seconds to run above query on a few thousand rows. How can I optimize this query?

I got this syntax from another site but it is giving error.

select id, url, word, count, 
sum(count) over(partition by url) as count_sum
from wordcounts where word in ('search', 'more') order by url

Error code 1064, SQL state 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(partition by url) as count_sum
from wordcounts where word in ('search', 'more')' at line 2
Line 1, column 1

Execution finished after 0 s, 1 error(s) occurred.

Upvotes: 2

Views: 225

Answers (3)

Gordon Linoff
Gordon Linoff

Reputation: 1269773

Your originally query runs slowly in MySQL because MySQL is executing the subquery for each row of the result set. You can fix this by doing the aggregation once and joining the results in:

select w.url, w.word, w.count, wsum.sumcount
from wordcoutns w join
     (select w.url, w.word, SUM(w.count) as sumcount
      from wordcounts w
      where w.word in ('search', 'more')
      group by w.url, w.word
     ) wsum
     on wsum.url = w.url and wsum.word = w.word
where w.word in ('search', 'more')

Other databases support a class of functions called window functions that make this easier. MySQL does not support these.

Upvotes: 1

Frank Schmitt
Frank Schmitt

Reputation: 30775

Use a JOIN instead of a subquery:

select w.url, w.word, w.count, sum(w2.count) as totalcount 
from wordcounts w
left join wordcounts w2  
  on w2.url = w.url and w2.word in ('search', 'more')
where w.word in ('search', 'more')
group by w.url, w.word, w.count

Upvotes: 1

RichardTheKiwi
RichardTheKiwi

Reputation: 107716

Pre-aggregate:

select w.url, w.word, w.`count`, w3.totalcount
from wordcounts w
join (
     select w2.url, sum(w2.`count`) totalcount
     from wordcounts w2
     where w2.word in ('search', 'more')
     group by w2.url) w3 on w3.url = w.url
where w.word in ('search', 'more')

Upvotes: 3

Related Questions