Reputation: 737
I have a table with around 6 million rows and has many columns with lot of data.
I want to implement a search functionality on this table mostly based on date range when the row was created and some other columns.
No other table is required so no question of joins in query. Also, I have implemented indexes on date column and other relevant columns.
Still the search takes long time if I search for large ranges, considering number of rows the table has. I have increased key_buffer_size to 1G and also mysql_query_cache to 500M.
I want to know if,
I want to explore all options before increasing RAM or CPU.
UPDATE:
Below are more details
CPU: Intel Quadcore RAM: 4GB OS: CentOS
+-----------------------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------------+---------------+------+-----+---------+----------------+
| deal_id | int(11) | NO | PRI | NULL | auto_increment |
| deal_group_id | int(11) | NO | PRI | NULL | |
| site_id | int(11) | YES | MUL | NULL | |
| datetime | char(14) | YES | MUL | NULL | |
| deal_category | varchar(200) | YES | MUL | NULL | |
| vendor_name | varchar(200) | YES | | NULL | |
| area | varchar(200) | YES | MUL | NULL | |
| address | text | YES | | NULL | |
| country_code | varchar(5) | NO | MUL | NULL | |
| expired | int(1) | NO | | 0 | |
| type | varchar(20) | NO | | REGULAR | |
| pn_id | int(11) | NO | | 0 | |
+-----------------------------+---------------+------+-----+---------+----------------+
Above is the schema. In the where clause the combination of datetime
in range, deal_category
, vendor_name
in like and country_code
is used. All are indexed. Below is one sample query
SELECT distinct t.deal_id, t.deal_group_id, t.site_name, UNIX_TIMESTAMP(t.datetime) as datetime, t.deal_category as deal_category, t.vendor_name, t.area as area, t.country_code FROM table t WHERE 1 AND t.country_code = 'US' AND datetime BETWEEN '20110601' AND '20120303' limit 10;
I have increased innodb_buffer_pool_size
to around 75%, it has improved the performance for some queries, but still takes lot of time if all filters are included.
Below is the my.cnf file settings,
query_cache_size=500M
max_allowed_packet= 1G
innodb_buffer_pool_size = 3G
key_buffer_size = 1G
This table is quite busy, there are hundreds of writes happening every minute.
Hope this gives clearer picture.
More Updates
Below is the explain output and indexes on table,
+----+-------------+-------+------+-----------------------+--------------+---------+-------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+-----------------------+--------------+---------+-------+--------+-------------+
| 1 | SIMPLE | t | ref | country_code,datetime | country_code | 17 | const | 521556 | Using where |
+----+-------------+-------+------+-----------------------+--------------+---------+-------+--------+-------------+
Indexes:
+-------------+------------+-------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------------+------------+-------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| table | 0 | PRIMARY | 1 | deal_id | A | 5810775 | NULL | NULL | | BTREE | |
| table | 0 | PRIMARY | 2 | deal_group_id | A | 5810775 | NULL | NULL | | BTREE | |
| table | 1 | site_id | 1 | site_id | A | 2109 | NULL | NULL | YES | BTREE | |
| table | 1 | area | 1 | area | A | 18927 | NULL | NULL | YES | BTREE | |
| table | 1 | country_code | 1 | country_code | A | 20 | NULL | NULL | | BTREE | |
| table | 1 | deal_group_id | 1 | deal_group_id | A | 1936925 | NULL | NULL | | BTREE | |
| table | 1 | deal_category | 1 | deal_category | A | 20 | NULL | NULL | YES | BTREE | |
| table | 1 | datetime | 1 | datetime | A | 5810775 | NULL | NULL | YES | BTREE | |
+-------------+------------+-------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
Thanks
Upvotes: 2
Views: 147
Reputation: 3523
For a database running Innodb you need to increase the value of innodb_buffer_pool_size to about 70 - 80% available RAM on a dedicated server so that as much of the data as possible can fit in memory
Create a single index containing the fields that you are going to search on
Searches with LIKE %value% do not use indexes
For additional tuning you need to provide more information on the schema and searches you will be carrying out
Remove the '1 AND' comparison in the WHERE clause
Change datetime BETWEEN '20110601' AND '20120303' to DATE(datetime) BETWEEN STR_TO_DATE('20110601', '%Y%m%d') AND STR_TO_DATE('20120303', '%Y%m%d'). I am assuming that the datetime field contains dates in the MySQL format
Consider changing the datetime field from a char(14) to a datetime field for faster computations
If the country_code column only has 2 digit country codes consider changing its size to char(2)
Remove DISTINCT before deal_id column which is an autoincrement and primary key
Upvotes: 3