MySQL Optimization - MySQL's maximum memory usage is dangerously high

Question

I ran the MySQLTuner script, and followed the recommendations that they gave, but it's still recommending increases, while telling me that MySQL's maximum memory usage is dangerously high.

• I have a CentOS 6.5 with 64 GB RAM.

• A vehicle geolocation application is deployed on this server (A huge stream of data/more than 4000 tags connected to this server)

• The size of the database exceeds 28GB (more than 18726 tables)

My current config in my my.cnf file is :

[mysqld]

server-id = 1
auto_increment_increment=2
log-slave-updates
slave-skip-errors =1146,1062,126
expire_logs_days        = 10
max_binlog_size         = 100M
relay-log = "/var/lib/mysql/mysql-relay-bin"
relay-log-index = "/var/lib/mysql/mysql-relay-bin.index"
log-error = "/var/lib/mysql/mysql.err"
master-info-file = "/var/lib/mysql/mysql-master.info"
relay-log-info-file = "/var/lib/mysql/mysql-relay-log.info"
datadir="/var/lib/mysql"
socket="/var/lib/mysql/mysql.sock"
symbolic-links=0

skip-host-cache
skip-name-resolve

open_files_limit=65535
query_cache_size=0
max_connections=500
max_user_connections=800
wait_timeout=120
tmp_table_size=32M
max_heap_table_size=32M
thread_cache_size=50
key_buffer_size=3G
max_allowed_packet=16M
table_cache=64K
join_buffer_size =32M
table_definition_cache=4096
table-open-cache=10240

myisam_sort_buffer_size=64M 
read_buffer_size=64M 
log-output=NONE 

innodb_file_per_table=1
innodb_flush_method=O_DIRECT
innodb_buffer_pool_size=38G
innodb_additional_mem_pool_size=2048M
innodb_log_buffer_size=1024M
innodb_thread_concurrency=16 
innodb_flush_log_at_trx_commit=0

[mysqld_safe]
log-error="/var/log/mysqld.log"
pid-file="/var/run/mysqld/mysqld.pid"

When I run mysqltuner.pl, this is what I get:

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mysqld.log(10B)
[OK] Log file /var/log/mysqld.log exists
[OK] Log file /var/log/mysqld.log is readable.
[OK] Log file /var/log/mysqld.log is not empty
[OK] Log file /var/log/mysqld.log is smaller than 32 Mb
[OK] /var/log/mysqld.log doesn't contain any warning.
[OK] /var/log/mysqld.log doesn't contain any error.
[--] 0 start(s) detected in /var/log/mysqld.log
[--] 0 shutdown(s) detected in /var/log/mysqld.log

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 3G (Tables: 2341)
[--] Data in InnoDB tables: 23G (Tables: 16806)
[OK] Total fragmented tables: 0

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 20m 8s (1M q [930.905 qps], 117K conn, TX: 1G, RX: 138M)
[--] Reads / Writes: 19% / 81%
[--] Binary logging is disabled
[--] Physical Memory     : 62.9G
[--] Max MySQL memory    : 92.1G
[--] Other process memory: 10.6G
[--] Total buffers: 44.0G global + 98.5M per thread (500 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 54.9G (87.30% of installed RAM)
[!!] Maximum possible memory usage: 92.1G (146.50% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (5/1M)
[OK] Highest usage of available connections: 22% (113/500)
[OK] Aborted connections: 0.00%  (3/117330)
[!!] Query cache is disabled
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 6K sorts)
[!!] Joins performed without indexes: 133
[OK] Temporary tables created on disk: 0% (541 on disk / 233K total)
[OK] Thread cache hit rate: 99% (132 created / 117K connections)
[!!] Table cache hit rate: 7% (10K open / 129K opened)
[OK] Open file limit used: 3% (2K/65K)
[OK] Table locks acquired immediately: 99% (304K immediate / 304K locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.8% (605M used / 3B cache)
[OK] Key buffer size / total MyISAM indexes: 3.0G/169.9M
[OK] Read Key buffer hit rate: 99.3% (1M cached / 10K reads)
[!!] Write Key buffer hit rate: 0.3% (24K cached / 24K writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 16
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 38.0G/23.5G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (0.0256990131578947 %): 5.0M * 2/38.0G should be equal 25%
[!!] InnoDB buffer pool instances: 1
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 95.56% (9270734 hits/ 9701289 total)
[OK] InnoDB Write log efficiency: 98.14% (65038 hits/ 66268 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1230 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.

-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.

-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[!!] This replication slave is not running but seems to be configured.

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    MySQL started within last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64

    Beware that open_files_limit (65535) variable
    should be greater than table_open_cache (10240)
    Performance should be activated for better diagnostics
    Consider installing Sys schema 

Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    query_cache_size (>= 8M)
    join_buffer_size (> 32.0M, or always use indexes with joins)
    table_open_cache (> 10240)
    performance_schema = ON enable PFS
    innodb_log_file_size * innodb_log_files_in_group should be equals to 1/4 of buffer pool size (=19G) if possible.
    innodb_buffer_pool_instances(=38)

What can I do to solve this problem? Thank you in advance

Answer 1

There is no valid formula for the maximum memory usage that MySQL could achieve. There are practical considerations.

• The tuner cannot make real judgments on a system that has been up only 20 minutes. Wait a day. (20m may explain "Table cache hit rate: 7%". And others.)
• You probably should move all of your tables to InnoDB. Tips.
• For production machines it is usually good to have the Query cache turned off.
• "Joins performed without indexes: 133" -- This is not necessarily bad. Are the tables tiny? Are the queries "fast enough"?
• "slave-skip-errors =1146,1062,126" -- This sweeps under the rug things that should be fixed.
• 19K tables? That may be an issue; what is going on?
• "InnoDB buffer pool instances: 1" -- 16 is preferred in your case.
• "This replication slave is not running but seems to be configured." -- Is it a slave?
• "Reduce your overall MySQL memory footprint for system stability" -- I don't know where it is headed with this.
• "Enable the slow query log to troubleshoot bad queries" -- I agree, also set long_query_time = 2. Then, let's see some of the worst queries.
• Since you are running a significant mixture of MyISAM and InnoDB, I would lower innodb_buffer_pool_size to 25G. This will silence some of tuner's complaints and be a little safer on memory, while not affecting performance much.

For another analysis, provide SHOW VARIABLES; and SHOW GLOBAL STATUS; after being up at least a day. I will investigate dozens of things that tuner misses.