mysqld performance high cpu and memory usage

Question

I appreciate if you can help with the mysqld performance problem as seen below. top shows 2 mysqld with 99% cpu usage and 73% mem usage.

I have 6CPU's and 8GB of RAM in total. You can easily see the CPU's performance in last 24h.

I am sending the mysql tuner's output as well.

CPU usage 24h

top output

--> mysqltuner output:

[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 10.1.24-MariaDB
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: (0B)
[!!] Log file  doesn't exist
[!!] Log file  isn't readable.

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in InnoDB tables: 15G (Tables: 121)
[--] Data in MEMORY tables: 0B (Tables: 2)
[OK] Total fragmented tables: 0

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User 'kepmark_readonly@%' hasn't specific host restriction.
[!!] User 'kpm_zeus_fsm@%' hasn't specific host restriction.
[!!] User 'root@%' hasn't specific host restriction.
[--] There are 612 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 11d 19h 19m 33s (37M q [36.944 qps], 3M conn, TX: 30G, RX: 3G)
[--] Reads / Writes: 50% / 50%
[--] Binary logging is disabled
[--] Physical Memory     : 11.7G
[--] Max MySQL memory    : 8.7G
[--] Other process memory: 702.6M
[--] Total buffers: 8.3G global + 2.9M per thread (151 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 8.5G (72.61% of installed RAM)
[OK] Maximum possible memory usage: 8.7G (74.32% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (18K/37M)
[OK] Highest usage of available connections: 52% (80/151)
[OK] Aborted connections: 0.35%  (10593/3009097)
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 16M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (165 temp sorts / 34K sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 0% (1K on disk / 4M total)
[!!] Thread cache is disabled
[!!] Table cache hit rate: 11% (398 open / 3K opened)
[OK] Open file limit used: 0% (66/16K)
[OK] Table locks acquired immediately: 99% (41M immediate / 41M locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 6 thread(s).
[--] Using default value is good enough for your version (10.1.24-MariaDB)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/126.0K
[OK] Read Key buffer hit rate: 100.0% (709K cached / 50 reads)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 8.0G/15.2G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (1.171875 %): 48.0M * 2/8.0G should be equal 25%
[!!] InnoDB buffer pool instances: 6
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 100.00% (299698901815 hits/ 299706339317 total)
[!!] InnoDB Write Log efficiency: 22.18% (1601644 hits/ 7221004 total)
[OK] InnoDB log waits: 0.00% (0 waits / 5619360 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/1B
[!!] Aria pagecache hit rate: 75.8% (4K cached / 1K reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.

-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.

-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] This is a standalone server.

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Restrict Host for user@% to user@SpecificDNSorIp
    Set thread_cache_size to 4 as a starting value
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64: 
    Beware that open_files_limit (16364) variable
    should be greater than table_open_cache (2000)
    Performance should be activated for better diagnostics
    Consider installing Sys schema from https://github.com/mysql/mysql-sys
Variables to adjust:
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 4M, or use smaller result sets)
    thread_cache_size (start at 4)
    table_open_cache (> 2000)
    performance_schema = ON enable PFS
    innodb_buffer_pool_size (>= 15G) if possible.
    innodb_log_file_size * innodb_log_files_in_group should be equal to 1/4 of buffer pool size (=4G) if possible.
    innodb_buffer_pool_instances(=8)

Answer 1

"99% cpu usage" -- "You cannot tune your way out of a performance problem."

So, find the slowest query. Fix it. Rinse and repeat. After (usually) 2-3 queries, the CPU will drop to maybe 10%.

How to find them? Turn on the slowlog, with long_query_time=1.

How to fix the queries? Show us a query and EXPLAIN ... and SHOW CREATE TABLE. Meanwhile, read about 'composite' indexes.

(73% mem usage is OK.)

Answer 2

In your top output, what would be the reason for two MYSQLD's?

Based on provided tuner information, suggestions to consider for your .cnf (or .ini) in [mysqld] section. Do NOT change more than 1 item per day. Keep a copy of last 3 .cnf just in case.

Purchase more RAM, if possible, minimum of 24G would be desirable

max_connections=100   # from 151 08/17/2017, only 80 used
have_query_cache=0   # from YES mm/dd/ccyy, to avoid QC overhead
query_cache_size=0   # is 0 mm/dd/ccyy, KEEP it at 0
thread_cache_size=4   # from 0 mm/dd/ccyy, to minimize thread creation overhead
table_open_cache=3000   # from 2000 mm/dd/ccyy, suggested by tuner
open_files_limit=1024   # from 16k, only 66 in use at 24 hr mark
key_buffer_size=(calc90%)   # from 134M until Read key buffer hit rate is less than 100% reported by tuner

For a more thorough analysis, provide current RAM (nnG) SHOW GLOBAL STATUS; SHOW GLOBAL VARIABLES; SHOW ENGINE INNODB STATUS

The 2.9M per thread reported by tuner should be analyzed with the assistance of mysqlcalculator.com to compare defaults to your values as reported by

SHOW GLOBAL VARIABLES;

some items appear to be excessive.