Reputation: 659
Here's the thing: I have an application server, called A, and a DB server, called B.
On B(DB), when I run netstat -ntp | grep 'A's IP' | wc -l
, I see over 400 TCP connections.
And when I connected to MySQL using MySQL client, and run select count(*) from processlist where host like 'A's IP%'
, I see over 400 results too, so it seems that both MySQL & OS agree there are over 400 open connections to A.
However, when I logon to A, run netstat -ntp | grep 'B's IP' | wc -l
, I see only over 100 TCP connections.
So how can this happen? Am I using netstat
wrong, or what? I just don't see why two hosts disagree on how many TCP connections that are between them.
Quick update:
We actually have 6 different projects on A(Some are Rails 3.2, Some are Rails 2.3), one of them redmine. I grab the show processlist
result on B, comparing it to netstat -ntp
on A, found out that the unmatched connections on B is from different projects, including redmine.
PS: Both OS is CentOS 5.4, MySQL is Percona 5.1.57-rel12.8-log
PS2: We rebooted all the Servers/Hardwares, but the problem's still there. Since it came out totally out of the blue, I'm guessing the problem/bug's always there, but we didn't notice it until it reached the 1000 max connections we set. So any wild guesses? Can it be an application code bug(activerecord or mysql gem)? OS bug? or what?
Upvotes: 1
Views: 1101
Reputation: 29759
Applications on A probably do not close their connections to MySQL properly on completion.
When a process terminates on A, the connections they own must be discarded automatically by the OS, but B is still awaiting for a signal from A.
B will eventually release these connections when the (MySQL) time-out delay is reached.
It is hard to be more specific without knowing more about the application on A, but there is nothing to be surprised of here (unless you didn't expect applications on A to be broken :)
Upvotes: 1