Can't connect to remote MySQL server on AWS, but ssh tunnel works

Question

I am trying to set up mysql so that a user 'imbnpandmkexby' can connect to database 'de0rllo43ct314' from any remote IP address, or locally.

=========== THESE ARE THE STEPS I'VE TAKEN: ===========

1) In my MySQL config, I have commented out the bind-address line, verified that skip-networking is not in the file, and restarted mysql:

#/etc/mysql/my.cnf:
#bind-address    = 127.0.0.1

2) I added remote permissions (by using the '%') for user 'imbnpandmkexby' on the desired database 'de0rllo43ct314':

[ remote ] > mysql -u root -p
[ mysql  ] > GRANT ALL PRIVILEGES ON de0rllo43ct314.* TO 'imbnpandmkexby'@'localhost' IDENTIFIED BY 'passwordhere' WITH GRANT OPTION;
[ mysql  ] > GRANT ALL PRIVILEGES ON de0rllo43ct314.* TO 'imbnpandmkexby'@'127.0.0.1' IDENTIFIED BY 'passwordhere' WITH GRANT OPTION;
[ mysql  ] > GRANT ALL PRIVILEGES ON de0rllo43ct314.* TO 'imbnpandmkexby'@'%' IDENTIFIED BY 'passwordhere' WITH GRANT OPTION;
[ mysql  ] > FLUSH PRIVILEGES;
[ mysql  ] > select * from mysql.user\G

This outputs:

*************************** 6. row ***************************
                  Host: localhost
                  User: imbnpandmkexby
              Password: *0000000000000000000000
...
*************************** 7. row ***************************
                  Host: 127.0.0.1
                  User: imbnpandmkexby
              Password: 
...
*************************** 8. row ***************************
                  Host: %
                  User: imbnpandmkexby
              Password: 
...

3) At this point I can connect with an SSH tunnel using Sequel Pro. The user appears to have all the right permissions.

4) Next I opened a firewall port and verified that MySQL is listening on that port:

[ remote ] > sudo iptables -I INPUT 10 -p tcp --dport 3306 -j ACCEPT
[ remote ] > sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state     RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp     dpt:4505
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp     dpt:4506
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http-alt
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-agent
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-trapper
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:mysql
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere    

Since this box is hosted on amazon ec2, I also opened up port 3306 in its security group:

5) I can telnet into the port:

Trying 00.00.00.000...
Connected to ec2-00.00.00.000.us-west-2.compute.amazonaws.com.
Escape character is '^]'.

=========== THIS IS WHERE I'M STUCK: ===========

00.00.00.000 shown instead of actual IP

When I try connecting to the database from my local machine, it doesn't work:

[ local ] > mysql -u imbnpandmkexby -h 00.00.00.000 -p
[ local ] > Enter password: 
[ local ] > ERROR 2003 (HY000): Can't connect to MySQL server on '00.00.00.000' (61)

I am able to connect to a database on a dreamhost server, so it doesn't seem to be a block on my side:

[ local ] > mysql -u dreamhost_user -h mysql.dreamhostdomain.com -p
[ local ] > Enter password: 
[ local ] > Welcome to the MySQL monitor.  Commands end with ; or \g.

Is there a layer of permissions that I'm missing?

Answer 1

Okay, finally figured it out! I had a combination of two problems:

1) My SQL rule was coming after a REJECT rule in the iptables:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:4505
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:4506
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http-alt
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-agent
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-trapper
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:mysql

What I did was remove the last rule, and re-add it at index 10:

[ remote ] > iptables -vnL --line-numbers ##Prints rules along with line numbers
[ remote ] > iptables -D INPUT 14
[ remote ] > sudo iptables -I INPUT 10 -p tcp --dport 3306 -j ACCEPT

I knew this was a step in the right direction because I was now able to connect to the box via telnet ("telnet 00.00.00.000 3306")

2) The second problem I has was that my MySQL user only had a password set on the 'localhost' user, not the users with access to '127.0.0.1' or '%'. It turns out that each user-host combination needs a password. Now when I run "select * from mysql.user\G" in the MySQL console, I get:

*************************** 6. row ***************************
                  Host: localhost
                  User: imbnpandmkexby
              Password: *0000000000000000000000
...
*************************** 7. row ***************************
                  Host: 127.0.0.1
                  User: imbnpandmkexby
              Password: *0000000000000000000000
...
*************************** 8. row ***************************
                  Host: %
                  User: imbnpandmkexby
              Password: *0000000000000000000000
...

Answer 2

run tcpdump on mysql server to ensure tcp/3306 is actually getting to that box, or to see where its being blocked.

if connecting to remote tcp/3306 hangs and timeouts, its being blackholed or denied by a firewall. if it comes back right away with cant connect, its most likely making it all the way to server, but being rejected (and tcp response is returned).