Why can't I ping the remote pox controller on port 6633 or port 6533 from mininet?

Question

Please help me troubleshoot my painpoint. I have mininet image running on a VM installed on Virtualbox that comes preinstalled with pox(ie the mininet vm image) and I have tried to run a simple mininet topology with one switch and four hosts with the controller set to remote on port 6633. I have pox controller running remotely on another terminal and when I try to ping the hosts using both pingall and the simple ping, the hosts are unable to reach each other. However, when I run mininet topology with the default controller ie local, I'm able to ping successfully with both commands. This has led me to understand that the remote controller is unreachable a fact further corroborated by the switch unable to reach the controller running on port 6633 or when I use port 6533 I get the same flag when topology is being created even if the remote controller is already up and running on the other terminal.

I have tried to check my host's firewall settings and enabled a few rules on both the inbound and outbound settings per the suggestions made online to no avail. I have tried to clear the controllers every time they have failed to connect using sudo fuser -k 6633/tcp or sudo fuser -k 6533/tcp before starting all over again. I have done the cleaning of mininet topologies through sudo mn -c before running new topologies. I have switched between mininet installations on VB since I have both mininet vm image as well the mininet standalone packages, etc. Nothing seems to work. However when I run mininet with default controller Im able to successfully ping and pingall. Just to add that I have been running pox with loadbalancer component in these instances. However when I run pox with the other components like l2 learning switch the controller is able to connect remotely. My loadbalancers are based of the default random loadbalancer in pox.

What could really be the problem with my setup?