CODEWITHSUNDEEP
rustp2plibp2p
donaldduck
donaldduck

Reputation: 1

How to use Rust libp2p behind a LB? Advertised External IPs are ignored

I am trying to write a P2P distributed app in Rust with 3 Nodes and deploy them in K8s. But K8s has the WorkerNodes in a private network so the Peers are not directly accessible with public IPs. However, I create Network Load Balancers in front of them as entry points for each peer.

In my scenario I have 3 nodes: BootNode Peer1 Peer2 and each of them has an NBL in front of them with public IPs and ports

BootNode ->  3.218.115.60:32100   
Peer1    ->  3.218.115.61:32200  
Peer2    ->  3.218.115.62:32300

They have an NLB which helps them get data from the internet, so if I test:

#telnet 3.218.115.61 32200                                           
Trying 3.218.115.61...
Connected to 3.218.115.61
Escape character is '^]'.

This shows me they are able to receive data.

Peer1 and Peer2 is configured to know on which IP ( 3.218.115.60:32100 ) to connect to BootNode. And they manage to connect. But when they connect, they do not use NLB (that is only for Ingress) they use the NAT Gateway.

And when they (Peer1 and Peer2) connect, I try to advertise / configure their public IP Addresses with swarm.add_external_address(external_addr), where external_addr to be the NLB one (3.218.115.61:32200)

swarm.add_external_address(external_addr);

I even try to print to see what external IP addresses has the Peer configured:

let externals: Vec<Multiaddr> = swarm
                    .external_addresses()
                    .cloned() 
                    .collect();
 info!("external addresses: {:?}", externals);

And these shows:

external addresses: [/ip4/3.218.115.61/tcp/32200, /ip4/102.28.17.127/tcp/32200]

But as you see above, it adds also /ip4/102.28.17.127/tcp/32200 which is the NAT GW IP Address.

Why? It looks like the P2P library performs the following sequence:

  1. Peer1 connects to BootNode and in that connection:

    1.1. Peer1 tells BootNode its peer ID, and hopefully its public IPs.
    1.2. BootNode replies and informs Peer1 that it received the message from 102.28.17.127 (which is the NAT Gateway IP).
    1.3. Then Peer1 becomes aware of its NAT Gateway IP and automatically adds it to the external_addresses list.

  2. BootNode attempts to connect to Peer1, but only on the NAT Gateway IP rather than the advertised NLB public IP where Peer1 is actually listening. As a result, the communication does not work.

The Problem:

How can I make the Nodes (Bootnode, Peer1 and Peer2) to connect to each other on the promoted external IP Address (the NLB External IP) not the NAT detected one?

Is there a way to do it, have you run into this trouble? I am using libp2p (version = "0.53.2")

Upvotes: 0

Views: 37

Answers (0)

Related Questions