Error connecting to private kubernetes cluster using oci bastion service

Question

I just created a private Kubernetes cluster on Oracle Cloud. The normal way to connect to cluster API is via the Bastion service. I've followed the exact steps as mentioned in this article: https://www.ateam-oracle.com/using-oci-bastion-service-to-manage-private-oke-kubernetes-clusters

After executing the ssh command port-forwarding (Step 4 in the article), the shell blocks as intended, but I don't get any sensible output from running kubectl:

$ kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: net/http: TLS handshake timeout

Here's the output when passing -v to ssh:

OpenSSH_8.4p1, OpenSSL 1.1.1k  25 Mar 2021
debug1: Reading configuration data /home/praj/.ssh/config
debug1: Reading configuration data /usr/etc/ssh/ssh_config
debug1: /usr/etc/ssh/ssh_config line 24: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /usr/etc/ssh/ssh_config line 26: Applying options for *
debug1: Connecting to host.bastion.ap-mumbai-1.oci.oraclecloud.com [192.29.162.226] port 22.
debug1: Connection established.
debug1: identity file /home/praj/.ssh/id_rsa type 0
debug1: identity file /home/praj/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug1: Authenticating to host.bastion.ap-mumbai-1.oci.oraclecloud.com:22 as 'ocid1.bastionsession.oc1.ap-mumbai-1.amaaaaaafvm2mgaa5inuqsfwe73eitjgead23h2avusdwryx5hlz6orz7jea'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:JTeqM8qvS9EO9reRIF/qyllvs6px8Y69LEveK9NFzZc
debug1: Host 'host.bastion.ap-mumbai-1.oci.oraclecloud.com' is known and matches the RSA host key.
debug1: Found key in /home/praj/.ssh/known_hosts:13
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: /home/praj/.ssh/id_rsa RSA SHA256:380ueVYrrzxGrkPRep4huj+pHdElPoz8iCTSYvKD5Hg explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/praj/.ssh/id_rsa RSA SHA256:380ueVYrrzxGrkPRep4huj+pHdElPoz8iCTSYvKD5Hg explicit
debug1: Server accepts key: /home/praj/.ssh/id_rsa RSA SHA256:380ueVYrrzxGrkPRep4huj+pHdElPoz8iCTSYvKD5Hg explicit
Enter passphrase for key '/home/praj/.ssh/id_rsa': 
debug1: Authentication succeeded (publickey).
Authenticated to host.bastion.ap-mumbai-1.oci.oraclecloud.com ([192.29.162.226]:22).
debug1: Local connections to LOCALHOST:6443 forwarded to remote address 10.0.0.14:6443
debug1: Local forwarding listening on 127.0.0.1 port 6443.
debug1: channel 0: new [port listener]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Connection to port 6443 forwarding to 10.0.0.14 port 6443 requested.
debug1: channel 1: new [direct-tcpip]
debug1: Connection to port 6443 forwarding to 10.0.0.14 port 6443 requested.
debug1: channel 2: new [direct-tcpip]
debug1: Connection to port 6443 forwarding to 10.0.0.14 port 6443 requested.
debug1: channel 3: new [direct-tcpip]
debug1: Connection to port 6443 forwarding to 10.0.0.14 port 6443 requested.
debug1: channel 4: new [direct-tcpip]
debug1: channel 1: free: direct-tcpip: listening port 6443 for 10.0.0.14 port 6443, connect from 127.0.0.1 port 44054 to 127.0.0.1 port 6443, nchannels 5
debug1: Connection to port 6443 forwarding to 10.0.0.14 port 6443 requested.
debug1: channel 1: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 6443 for 10.0.0.14 port 6443, connect from 127.0.0.1 port 44056 to 127.0.0.1 port 6443, nchannels 5
debug1: channel 3: free: direct-tcpip: listening port 6443 for 10.0.0.14 port 6443, connect from 127.0.0.1 port 44058 to 127.0.0.1 port 6443, nchannels 4
debug1: channel 4: free: direct-tcpip: listening port 6443 for 10.0.0.14 port 6443, connect from 127.0.0.1 port 44060 to 127.0.0.1 port 6443, nchannels 3
debug1: channel 1: free: direct-tcpip: listening port 6443 for 10.0.0.14 port 6443, connect from 127.0.0.1 port 44062 to 127.0.0.1 port 6443, nchannels 2
^Cdebug1: channel 0: free: port listener, nchannels 1
Killed by signal 2.

My cluster is running on two ARM-based nodes (A1 Flexible VM), with the default Oracle Linux 7.9 as OS, and Kubernetes version 1.20.8

Can anyone tell me where's the issue? Does it need any additional configuration to connect to Kubernetes API?

Answer 1

I was able to connect to private K8s cluster from my local machine via SSH with these steps:

1. Create a separate public subnet inside the same VCN of the cluster for the bastion target subnet.
2. Use the default public route table of the cluster network for the new subnet. This route table already has a route rule with internet gateway target that will allow traffic from local machine to the cluster API and vice versa.
3. Add security list to the subnet with egress rule of 0.0.0.0/0 as destination CIDR, allowing all ports.
4. Make sure to add the port CIDR of the subnet to the security list ingress of the subnet used by the cluster API.
5. Create the OCI bastion. Remember to use the new subnet as the target subnet.

Answer 2

Quick question:

• Assuming you have set up right ingress rules to Bastion subnet's security list and the session and Port Forwarding (SSH Tunnel) have been created, are you able to login to the worker node from your PC through simple SSH command on port 6443?

Note: You will need private key to login to the worker node and that the worker node should have been created with a public key for interactive login.