Monitor AKS container/pod status from pod running in different cluster/context

Question

I have applications running in different cluster, like below (say as example)

 cluster1 - for scratch work
 cluster2 - as staging env (pods running application )
 cluster3 - as testing env (pods running application )
 cluster4 - monitor app

All the AKS cluster are connected in the network, a monitoring app is running on cluster4, which has custom monitoring dashboard. I am trying to add the container/pod status of app running on cluster2 and cluster3 to that dashboard.

Is there a way to fetch the container/pod status from the app running in cluster4 either using java/REST API/shell.

I came across Kubernetes java client InclusterClientExample.

Is there any better approach for this situation to fetch the container/pod status of the application running on different cluster.

Answer 1

Another options was to enable Log Analytics API.

• create a workspace and use REST API to fire query. below is at a high level of how to do it.

  • If AAD is setup, get the TENENT ID info.
  • Register App, create a Service principal. (provides application (client) id)
    • Within the SP, add a secret key. (note this down, since it won't be viewable later)
    • Add API Permission, select Log Analytics
      • Grant Delegate permission (Data.Read) and Application Permissiong (Data.Read) which was in my case.
  • In Log Analytics workspace, Access Control (IAM), add the Service principal and provide Reader access. ( workspace-name, resourceGroup-name, workspace-id, subscription-id to be noted)

Per the link. First we need to fetch the access_token using the SP and Tenet id. Few of the screenshot in this link is old.

Using curl command:

1. Fetch token

curl -vX POST -d 'grant_type=client_credentials&client_id=[SP application(client) id]&client_secret=[Client secret created in SP]&resource=https://management.azure.com/' https://login.microsoftonline.com/[TENENT_ID]/oauth2/token

2. Fetch log info

curl -vX post -H "Authorization: Bearer [TOKEN-FROM-ABOVE]" -H "Content-Type: application/json" -H "Prefer: response-v1=true"-d @samplequery.json https://api/loganalytics.io/v1/subscriptions/[subscription-id-of-workspace]/resourceGroups/[Resource-group-name-of-workspace]/providers/Microsoft.OperationalInsights/workspaces/<workspace-name>/api/query?api-version=2020-08-01

Sample query file

{ 
 "query": "Perf | where CounterName == 'Available MBytes' | summarize avg(CounterValue) by bin(TimeGenerated, 1h)",
  "timespan": "PT12H"
}
}

output would be the result of the query.

Answer 2

With many other options, tried out camel-kubernetes component. This uses kubernetes-client managed by fabric8.

• If using minikube, set and start the application.

• use minikube dashboard or kubectl proxy --port=8080 so the cluster can be accessed from host machine.

• add the camel-core and camel-kubernetes dependencies to the project

package com.learning.camel.examples.prog3;

import org.apache.camel.CamelContext;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.impl.DefaultCamelContext;

public class ListPodsInK8s {
    
    public static void main(String[] args) throws Exception {
        
        CamelContext context = new DefaultCamelContext();
        try {
        context.addRoutes(new RouterToAccessK8s());
        context.start();
        
        ProducerTemplate template = context.createProducerTemplate();
        template.start();
        template.sendBody("direct:input1","example");
        Thread.sleep(3000); // sleep 3 seconds
        }finally {
            context.stop();
        }       
    }
}

• code that will route to fetch the data

package com.learning.camel.examples.prog3;

import java.util.List;
import org.apache.camel.builder.RouteBuilder;
import io.fabric8.kubernetes.api.model.Pod;

public class RouterToAccessK8s  extends RouteBuilder{

    public String host = "http://127.0.0.1:8080"; // use az command to get server url
    public String authToken = "";  // fetch the token using az command in case AKS
    public String certPath = "C:\\Users\\tim\\.minikube\\profiles\\minikube\\client.crt"; //minikube 
    public String certKey = "C:\\Users\\tim\\.minikube\\profiles\\minikube\\client.key"; // minikube
    @Override
    public void configure() throws Exception {
        
        from("direct:input1")
        .toF("kubernetes-pods://%s?clientCertFile=%s&clientKeyFile=%s&namespace=default&operation=listPods", host,certPath,certKey)
        .log("pod size: ${body.size()}")
        .process(exchange -> {
            List<Pod> pods = exchange.getIn().getBody(List.class);
            System.out.println("NameSpace | PodName | Status");
            pods.stream()
            .forEach(pod -> {System.out.println(pod.getMetadata().getNamespace()+ " |  "+ pod.getMetadata().getName()+" | "+pod.getStatus().getPhase());});
        })
        .end();
    }
}

output:

NameSpace | PodName | Status
default |  ngnix | Succeeded
kube-system |  coredns-74ff55c5b-26drq | Running
kube-system |  etcd-minikube | Running
kube-system |  kube-apiserver-minikube | Running
kube-system |  kube-controller-manager-minikube | Running
kube-system |  kube-proxy-b97ss | Running
kube-system |  kube-scheduler-minikube | Running
kube-system |  storage-provisioner | Running
kubernetes-dashboard |  dashboard-metrics-scraper-c95fcf479-x4bgq | Running

For Azure, az aks get-crediential --file - can be used to fetch the master server url (host) and access token info.