CODEWITHSUNDEEP
java
JayC
JayC

Reputation: 2292

How to retrieve data from remote server faster in java?

I have a enhanced for loop that enters ch.ethz.ssh2.connection to obtain over 200 values. Every time it goes into the loop a new server is being authenicated and it only retrieves one value from that server. Each time it's looped the data are being saved into an arraylist to be displayed in html tables using thymeleaf. But this method takes forever for eclipse to run through all 200 values one at a time, then it have to restart when I open up localhost:8080 to load up all the tables with all the data. It takes over 5 mins to load the page up. What can I do to speed things up?

Problem in code

List<DartModel> data = new ArrayList<DartModel>();
for(String server:serverArray) {    

    try {

        conn = new ch.ethz.ssh2.Connection(server);
        conn.connect();
        boolean isAuthenticated = conn
                .authenticateWithPassword(username_array[j],
                        password_array[j]);
        if (isAuthenticated == false) {
            throw new IOException("Authentication failed.");
        }

I need to somehow recode the code above so I can obtain the data all in super quickly.

Output

Loop1: Server1
Loop2: DifferentServer2
Loop3: AllDifferentSever3
and goes on......

Alternative

I was thinking to let the java program run several times while saving the data into redis. Then Auto refresh the program, when it runs it sends the data into redis. Set an expiration time, But I was unable to get the data into thymeleaf html tables. Would this work? If so how can I display this into thymeleaf.

Upvotes: 2

Views: 362

Answers (3)

DwB
DwB

Reputation: 38300

Short answer:
Create a message protocol that sends all values in one response.

More Info:
Define a simple response message protocol. One simple example might be this:

count,value,...

count: contains the number of values returned.
value: one of the values.

Concrete simple example:

5,123,234,345,456,567

You can go bigger and define the response using json or XML. Use whatever seems best for your implementation.

Edit: My bad, this will not work if you are polling multiple servers. This solution assumes that you are retrieving 200 values from one server, not one value from 200 servers.

Upvotes: 2

Alberto
Alberto

Reputation: 841

At face value, it's hard to tell without looking at your code (recommend sharing a gist or your code repo).

I assume you are using library. In general, a single SSH2 operation will make several attemtps to authenticate a client. It will iterate over several "methods". I you are using ssh over a command line, you can see these when you use the flag -vv. If one fails, it tries the next. The java library implementation that I found appears to do the same.

In the loop you posted (assuming you loop 200 times), you'll try to authenticate 200 x (authentication method order). I suspect the majority of your execution may be burned in SSH handshakes. This can be avoided by making sure you use your connection only once and get as much as you can from your (already authenticated) opened socket.

Consider moving you connection outside the loop. If you absolutely must do ssh, and the data you are using is too large, parallelism may help some, but that will involve more coordination.

Upvotes: 1

Martin Milichovsky
Martin Milichovsky

Reputation: 730

You can query multiple servers at once (in parallel).

If your framework for remote connections is blocking (the methods you call actually wait until the response is received), you'd have to start handful of threads (one thread for one server in the edge case) to do that in parallel (which doesn't scale very well).

When you can use some Future/Promise based tool, you can do it without much overhead (convert 200 futures into one future of 200 values/responses).

Note: In case you would query single server for 200 responses, it is not good idea to do it this way, because you would flood it with too many requests at once. Then you should implement some way to get all the data by one request.

Upvotes: 4

Related Questions