CODEWITHSUNDEEP
javamultithreadingspring-boottwittertwitter4j
stzoannos
stzoannos

Reputation: 938

Twitter4j - AsyncTwitter: Associate the response in the TwitterListener with the initiating request

I have implemented a Spring Boot HTTP method to allow fetching 5000 followers for any Twitter account.

In the backend, first, I am retrieving the 5000 follower Ids with 

IDs ids = twitter.getFollowersIDs(username, -1);

After that, I am dividing the 5000 ids in chunks of 100.

Then, I am using AsyncTwitter to send several "lookupUsers" HTTP requests to Twitter asynchronously and I pause the thread with LOCK.wait();

Then, with the following listener method: 

    @Override
    public void lookedupUsers(ResponseList<User> users)

I read each response and add the Users list returned in a class property "followerAccounts".

In the end, when I get back all the responses, I unblock the main thread with LOCK.notify() and I am returning the "followerAccounts" as a response of the HTTP method.

The problem is that when two external users call the same spring HTTP method simultaneously, there is no way to check in the TwitterListener inside any "lookedupUsers" method which of the two HTTP requests it is referring to.

Is there any way to get e.g. some headers of the Twitter HTTP response inside lookedupUsers(ResponseList users) method or any way to pass parameters that indicate somehow the "lookupUsers" request that is linked to the response?

I don't want to make my HTTP method synchronized because then it doesn't scale up to support a big number of parallel requests.

Upvotes: 1

Views: 149

Answers (1)

Mikita Harbacheuski
Mikita Harbacheuski

Reputation: 2253

Unfortunately, twitter async api doesn't look flexible enough to easily meet your needs. All of its methods return nothing (void) so it's not possible to collect responses (Futures) for a particular http request. In fact, it provides only global listeners for all api methods instead for finer-grained callbacks for each method. Otherwise it would be possible to provide a separate callback instance for each http request.

It's still possible to separate responses for each http request using only global listener, but it requires a lot of shared so thread-safe state with pretty complex logic and i don't think it will be effective. Instead i suggest not to use this async api at all and provide your own implementation on top of sync api. Create a separate task for each chunk of ids, submit it to a thread pool and save futures to a list. Than iterate over them and collect all results. It will provide you more control over concurrency level, exception handling and timeouts for a particular twitter api method than existing implementation (AsyncTwitterImpl and DispatcherImpl) which provides a single fixed thread pool (with only 1 thread by default) for all api methods, swallows all exceptions and doesn't provide any timeouts.

Upvotes: 2

Related Questions