CODEWITHSUNDEEP
javascriptajaxasynchronousdojo
Mojimi
Mojimi

Reputation: 3161

Call hierarchy of async functions inside a loop?

There's a async call I'm making that queries a database on a service, but this service has a limit of how many it can output at once, so I need to check if it hit its limit through the result it sends and repeat the query until it doesn't.

Synchronous mockup :

var query_results = [];

var limit_hit = true; #While this is true means that the query hit the record limit
var start_from = 0; #Pagination parameter

while (limit_hit) {
    Server.Query(params={start_from : start_from}, callback=function(result){
        limit_hit = result.limit_hit;
        start_from = result.results.length;
        query_result.push(result.results);
    }
}

Obviously the above does not work, I've seen some other questions here about the issue, but they don't mention what to do when you need each iteration to wait for the last one to finish and you don't know before hand the number of iterations.

How can I turn the above asynchronous? I'm open to answers using promise/deferred-like logic, but preferably something clean.

I can probably think of a monstruous and horrible way of doing this using waits/timeouts, but there has to be a clean, clever and modern way to solve it.

Another way is to make a "pre-query" to know the number of features before hand so you know the number of loops, I'm not sure if this is the correct way.

Here we use Dojo sometimes, but the examples I found does not explain what to do when you have an unknown amount of loops https://www.sitepen.com/blog/2015/06/10/dojo-faq-how-can-i-sequence-asynchronous-operations/

Upvotes: 7

Views: 522

Answers (5)

Josh Lin
Josh Lin

Reputation: 2437

although many answers already, still I believe async/await is the cleanest way.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

and you might need babel

https://babeljs.io/

JS async logic syntax changed from callback to promise then to async/await, they all do the same thing, when callback nests a lot we need something like a chain, then promise come, when promise goes in loop, we need something make the chain more plain more simple, then async/await come. But not all browsers support the new syntax, so babel come to compile new syntax to old syntax, then you can always code in new syntax.

getData().then((data) => {
  //do something with final data
})

async function getData() {
  var query_results = [];

  var limit_hit = true;
  var start_from = 0;
  
  //when you use await, handle error with try/catch
  try {
    while (limit_hit) {
      const result = await loadPage(start_from)
      limit_hit = result.limit_hit;
      start_from = result.results.length;
      query_result.push(result.results);
    }
  } catch (e) {
    //when loadPage rejects
    console.log(e)
    return null
  }
  return query_result
}

async function loadPage(start_from) {
  //when you use promise, handle error with reject
  return new Promise((resolve, reject) => Server.Query({
    start_from
  }, (result, err) => {
    //error reject
    if (err) {
      reject(err)
      return
    }
    resolve(result)
  }))
}

Upvotes: 4

MR0
MR0

Reputation: 164

You must await for the server response every time. Here a encapsulated method

var query = (function(){
  var results = [];
  var count = 0;
  return function check(fun){
    Server.Query({ start_from: count}, function(d){
      count = d.results.length;
      results.push(d.results);
      if (d.limit_hit && fun) fun(results);
      else check(fun);
    });
  };
})();

// Call here
var my_query = query(function(d){
  // --> retrive all data when limit_hit is true)
});

Upvotes: 0

user5886944
user5886944

Reputation: 174

You don't understand callbacks until you have written a rate limiter or queue ;) The trick is to use a counter: Increment the counter before the async request, and decrement it when you get the response, then you will know how many requests are "in flight".

If the server is choked you want to put the item back in the queue.

There are many things you need to take into account: What will happen to the queue if the process is killed ? How long to wait before sending another request ? Make sure the callback is not called many times ! How many times should you retry ? How long to wait before giving up ? Make sure there are no loose ends ! (callback is never called)

When all edge cases are taken into account you will have a rather long and not so elegant solution. But you can abstract it into one function! (that returns a Promise or whatever you fancy). If you have a user interface you also want to show a loading bar and some statistics!

Upvotes: 2

Faaiz Khan
Faaiz Khan

Reputation: 332

You can use a generator function Generators to achieve this For POC:

some basics - You define a generator with an asterick * - it exposes a next function which returns the next value - generators can pause with yield statement internally and can resume externally by calling the next() - While (true) will ensure that the generator is not done until limit has reached

function *limitQueries() {
  let limit_hit = false;
  let start_from  = 0;
  const query_result = [];

  while (true) {
    if (limit_hit) {break;}
    yield Server.Query(params={start_from : start_from}, 
        callback=function* (result) {
        limit_hit = result.limit_hit;
        start_from = result.results.length;
        yield query_result.push(result.results);
    }
  }
}

So apparently, the generator function maintains its own state. Generator function exposes two properties { value, done } and you can call it like this

const gen = limitQueries();
let results = [];
let next = gen.next();

while(next.done) {
  next = gen.next();
}
results = next.value;

You might have to touch your Server.Query method to handle generator callback. Hope this helps! Cheers!

Upvotes: -1

Lukilas
Lukilas

Reputation: 176

If you want to use a loop then I think there is no (clean) way to do it without Promises.

A different approach would be the following:

var query_results = [];

var start_from = 0;

funciton myCallback(result) {  
  if(!result) {
    //first call
    Server.Query({ start_from: start_from}, myCallback);
  } else {
    //repeated call
    start_from = result.results.length
    query_result.push(result.results);
    
    if(!result.limit_hit) {
      //limit has not been hit yet
      //repeat the query with new start value
      Server.Query({ start_from: start_from}, myCallback);
    } else {
        //call some callback function here
    }
  }
}

myCallback(null);

You could call this recursive, but since the Query is asynchronous you shouldn't have problems with call stack limits etc.

Using promises in an ES6 environment you could make use of async/await. Im not sure if this is possible with dojo.

Upvotes: 3

Related Questions