CODEWITHSUNDEEP
marklogicmarklogic-8
Dixit Singla
Dixit Singla

Reputation: 2620

Is there any way to check if all the spawned tasks on the task server is completed in marklogic?

I am using ML 8.

Is there any way to check if the queue size is zero (all the spawned task got consumed) on task task server?

Why I need this:

I have to spawn lets say 10k task, these 10K task will create 10K xml files in D drive(using xdmp:save)

once these 10K files are generated I have spawn another 20k task and have to save 20K xml files in D drive, in these 20K tasks I have to make use of the results of previous 10K spawned task (10k files saved in D drive)

then spawn another 25k task, again will save 25K files in D drive and while processing need the result of previously spawned task which is 20K xml files.

and so on...

currently I am doing this manually, I mean I will spawn the first set of 10K files and will monitor the task server status at 8001 port. once the queue size reaches zero I will go to qconsole and spawn the another 20K files and so on...

I want to achieve this using code.

something like:

spawn 10K files; wait till all the spawned 10k tasks completed (this has to be checked using code); spawn another 20k tasks and so on...

Please help in the same, if more detail is required please let me know.

Upvotes: 1

Views: 534

Answers (1)

David Ennis -CleverLlamas.com
David Ennis -CleverLlamas.com

Reputation: 7770

When reading your post in its entirety, I would strongly suggest that you look into options such as:

  • CORB2 (using xdmp:save)
  • MLCP (The natural choice if you are just trying to export content as-is using a shell command and config file)
  • Data Movement SDK (a robust Java toolkit encompassing the goodness of both of the above)

But to answer the specific question in your title, see below:

Please keep in mind that as part of the share-nothing architecture of MarkLogic, the task queue is per host and not a cluster-wide number. However, since you are using query console for your current approach, you are already isolating yourself to a single host.

The actual command you want is:

xdmp:server-status()

This will give you quite a bit of information. One of the items in the return XML is called request-statuses. A count on that will give you what you are looking for in terms of the current tasks running. You can also find out the full queue size. Of course, none of this helps if there is anything else using the task queue as you would not know what is related to your processing and other items on the task queue.

The following code example loads up some tasks into the queue and demonstrates how to get the queue size and the current tasks running.

xquery version "1.0-ml";
for $x in 1 to 10
  for $y in 1 to 10
    return xdmp:spawn-function(function(){
      xdmp:sleep(1000)
    })
;

let $id := fn:data(xdmp:host-status(xdmp:host())//*:task-server/*:task-server-id)
return for $x in 1 to 10
 let $_ := xdmp:sleep(1000)
 return 
   element {"current-status"} {
     attribute {"queued"} {xdmp:server-status(xdmp:host(),$id)//*:queue-size/text()},   
     attribute {"being-processed"} {fn:count(xdmp:server-status(xdmp:host(),$id)//*:request-statuses/*)} 
}

On my machine with 16 threads for the task server, I get the following example output showing both numbers being exhausted.

<current-status queued="68" being-processed="16"/>
<current-status queued="52" being-processed="16"/>
<current-status queued="36" being-processed="16"/>
<current-status queued="20" being-processed="16"/>
<current-status queued="4" being-processed="16"/>
<current-status queued="0" being-processed="4"/>
<current-status queued="0" being-processed="0"/>
<current-status queued="0" being-processed="0"/>
<current-status queued="0" being-processed="0"/>
<current-status queued="0" being-processed="0"/>

Upvotes: 1

Related Questions