CODEWITHSUNDEEP
google-app-enginepostmapreduce
Sergey
Sergey

Reputation: 1208

Gae MapReduce. How to send post request programmatically

I am trying to send post request to mapreduce in GAE. Google provided 

private String generateHtml(String configXml) {
return "<html>"
    + "<body>"
    + "<form action=\"/mapreduce/start\" method=\"POST\">"
    + "<textarea name=\"configuration\" rows=20 cols=80>"
    + configXml
    + "</textarea>"
    + "<input type=\"submit\" value=\"Start\">"
    + "</form>";

}

How I configure:

 String configString = ConfigurationXmlUtil.convertConfigurationToXml(config);
 String body = "configuration=" + configString;

Doing so I get null pointer exception when server tries to get parameter "configuration";

How should I build request so it would be identical to the html scenario with textarea?

Upvotes: 1

Views: 388

Answers (1)

alex
alex

Reputation: 2451

Here's how I did it once using TaskQueue:

import static com.google.appengine.api.datastore.FetchOptions.Builder.withLimit;
import static com.google.appengine.api.taskqueue.TaskOptions.Builder.withUrl;

import com.google.appengine.api.taskqueue.TaskOptions;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.Mapper;
//... other imports

// mapreduce config
Configuration conf = new Configuration(false);
conf.setClass("mapreduce.map.class", YourMapperClass.class, Mapper.class);
conf.setClass("mapreduce.inputformat.class", DatastoreInputFormat.class, InputFormat.class);

// setup callbacks if you need to know when the job is done.
//conf.set(AppEngineJobContext.DONE_CALLBACK_URL_KEY, "/job-done-callback");

// you can also specify a queue. Defaults to "default" :)
//conf.set(AppEngineJobContext.CONTROLLER_QUEUE_KEY, "mrqueue");
//conf.set(AppEngineJobContext.DONE_CALLBACK_QUEUE_KEY, "mrqueue");
//conf.set(AppEngineJobContext.WORKER_QUEUE_KEY, "mrqueue");


conf.set(DatastoreInputFormat.ENTITY_KIND_KEY, 'YourEntityKind');
String xml = ConfigurationXmlUtil.convertConfigurationToXml(conf);

TaskOptions opts = withUrl("/mapreduce/start")
    .param("configuration", xml)
    .method(TaskOptions.Method.POST)
    .header("X-Requested-With", "XMLHttpRequest");

// use our dedicated dbm4g queue
// this needs to be configured by a user
QueueFactory.getQueue("mrqueue").add(opts);

I used a non default queue to make sure my MapReduce jobs won't hit limits too much. But you can safely use the default queue too.

Upvotes: 3

Related Questions