CODEWITHSUNDEEP
hadoopcassandraapache-pig
Paul C
Paul C

Reputation: 8457

Pig integrated with Cassandra: simple distributed query takes a few minutes to complete. Is this normal?

I set up a test integration of Cassandra + Pig/Hadoop. 8 nodes are Cassandra + TaskTracker nodes, 1 node is the JobTracker/NameNode.

I fired up the cassandra client and created a the simple bit of data listed in the Readme.txt in the Cassandra distribution:

  [default@unknown] create keyspace Keyspace1;
  [default@unknown] use Keyspace1;
  [default@Keyspace1] create column family Users with comparator=UTF8Type and default_validation_class=UTF8Type and key_validation_class=UTF8Type;
  [default@KS1] set Users[jsmith][first] = 'John';
  [default@KS1] set Users[jsmith][last] = 'Smith';
  [default@KS1] set Users[jsmith][age] = long(42)

Then I ran the sample pig query listed in CASSANDRA_HOME (using pig_cassandra):

grunt> rows = LOAD 'cassandra://Keyspace1/Users' USING CassandraStorage() AS (key, columns: bag {T: tuple(name, value)});
grunt> cols = FOREACH rows GENERATE flatten(columns);
grunt> colnames = FOREACH cols GENERATE $0;
grunt> namegroups = GROUP colnames BY (chararray) $0;
grunt> namecounts = FOREACH namegroups GENERATE COUNT($1), group;
grunt> orderednames = ORDER namecounts BY $0;
grunt> topnames = LIMIT orderednames 50;
grunt> dump topnames;

It took about 3 minutes to complete.

    HadoopVersion   PigVersion      UserId  StartedAt                FinishedAt                            Features
    1.0.0             0.9.1          root    2012-01-12      22:16:53     2012-01-12 22:20:22     GROUP_BY,ORDER_BY,LIMIT
Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime      MinMapTIme      AvgMapTime      MaxReduceTime   MinReduceTime   AvgReduceTime   Alias   Feature Outputs
job_201201121817_0010   8       1       12      6       9       21      21      21      colnames,cols,namecounts,namegroups,rows        GROUP_BY,COMBINER       
job_201201121817_0011   1       1       6       6       6       15      15      15      orderednames    SAMPLER 
job_201201121817_0012   1       1       9       9       9       15      15      15      orderednames    ORDER_BY,COMBINER       hdfs://xxxx/tmp/temp-744158198/tmp-1598279340,

Input(s):
Successfully read 1 records (3232 bytes) from: "cassandra://Keyspace1/Users"

Output(s):
Successfully stored 3 records (63 bytes) in: "hdfs://xxxx/tmp/temp-744158198/tmp-1598279340"

Counters:
Total records written : 3
Total bytes written : 63
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

There were no errors or warnings in the logging.

Is this normal, or is there something wrong?

Upvotes: 0

Views: 1392

Answers (1)

Brainlag
Brainlag

Reputation: 628

Yes this is normal because running a Map/Reduce job on Hadoop usually takes about 1 minute just for startup. Pig generates multiple Map/Reduce jobs dependent on the complexity of the script.

Upvotes: 2

Related Questions