CODEWITHSUNDEEP
apache-sparktpc
user9332151
user9332151

Reputation: 57

How to generate tables in parallel with tpc-ds benchmark?

How do I run tpc-ds data generation (dsdgen) and then run queries on these data (dsqgen) in a parallel distributed mode. I am using Spark on yarn configuration (spark.master yarn) and storing data on a burst buffer storage system.

Upvotes: 0

Views: 998

Answers (1)

Mageswaran
Mageswaran

Reputation: 450

Please check my current exploration @ https://github.com/dhiraa/spark-tpcds. t There is application under t which can be used to generate data in parallel.

Or you could check out my reference @ https://github.com/maropu/spark-tpcds-datagen

In both the cases don't forget to use the option "--partition-tables" to make use of the parallel generation.

Upvotes: 0

Related Questions