CODEWITHSUNDEEP
amazon-s3apache-sparkhdfs
member555
member555

Reputation: 807

Running Spark application using HDFS or S3

In my spark application, I just want to access a big file, and distribute the computation across many nodes on EC2.

Initially, my file is stored on S3.

It's very convenient for me to load the file with sc.textFile() function from S3.

However, I can put some efforts to load the data to HDFS and then read the data from there.

My question is, will the performance be better with HDFS?

My code involves the spark partitions(mapPartitions transforamtion), so does it really matter what is my initial file system?

Upvotes: 1

Views: 459

Answers (1)

vanekjar
vanekjar

Reputation: 2406

Obviously when using S3 the latency is higher and the data throughput is lower compared to HDFS on local disk.

But it depends what you do with your data. It seems most of programs are limited more by CPU power than network throughput. So you should be fine with the 1Gbps throughput that you get from S3.

Anyway you can check recent slides from Aaron Davidson's talk on Spark Summit 2015. This topic is discussed there.

http://www.slideshare.net/databricks/spark-summit-eu-2015-lessons-from-300-production-users/16

Upvotes: 2

Related Questions