CODEWITHSUNDEEP
hadoopapache-sparkhivehbasedataformat
Flibidi
Flibidi

Reputation: 153

Data format and database choices Spark/hadoop

I am working on structured data (one value per field, the same fields for each row) that I have to put in a NoSql environment with Spark (as analysing tool) and Hadoop. Though, I am wondering what format to use. i was thinking about json or csv but I'm not sure. What do you think and why? I don't have enough experience in this field to properly decide.

2nd question : I have to analyse these data (stored in an HDFS). So, as far as I know I have two possibilities to query them (before the analysis):

  1. direct reading and filtering. i mean that it can be done with Spark, for exemple:

    data = sqlCtxt.read.json(path_data)

  2. Use Hbase/Hive to properly make a query and then process the data.

So, I don't know what is the standard way of doing all this and above all, what will be the fastest. Thank you by advance!

Upvotes: 1

Views: 87

Answers (1)

Dee Carter
Dee Carter

Reputation: 433

Use Parquet. I'm not sure about CSV but definitely don't use JSON. My personal experience using JSON with spark was extremely, extremely slow to read from storage, after switching to Parquet my read times were much faster (e.g. some small files took minutes to load in compressed JSON, now they take less than a second to load in compressed Parquet).

On top of improving read speeds, compressed parquet can be partitioned by spark when reading, whereas compressed JSON cannot. What this means is that Parquet can be loaded onto multiple cluster workers, whereas JSON will just be read onto a single node with 1 partition. This isn't a good idea if your files are large and you'll get Out Of Memory Exceptions. It also won't parallelise your computations, so you'll be executing on one node. This isn't the 'Sparky' way of doing things.

Final point: you can use SparkSQL to execute queries on stored parquet files, without having to read them into dataframes first. Very handy.

Hope this helps :)

Upvotes: 2

Related Questions