CODEWITHSUNDEEP
scalacsvapache-sparkapache-spark-sql
whatsinthename
whatsinthename

Reputation: 2157

Unable to create dataframe using SQLContext object in spark2.2

I am using spark 2.2 version on Microsoft Windows 7. I want to load csv file in one variable to perform SQL related actions later on but unable to do so. I referred accepted answer from this link but of no use. I followed below steps for creating SparkContext object and SQLContext object: 

import org.apache.spark.SparkContext  
import org.apache.spark.SparkConf  
val sc=SparkContext.getOrCreate() // Creating spark context object 
val sqlContext = new org.apache.spark.sql.SQLContext(sc) // Creating SQL object for query related tasks

Objects are created successfully but when I execute below code it throws an error which can't be posted here. 

val df = sqlContext.read.format("csv").option("header", "true").load("D://ResourceData.csv")

And when I try something like df.show(2) it says that df was not found. I tried databricks solution for loading CSV from the attached link. It downloads the packages but doesn't load csv file. So how can I rectify my problem?? Thanks in advance :)

Upvotes: 2

Views: 2664

Answers (3)

whatsinthename
whatsinthename

Reputation: 2157

I solved my problem for loading local file in dataframe using 1.6 version in cloudera VM with the help of below code: 

1) sudo spark-shell --jars /usr/lib/spark/lib/spark-csv_2.10-1.5.0.jar,/usr/lib/spark/lib/commons-csv-1.5.jar,/usr/lib/spark/lib/univocity-parsers-1.5.1.jar  

2) val df1 = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("treatEmptyValuesAsNulls", "true" ).option("parserLib", "univocity").load("file:///home/cloudera/Desktop/ResourceData.csv")

NOTE: sc and sqlContext variables are automatically created But there are many improvements in the latest version i.e 2.2.1 which I am unable to use because metastore_db doesn't gets created in windows 7. I ll post a new question regarding the same.

Upvotes: 3

ReKx
ReKx

Reputation: 1066

Spark version 2.2.0 has built-in support for csv.

In your spark-shell run the following code

val df= spark.read
             .option("header","true")
             .csv("D:/abc.csv")

df: org.apache.spark.sql.DataFrame = [Team_Id: string, Team_Name: string ... 1 more field]

Upvotes: 0

deadbug
deadbug

Reputation: 444

In reference with your comment that you are able to access SparkSession variable, then follow below steps to process your csv file using SparkSQL.

Spark SQL is a Spark module for structured data processing.

There are mainly two abstractions - Dataset and Dataframe :

A Dataset is a distributed collection of data.

A DataFrame is a Dataset organized into named columns. In the Scala API, DataFrame is simply a type alias of Dataset[Row].

With a SparkSession, applications can create DataFrames from an existing RDD, from a Hive table, or from Spark data sources.

You have a csv file and you can simply create a dataframe by doing one of the following:

  • From your spark-shell using the SparkSession variable spark:

    val df = spark.read .format("csv") .option("header", "true") .load("sample.csv")

  • After reading the file into dataframe, you can register it into a temporary view.

    df.createOrReplaceTempView("foo")

  • SQL statements can be run by using the sql methods provided by Spark

    val fooDF = spark.sql("SELECT name, age FROM foo WHERE age BETWEEN 13 AND 19")

  • You can also query that file directly with SQL:

    val df = spark.sql("SELECT * FROM csv.'file:///path to the file/'")

  • Make sure that you run spark in local mode when you load data from local, or else you will get error. The error occurs when you have already set HADOOP_CONF_DIR environment variable,and which expects "hdfs://..." otherwise "file://".

  • Set your spark.sql.warehouse.dir (default: ${system:user.dir}/spark-warehouse).

    .config("spark.sql.warehouse.dir", "file:///C:/path/to/my/")

It is the default location of Hive warehouse directory (using Derby) with managed databases and tables. Once you set the warehouse directory, Spark will be able to locate your files, and you can load csv.

Reference : Spark SQL Programming Guide

Upvotes: 2

Related Questions