CODEWITHSUNDEEP
scalaapache-spark
jaggu
jaggu

Reputation: 31

Reading comma separated text file in spark 1.6

I have a text file which is similar to below 

20190920

123456789,6325,NN5555,123,4635,890,C,9

985632465,6467,KK6666,654,9780,636,B,8

258063464,6754,MM777,789,9461,895,N,5

And I am using spark 1.6 with scala to read this text file 

val df = sqlcontext.read.option("com.databricks.spark.csv")
              .option("header","false").option("inferSchema","false").load(path)

df.show()

When I used above command to read it is reading only first column. Is there anything to add to read that file with all column values.

Output I got :

20190920

123456789

985632465

258063464

3

Upvotes: 1

Views: 725

Answers (1)

chlebek
chlebek

Reputation: 2451

In this case you should provide schema,So your code will look like this

val mySchema = StructType(
  List(
    StructField("col1", StringType, true),
    StructField("col2", StringType, true),
    // and other columns ...
  )
)

val df = sqlcontext.read
.schema(mySchema)
.option("com.databricks.spark.csv")
.option("header","false")
.option("inferSchema","false")
.load(path)

Upvotes: 1

Related Questions