CODEWITHSUNDEEP
scalaapache-flink
igx
igx

Reputation: 4231

flink what is the equivalent to parseQuotedStrings in scala api

I am trying to convert this jave code to scala:

DataSet<Tuple3<Long, String, String>> lines = env.readCsvFile("movies.csv")
            .ignoreFirstLine()
            .parseQuotedStrings('"')
            .ignoreInvalidLines()
            .types(Long.class, String.class, String.class);

to scala. I couldn't find any alternative in scala to parseQuotedStrings I will appreciate any assistance here

Upvotes: 1

Views: 104

Answers (1)

Biswanath
Biswanath

Reputation: 9185

This is following code uses flink's java api, literal translation of the code provided by you.

import org.apache.flink.api.java._

val env = ExecutionEnvironment.getExecutionEnvironment
val movies = env.readCsvFile("movies.csv")
  .ignoreFirstLine()
  .parseQuotedStrings('"')
  .ignoreInvalidLines()
  .types(classOf[Long], classOf[String], classOf[String])

Also you can use flink's scala api, something like this 

import org.apache.flink.api.scala._

val env = ExecutionEnvironment.getExecutionEnvironment
val movies = env.readCsvFile[(Int,String,String)]
  ("movies.csv", ignoreFirstLine = true, quoteCharacter = '"', lenient = true)

AFAIK Scala api does not have the fluent api of the java version. "lenient" options is same as "ignoreInvalidLines" and the other options should be self explanatory.

Upvotes: 2

Related Questions