CODEWITHSUNDEEP
urlftpapache-spark
Pawan B
Pawan B

Reputation: 4613

Read csv file in Apache Spark from remote location (ftp)

I have a file in a Ubuntu machine which I want to read in Apache spark .

I found this example :

object BasicTextFromFTP {
    def main(args: Array[String]) {
      val conf = new SparkConf
      conf.setMaster(args(0))
      val sc = new SparkContext(conf)
      val file = sc.textFile("ftp://anonymous:[email protected]/ubuntu/ls-LR.gz")
      println(file.collect().mkString("\n"))
    }
}

on this link :

https://github.com/databricks/learning-spark/blob/master/src/main/scala/com/oreilly/learningsparkexamples/scala/BasicLoadTextFromFTP.scala

I don’t understand how the URL is created. Please help me with this.

Upvotes: 0

Views: 4419

Answers (1)

zero323
zero323

Reputation: 330063

Basic structure of the URL is a schema type (here ftp) followed by 

//<user>:<password>@<host>:<port>/<url-path>

where every part excluding host can be omitted.

Upvotes: 3

Related Questions