CODEWITHSUNDEEP
apache-spark
psmith
psmith

Reputation: 1813

saveAstextFile is Spark on Windows

I know it's unusual configuration, but I I have Spark installed on Windows just for tests

I can easily use 

val file_in = sc.textFile("UDP_file.txt")

on local filesystem. But when I try to use 

names.saveAsTextFile("file:///file_out.txt")

I get an error:

14/08/21 13:06:12 ERROR FileOutputCommitter: Mkdirs failed to create file:/file_out.txt/_temporary/0

but actualy the path is created:

file_out.txt\_temporary\0\_temporary\attempt_201408211306_0000_m_000000_34

and with file part-00000 insite the catalog.

Is it possible to use saveAsTextFile method on Spark installed on Windows?

Regards
Pawel

Upvotes: 1

Views: 2118

Answers (1)

Devan M S
Devan M S

Reputation: 702

Instead of "file:///file_out.txt" try "file_out.txt".

It will create one folder (not file) "file_out.txt", so you can specify "file_out" also as the name.

Inside that folder you can see part-000* files.

Upvotes: 2

Related Questions