CODEWITHSUNDEEP
javaapachecsv
Suganthan Madhavan Pillai
Suganthan Madhavan Pillai

Reputation: 5655

Apache spark - convert JavaRDD to csv file

I am try to write my org.apache.spark.api.java.JavaRDD<Object> as pipe separated text to .txt file using Apache spark. For that I am using the spark provided saveAsTextFile method. But issue with this method is, it directly writes the object to file with no formatting and I could not able to give proper file name too.

So which is the best way to format/convert my object into pipe separated string and write that to .txt file and also what is proper way to name file.

This is dependencies in build.gradle

dependencies {

provided(
        [group: 'org.apache.spark', name: 'spark-core_2.10', version: '1.4.0'],
        [group: 'org.apache.spark', name: 'spark-sql_2.10', version: '1.4.0'],
        [group: 'com.datastax.spark', name: 'spark-cassandra-connector-java_2.10', version: '1.4.0']
)

    compile([

            [group: 'com.databricks', name: 'spark-csv_2.10', version: '1.4.0'],
    ])


}

Upvotes: 2

Views: 1722

Answers (1)

KayV
KayV

Reputation: 13835

Here is the complete code, using which you can format the data:

    String args[] = {"/Users/***/Documents/hContent/input/***/micro-/sample.txt",
                    "Users/**/Documents/hadoop/output"}; 


    SparkConf conf = new SparkConf();
    JavaSparkContext sc = new JavaSparkContext("local", "MaxTemperatureSpark", conf);
    JavaRDD<String> lines = sc.textFile(args[0]);


    JavaRDD<String[]> records = lines.map(new Function<String, String[]>(){

        public String[] call(String t){
            return t.split("\t");
        }
    });


    JavaRDD<String[]> filtered = records.filter(new Function<String[], Boolean>() {

        public Boolean call(String[] rec) throws Exception {
            // TODO Auto-generated method stub
            return rec[0] != "9999" && rec[1].matches("[01459]");
        }
    });

    JavaPairRDD<Integer, Integer> tuples = filtered.mapToPair(
            new PairFunction<String[], Integer, Integer>() {

                public Tuple2<Integer, Integer> call(String[] rec) throws Exception {
                    // TODO Auto-generated method stub
                    return new Tuple2<Integer, Integer>(
                            Integer.parseInt(rec[0]), Integer.parseInt(rec[1]));
                }

            }

    );

    JavaPairRDD<Integer, Integer> maxTemps = tuples.reduceByKey(
                new Function2<Integer, Integer, Integer>() {

                    public Integer call(Integer arg0, Integer arg1) throws Exception {
                        // TODO Auto-generated method stub
                        return Math.max(arg0, arg1);
                    }
                }
            );

    maxTemps.saveAsTextFile(args[1]);

Note that the output folder specifies the location where the output files will be created and the name of the files starts as "part-00000". So you can parse the output folder and search for the files.

Upvotes: 1

Related Questions