CODEWITHSUNDEEP
javaapache-spark
Oxana Grey
Oxana Grey

Reputation: 387

how to put HashMap in BroadCast variable?

I'm trying to put HashMap in BroadCast Variable, but getting below error

The method broadcast(T, ClassTag<T>) in the type SparkContext is not applicable for the arguments (Map<String,String>, ClassTag<HashMap>)

on below code snippet

Broadcast<HashMap<String, String>> br = ss.sparkContext().broadcast(mp, classTag(HashMap.class));

complete code

        private static <T> ClassTag<T> classTag(Class<T> clazz) {
                return scala.reflect.ClassManifestFactory.fromClass(clazz);
    }

        Dataset<Row> schema = ss.read().format("csv").option("delimeter", ",").option("header", "true")
                .option("quote", "").load("D:\\Data\\schema.csv");
        
        Map<String, String> mp = new HashMap<String, String>();
        
        schema.foreach((ForeachFunction<Row>) row -> {
            String[] names = row.schema().fieldNames();
            
            for (int i = 0; i< names.length; i++) {
                mp.put(names[i], row.getAs(names[i]));
            }
                
        });
        
        Broadcast<HashMap<String, String>> br = ss.sparkContext().broadcast(mp, classTag(HashMap.class));

Am literally stuck on this. Can anyone suggest what am doing wrong ?

Upvotes: 1

Views: 379

Answers (1)

Erwan Daniel
Erwan Daniel

Reputation: 1537

You are using the Scala version of the SparkContext, you can get the Java version like this :

JavaSparkContext.fromSparkContext(sc).broadcast(new HashMap<String, String>());

No need for a class tag !

Upvotes: 2

Related Questions