CODEWITHSUNDEEP
javafileapache-sparkhadoop
kecso
kecso

Reputation: 2485

Spark - how to write files with a given permission

I tried to write some files from spark with 750 permission with the following way I updated the fs.permissions.umask-mode in the code

jsc.hadoopConfiguration().set("fs.permissions.umask-mode", "022");

It's successfully updated the default umask.

Than I tried to write some RDD on disk, but the file permissions weren't align the the mask I applied. The files didn't have the expected 750 permission.

Code example:

public class Bla {
    public static void main(String[] args) throws Exception {
    SparkConf sConf = new SparkConf().setAppName("test hadoop config ");
    JavaSparkContext jsc = new JavaSparkContext(sConf);
    JavaRDD<String> stringJavaRDD = jsc.textFile("/path/a.txt");
    stringJavaRDD.saveAsTextFile("/path/def_umask");
    System.out.println("fs.permissions.umask-mode " +
            jsc.hadoopConfiguration().get("fs.permissions.umask-mode"));
    jsc.hadoopConfiguration().set("fs.permissions.umask-mode", "022");
    System.out.println("after mod -- fs.permissions.umask-mode " +
            jsc.hadoopConfiguration().get("fs.permissions.umask-mode"));
    // < this succeed
    stringJavaRDD.saveAsTextFile("/path/updated_umask");
    // < files has the same permission as before the umask change :(

    jsc.stop();
}

What do I miss here? How should I do it? Spark 1.6.3

Upvotes: 2

Views: 10871

Answers (1)

kecso
kecso

Reputation: 2485

Actually I get the answer from another source.

--conf spark.hadoop.fs.permissions.umask-mode=022

This setting is coordinating permission in the submitted job (all writes will have this permission). I tested this and works fine.

OR

It's possible to set the permission for a given path in the code

FileSystem.get(path, conf).setPermission(...)

Upvotes: 10

Related Questions