CODEWITHSUNDEEP
apache-sparkcassandraspark-cassandra-connector
Jintao Guan
Jintao Guan

Reputation: 101

How can I customize column-mapping when save RDD to Cassandra?

I am using Java to write Spark application. If I have a customized tuple, let's say class "Person". 

Class Person { 
  public String name1; 
  public String name2; 
  public String name3; 
}

And I have a

JavaRDD<Person> rdd;

Now I want to save it to Cassandra.

Assume that I have a table called "people" with three columns "name1", "name2" and "name3", "name4", ..., "name10" in Cassandra. According to the tutorial, the default column-mapping is using this code: 

javaFunctions(rdd).writerBuilder("test", "person", mapToRow(Person.class)).saveToCassandra();

This will use the default column-mapping like:

Person.name1  --> "name1"    
Person.name2  --> "name2"     
Person.name3  --> "name3"

But I want to customize the column mapping, the new mapping looks like this: 

Person.name1  --> "name3"       
Person.name2  --> "name2"  
Person.name3  --> "name1"

or even that I want to discard Person.name2

Person.name1  --> "name3"
Person.name3  --> "name1"

Anyway, I wonder if there is way to override or replace the default RowWriter?
What should I do to modify the column-mapping ?
I cannot find any good material about customized column-mapping in Java.

Upvotes: 0

Views: 1319

Answers (1)

ayan guha
ayan guha

Reputation: 1257

Please find the signature of saveTOCassandra

def saveToCassandra(keyspaceName: String, 
                    tableName: String, columns: 
                    ColumnSelector = AllColumns, 
                     writeConf: WriteConf = WriteConf.fromSparkConf(sparkContext.getConf))

Explanation:

@param table table definition used to create a new table

@param columns Selects the columns to save data to. Uses only the unique column names, and you must select at least all primary key columns. All other fields are discarded.Non-selected property/column names are left unchanged.

If I understood your need correctly, you can use the parameter "column" to achieve your result.

Upvotes: 1

Related Questions