CODEWITHSUNDEEP
sql-serverapache-sparkhadoop
VirallyNerd
VirallyNerd

Reputation: 11

Export Data from Hadoop using sql-spark-connector (Apache)

I am trying to export data from Hadoop to MS SQL using Apache Spark SQL Connector as instructed here sql-spark-connector which fails with exception java.lang.NoSuchMethodError: com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer (Lcom/microsoft/sqlserver/jdbc/ISQLServerBulkRecord;)V

According to official documentation Supported Versions

My Development Environment:
Hadoop Version: 2.7.0
Spark Version: 2.4.5
Scala Version: 2.11.12
MS SQL Version: 2016

My Code:

package com.company.test

import org.apache.spark.sql.SparkSession

object TestETL {   
  def main(args:Array[String]):Unit = {  
    val spark:SparkSession = SparkSession  
      .builder()  
      .getOrCreate()  
    import spark.implicits._  
  
//  create DataFrame  
    val export_df = Seq(1,2,3).toDF("id")  
    export_df.show(5)  
  
//  Connection String  
    val server_name = "jdbc:sqlserver://ip_address:port"  
    val database_name = "database"  
    val url = server_name + ";" + "databaseName=" + database_name + ";"  
    export_df.write  
      .format("com.microsoft.sqlserver.jdbc.spark")  
      .mode("append")  
      .option("url", url)  
      .option("dbtable", "export_test")  
      .option("user", "username")  
      .option("password", "password")  
      .save()  
  }  
}

My SBT
build.sbt

Command line argument I executed
/mapr/abc.company.com/user/dir/spark-2.4.5/bin/spark-submit --class com.company.test.TestETL /mapr/abc.company.com/user/dir/project/TestSparkSqlConnector.jar

JDBC Exception

I de-compiled the mssql-jdbc-8.2.0.jre8.jar to check if it is missing the SQLServerBulkCopy.writeToServer method implementation but that doesn't see to be the case.

Any insights on how I can fix this?

Upvotes: 1

Views: 476

Answers (1)

Nassereddine BELGHITH
Nassereddine BELGHITH

Reputation: 640

it is a compatability error please to reffer to this link it will explain the error or just choose compatible versions. gitHub link

Upvotes: 0

Related Questions