Spark JDBC SQLException

Question

I keep getting SQLException but I suspect that it is not the problem. Table is :

create table person (first varchar(30) DEFAULT NULL, last varchar(30)      DEFAULT NULL, gender char(1) DEFAULT NULL, age tinyint(4) DEFAULT NULL);

Insert statements:

insert into person values('Barack','Obama','M',54);
insert into person values('Hillary','Clinton','f',34);

Spark code:

public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("Stackoverflow")
                            .setMaster("local[4]");
        JavaSparkContext sc = new JavaSparkContext(conf);
        SQLContext sqlContext = new SQLContext(sc);
        Map<String, String> options = new HashMap<>();
        options.put("url", "jdbc:mariadb://localhost:3306/persondb");
        options.put("user", "user");
        options.put("password", "password333");
        options.put("dbtable", "(select * from person where gender = 'M') as someone");

        DataFrame jdbcDF = sqlContext.read().format("jdbc"). options(options).load();
        jdbcDF.show();

Error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.sql.SQLException: Out of range value for column 'age' : value age is not in Integer range

I tried changing table stmt(@jmj):

create table person (first varchar(30) DEFAULT NULL, last varchar(30)      DEFAULT NULL, gender char(1) DEFAULT NULL, age int DEFAULT NULL);

Then it worked for some queries but mostly it is giving:

Caused by: java.sql.SQLException: Out of range value for column 'age' : value age is not in Integer range

Answer 1

The source of your problem is the use of TINYINT(4) to storage the age.

Change the type by INT insead TINYINT(4).

To understand why check this post.

Hope this Helps.