CODEWITHSUNDEEP
springspring-boothivespring-jdbcjdbctemplate
Arun
Arun

Reputation: 3680

Spring boot app connecting Hive Datasource : java.sql.SQLException: org.apache.thrift.transport.TTransportException

I am connecting to a Hive datasource with my spring boot app. Below is the datasource-config

import org.apache.commons.dbcp.BasicDataSource;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.JdbcTemplate;

import javax.sql.DataSource;
import java.io.IOException;

@Configuration
public class HiveDataSourceConfig {

    @Value("${hive.url}")
    private String url;

    @Value("${hive.username}")
    private String username;

    @Value("${hive.password}")
    private String password;

    public DataSource getHiveDataSource() {

        BasicDataSource dataSource = new BasicDataSource();
        dataSource.setUrl(url);
        dataSource.setDriverClassName("org.apache.hive.jdbc.HiveDriver");
        dataSource.setUsername(username);
        dataSource.setPassword(password);

        return dataSource;
    }

    @Bean(name = "jdbcTemplate")
    public JdbcTemplate getJDBCTemplate() throws IOException {
        return new JdbcTemplate(getHiveDataSource());
    }
}

This is the JDBC-Url that i connect to

jdbc:hive2://hpeeeee.hpc.company.com:8443/;ssl=true;sslTrustStore=/Users/arun/Downloads/truststore.jks;trustStorePassword=password123;transportMode=http;httpPath=one/default/hive

I connect to hive database to get 8000+ records.. The connection works perfectly well and some times i get the below exception

java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.NoHttpResponseException

and some time i get the below exception

org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:

When i googled the reason for these exception, i can see some responses telling that it is due to the issue with the Hive-Server...

However when i restart my app (that connects to hive DB), this failure is gone and coming back after some time.

Any idea ?

Upvotes: 0

Views: 5254

Answers (1)

Amit Vyas
Amit Vyas

Reputation: 790

@Arun As one of your comment states "it works perfectly fine.. only after an idle period of certain time, i get the exception" This seems to me and connection pool issue. Due to active/idle/timeout properties after some idle time when a new request comes in it is not having a connection object properly available in the pool.

With Spring boot you can use and pool in a much better way like below then you have shown in the sample code above.

In application properties file

// These are tomcat connection pool properties.
// Sprign boot properties document link provided below 
spring.datasource.hivedb.username=
spring.datasource.hivedb.password=
spring.datasource.hivedb.url=
spring.datasource.hivedb.driver-class-name=
spring.datasource.hivedb.initial-size=<<define as per your load>>
spring.datasource.hivedb.validation-query=<<for hive select query>>
spring.datasource.hivedb.validation-query-timeout=3
spring.datasource.hivedb.test-on-borrow=true
spring.datasource.hivedb.max-wait=60000

In Java config file

@Configuration
public class HiveDataSourceConfig {

@Bean(name = "hiveDataSource")
@ConfigurationProperties(value = "spring.datasource.hivedb",ignoreUnknownFields = false)   public DataSource hiveDataSource() {
return DataSourceBuilder.create().build();
}
    
@Bean(name = "jdbcTemplate")
public JdbcTemplate jdbcTemplate(){
return new JdbcTemplate(hiveDataSource());
}
}

Spring boot data properties document link

https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#data-properties

This document provide details of properties defined by spring framework for multiple connection pools supported by the framework. Use them as a reference to tweak them.

As shown in my example application properties you need to test and tweak them as per the need.

Upvotes: 2

Related Questions