CODEWITHSUNDEEP
hivedocker-composeprestohive-metastore
vinny
vinny

Reputation: 55

Presto Cluster unable to connect to Hive metastore service

Facing issues when building a presto cluster using docker compose which also runs a Hive metastore as a standalone service. Cluster is not able to connect to Hive metastore.

It throws an error:

coordinator       |     Caused by: org.apache.thrift.transport.TTransportException: localhost:9083: java.net.ConnectException: Connection refused (Connection refused)
coordinator       |         at io.prestosql.plugin.hive.metastore.thrift.Transport.rewriteException(Transport.java:96)
coordinator       |         at io.prestosql.plugin.hive.metastore.thrift.Transport.create(Transport.java:55)
coordinator       |         at io.prestosql.plugin.hive.metastore.thrift.DefaultThriftMetastoreClientFactory.createTransport(DefaultThriftMetastoreClientFactory.java:110)
coordinator       |         at io.prestosql.plugin.hive.metastore.thrift.DefaultThriftMetastoreClientFactory.create(DefaultThriftMetastoreClientFactory.java:97)
coordinator       |         at io.prestosql.plugin.hive.metastore.thrift.StaticMetastoreLocator.getClient(StaticMetastoreLocator.java:110)
coordinator       |         at io.prestosql.plugin.hive.metastore.thrift.StaticMetastoreLocator.createMetastoreClient(StaticMetastoreLocator.java:98)

Docker-compose.yml used to create one worker cluster with a standalone hive metastore service.

version: '3'

services:
  coordinator:
    image: "presto-coordinator:345"
    ports:
      - "8080:8080"
    container_name: "coordinator"
    command: http://coordinator:8080 coordinator
  worker0:
    image: "presto-worker:345"
    container_name: "worker0"
    ports:
      - "8081:8081"
    command: http://coordinator:8080 worker0
  mariadb:
      image: mariadb:latest
      ports:
        - 3306:3306
      environment:
        MYSQL_ROOT_PASSWORD: admin
        MYSQL_USER: admin
        MYSQL_PASSWORD: admin
        MYSQL_DATABASE: metastore_db
  hive-metastore:
      image: "hive:3.0.0"
      ports:
      - "9083:9083"
      depends_on:
        - mariadb

hive.properties in Presto cluster

connector.name=hive-hadoop2
hive.metastore.uri=thrift://localhost:9083
hive.metastore.username=hive
hive.allow-drop-table=true

metastore-site.xml

<configuration>
    <property>
        <name>metastore.thrift.uris</name>
        <value>thrift://localhost:9083</value>
        <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
    </property>
    <property>
        <name>metastore.thrift.port</name>
        <value>9083</value>
    </property>
    <property>
        <name>metastore.task.threads.always</name>
        <value>org.apache.hadoop.hive.metastore.events.EventCleanerTask,org.apache.hadoop.hive.metastore.MaterializationsCacheCleanerTask</value>
    </property>
    <property>
        <name>metastore.expression.proxy</name>
        <value>org.apache.hadoop.hive.metastore.DefaultPartitionExpressionProxy</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://mariadb:3306/metastore_db</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>admin</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>admin</value>
    </property>

    <property>
        <name>metastore.warehouse.dir</name>
        <value>/user/hive/warehouse</value>
    </property>

Port 9083 is exposed in the Hive metastore image so not sure why the connection is being refused. Can someone help please?

Upvotes: 1

Views: 1300

Answers (1)

Ohad Chaet
Ohad Chaet

Reputation: 519

Although my answer is probably too late, looks like your hive.metastore.uri property in hive.properties is incorrect. You configured it to thrift://localhost:9083, but your Metastore runs in a dedicated service inside your docker compose and not on the presto coordinator localhost. Hence, you should configure it to the Metastore DNS, aka hive.metastore.uri=thrift://hive-metastore:9083.

In addition, I tend to believe that the metastore.thrift.uris inside metastore-site.xml should be configured to thrift://0.0.0.0:9083 and not thrift://localhost:9083.

Upvotes: 0

Related Questions