Reputation: 645
I'm experiencing issues with a HiveServer2 pod in my Kubernetes cluster. The pod is failing due to readiness probe errors, specifically being unable to connect to port 10000.
Readiness probe failed: dial tcp 10.42.0.63:10000: connect: connection refused
I've attempted to modify the readiness probe configuration, but the core issue appears to be with the port 10000 binding itself.
Following is the yaml file for hiveserver2 deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hiveserver2
namespace: hadoop-ecosystem
spec:
selector:
matchLabels:
app: hiveserver2
replicas: 1
template:
metadata:
labels:
app: hiveserver2
spec:
initContainers:
- name: wait-for-metastore
image: busybox:1.36
command: ['sh', '-c', 'until nc -z hive-metastore 9083; do echo waiting for metastore; sleep 2; done;']
containers:
- name: hiveserver2
image: apache/hive:3.1.3
command:
- "/bin/bash"
- "-c"
- |
export HADOOP_CLIENT_OPTS="-Xms1g -Xmx2g -XX:+UseG1GC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
export HIVE_LOG_DIR=/tmp/hive/logs
export HADOOP_ROOT_LOGGER="DEBUG,console"
export HIVE_ROOT_LOGGER="DEBUG,console"
# Debug information
echo "Checking configurations:"
echo "HIVE_HOME: $HIVE_HOME"
echo "HADOOP_HOME: $HADOOP_HOME"
echo "HADOOP_CONF_DIR: /etc/hadoop"
# Export HADOOP_CONF_DIR
export HADOOP_CONF_DIR=/etc/hadoop
# Start HiveServer2
$HIVE_HOME/bin/hiveserver2 \
--hiveconf hive.root.logger=DEBUG,console \
--hiveconf hive.server2.thrift.port=10000 \
--hiveconf hive.server2.thrift.bind.host=0.0.0.0 \
--hiveconf hive.server2.enable.doAs=false \
--hiveconf hive.metastore.uris=thrift://hive-metastore:9083 \
--hiveconf hive.metastore.event.db.notification.api.auth=false \
--hiveconf hive.server2.logging.operation.enabled=true \
--hiveconf hive.server2.thrift.http.port=10001 \
--hiveconf hive.server2.webui.port=10002 \
--hiveconf hive.server2.webui.host=0.0.0.0 \
--hiveconf hive.server2.transport.mode=binary \
--hiveconf hive.server2.thrift.min.worker.threads=5 \
--hiveconf hive.server2.thrift.max.worker.threads=500 \
--hiveconf hive.server2.thrift.sasl.qop=auth \
--hiveconf hive.server2.authentication=NONE \
--hiveconf hive.server2.use.SSL=false
ports:
- containerPort: 10000
env:
- name: HADOOP_HOME
value: "/opt/hadoop"
- name: HADOOP_ROOT_LOGGER
value: DEBUG,console
- name: HIVE_HOME
value: "/opt/hive"
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "3Gi"
cpu: "1000m"
startupProbe:
tcpSocket:
port: 10000
failureThreshold: 30
periodSeconds: 10
initialDelaySeconds: 60
readinessProbe:
tcpSocket:
port: 10000
initialDelaySeconds: 60 # Increased from 30 to 120
periodSeconds: 20 # Increased from 10 to 20
timeoutSeconds: 5 # Added explicit timeout
failureThreshold: 6 # Increased failure threshold
volumeMounts:
- name: hive-config
mountPath: /opt/hive/conf/hive-site.xml
subPath: hive-site.xml
- name: hadoop-config
mountPath: /etc/hadoop/core-site.xml
subPath: core-site.xml
- name: hadoop-config
mountPath: /etc/hadoop/hdfs-site.xml
subPath: hdfs-site.xml
- name: hive-config
mountPath: /etc/hadoop/log4j.properties
subPath: log4j.properties
volumes:
- name: hive-config
configMap:
name: hive-config
- name: hadoop-config
configMap:
name: hadoop-config
---
apiVersion: v1
kind: Service
metadata:
name: hiveserver2
namespace: hadoop-ecosystem
spec:
selector:
app: hiveserver2
ports:
- port: 10000
targetPort: 10000
name: thrift
- port: 10002
targetPort: 10002
name: webui
type: ClusterIP
What could be preventing HiveServer2 from binding to port 10000? Are there any common configuration issues that might cause this port binding failure?
Upvotes: 0
Views: 37