CODEWITHSUNDEEP
hadoophbasehortonworks-data-platformapache-phoenixambari
e123
e123

Reputation: 11

HORTONWORKS - Hbase/Phoenix - WALEditCodec - missing

I am receiving the following error while trying to run Phoenix on top of Hbase:

EXCEPTION #1:

2017-11-07 12:40:12,620 WARN  [RS_LOG_REPLAY_OPS-XXX:16020-0] 
regionserver.SplitLogWorker: log splitting of 
WALs/XXX.XXX.XXX.XXX,16020,1507179047656-
splitting/XXX.XXX.XXX.XXX%2C16020%2C1507179047656.default.1507179049782 failed, returning error
java.io.IOException: Cannot get log reader
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:355)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:267)
    at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:839)
    at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:763)
    at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
    at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
    at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
    at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
    at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
:$
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:355)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:267)
    at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:839)
    at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:763)
    at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
    at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
    at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
    at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
    at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unable to find  org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
    at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:36)
    at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:103)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:297)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:307)
    at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:164)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
    ... 11 more
Caused by: java.lang.ClassNotFoundException:  org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:32)
    ... 17 more

APPLIED PATCHES #1: I have applied the following settings through the Ambari web UI for the advanced Hbase configs as specified by the Hortonworks document: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-upgrade/content/configure-phoenix-25.html

EXCEPTION #2

FATAL [RS_LOG_REPLAY_OPS-XXX:16020-1] conf.Configuration: error parsing conf core-site.xml
java.io.FileNotFoundException: /etc/hadoop/2.6.1.0-129/0/core-site.xml (Too many open files)

APPLIED PATCHES #2 I checked each 'core-site.xml' file on each server that contained a Hbase region server and made sure it ended with a </configuration>. As well as the core-site.xml files in the specified directory '/etc/hadoop/2.6.1.0-129/0/core-site.xml'

Haven't been able to find any other information regarding this issue.

Upvotes: 0

Views: 518

Answers (1)

e123
e123

Reputation: 11

I went into the HDFS and deleted all WAL split logs using the following command:

hdfs dfs -rm -r /apps/hbase/data/WALs/*splitting*

This resolved exception #1. Keep in mind from what I've read this will incur data loss.

For exception #2 I went back and checked the open file limits for each server (ulimit -n) and updated where applicable with respect to the Hortonworks doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/kerb-config-limits.html

Upvotes: 0

Related Questions