CODEWITHSUNDEEP
hadooplzo
Brugere
Brugere

Reputation: 436

Hadoop compression : "Loaded native gpl library" but "Failed to load/initialize native-lzo library"

after several try installing Lzo compression for hadoop, I need help because I have really no idea why it doesn't work.

I'using hadoop 1.0.4 on CentOs 6. I tried http://opentsdb.net/setup-hbase.html, https://github.com/kevinweil/hadoop-lzo and some others but i'm still getting error :

13/07/03 19:52:23 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/03 19:52:23 WARN lzo.LzoCompressor: java.lang.NoSuchFieldError: workingMemoryBuf

13/07/03 19:52:23 ERROR lzo.LzoCodec: Failed to load/initialize native-lzo library

even if native gpl is loaded. I've updated my mapred-site and core-site according to links below, I've copy/paste libs in right path (still according to links).

The real problem is that the lzo test works on the namenode :

13/07/03 18:55:47 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/03 18:55:47 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev ]

I've try setting several path in haddop-env.sh but there seems to be no right solution... So, if you have any idea, link ... ? I'm really interested

[edit] after a week, i'm still trying to make it functionnal. I've try sudhirvn.blogspot.fr/2010/08/hadoop-lzo-installation-errors-and.html but removing all Lzo and gplcompression libraries and then making a nez install was not better at all.

Is that due to my hadoop core version ? Is it possible to have hadoop-core-0.20 and hadoop-core-1.0.4 at the same time ? Should i compile Lzo on a 0.20 hadoop in order to use lzo ? By the way I already tried compiling hadoop-lzo like this :

CLASSPATH=/usr/lib/hadoop/hadoop-core-1.0.4.jar CFLAGS=-m64 CXXFLAGS=-m64 ant compile-native tar

If it helps the full error is :

    INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
    WARN lzo.LzoCompressor: java.lang.NoSuchFieldError: workingMemoryBuf
    ERROR lzo.LzoCodec: Failed to load/initialize native-lzo library
    INFO lzo.LzoIndexer: [INDEX] LZO Indexing file test/table.lzo, size 0.00 GB...
    WARN snappy.LoadSnappy: Snappy native library is available
    INFO util.NativeCodeLoader: Loaded the native-hadoop library
    INFO snappy.LoadSnappy: Snappy native library loaded
    Exception in thread "main" java.lang.RuntimeException: native-lzo library not available
            at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:87)
            at com.hadoop.compression.lzo.LzoIndex.createIndex(LzoIndex.java:229)
            at com.hadoop.compression.lzo.LzoIndexer.indexSingleFile(LzoIndexer.java:117)
            at com.hadoop.compression.lzo.LzoIndexer.indexInternal(LzoIndexer.java:98)
            at com.hadoop.compression.lzo.LzoIndexer.index(LzoIndexer.java:52)
            at com.hadoop.compression.lzo.LzoIndexer.main(LzoIndexer.java:137)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

I really want to use lzo because I have to deal with very large files on a rather small cluster (5 nodes). Having splittable compressed files could make it run really fast. Every remark or idea is welcome.

Upvotes: 2

Views: 5541

Answers (2)

hobgoblin
hobgoblin

Reputation: 935

I had this same issue on my OSX Machine. The problem was solved when I removed hadoop-lzo.jar (0.4.16) from my classpath and put hadoop-gpl-compression jar instead.

Upvotes: 2

Carl Sagan
Carl Sagan

Reputation: 982

I was having the exact same issue and finally resolved it by randomly choosing a datanode, and checking whether lzop was installed properly.

If it wasn't, I did:

sudo apt-get install lzop

Assuming you are using Debian-based packages.

Upvotes: 2

Related Questions