CODEWITHSUNDEEP
hadoopavro
Abhishek Pathak
Abhishek Pathak

Reputation: 1569

Hadoop giving error on avro tools concat

I'm trying to concatenate multiple avro files in an HDFS directory using avro tools. I'm following the approach as this question. However, I'm getting the following error:

Exception in thread "main" java.io.FileNotFoundException: /user/myuser/output/output.avro (No such file or directory)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
    at org.apache.avro.tool.Util.fileOrStdout(Util.java:79)
    at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:58)
    at org.apache.avro.tool.Main.run(Main.java:80)
    at org.apache.avro.tool.Main.main(Main.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Theoretically, it should work, however it does not. I feel as if avro-tools is checking the local filesystem for the file(directory?) and that is causing the error. Any pointers?

Upvotes: 0

Views: 1034

Answers (1)

Cl&#233;ment MATHIEU
Cl&#233;ment MATHIEU

Reputation: 3161

Try using Avro 1.7.5 or later or prefix your argument with "hdfs://".

See https://issues.apache.org/jira/browse/AVRO-867 for more details.

Basically, AVRO-867 rewrote the utility methods used by avro-tools to deal with the file system. They now work as you expect, ie. no longer require the hdfs:// prefix and support any file system.

Upvotes: 0

Related Questions