CODEWITHSUNDEEP
pythonhadoophdfs
Selin
Selin

Reputation: 646

copying file to hdfs using pydoop

i'm trying to write python script for copying file to hdfs. i'm working on ubuntu and installed hadoop and pydoop. The following code is my script:

import pydoop.hdfs as hdfs

class COPYTOHDFS():

    local_path = '/home/user/test.txt'
    hdfs_path = '/testfile'
    host = 'master'
    port = 9000
    hdfsobj = hdfs.hdfs(host, port, user='cloudera-user', groups=['supergroup'])
    hdfsobj.copy(local_path, hdfsobj, hdfs_path)

And error is here :

Traceback (most recent call last):
  File "COPYTOHDFS.py", line 3, in <module>
    class COPYTOHDFS():
  File "COPYTOHDFS.py", line 10, in COPYTOHDFS
    hdfsobj.copy(local_path, hdfsobj, hdfs_path)
  File "/usr/local/lib/python2.7/dist-packages/pydoop-0.5.2_rc2-py2.7-linux-x86_64.egg/pydoop/hdfs.py", line 458, in copy
    return super(hdfs, self).copy(from_path, to_hdfs, to_path)
IOError: Cannot copy /home/user/test.txt to filesystem on master

Error hasn't detail. Any idea?

Upvotes: 2

Views: 4630

Answers (1)

subiet
subiet

Reputation: 1399

In your conf/core-site.xml you would have set the tmp directory for fs operations. If you have forgotten to set ownership and permission of the running user on those directories then that gives an IO exception, check that.

Upvotes: 2

Related Questions