HHH
Reputation: 6485
How to connect to hdfs using pyarrow in python
I've installed pyarrow and would like to connect to hdfs file in a Hadoop cluster. I'm having the following line and it give me error.
fs = pa.hdfs.connect(host='...', 50057, user='...', kerb_ticket='/tmp/krb5cc_0')
this is the error message I'm getting
ArrowIOError: Unable to load libhdfs
How should I install libhdfs? What other dependecies/settings I need to do?
Upvotes: 3
Views: 5901
Answers (0)
Related Questions
- read a parquet files from HDFS using PyArrow
- I can't load data from hdfs, I use the python pyarrow library. In a docker container
- How to write on HDFS using pyarrow
- How to properly setup pyarrow for python 3.7 on Windows
- Unable to load libhdfs when using pyarrow
- PyArrow 0.16.0 fs.HadoopFileSystem throws HDFS connection failed
- How to replace old legacy hdfs connector with new fs.HadoopFileSystem?
- Hadoop: Failed to connect to HDFS(Hadoop) using python
- connecting pyarrow with libhdfs3
- hdfs.connect() vs HdfsClient in PyArrow