CODEWITHSUNDEEP
hadoophdfsfiwarefiware-cygnus
yeuk0
yeuk0

Reputation: 43

FIWARE Cygnus: File exists in HDFS, but could not write the data

I am using NGSIHDFSSink to persist on an HDFS through Cygnus. When the first notification from Orion is received, it creates the directory and the correct file with the data:

/user/hdfs/openiot/Lamp_001_Lamp/Lamp_001_Lamp.txt

However, for the next notifications received it shows the following error:

There was some problem with the current endpoint, trying other one. Details: CygnusPersistenceError. /user/hdfs/openioLamp/Lamp_001_Lamp.txt file exists in HDFS, but could not write the data. Server response: 403 Forbidden.

This is my Sink configuration:

cygnus-ngsi.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.NGSIHDFSSink
cygnus-ngsi.sinks.hdfs-sink.channel = hdfs-channel
cygnus-ngsi.sinks.hdfs-sink.enable_encoding = false
cygnus-ngsi.sinks.hdfs-sink.enable_grouping = false
cygnus-ngsi.sinks.hdfs-sink.enable_lowercase = false
cygnus-ngsi.sinks.hdfs-sink.enable_name_mappings = false
cygnus-ngsi.sinks.hdfs-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
cygnus-ngsi.sinks.hdfs-sink.backend.impl = rest
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns = 500
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns_per_route = 100
cygnus-ngsi.sinks.hdfs-sink.hdfs_host = my_hdfs_ip
cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 50070
cygnus-ngsi.sinks.hdfs-sink.hdfs_username = hdfs
cygnus-ngsi.sinks.hdfs-sink.hdfs_password =
cygnus-ngsi.sinks.hdfs-sink.oauth2_token =
cygnus-ngsi.sinks.hdfs-sink.service_as_namespace = false
cygnus-ngsi.sinks.hdfs-sink.batch_size = 100
cygnus-ngsi.sinks.hdfs-sink.batch_timeout = 30
cygnus-ngsi.sinks.hdfs-sink.batch_ttl = 10
cygnus-ngsi.sinks.hdfs-sink.batch_retry_intervals = 5000
cygnus-ngsi.sinks.hdfs-sink.hive = false
cygnus-ngsi.sinks.hdfs-sink.krb5_auth = false

Any ideas?

Thank you so much

Upvotes: 1

Views: 147

Answers (1)

fgalan
fgalan

Reputation: 12322

According to HDFS sink documentation, Cygnus uses the WebHDFS to write in HDFS.

I'm not sure but maybe WebHDFS considers "create" and "write" as different permissions, so your WebHDFS instance is configured to allow only the former but not the latter. In that case, it would be an issue with permissions at the WebHDFS server.

I'd suggest to try the WebHDFS itself without Cygnus (i.e. using Postman or curl) to ensure that WebHDFS is working properly.

Upvotes: 0

Related Questions