CODEWITHSUNDEEP
amazon-s3marklogicmarklogic-8mlcp
Peter
Peter

Reputation: 29

CSV Files from AWS S3 to MarkLogic 8

Can csv files from the AWS S3 bucket be configured to go straight into ML or do the files need to land somewhere and then the CSV files have to get ingested using MCLP?

Upvotes: 0

Views: 229

Answers (1)

Harry Bakken
Harry Bakken

Reputation: 793

Assuming you have CSV files in the S3 Bucket and that one row in the CSV file is to be inserted as a single XML record...that wasn't clear in your question, but is the most common use case. If your plan is to just pull the files in and persist them as CSV files, there are undocumented XQuery functions that could be used to access the S3 bucket and pull the files in off that. Anyway, the MLCP documents are very helpful in understanding this very versatile and powerful tool.

According to the documentation (https://developer.marklogic.com/products/mlcp) the supported data sources are:

  • Local filesystem
  • HDFS
  • MarkLogic Archive
  • Another MarkLogic Database

You could potentially mount the S3 Bucket to a local filesystem on EC2 to bypass the need to make the files accessible to MLCP. Google's your friend if that's important. I personally haven't seen a production-stable method for that, but it's been a long time since I've tried.

Regardless, you need to make those files available on a supported source, most likely a filesystem location in this case, where MLCP can be run and can reach the files. I suppose that's what you meant by having the files land somewhere. MLCP can process delimited files in import mode. The documentation is very good for understanding all the options.

Upvotes: 1

Related Questions