CODEWITHSUNDEEP
marklogicmlcp
Martin Thompson
Martin Thompson

Reputation: 3745

How to Import MarkLogic one file many documents with content Pump

I have a correctly formatted single json file which is an array of documents ( about 60,000 ). I can import it using node api, but just wondering how I would import this file using the Marklogic content pump ( windows ).

I tried using basic options and it just creates one document called orders.json

Upvotes: 1

Views: 217

Answers (1)

David Ennis -CleverLlamas.com
David Ennis -CleverLlamas.com

Reputation: 7770

For MLCP, a properly formatted JSON aggregate(Called Line-delimited JSON) file is not an array of objects. It is a file with one stringified object per line. Think CSV, but JSON stuff instead.

Imagine if you passed an array - the MLCP process would have to parse the entire file into a JSON array to process. One object per line means that MLCP can easily shard the file and process in parallel. In fact, I do not think it even parses to JSON if you use automatic URIs.

Bad:

[
  {
    "bar": "baz"
  },
  {
    "bar": "buz"
  }
]

Good:

{"bar":"baz"}
{"bar":"buz"}

All the goodies about this can be found here:
MLCP User Guide

Relevant section are: (Accurate for document version: '9.0-3, September, 2017')

  • 4.10
  • 4.10.1

Upvotes: 2

Related Questions