CODEWITHSUNDEEP
jsonmulemule-studiodataweavemulesoft
Triumph Spitfire
Triumph Spitfire

Reputation: 645

Convert Json objects Collection to Array

I have a json file which contains a collection of numerous JSON objects. A sample format is given below:

{"ID": 123,"Name": "TEST-1","val11": {},"url": "test1.com","val12": []}
{"ID": 456,"Name": "TEST-2","val21": {},"url": "test2.com","val22": []}
{"ID": 789,"Name": "TEST-3","val31": {},"url": "test3.com","val32": []}

As you see, it is not an array ([ ] and commas missing). I need to convert this into a valid JSON array. The code that I tried is:

%dw 2.0
output application/json
var PayloadSplit = payload splitBy('\n')
var PayloadArray = (PayloadSplit map (value, index) -> read(value, 'application/json'))
---
PayloadArray

This works fine for a small sized payload. However, if I try to perform this on the entire file (size about 320 MB with ~20k JSON objects), it fails with a java.lang.OutOfMemoryError: Java heap space error. Is there a way to overcome this? Or can I split the main file into multiple files and then try this (in a ForEach Loop perhaps?). Please advise

Edit1 - Attaching the mule flow below:

    <flow name="convert-object-to-array-test1Flow" doc:id="0645e9bd-7f77-4b1e-93d0-dedd9d154ef7" >
        <http:listener doc:name="Listener" doc:id="551cd3b6-e4c8-4b7a-aff3-305effbe8a8b" config-ref="HTTP_Listener_config" path="/file"/>
        <file:read doc:name="Read" doc:id="21a310c1-5887-4bc0-83b9-b8968e145f0d" path="C:\Desktop\NDJsonSample.json" outputMimeType="application/ndjson" />
        <ee:transform doc:name="Transform Message" doc:id="95235c56-2f5a-4f39-ba96-8be7c4e501b5" >
            <ee:message >
                <ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
            </ee:message>
        </ee:transform>
        <logger level="INFO" doc:name="Logger" doc:id="935530dd-17fd-41c9-8fe0-1561ba3de703" />
    </flow>

Upvotes: 1

Views: 1518

Answers (1)

machaval
machaval

Reputation: 5059

DW already have support for this format. It is called ndjson. Please visit the documentation. You just need to set application/ndjson to the payload.

Upvotes: 7

Related Questions