CODEWITHSUNDEEP
hpcc-eclhpcc
encoded150
encoded150

Reputation: 39

How to deal with concatenated logical file obtained from spraying a folder containing JSON files in ecl?

Using DFU plus I executed the following command to spray multiple json files in a folder:

dfuplus action=spray server=http://localhost:8010/ srcip=10.0.2.15 prefix=FILENAME format=json srcfile=/var/lib/HPCCSystems/mydropzone/CSV/NEW2/*.json dstname=hthor::test dstcluster=mythor overwrite=1 

But the problem is json representation in ecl is in the form of json array, example

[
    {
        "name": "hauserGermany",
        "place": 56.08
    },
    {
        "name": "hyou",
        "place": 59.03535
    }
]

But the problem if the logical file is being display in the following way

{
    "name": "hauserGermany",
    "place": 56.08
},
{
    "name": "hyou",
    "place": 56.03535
}

Notice that the square brackets are missing

Due to this, I am able to access only the first file and not all the files in the directory.

Is there any solution to how I can access all the files?

How can I add the square brackets in logical file, or is there any other way of multispraying in which the brackets get automatically added?

Upvotes: 1

Views: 44

Answers (1)

Richard Taylor
Richard Taylor

Reputation: 780

If it is your intention to end up with a valid JSON file on your Thor cluster, then this approach will not work.

The dfuplus command line you're using is doing a BLOB spray, which puts each file's content into a DATA field which is preceded by a STRING field containing the name of the file in that blob. So you end up with a two-field flat file dataset, not a valid JSON file.

I suggest you combine the files BEFORE doing a standard JSON spray operation to put data on your HPCC Systems cluster.

Upvotes: 1

Related Questions