CODEWITHSUNDEEP
pythonpython-2.7python-behave
natn2323
natn2323

Reputation: 2061

JSON file isn't finished writing to by the time I load it, behave BDD

My program is writing to a JSON file, and then loading, reading, andPOSTing it. The writing part is being done by the behave BDD. 

# writing to the JSON file is done by behave
data = json.load(open('results.json', 'r'))
r = requests.post(MyAPIEndpoint, json=data)

I'm running into an issue since the writing is not being completed before I begin loading. (It's missing the closing [ after the final {.)

HOOK-ERROR in after_all: ValueError: Expecting object: line 2 column 2501 (char 2502)

Is there a way of getting by this, either by changing something with my call to behave's __main__ or by a change in how or when I'm loading the JSON file?

Upvotes: 0

Views: 905

Answers (2)

Taylor D. Edmiston
Taylor D. Edmiston

Reputation: 13036

One way to address this problem is to change your file format from being JSON at the top level to newline-delimited JSON (NDJSON), also called line-delimited JSON (LDJSON) or JSON lines (JSONL).

https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON

For example, this JSON file:

{
  "widgets": [
    {"name": "widget1", "color": "red"},
    {"name": "widget2", "color": "green"},
    {"name": "widget3", "color": "blue"}
  ]
}

Would become this NDJSON file:

{"name": "widget1", "color": "red"}
{"name": "widget2", "color": "green"}
{"name": "widget3", "color": "blue"}

It's especially useful in the context of streaming data, which kind of sounds like the use case you have where you might have one process writing to a file continuously while another is reading it.

You could then read the NDJSON file like so:

import json
from pprint import pprint

with open('widgets.json') as f:
    all_lines = [json.loads(l) for l in f.readlines()]
    all_data = {'widgets': all_lines}
    pprint(all_data)

Output:

{'widgets': [{'color': 'red', 'name': 'widget1'},
             {'color': 'green', 'name': 'widget2'},
             {'color': 'blue', 'name': 'widget3'}]}

Upvotes: 1

Latot
Latot

Reputation: 307

i think the problem here is little mixed, in one part, you can wait to the file finish to be written, and close it when is not in use, you can do that inside your code or somwthing like this check if a file is open in Python

In other part, for the computer, the data is data, don't will analysis that, i means, you know where is the error, because you analise that, when you think, so for you is obvs, but for the computer isn't obvs, how much errores there is?, and where or the structure of it?, is there all the data we need?, for a computer is hard know all of this, you will need write a program to can deduce all of this data and check it.

If your program uses multiples results, i think the better ways is use temp files, so you can freely create one, write, check when is ready and use it, and don't will care if you have other similar process using it.

Other way, check if the json is valid before call it, Python: validate and format JSON files, when is valid load it.

Hope can help.

Cya.

Upvotes: 1

Related Questions