CODEWITHSUNDEEP
xmljson
rwallace
rwallace

Reputation: 33465

Incremental interchange format

I need a format to transfer data from program A to program B, and obviously I'd prefer to use a standard format. On the face of it XML would be perfect for this job, or possibly JSON.

The fly in the ointment is that I need the format to be incremental: program A appends data to the file from time to time, then program B slurps the whole lot when it is ready. From what I've read, neither XML nor JSON can do this, because they both require exactly one top-level element in a file.

Is there anything I'm missing?

Clarification:

I need structure of the kind XML provides and CSV doesn't. (Well I could make a mutant variant of CSV that nothing else would be able to read, but that's what I'm trying to avoid.)

A doesn't know when it will be finished (depends on user actions, availability of network connections and other such unpredictable variables), so A can't say "now I will write the last end tag and handover to B" because A might be called again before B.

Upvotes: 0

Views: 99

Answers (3)

Brian Agnew
Brian Agnew

Reputation: 272347

So A always needs to write a well-formed file such that B can read it at any time.

If you use XML (and that seems appropriate) you will always have to write a well-formed document (e.g. containing one root node and subnodes within). So you'll have to maintain a DOM object in your process. Depending on memory consumption, that may well be fine.

A can write this whenever an update comes in. The problem here is that writing a file will take a finite amount of time, and you don't want B to read a partially-written file. Some form of signalling that the file is complete will be required here (using temp files as semaphores, renaming etc.)

Upvotes: 1

Ryan Brunner
Ryan Brunner

Reputation: 14851

If I'm understanding you correctly, the workflow is as follows:

  • Over some period of time, A is building up data to put in a file
  • Once that is ready, B consumes the whole thing.

In this case, both XML and JSON are fine, since there's a distinct point where A is working and B doesn't start consuming the file until it is processed (as opposed to something like an always-open stream that continually feeds data).

In something like .NET, you could create an XmlDocument with a root element, and continue to add nodes to it until you are done.

Upvotes: 0

B.E.
B.E.

Reputation: 5080

How about CSV? Easy to write - easy to parse if you know the format and don't need extra metainformation.

Upvotes: 0

Related Questions