CODEWITHSUNDEEP
c#jsonazurejson.netazure-eventhub
Steph Locke
Steph Locke

Reputation: 6146

How to write JSON to Event Hub correctly

I'm batching serialized records (in a JArray) to send to Event Hub. When I'm writing the data to Event Hubs it seems to be inserting extra speech marks around the JSON i.e. what is written "{"myjson":"blah"}" not {"myjson":"blah"} so downstream I'm having trouble reading it.

Based on this guidance, I must convert JSON to string and then use GetBytes to pass it into an EventData object. I suspect my attempt at following this guidance is where my issue is arising.

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public static class EventDataTransform
{
    public static EventData ToEventData(dynamic eventObject, out int payloadSize)
    {
        string json = eventObject.ToString(Formatting.None);
        payloadSize = Encoding.UTF8.GetByteCount(json);
        var payload = Encoding.UTF8.GetBytes(json);
        var eventData = new EventData(payload)
        {
        };
        return eventData;
    }
}

How should an item from a JArray containing serialized data be converted into the contents of an EventData message?

Code call location - used in batching upto 256kb parcels

public bool MoveNext()
{
        var batch = new List<EventData>(_allEvents.Count);
        var batchSize = 0;
        for (int i = _lastBatchedEventIndex; i < _allEvents.Count; i++)
        {
            dynamic evt = _allEvents[i];
            int payloadSize = 0;
            var eventData = EventDataTransform.ToEventData(evt, out payloadSize);
            var eventSize = payloadSize + EventDataOverheadBytes;
            if (batchSize + eventSize > MaxBatchSizeBytes)
            {
                break;
            }

            batch.Add(eventData);
            batchSize += eventSize;
        }
        _lastBatchedEventIndex += batch.Count();
        _currentBatch = batch;
        return _currentBatch.Count() > 0;
}

Upvotes: 2

Views: 5808

Answers (1)

Anthony Chu
Anthony Chu

Reputation: 37520

Sounds like the JArray already contains serialized objects (strings). Calling .ToString(Formatting.None) will serialize it again a second time (wrapping it in quotes).

Interestingly enough, if you call .ToString() without passing in a Formatting, it would not serialize it again.

This fiddle demonstrates this: https://dotnetfiddle.net/H4p6KL

Upvotes: 4

Related Questions