CODEWITHSUNDEEP
c#.netelasticsearch
M.Y. Babt
M.Y. Babt

Reputation: 2911

C# HttpClient.PostAsJsonAsync() fails, even though the exact same request is accepted when made through PostMan

I have found similar questions both here as well as on the Elastic discussion forum, but unfortunately none of the answers helped.

I am currently using ElasticSearch 7.0.

I want to make a bulk request to my ElasticSearch server. My JSON file contains information that looks something like this:

{ "index": { "_index": "website", "_id": "link1" }}
{ "label":    "Link1" }

Each line is terminated by an LF line break, and there is also an additional LF line break at the end of the document.

In C#, here is how I make a POST request for my bulk data:

HttpResponseMessage response = await httpClient.PostAsJsonAsync($"http://127.0.0.1:9200/website/_bulk", jsonDocumentContents);

And yet I keep seeing this error message:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"},"status":400}

How can I fix this error?

UPDATE:

A short description of how I read the JSON document contents into the jsonDocumentContents variable: The JSON document was stored inside a zipped folder, so retrieving it requires unzipping:

ZipArchive archive = new ZipArchive(zippedFolderStream);
foreach (ZipArchiveEntry entry in archive.Entries)
{
    string jsonDocumentContents = new StreamReader(entry.Open()).ReadToEnd();
    HttpResponseMessage response = await httpClient.PostAsJsonAsync($"http://127.0.0.1:9200/website/_bulk", jsonDocumentContents);
    Console.WriteLine(await response.Content.ReadAsStringAsync());
}

UPDATE:

I just made a bulk request with the exact same contents using PostMan, and the request was successful. However, the error message persists when I make the same bulk request in C# using httpClient.PostAsJsonAsync(...).

Upvotes: 5

Views: 1421

Answers (1)

M.Y. Babt
M.Y. Babt

Reputation: 2911

I got it working by changing my code to the following:

ZipArchive archive = new ZipArchive(zippedFolderStream);
foreach (ZipArchiveEntry entry in archive.Entries)
{
    string jsonDocumentContents = new StreamReader(entry.Open()).ReadToEnd();
    StringContent content = new StringContent(jsonDocumentContents, Encoding.ASCII, mediaType: "application/json");
    HttpResponseMessage response = await httpClient.PostAsync($"http://127.0.0.1:9200/website/_bulk", content);
    Console.WriteLine(await response.Content.ReadAsStringAsync());
}

Notice that I am using HttpClient.PostAsync() instead of HttpClient.PostAsJsonAsync(), with a StringContent instance that specifies "application/json" as its media type.

I looked into the source code for HttpClient, and noticed that a new instance of JsonMediaTypeFormatter is created every time HttpClient.PostAsJsonAsync is called.

Since my POST requests are successful when I make them through PostMan, the issue must be caused by how PostAsJsonAsync() is implemented. I suspect, but have not verified, that the problem is due to the default properties in the JsonMediaTypeFormatter class.

To circumvent the problem I decided to use Http.PostAsync() with a correctly-configured StringContent instance.

Lo and behold, I can now send bulk requests to my ElasticSearch server using C#.

Upvotes: 7

Related Questions