CODEWITHSUNDEEP
c#dotnet-httpclientvisual-studio-2022
Matheus Rocha
Matheus Rocha

Reputation: 1204

HttpClient.GetAsync() selectively ignoring query parameter

I have a very simple method for querying a webpage and parsing the results. The body of the response isn't the problem, the problem is that the request generated by the GetAsync(string) or GetAsync(Uri) methods seem to selectively ignore a certain query parameter. I say it is selectively ignoring it, because it is ignoring the same parameter regardless of the order in which they appear.

When I check the RequestMessage property of the returned HttpResponseMessage, the RequestUri shows the complete uri correctly, except that it's missing the very specific "o=data" parameter, and the response body confirms that the parameter wasn't present in the request because the content should be sorted by date if it was present (but it isn't).

static readonly HttpClient client = new() { BaseAddress = new("https://www.jusbrasil.com.br/diarios/busca") };

public static async Task<ResultDocument[]> GetResults(string name)
        {
            var query = new Uri($"https://www.jusbrasil.com.br/diarios/busca?q={Uri.EscapeDataString(name)}&o=data");
            //var query = "?q={name}&o=data" the exact same problem happens whether I use BaseAddress or not.
            var resp = await client.GetAsync(query);
            // Rest of the function parsing the result's body.
        }

Specifically the "o" query parameter disappears from the request.

UPDATE: I've tried to rename the parameter. It works if the parameter is named "w", and it works if the parameter is named "or". It ONLY disappears if it has the name "o" (which is the one I need). Doesn't work. Any other parameter disappears if name has a space character.

UPDATE: Turns out the problem only happens when name has a space character. Even if it escaped. Code updated.

No space: No space

With space: With space

Showing name.ToCharArray() when "o" disappears. (The method here is being called directly from Program.cs): enter image description here

UPDATE: It seems the problem might be with the default HttpMessageHandler. When I implement a custom handler just to see how HttpClient is generating the HttpRequestMessage, the RequestUri is correct in the request. And also notice that here the space character shows escaped as "%20" and not as '+', which is how it should be, since Uri.EscapeDataString(string) escapes it with "%20" and not '+'. enter image description here

Upvotes: 1

Views: 1086

Answers (1)

Matheus Rocha
Matheus Rocha

Reputation: 1204

With a lot of help from @Dai, we finally figured out the problem. The server responded with a redirect if the space character in the name was escaped with %20, and the redirection destination ignored the other parameters. It turns out the problem was with the server, and not the code.

To fix this I simply changed the line:

var query = new Uri($"https://www.jusbrasil.com.br/diarios/busca?q={Uri.EscapeDataString(name)}&o=data");

To:

var query = new Uri($"https://www.jusbrasil.com.br/diarios/busca?q={Uri.EscapeDataString(name).Replace("%20", "+")}&o=data");

I left the Uri.EscapeDataString(string) and only replaced %20 because other characters escaped with %xx (such as double quote %22) still works fine and causes no faulty redirection.

Upvotes: 2

Related Questions