CODEWITHSUNDEEP
c#character-encodingasp.net-core-webapi
Txaran
Txaran

Reputation: 164

DecoderFallbackException: Unable to translate bytes [BF] at index 75 from specified code page to Unicode

On a .NET Core 2.1 Web API I am working on, I've a POST method receiving a JSON object with just one property, like this:

{
    "longURL": "http://foo.example.com/path/path/path/path/path/pfad"
}

And the method's header:

public async Task<ActionResult<ShortenerOutputDto>> CreateAsync([FromBody] 
ShortenerInputDto input)

But I'm getting an exception if the input JSON contains some special characters, like this:

{
    "longURL": "http://foo.example.com/path/path/path/path/path/pfad¿"
}

Please, note that the last (¿) is the offending character. The exception I'm getting is:

System.Text.DecoderFallbackException: Unable to translate bytes [BF] at index 75 from specified code page to Unicode.
   at System.Text.DecoderExceptionFallbackBuffer.Throw(Byte[] bytesUnknown, Int32 index)
   at System.Text.DecoderExceptionFallbackBuffer.Fallback(Byte[] bytesUnknown, Int32 index)
   at System.Text.DecoderFallbackBuffer.InternalFallback(Byte[] bytes, Byte* pBytes, Char*& chars)
   at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount, DecoderNLS baseDecoder)
   at System.Text.DecoderNLS.GetChars(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Char[] chars, Int32 charIndex, Boolean flush)
   at System.Text.DecoderNLS.GetChars(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Char[] chars, Int32 charIndex)
   at Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.ReadIntoBuffer()
   at Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.Read(Char[] buffer, Int32 index, Int32 count)
   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)

The problem is that the exception is being thrown before entering the controllers method.

So, is there a way to avoid this? I'd like to send a BadRequest in case the input can't be decoded.

UPDATE

Based on @jdweng's answer, I added the following converter class: 

public class HtmlEncodingConverter : JsonConverter
{
        public override bool CanConvert(Type objectType)
        {
            return objectType == typeof(String);
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            return System.Web.HttpUtility.HtmlDecode((string)reader.Value);
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {

writer.WriteRawValue(System.Web.HttpUtility.HtmlEncode((string)value));
        }
}

and Registered as a JSONOptions:

services.AddMvc()
    .AddJsonOptions (opt => opt.SerializerSettings.Converters.Add(new HtmlEncodingConverter()));

But ReadJson nor WriteJson are not being hit if some special character is on the request.

So it seems that decoding or decoding happens before Core's tries to Convert the input. Really weird.

UPDATE II

Raw HTTP Request Message:

POST /create HTTP/1.0
Host: localhost:5000
Content-Length: 80
Content-Type: application/json

{
    "longURL" : "http://foo.example.com/path/path/path/path/path/pfad¿"
}

Upvotes: 4

Views: 8874

Answers (1)

Txaran
Txaran

Reputation: 164

Finally I ended up adding an Exception filter like this:

public class DecoderFallbackExceptionFilter : IExceptionFilter
{
        public void OnException(ExceptionContext context)
        {
            if (context.Exception.GetType() == typeof(DecoderFallbackException))
                context.Result = new BadRequestObjectResult(ShortURLResponse.InvalidURL());
        }
}

...and register in startup.cs:

services.AddMvc(opt =>
        {
            opt.Filters.Add(new DecoderFallbackExceptionFilter());
        });

Maybe it is not the solution I initially expected but works and allows me to take control of whatever action I should take for each case.

Furthermore, I can add input parameters to the method again, and re-enable the unit tests. https://github.com/aspnet/AspNetCore/issues/8676

BTW, it seems that this behavior will be enhanced in ASP.NET Core 3.0. https://github.com/aspnet/AspNetCore/issues/3959

Upvotes: 6

Related Questions