CODEWITHSUNDEEP
restgrailscharacter-encoding
john
john

Reputation: 613

UTF-8 encoded characters from REST-query not rendered properly

I'm consuming an external REST service that provides all content as UTF-8 encoded.

For some reason my application cannot properly handle the response. If I dump the response I will se things like LuleÃ¥ (should be Luleå).

EDIT: The same behavior happens if i forward (without altering) the string to the UI, ex.:

flash.message = "Test" + integrationService.testEncoding()

What I did was to create a _Events.groovy file in the /script folder and specifying there that 

eventConfigureTomcat = { tomcat ->
    tomcat.connector.URIEncoding = "UTF-8"
    tomcat.connector.useBodyEncodingForURI = true
}

I also have the following in my Config.groovy:

grails.views.gsp.encoding = "UTF-8"
grails.converters.encoding = "UTF-8"

But that changed nothing. The response is still wrongly shown. I'm not sure if this is a configuration issue with Grails, with the embedded tomcat or with something else. I'm currently running my test setup on windows 7, but the same issue happens on my server running on Centos. Please advice.

EDIT2: If i consume the REST service using curl, everything is rendered correctly in the output.

EDIT3: I'm using org.springframework.web.client.RestTemplate and HttpComponents to consume the service:

private static final HttpHeaders requestHeaders
static{
    requestHeaders = new HttpHeaders()
    requestHeaders.set(HttpHeaders.CONTENT_TYPE, "application/json")
    requestHeaders.set(HttpHeaders.ACCEPT, "application/json")
    requestHeaders.set("Accept-Encoding", "gzip")
}

private final static RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory(
    HttpClientBuilder.create().build()))
...
...
public def testEncoding(){
    ResponseEntity<String> response = restTemplate.exchange(
            "https://www.url.com", HttpMethod.GET, new HttpEntity<Object>(requestHeaders),
            String.class)
    def gamesJson = JSON.parse(response.getBody())
    //...
    //parse value from gamesJson
    //...
    return testValue
}

Upvotes: 3

Views: 11591

Answers (5)

Sobhan
Sobhan

Reputation: 1460

In my case that i had the same problem with contents received from my REST web service from the server and not my local enviroment. I had search a lot and finally i found a solution that resolved my issue. In Windows I added a new environment variable with key: JAVA_TOOL_OPTIONS and set its value to: -Dfile.encoding=UTF8. The (Java) System property will be set automatically every time a JVM is started. You will know that the parameter has been picked up because the following message will be posted to System.err:

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8

Upvotes: 0

beerbajay
beerbajay

Reputation: 20270

Per my previous answer:

You just need to add the StringHttpMessageConverter to the template's message converters:

RestTemplate template = new RestTemplate();
template.getMessageConverters()
        .add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
ResponseEntity<Object> response = template.exchange(endpoint, method, entity, 
                                                    Object.class);

Upvotes: 8

john
john

Reputation: 613

The answer to my problem is already found on Stack Exchange.

You just need to add the StringHttpMessageConverter to the template's message converters:

restTemplate.getMessageConverters()
    .add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));

Upvotes: 2

khakiout
khakiout

Reputation: 2400

Our team experienced a similar issue before, we have a 3rd party service and they said their output is encoded in UTF-8. But the strings returned are still garbled. After a bit of testing, it turns out that they were returning ISO-8859-1 encoded strings. What we did was to decode/encode their input into UTF-8 encoded characters so we can use those properly.

For your case, I think this is a similar issue:

UTF-8: Luleå

ISO-8859-1: Luleå

In Java, we did something like this:

Charset initialEncoding = Charsets.ISO_8859_1;
Charset outputEncoding = Charsets.UTF_8; 
byte[] byteArray = input.getBytes(initialEncoding);
String output = new String(new String(byteArray, outputEncoding));

In Groovy, I think you could do something like

import groovy.json.JsonSlurper;

def main = {
    def response = '{"name":"Luleå"}' 

    def slurper = new JsonSlurper()
    def result = slurper.parse(response.getBytes(), 'UTF-8')

    println result.name // prints Luleå
}

Upvotes: 2

J.P
J.P

Reputation: 565

The encoding type can be enforced in the environment itself.

JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8 -Dclient.encoding.override=UTF-8

Just try setting the above encoding settings in windows/linux. I hope this should resolve the issue.

In this case, JVM will pickup the default encoding type from environment variable.

Upvotes: 3

Related Questions