UTF-8 | issue with special character on form submit | java.io.CharConversionException

Question

I have a form where I have textbox and a submit button. I am using burp tool to change the value submitted in the form to test my server side validation.

The server side validation works fine, unless there is a % character entered from the burp tool, on entering % character the server shows following exception.

2012-12-11 11:37:07,860 WARN  [org.apache.tomcat.util.http.Parameters] (ajp-0.0.0.0-8109-19) Parameters: Character decoding failed. Parameter skipped.
java.io.CharConversionException: EOF
                at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:83)
                at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:49)
                at org.apache.tomcat.util.http.Parameters.urlDecode(Parameters.java:429)
                at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:412)
                at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:363)
                at org.apache.catalina.connector.Request.parseParameters(Request.java:2562)
                at org.apache.catalina.connector.Request.getParameter(Request.java:1060)
                at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:355)
                at org.displaytag.filter.ResponseOverrideFilter.doFilter(ResponseOverrideFilter.java:118)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
                at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
                at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
                at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
                at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
                at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
                at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                at java.lang.Thread.run(Thread.java:662)

when i submit the form with % character, it changes it to %25 correctly, but how do i handle this server side?

Answer 1

The other answers are correct in considering it safe to ask you to encode your input since all the browsers would do so as well and you would ideally never receive an un-encoded plain % character.

But, with that said, if you still insist on seeing an error page; it's possible to write a custom filter to catch this CharConversionException and send the client an error page or forward the request to the same URI but with an errorMessage attribute set so that the request is handled differently.

This is being configured here through a filter parameter named useErrorPage.

EDIT: In light of the source code shared by @Santosh which reveals that CharConversionException is actually being suppressed by Tomcat I've modified the filter to inject parameter validation and check all request parameters for a list of forbidden characters that can be configured through a filter parameter named forbiddenChars.

Place this filter above DisplayTag's ResponseOverrideFilter in web.xml to make sure it intercepts everything. Ordering can have its side effects when it comes to filters.

I would take this edit as an opportunity to stress on the fact that Servlet Filters is a very powerful concept that lets you pre or post process any request in any way you like. Most of the web frameworks (like Struts 2 for MVC) are using filters to serve as their entry points.

So, there's very little that you can't do with filters. Just make sure you use them for the right purpose like business logic should obviously not go there though application wide authentication can.

IOException Filter

public class CharConversionExpFilter implements Filter {

    private char[] forbiddenChars;  // ADDED
    private FilterConfig filterConfig;

    @Override
    public void init(FilterConfig filterConfig) throws ServletException {
        this.filterConfig = filterConfig;
        forbiddenChars = filterConfig.getInitParameter("forbiddenChars")
                               .replace(",", "").toCharArray(); // ADDED
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response,
            FilterChain chain) throws IOException, ServletException {
        String requestURI = ((HttpServletRequest) request).getRequestURI();
        try {
            validateParameters((HttpServletRequest) request);  // ADDED
            chain.doFilter(request, response);
        } catch (IOException e) {
            if (e instanceof CharConversionException) {
                if ("true".equalsIgnoreCase(filterConfig.getInitParameter("useErrorPage"))) {
                    if (response instanceof HttpServletResponse) {
                        ((HttpServletResponse) response).sendError(400,
                        "The request cannot be fulfilled due to bad input.\nError:" + e.getMessage());
                    }
                } else {
                    request.setAttribute("errorMessage", e.getMessage());
                    filterConfig.getServletContext().getRequestDispatcher(requestURI).forward(request, response);
                }
            }
        }
    }

    // ADDED
    private void validateParameters(HttpServletRequest request) throws CharConversionException {
        Enumeration<String> parameterNames = request.getParameterNames();
        while (parameterNames.hasMoreElements()) {
            String parameter = request.getParameter(parameterNames.nextElement());
            if (parameter != null && parameter.length() > 0) {
                for (char forbidChar : forbiddenChars) {
                    if (parameter.indexOf(forbidChar) != -1) {
                        throw new CharConversionException(
                                String.format(
                                        "Parameter: [%s] contains the forbidden character [%c]",
                                        parameter, forbidChar));
                    }
                }
            }
        }
    }

    @Override
    public void destroy() {}
}

web.xml

<filter>
  <filter-name>CharConversionExpFilter</filter-name>
  <filter-class>servlet.filter.CharConversionExpFilter</filter-class>
  <init-param>
    <param-name>useErrorPage</param-name>
    <param-value>true</param-value>
  </init-param>
  <init-param> <!-- ADDED -->
    <param-name>forbiddenChars</param-name>
    <param-value>%,*,?,#,$</param-value> <!-- filtering wildcards and EL chars -->
  </init-param>
</filter>

<filter-mapping>
  <filter-name>CharConversionExpFilter</filter-name>
  <url-pattern>/*</url-pattern> <!-- * = ALL; "/servlet-name" if required -->
</filter-mapping>

<error-page>
  <error-code>400</error-code>
  <location>/errorPage.jsp</location> <!-- isErrorPage = true; use "exception" obj -->
</error-page>

Hope this gives you enough pointers to work out a solution as per your needs. (Note: The syntax highlighting is interpreting <url-pattern>/* incorrectly as the start of a multi-line Java comment. Please ignore that.)

Answer 2

If you closely examine the stack trace, all the classes involved (in parsing HTTP text to Java request object) are from the framework and there is no user control over it.

Now the basic premise behind this is: The server will always assume that the data it received is always URL encoded and HENCE it will always try to decode.

If the decoding fails, that implies that the basic tenets of the HTTP protocol were not adhered to and hence rejects the request outrightly without giving user a chance to handle it.

So IMHO, you cannot handle this condition as the server is rightly handling it by itself.

EDIT:

One the solution proposed by @Ravi above looks like it solves the problem. I tried the same code but I could never catch the exception !

Digging down deeper, it became clear why that exception (java.io.CharConversionException) could not be handled in any user defined code.

Though java.io.CharConversionExceptionis thrown but its never propagated in the call hierarchy. Here is why,

In the stack trace

        at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:83)
        at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:49)
        at org.apache.tomcat.util.http.Parameters.urlDecode(Parameters.java:429)
        at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:412)

Look at the source of the method processParameters() in class org.apache.tomcat.util.http.Parameters (Code taken from grepcode)

public void processParameters( byte bytes[], int start, int len, 
                                   String enc ) {
        int end=start+len;
        int pos=start;

        if( debug>0 ) 
            log( "Bytes: " + new String( bytes, start, len ));

        do {
            boolean noEq=false;
            int valStart=-1;
            int valEnd=-1;

            int nameStart=pos;
            int nameEnd=ByteChunk.indexOf(bytes, nameStart, end, '=' );
            // Workaround for a&b&c encoding
            int nameEnd2=ByteChunk.indexOf(bytes, nameStart, end, '&' );
            if( (nameEnd2!=-1 ) &&
                ( nameEnd==-1 || nameEnd > nameEnd2) ) {
                nameEnd=nameEnd2;
                noEq=true;
                valStart=nameEnd;
                valEnd=nameEnd;
                if( debug>0) log("no equal " + nameStart + " " + nameEnd + " " + new String(bytes, nameStart, nameEnd-nameStart) );
            }
            if( nameEnd== -1 ) 
                nameEnd=end;

            if( ! noEq ) {
                valStart= (nameEnd < end) ? nameEnd+1 : end;
                valEnd=ByteChunk.indexOf(bytes, valStart, end, '&');
                if( valEnd== -1 ) valEnd = (valStart < end) ? end : valStart;
            }

            pos=valEnd+1;

            if( nameEnd<=nameStart ) {
                log.warn("Parameters: Invalid chunk ignored.");
                continue;
                // invalid chunk - it's better to ignore
            }
            tmpName.setBytes( bytes, nameStart, nameEnd-nameStart );
            tmpValue.setBytes( bytes, valStart, valEnd-valStart );

            try {
                addParam( urlDecode(tmpName, enc), urlDecode(tmpValue, enc) );
            } catch (IOException e) {
                // Exception during character decoding: skip parameter
                log.warn("Parameters: Character decoding failed. " + 
                         "Parameter skipped.", e);
            }

            tmpName.recycle();
            tmpValue.recycle();

        } while( pos<end );
    }

In the method, there is do-while loop for parsing all the request parameters. Look at the try-catch block at the end of loop in the method. There is a comment which says

// Exception during character decoding: skip parameter

So even if the exception is thrown but its not propagated. Its logged as a warning message and parameter value is set to null. So it's clear that you will never be able to catch that exception (java.io.CharConversionException)

Answer 3

With this test (including a raw '%' in what is supposed to be URL-encoded data) you are testing that your app appropriately rejects malformed input. Nothing wrong with that.

If you want to push data using burp, you need to encode it, meaning you need to put in '%25' where you want a '%'.

Answer 4

The problem you're running into is that the decoder expects a valid, decodable paramter when it begins with the % symbol. When submitting the form, the input parameter % is correctly encoded into %25, when you edit this request with Burp, this encoding does not happen as far as I know. Burp willingly sends your server the % symbol, and your server side validation assumes this is the beginning of an encoded value, but fails any decoding because its, well, basically a broken parameter.

My best guess is not to send %-values alone (ie without accompying number value). Its like calling a method on a null reference..some things just do not work. I'd say you try to check whether or not the percentage-symbol gets approved or rejected by your own input validation, but as long as it goes through the decoding first, it has to be encoded.