Java library for URL encoding if necessary (like a browser)

Question

If I put the http://localhost:9000/space test URL to the address bar of a web browser it calls the server with http://localhost:9000/space%20test. http://localhost:9000/specÁÉÍtest will be also encoded to http://localhost:9000/spec%C3%81%C3%89%C3%8Dtest.

If put the encoded URLs to the address bar (i.e. http://localhost:9000/space%20test and http://localhost:9000/spec%C3%81%C3%89%C3%8Dtest) they remain the same (they won't be double-encoded).

Is there any Java API or library which does this encoding? The URLs comes from the user so I don't know if they are encoded or not.

(If there isn't would it be enough to search for % in the input string and encode if it's not found, or is there any special case where this would not work?)

Edit:

URLEncoder.encode("space%20test", "UTF-8") returns with space%2520test which is not what I would like since it is double-encoded.

Edit 2:

Furthermore, browsers handle partially encoded URLs, like http://localhost:9000/specÁÉ%C3%8Dtest, well, without double-encoding them. In this case the server receives the following URL: http://localhost:9000/spec%C3%81%C3%89%C3%8Dtest. It is same as the encoded form of ...specÁÉÍtest.

Answer 1

What every web developer must know about URL encoding

Url Encoding Explained

Why do I need URL encoding?

The URL specification RFC 1738 specifies that only a small set of characters 
can be used in a URL. Those characters are:

A to Z (ABCDEFGHIJKLMNOPQRSTUVWXYZ)
a to z (abcdefghijklmnopqrstuvwxyz)
0 to 9 (0123456789)
$ (Dollar Sign)
- (Hyphen / Dash)
_ (Underscore)
. (Period)
+ (Plus sign)
! (Exclamation / Bang)
* (Asterisk / Star)
' (Single Quote)
( (Open Bracket)
) (Closing Bracket)

How does URL encoding work?

All offending characters are replaced by a % and a two digit hexadecimal value 
that represents the character in the proper ISO character set. Here are a 
couple of examples:

$ (Dollar Sign) becomes %24
& (Ampersand) becomes %26
+ (Plus) becomes %2B
, (Comma) becomes %2C
: (Colon) becomes %3A
; (Semi-Colon) becomes %3B
= (Equals) becomes %3D
? (Question Mark) becomes %3F
@ (Commercial A / At) becomes %40

Simple Example:

import java.util.logging.Level;
import java.util.logging.Logger;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;

public class TextHelper {
    private static ScriptEngine engine = new ScriptEngineManager()
        .getEngineByName("JavaScript");

/**
 * Encoding if need escaping %$&+,/:;=?@<>#%
 *
 * @param str should be encoded
 * @return encoded Result 
 */
public static String escapeJavascript(String str) {
    try {
        return engine.eval(String.format("escape(\"%s\")", 
            str.replaceAll("%20", " "))).toString()
                .replaceAll("%3A", ":")
                .replaceAll("%2F", "/")
                .replaceAll("%3B", ";")
                .replaceAll("%40", "@")
                .replaceAll("%3C", "<")
                .replaceAll("%3E", ">")
                .replaceAll("%3D", "=")
                .replaceAll("%26", "&")
                .replaceAll("%25", "%")
                .replaceAll("%24", "$")
                .replaceAll("%23", "#")
                .replaceAll("%2B", "+")
                .replaceAll("%2C", ",")
                .replaceAll("%3F", "?");
    } catch (ScriptException ex) {
        Logger.getLogger(TextHelper.class.getName())
            .log(Level.SEVERE, null, ex);
        return null;
    }
}

Answer 2

This is a Scala code snippet. This encoder will encode non-ascii characters and reserved characters in the URL. Also, as the operation is idempotent, the URL won't be double-encoded.

import java.net.URL
import scala.util.parsing.combinator.RegexParsers

object IdempotentURLEncoder extends RegexParsers {
  override def skipWhitespace = false
  private def segment = rep(char)
  private def char = unreserved | escape | any ^^ { java.net.URLEncoder.encode(_, "UTF-8") }
  private def unreserved = """[A-Za-z0-9._~!$&'()*+,;=:@-]""".r
  private def escape = """%[A-Fa-f0-9]{2}""".r
  private def any = """.""".r
  private def encodeSegment(input: String): String = parseAll(segment, input).get.mkString
  private def encodeSearch(input: String): String = encodeSegment(input)
  def encode(url: String): String = {
    val u = new URL(url)
    val path = u.getPath.split("/").map(encodeSegment).mkString("/")
    val query = u.getQuery match {
      case null      => ""
      case q: String => "?" + encodeSearch(q)
    }
    val hash = u.getRef match {
      case null      => ""
      case h: String => "#" + encodeSegment(h)
    }
    s"${u.getProtocol}://${u.getAuthority}$path$query$hash"
  }
}

Example usage(test code)

import org.scalatest.{ FunSuite, Matchers }

class IdempotentURLEncoderSpec extends FunSuite with Matchers {
  import IdempotentURLEncoder._

  test("Idempotent operation") {
    val url = "http://ja.wikipedia.org/wiki/文字"
    assert(encode(url) == encode(encode(url)))
    assert(encode(url) == encode(encode(encode(url))))
  }

  test("Segment encoding") {
    encode("http://ja.wikipedia.org/wiki/文字")
      .shouldBe("http://ja.wikipedia.org/wiki/%E6%96%87%E5%AD%97")
  }

  test("Query string encoding") {
    encode("http://qiita.com/search?utf8=✓&sort=rel&q=開発&sort=rel")
      .shouldBe("http://qiita.com/search?utf8=%E2%9C%93&sort=rel&q=%E9%96%8B%E7%99%BA&sort=rel")
  }

  test("Hash encoding") {
    encode("https://www.google.co.jp/#q=文字")
      .shouldBe("https://www.google.co.jp/#q=文字")
  }

  test("Partial encoding") {
    encode("http://en.wiktionary.org/wiki/français")
      .shouldBe("http://en.wiktionary.org/wiki/fran%C3%A7ais")
  }

  test("Space is encoded as +") {
    encode("http://example.com/foo bar buz")
      .shouldBe("http://example.com/foo+bar+buz")
  }

  test("Multibyte domain names are not supported yet :(") {
    encode("http://日本語.jp")
      .shouldBe("http://日本語.jp")
  }
}

This code is from Qiita.

Answer 3

Finally, I've checked what Firefox and Chrome do. I've used the following URL with both browsers and capture the HTTP request with netcat (nc -l -p 9000):

http://localhost:9000/!"$%&'()*+,-./:;<=>?@[\]^_`{|}~

This URL contains every character from ASCII 32 to 127 except [0-9A-Za-z#].

The captured request is the following with Firefox 18.0.1:

GET /!%22$%&%27()*+,-./:;%3C=%3E?@[\]^_%60{|}~%7F HTTP/1.1

With Chrome:

GET /!%22$%&'()*+,-./:;%3C=%3E?@[\]^_`{|}~%7F HTTP/1.1

Firefox encodes more characters than Chrome. Here is it in a table:

Char | Hex    | Dec     | Encoded by
-----------------------------------------
"    | %22    | 34      | Firefox, Chrome
'    | %27    | 39      | Firefox
<    | %3C    | 60      | Firefox, Chrome
>    | %3E    | 62      | Firefox, Chrome
`    | %60    | 96      | Firefox
     | %7F    | 127     | Firefox, Chrome

I've found some code in their source tree which does something similar but I'm not quite sure that these are the actually used algorithms or not:

• Chrome: http://src.chromium.org/viewvc/chrome/trunk/src/net/base/escape.cc?revision=HEAD&view=markup
• Firefox: toolkit/components/url-classifier/nsUrlClassifierUtils.cpp

Anyway, here is a proof of concept code in Java:

// does not handle "#"
public static String encode(final String input) {
    final StringBuilder result = new StringBuilder();
    for (final char c: input.toCharArray()) {
        if (shouldEncode(c)) {
            result.append(encodeChar(c));
        } else {
            result.append(c);
        }
    }
    return result.toString();
}

private static String encodeChar(final char c) {
    if (c == ' ') {
        return "%20"; // URLEncode.encode returns "+"
    }
    try {
        return URLEncoder.encode(String.valueOf(c), "UTF-8");
    } catch (final UnsupportedEncodingException e) {
        throw new IllegalStateException(e);
    }
}

private static boolean shouldEncode(final char c) {
    if (c <= 32 || c >= 127) {
        return true;
    }
    if (c == '"' || c == '<' || c == '>') {
        return true;
    }
    return false;
}

Since it uses URLEncoder.encode, it handles ÁÉÍ characters as well as ASCII characters.

Answer 4

Use the java java.net.URLEncoder#encode():

String page = "space test";
String ecodedURL = "http://localhost:9000/" + URLEncoder.encode(page, "UTF-8");

Note: encoding the complete URL would result in an undesired situation, for example http:// encodes in http%3A%2F%2F!

Edit: to prevent encoding an URL twice you could check whether the URL contains a % as it is only valid for encodings. But if a user wrongly messes up the encodings (like, only encode the URL partially or use a % in an URL without it being used for encoding something) then there is not much to do using this method...

Answer 5

Standard Java api's it self will do the URL encoding and decoding.

java.net.URI

try the classes URLDecoder and URLEncoder

To encode text for safe passage through the internets:

import java.net.*;
...
try {
    encodedValue= URLEncoder.encode(rawValue, "UTF-8");
} catch (UnsupportedEncodingException uee) { }

And to decode:

try {
    decodedValue = URLDecoder.decode(rawValue, "UTF-8");
} catch (UnsupportedEncodingException uee) { }