JavaQuestions on HttpURLConnection and Proxies

Questions on HttpURLConnection and Proxies

Developer.com content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

There’s no shortage of Internet protocols. I was recently reviewing the official list of port assignments (www.iana.org/assignments/port-numbers) and I was surprised at how imaginative protocols designers have been.

Yet only a handful of those protocols are commonly used, chief amongst them is HTTP. HTTP is particularly attractive because it is accepted by corporate firewalls. Consequently, HTTP has outgrown its origins in Web browsing and has been adopted by many applications. For example, the latest XML protocols, such as SOAP, are also build on HTTP.

Java Support

Java supports HTTP through two APIs. The servlet API (and JSP) covers server-side programming while the java.net package offers client-side support through HttpURLConnection.

While I have had few problems with the servlet API, I have found that HttpURLConnection requires some work to interface with firewalls. All the services you need are available but the documentation is sketchy at best. Here are some of the problems I have encountered… and the solutions.

Typically, a client on a corporate network has no direct connection to the Internet. Access goes through proxy servers that monitor the traffic and enforce security rules.

The Basics

In the java.net API, proxies are supported through two system properties: http.proxyHost and http.proxyPort. They must be set to the proxy server and port respectively. The following code fragment illustrates it:

String url = "http://www.marchal.com/",
       proxy = "proxy.mydomain.com",
       port = "8080";
URL server = new URL(url);
Properties systemProperties = System.getProperties();
systemProperties.setProperty("http.proxyHost",proxy);
systemProperties.setProperty("http.proxyPort",port);
HttpURLConnection connection = (
    HttpURLConnection)server.openConnection();
connection.connect();
InputStream in = connection.getInputStream();
readResponse(in);

Of course, you would need to use the proxy and port values suitable for your network. You can most likely find them in the configuration of your browser. If unsure, ask your system administrator.

Using Authentication

Increasingly, companies require employees to log in to the proxy before accessing the Internet. Login is used to better monitor Internet usage; for example, to monitor what sites are visited.

HttpURLConnection supports proxy authentication through the Authenticator class. To enable authentication, your application subclasses Authenticator and defines the getPasswordAuthentication() method. A minimalist implementation is as follows:

public class SimpleAuthenticator
   extends Authenticator
{
   private String username,
                  password;
                     
   public SimpleAuthenticator(String username,String password)
   {
      this.username = username;
      this.password = password;
   }
   
   protected PasswordAuthentication getPasswordAuthentication()
   {
      return new PasswordAuthentication(
             username,password.toCharArray());
   }
}

Next, it must register the authenticator through Authenticator.setDefault(). If we adapt the previous code sample to use Authenticator, it looks like this:

String url = "http://www.marchal.com/",
       proxy = "proxy.mydomain.com",
       port = "8080",
       username = "usr",
       password = "pwd";
Authenticator.setDefault(new SimpleAuthenticator(
                        username,password));
URL server = new URL(url);
Properties systemProperties = System.getProperties();
systemProperties.setProperty("http.proxyHost",proxy);
systemProperties.setProperty("http.proxyPort",port);
HttpURLConnection connection = (
    HttpURLConnection)server.openConnection();
connection.connect();
InputStream in = connection.getInputStream();
readResponse(in);

A More Serious Issue

So far, I have covered the most common situations where HttpURLConnection works properly. However, I have encountered a few networks where HttpURLConnection is not usable.

It appears that the problem is linked to a faulty configuration of the DNS server. For some reason, HttpURLConnection always attempts to resolve the host name against the DNS server. Normally, it fails gracefully and the connection goes is rerouted through the proxy server. A few DNS servers return an inappropriate answer that results in a UnknownHostException.

There’s an interesting theoretical debate as to whether the DNS server behaviour is acceptable. Although I am no expert on the topic, it would appear it is not. However, as a developer, you seldom have the option to reconfigure the DNS server, so you have to find a workaround.

My solution is to roll out my own implementation of the HTTP protocol. In its simplest form, a GET request looks like the following listing:

String url = "http://www.marchal.com/",
       proxy = "proxy.mydomain.com",
       port = "8080",
       authentication = "usr:pwd";
URL server = new URL(url);
Socket socket = new Socket(proxy,port);
Writer writer = new OutputStreamWriter(socket.getOutputStream(),
                                       "US-ASCII");
writer.write("GET " + server.toExternalForm() + " HTTP/1.0rn");
writer.write("Host: " + server.getHost() + "rn");
writer.write("Proxy-Authorization: Basic "
             + new sun.misc.BASE64Encoder().encode(
               authentication.getBytes())
             + "rnrn");
writer.flush();
BufferedReader reader = new BufferedReader(new InputStreamReader(
                            socket.getInputStream(),"US-ASCII"));
String line = reader.readLine();
if(line != null && line.startsWith("HTTP/"))
{
   int sp = line.indexOf(' ');
   String status = line.substring(sp + 1,sp + 4);
   if(status.equals("200"))
   {
      while(line.length() != 0)
         line = reader.readLine();
      readResponse(reader);
   }
   else
      throw new FileNotFoundException("Host reports error " +
                                       status);
}
else
   throw new IOException("Bad protocol");
reader.close();
writer.close();
socket.close();

Notice that the proxy username and password are given as username:password and are later encoded in base 64. For a complete description of the HTTP protocol, turn to http://www.ietf.org/rfc/rfc2616.txt.

Conclusion

HTTP is an interesting protocol because it is supported by all corporate firewalls. The Java developer needs to pay special attention to proxies and other authentication issues, though.

Benoît Marchal is a Belgian developer and writer. He is the author of XML by Example (two editions), Applied XML Solutions, and Java Web Services.

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Latest Posts

Related Stories