Channels ▼

Eric Bruno

Dr. Dobb's Bloggers

Making HTTP Requests From Java

September 09, 2013

Continuing the Java network communication theme, let's examine how to make HTTP requests from a Java application, including how to add header parameters and proxy information. I've made requests to HTTP servers from Java in the past, but believe it or not I only recently had to consider adding proxy server information. As usual for me, I started at a low level, making the requests using a Java Socket.

To begin, you need to parse the given URL request to extract the host, path, port, and protocol (i.e., HTTP, HTTPS, and so on). For example, let's parse the host and path first:


            String urlStr = "http://www.someserver.com/getdata?param1=abc¶m2=xyz"; // some URL
            URI uri = new URI( urlStr); 
            String host = uri.getHost( ); 
            String path = uri.getRawPath( ); 
            if (path == null || path.length( ) == 0) {
                path = "/";
            } 

            String query = uri.getRawQuery( ); 
            if (query != null && query.length( ) > 0) {
                path += "?" + query;
            } 

Next, let's extract the protocol and port, and make sure they match:


            String protocol = uri.getScheme( ); 
            int port = uri.getPort( ); 
            if (port == -1) {
                if (protocol.equals("http")) { 
                    port = 80; // http port 
                }
                else if (protocol.equals("https")) {
                    port = 443; // https port 
                }
                else {
                    return null;
                }
            }

Now that the required information has been extracted, we need to do three main things. First, make a socket connection to the server; second, send a correctly formatted HTTP request; third, listen for the response. Connecting is simple; just create a new Java Socket with the host and port:

Socket socket = new Socket( host, port ); 

Sending the request is a little trickier. If the HTTP server requires authentication for the request, you need to encode this so it isn't sent in plain text (and hence viewable by anyone looking at the network traffic). The HTTP specification requires you to format it as ":", as in "ericbruno:mypassword" (see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html). To encode it using Base64 encryption, I used the Apache Commons Codec library. Here's the code:

            import org.apache.commons.codec.binary.Base64;
            …

            String username = "ericbruno";
            String password = "mypassword";
            String auth = username + ":" + password;
            String encodedAuth = Base64.encodeBase64String(auth.getBytes());

The HTTP request format for this request needs to look like this, according to the spec:


            GET /getdata?param1=abc¶m2=xyz HTTP/1.1
            Host: www.someserver.com
            Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
            Connection: close

The code to send this request uses the Socket's OutputStream and simply "prints" the request as a String to it:


            PrintWriter request = new PrintWriter( socket.getOutputStream() );
            request.print(  "GET " + path + " HTTP/1.1\r\n" + 
                                   "Host: " + host + "\r\n" + 
                                   "Authorization: Basic " + encodedAuth + "\r\n" +
                                   "Connection: close\r\n\r\n"); 
            request.flush( ); 

The "\r\n" (line feed, new line) combinations at the end of each line in the String ensure that the request meets the HTTP specification formatting requirements. To get the response, use the Socket's InputStream and read the resulting text line-by-line:


            InputStream inStream = socket.getInputStream( ); 
            BufferedReader rd = new BufferedReader(
                    new InputStreamReader(inStream));
            String line;
            while ((line = rd.readLine()) != null) {
                // ...
            }

A Better Way

Using this code works, and it's obvious what's happening since it involves low-level socket communication. There are cases when understanding the low-level workings is important, and for some reason I tend to feel comfortable working at this level. I wrote the basis for this code years ago, and I simply copy/paste from my personal code library each time I need it. However, I ran into a snag when I had to add the proxy information. I'm sure it can be done using this low-level code, but I decided to find a better way. For me, that search ended with the Apache HTTP Client classes, part of the Apache HTTP Components library. Using this library, the code to make the request via a proxy server looks like this:


            DefaultHttpClient httpclient = new DefaultHttpClient();
            if ( useProxy == true ) {
                HttpHost proxy = new HttpHost(proxyStr, 80, "http");
                httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
            }

            HttpGet httpget = new HttpGet(urlStr);
            httpget.addHeader("Authorization", "Basic " + encodedAuth);

            HttpResponse response = httpclient.execute(httpget);

Here, the proxy I needed to use was a basic HTTP proxy, but there's support for others (i.e., SOCKS). If you don't require a proxy, simply set the useProxy flag to false. The response is returned when you make the request via the execute() method. Here's the code to extract the call status (i.e., "HTTP/1.1 200 (OK)"), the headers, and the body of the response:


            String status = response.getStatusLine().toString();

            Header[] headers = response.getAllHeaders();

            HttpEntity entity = response.getEntity();
            BufferedReader rd = new BufferedReader(
                    new InputStreamReader(entity.getContent()));
            String line;
            while ((line = rd.readLine()) != null) {
                // ...
            }

That's much better! Happy coding!
-EJB

Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Comments:

ubm_techweb_disqus_sso_-06a6faddc430162ab6c827d900667643
2013-09-30T18:05:21

Thanks for the feedback and useful information. I apologize if you feel misled. This code was part of a larger project where the Apache classes were used for multiple reasons. Additionally, I wanted to focus on what could be done with the current version of Java, not the next one (JDK 8). However, you've raised some good points and I plan to update this blog with a fresh approach. Thank you.


Permalink
ubm_techweb_disqus_sso_-4a812d7ff9c80b353fd827c838936b7f
2013-09-26T11:14:40

Wauw. Somewhat surprised to see you pull in so many external libraries. I must admit that I feel you lead people a bit astray in your posting.

1. As for Base64 encoding.

This can be done using JAXB class which is part of standard JDK since Java 6 so no need to resort to something else.
Just do:

String encodedString = javax.xml.bind.DatatypeConverter.printBase64Binary(myString.getBytes());

In Java8 there will finally be a base64 encoder as part of the Util class. Until then you can use the JAXB class (even if your application doesn't use JAXB at all).

2. Non-disclosure of password

The only way to secure that your password remains secure between you and the endpoint is to use HTTPS rather than HTTP. The fact that the password is Base64 encoded doesn't really help much in terms of security.

3. HTTP client library

It is true that many people resort to Apache Http Client library for making http requests from Java but the alternative to that is not low-level Sockets but to use the regular JDK classes for HTTP client communication, namely HttpURLConnection and HttpsURLConnection from the standard Java lib. These classes have improved over the years and I personally do not have a need to use Apache Http Client library. These standard JDK classes do in fact support proxy.... as you would expect. Just Google it.


Permalink


Video