Channels ▼

Eric Bruno

Dr. Dobb's Bloggers

Making HTTP Requests From Java

September 09, 2013

Continuing the Java network communication theme, let's examine how to make HTTP requests from a Java application, including how to add header parameters and proxy information. I've made requests to HTTP servers from Java in the past, but believe it or not I only recently had to consider adding proxy server information. As usual for me, I started at a low level, making the requests using a Java Socket.

To begin, you need to parse the given URL request to extract the host, path, port, and protocol (i.e., HTTP, HTTPS, and so on). For example, let's parse the host and path first:


            String urlStr = "http://www.someserver.com/getdata?param1=abc¶m2=xyz"; // some URL
            URI uri = new URI( urlStr); 
            String host = uri.getHost( ); 
            String path = uri.getRawPath( ); 
            if (path == null || path.length( ) == 0) {
                path = "/";
            } 

            String query = uri.getRawQuery( ); 
            if (query != null && query.length( ) > 0) {
                path += "?" + query;
            } 

Next, let's extract the protocol and port, and make sure they match:


            String protocol = uri.getScheme( ); 
            int port = uri.getPort( ); 
            if (port == -1) {
                if (protocol.equals("http")) { 
                    port = 80; // http port 
                }
                else if (protocol.equals("https")) {
                    port = 443; // https port 
                }
                else {
                    return null;
                }
            }

Now that the required information has been extracted, we need to do three main things. First, make a socket connection to the server; second, send a correctly formatted HTTP request; third, listen for the response. Connecting is simple; just create a new Java Socket with the host and port:

Socket socket = new Socket( host, port ); 

Sending the request is a little trickier. If the HTTP server requires authentication for the request, you need to encode this so it isn't sent in plain text (and hence viewable by anyone looking at the network traffic). The HTTP specification requires you to format it as ":", as in "ericbruno:mypassword" (see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html). To encode it using Base64 encryption, I used the Apache Commons Codec library. Here's the code:

            import org.apache.commons.codec.binary.Base64;
            …

            String username = "ericbruno";
            String password = "mypassword";
            String auth = username + ":" + password;
            String encodedAuth = Base64.encodeBase64String(auth.getBytes());

The HTTP request format for this request needs to look like this, according to the spec:


            GET /getdata?param1=abc¶m2=xyz HTTP/1.1
            Host: www.someserver.com
            Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
            Connection: close

The code to send this request uses the Socket's OutputStream and simply "prints" the request as a String to it:


            PrintWriter request = new PrintWriter( socket.getOutputStream() );
            request.print(  "GET " + path + " HTTP/1.1\r\n" + 
                                   "Host: " + host + "\r\n" + 
                                   "Authorization: Basic " + encodedAuth + "\r\n" +
                                   "Connection: close\r\n\r\n"); 
            request.flush( ); 

The "\r\n" (line feed, new line) combinations at the end of each line in the String ensure that the request meets the HTTP specification formatting requirements. To get the response, use the Socket's InputStream and read the resulting text line-by-line:


            InputStream inStream = socket.getInputStream( ); 
            BufferedReader rd = new BufferedReader(
                    new InputStreamReader(inStream));
            String line;
            while ((line = rd.readLine()) != null) {
                // ...
            }

A Better Way

Using this code works, and it's obvious what's happening since it involves low-level socket communication. There are cases when understanding the low-level workings is important, and for some reason I tend to feel comfortable working at this level. I wrote the basis for this code years ago, and I simply copy/paste from my personal code library each time I need it. However, I ran into a snag when I had to add the proxy information. I'm sure it can be done using this low-level code, but I decided to find a better way. For me, that search ended with the Apache HTTP Client classes, part of the Apache HTTP Components library. Using this library, the code to make the request via a proxy server looks like this:


            DefaultHttpClient httpclient = new DefaultHttpClient();
            if ( useProxy == true ) {
                HttpHost proxy = new HttpHost(proxyStr, 80, "http");
                httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
            }

            HttpGet httpget = new HttpGet(urlStr);
            httpget.addHeader("Authorization", "Basic " + encodedAuth);

            HttpResponse response = httpclient.execute(httpget);

Here, the proxy I needed to use was a basic HTTP proxy, but there's support for others (i.e., SOCKS). If you don't require a proxy, simply set the useProxy flag to false. The response is returned when you make the request via the execute() method. Here's the code to extract the call status (i.e., "HTTP/1.1 200 (OK)"), the headers, and the body of the response:


            String status = response.getStatusLine().toString();

            Header[] headers = response.getAllHeaders();

            HttpEntity entity = response.getEntity();
            BufferedReader rd = new BufferedReader(
                    new InputStreamReader(entity.getContent()));
            String line;
            while ((line = rd.readLine()) != null) {
                // ...
            }

That's much better! Happy coding!
-EJB

Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Comments:



Video