Enhancing .NET Web Services

Extending and debugging web services

Eric has developed everything from data reduction software for particle bombardment experiments to software for travel agencies. He can be contacted at ericterrell@ comcast.net.

Calling a web service from a .NET application couldn't be easier. Just select the Project/Add Web Reference menu item. Add a URL to the WSDL file that describes the web service. Press Add Reference and a proxy class is automatically created. This proxy class includes methods corresponding to each of the web service's methods. Calling any of these proxy class methods generates a SOAP (XML) request, sends it to the server, retrieves the SOAP response, and converts it into a response object. But what if you want to request compressed responses, or access or even change the requests and responses? In this article, I show how to ask for compressed responses, capture and optionally modify SOAP requests/responses, and debug web-service calls with a network packet analyzer program. I've included SoapEx, a sample application (available electronically; see "Resource Center," page 5) that demonstrates these techniques. SoapEx (see Figure 1) is a .NET Windows Forms program written in C#. It requires the .NET 1.1 framework and Windows 2000/XP or later. Build it in Visual Studio .NET 2003 by opening SoapEx\SoapEx.sln. You can also install the app by running setup.exe in the setup folder.

When you run SoapEx, it calls the Google web service. Check the Compress Response checkbox to request a compressed response. Check the Modify Soap Request and Response checkbox to change the SOAP requests and responses. You'll need a license key to use the Google web service. Press the Get License Key button to get one, or go to http://www.google.com/apis/. In either case, it will take only a minute or two to apply for a license key and receive it via e-mail.

Press Call Web Service to send the web-service request and retrieve the response. The SOAP request/responses are displayed in the Results text box. Pressing the IP Addresses button displays the IP address of your machine and the server hosting the web service. These IP addresses are useful for troubleshooting web-service calls with Packetyzer.

Many web services can return results in compressed format to conserve bandwidth. Given the verbosity of XML, compression can drastically reduce bandwidth consumption. Unfortunately, .NET web-service proxies do not ask for compressed results by default. But it's easy to correct this situation. The first step is to create a new class derived from the web-service proxy class. (For example, SoapEx has the CompressedGoogleSearch class, which is derived from the GoogleSearchService proxy class.) Then override the derived class's GetWebRequest method to specify an HTTP header Accept-Encoding value of gzip, deflate (Listing One). The Accept-Encoding header specifies the compression formats that the client can cope with. The SoapEx application can decompress both gzipped and deflated responses. The second step is to override your derived class's GetWebResponse method to return a CompressedWebResponse object. The CompressedWebResponse object will be able to decompress web-service responses.

When the server returns the results, they are in gzip format, deflated format, or uncompressed. Bear in mind that the Accept-Encoding header is only a suggestion, not a demand. The CompressedWebResponse.GetResponseStream method decompresses the response Stream if it is actually compressed (Listing Two). GetResponseStream determines the response format by inspecting webResponse.ContentEncoding (see the code inside the try block). If the response is compressed, the correct inputStream is selected: GZipInputStream for gzip, InflaterInputStream for deflate. A BinaryReader object is created to read bytes from the inputStream. As the inputSource is read, the uncompressed data is written to the result Stream using a BinaryWriter. If the response was not compressed, the result Stream is set to the original, uncompressed Stream. In any case, the result Stream is returned at the end of GetResponseStream.

The CompressedWebResponse class uses the #ziplib library (available here) to decompress responses. .NET 2.0 will include built-in libraries for gzip compression and decompression.

Strictly speaking, you can enable compressed responses without deriving a class from the proxy class. You could override GetWebRequest and GetWebResponse in the original proxy class. But changes to the proxy class will be undone if you ever need to regenerate the proxy in the future. Compressing responses on the server and decompressing them on the client consumes additional CPU cycles on both server and client, but the savings in bandwidth is likely to justify the processing cost. At least at the moment, the Google web service does not return compressed responses, perhaps because the responses tend to be small. SoapEx requests compressed responses, and is ready to decompress them if they are returned in the future.

SoapExtensions

A SoapExtension enables your program to wiretap web-service calls and even change the SOAP requests and responses. SoapExtensions are not tied to a particular web service. If your application calls multiple web services, they will all be manipulated by a SoapExtension. .NET's SoapExtension mechanism lets custom code be called at various stages of the web-service call process. If you derive a class from SoapExtension, your class's ProcessMessage method (Listing Three) will be called at the stages of the web-service call process listed in Table 1.

If you intend to capture and/or change SOAP requests and responses, override the ChainStream method in Listing Three. The argument to ChainStream is a Stream containing the current SOAP request or response. In your overridden ChainStream, save the Stream argument and create, save, and return another Stream argument. For example, the sample application saves the argument as oldStream, and creates and returns a new Stream named "newStream."

After .NET has created a SOAP request by serializing the request proxy object, you can capture the SOAP request, and even modify it. Just take the original request, which is stored in newStream, change the XML, and store it in oldStream (see ProcessRequest in Listing Four). When the client receives a SOAP response, the XML can be captured and changed before it's deserialized into a response proxy object. To modify a response, take the original response stored in oldStream, change the XML, and store it in newStream (see ProcessResponse).

You can see SoapEx capture and display SOAP requests and responses by pressing the Call Web Service button and looking in the Results text box. To see SoapEx modify requests and responses, check the Modify Soap Request and Response checkbox, and press the Call Web Service button. Then look at the Results text box. The request and response will be modified by "pretty-printing" them with indentation. I confess: SoapEx's modifications to SOAP requests/responses are somewhat contrived. But if you have control over both the client and server, you can use SoapExtensions to modify requests and responses to implement proprietary compression, encryption, and so on.

Configuring SoapExtensions

A SoapExtension can be wired into an application using a configuration file. For Windows Forms applications, put the configuration settings in a file with the same name as the executable, with an extra .config on the end. For example, the SoapEx's configuration file is named SoapEx.exe.config (Listing Five). For ASP.NET applications, store the same configuration information in a Web.config file.

The <add> element is where the SoapExtension is specified. The type attribute specifies the full name of the SoapEx class in namespace.class format, followed by the name of the assembly containing the SoapExtension. If there are multiple SoapExtensions, the priority attribute determines the order in which they are applied. SoapExtensions with higher priorities are applied before ones with lower priorities. Priority 0 is the highest, 1 is the next highest, and so on. SoapExtensions with a group attribute of 0 have the highest relative priority.

Debugging Web Services

Sometimes, the Visual Studio debugger is not sufficient to debug web services. It's often important to see the exact bytes going over the wire during a web-service call. For example, when a web-service response isn't well-formed XML, the response proxy object will be null, and you'll want to inspect the actual response that the server returned. In these sorts of situations, I recommend Network Chemistry's Packetyzer program (Figure 2), available here. When you launch Packetyzer 3.0.1, the Capture Options dialog lets you "Capture packets in promiscuous mode," which I don't recommend. If your machine has a promiscuous network card, Packetyzer can display traffic from your machine and other machines on the network. Unless you need to monitor other machines' traffic, promiscuous mode just slows down the debugging process.

Launch Packetyzer and then launch SoapEx. Click on the IP Addresses button, which will launch a dialog box (Figure 3). Select the line below the "Packetyzer Advanced Filter" and press Ctrl+C to copy it into the clipboard. Then go to Packetyzer and click the Display Filter drop-down combobox. Press Ctrl-V to paste the filter and press Apply to apply it. This filter specifies that Packetyzer will display HTTP network traffic between your machine and the web server that implements the web service. I recommend taking the time to learn Packetyzer's filtering mechanism—filters can make Packetyzer much more convenient and efficient.

Select Session/Start Capture in Packetyzer's main menu. Then press SoapEx's Call Web Service button. Give Packetyzer a few seconds to capture the traffic. Select Session/Stop capture. Then click on the rows in the list control in the top, right corner of the screen (Figure 2). As you click on the rows, you'll see the entire conversation that SoapEx had with the web server. To see the entire conversation in one stream, right-click any row in the list control and select Follow TCP Flow. Then select the Decode tab and you'll see the web-service request and response in a convenient textual format.

Conclusion

The next time you use Visual Studio's Add Web Reference feature, consider creating derived proxy classes to specify compressed responses. The bandwidth that you save may pay for the modest effort of creating a derived class that requests compressed responses. And consider using a SoapExtension to capture your application's SOAP requests and responses for debugging purposes. Finally, download Packetyzer and install it. You'll want to have it handy the next time you need to do any serious web-service debugging.

protected override WebRequest GetWebRequest(Uri uri)
// Update the request's HTTP headers to specify that
// we can accept compressed (gzipped, deflated) responses
{
  WebRequest request = base.GetWebRequest(uri);
  // If user checked the Compressed Response checkbox
  if (compressResponse)
  {
    request.Headers.Add("Accept-Encoding", "gzip, deflate");
  }
  return request;
}
protected override WebResponse GetWebResponse(WebRequest request)
// If we've requested compressed responses, return a WebResponse
// derivative that's capable of uncompressing it.
{
  WebResponse result;
  // If user checked the Compressed Response checkbox
  if (compressResponse)
  {
    result = new CompressedWebResponse((HttpWebRequest) request);
  }
  else // no compression requested, return stock WebResponse object
  {
    result = base.GetWebResponse(request);
  }
  // Keep track of content length to measure bandwidth savings.
  responseContentLength = result.ContentLength;
  return result;
}

public override Stream GetResponseStream()

// Decompress the web service response and return it in a stream.
{
  Stream result = null;
  // Clean up previously-used BinaryReader and BinaryWriter.
  if (reader != null)
  {
    reader.Close();
    reader = null;
  }
  if (writer != null)
  {
    writer.Close();
    writer = null;
  }
  try
  {
    // Get response.
    HttpWebResponse webResponse = (HttpWebResponse) request.GetResponse();
    Stream inputStream = null;
    bool decompress = true;
    // Get an input stream based on the type of compression, if any.
    if (webResponse.ContentEncoding == "gzip")
    {
      inputStream = new
        GZipInputStream(webResponse.GetResponseStream());
    }
    else if (webResponse.ContentEncoding == "deflate")
    {
      inputStream = new
        InflaterInputStream(webResponse.GetResponseStream());
    }
    else
    {
      // Response wasn't compressed, return the original, uncompressed stream.
      result = webResponse.GetResponseStream();
      decompress = false;
    }
    // If response was actually compressed, decompress it.
    if (decompress)
    {
      // Decompress the input stream.
      reader = new BinaryReader(inputStream);
      result = new MemoryStream();
      writer = new BinaryWriter(result);
      int bytesRead;
      byte[] buffer = new byte[1024];
      do
      {
        // Read from compressed buffer and store the decompressed
        // bytes in the buffer.
        bytesRead = reader.Read(buffer, 0, buffer.Length);
        // Write decompressed buffer to the result stream.
        writer.Write(buffer, 0, bytesRead);
      } while (bytesRead > 0);
      writer.Flush();
      result.Position = 0;
    }
  }
  catch (Exception ex)
  {
    MessageBox.Show(ex.Message, "Exception", 
      MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
  }
  // Returned decompressed response.
  return result;
}

public override void ProcessMessage(SoapMessage message) 
// Hook into the web service process at various stages.
{
  switch (message.Stage) 
  {
    case SoapMessageStage.BeforeSerialize:
      break;
    case SoapMessageStage.AfterSerialize:
      // Capture and optionally modify SOAP request.
      ProcessRequest(message);
      break;
    case SoapMessageStage.BeforeDeserialize:
      // Capture and optionally modify SOAP response.
      ProcessResponse(message);
      break;
    case SoapMessageStage.AfterDeserialize:
      break;
    default:
      throw new Exception("invalid stage");
  }
}
public override Stream ChainStream(Stream stream)
// Save the stream representing the SOAP request or response

{
  oldStream = stream;
  newStream = new MemoryStream();
  return newStream;
}

private void ProcessRequest(SoapMessage message)
// Capture and optionally modify the SOAP request.
{
  newStream.Position = 0;
  using (MemoryStream memoryStream = new MemoryStream())
  {
    CopyStream(newStream, memoryStream);
    // Capture original SOAP request.
    OriginalSoapRequest = GetStreamText(memoryStream);
    // If user has checked the Modify SOAP Request and Response checkbox...
    if (modify)
    {
      // "Pretty-print" SOAP request XML with indentation.
      ModifiedSoapRequest = ModifySOAP(memoryStream, oldStream);
    }
    else
    {
      CopyStream(memoryStream, oldStream);
    }
  }
}
private void ProcessResponse(SoapMessage message)
// Capture and optionally modify the SOAP response.
{
  using (MemoryStream memoryStream = new MemoryStream())
  {
    CopyStream(oldStream, memoryStream);
    OriginalSoapResponse = GetStreamText(memoryStream);
    // If user has checked the Modify SOAP Request and Response checkbox
    if (modify)
    {
      ModifiedSoapResponse = ModifySOAP(memoryStream, newStream);
    }
    else
    {
      CopyStream(memoryStream, newStream);
    }
  }
  newStream.Position = 0;
}

<configuration>
 <system.web>
   <webServices>
     <soapExtensionTypes>
      <add type="SoapExDLL.SoapTraceModify, SoapExDll"
           priority="1"
           group="0" />
     </soapExtensionTypes>
    </webServices>
 </system.web>
</configuration>