|
Eric uses Amazon.com's S3 web service and .NET 2.0's FtpWebRequest class to securely synchronize files on multiple machines.
August 02, 2006
URL:http://www.drdobbs.com/database/synchronize-now/database/synchronize-now/191601621
Eric has developed everything from data-reduction software for particle bombardment experiments to software for travel agencies. He can be contacted at [email protected].
There are several files I need to access and update from practically any computer I use—my web browser favorites, contact lists, web-site passwords, and my .NET programming tips. Although I keep these files on a flash drive, it's not always convenient to take the drive everywhere I go. I'd rather store the files on a server and access them over the Internet, as long as it's reliable, secure, and inexpensive. Xdrive.com offers 5 GB of Internet-accessible storage, but the $10.00 monthly fee is more than I pay for hosting my entire web site. Fortunately, Amazon's S3 web service offers secure Internet file access at modest prices. Additionally, .NET 2.0's FtpWebRequest class offers access by programs to FTP sites. In this article, I show how to use S3 and FtpWebRequest to securely synchronize files on multiple machines.
To illustrate, I use Visual Studio 2005 to develop a sample application named "Synchronize Files" (available at http://www.ddj.com/code/). You can build it with Visual Studio 2005 or Visual C# 2005 Express Edition (http://msdn.microsoft.com/vstudio/express/visualcsharp). This program encrypts files for security, and compresses them to conserve bandwidth and storage space. It takes advantage of the Microsoft Enterprise Library for cryptography, logging, and exception management.
The sample app uses the Cryptography, Exception Handling, and Logging "Application Blocks" from the January 2006 Microsoft Enterprise Library for .NET Framework 2.0. Before building the app, download the Enterprise Library (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/entlib2.asp) and run the install. After installing the library, specify an encryption key to encrypt/decrypt your files. To specify your key, first ensure the sample app's app.config file is writable. Run the Enterprise Library Configuration program. Select File/Open Application and open the sample app's app.config file. Right-click the Symmetric Provider on the left and select Remove. Then right-click Symmetric Providers and select New/Symmetric Algorithm Provider. Select RijndaelManaged and press OK. Make sure the Create a New Key radio button is selected, then click Next. Click the Generate button, then Next. Specify the key file path and press Next. Specify the data-protection mode and press Finish. On the right, change the Name from "RijndaelManaged" to "Symmetric Provider," which is the name the sample app expects. Finally, select File/Save Application.
Now it's time to build the app. Open SynchronizeFiles.sln in Visual Studio 2005, build the app, and run it. Select Options/Settings and select the S3 or FTP radio button to specify where your files will be stored. If you're using S3, enter the Access Key ID and Secret Access Key you received when signing up. Then press Get Canonical User ID and your Canonical User ID and Display Name is filled in. For an FTP site, first use an FTP program to create a folder on the FTP site for your home directory. Then go back to Synchronize Files and enter the FTP site (ftp://upload.comcast.net, for instance), the home directory you created, and your User ID and Password.
After entering your configuration information, press OK. Click the New button to create a bucket (folder) to hold your files. Then click Upload to upload your files. When files are selected by checking the checkbox in the DataGridView's leftmost column (Figure 1), they can be downloaded or deleted. When you press the Synchronize button, all files with a check in the Synchronize column are uploaded to the server if they've changed on the local machine.
The S3 web service (http://www.amazon.com/s3) costs $0.15 per month per GB of storage, and $0.20 per GB of data transferred. If you already have an Amazon account, your existing account is billed. The price is reasonable; so far I've run up a $0.03 bill. All calls to S3 go over an encrypted SSL connection (although you can use a nonSSL connection for debugging or troubleshooting). When signing up for S3, you are assigned an Access Key ID that identifies you in every S3 call. Your Access Key ID is not secret. You are also given a Secret Access Key that creates a signature that authenticates you.
S3 organizes data in "buckets" that are similar to disk folders. Although individual S3 users can have no more than 100 buckets, each bucket can hold an unlimited number of objects. Unlike disk folders, buckets cannot contain other buckets—just objects. Objects are blobs of data from 1 byte to 5 GB. Each object in a given bucket is identified by a unique key. Although my sample app only grants you access to the objects it stores, the S3 API lets you grant access to other users via access-control lists. See S3DataTransfer.AccessControlList in the sample app source code for an example of an access-control list. You can grant yourself access to an object with an access-control list containing either the e-mail address you use for your Amazon account, or your Canonical User ID. I use my Canonical User ID because my e-mail address doesn't work (because my Amazon account somehow ended up with multiple passwords). You can determine your Canonical User ID and display name by calling the ListAllMyBuckets method and inspecting the Owner.ID and Owner.DisplayName properties in the response object. That's how the Options/Settings dialog box does it.
To use S3 in a Visual Studio 2005 application, add a web reference to http://s3.amazonaws.com/doc/2006-03-01/AmazonS3.wsdl. Download the "Amazon S3 Library for SOAP" in C# from Amazon's S3 Resource Center. Add the AWSAuthConnection.cs and AWSDateConnection.cs files to your project. You'll need to add a using statement to the AWSAuthConnection.cs file that points to the web service proxy that you created when you added the web reference. See the sample app's version of AWSAuthConnection.cs for an example of this. Synchronize Files uses the AWSAuthConnection class to make all of its S3 calls because it takes care of all the security details.
Reading through the S3DataTransfer source code, you notice a few S3 API quirks. Currently the AWSAuthConnection.createBucket method always returns a null response object, even when it successfully creates the bucket. That's why S3DataTransfer.CreateBucket calls S3DataTransfer.BucketExists to determine if the bucket was created. Another quirk is that AWSAuthConnection.put stores a string, but AWSAuthConnection.get retrieves a byte array. To overcome this asymmetry, S3DataTransfer.UploadBuffer converts its byte array into a string using UTF8Encoding.GetString. When that byte array is subsequently retrieved using AWSAuthConnection.get in S3DataTransfer.DownloadBuffer, the byte array will be identical to the one previously stored.
Although the S3 fees are not high, if you already have access to an FTP site, you may prefer to host your files there. But beware! Unless your FTP site is running over SSL, your FTP credentials (user ID and password) are transmitted over the Internet in plaintext, and can be intercepted and abused by bad guys. This vulnerability applies to every program that uses the FTP protocol, not just the sample app. You can see this vulnerability with Packetyzer, a freely available network analysis program (http://www.networkchemistry.com/products/packetyzer.php). Start a network capture in Packetyzer. Then use Synchronize Files to download a file from an FTP site. You'll see a trace like the following in Packetyzer:
USER fredjones ..UR..331 Password required for fredjones. PASS fredspassword ...e..230 User fredjones logged in.
Bad guys monitoring packets on any of the machines through which the transfer travels will be able to capture your credentials. Scary! Although the sample app can't prevent this vulnerability, it encrypts your files before uploading them to the FTP site, and decrypts them only after they've been downloaded to your machine. Consequently, there's little risk that bad guys could decrypt your files by intercepting them during transfers, or by downloading them directly from your FTP site.
.NET 2.0 provides native FTP access with FtpWebRequest, but this class is little more than a thin wrapper over the FTP protocol. See FTPDataTransfer for examples of using the FtpWebRequest to synchronize files between the user's machine and an FTP site. FTPDataTransfer.GetFtpWebRequest (Listing One) returns an FtpWebRequest object. The KeepAlive property is set to False to ensure that the connection to the FTP server is closed after every FTP operation. If this isn't done, some FTP operations may fail, depending on the FTP command that was previously executed, and the server's FTP software. Although the KeepAlive = false setting allows arbitrary command sequences to succeed, creating a new FTP connection for each operation is slow. Furthermore, your FTP credentials will be sent to establish each new connection. GetFtpWebRequest includes a Method parameter that specifies the FTP operation that will be performed. There are several FTP operations including UploadFile, DownloadFile, DeleteFile, ListDirectory, and so on. Finally, the UseBinary property should be set to True. Otherwise, binary files will not be correctly transferred.
...
class FTPDataTransfer : DataTransfer
{
...
private FtpWebRequest GetFtpWebRequest(string FTPAddress, string Method)
{
FtpWebRequest ftpWebRequest = null;
try
{
ftpWebRequest = (FtpWebRequest)WebRequest.Create(FTPAddress);
// Start a new FTP session.
ftpWebRequest.KeepAlive = false;
// Data will be transferred in binary format.
ftpWebRequest.UseBinary = true;
ftpWebRequest.Credentials =
new NetworkCredential(userID, password);
ftpWebRequest.Method = Method;
}
catch (UriFormatException ex)
{
// Display error message if FTP address is invalid.
MessageBox.Show(ex.Message, Application.ProductName,
MessageBoxButtons.OK, MessageBoxIcon.Error);
}
return ftpWebRequest;
}
...
protected override byte[] DownloadBuffer(string BucketName,
string FileName)
{
WebResponse response = null;
// Convert bucket and filename into legal FTP folder and filenames.
BucketName = EncodeName(BucketName);
FileName = EncodeName(FileName);
byte[] result = null;
try
{
string ftpAddress = ftpSite + "/" + ftpHomeDirectory +
"/" + BucketName + "/" + FileName;
FtpWebRequest
ftpWebRequest = GetFtpWebRequest(ftpAddress,
WebRequestMethods.Ftp.DownloadFile);
response = ftpWebRequest.GetResponse();
// Prepare to read the file from the response stream.
Stream responseStream = response.GetResponseStream();
BinaryReader binaryReader = new BinaryReader(responseStream);
const int bufferSize = 1024;
byte[] buffer;
List<byte> bufferList = new List<byte>();
// Read the file, one buffer at a time.
do
{
buffer = binaryReader.ReadBytes(bufferSize);
if (buffer.Length > 0)
{
bufferList.AddRange(buffer);
}
} while (buffer.Length == bufferSize);
result = bufferList.ToArray();
binaryReader.Close();
}
catch (WebException)
{
// A WebException will be thrown if file does not exist on server.
}
finally
{
if (response != null)
{
response.Close();
}
}
return result;
}
...
}
FTPTransfer.DownloadBuffer calls GetFtpWebRequest to download a file's contents (Listing One). First the EncodeName method is called to convert the bucket name and file name into characters that FTP supports. Then GetFtpWebRequest is called with the FTP address of the file that will be downloaded, and the DownloadFile method. GetResponse is called to access the FtpWebRequest's response stream. The contents of the file are then read from the stream using a BinaryReader. After the stream is closed, the file's contents are returned. When a file is uploaded to an FTP site, its contents are written to the FtpWebRequest's request stream (see FTPTransfer.UploadBuffer for details).
Synchronize Files uses the DataTransfer abstract class to transfer files to and from S3 or an FTP site. The DataTransfer class includes nonabstract methods such as UploadFile that are independent of the transfer mechanism (S3 or FTP), and abstract methods such as UploadBuffer that are totally dependent on the transfer mechanism, and implemented in the S3DataTransfer and FTPDataTransfer classes. The sample app is able to transfer files using a DataTransfer object, without having to be aware of the differences between S3 and FTP.
DataTransfer maintains metadata for each file that is uploaded to the server. The metadata includes an SHA1 hash of the file's contents. This hash is a 160-bit binary number that is computed from the file's contents. Because it is extremely unlikely that two different files will have the same hash, the hash is used to avoid unnecessary file transfers. If the user requests a file download, the file will only be downloaded if the server's copy has a different hash than the local file. And a file will only be uploaded if its hash differs from the hash of the corresponding server file. The sample app also uses the hash to detect file corruption. When a file is downloaded, the sample app knows the file's contents have changed due to corruption or sabotage if its hash differs from the one stored on the server. To compute a hash from an array of bytes, instantiate an SHA1Managed object and call its ComputeHash method.
DataTransfer compresses and encrypts all files uploaded to the S3 or FTP server, and decrypts and decompresses all downloaded files. The Serialization class does the compression/decompression and encryption/decryption. For example, Synchronize Files calls Serialization.ObjectToBase64String (Listing Two) to compress and encrypt a file prior to uploading it. First the ObjectValue parameter is serialized to a MemoryStream using a BinarySerializer. The serialized data is extracted from the MemoryStream into a byte array. The byte array is then compressed using the Compression class. Thanks to the Enterprise Library's Cryptography Application Block, encrypting the byte array is just a simple call to Cryptographer.EncryptSymmetric. Decryption is just as easy, just a call to Cryptographer.DecryptSymmetric. In addition to simplifying encryption and decryption, the Cryptography application block decouples the cryptography algorithm from the code that uses it. The sample app is configured to use the Rijndael algorithm, but a different algorithm can be substituted with no code changes. Just run the Enterprise Library Configuration program. Select File/Open Application and open the sample app's app.config file. Right-click the Symmetric Provider and select Remove. Then right-click Symmetric Providers and select New/Symmetric Algorithm Provider. Specify a different algorithm (say, TripleDESCryptoServiceProvider), create a key, specify a key file path, and specify the data-protection mode. Finally, change the name of the TripleDESCryptoServiceProvider on the right to Symmetric Provider.
...
public static class Serialization
{
public static string ObjectToBase64String(object ObjectValue)
{
string objectString = null;
using (MemoryStream memoryStream = new MemoryStream())
using (StreamReader streamReader = new StreamReader(memoryStream))
{
BinaryFormatter binaryFormatter = new BinaryFormatter();
// Serialize the object into the memory stream.
binaryFormatter.Serialize(memoryStream, ObjectValue);
// Rewind the memory stream.
memoryStream.Position = 0;
// Extract the object's data into a byte array.
byte[] buffer = new byte[memoryStream.Length];
Array.Copy(memoryStream.GetBuffer(), buffer, buffer.Length);
// Compress the object's data.
buffer = Compression.Compress(buffer);
try
{
int plainTextLength = buffer.Length;
// Encrypt the object's data.
buffer = Cryptographer.EncryptSymmetric(
StringLiterals.SymmetricProvider, buffer);
int cipherTextLength = buffer.Length;
Trace.WriteLine(string.Format(
"{0} bytes encrypted into {1} bytes",
plainTextLength, cipherTextLength));
// Convert the compressed, encrypted object data
// into a base 64 string.
objectString = Convert.ToBase64String(buffer);
}
catch (Exception ex)
{
objectString = null;
if (ExceptionPolicy.HandleException(ex,
StringLiterals.CaughtExceptionPolicy))
{
throw;
}
}
}
return objectString;
}
...
}
Encrypting your files is useless if a bad guy gets hold of your private key. That's why the Cryptography Applicaton Block encrypts keys using DPAPI. This makes the key useless on any machine other than the one on which it was originally encrypted. This extra level of security makes it less convenient to transfer your key to other machines, which is exactly what you'll need to do if you intend to run Synchronize Files on multiple computers. To transfer a key to another machine, first make sure your key file is writable. Then run the Enterprise Library Configuration program, select File/Open Application, and open the sample app's app.config file. Right-click the Symmetric Provider on the left and select Export key. Specify a filename and a password for the exported key and press OK. Then copy the exported key file to the second machine. On the second machine, run the Enterprise Library Configuration program. Remove the Symmetric Provider on the left. Right-click Symmetric Providers on the left and select New/Symmetric Algorithm Provider. Choose RijndaelManaged and press OK. Then select the "Import a password-protected key file" radio button and press Next. Choose the exported key file and enter the password you used to encrypt it. Click Next and specify where the imported key file should be stored. After completing the key import process, change the Name field on the right to Symmetric Provider.
The Compression class (Listing Three) uses the .NET 2.0 GZipStream class to compress and decompress files. Compression.Compress instantiates a MemoryStream and associates it with a GZipStream with a CompressionMode of compress. When a byte array is written into the GZipStream, then retrieved using the MemoryStream's GetBuffer method, it is compressed. Attempting to compress very small files, or already compressed files such as Zip or JPEG files, will tend to result in larger, not smaller, files.
...
public static class Compression
{
...
public static byte[] Compress(byte[] Buffer)
{
byte[] result = null;
using (MemoryStream memoryStream = new MemoryStream())
using (GZipStream gzipStream = new GZipStream(memoryStream,
CompressionMode.Compress, true))
{
// Write the buffer into the GZip stream.
gzipStream.Write(Buffer, 0, Buffer.Length);
gzipStream.Close();
// Extract the compressed data from the GZip stream.
result = new byte[memoryStream.Length];
Array.Copy(memoryStream.GetBuffer(), result, result.Length);
}
if (Buffer.Length > 0)
{
Trace.WriteLine(string.Format(
"{0} bytes compressed to {1} bytes ({2}%)", Buffer.Length,
result.Length, ((double)result.Length /
(double)Buffer.Length) * 100.0));
}
return result;
}
public static byte[] Decompress(byte[] Buffer)
{
byte[] result = null;
using (MemoryStream memoryStream = new MemoryStream(Buffer))
using (GZipStream gzipStream = new GZipStream(memoryStream,
CompressionMode.Decompress, true))
{
result = ReadBytes(gzipStream);
}
return result;
}
}
The DataTransfer class includes two methods—EncodeName and DecodeName—that relax restrictions in S3 and FTP bucket and file names. For example, S3 Bucket Names currently only allow ASCII letters, numbers, underscores, and dashes. This is a surprising restriction considering the popularity of Unicode for web-service text encoding. The sample application can handle bucket names and filenames with no restrictions because EncodeName converts unsupported characters to supported characters, and DecodeName does the reverse. This lets the sample application display bucket and filenames in high market-share languages such as Chinese.
DataTransfer the Enterprise Library's Logging Application Block to log file transfers and errors. The log file is named trace.log and is stored in the same folder as the .exe. Most messages go to this log file. Serious errors are sent to the system event log that you can inspect by going to Control Panel/Administrative Tools/Event Viewer. The Logging Block provides two major advantages to applications that use it:
protected void Log(string Category, string Message)
{
LogEntry logEntry = new LogEntry();
logEntry.Categories.Add(Category);
logEntry.Title = Application.ProductName;
logEntry.EventId = ++eventId;
logEntry.Priority = 0;
logEntry.Message = Message;
logEntry.ExtendedProperties.Add("Protocol", ProtocolName());
Logger.Write(logEntry);
}
catch (Exception ex)
{
if (ExceptionPolicy.HandleException(ex, "policy"))
{
throw;
}
}
The effect of this code depends on the configuration file. The policy can specify that the exception is rethrown, replaced with a different exception, wrapped in another exception, or logged. If the exception should be thrown, HandleException will return True.
The S3 web service is a simple way to store and retrieve data over the Internet. But if you already have access to an FTP site and can live with the security drawbacks, you can use the .NET 2.0 FtpWebRequest class instead. In either case, the .NET 2.0 GZipStream class is a convenient way to compress data to conserve bandwidth and storage space. The Microsoft Enterprise Library contains some invaluable Application Blocks that simplify application development, and let many important modifications be accomplished with configuration updates rather than code changes. The cryptography, logging, and Exception Handling Application Blocks simplified the sample app's design and coding. I recommend you download the Enterprise Library and familiarize yourself with all of its capabilities. And if you ever troubleshoot Internet applications, you're "flying blind" without a network capture and analysis tool like Packetyzer.
Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.