Channels ▼


Distributed Computing: Windows and Linux

Source Code Accompanies This Article. Download It Now.

System1 and System2

The Windows-based System1 consists of the following programs:

  • The Generator, which generates statistical data sets, written in Visual C++.
  • The NIST statistical suite, which processes the data sets generated with the Generator, written in C.
  • The Extractor, which extracts and summarizes the results generated from the NIST statistical suite, written in Visual Basic 6.

For its part, System2 works as follows:

  • Data sets are generated with the Generator on the Windows machine.
  • These data sets are then transferred to a shared hard-disk partition on the Linux cluster.
  • The NIST statistical suite processes the data sets on the Linux cluster.
  • The results are then transferred back to the Windows machine.
  • Finally, the data are extracted and summarized by the Extractor on the Windows machine.

Our biggest challenge here was how to get System2 programs to communicate with each other, and how to automate the system as much as possible. To accomplish this, we had to modify the current programs and write new ones. In a nutshell, we decided to divide System2 into three parts:

  • Data sets generation and transmission to the Linux cluster.
  • Processing data on the Linux cluster.
  • Transmitting results to the Windows machine for analysis.

Once we generated the data using the Generator, our next problem was how to transfer it to the Linux cluster. We tried several techniques, including generating all data sets, then transferring them via the Samba client. But the problem here is that we had to wait until all the data is generated (which could take a couple of days), then start the Samba client from a Linux emulator under Windows (cgywin). Clearly, this didn't suit our needs.

In another attempt, during generation of the data sets, we started WinSCP using the command, "Keep the remote directory up to date," and changed the priority of the WinSCP to a higher priority than that of the Generator, to transfer the data once it is created. The problem with this approach was that we ran out of space on the Windows machine, as all the generated data sets are stored on the Windows machine.

Finally, we modified the Generator by adding the function MoveData() (Listing One), which calls a WinSCP script (Listing Two) to copy the data sets from the Windows machine to the Linux cluster. It then deletes these data sets from the Windows machine (to free up space for more generated data sets). MoveData() is then called after each group of data sets is generated.

void MoveData()
system("F:\\WinSCP3\\WinSCP3 /console /script=c:\\move.txt");
system("del c:\\test\\* /q");

Listing One

# Automatically answer all prompts negatively not to 
# stall the script on error
#     option batch on
# Disable overwrite confirmations that 
# conflict with the previous
#     option confirm off
# Connect a stored connection called mido 
# (where the password was saved)
#     open mido
# Change remote directory
#    cd /users/disk2/et/midono1/test
# Force binary mode transfer (It is very 
# important for us to transfer the data in binary
# format) 
#    option transfer binary
# Upload the file to current working directory
put c:\test\*
# Disconnect
# Exit WinSCP

Listing Two

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.