Channels ▼

Jack Woehr

Dr. Dobb's Bloggers

Pipes vs Pipelines

November 30, 2008

CMS Pipelines Runtime Library logo Sometimes one hardly recognizes one's own children! CMS Pipelines was inspired by Unix pipes but morphed into something recognizably different under the CMS guest operating system running under IBM VM .

Once you get used to the idea that topologies need not be simple left-to-right, there's a lot of things that can be done in a very natural efficient way. - Jeff Savit

The theoretical difference between Pipelines and Unix pipes is exemplary of the difference between the Unix viewpoint and the Mainframe viewpoint. On Unix, data goes from Here to There. On Mainframes, data goes from Lotsa Places to Everywhere.

When Roland Mainz asked me on the SOL-390 list:

Jack Woehr wrote:<br />
<span class="moz-txt-citetags">> </span>The main coolosity about Pipelines vs Unix pipes (from a Unicist point<br /><span class="moz-txt-citetags">> </span>of view) is that Pipelines are inherently "tees".<br />
<!---->Can you describe this from an Unix standpoint somehow, e.g. a short<br />pseudocode example would be nice...

Jeff Savit, a Pricinpal Field Technologist at Sun Microsystems, jumped in with the following. It's quoted with Jeff's kind permission:

---

I'll describe with actual code. Here's a trivial example. In VM, the command "QUERY NAMES" (something like the Unix command "who", and can be abbreviated to "Q N", and typically is entered in lower case from the terminal) yields output like:

q n
LOGL0022 -L0022, MAINT4 -L0007, MONWRITE - DSC , EREP - DSC
DISKACNT - DSC , VMBACKUP - DSC , VMTAPE - DSC , VMSCHED - DSC
TCPVSW1A - DSC , ZIP05LNX - DSC , VM - 00C0, ZIP00SOL - DSC
ZIPK - DSC , ZIPL - DSC , ZIPA - DSC , SMTP - DSC
NAMESRV - DSC , FTPSERVE - DSC , UFTD - DSC , REXECD - DSC
TCPIP - DSC , ISPVM - DSC , VSM - DSC , PERFSVM - DSC
VMPRF - DSC , VMSECURE - DSC , OPERATOR - DSC , V102812 -L000B
VSM - TCPIP
Ready;

Given this, I'd like to see how many users are logged on at a terminal, or how many are service virtual machines (the VM equivalent of a daemon), and how many sessions are at a terminal. It's easy to issue a command, see who is at the "not really a terminal" identifier "DSC" (for "disconnected"), and filter out any special userid marked as a "VTAM service machine" (a misnomer actually) that creates a logical terminal, and is marked by the tag 'VSM'. So a pipe could look like:

/* Sample pipe */
address COMMAND

'PIPE (end ?) cp query names',
'| split at ,',
'| a: locate word 3 /DSC/',
'| count lines',
'| spec word 1 1 /disconnected users/ nextw',
'| console',
'? a:',
'| nlocate word 1 /VSM/',
'| count lines',
'| spec word 1 1 /Connected users/ nextw',
'| console'

and generate the following output:

4 Connected users
24 disconnected users

Notice several things:

- the "(end ?)" pieces specifies the character that marks the end of a pipe segment
- the cp stage issues the CP command and retrieves its output, passing it to its standard output
- the split command spits the incoming text at the marked item
- the locate command matches all the lines containing the specified string (here, I require it be in the 3rd word of the line)
- lines containing the desired string go to standard output, where I count the number (like a "wc -l") and then format an output line.
- "spec" is the all-purpose text formatting pipe stage with lots of features for formatting and selecting fields.
- lines not comtaining that "DSC" are sent to a pipe segment with the label "a:", which works in similar manner.
- the generalization is that a stage can have any number of input streams and output streams (primary, secondary, tertiary)
- stages like "count" are "blocking stages", since they delay putting an output record till they consume all their input records.When the primary input on the count stage is closed (end of input), then count emits its record.
- stages like "locate", "nlocate" ("NOT" applied to locate) do not block, but pass data to its primary or secondary output as needed.

So, the locate stage has a primary output stream ("matched") and a secondary stream ("unmatched"), and I can direct that to another part of the pipe for different processing.

This is just a trivial case. Once you get used to the idea that topologies need not be simple left-to-right, there's a lot of things that can be done in a very natural efficient way.

Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 


Video