Linux on the IBM S/390

Conversation at SHARE 94

Anaheim, California, March 7, 2000 — Jack Woehr sat down with a select group of crossover VM-Linux/390 folks about the genesis of this project. DDJ thanks Marty Zimelis of Sterling Software for assistance in arranging this conversation.

Participants:

Neale Ferguson, Sterling Software

Phil Smith, Sterling Software

Romney White, IBM

David Steinhoff, IBM

Stephen Record, IBM

Rob van der Heij, independent contractor

Rick Troth, BMC

Jack Woehr, DDJ

JW: How did Linux come to run on S/390?

RW: Through a series of fortunate accidents. Some people had an idea that it was possible, and they saw it through.

NF: There were a number of false-start attempts in the community.

Rick sent a number of notes about why you would want to, and it started to fall into place.

RW: Rick was talking about this almost ten years ago.

NF: It needs a critical mass. In one of the independent 390 efforts, there were people with the skillset who were Linux devotees who recognized that there was this thing called S/390, which all these Fortune 500 companies continue to run and that it might be interesting to see what Linux could do on this platform. Through luck and good fortune people had the skills to do the GCC compiler, the glibc, had enough familiarity with how Linux is laid out, its platform-independent parts, its platform-dependent parts, and a crew got on board and started to do the work.

I imagine it was the same case within IBM. People did things on their spare time, got a critical mass of very devoted people who like to share ideas.

SR: The IBM team started out with a need to have a GCC compiler that would produce 390 code. They were porting GCC to VM/CMS Open Edition [see sidebar, "About Open Edition"]. Having got a compiler that would generate that code, it then became obvious to them that there were things they could do other than what they had originally needed it for.

JW: The desire to port GCC to VM, which Neale, outside IBM, was already doing ...

NF: Attempting to do!

JW: ... Attempting to do, and there were some files in the GCC tree already aimed at MVS ...

NF: David Pitts did a damned good job, had GCC working under MVS. I had three or four goes at it, but had insufficient knowledge of the machine-dependent definitions to get much further than the phase-two compilation. Obviously the people at IBM had better skills!

RW: They started earlier and were very single-minded. They had this one key application that needed to be ported. Being able to compile Linux was an unexpected bonus.

NF: Were they Linux guys by nature?

RW: No, they were pretty much hardware engineers.

SR: They worked in hardware simulation, hardware performance. They're fairly young, so they had exposure to Linux, but it had nothing to do with their day job.

JW: What were they trying to build? Did they just want to say that GCC ran on Open Edition?

RW: They were trying to get away from reliance on a proprietary language, partly because you couldn't hire anybody who knew that language.

NF: PL/X?

RW: No, even more arcane than that.

NF: The microcode stuff.

RW: Yeah ... they were trying to solve an IBM business problem.

JW: So they wanted to actually use GCC to compile code for some part of the 390 system?

RW: Right. Then someone said, "You have GCC? Then Linux is a small step." A lot of the heavy lifting is in the compiler itself.

NF: Linux is nicely laid out so you can identify the system-dependent parts. With GCC, it's a major effort.

JW: Neale, you were porting GNU software to VM Open Edition before GCC was running.

NF: I ported SAMBA, Apache, and other killer apps, so the VM community didn't have to buy this sort of stuff. I wanted GCC because I wanted a C++ compiler, but I didn't have the skills to finish. I found David Pitts who had done it, and his web site pointed to Linas Vepstas who was working with Linux. The parts started to fall into place when people started finding each other. Rick found us again, Rob found us, Peter Schulte-Stracke in Germany ...

RT: Linas Vepstas had been doing big Linux for quite some time. He has a bit of a reputation lashing together 486s since before the Pentium. He's also done contracting on S/390, I don't know how much.

NF: He wanted to learn more about S/390. The best way to know an architecture to look at an operating system. So he took an operating system he knew and did some amazing stuff for a guy who wasn't schooled in the S/390 architecture. We came along and helped him with a lot of concepts we knew about from VM days. It was quite an amazing effort of the group to get as far as we did from scratch.

RW: It's to Linas' credit that, judging by what I read of the interactions on the mailing list [[email protected]], that he recognized what VM brought to the table.

RT: I believe he originally was doing this work without VM. I'm a metalhead, but not that much of a metalhead, to do all my testing with the actual hardware console. Having VM in there makes it much easier.

NF: Melinda Varian, from Princeton University, offered him an account where he could do all his kernel testing. That's where we all collaborated. I developed some debugging tools for him so he could do the tracing, see the page translation ... we really accelerated once we got onto Princeton and started using VM.

JW: You didn't know what was happening at IBM in Boeblingen, Germany?

NF: As large as IBM is, you can't help but hear the rumours coming out. We'd hear bits and pieces that indicated that something was happening under the covers. One of the nice things about the 370 stuff we were doing with Linux was hoping we could flush out IBM's hand by advancing. At every opportunity we were putting console logs on the list server to say, "We've booted this beast, we're getting a user process here." We were hoping IBM would say, "We had better mosey along, these people are getting a bit too far."

RvdH: Linas was pushing us to do it. I personally felt very awkward posting to the list seeing that there are 200 IBMers subscribed to the list lurking, knowing that if IBM wanted to do it, they could do it, that they have done it already and were just laughing at me!

NF: It was easy to tell Linas was working on it on his own time. He was corresponding at one o'clock in the morning, his time. It was good for me, because at that time I was in Australia. Rob would come on, again before and after work hours.

RvdH: Although I was embarrassed doing this on the list, Linas convinced me you should do this in the open. Just show the people.

JW: So did anything from IBM's internal project become public before the time everyone noticed it, in December, 1999?

SR: Last fall Karl-Heinz Strassemeyer, an IBM distinguished engineer made a presentation at WAVV, a VSE and VM worldwide user's group, in which he spent about a half hour talking about Linux.

JW: How did the decision take place that IBM was actually going to put this up on the Web?

RW: It wasn't an easy process; it was a lot easier to do the development work!

JW: Was there concern that the well was being poisoned with respect to Open Edition?

RW: Open Edition bring a lot more to the table than a bunch of Unix APIs. It was more a consideration of "how do we do this," because we have never done open source in IBM. We write software to sell to people. It was difficult for people to get their minds around the concept.

There are, to IBM, important questions about intellectual property relating to Linux and the licenses that surround it. We can't just cavalierly throw something out there and hope it will turn out all right. We have to understand these things.

DS: There was the issue of previous patents. For example, in porting DB2 to Linux, we had to consider how much of DB2 we could keep protected. This had to be thought through.

RW: And you'll notice that part of the "how" is that there's a network driver for Linux that is object code only, because it embodies IBM proprietary technology that we are not prepared to give away to the community.

JW: So as umpteen lawyers looked at this, you discovered that the GNU Public License is compatible with proprietary software plugged into it?

RW: I think what we discovered is that the GNU Public License is sufficiently ambiguous to make any attorney worth his salt incredibly nervous. Part of the problem was that it wasn't written by an attorney, so it doesn't use the terms of art that make lawyers comfortable. There are four or five places in the license that it is not terribly precise. If you don't know what rights you are giving up, that ought to make you nervous.

JW: So how does IBM grit its collective teeth and make such a decision?

RW: I think it's safe to say that there are several factors. One is the efforts by the customers. We knew that there was a race on, no question about it. We were also motivated by wanting very much to participate in the open source community, particularly with Linux, and stop having to keep what we were doing IBM confidential. Finally, it was the realization that it wasn't a matter of "if" Linux would run on S/390, it was a matter of when.

JW: And whose version. If IBM releases a distribution, that would become the canonical release to which most of your customers will tend to gravitate.

RT: What IBM has released is patches, patches to the kernel, patches to GCC, patches to GLIBC, GDB ... Based on those patches, which hopefully will get rolled in with the official sources ...

NF: I see from Alan Cox's change history the 390 changes are listed as 2.2.15.

RT: So it could be anybody's distribution. Red Hat may or may not compile it, but the sources that will be on a Red Hat CD will eventually have everything one will need to produce it.

NF: A company in Germany, Millenux, have put out their "Iron Penguin" or "Think Blue" release already, with 420 RPMs [RedHat Package Manager archives] ready to download [see sidebar: "About Linux/390"].

RW: Let's be clear about this: IBM has not made a Linux distribution for S/390 or any other platform.

JW: But you have empowered the community to create such a distribution.

RW: Definitely.

JW: At some point, isn't IBM going to want to take over the stewardship of such a distribution, in its own and in its customers' interests?

RW: That depends on what you mean by "IBM." I can find you an IBMer who will say no, and I can find you an IBMer who will say yes!

JW: Do things like this go all the way up to Lou before they get decided? How does the decision-making process work?

RW: We're discovering that as we go forward. [General laughter.]

DS: We might be just as comfortable having another open source distributor be our distributor, Caldera, Red Hat, whoever.

NF: I'm glad IBM didn't come out with a distribution as their first public offering. It allowed the community to have a look at the diffs, see what they did, build their own kernels, put a file system down, getting a hands-on feel, taking collective ownership. IBM put this thing here, and we've been able to embrace it, whereas if IBM would have come out with a full "Big Blue" distribution first off, that would have frightened a lot of people with "Thou Shalt" instead of "Here Is".

JW: That's the beauty of open source. Perhaps it appears to IBM that people are taking intellectual property, but they'll also do a lot of the work for you when you play as an equal partner in the open source community, doing testing, bug finding, general design without being paid to do it. Didn't IBM have much that same experience in the evolution of VM itself? Your users were perfecting your software for you, not because it was open source, but because they had source?

RW: I have to tell you, in the days when that was happening, I was a customer, and I'd say it was like pulling teeth to get IBM to actually adopt any customer software.

SR: It wasn't easy, but it did happen.

RW: Well, if you were prepared to give up all right, title, and interest in what modifications you had developed, IBM might have considered looking at what you had done. But it was a long, long way from the spirit of open source.

JW: Is open source an improvement for IBM and its customer base? Does the opening up of this gateway mark a new era in the relations of IBM and its customers?

RW: My personal opinion is that IBM has always made an effort to understand what customers want and deliver on that. At the same time, IBM needs to make a profit. Fundamentally, open source is a huge change in IBM's standard way of doing business. We're still digesting the effects of that, trying to understand what it means.

RT: Steps were beginning to happen prior to Linux. How long ago was it — eighteen months ago — when IBM made a contribution to Apache?

Was that to make it run better on Open Edition?

RW: We have a couple, I believe, of employees whose full-time job is to work on Apache. Of course, they contribute their efforts to the open source community.

JW: Why was IBM interested in working on Apache?

RW: Because customers are interested in Apache.

JW: Running on what?

RW: Almost anything. Apache was clearly the standard to aspire to.

JW: What was the change that made customers being interested in something you can't charge for make you devote employees to maintain this open source?

RT: I run Apache on AIX. Some people run Apache on Open Edition, some people run Apache on whatever IBM operating system they want to, and even if it's not an IBM operating system, it may be IBM hardware.

DS: So it's becoming a de facto standard. If you didn't support the de facto standard, you're either in the market or out of the market.

It's a simple decision we want to be in the market.

NF: It's not exactly a simple business decision. There's not an IBM software business, an IBM hardware business, an IBM services business, it's all one, interconnected.

JW: So is IBM being dragged by its little toe ... the next step is Linux, and the next step is you've got to put the 390 hardware diagrams on the Web ...

RW: Not at all. With Apache, we never set out to provide the best-of-breed web server for S/390. We had a web server, but I don't think we were trying to go head-to-head with other people providing that function. What we wanted to do was add value to the web serving environment by surrounding it with application enablers we call "Websphere". The web server is part of that, but it's okay that it's Apache. It made sense to go where the standards are going and be able to interoperate, to allow customers to use software from other companies that was written to the Apache standard.

JW: So you dipped your toes in the water with Apache, then GCC was needed for an internal project, and now you're contributing to the Linux source tree. You've given us the sense that it's still scary, but does it look like there's going to be deeper involvement in open source?

NF: They've already released JFS, Journaling File System, to the community. That's something that's really needed at the high end.

JW: Absolutely, that's really a great service to the community.

NF: The S/390 people have to talk about gigabytes and terabytes of data; can you imagine running 'fsck' on something like that?

RT: Linux on the S/390 is not some weird animal, it's Linux. It's not an EBCDIC system, it's ASCII. An S/390 is a computer, it's got a CPU, memory, disk, ethernet ...

JW: With this feature that you can run multiple systems on the same machine.

RT: Another feature is channelized I/O.

JW: Why is a mainframe different than, say, a big HP or Sun machine?

Why do banks and the IRS and so forth use them?

NF: The realization that business is about moving data, lots of data, quickly, securely and reliably, 99.99% uptime on these boxes.

RT: With Linux on top of VM, VM can handle this wonderful mainframe concept that you can have a processor go down and the system running doesn't feel it, you can bring another on line, and the system is still okay. Try that on a PC!

JW: And great I/O capabilities, in the realm where quantity becomes quality.

RW: The word is "awesome" I/O capabilities. Also, S/390 memory has parity checking. That makes it a little slower, but fundamentally error free. We detect and correct every single-bit error, and most double-bit and triple-bit errors. A VM system stays up for six months, routinely.

PS: You bring it down to add hardware.

RvdH: Though normally you can add hardware while the system is still running.

NF: There's no such thing as blue screen death on a VM system. It's a conscious decision to take it down.

DS: For over 25 years, we've been building this reliability, this availability, figure out what the operator meant instead of what the operator said. This is both hardware and the VM software. You cannot afford to be down. Windows and other systems have never felt the pressure to deliver that kind of availability. That continuous pressure cooker of deliver, deliver, deliver has put us ahead of the rest of the industry in those sort of reliability, availability, security, integrity issues.

JW: Linux itself is considerably less reliable than this! But what you're saying is that you can have a machine where there are 25 images of Linux running, each serving a few users, a few images can go down and other processes will bring them back up.

RT: And that you can have a hardware error, VM recovers the hardware error, and Linux never saw the error. Linux under VM is inherently more reliable than Linux on bare metal.

NF: Introducing the concept of VM to Linux users in Sweden, I called VM a "very intelligent BIOS". It does things intelligently between you and the hardware so Linux can get on with the job of serving Web pages and SAMBA and not worry about recovering CPUs and disk and memory checks and tape ...

JW: So some of the stuff that people might want to write into Linux doesn't have to be written if it's running on top of VM?

SR: VM is like the hardware abstraction layer.

RvdH: You also have the human factor contributing to mainframe reliability. If your application on the mainframe breaks, you don't just reboot the system, several people are sent out to debug the dump, you find out what caused the problem and get it fixed and make sure it doesn't happen again. That's an important aspect of the quality of the system.

PS: The MIS guys get called by PC users who have applications go down. The users get told, "Reboot," and they get angry and say, "MIS isn't doing its job." There's no way to fix some of these problems, because there's a fundamentally different mindset in the PC world.

You can get an ICE and get in there and figure it out, but you're not going to do it just because one day a year a word processor fails.

JW: What makes that kind of diligence economical in the mainframe world?

NF: Thousand of users on an application.

SR: And hundreds of applications running at once.

JW: At the peak of PC hubris, that was precisely the argument. "Let's escape from this kind of liability. Let's give everybody a computer, then if one goes down, just one goes down." What's the mainframe answer to that?

RW: The problem with that approach is that you then need a lot more people to administer. For every ten, twenty, fifty PCs you have, you need someone to look after them.

JW: But many DDJ readers do administer large networks of PCs and don't see this as being impossible.

RW: It's not impossible, but is it fun?

JW: Is administering mainframes more fun than administering PCs?

RT: Yes!

RW: One administrator can administer thousands of users. If you run VM, not only can one administrator administer thousands of users, but they can each have their own computer.

JW: So each user can be running their own Linux image?

RvdH: And the system would not have to be installed for each user.

The administrator provides one copy of, say, each of seven versions and a user can choose to IPL [initial program load] a particular version.

DS: And share one program library, so they don't have to be copied over and over again.

NF: And the disciplines of system management that have been part of the mainframe world for ages are applied, where you've got operators trained in disaster recovery procedures, and you have your backup sets taken offline. You can recover from the worst situations very quickly and securely. You just don't lose data that you can't afford to lose.

RT: And of course each user can configure his or her own virtual machine, override defaults configured by the system administrator. It is very much a sandbox. They can have their very own Linux environment.

JW: But you can still have multiple users logging on to one instance?

RW: You have that flexibility. Either way.

RvdH: Those users logging into that Linux machine will not notice the difference [between logging into VM and launching a Linux instance versus logging into to a shared instance via telnet] until they do a 'cat /proc/cpuinfo'.

JW: How are people going to evaluate how much Linux work they can get out of spare process space on their already-installed S/390? In the mainframe world you have all these scientists doing precise workload measurements, now all of a sudden you have this weird leak in your model, a multiuser system where one guy is running a Linux image for his own use, another with twenty-five users, another putting it out on the Web. Who is going to tell users what they can do with Linux in a practical sense on a given platform?

RT: This wonderful "intelligent BIOS" has all kind of knobs you can turn that have a real effect on how far the sandbox is going to stretch.

SR: And it also presents means to measure realistically.

RW: IBM is very interested in the answer to these questions. Resources are being committed to this.

RT: There are also some tools that need to come forward. There's a tool in VM called TRACK that lets watch what happens in a virtual machine. I'm looking forward to having more panels in TRACK that will allow you to pick apart more accurately a Linux virtual machine.

NF: I'm writing an experimental device driver that allows Linux to generate monitor data that the VM system can pick to do some metrics on internal stuff from the /proc file system. That data can be coalesced from a whole lot of virtual machines and we can accurately account not only for the Linux virtual machine, but also for the applications and processes running therein.

SR: There are two other things that come to mind about performance.

One is the issue of how VM can enhance the performance of the Linux guest. The other is the kind of horizontal scaling VM allows, where it can be appropriate to consider running individual applications in individual Linux virtual machines rather than bundling a number of applications in a single instance of Linux.

JW: There seem to be almost infinite possibilities. So IBM is going to help the customers sort out the optimal configurations?

RW: IBM is going to make some measurements and IBM is going to say something about them.

JW: In a redbook [an IBM-commissioned and co-authored treatise on a particular subject, http://www.redbooks.ibm.com]?

RW: There's going to be at least one redbook, and judging from the high interest of the customer community, more than one redbook.

NF: Rebooks are great, they are really right in the philosophy of open source, getting a group of people together to sit down and dedicate themselves to a task and publish all the results, publishing the code they used.

PS: You know that saying about, "Lead, follow, or get out of the way?" Ten years ago, IBM would have gotten out of the way. Five years ago, they would have followed. Now they're leading.

JW: So far, is IBM satisfied with the reaction of the customer and the open source world to their activities, or is it still controversial inside IBM?

RW: I think everything is controversial inside IBM! There is always more than one opinion. But it's fair to say that there have been several events that have produced interest and encouraged people inside IBM, not the least of which is [David Boyes at Dimension Enterprises managing to achieve] 41,000 virtual machines running Linux on one S/390. That certainly got attention in places where VM isn't necessarily mentioned very often. Also, the interest and enthusiasm shown by the VM community for Linux.

RT: This is something a lot of people have put personal time into.

There's a big passion behind this. People who understand the mainframe and understand Linux see the two coming together and get worked up.

This is cool. This is extremely cool!