Ed's an EE, PE, and author in Poughkeepsie, NY. Contact him at [email protected] with "Dr Dobbs" in the subject to avoid spam filters.
Back in the mid 1800s, Thomas Carlyle described economics as "the dismal science" because, according to folks like John Stewart Mill, everybody was, or at least should be, pretty much equal in the eyes of employers. Carlyle held that supply and demand should not determine wages and prices and that, if you weren't willing to work for the proffered wages, then you should be compelled to do so. That is, if you weren't European, of course.
An information undercurrent at the Embedded Systems and Game Developers Conferences had to do with economics—wages, prices, working conditions, offshoring, security, and so forth. I saw many different aspects of economics popping up here and there, with each speaker addressing a different aspect of the same beast.
An attempt to look for the lump from which all those loose ends originate: There's more here than meets the casual glance.
Hardware: Fled
CEOs from Micron, Altera, and Cypress Semiconductor talked about "After the Storm: How the Industry Changed Forever." Although they covered many topics relating to the semiconductor crash in the early naughts of this century, a central concern seems to be the economics of building factories that crank out ever-more-complex, ever-smaller chips.
Some background is in order. In very round numbers, a 300-mm fab produces 150,000 wafers a year. Each wafer has an area of 700 cm2 and high-end desktop microprocessors are under 1 cm2, which means each wafer holds on the order of 500 processors and the fab cranks out 75 million chips per year. The harsh reality of semiconductor manufacturing is that not all the chips work, so that's an overestimate.
One of my managers with a deep background in semiconductor production once observed that if the yield for a new fab exceeded 10 percent, you weren't pushing the technology nearly hard enough. Obviously, you want much higher yields in a mature line, but manufacturers regard their actual yield numbers as Top-Secret-To-Die-For information.
I did find some hints that give an order of magnitude, though. Defect densities on raw 300-mm wafers may be around 0.1-0.2/cm2, so call it 100 per wafer. If you assume manufacturing defects kill another 100 chips, you're looking at 50 percent yield and a fab can produce 40 million CPUs per year. Forecasting yield is much more complex than that, of course, but I suspect it's neither twice as good nor half as bad.
Let's say desktop CPUs wholesale for $100 each, so gross revenue is around $4×109/year. Nice work if you can get it, eh? Fairly obviously, yield forecasts have a direct effect on revenue, which is why careers are made and broken on their accuracy.
A new 300-mm wafer fab costs $2×109 regardless of its location. Most of that money goes for the equipment inside, a slice goes to real estate and construction, and another chunk may, depending on the location, go to taxes and baksheesh.
Although tales of offshoring generally revolve around labor costs, the reason 50 percent of all fabs are now in Taiwan and China has little to do with payroll. At a fully burdened rate of $50/hour, 1000 direct employee equivalents add up to $100×106/year, or only 2 percent of gross revenue. That's probably an overestimate, too.
The single largest cost for a fab turns out to be equipment depreciation. Applying five-year, straight-line depreciation to $1.5×109 of equipment costs $300×106/year. After that, the fully depreciated fab can be relegated to cranking out embedded-system CPUs.
Ever wonder why embedded CPUs are so cheap? It's because the fab line is basically free. The CPUs are smaller, so relaxed design rules can increase the yield. More chips per wafer amortize the fixed costs of production over a bigger base. Variable costs are nearly zero per chip, too. Shazam! New life for old equipment!
With most of the money in hardware, fab startup costs are largely fixed, so taxes and legislation become the swing issue. California evidently levies an investment tax on new tech construction, but, in the past, has refunded that tax in order to attract new business: You paid The Man up front and got a promise that your coin would return. When one of the CEOs was trying to establish a new fab, there was talk of not rebating already collected money in order to help balance California's budget. That means the new fab's cost might take a $200×106 hit, but he couldn't tell.
Plus, California hands out a stack of environmental impact statement forms, with the promise of an uncertain result after two years of review. Everyone wants juicy high-priced tech development, but not in their backyard when it involves arsenic and organic solvents.
So, if I have the story correct, that fab wound up in Texas, where the building process went smoothly. The CEO now recites his ABCs when the topic of new construction comes up: "Anywhere But California."
Why not in Taiwan? That company wanted to build in the U.S. and was willing to pay a relatively small premium. In general, however, fabs wind up where capital and taxes dictate, not where politicians (claim to) want them.
Software: Fought
Hank Howie's Game Developers presentation tells of a seemingly different problem: "Better Games (and Quality of Life) in 40 Hours Per Week." Although embedded systems engineers and programmers tend to put in long hours during the "crunch time" near the end of the project, it seems that game-design scheduling assumes massive overtime right from the start. A developer survey showed that half of them had managers who regarded crunches as normal and half (perhaps the same ones?) expected to be working in another field within 10 years.
Howie attributes this problem to some factors that may sound familiar. Game developers tend to be young, unmarried enthusiasts with a surplus of dedication and a deficit of real-world experience. They're working on pure software projects with the usual unrealistic schedules and estimates, pressurized with tens of megabucks of venture capital per title.
Worse, game programming has only recently catapulted from the small-shop model into the million-LOC arena that requires far more rigid management techniques than the programmers are willing to accept. Indeed, it sounds quite familiar, doesn't it?
Howie emphasized the fact that most people cannot actually produce eight solid hours of work a day, let alone 10 or 12, for endless seven-day weeks. At some point, external life intrudes, even for young, single, over-dedicated worker bees. When that happens, bad stress can do bad things to good people, as well as make good people do bad things.
Howie has structured his company, Blue Fang Games (http://www.bluefang.com/), around a 40-hour work week with predictable, limited-duration crunches at known points in the schedule. He finds that his workers produce better-quality code in roughly the same calendar time, with less stress and chaos. If that sounds reasonable to you, it seems rather innovative in game-biz terms.
The fundamental problem with software development is that productivity, measured on any scale you like, differs by at least an order of magnitude between developers and the ranks are pretty thin at the high end. I've known three superstar programmers in as many decades; none were paid anywhere near what they were actually worth.
Here's where management techniques come into play. You can safely regard all software development methodologies as ways to raise a group's average productivity to get better results from typical employees. After some experience, you also get more predictable results from the group and can set schedules with better confidence. At least, that's the plan.
Given a process that transforms specs into code, which is what all methodologies claim to be, you can then get predictable results from any group of programmers. But then the same economic forces driving fabs to Taiwan and China suddenly apply to programming shops: Your business simply cannot afford to pay more for the same commodity. In this case, labor is the only expense, so the pressure is even greater.
Game developers also worry about having their jobs vanish offshore, as witnessed by several presentations on the subject. The game industry has become closely allied with (or, depending on who you ask, is being Borged by) the entertainment industry, which makes unionization not only talked about, but possible. It seems that organizations such as the Screen Writers Guild have begun eyeing the legions of oppressed developers.
As nearly as I can tell, though, unionization simply accelerates the shift, as demonstrated by any number of once-flourishing industries that chose between well-paid jobs and plentiful jobs. A business can't afford to have lots of well-paid workers anymore, unless they're adding tremendous value to the product. If they are, fine, but another group can probably do the same thing at a lower cost.
Firmware: Freed
Paul Knot, from New Zealand's 4RF Communications (http://4rf.com/), described "Selecting Linux as an Embedded Operating System" for its line of high-bandwidth digital microwave data link radios. These are not cute table radios with faux wood trim and knobs. They're rack-mounted boxes funneling torrents of digital data through line-of-sight RF links. In addition to all the usual computer issues, the radios must contend with the vagaries of RF communication.
Their previous radios used traditional homebrew operating systems, but because the new radios would handle much more varied network traffic, a network-aware OS was in order. The choices boiled down to a commercial embedded RTOS or Linux.
They chose Linux based on both technical and economic factors. Technically, they were familiar with the inner workings of an OS, so the prospect of adapting the Linux kernel wasn't frightening. Economically, the prospect of paying either a massive up-front charge or a per-unit royalty wasn't appealing, particularly because they'd already been doing their own support for years.
It's worth noting that the radio runs just the Linux kernel and a few utilities, not the entire collection of GNU and open-source software that accompanies the usual desktop Linux distribution. Although they could have used a commercial embedded Linux distribution, they elected to build a completely customized system for their unique application.
Previous 4RF radios used PowerPC CPUs, so the design team knew its peculiarities. The Linux kernel runs well on PowerPCs, eliminating the daunting challenge of porting it to a completely new architecture. They had experience interfacing radio hardware with their own software, reducing the number of integration mistakes.
Best of all, microwave data link radios have somewhat different design parameters than mass-market consumer devices. In exchange for full-time operational reliability with minimal attention in remote locations, they're not as relentlessly cost-sensitive as, say, a pocket-sized MP3 player with an FM receiver. That means 4RF's engineers could include hardware features that simplify development, improve reliability, or enhance compatibility, at least if they didn't get too carried away.
Their minimum configuration can run in 4 MB of RAM and 2 MB of flash ROM, which is aggressive for an embedded Linux kernel, but 16+8 makes life much easier. The development boards sported 32+8 to allow all manner of debugging and tracing aids. Production boards can handle RAM chips up through 16 MB because they expect RAM prices to drop during the radio's life and, at some point, bigger chips will cost less than smaller ones.
The stock Linux kernel can provide, at best, soft real-time performance. The engineers implemented all the hard real-time functions in hardware buffers and custom digital modulator-demodulator chips, leaving only very soft real-time requirements for the kernel. This is an excellent practice for products that can absorb the additional cost, because it simplifies software development and improves the overall reliability of the design: It's easier to verify hardware than software.
He emphasized that, even though the Linux kernel has a reputation for reliable operation, you must test the entire system under your peculiar operating conditions. For example, memory allocation poses a challenge for always-on, never-restarted systems: You may need a custom memory allocator to prevent bizarre runtime failures due to heap fragmentation.
Michael Anderson from the PTR Group provided some manpower estimates during his talk on "Migrating from a Legacy RTOS to Embedded Linux." In round numbers, figure on two man-years of effort to create your own "embedded Linux distro" starting at http://www.linuxfromscratch.org/, then adding your own tools and code.
You can short-circuit some of that effort with a commercial embedded Linux distribution or even a proprietary OS, but don't expect the knowledge required to use the system will emerge from the shrink wrap! The key advantages of do-it-yourself adaptation are that you both understand the code and can use it without royalties.
Anderson observes that per-unit royalties are the driving force in high-volume designs, making embedded Linux a clear win over any commercial alternative.
Economics: Freestyled
The history of industrial processes, most recently that of fabs, shows that factories will gravitate to low-cost locations with stable economies and available labor. Some factories may remain in high-cost locations, but only for niche products that can support a high production cost.
Software development has reached the knee of the "industrialization" curve, where teams can barely predict code productivity and quality based on specifications. As soon as that improves, programming will follow the fabs. Because capital expense is minimal for programming operations, labor costs will trump nearly everything; India and China become just stopping points.
The commercial embedded operating-system industry may lag a decade behind ordinary application software, but open-source techniques have redefined how firmware gets built. Basically, any hardware that can run Linux will run Linux, so that there's no money to be made in proprietary infrastructure code. You'll see lots of proprietary firmware built atop open-source code, although the same forces driving software applications will begin applying to firmware, as well.
One outcome may not be so dismal, though. Tech money can drive social progress, too, as we're effectively funneling money directly to people performing first-world-style jobs in third-world-style countries. That's in contrast to governmental financing that seems funneled directly to despots and thugs.
Perhaps if we got enough people running fabs and writing code in the poorest HASH(0x1876360) of the world, we might reduce the number of countries in need of adult supervision.
Yeah, I can dream, can't I?
Reentry Checklist
Discover that everything you think you know about Carlyle's famous "dismal science" quote is completely wrong at http://www.economics.unimelb.edu.au/ TLdevelopment/econochat/Dixonecon00.html and http://www.victorianweb.org/authors/carlyle/kennedy1.html.
You will certainly find many different silicon processing numbers than the ones I used for wafer fab production and employment. Let me know how far off I am!
Howie's "Better Games in 40 Hours" presentation is available at http://www.bluefang.com/images/gdc2005_Hank_ Howie%20Final.pdf. He highly recommends, as do I, Steven McConnell's excellent books. Start with Code Complete and go on from there. Do it now.
All the GPL and LGPL code modified by 4RF is available directly from them for a production and shipping charge that makes perfect sense given that you're probably not in New Zealand. Remember that "free software" is free-as-in-speech.
DDJ
