Ed's an EE, PE, and author in Poughkeepsie, NY. Contact him at [email protected] with "Dr Dobbs" in the subject to avoid spam filters.
Midwinter in the Northeastern U.S. may not be quite so cold and dark as in Finland, where Linux originated, but you do tend to spend a lot of time indoors playing with your toys. Some folks play with computers, while others retreat to their basement and turn metal into works of mechanical art. When solitary confinement palls, January's Cabin Fever Exposition in York, Pennsylvania lures machine-shop wizards from their dens to show off their miniature creations.
While this isn't the place for a machine-tool pep talk, you should admire the award-winning creations from a similar show at http://sherline.com/names03.htm. Essentially, all of the craftsmen use manual machine tools, if only because they enjoy turning the cranks while watching their ideas become metal.
What has this to do with embedded systems? Some craftsmen (basically, it's a guy thing) now use Computer Numerical Control (CNC) machines that convert software into physical objects. CNC pushes the creativity one step further back from the metal: While machinists still watch their ideas become metal, they're no longer directly turning the cranks.
A CNC machine tool is a computer-controlled robot executing a specialized language that describes where the cutter should remove material. The operators generally care very little about how the computer part of the system works, preferring to worry about spindle speeds and feed rates. They're happy to leave the intricacies of binary logic and program design to folks who care about that stuff.
I attended Cabin Fever this year to see a new CNC milling machine by Sherline Products. All of the control functions come from a stock PC and, surprisingly, they don't make a big deal of what's in the PC. Configuration consists of simply plugging in the usual cables, plugging the milling machine into the motor driver box, and turning on the power.
Equally surprisingly, that PC runs a GUI tool controller program atop KDE atop Linux atop RTLinux. However, the machinist interacts with just one full-screen program that performs one function—converting raw metal into a finished part.
Let's see what sort of software gets used in a world that sweeps chips into the trash when the job's done and how satisfying customers and making money with free software means staying well away from the cutting edge. First, let's see why this seemingly simple application requires true, hard real-time performance.
Real-Time For Real
Figure 1 shows the Sherline CNC milling machine posed next to its PC. The relative size of the beige box indicates that this mill handles relatively small parts: Its "work envelope" measures about 9×5×6 inches, although you can cut sections of somewhat larger parts.
You can get a good idea of how a milling machine works if you think of it as a three-dimensional plotter. A slide moves vertically along the column and holds a chuck that spins a cutting tool, which is shaped much like an ordinary twist drill. Below that, the part to be machined moves in a horizontal plane on two perpendicular slides atop the base.
Unlike a 2D pen-and-ink plotter, you cannot simply wave the cutter through the material and make it go away. Well, you can try, but you'll snap off the cutter and destroy the part. You must make a series of relatively shallow cuts that carve out the desired shape. A process called "tool path planning" prevents deep gouges and similar errors, but at the level of the CNC machine itself, there are essentially no restrictions. Like Linux, you're supposed to know what you're doing.
A stepper motor (or stepping motor, depending on where you first met them) on each axis turns a long leadscrew that moves a follower nut attached to the slide, which glides freely along lubricated guide ways. One motor revolution corresponds to one leadscrew revolution and, because the leadscrews have 20 threads per inch, the table moves 0.050 inch for each revolution.
You're probably familiar with AC and DC motors that rotate smoothly, but stepper motors move in discrete steps that are ideally suited for digital control. The Sherline motors have 200 steps per revolution, so each step moves the slide 0.050/200= 0.000250 inch. The handwheels used for manual control have 0.001-inch resolution—four motor steps correspond to one handwheel division. The actual positioning accuracy depends more on the leadscrew and follower nut than the motor, but the CNC controller can get pretty close to the desired spot.
For reasons beyond the scope of this column, the Sherline motor interface box actually runs the motors in quarter-step mode: Each output pulse turns the motor one-fourth of a complete step, for a total of 800 quarter-steps per revolution or 62.5 m-in per quarter-step. As far as the PC is concerned, the motors require 800 steps to move the slides 0.050 inch.
Although in this day and age you'd expect a network or USB connection from the PC to the motors, the interface uses the venerable parallel port. Each motor requires two bits: One sets the direction, the other triggers a single step. The controller can handle up to four axes through one parallel port: x, y, and z corresponding to the three slides, plus an a (angular) axis for an optional rotary table.
The Sherline specs set the full-speed travel at 0.36 in/sec, which corresponds to 5760 steps/sec. That may not sound like much, but it works out to 175 microseconds per pulse and 88 microseconds per interrupt.
You must feed pulses to a stepper motor at a fairly constant rate: If you don't send a pulse when the motor expects one, it won't stop abruptly and wait for you to catch up. Instead, it will continue rotating, at least for a few steps, so the leadscrew is no longer at the position the software expects. You know the saying: 0.001 inch here, 0.001 inch there, and pretty soon you're missing your target.
A rule of thumb says the pulse-to-pulse jitter shouldn't exceed 10 percent of the pulse period, about 18 ms at full throttle. That's hard real time with a vengeance!
The Sherline CNC configuration uses a 16 ms basic time step, so that all pulse durations will be a multiple of that value. Figure 2 shows a multiple exposure of the step signal for one axis, with most of the edges at regular 16-ms intervals. The leading-edge jitter is under 10 ms, although this is certainly not an exhaustive test.
Linux kernels in the 2.2 and 2.4 series, even with low-latency patches, still have unpredictable interrupt handler latencies ranging upward beyond tens of milliseconds. Because the delays depend on the path length through the kernel and device drivers, the maximum latency can be unbounded, even when the average latency looks quite low.
A paper in the July 2003 Proceedings of the IEEE ("The Design and Implementation of Real-Time Schedulers in RED-Linux," Lin and Wang) pointed out several causes of extreme latency. My favorite is the Caps-Lock Key: "In Linux, the kernel code for switching keyboard caps-lock and num-lock waits for an acknowledgment from the keyboard." Firmware in a generic $5.00 keyboard isn't designed for low latency, so the communication path has tremendous round-trip delay.
Ooops.
The best way to convert Linux into a hard real-time operating system involves sliding a hard real-time operating system underneath it and running the time-critical code as an RTOS task. The Linux kernel, plus all its user-space code, then runs as the lowest priority task when the RTOS has nothing better to do.
Pulling that off requires serious software expertise. Who ya gonna call?
Assembling the Stack
The National Institute of Science and Technology developed an open-architecture machine-tool control system in the 1990s. The result of that effort, the Enhanced Machine Controller (EMC) had capabilities well suited for high-volume factories with a suite of high-end CNC machines on the shop floor.
EMC solves an extremely complex embedded-system problem in a very configurable manner. Everything, from the user interface to the number of axes to the spatial resolution of the drivers, is tweakable somewhere in a myriad of scripts and configuration files. NIST eventually turned the source code over to a group of dedicated folks who have further adapted it to a wide variety of machines.
You won't be particularly surprised to learn that EMC has become a Linux-based program with an underlying RTOS that wrings hard real-time performance from a stock PC. The entire Linux-RTOS-EMC installation process has been reduced to what's cheerfully called a "Brain-Dead Install": One CD and a simple Q&A session converts a spare PC into a dedicated EMC station.
Sherline created a set of modifications and patches that transmute a BDI-created EMC machine into a customized Sherline CNC mill controller. The modifications replace the original EMC UNIX-flavored GUI into something that resembles a Windows application, as in Figure 3, so as to avoid terrifying new users.
Sherline sells a complete CNC milling system with a preloaded PC and an hour of Linux/EMC phone assistance. They tell me the phone generally remains silent, except for occasional calls from folks who are completely new to CNC machining. Those users are sometimes distressed to discover that CNC training isn't included in the phone support package.
You'd expect that a new product would use the most recent releases of all the software. Nope: the BDI/EMC package installs Red Hat 6.2 (with the 2.2.18 kernel) and RTLinux 3.0, all of which date back to the turn of the millennium. Indeed, Red Hat recently announced that RH6.2 had entered its end-of-life stage and would receive no further updates. What's up with vintage code?
New Versus Old
The long-awaited December 2003 release of Linux kernel 2.6 brought improved performance to systems ranging from rack-mount monsters to hand-held gizmos. CPU clusters with local and global memory benefit from the kernel's Nonuniform Memory Architecture support, while embedded CPUs lacking memory-management units gain mainline support from the uCLinux code. Most users won't notice either improvement, but should you need those features, you need them badly!
Most users will, however, notice that Kernel 2.6 is fully preemptable because that feature improves desktop responsiveness in the face of heavy I/O and CPU loading. The kernel and I/O drivers also incorporate rewritten code to reduce interrupt latency and overall path length. Linux, while still a soft real-time operating system, has firmed up enough for projects closer to the edge of the hard real-time field.
Unlike new versions of Windows, you can download, install, and use the new kernel on your existing Linux system right now, even before your favorite Linux distribution picks it up. Although that's probably not a good idea for most folks, large-system users can begin validating their code and embedded-systems users can begin their hardware tweaks. Any self-respecting geek would immediately download the latest kernel updates. Right?
Embedded-systems geeks might not leap at the chance, however, unless they're already using the preemption and low-latency patches for the 2.2 or 2.4 kernel, because embedded-systems development marches to a much different drummer than desktop or enterprise projects. Longer product lifetimes, software inextricably burned into hardware, and the difficulty of making updates combines to keep embedded software far from the field's bleeding edge.
The Sherline CNC mill controller runs exactly one program, full-time and full-screen. When you boot the box, you wind up looking at the usual KDE desktop with a conspicuous Sherline icon. Click the icon and the customized EMC program springs to life. Load your part file, perhaps from a floppy (remember floppies?), click the Run button, and the machine starts gnawing your stock.
Assuming that Sherline configured EMC correctly for their hardware, which seems quite likely, everything will operate correctly on each system. The OS and EMC code have been fairly well wrung out, the hardware won't change (unless you simply wear it out), the Sherline software won't require any updates, and, for an isolated Linux box, who cares about viruses or worms?
So Red Hat 6.2, Linux kernel 2.2.18, RTLinux 3.0, and the Sherline modifications will be perfectly fine forever more. As long as it works, don't mess with it!
Financing the Revolution
Sherline briefly considered a Windows-based PC controller for their milling machine, but concluded that the only way to get anything close to real-time performance out of Windows involved external hardware. Adding more hardware complexity to their motor driver box would bump their cost up; plus, they'd need a Windows license for each PC.
Companies with most of their IP in hardware looove GPL code, because they can take advantage of the work of all those dedicated programmers out there who looove solving problems for free. Combining GPL code with real, physical hardware allows them to ship essentially free software with their milling machines. Nothing wrong with that, of course, and Sherline includes all the source for their mods, as well as the BDI CD, in case you must reinstall it.
So, Sherline buys custom-built PCs preloaded with the EMC BDI code and their own modifications. They do a final test on each PC with its hardware and pass the savings on to their customers.
Pop Quiz: If you needed a steady supply of Linux-preloaded PCs, who would you call? Hint: There's probably a store using the same vendor near your town, wherever you are. 'Nuff said.
Sherline also sells all the individual pieces required to build a complete CNC system around your own PC. Because they're not in the software business, you're on your own for support if you decide to roll your own from their catalog of stepper motors, drive boxes, and CNC-ready mills.
Bottom line: For little more than the additional cost of a beige-box PC, Sherline added a precision CNC milling machine to their list of products. From what I heard at Cabin Fever, it's working out wonderfully well for them.
Reentry Checklist
There's more to the story of releasing GPL with your hardware, particularly with respect to the RTLinux patent. I don't understand this well enough to explain it, but I'll keep digging.
Get more info on Cabin Fever Expo at http://www.cabinfeverexpo.com/. Sherline's CNC links appear right on their home page at http://www.sherline.com/, with BDI ISO files and so forth at http://sherline.com/emc/. I bought my CNC mill from http://www.thesherlineshop.com/.
The definitive magazine for home shop machinists is, naturally, Home Shop Machinist, with more information at their eponymous web site http://www.homeshopmachinist.net/.
The definitive list of what's new in the Linux Kernel world is at http://www .kernel.org. I perused the 2.6.1 patch history at http://www.kernel.org/pub/linux/ kernel/v2.6/ChangeLog-2.6.1. Joe Pranevich presents an excellent technical writeup of the changes and improvements in Linux Kernel 2.6 at http://www.kniggit.net/wwol26.html. Review the History of Linux at http://ragib.hypermart.net/linux/.
Current EMC code, various BDI ISO images, and the doc live at http://www .linuxcnc.org/. Some of the original EMC papers are available from the NIST web site at http://www.isd.mel.nist.gov/documents/ publist.htm.
RTLinux comes from FSM Labs at http://www.fsmlabs.com/. I think the licensing situation has changed somewhat since RTLinux 3.0 days.
DDJ
