Embedded Linux Router
Tom Erjavec
Embedding Linux to create a networking device has long been a desired project of mine. With the advent of many micro-Linux distributions, it has become an easy task that can be done by any Linux lover. My goal was a high-performance secure Internet gateway, so I built a PC/Linux device that could offer full speed on an Ethernet port and provide a firewalled connection to the Internet.
It would be easy to run just one of the Linux Router Project solutions off a floppy disk. But the 1.44 or 1.68 MB floppy space that most LRP solutions provided was not sufficient for the router software configuration I wanted. One possibility was to move the software to a hard disk or to burn the software image onto a CD and run the router off the CD drive. But I decided to go for a solution with no moving mechanical parts. I wanted a flash-disk solution without any drives, connected devices, or even a cooling fan in order to provide an "install-and-forget" and completely silent operation.
The goal of my project is sketched in Figure 1. The routing device must support NAT, DHCP, DNS, and packet filtering with stateful inspection. More than two network segments can be configured, thereby creating one or more demilitarized zones. Inbound NAT enables packet forwarding to virtually provide services on the router while they are really running on a machine in the demilitarized segment. All this and more (using additional LRP packages) is possible with an LRP distribution.
Hardware
The hardware I found suitable for my purpose was an older Cyrix MediaGX 233 set-top box, Allwell STB3036. It's a slim-line, small footprint PC with power consumption low enough to survive without a power-supply fan, and a processor slow enough not to need a cooling fan (Figures 2 and 3). The STB3036 is equipped with M-Systems' DiskOnChip (DoC) device providing 16 MB of solid-state disk space in a flash onboard, but it lacks both a floppy drive and a CD drive. The DoC module is situated in a corner of the motherboard, making it easy to extract and insert the module into the socket (Figure 4).
Software
My software selection was Jacques Nilo's and Eric Wolzak's LEAF/Bering, a derivative of Charles Steinkuehler's Dachstein, itself a derivative of Dave Cinege's original Linux Router Project. The underlying Linux is a dwarfed Debian distribution. I chose Bering because it included Tom Eastep's Shoreline Firewall, a handy configuration utility around iptables. Additionally, Jacques Nilo's archive offers a number of fine software packages that are compatible with Bering. The documentation that comes with Bering is very well written, and it is not difficult to set up a customized environment. The prefixed acronym LEAF (Linux Embedded Appliance Firewall) suggests the purpose itself.
The first obstacle I encountered was booting my hardware platform without a floppy drive from a Bering startup floppy disk. I didn't have an external floppy drive so I used my desktop PC instead. I disconnected the floppy data cable from my desktop PC's motherboard and connected it to the STB3036 motherboard. After powering desktop PC first and STB3036 second, it looked like a blood transfusion but it worked fine (Figure 5). STB3036 woke up in the micro-Linux environment of Bering. As a candidate for embedding, it is clear that Bering does not run off the boot media but rather generates a RAM disk in the system's memory, transfers all the software from the boot media into the RAM disk, and runs from there.
Once my black box with micro-Linux was running, it was still far from operational as a router. Because a micro-Linux kernel supports very few devices straight out of the box, it is necessary to provide all the drivers for the specific hardware being used. The kernel that comes with Bering 1.2 is 2.4.20 and is modular. Jacques Nilo provides a broad range of various modular device drivers in his repository:
http://leaf.sourceforge.net/devel/jnilo/bering/latest/modules/To support my Ethernet interfaces, I used the drivers for RealTek and 3C905c-TX:
mii.o, 8193too.o and 3c59x.oLooking ahead to the end of my project, I downloaded also the modules to support the DiskOnChip flash:
mtdcore.o, docecc.o, doc2000.o, docprobe.o and nftl.oBering also contains support for PPP, PPPoE, HDLC, and ASYNC communications, which were of no interest to me. So, I deleted these modules from the distribution floppy and replaced them with the ones I needed. If you need more modules than can fit on the distribution floppy, an alternative approach would be to provide support for the flash first, then boot off the flash (as described later on) and add the remaining modules manually by copying them from the mounted floppy disk(s) to the RAM-disk. Then, you could re-run the configuration procedure and save the new configuration back to the flash boot image. I activated the modules with the insmode command manually to test whether everything worked. It did. The docprobe module revealed the segments of my flash memory, and ntfl module gave life to the /dev/nftla device. However, the flash was still unusable without partitions and a file system. Configuring Modules So far, all the work has been done manually. To make Bering set the customized environment automatically, the system must be configured. This can easily be done with lrcfg, the Bering configuration menu system. Modules can be added at two points in the micro-Linux file system. The modules that do not need to be active at the beginning of the boot procedure (e.g., Ethernet drivers) are placed in /lib/modules using the "Packages configuration -- modules" menu selection in lrcfg. However, the drivers that are necessary for reading from the boot device (e.g., flash memory drivers in my case) must be loaded prior to the boot procedure. These are placed in /boot/lib/modules using the "Packages configuration -- initrd" menu selection. When the flash image is saved, these will be put into the initial RAM disk for startup purposes. Because Bering runs off the RAM disk, all the configuration changes done in RAM disk must be saved back to the target boot media before restarting the system. This can be done using the backup options in the lrcfg menu (Figure 6). Configuring Services The future services of the routing platform are defined and configured by adding software packages to the bare-bone routing platform. Many of them are already included in the distribution, such as:
- IPtables packet filtering
- Pump (DHCP)
- Shorewall configuration tool for iptables
- Logging utility
- DNS cache
- libz.lrp
- sftp.lrp
- sshd.lrp
- sshkey.lrp
- A tiny FreeSWAN package for IPsec VPN tunneling
- A tiny HTTP server (a question of security)
- A tiny SMTP server (again, a question of security)
- fdisk.lrp
- mkdosfs.lrp
- syslinux.lrp
LRP=root,etc,local,modules,iptables,pump,keyboard,shorwall, ulogd,dnscache,weblet,libz,sshkey,sshd,sftpAfter editing the syslinux.cfg file and saving the configuration changes, it's time to start the embedding. Note that the procedure described so far is sufficient to boot the final configuration from a floppy. This setup will be able to see the flash disk as a device, but the flash will still be empty. Embedding With flash modules in /boot/lib/modules, the system reset will bring up all software needed for accessing the flash in DoC. The first tasks are to create a partition table in the master boot record of the flash disk and to create a file system. The tools for this are installed with:
lrpkg -i fdisk.lrp lrpkg -i mkdosfs.lrpfdisk is used on a flash disk in the same way as on any hard drive:
fdisk /dev/nftlaA partition table is created in the MBR of the flash disk, then a new primary partition is created. Its partition type must be changed from Linux type 83 to type 01 (FAT-12). The active flag must be set for this partition and the table saved to the flash disk /dev/nftla. Now a FAT-12 partition exists on the/dev/nftla1, but it is empty. A FAT-12 filesystem is created in it by running:
mkdosfs -F 12 /dev/nftla1This makes the flash disk ready to use. It can be mounted to the root file system in the mountpoint /doc by:
mkdir /doc mount -t msdos /dev/nftla1 /docThe configuration that was created in the RAM disk and saved to the boot media (curently still the floppy disk) must now be transferred onto the flash. The floppy is mounted and then the files are copied:
mount -t msdos /dev/fd0u1680 /mnt cp /mnt/* /docEffectively this transfers the contents of the floppy disk into the flash. The contents of the flash are almost ready, but it still needs some tweaking. Let's look at a complete syslinux.cfg file:
display syslinux.dpy timeout 0 default linux initrd=initrd.lrp init=/linuxrc rw root=/dev/ram0 \ boot=/dev/fd0u1680:msdos PKGPATH=/dev/fd0u1680 LRP=root,etc,local,modules,iptables,pump,keyboard,shorwall, \ ulogd,dnscache,weblet,libz,sshkey,sshd,sftpThe pointers that refer to the floppy drive must be replaced by pointers to the flash disk:
boot=/dev/nftla1:msdos PKGPATH=/dev/nftla1Now the pointers are fine, but there is no bootstrap code in this partition. The syslinux package is used for that purpose. Syslinux version 1.62 never worked for me, so I used version 1.42, which worked fine. There are newer versions around but I had no need for them. To prepare the bootstrap code in the target partition, it must first be unmounted from the RAM disk:
umount /doc syslinux -s /dev/nftla1The last command created a ldlinux.sys file in the root of the target filesystem. It contains Step 3 of the bootstrap code, executed just after Step 2 of the bootstrap code in the boot record of the active partition in the flash, which runs after Step 1 bootstrap code in the MBR. The big moment has arrived. We do one last check: the partition table is OK; the file system is OK; the bootstrap code should be fine; the modules configuration is OK; the packages are OK. Power down. Power up. Change the boot device order in BIOS from floppy drive to flash. Reset. The screen flickers after the BIOS test and then comes the infinite darkness... Debugging the Bootstrap Procedure When trying to determine what went wrong, I suspected syslinux. I tried three different releases without success. When all reasonable resources had been tried, nothing was left but a HEX editor. I zeroed the Master Boot Record of the flash, restarted the setup from the beginning for the fourth time, and took snapshots of the MBR with dd after each step:
dd if=/dev/nftla of=/mnt/<snapshot-name> count=1After this procedure, I inspected the 512-byte snapshots one by one with a HEX editor. I found out that fdisk didn't produce the Step 1 bootstrap code in the MBR after saving the partition table. With no bootstrap code in MBR at boot time, the execution was never transferred to the active partition's boot record and that was the reason for the "infinite darkness" I experienced (Figure 8). I looked around and found a 16-year-old floppy disk with MS-DOS 6.2. The floppy still worked (I am not sure about MS-DOS). Microsoft did implement a non-documented switch with their FDISK.COM to provide the Step 1 bootstrap code in the MBR by issuing the command:
FDISK /MBRHowever, this command does not address any specific device and, as one might expect, the bootstrap code is written to the MBR of the first physical disk. I could not address my flash in this way so I had to do the procedure manually. I made FDISK /MBR on a spare disk of another machine, but any other healthy disk with an MBR should be fine. Then I ripped the bootstrap code from the MBR off that hard drive by using:
dd if=/dev/hda of=/mnt/mbr.bin bs=512 count=1I used a floppy disk as the transfer media and patched the bootstrap code over the first sector of my flash:
mount -t msdos /dev/fd0 /mnt dd if=/mnt/mbr.bin of=/dev/nftla bs=512 count=1Then I continued the normal setup procedure on the flash for the fifth time, starting with the creation of a new partition table in the patched flash MBR with fdisk. The rest is a replay of the previous section.
For those that don't keep old floppies around or don't want to dd MBRs of healthy (usually non-backuped) hard drives, I must add that, obviously, I did this the hard way. If I had thoroughly read the documentation of syslinux, I'd have known that Peter Anvin provided a generic MBR, including the assembly language source code that contains the Step 1 bootstrap code. However, the Bering step-by-step instructions keep this fact as a secret, and you must know the bootstrap procedure pretty well to locate the problem. I did not experiment with this generic MBR code since I trusted my patchwork was flawless.
This time the reboot didn't fail. In comparison to a floppy disk, the flash provided a lightning-fast bootstrap.
Performance Evaluation
I installed a commercial SOHO router and my embedded Linux router side-by-side to operate in the same environment. My evaluation test was feeding traffic in increments of 1 Mb/s on the internal side through the router to the external side. A probe measured the achieved throughput. The generated packets had the maximum size of 64 KB and were fragmented by the source machine to the largest acceptable framesize on Ethernet -- 1518 bytes. In real life, the packets would be smaller and the routers would be busier with NAT address translation on additional smaller-sized packets. Still, the measurement can give us a feeling for performance comparison.
Figure 9 shows the throughput graph for the commercial router. It was incrementally loaded with eight 1-Mb/s streams. In the beginning, the increments can be seen from the graph, but after the fourth increment (at 4 Mb/s), the linear response of the router is broken, the graph starts oscillating, and the router starts dropping IP packets. The packet loss went as high as 86% with router saturation reached at just over 4 Mb/s.
The situation is much different in Figure 10, which shows the response of the embedded Linux router. Note that the scales of the two graphs are different -- the graph of the commercial router scales to 7 Mb/s, while the scale of the Linux router graph is almost 10 times higher (60 Mb/s). The Linux router was incrementally loaded with 40 simultaneous streams of 1 Mb/s. The graph shows a nice linear response without any sign of saturation. At 42 Mb/s, ten times exceeding the threshold of the commercial router, the packet loss of the Linux router was 0% while the packet delay increased from 1 ms to 75 ms, which is still a quite acceptable response time.
Conclusion
An embedded LRP Linux device can be an appealing security gateway. This solid-state box with no moving parts and Linux hidden in a flash is an "install-and-forget" solution with automatic log rotation that needs very little or no maintenance. Superior performance, compared to commercial devices, and an availability of various configurable networking services make it ideal for an advanced SOHO user.
The Bering distribution, a flavor of LRP, is one of the more advanced LRP branches. However, Bering already has a descendant called Lince, developed by Juan Jesus Prieto. It includes a downsized Squid proxy server and Dansguardian Web content filtering package in the standard distribution. The sacrifice (or rather evolution) is that Lince no longer fits on a floppy but on a CD-ROM.
FreeSWAN IPsec VPN package is another logical choice for an embedded Linux router. But, it should be noted that 3-DES algorithm implemented in FreeSWAN drains the processor power significantly. Loaded with encrypted IPSec traffic, the performance of a system would be expected to drop by a factor of 10.
References
1. LEAF project (Bering, Dachstein, Lince, Oxygen, etc.) -- http://leaf.sourceforge.net
2. Tom Eastep's firewall project Shorewall -- http://www.shorewall.net
3. Repository of the Bering project -- http://sourceforge.net/project/showfiles.php?group_id=13751
4. Compiled modules for Bering, Linux kernel 2.4.20 -- http://leaf.sourceforge.net/devel/jnilo/bering/latest/modules/
5. Peter Anvin's Syslinux -- http://syslinux.zytor.com/
6. Repository of Syslinux releases -- http://www.kernel.org/pub/linux/utils/boot/syslinux/
7. Charles Steinkuehler's home page -- http://lrp.steinkuehler.net/
8. Original LRP homepage -- http://www.linuxrouter.org/
Tom Erjavec was awarded his M.Sc. from computer science at the University of Ljubljana, Slovenia. In the past, he was an assisting teacher at Faculty of computer science in Ljubljana. He then worked with IBM Slovenia as a software developer, with SRC as an electronic meeting systems manager. He co-founded a network integration company MIBO Integra, and is now a marketing director at Iskra Transmission, a microwave telecommunications equipment vendor. Linux remains his hobby.
