Development Tools for Embedded Systems
To develop solid embedded applications you need good tools. Luckily, development tool suites are available from most uC vendors and other third parties. Some of these tools are free, some are free but have built in limits as to the size of code they can compile and some tool suites come with the purchase of a development kit. These development tool suites usually come with an assembler, a C or C++ compiler, a linker/code librarian and most importantly a debugger. For many uCs the most expensive item one needs to purchase is a programmer for the chip itself. You need some way of getting the code you develop onto the uC and in general a device programmer is how that is done.
Most development tools are manufacturer proprietary but some are based on GNU tools within an Eclipse environment. All of the development tools I have used recently have been powerful and robust.. I've yet to run into a situation where the tools have been limiting or buggy.
In terms of cost, things have never been better. The development system I bought for my digital color organ project cost $20.00 from TI (see Figure 1) and this included the device programmer, the development tools and a single target uC board. For my PIC projects I bought a device programmer for approximately $100.00 but the development tools were free.

The Microchip company which makes PIC uCs also provided a free TCP/IP stack that greatly simplifies any networking development efforts you may undertake. My recent projects have been based on the NetBurner development kit (Mod5270LC, $99.00) which includes a target board with network connectivity, power supply, development tool suite and even the cables one needs. The development tools from NetBurner get my personal recommendation as they provide an Eclipse environment for development and the ability to program their devices over the network and do remote debugging.
If the tools available from the manufactures are insufficient, many third-party companies provide extras. Micro Engineering Labs, for example, has a line of BASIC compilers for PIC devices and I personally provide a FORTH programming environment for NetBurner devices. A quick search on the web will turn up many tool options for all of the popular uCs.
What can you Interface to a uC?
The short answer is just about everything and anything. With a modicum of hardware knowledge you can connect GPS devices, USB devices, LCD displays, LEDs, switches, sensors, MP3 music players, digital and serial I/O devices, file system subsystems, networking hardware, memory cards, robotic hardware, a cappuccino machine and just about anything else. If you do some research you can probably find a uC board with the facilities you need for your application eliminating the need to develop any custom hardware at all. In going this route you may miss out on some of the fun, however.
The Arduino uC board was developed specifically for low cost and with simple device control in mind. Arduino boards are being used to control a vast array of things from industrial equipment to kinetic art pieces. See http://blog.makezine.com/archive/arduino/ for some cool examples. As an interesting aside, the Arduino programming language is an implementation of Wiring which is based on the Processing multimedia programming environment. This was done to allow non programmers into the device control arena.