Connected to the language question is what are you going to use as a microcontroller? Your uC might limit your choices of languages- especially if you want free compilers. Also consider how quickly do you wantto go forward.
Easiest solution right now might be something like the Parallax Basic Stamp. The programming language, PBasic is simple and it wont take you long to get up and running. However- if you ever want to get more advanced, then you will have to reinvest again in programmers and such. The concepts you learn will help you, but you will have to relearn a new language, relearn how to program a new controller etc. But if you need to be up and running quickly its the way to go
C based language will set you up much better for the future but might be a little bit harder right now. BUt there is a lot of help out there. IF you go with an ATMEL microcontroller with ISP (in system programing) you can download the C compiler for free. See the $50 robot tutorial. There are Basic compilers available also that would work with these same microcontrollers. Not sure I know of any good ones.
If you dont want to get into a lot of electronic circutry- but do want to get started with something that will set you up to become more advanced- I would suggest you look at the Orangutan series of controllers from pololu http://www.pololu.com/catalog/category/8
They all use ATMEL AVR microcontrollers so the programing is the same you would use if you built eveyrthing from scratch, but the board include motor controllers, leds, power circuits etc. Make sure to order a programmer. The Baby Orangutan would be something like the basic stamp.
Whatever you decide- best of luck.