When you say the green light is on, how many green lights? When everything is connected correctly, there should actually be two green lights. One signals whether the USB powering aspect is correct for the programmer. The represents everything on the AVR side. So if you only have 1 green light, that means you don't have your AVR programming header wired up correctly to the AVR.
I attached some useful files I've used before to hook it up correctly. (applies only to ATmega8 as far as I know)