So I think I have a working bootloader (it compiles), but I have a question with respect to the reset pin.
Tell me if I'm understanding this right . . .
I install the bootloader using the normal programmer method.
I toggle the reset pin.
Then I upload a new .hex file using serial (I have X seconds to do this).
Is that right?
Your bootloader works however you program your bootloader to work. You can have your main application jump to the start of the bootloader section whenever you want using a function pointer. I guess typically bootloaders give you X seconds after reset to provide the appropriate input sequence that will keep execution in the bootloader section, but I've written bootloaders that behave differently (e.g. if the program starts with a certain button pressed or other external input in a predetermined state, remain in the bootloader, otherwise jump to the main application). You could just as easily write a bootloader that won't execute the main application until a certain input is received.
Is there any way the bootloader could just detect a new program being uploaded without having to push the reset button?
The bootloader can't detect anything while your main application is running since your main application is running, though you can program your AVR so the UART is constantly monitoring for a special byte sequence. When that sequence is received, call your function pointer that jumps you to start of the bootloader section.
Can my bootloader toggle the reset pin in software whenever I power up my AVR, meaning I don't need a hardware button?
Yes, if you tied one of your I/O lines to your reset line, you could reset your device by pulling that I/O line low. You would need to use EEPROM to avoid the situation where your program restarts, jumps to the bootloader, and resets itself again. One thing you should note is that if your bootloader is the only thing in your AVR's flash, when it jumps to the application section after X seconds, it almost immediately automatically reenters the bootloader (the AVR looks through the flash for the first instruction sequence, which it encounters when it gets to the bootloader section).
