I want to design [...]
Can someone please suggest some good resource with circuit diagrams and theory??
If you want to design a system, you
are the one designing and drawing the schematics and developing the theories for this system.
If you just want to copy someones else's design, don't claim you want to design it.
Copying andf designing are two very different disciplines.
a simple system based on Line of Sight communication technique using LED in the visible spectrum, the range should be about 2-3 m.
There's more power in IR-LEDs.
I am thinking on the lines of using RJ 45 to serial
"RJ45" is a connection standard (RJ=Registered Jack and strictly speaking, RJ45 is an 8P2C connector [8 positions 2 conductors]), "serial" is just a description of how multiple bits are transmitted one by one, so what do you mean?
Are you perhaps thinking Ethernet (which is
serial) to EIA/TIA-232 ("RS-232")?
and then Frequency modulation (VCO). On the receiver end demodulation (PLL) and again interface to RJ 45.
What bit rate do you need (absolutely)?
FM might not be the best solution.
Did you consider IRDA or something like a TV-remote NRZ-scheme?