The shifter board is actually doing nothing with the serial data, except amplifying 0or reducing the voltage where necesarry.
yup. a level shifter also inverts the signal.
The problem may lie more with the usb serial adapter. If this needs a serial signal of +/- 12v(Like the old serial standard) then it wont work as the shifter that you bought only converts this to a signal voltage of 0/5v for the newer serial port standard. So if the usb adapter needs a 12v signal to register high, the shifter that you have is only regulating the 12v down to 5v and not being picked up.
this is rarely a problem.
as you say, it is common to find serial devices that don't transmit at as high a voltage as they used to back in the day.
the reasons for this is that it is rare these days that people use serial comms for long distance communication the way they were first envisaged.
all these serial port devices are still compatible for short ranges though. the RX side will still accept a wide range of voltages.
what does all this mean?
as long as your serial cable is less than 5 meters or so you don't have to worry about any of what i've just been talking about. it should Just Work (tm).
Find out if the usb adapter works with the old or new serial standard (the old is rs-232 needing +/- 12v for high/ low signals and the new is eia-232 which uses 5v/0v for high/low - your rs232 shifter changes the old standard to the new.)
sorry Paul, this is just plain wrong.
there is relatively little difference between rs232 and eia-232 for what Admin is trying to do.http://www.camiresearch.com/Data_Com_Basics/RS232_standard.html#anchor1181130
both rs232 and eia-232 will need to go through a level shifter (and inverter) to reach the TTL UART levels that a microcontroller can deal with.
the level shifter Admin has bought should work fine for rs232 or eia-232.
the fact a USB dongle is being used should have no impact on the situation.
so, to trouble shoot this i would do some loop back tests.
plug in your USB serial thing, open hyperterminal (or whatever comms program you are using) and connect to your serial port.
disable all flow controll. it doesn't matter which baud rate you choose for this test.
now, connect pins 2 and 3 on your serial port.
everything you type in hyperterminal should be sent down the TX line and come back in the RX line and appear on your screen.
if that works, plug in your sparkfun level shifter, power it and join the TX and RX lines on it (the TX and RX wires that will go to your microcontroller).
you should again see the same results, this time transmitting from your serial port, through the level shifter, through the loop, back into the level shifter, back into the serial port and appear on your screen.