The first thing I always check when someone's circuit is acting funny is if they have uncommon grounds.
yeap, connect the negative end of both batteries . . . but do not connect the positive ends together of any different type battery (or even the same type batteries with different levels of charge). What happens is one battery keeps trying to charge the other because they discharge at different rates . . .
6V 20Ah battery
sounds dangerous . . . is it lithium ion? or a typo?