Is it fine to check the voltage across the batteries while connected to the 12V supply?
Yes it is, as long as you don't short it out :-) The multimeter is a very light load, and won't affect the charging.
If the batteries are deep discharged, then the 12V power supply will do one of three things:
1) Deliver its maximum current at a lower voltage. If the maximum current is 5A and the batteries are rated at 1000 mAh, you'd be charging at 5C rate. This may not be good for the batteries -- they could overheat and either go bad, or, worst case, catch fire! Make sure to match the capacity of the supply to the charge rate of the batteries.
2) Go into "hiccup" mode where it tries to deliver charge, notices the overload, disconnects, then repeats the cycle. (Sometimes, it won't repeat the cycle, in which case you can't use that supply.) This will slowly put a charge into the battery, without overheating it, but it's not very efficient.
3) Attempt to over-deliver on current, and burn out itself (rather than the batteries.) This will ruin the wall adapter.
So, hopefully you have wall adapter of type 1, and it's reasonable matched to the batteries. LiPos can often be charged at more than 1C, but how well that works depends on the specific battery. Monitor the heat of the battery!
I'd just get a B6 charger knock-off for $20 and solder in the appropriate JST balancing connector. That way, you know it'll be done right, and have less risk of burning the house down.