Companies that build robots employ electrical engineers, mechanical engineers, computer engineers, computer science graduates, and more. Some schools are now offering degrees in robotic engineering too.
They are all relevant degrees - the question becomes - what interests you the most? What kind of robotics do you want to do? Industrial, autonomous vehicles, household, medical, or???
Pick the degree that best matches your skill set.
BTW - lots of people have a "dislike" for math - but you can't really get away from it if you want to design things for the real world. Sir Issac Newton didn't invent calculus just to make life miserable for engineering students - he (and others) invented it because it was needed to describe his laws of motion and gravity.