### Author Topic: Reasoning Robot  (Read 6624 times)

0 Members and 1 Guest are viewing this topic.

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Reasoning Robot
« on: March 29, 2009, 02:53:52 PM »
I want to make a robot that takes input from its surroundings via ultrasonic and Sharp IR sensors. I then want it to use this information with the input of a three axis accelerometer, gyro, and temperature sensor (included in the gyro), to figure out how to avoid walls.

I have done a lot of reading on reinforcement and Q learning and I understand the outline of what needs to be done.

I am going to use the accelerometer to give negative and positive feed back. If there is a spike of deceleration then that is negative. I haven't quite decided what is positive yet.

I want to program into the robot as little as possible, ie not telling it that values less than 3000 for the motor will make it go backward. However this creates a problem of a massive amount of choices for it to make, the motor and turning values go from 2000-4000 roughly so around 4,000,000 possible combinations.

As this will require a lot more power then an AVR has I am using a Gumstix.

Does anyone here have experience with reinforcement learning and is willing to help me out?

I attached some pictures of the robot.

I have all the code for the AVR (on the robostix) done and now I just need to get the code going. All the stuff I have read about reinforcement learning is just theory and no actually code or how to implement it, so I need some help.

Thanks,
Justin

• Supreme Robot
• Posts: 11,696
##### Re: Reasoning Robot
« Reply #1 on: April 14, 2009, 12:19:34 AM »
Quote
input of a three axis accelerometer, gyro, and temperature sensor (included in the gyro), to figure out how to avoid walls.
I'm a bit confused . . . accelerometers, gyros, and temperature sensors are useless for avoiding walls . . .

The wall avoiding algorithm is as simply as a photovore.

photovore:
if light found
turn to light()

wall avoider:
if wall found
turn away()

Quote
I want to program into the robot as little as possible, ie not telling it that values less than 3000 for the motor will make it go backward. However this creates a problem of a massive amount of choices for it to make, the motor and turning values go from 2000-4000 roughly so around 4,000,000 possible combinations.
Do you mean you want it to learn how to control its own motors? If so, I recommend reading a ton of AI theory before starting! Oh and learning can be done on a microcontroller. The limitation isn't the processor speed, its the time it takes for a robot to physically complete an action and memory to record what happens.

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #2 on: April 14, 2009, 04:38:44 PM »
I'm a bit confused . . . accelerometers, gyros, and temperature sensors are useless for avoiding walls . . .

What I want it to do is use the accelerometer and gyro (more the accelerometer) to determine when it hit the wall. There should be a rather large spike of negative acceleration when the wall is hit. I then want it to figure out what it did that caused it to hit the wall and avoid that.

I know it is easy to avoid walls, but I want it to learn how to avoid walls. However, I don't want to code it like that. I want to code it as general as possible, so I could teach it things.

Oh and learning can be done on a microcontroller. The limitation isn't the processor speed, its the time it takes for a robot to physically complete an action and memory to record what happens.

The main reason I am using the Gumstix is for more memory. I imagine a lot of RAM is needed to store the knowledge gained, as the longer it runs the more it accumulates.
« Last Edit: April 14, 2009, 04:40:34 PM by Commanderbob »

#### SmAsH

• Supreme Robot
• Posts: 3,959
• SoR's Locale Electronics Nut.
##### Re: Reasoning Robot
« Reply #3 on: April 14, 2009, 04:48:19 PM »
However, I don't want to code it like that. I want to code it as general as possible, so I could teach it things.
then what language do you want to code it in?
Howdy

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #4 on: April 14, 2009, 04:55:04 PM »
However, I don't want to code it like that. I want to code it as general as possible, so I could teach it things.
then what language do you want to code it in?

I was going to code it in C, but that is not what I meant. I meant that I don't want the algorithm to know it is learning to avoid walls, ie tell it before hand that something coming closer is bad. I just want to have it know that spikes of negative acceleration is bad and to avoid that.

#### SmAsH

• Supreme Robot
• Posts: 3,959
• SoR's Locale Electronics Nut.
##### Re: Reasoning Robot
« Reply #5 on: April 14, 2009, 04:58:40 PM »
spikes of negative acceleration on the accelerometer? if so it will only know that there is a neg spike once its hit a wall so it will keep smashing into walls?
Howdy

#### Razor Concepts

• Supreme Robot
• Posts: 1,855
##### Re: Reasoning Robot
« Reply #6 on: April 14, 2009, 05:00:31 PM »
No it will learn that whatever it did before smashing into the first wall is bad and will not do that again.

#### SmAsH

• Supreme Robot
• Posts: 3,959
• SoR's Locale Electronics Nut.
##### Re: Reasoning Robot
« Reply #7 on: April 14, 2009, 05:05:46 PM »
i can understand if it uses a wall sensor for that but if it uses a gyro/accel to read this it will only be able to see the negative spikes once its too late.
Howdy

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #8 on: April 14, 2009, 05:06:35 PM »
No it will learn that whatever it did before smashing into the first wall is bad and will not do that again.

There in lies the problem. I am not sure how to make it realise what led up to the event of hitting the wall. Was it something it just did, or something that happened a few minutes ago?

I was thinking of storing the scans form the past few minutes, finding trends in them, saving those trends as bad, then if it hits a wall again compare the trends to see if some are similar.

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #9 on: April 14, 2009, 05:08:07 PM »
i can understand if it uses a wall sensor for that but if it uses a gyro/accel to read this it will only be able to see the negative spikes once its too late.

As you can see in the pictures it also has an IR sensor and ultra sonic sensor. I want it to learn what the inputs from those sensors mean, by using the accelerometer as a base of knowledge.

#### SmAsH

• Supreme Robot
• Posts: 3,959
• SoR's Locale Electronics Nut.
##### Re: Reasoning Robot
« Reply #10 on: April 14, 2009, 05:13:06 PM »
No it will learn that whatever it did before smashing into the first wall is bad and will not do that again.

There in lies the problem. I am not sure how to make it realise what led up to the event of hitting the wall. Was it something it just did, or something that happened a few minutes ago?

I was thinking of storing the scans form the past few minutes, finding trends in them, saving those trends as bad, then if it hits a wall again compare the trends to see if some are similar.
but then there lies a problem as most of the time events before hitting a wall are the same as regular driving.
Howdy

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #11 on: April 14, 2009, 05:18:24 PM »
That is the whole point of the program, to pick out which events are different, which ones lead to crashes.

#### SmAsH

• Supreme Robot
• Posts: 3,959
• SoR's Locale Electronics Nut.
##### Re: Reasoning Robot
« Reply #12 on: April 14, 2009, 08:32:21 PM »
ohhh, so will it record the events and if there is a crash store it and if those events start happening again know it may crash again?
Howdy

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #13 on: April 14, 2009, 08:51:45 PM »
That's the basics of it. The problem is recognising what in the recorded events led to the crash. If anyone has any suggestions on how to do this I might actually make some working code  .

#### chelmi

• Supreme Robot
• Posts: 496
##### Re: Reasoning Robot
« Reply #14 on: April 14, 2009, 09:13:11 PM »
That's the basics of it. The problem is recognising what in the recorded events led to the crash. If anyone has any suggestions on how to do this I might actually make some working code  .

Take a look at artificial neural networks. They are very good at recognizing patterns in complex data sets. But I don't know if you will be able to use this kind of tools for real time data processing on a MCU...

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #15 on: April 14, 2009, 10:07:22 PM »
Well, I'm using the Gumsitx, 600MHz.

I've read some about neural networks, but I don't quite seem to understand how they help. I am going to read some more tonight. If you have any info that would help me code one that would be great.

#### GearMotion

• Supreme Robot
• Posts: 489
• Two decades+ of Embedded Design
##### Re: Reasoning Robot
« Reply #16 on: April 15, 2009, 07:38:32 AM »
Commanderbob: Perhaps you would want to simulate algorithms on a PC before using them on your robot. You can simulate and repeatedly set up scenarios for your algorithm testing.

(Something like robotbasic.org might help)

#### hoosier122

• Full Member
• Posts: 84
##### Re: Reasoning Robot
« Reply #17 on: May 17, 2009, 06:53:27 AM »
Ok, so basically your refusal in wanting to use any sort of visual sensors (and only using touch) means you want to build a blind robot?.?.?

#### Canabots

• Contest Winner
• Robot Overlord
• Posts: 245
• It's not a bug, it's a feature!
##### Re: Reasoning Robot
« Reply #18 on: May 17, 2009, 07:55:16 AM »
I want to make a robot that takes input from its surroundings via ultrasonic and Sharp IR sensors. I then want it to use this information with the input of a three axis accelerometer, gyro, and temperature sensor (included in the gyro), to figure out how to avoid walls.

I'm a bit confused . . . accelerometers, gyros, and temperature sensors are useless for avoiding walls . . .

What I want it to do is use the accelerometer and gyro (more the accelerometer) to determine when it hit the wall. There should be a rather large spike of negative acceleration when the wall is hit. I then want it to figure out what it did that caused it to hit the wall and avoid that.

I know it is easy to avoid walls, but I want it to learn how to avoid walls. However, I don't want to code it like that. I want to code it as general as possible, so I could teach it things.

There are visual sensors used (the ultrasonic and IR rangefinders). It's just that this robot is essentially learning how to avoid obstacles. If I'm not mistaken, it will relate what occurs when it collides with something (using a gyros and accelerometers) to what occurs visually (using the Ultrasonic and IR sensors) and to not let it happen again.

Imagine it like we are just learning the different textures (smoothness/roughness/softness, ect.) an object posseses. When we first  touch the object, we will relate visually to what we sense on our fingertips, so if it's a hard, flat surface that we view, and it feels smooth.

The next time we see a surface similar to that, we don't need to reach out and touch it before identifying the texture of the object. We can just say its texture is smooth because of the similarities it possesed with the first object.

BTW, Cool project CommanderBob!
« Last Edit: May 17, 2009, 08:04:04 AM by Canabots »
My robotics, electronics, software, or other stuff blog:
www.saltech.wordpress.com

#### guncha

• Jr. Member
• Posts: 40
##### Re: Reasoning Robot
« Reply #19 on: May 17, 2009, 01:57:20 PM »
Comanderbob, I'm thinking that you'll first have to clearly define the "problem" your robot is going solve and use it's performance as a positive feedback and hitting a wall as a negative one. Otherwise, if you program your robot just to avoid walls, it will quickly solve this problem and devise world's best wall avoiding algorithm - staying on the spot.

#### arixrobotics

• Full Member
• Posts: 119
##### Re: Reasoning Robot
« Reply #20 on: May 20, 2009, 07:31:20 AM »
Comanderbob, I'm thinking that you'll first have to clearly define the "problem" your robot is going solve and use it's performance as a positive feedback and hitting a wall as a negative one. Otherwise, if you program your robot just to avoid walls, it will quickly solve this problem and devise world's best wall avoiding algorithm - staying on the spot.

In other words, the robot has to have a goal / target. For example, it could be to go to a certain spot, while avoiding obstacles on the way.

At the very least, the goal should be to always move in a straight line, I think. That way the robot should never stop and do nothing.

#### SmAsH

• Supreme Robot
• Posts: 3,959
• SoR's Locale Electronics Nut.
##### Re: Reasoning Robot
« Reply #21 on: May 20, 2009, 03:57:47 PM »
can it stop to turn?
Howdy

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #22 on: May 20, 2009, 04:52:26 PM »
Sorry for not replaying in so long. For some reason all my email were going into the spam folder, so I didn't know anyone posted.

I have not done any more actual work as school has kept me busy. However with only 15 days left I will have more time. What I did do was find http://leenissen.dk/fann/ it is basically what I want.
At the very least, the goal should be to always move in a straight line, I think. That way the robot should never stop and do nothing.

Ha ha, that is what I was thinking, if it's goal was only to avoid walls then the best was to do that would be to never move and it should eventually just stop.

Commanderbob: Perhaps you would want to simulate algorithms on a PC before using them on your robot. You can simulate and repeatedly set up scenarios for your algorithm testing.

(Something like robotbasic.org might help)

Thanks for the suggestion but I find it a lot more work to do that than to just dive into the real thing. I already have the robot built, I just need the code and to simulate the real input it will receive will be harder than just letting it run.

Canabots, that is exactly what I am trying to do.

guncha, that again is what I was thinking but the only real problem I can think of is avoiding walls. I have a GPS that I could hook up later and try to make it learn how to get to places sent wirelessly to it.

I am getting a new computer in a few days so I'll wait for that before I start the coding, as I will need to rebuild the gumstix stuff.

Justin Rajewski

#### Webbot

• Expert Roboticist
• Supreme Robot
• Posts: 2,165
##### Re: Reasoning Robot
« Reply #23 on: May 20, 2009, 10:06:14 PM »
I know it is easy to avoid walls, but I want it to learn how to avoid walls. However, I don't want to code it like that. I want to code it as general as possible, so I could teach it things.

Here's some food for thought.. http://www.societyofrobots.com/robotforum/index.php?topic=7.msg30869#msg30869
Webbot Home: http://webbot.org.uk/
WebbotLib online docs: http://webbot.org.uk/WebbotLibDocs
If your in the neighbourhood: http://www.hovinghamspa.co.uk

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #24 on: May 22, 2009, 08:33:07 PM »
Some other interesting places to visit.
FANN documentation/tutorial
http://softlayer.dl.sourceforge.net/sourceforge/fann/fann_doc_complete_1.0.pdf

The Next Generation of Neural Networks

I definitely recommend checking out FANN, I haven't tried it yet but it looks promising.

#### Commanderbob

• Robot Overlord
• Posts: 146
• Embeddedmicro.com
##### Re: Reasoning Robot
« Reply #25 on: May 31, 2009, 03:44:46 PM »
I was messing around with the FANN library on my computer and it is really cool. I made a training file with 10 inputs and 2 outputs (speed and turning). The training file contained stuff like 9 8 5 3 2 0 1 0 0 0 (larger number = closer) the output would be something like .94 .75 meaning turn 94% right and forward at 75% max speed. Once it was trained off the sample data I gave it, it really seemed to work well.

Now I need to figure out a way to train it on the fly, without a training file.

My new computer does not have a serial port   so I have to go buy a USB->Serial adapter before I can work with the Gumstix.

Anyone know where I can get a good cheap one? Radio Shack sells them for \$30 which seems really over priced to me.
« Last Edit: May 31, 2009, 03:46:32 PM by Commanderbob »

#### Razor Concepts

• Supreme Robot
• Posts: 1,855
##### Re: Reasoning Robot
« Reply #26 on: May 31, 2009, 04:01:00 PM »
I bought a usb to serial adapter on ebay for \$2 shipped

#### SmAsH

• Supreme Robot
• Posts: 3,959
• SoR's Locale Electronics Nut.
##### Re: Reasoning Robot
« Reply #27 on: May 31, 2009, 04:02:46 PM »
does that online store that has free shipping have them?
y'know the one that's really cheap...
Howdy

#### lemontree

• Jr. Member
• Posts: 38
##### Re: Reasoning Robot
« Reply #28 on: June 13, 2009, 06:44:58 PM »
I would be inclined to use a combination of a hash table or bloom filter with locality sensitive hashing.
I would get the sensor readings every .1 sec for example.  If I got some bad news from my accelerometer I would go back through the last 10 sensor reading (ie the last 1 sec) . Do the locality sensitive hash of all the sensor reading for each of the past 10 sensor reading in turn. Eg int x=LShash(speed X, speedY, light level, ......).  Then I would put an entry into a hash table for each of the 10 values of x indicating something bad had happened.
Then it is a matter of constantly monitoring your input sensor readings, doing the locality sensitive hash to get x, then putting x into the hash table to see if anything bad happened before  with approximately those sensor readings.
There is an overly technical Wiki page on LS hashing:
http://en.wikipedia.org/wiki/Locality_sensitive_hashing

#### Samuel

• Jr. Member
• Posts: 18
##### Re: Reasoning Robot
« Reply #29 on: June 15, 2009, 01:21:02 AM »
Chapter 16 in the book Robot Programmer's Bonanza does EXACTLY what you are trying to do
with a program and all.....

http://www.amazon.com/Robot-Programmers-Bonanza-John-Blankenship/dp/0071547975/ref=pd_bxgy_b_img_b

it does it on a simulated robot instead of a real one but the ALGORITHMS are what you need and
the program shows all the algorithms you want. The chapter also explains the theory and
how to improve on the given program.

You can see the program in action in this video
Simulator for Robotics