tracking a moving object from a moving UAV robot

[gratefulfrog]

Hi SoR team!

I'm sorry to jump in as a complete newbee, but I am.  Here's the first of my dilemmas:

I'm about to start building an autonomous UAV that should be able to fly ahead of a moving person guided by a video stream from an on board camera. The idea is to maintain the person in the center of the frame, while keeping a fixed distance ahead of him.

My questions is: How could I extract the position of my target person from the image stream, given that the UAV is moving as well as the target person?

All suggestions would be great - I'm sorry if this is a silly question...

I am more than available to help answer it.

With great appreciation for all the expertise I see here!

thanks,
Bob

[Admin]

A camera is the only way to do it. Have the person wear a red shirt, and then have your system do blob detection on the color red. If the blob gets smaller, that means the person is walking away. If it gets bigger, the person is moving towards the robot. etc.

Make a ground-based mock-up first, using a simple robot, so you can test out your vision algorithms without worrying about crashing or the ~10 min battery lifetime of your UAV.

[gratefulfrog]

Hi Admin!

Thanks for the tips!

There is one issue, that the target will be dressed as it wants.  Do you think there's a way of identifying him at first, then using blob detection (or optical flow?) to track the "identified" target?

I'm really at a loss as to where to start here. Would you try to run OpenCV  on an arduino-based UAV? or something else?

Thanks for your help!
Ciao,
Bob

[Admin]

What you are asking for is *really* hard, to the point even the experts will be challenged. I've seen a Kinect on a UAV, so in theory it is possible. However, the Kinect doesn't work outside and the range isn't that great . . .

OpenCV will require a decent processor to run . . . Arduino isn't going to cut it

You could try getting a Raspberry Pi to run OpenCV, but expect a lot of pain if this is your first project . . .

[gratefulfrog]

ok, so let's forget the identification step.  Maybe I can give the target a red hat, or something like that?  Or even an infra-red beacon I was thinking...

Then the blob detection, would I just try to code it from scratch? Without the help of all those libraries?

What processor board would you recommend? Propeller? Arduino 32bit? Ardu-copter?

Thanks again.
Bob

[Admin]

What is your previous experience with such things? Are you a skilled programmer?

If I were to do it, I'd use a Blackfin Camera combined with my Axon Mote microcontroller, and a sonar to verify the target distance. Probably use a construction worker reflector vest to stand out.

An infrared beacon is fine, but most cameras have infrared filters built in already. A red hat may be too small, especially compared to a red car in the background.

[gratefulfrog]

Hi,
Yes,  I'm an expert programmer in many many languages, but I have nearly no idea of how the hardware will all work together, nor do I know anything about image processing.  But it doesn't scare me.

I am a keen off-piste skier and was thinking about how I would build a quad-copter that would be able to fly ahead of a skier, and film him from above.  I would not consider any obstacle avoidance, assuming a clear sky...  This could potentially give great shots of my ski trips, compared to the boring photos & films I get today...

I only discovered your site this afternoon, until then I had been mainly surfing on DIYdrones.

Algorithmically, it seems quite straightforward, using any of the autopilot boards based on an arduino-mega? taking a video feed from an intova camera http://www.intova.net/products/sport-hd/ and I would have the skier carry some altimeter, either an android phone or other, to send alt info to the drone, and to call it back when finished.

However, I have not seen anyone do this yet, so I wonder if I am just too naive?

I have been debating on how to start, maybe with an ar.drone at first, for simplicity and reliability... I hadn't considered a ground based prototype - I'm also quite expert in RC heli flying.

But I haven't got a clear plan yet.

Discussing with you has already been helpful!
Thanks,
Bob

[stridera]

i've done something similar using aForge.net.  (They have lots of good tutorials on their site.  Like OpenCV, you won't be able to run it on a arduino.  (aforge would take a laptop most likely.)  However, the articles and concepts should transfer over nicely.

Basically what you'll want to do is take a image of the background, and then watch as things change.  (Check out this article for a simplified version.)  The problem is that if the robot moves, the background will change.  (Also, if the user wears the same shades as the background, you'll lose them.)    If the user is in a white room wearing dark clothes, it's easy since the background remains the same.  Otherwise, you'll want to track new items as they come into the frame and mark them as background.

As mentioned by Admin, it's not trivial unless your conditions are perfect. 

[Admin]

If you are skiing and there are no obstacles (no trees, rocks, or other people), this might be relatively easy.

It would be "easy" because it's a dark colored person on a perfectly clean/white background (ie snow).

What normally would make this problem hard is the person getting confused with the background clutter - I originally assumed you wanted to follow a person walking around in a neighborhood.

An image is just an x/y matrix with with color values.

I recommend reading up on vision processing before continuing, but this is basically what I'd do (google anything you don't understand):
1) read the image into a large matrix (x, y, color)
2) convert the image to 256 bit greyscale (or greenscale, if the human has a green vest on, etc)
3) run a histogram to calculate a threshold value
4) using threshold value, convert image to black/white (not greyscale)
5) run blob detection, filter out anything that isn't about the size/shape of a person
6) determine x/y coordinates in the image matrix of the center of that blob
7) knowing size of the blob, and info from the IMU of the drone, calculate location of that person

Then it gets a bit harder . . . the robot needs to predict ahead of time which direction the human is going so that it may keep up and not fall behind. So then you got complicated math to calculate the trajectory and speed of the person wrt the speed/trajectory of your robot.

this might help:
http://www.societyofrobots.com/programming_computer_vision_tutorial.shtml

Make a cheap ghetto two wheel robot using the camera and processor you plan to use. Program it to chase a ball that roles around your floor. Then when you think you're ready and have things figured out, just move that same software to a quadrotor.

[gratefulfrog]

Hi Admin!

Thanks for those detailed instructions!  I am studying the vision tutorial!

It sounds like the programming shouldn't be so hard, the math doesn't scare me ;-)

On the other hand, identifying the moving skier may not be as easy as you suggest since snow isn't really "white" and mountains are covered with lots of stuff that's not snow, like rocks, trees, and people. In any event, this looks like a good starting point for some prototyping!

I plan to use "Wizard of Oz" systems,; ie human replacing machine at first, then  build robots working in 2D  and finally 3D.   I  have quite an AI background and know the importance of reducing complexity early on to see if the problem is solvable.

This forum has been really helpful and you can count on me for any support and as a customer on the shop!

With my greatest appreciation and thank,
Ciao,
Bob

[Admin]

Awesome

Keep us updated. I'm actually working on a flying robot myself