How to pick up an object using mech arm and camera

[iamaldrin]

How can I do this using C#, a microcontroller and a camera? First, it looks for an object, when it is found, it will pick it up. It should also identify the shapes. I've seen shape identification, but it is only for pictures, i've tried changing the input from picture to camera but it doesnt work. Here's my code:

Code: [Select]

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using Emgu.CV;
using Emgu.CV.Structure;
using Emgu.Util;
using Microsoft.Win32;


namespace Camera
{
    public partial class Form1 : Form
    {
        // The path to the key where Windows looks for startup applications
        RegistryKey rkApp = Registry.CurrentUser.OpenSubKey("SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run", true);

        private Capture _capture;
        private bool _captureInProgress;

        public Form1()
        {
            InitializeComponent();

            // Check to see the current state (running at startup or not)
            if (rkApp.GetValue("MyApp") == null)
            {
                // The value doesn't exist, the application is not set to run at startup
                checkBox1.Checked = false;
            }
            else
            {
                // The value exists, the application is set to run at startup
                checkBox1.Checked = true;
            }
        }

        private void ProcessFrame(object sender, EventArgs arg)
        {
            Image<Bgr, Byte> frame = _capture.QueryFrame();
            imageBox1.Image = frame;

        }

        private void button1_Click(object sender, EventArgs e)
         {
         #region if capture is not created, create it now
         if (_capture == null)
         {
            try
            {
               _capture = new Capture();
            }
            catch (NullReferenceException excpt)
            {
               MessageBox.Show(excpt.Message);
            }
         }
         #endregion

         if (_capture != null)
         {
            if (_captureInProgress)
            {  //stop the capture
               button1.Text = "Start Capture";
               Application.Idle -= ProcessFrame;
            }
            else
            {
               //start the capture
               button1.Text = "Stop";
               Application.Idle += ProcessFrame;
            }
           
            _captureInProgress = !_captureInProgress;
         }
      }

        private void button2_Click(object sender, EventArgs e)
        {

            if (checkBox1.Checked)
            {
                // Add the value in the registry so that the application runs at startup
                rkApp.SetValue("MyApp", Application.ExecutablePath.ToString());
            }
            else
            {
                // Remove the value from the registry so that the application doesn't start
                rkApp.DeleteValue("MyApp", false);
            }

        }

        private void RefreshStatus()
        {
            PowerStatus power = SystemInformation.PowerStatus;

            // Show the main power status
            switch (power.PowerLineStatus)
            {
                case PowerLineStatus.Online:
                    MainsPower.Checked = true;
                    break;

                case PowerLineStatus.Offline:
                    MainsPower.Checked = false;
                    break;

                case PowerLineStatus.Unknown:
                    MainsPower.CheckState = CheckState.Indeterminate;
                    break;
            }

            // Power level
            int powerPercent = (int)(power.BatteryLifePercent * 100);
            if (powerPercent <= 100)
                BatteryIndicator.Value = powerPercent;


            else
                BatteryIndicator.Value = 0;
            label3.Text = Convert.ToString(powerPercent);
            // Battery Remaining
            int secondsRemaining = power.BatteryLifeRemaining;
            if (secondsRemaining >= 0)
            {
                BatteryLifeRemaining.Text = string.Format("{0} min", secondsRemaining / 60);
            }
            else
            {
                BatteryLifeRemaining.Text = string.Empty;
            }

            // Battery Status
            BatteryStatus.Text = power.BatteryChargeStatus.ToString();
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            RefreshStatus();
            RefreshTimer.Enabled = true;
        }

        private void RefreshTimer_Tick(object sender, EventArgs e)
        {
            RefreshStatus();
            PerformShapeDetection();
        }

        public void PerformShapeDetection()
        {
            //Load the image from file and resize it for display
            Image<Bgr, Byte> img = _capture.QueryFrame();

            //Convert the image to grayscale and filter out the noise
            Image<Gray, Byte> gray = img.Convert<Gray, Byte>().PyrDown().PyrUp();

            Gray cannyThreshold = new Gray(180);
            Gray cannyThresholdLinking = new Gray(120);
            Gray circleAccumulatorThreshold = new Gray(500);
           
            CircleF[] circles = gray.HoughCircles(
                cannyThreshold,
                circleAccumulatorThreshold,
                4.0, //Resolution of the accumulator used to detect centers of the circles
                15.0, //min distance
                5, //min radius
                0 //max radius
                )[0]; //Get the circles from the first channel

            Image<Gray, Byte> cannyEdges = gray.Canny(cannyThreshold, cannyThresholdLinking);
            LineSegment2D[] lines = cannyEdges.HoughLinesBinary(
                1, //Distance resolution in pixel-related units
                Math.PI / 45.0, //Angle resolution measured in radians.
                20, //threshold
                30, //min Line width
                10 //gap between lines
                )[0]; //Get the lines from the first channel

            #region Find triangles and rectangles
            List<Triangle2DF> triangleList = new List<Triangle2DF>();
            List<MCvBox2D> boxList = new List<MCvBox2D>(); //a box is a rotated rectangle

            using (MemStorage storage = new MemStorage()) //allocate storage for contour approximation
                for (
                   Contour<Point> contours = cannyEdges.FindContours(
                      Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
                      Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_LIST,
                      storage);
                   contours != null;
                   contours = contours.HNext)
                {
                    Contour<Point> currentContour = contours.ApproxPoly(contours.Perimeter * 0.05, storage);

                    if (currentContour.Area > 250) //only consider contours with area greater than 250
                    {
                        if (currentContour.Total == 3) //The contour has 3 vertices, it is a triangle
                        {
                            Point[] pts = currentContour.ToArray();
                            triangleList.Add(new Triangle2DF(
                               pts[0],
                               pts[1],
                               pts[2]
                               ));
                        }
                        else if (currentContour.Total == 4) //The contour has 4 vertices.
                        {
                            #region determine if all the angles in the contour are within [80, 100] degree
                            bool isRectangle = true;
                            Point[] pts = currentContour.ToArray();
                            LineSegment2D[] edges = PointCollection.PolyLine(pts, true);

                            for (int i = 0; i < edges.Length; i++)
                            {
                                double angle = Math.Abs(
                                   edges[(i + 1) % edges.Length].GetExteriorAngleDegree(edges[i]));
                                if (angle < 80 || angle > 100)
                                {
                                    isRectangle = false;
                                    break;
                                }
                            }
                            #endregion

                            if (isRectangle) boxList.Add(currentContour.GetMinAreaRect());
                        }
                    }
                }
            #endregion

            imageBox3.Image = img;

            #region draw triangles and rectangles
            Image<Bgr, Byte> triangleRectangleImage = img.CopyBlank();
            foreach (Triangle2DF triangle in triangleList)
                triangleRectangleImage.Draw(triangle, new Bgr(Color.DarkBlue), 2);
            foreach (MCvBox2D box in boxList)
                triangleRectangleImage.Draw(box, new Bgr(Color.DarkOrange), 2);
            imageBox3.Image = triangleRectangleImage;
            #endregion

            #region draw circles
            Image<Bgr, Byte> circleImage = img.CopyBlank();
            foreach (CircleF circle in circles)
                circleImage.Draw(circle, new Bgr(Color.Brown), 2);
            imageBox2.Image = circleImage;
            #endregion

            #region draw lines
            Image<Bgr, Byte> lineImage = img.CopyBlank();
            foreach (LineSegment2D line in lines)
                lineImage.Draw(line, new Bgr(Color.Green), 2);
            imageBox3.Image = lineImage;
            #endregion

        }

        }
    }



[rbtying]

There are relatively few people on this forum (or in the world in general...) that use C# to program their robots. This is mostly because C# is not at all suited for the task... learn to use C++ or Python and you'll find tutorials and help all over the web.

As for shape identification from pictures, you could try getting a picture from the camera, processing it, and then getting the next picture. Not too sure how to help you beyond that.

[mstacho]

Two things:

a) I use C# for the PC code that talks to my robot hand.  This is because everything else is set up in C# for the project, so I had to choose either write one piece of code or re-write everything in c++

b) What you're talking about doing is hard.  Really, really hard.  And you don't seem to have defined the problem particularly well.  Let me try to re-define (and then tell you to just do it in C++ using OpenCV  )

- rbtying is correct -- the way you do it is to take a picture from the camera and process it.  This appears to be what Emgu.CV library is capable of doing.  But OpenCV can do it continuously with an image stream from a camera, which is another reason to use OpenCV directly instead of a wrapper class that works in .Net.

- When you say "shape", what do you mean?  Do you mean the outline of the object?  If so, I'm going to assume that the object is the only thing on an otherwise monochrome (preferably white) table.  Otherwise you'll have to do some kind of pattern recognition (which OpenCV can do quite well).  If you just mean the outline, then again OpenCV can help here, and it's actually really easy.  Basically, as long as the object/table set up is like I describe, you take a picture, analyze it to extract all non-white pixels, turn it into a binary image, and run a contour algorithm on it.  The contour is the outline of the object almost always (unless your camera sucks).

If you mean anything other than the outline (like some parameterized mathematical description of the object)...good luck.  It's not impossible, but it's hard, almost certainly computationally intractable in real-time, and probably not something that's easily explained by a forum post haha, sorry!

But if you mean just object recognition (like "identify the shapes" seems to suggest) then OpenCV is once again good for this.

- What are the objects you are going to "pick up"?  There are hundreds of grasping algorithms out there, some more robust than others.  But if your objects are going to be things like balls and squares, and the gripper is simple enough, it would suffice to just move the arm to hover over the centroid of the object and close the gripper.  More detail?

MIKE

[iamaldrin]

Thank you for your replies. I used C# since the examples in EMGU CV was in C#, im not that familiar in both languages, since i've only been using VB. Now i'll try using C++. Anyway, is there a GUI to setup in C++?

I also need these for C++ programming, I'll try to search for it but help is always appreciated

1. How to detect laptop battery status
2. Run at start up
3. How to use microcontroller

I'm talking about basic shapes (Rect, Square, Tri, Circle...etc. ). How can I use the program for the coordinates of the object? The object will be in a black or white floor, and the robot will search for it using the camera. I've seen blob detection, is it good for recognition? 

Also, which is better for adaptive mapping? Camera or IR? Since i'll be using a camera, i thought i'll use it too for adaptive mapping or obstacle avoidance.

I really need help since it's my first time doing all these. Thanks a lot!

[mstacho]

well, EMGU CV is basically OpenCV that works in C#. However, I'd REALLY suggest using OpenCV and C++, since that way you can port your code to something that isn't a windows machine.  Plus, Microsoft is planning on moving away from the .Net framework, so it's a bad choice to develop for that one :-P

What do you mean "is there a GUI to set up in C++"?

What you appear to be asking is too general for any help to be given.  For instance, "how to use microcontroller" depends entirely on which uC you're using and what you intend to do with it! :-) Please be much more specific.  It's basically impossible for any of us to answer all of these questions in a single thread.  I suggest you start doing some work, then when you have specific questions (like "how do I program an Arduino") we'd be able to help you a lot more clearly.

Your shape detector is still going to be hard, I'm afraid, unless all you want to do is draw a box around the object.  but OpenCV has a LOT of stuff you can use.  For instance, for the coordinates of the object, you just need to detect the object (which will be a blob detector) then find the centroid of that blob (for which OpenCV has the functions as well).  But...it'll still be a big project. 

I don't know what you mean by "adaptive mapping".  I assume you mean SLAM?

From this post, I can tell you have a lot of work ahead of you and aren't really sure where to start.  I suggest starting small -- if you've never used C++ before, it doesn't make sense to jump right into OpenCV and get things done.  I think you need to sit down and plan out how you're going to attack your project, then start small!

MIKE

[iamaldrin]

I don't have a choice, I need to finish the program in less than a month then it'll be integrated with the robot. We're in a rush because it's a school project. I'm learning c++ now, downloaded 50 tuts for that.

Adaptive mapping, I can create a map through c++ and determine where the robot will go. I've seen it done in the SOR tutorials.

Nevermind the GUI, i think it's all in the command prompt. I'll be starting at camera input, then shape determination.

[corrado33]

Look at the wavefront tutorial on this site.  It talks about a couple lines of code to account for adaptive mapping.  Basically if no solution is found, it starts over and scans everything again.  That's adaptive.   

[Admin]

Here is how I did it, using C:

http://www.societyofrobots.com/robot_ERP.shtml

Source code and designs are all included.

Robot Grabs & Throws ERP test

[iamaldrin]

Here is how I did it, using C:

http://www.societyofrobots.com/robot_ERP.shtml

Source code and designs are all included.

Robot Grabs & Throws ERP test

That's what I want! It goes to the object then picks it up. I want to learn that!

It's a bit overwhelming though, i am really new at this. I'm limited to using only C++, uC don't know which yet, a Sharp IR, and a camera and a motherboard for the program. We can't buy an ARM here in the philippines.

So, what I want to learn should be in order.

Camera Input, Shape identification, Picking up of object then adaptive mapping.

Can you guide me please. I have lots to read now.
Thank you for your patience!

[Admin]

errrr if this is your first robot, I'd suggest not building an expert level robot on your first try lol . . .

And if this is exactly what you want, you could just copy my open source designs like this guy is doing:
http://www.societyofrobots.com/robotforum/index.php?topic=13599.0


Start off with just a robot arm, and after you finish, then work on the next thing. Basically do it bit by bit, project by project, and at the end then combine it all together. Don't bite off more than you can chew, etc.

[iamaldrin]

errrr if this is your first robot, I'd suggest not building an expert level robot on your first try lol . . .

And if this is exactly what you want, you could just copy my open source designs like this guy is doing:
http://www.societyofrobots.com/robotforum/index.php?topic=13599.0


Start off with just a robot arm, and after you finish, then work on the next thing. Basically do it bit by bit, project by project, and at the end then combine it all together. Don't bite off more than you can chew, etc.

Yes, that's what i'm supposed to do, but it's a thesis.. so i'm a little short in time. I'm only focusing on the programming part, building the robot will be my group's problem. So, I want to learn those things, then when i'm done, we'll integrate it with the robot.

[Admin]

ah, in that case . . . use RoboRealm + laptop + microcontroller.

or with more programming:
OpenCV + laptop + microcontroller

[iamaldrin]

ah, in that case . . . use RoboRealm + laptop + microcontroller.

or with more programming:
OpenCV + laptop + microcontroller

Yes, i will use OpenCV, C++  using VS2010, an atom motherboard to store the program, and a microcontroller through USB port.



This is the robot that we will use, we'll replace the arm with the one that my group is making, and attach an IR sensor and a camera.