What shapes are you looking for? I would assume just the basic shapes, circle, square, triangle, right? 3D shapes or just 2D?
Each of those shapes has a different number of lines. Your algorithm can do what is called edge detection, which is a very simple and well known algorithm to implement. Edge detection finds the outside lines of the shape, then you can count the number of lines. A circle has 1 line, a triangle has 3, a square has 4, etc.
For more complicated shapes, your algorithm can then calculate the angle between each line. Obtuse triangle, right triangle, etc.
For a rectangle, see if 2 lines consist of more pixels than the other two lines.