Alysha Talking Head "Open Source" Project

[Robo_Pi]

Hello everyone.  I've decided to start a project to develop a robotic talking head.  I'm using a female head and I've named her Alysha.  From a previous thread I started there appears to be some interest in robotic human heads.  So I thought I'd start a thread dedicated to this type of project.  Everyone is welcome to comment, contribute, or add information from similar projects.

In the meantime I thought I'd share what I've done thus far.   To begin with I just found a fairly decent female mask that I might be able to use for the actual project.  CyberJeff contributed a link to a "Party City" mask that may prove to be useful.  So I would like to thank CyberJeff for that important contribution.   The mask it designed to be worn by an actual human however.  It's meant to be a Halloween mask even though it appears to cover the entire head and even has its own ears and hair!   But it has no eyes, teeth, or tongue.  However, I already have plans for securing those additional items.

In the meantime I thought I would begin with the actual design work.  This is going to be a HUGE project.  Make no mistake about that.  But at the same time I believe that it is doable and within the reach of the average hobbyist.  The main thing to realize is that this head is indeed going to appear as a "Talking Manikin".  It's in no way going to be convincing as an actual living entity.   But hey, what should hobbyists expect right? 

In any case, I've been working out what it might take to make this thing talk.  I'm not concerned with speech synthesis, since computers already do that.  I plan on using standard speech synthesis and a speaker.  So the actual speech synthesis is basically already done.   There are tons of resources for that.

In this project the goal is to make a head that merely "appears" to be talking.   It is with this goal in mind that I offer the following brainstorming ideas:

To begin with I started a spreadsheet and simply typed down one column the letters of the alphabet.  Then I set about saying each letter and trying to decide how many articulation motions I would need.  Obviously I could get carried away if I got lost in trying to recreate too many details.  So instead I chose the exact opposite extreme.  What is the absolute LEAST motion that I could get away with and still have a fairly convincing talking head?

Well, after analyzing the entire alphabet and the first ten numerical digits I came to the realization that I only need 6 basically motions.    Imagine that.   This means that the entire speech mechanism could be accomplished with as few as 6 servos.

Here's the six motions that I came up with:



The first column marked Key_# is just the number of the important motions.  As I went through the alphabet I realized that most letters would look the same in terms of visual appearance so I marked those as duplicates of previous letters.  When all is said and done I only ended up with these six "Primal Motions".

The second column is just the letter of the Alphabet where these motions FIRST appeared.   Believe it or not you can visually reproduce the entire alphabet with just these six motions of your mouth, teeth, and tongue.   And it does require all three of these objects.   The third column tells how many motions are required for a letter.  Notice that I have A marked as zero.  That's because A will be the "resting position" or the "home" position of the mouth.  So nothing needs to be done to articulate the letter A (or any other key #1 letter).  (by the way I have the full spreadsheet if anyone would like to see the entire 26 letters and how they are made up of just these six motions).  Keep in mind we are only talking about "Visual appearance" here.  Not actually making any sounds.   Sound making is not required for this project.  That will be taken care of by speakers and a computer simulated voice.

All we need to do here is construct at head that "appears" to be making these sounds.   And these six motions will suffice.

The column marked Comb_1 describes how the motions are made up from other primal motions.

The column marked "Mouth Pos" is the position of the mouth (simply O-pen, or C-losed).  That's all we need.  Except for the O sound which requires a special effects on the lips.

The column marked "Teeth Pos" is the position of the teeth (again simply O-pen, or C-losed). That's all we need except for the special F sound which requires the lower jaw to retreat under the upper teeth.  So that's another "special effect".

And finally we have the "Tongue Pos" where the tongue is either laying on the bottom of the mouth (its home or N-ormal position) or it curves up to touch the roof of the mouth, the UP position.  Moving the tongue will be subtle but could make the head look quite a bit more realistic having a tongue actually taking part in speech.

So that's it.  Just six motions for the speech.   Six Servos!  Not too bad.

I also have in mind an idea for a jaw that kind of floats around a bit on it's own to add a more realistic bounce to the speech.  The head itself may have other feature that could change even during speech.  But right now I'm just focusing on what it takes to articulate speech motion.

What's Next?

Well, after I figured out these motions. I decided to draw them up and then make a simple animated GIF of them.

Here are the six basic motions:



You'll probably notice there are actually seven drawings here, but E is the same as A.  Visually they are the same.  And keep in mind, I'm reducing this to the bare essentials.  Clearly a person could easily get carried away trying to reproduce every little human motion.  But let's not go there.   It's simply not required.  These Six motions will do, at least for a hobby head. 

We're not working on a Hollywood movie BUDGET! (or at least I'm not)  Gotta keep things simple.

I made an animated GIF of these six motions.  But my photo hosting site doesn't seem to handle animated GIFs, I'll try to add it here as an attachment.  Not sure if that will work or not.

Where do we go from here?

I guess the next step is to get out some servos and some prototyping materials to see what it's going to take to articulate these six motions. 

[cyberjeff]

Pretty good research working that out. Compares well with the animation guides:

http://animation.about.com/od/flashanimationtutorials/a/animationphonem.htm
http://www.garycmartin.com/mouth_shapes.html

Been busy. I meant to mention that when you go looking for jaws and teeth  that male and female head shapes are different. Men have a more squared off chin:

https://www.google.com/search?q=male+vs+female+head

You probably know that already, but most of what I have seen look distinctly male.

On another note. What do you think of CX?:
http://cx-lang.org/

Also, add Ivona to your list of speech possibilities if you haven't already:
http://developer.ivona.com/en/speechcloud/introduction.html

I like the name Alysha.

[Robo_Pi]

Pretty good research working that out. Compares well with the animation guides:

Thanks, I've actually done quite a bit more work on it since I started this thread.  I actually wrote a program that will construct animated GIFs from my six basic motions automatically for sentences of arbitrary length.   Unfortunately the graphics I drew up to this point isn't real great.  I'll have to take some time to draw up better pictures for the animations. In fact, I might post a request for help with that in this thread, but this forum doesn't seem to be very well populated.  The other inconvenient issue is that GIFs don't provide for sound.  The up side is that they can be converted to AVI movies and then sound can be added.   It's nice to get a visual idea of how to best model these particular motions.

Been busy. I meant to mention that when you go looking for jaws and teeth  that male and female head shapes are different. Men have a more squared off chin:

I'll be building the jaw from scratch, so I'll make it to simply fit the mask.  I imagine the mask already has a pretty decent shape.  At least I hope it does.  Actually I hope it's so flexible that it basically collapses without a frame.  But even so it will still have a natural shape that it tends to fit over.  I'll try to match up with that.

For the teeth I'm actually thinking that one of those cheap plastic "Chattering Teeth" might do pretty well. If they are plastic they may be malleable if heated.  If they are then I could shape them to fit precisely the way I want them to fit.   Only time will tell on that one.  I can't know until I get this stuff in hand and see what potential it has.

You probably know that already, but most of what I have seen look distinctly male.

My original goal was to build two heads side-by-side.  One male and one female.  Alysha and Arathoon.  But thus far I haven't found a decent face mask for Arathoon.    I may still build two skull armatures side-by-side, maybe a face for Arathoon will show up down the road.

On another note. What do you think of CX?:

This is the first I've heard of it.   I think I'm going to get the Altera Cyclone II board.  First I'm going to download their programming software and see if I like that.  The software is free.  I actually have some antiquated FPGA and CPLD development boards that I might get out and play with first.  I used to program these things years ago.  But it's been a while.  I thought I'd get the Altera Cyclone II just so I can be on the same page with what some other people are doing.

Also, add Ivona to your list of speech possibilities if you haven't already:

Actually my main interest in robotics is AI.  So I'm looking to program from the ground up.  Not for me, but for the sake of the robot.  If you've read Marvin Minsky's "Society of Mind" you'll know why it's important to start with "The Builder" paradigm for AI development.   Everything evolves from that seed.

I like the name Alysha.

I originally named her Lisa, but when I started making up her personality dictionary I ended up changing it to Alysha.  In part because I wanted her name to start with an "A" since she will be the first one.   

[cyberjeff]

We have much the same interest in developing AI. I've had some time to think about personalities and stumbled across Myers Briggs Type Indicator (MBTI). Think Carl Jung, not Freud.

About 3% of the population is of the ENTP type, often called the Originators. Here is a simple test:

http://www.25quiz.com/

Something more here on ENTP:

http://psychology.about.com/od/trait-theories-personality/a/entp.htm

I mention this because I believe that is your type, as well as mine. Three quarters of the population will have not a clue about what you are about. So, well populated or not, it will be hard to run with this until it becomes more mainstream, and that is where people will be able to "connect". Here is a more populated forum:

http://letsmakerobots.com/

But it will suffer from the same type of issues.  The originators of that group largely left (or got thrown out) in mass a couple months ago. Still there is some interesting work there:

http://letsmakerobots.com/robot/project/super-droid-bot-anna-w-learning-ai

It's worth keeping this project here at SOR, but look around there.

[Robo_Pi]

About 3% of the population is of the ENTP type, often called the Originators. Here is a simple test:

These are my results.

41% INTP   - Mostly independent.
37% INFP - Very Intimate in my feelings.
5% ENFP - Slightly Inspirational.
5% ENTP - Slightly Explorational
3% INFJ -  And apparently I have very little concern or thoughts of humanity.

I don't put a lot of faith in these kinds of tests.  In fact, I don't give the study of psychology itself a lot of merit in any case.  What psychology does is look at large groups of people and then tries to define "Scientific Stereotypes" from that.  And then it turns around with tests like these and tries to fit those broader stereotypes onto specific individuals.  I personally feel that this is "Bad Science" and therefore I don't give psychology much credence for even being a valid  science.   It's more like astrology.  It can be amusing but don't take it too seriously.  You might get better results from reading Tarot Cards.

Something more here on ENTP:

http://psychology.about.com/od/trait-theories-personality/a/entp.htm

I think anyone who knows me would say that the terms innovative, clever, and expressive can definitely be used to describe me, yet look how low my score was for ENTP.   So, like I say, these psychology tests are more like reading Sun Sign astrology descriptions.   Also people have a very innate tendency to be attracted to tests that produce results they would like to identify with whilst ignoring or playing down other tests where they don't care for the results.  One thing for sure if you go around the Internet taking every "Psychology Test" you can find you will indeed come back with many different results.  In fact, I've done that in the past.  And the results were quite often as different as night and day from once site to another.  Different psychologist use different types of "Bad Science" to do their astrology predictions. 

I mention this because I believe that is your type, as well as mine. Three quarters of the population will have not a clue about what you are about. So, well populated or not, it will be hard to run with this until it becomes more mainstream, and that is where people will be able to "connect". Here is a more populated forum:

Well, according to the quiz I'm very far from being ENTP.  I only got 5% for that  particular label.  But I'm definitely innovative, clever, and expressive.

I think also, there tends to be two main types of individuals.  Those who have innovative and clever ideas, and those who don't.  Those who do, don't always cooperate well together because they prefer their own clever innovative ideas over those of others.   Those who don't have clever innovative ideas may be far more willing to contribute their other skills to those who do.   A good example would be this particular talking head project.  Right now I could use a graphic artist to draw up the frames of talking head animations.  An artist who is really good at drawing, but who has no clue how to actually build a talking head, might be very  pleased to contribute the artwork.  There is no "conflict" in terms of creative innovative ideas in this case.

So everyone who contributes to a project doesn't need to be in competition for the actual design ideas.  In fact, it's better for the project as a whole if many of the actual contributors are just contributing skills rather than design ideas.

http://letsmakerobots.com/

But it will suffer from the same type of issues.  The originators of that group largely left (or got thrown out) in mass a couple months ago. Still there is some interesting work there:

And this probably happened because everyone in that group was most likely "competing" to have their robotics ideas win favor.   This happens a lot when you get a whole bunch of innovative designers together, they all want to be the innovators, and there's no one left to actually contribute to getting things done.

Actually if I want a graphic artist to draw up my robot face I'd be better off joining an art forum where the artists would love to contribute their artistic skills to a project that they otherwise know nothing about.

It's worth keeping this project here at SOR, but look around there.

Well, this seems like a nice forum.  And hopefully some other people will show up along the way.  So I'll keep the project here, but I might post it other places as well.   Like I say, going  to an art forum instead of a robot forum might actually pan out way better if my immediate goal is to obtain some artistic contributions. 

In fact, I think I'll go look for an art forum right now.  I would really like to get some better GIF frames to animate.  Ideally I would like to have at least two views of the face simultaneously, head-on and profile.   I could actually draw something up better myself, but finding the time to do it is another question entirely.

Currently I actually have a lot of mechanical designs in mind that I would like to start drawing up too.   The actual mechanisms that are going to control the actual head.   I might try drawing those up in 3-D.  But again, there's the time issue.  I have other things to do besides work on the robot.  Unfortunately.

[cyberjeff]

About 3% of the population is of the ENTP type, often called the Originators. Here is a simple test:

These are my results.

41% INTP   - Mostly independent.
37% INFP - Very Intimate in my feelings.
5% ENFP - Slightly Inspirational.
5% ENTP - Slightly Explorational
3% INFJ -  And apparently I have very little concern or thoughts of humanity.

I don't put a lot of faith in these kinds of tests.  In fact, I don't give the study of psychology itself a lot of merit in any case.  What psychology does is look at large groups of people and then tries to define "Scientific Stereotypes" from that.  And then it turns around with tests like these and tries to fit those broader stereotypes onto specific individuals.  I personally feel that this is "Bad Science" and therefore I don't give psychology much credence for even being a valid  science.   It's more like astrology.  It can be amusing but don't take it too seriously.  You might get better results from reading Tarot Cards.

Thanks, but I hold with my opinion. Tests can be off. In fact your tests show you to be an extrovert (E), show you are a pattern and not a detail person (N), that you make your judgements not on how you feel but on what you think is best (T), and finally that you are perceptive and not judgemental (P). See how that works?

The brain is designed to process information and it can be wired multiple ways to do so. As we develop we favor some paths over other and it works to their preferred nature. There is always flexibility but there is a preferred path.

Some paths are better for some work than other. Athletes and police need to make quick decisions, not get hung up in processing options. Artists need to very involved up in their work, so much so that they avoid external influences that over stimulate them. A classic artist is very introverted and has difficulty explaining their work and greater difficulty working beyond their frame of reference.

No matter.

As far as what happened in LMR. The site was sold and new owners laid down rules and restrictions that never existed before. The people that left were those that were the most likely to help others and for the most part had opened up fully their projects. They weren't competitors.

Now back to the task at hand and I certainly wish you success.

[Robo_Pi]

Thanks, but I hold with my opinion.

And that's fine because I definitely have no desire to change your opinions. But like you, I hold my own. 

Tests can be off. In fact your tests show you to be an extrovert (E),

I don't think anyone who knows me would ever suggest that I'm an extrovert.  My sister is pretty heavy into psychology as this is her profession, and she has made it crystal clear that I'm definitely and introvert with no room for doubt.    I'm not going to argue with her because I actually agree with her assessment.

show you are a pattern and not a detail person (N),

Isn't it possible to be both simultaneously?  I have always felt that I do both quite well.  I tend to work in the abstract "bird's-eye" view of things very well to get my projects organized quickly.  But then I often end up zooming in on the details almost to the point of spending too much time on the details.  I've learned that it's best to keep zooming back out to the more abstract "pattern" view when I find that I'm getting too bogged down in the details.  So I would suggest that I'm both pattern and detail oriented. 

I've also been told by several people (typically employers) that tell me that I'm a "Results-Oriented" worker.  When given a project I tend to work in ways that produce the quickest results, and then deal with the details after the basic project is up and running.  Employers tend to like that type of worker, especially if they have clients who are anxious to see some results.

that you make your judgements not on how you feel but on what you think is best (T)

Again, I think I do both.  Typically I do consider the "best" solution whenever possible.  However, how I feel about something can be a big factor in some cases.  Sometimes I will chose what I like best, even if it's not necessarily the best choice available.

and finally that you are perceptive and not judgemental (P).

I like to think so.  But I also feel that this is something that has come with maturity more than anything.  I wasn't always so perceptive, and I used to be quite judgmental.  I've become far more considerate,  empathetic, and less judgmental as I have matured.  And, unfortunately this doesn't seem to be an automatic trend for everyone.  So why I have grown to be who I am today is anyone's guess. 

The brain is designed to process information and it can be wired multiple ways to do so. As we develop we favor some paths over other and it works to their preferred nature. There is always flexibility but there is a preferred path.

And this is certainly a very deep and controversial topic for sure.  I respect your views on this, but at the same time I reserve the right to consider other possibilities as well.    I think too that these types of considerations become highly philosophical and whether a person is oriented toward a purely secular materialistic existence versus a potentially mystical or spiritual existence can influence how they might consider these questions.

Someone who is convinced of a purely secular materialistic existence is naturally going to begin with premises that support that paradigm.   Those who are open to a potentially mystical or spiritual existence are going to allow for that premise.  So which premises we allow for can vastly influence which conclusions we are likely to arrive at.

And now back to Alysha

Putting humans aside for the moment and considering a potential Robot AI scheme,... then clearly the "design" of the brain is going to play a far different role.  In the case of Alysha's brain, I will definitely be constructing her primal personality, and hopefully she will turn out to have the personally her creator intended for her. 

But at the same time, it is my hope that I construct her brain in such a way that she will be able to modify her own personality over time herself, and thus potentially change her own personality as she grows.   It will be quite interesting to see how that unfolds.   And if she does become "independent" at what point do we call that "Sentience"?

Interesting questions to be sure. 

It would be great if she could someday be able to actually take one of these psychological tests.  What do we do then?  Accept the results and claim that she does indeed have the psychological profile that the test indicates?

And if she does, from whence did that personality arise?

Very interesting questions to be sure.

[cyberjeff]

Thanks, but I hold with my opinion.

And now back to Alysha

Putting humans aside for the moment and considering a potential Robot AI scheme,... then clearly the "design" of the brain is going to play a far different role.  In the case of Alysha's brain, I will definitely be constructing her primal personality, and hopefully she will turn out to have the personally her creator intended for her. 

But at the same time, it is my hope that I construct her brain in such a way that she will be able to modify her own personality over time herself, and thus potentially change her own personality as she grows.   It will be quite interesting to see how that unfolds.   And if she does become "independent" at what point do we call that "Sentience"?

Interesting questions to be sure. 

It would be great if she could someday be able to actually take one of these psychological tests.  What do we do then?  Accept the results and claim that she does indeed have the psychological profile that the test indicates?

And if she does, from whence did that personality arise?

Very interesting questions to be sure.

Here is the way I see it.

We accumulate and store and sort what we like and what we don't. How we feel about something can be drawn from how feel about things we deem to be similar. So it is that to be build an individuality depends on how we store information and how that  information is retrieved.

So, I see a couple of ways to collate and retrieve.

Take a look at hypergraphs:

https://en.wikipedia.org/wiki/Hypergraph

I can also see  that as being expressed in Barycentric coordinates in much the same way. These are both ways of mapping how close one thing is to another. I can forsee a way to do this in a run of the mill SQL database. And another in an FPGA.

As far as personality, I don't believe it is directly related to capacity for intelligence or the available knowledge. Cats and dogs and even goldfish can have individual personalities, none of it based on what they know, but on on how they go about interacting and processing.

[Robo_Pi]

Just to clarify the "Alysha Talking Head Project" for anyone reading this thread.

AI is a fascinating topic.  And my own personal real "Alysha Project" ultimately is an AI project.   Of course for the purposes of this thread the "Alysha Talking Head" project is only concerned with lip-syncing a mechanical face with a speech synthesizer program.   A lot of people would be happy to have a robot that could just do that much alone, never mind that they will program into it precisely what it will say (i.e. no AI required just to have a robot that appears to be talking).

So for the purposes of this "Open Source" project, the only goal here is to create a convincing lip-syncing machine.

The AI aspect of the project is a totally separate issue and far beyond the scope of the "Alysha Talking Head" project.

I just want to make this clear for others who might be reading this thread.  This thread itself it not about AI.  And the "Alysha Talking Head" Open Source Project, is aimed solely at the mechanics of creating a lip-syncing face.

Ideas, for AI, should be addressed in threads specifically addressing those topics.

This thread is really aimed at just the mechanical construction of a lip-syncing mechanical skull and face.  What the robot might actually have to say is beyond the scope of this project.

[mklrobo]

  Hello!
I found a product on robotmarketplace.com, that sell a automated face for
approx. $20,000!!! It has a lot of features, very impressive, and beautiful. Looks like a face out of
I ROBOT, the movie in which Will Smith was in.
I like the face and product, but pardon me if I do not buy it.
I think RoboPi is doing great work with his  face project, and giving robot hobbyists great opportunity
to have an awesome robot that is affordable. I understand money has to be made to continue technology
to even have robots, but there has to be a "sweet spot" that we all can enjoy. This is the true heart of robots, I think, and I appreciate people like RoboPi who are doing work that they love, and share it freely. Good JOB Robo Pi!! 

[Robo_Pi]

I'm not very impressed by the automated face for $20,000.  And the reason I'm not impressed is because of the price.  It would seem to me that a manufacturer should be able to do better than that for 20 grand.   The one I'm working on could sell for more like $2,000.  Especially if it were being mass produced by a competent factory.

One difference I have noted however, is that they do claim that there face has stereoscopic vision through the eyeballs using micro cameras.   That's a feature that I'm not sure I could emulate in a $2000 product.  The idea I currently have for the eyeball cameras is to use a flexible fiber optic cable the size of the pupil that actually runs from the pupil to a small camera.  The cost could be kept down considerably using this method, but I'm not sure how well it would work either.  I don't know how well a fiber optic cable would work for this purpose.  And there's also the problem that it might not be cosmetically pretty at the eyeball end.

In any case, unlike a factory, I don't have long work days to devote to just developing this head.  Nor do I have a team of technicians working under me to do the grunt work.  Lately I've been working on preparing my house for the winter instead of working on the robot head.

I've also been learning how to use the Microsoft.Speech Namespace in C#.   Microsoft offers some truly fantastic speech code to play with.  They have an entire Grammar System called SRGS (Speech Recognition Grammar Specification).  It's really cool.   It allows you to organize subjects, objects, verb, adjectives, and even has a neat way to deal with prepositions.   It also has a PromptBuilder which is related to the GrammarBuilder but is used for speech synthesis instead of speech recognition.  And they even have an entire semantic system so the works can actually be recognized in semantic context.  It's pretty amazing.

In any case, I haven't done much of anything toward actually starting to build the mechanical head yet.  Ironically I wasn't all that impressed with the Party City Mask I'm thinking of using, but after looking at the $20,000 head the Party City Mask actually looks quite sufficient.

Unfortunately I'm not sure if I'll even be able to do this at all.  My health is failing and I'm no spring chicken anymore.  I have some good ideas, but not sure if I'll ever be able to actually bring them into physical reality.