Predicting People’s Eye and Hand Movements

Steve, tell us about yourself. What is your background and what do you do today in your role at NASA?

I studied behavioral science at UC Berkeley, which was a combination of some elementary physics and quantitative social science, especially demography and economics. I almost majored in economics but became interested in social and experimental psychology because of their more empirical approach to the study of human behavior. I eventually became mainly interested in spatial perception and sensory motor interaction which I studied in graduate school at McGill University and later as a postdoc a Brown and UC Berkeley.

During a later research job in Professor Lawrence Stark’s lab at UC Berkeley, I became interested in user physical interaction with moving systems. This work led to a NRC Research Associateship and ultimately a civil service job at NASA Ames. I currently use virtual environments to study phenomena relevant to the design of telerobotic systems, as might be used by astronauts during planetary surface exploration..

Why is NASA interested in virtual environments. Is it to be able to train astronauts and understand astronaut behavior?

My work on telerobotics is just one example. NASA as a whole is interested in using vehicle and other simulators for a wide variety reasons. They’re interested in the perceptual accuracy operators may have or need  during aircraft or spacecraft flight. They are also interested in how pilots or other users of aerospace systems, such as  airtraffic controllers, can safely and effectively use the computer-based interfaces that are now becoming common.  A recent example of these kinds of user interfaces is that being designed by Google for their driverless car.

What errors do you see in the research that you’ve done that people are making? How do we address these errors?

I am particularly interested in geometric errors, but it’s not just a question of error, it’s also a question of how long it takes.

To address these errors, we try to measure and model them. What we’re doing now is to build a virtual environment as a kind of telerobotic simulator. We are in fact building it as a tool that can understand user responsiveness and certain types of distortions so that we can thereafter use computational modeling to account for these errors.

Steve, what are some everyday applications of your research?

That’s a good question. But before answering it we must step back and define what a virtual environment is.  It is a kind of visual display that can give its user the impression that they are somewhere else than where they are physically present.  It does this by presenting its user with the sights, sound, and possibly touch they would sense were they to actually be in the simulated environment.   As such it could called a direct personal simulator.  It contrasts with the vehicle simulators that preceded it, such as aircraft simulator, which are indirect in that their users are put in a room that is in then made to appear to be somewhere else.  The virtual environment is a medium for communication. It’s artificial and imperfect so that’s one of the reasons why we have to study the distortions in our ability to use it to perceive and act within the simulated environment. Virtual environments are hardly evident in everyday applications. But they are used for training of telerobot operators, for spacesuit operations, and for a variety of technical aspects of military and civilian mission operations.  We are beginning to see them as media for video games but there may still be some technical issues related to user safety that will need to be resolved before their use is widespread.

Because you’re modeling behavior, it sounds like you can predict where people will be moving? For example, with a surgeon, can you predict how they will be moving their hands during an operation?

That’s literally what we’re doing. More importantly, we’re developing analytic models to predict patterns of hand movement during some aspects of difficult teleoperation and how difficulty interacts with the time delays potentially present during space telerobotics. The second thing is, we’re using our understanding for another investigation into latency. It’s a very difficult task have to have immediate control when significant time lag is present and so we’re trying to come up with a description of that effect as well.

Any other insights that you would share with the broader public on how your research can be helpful in their lives?

There’s all of the technical stuff about the work we do. I don’t know that would be appropriate, but there are some things that are more general application. For example, one of the things that NASA did focuses on the very important role that simulation plays. You need some type of test or simulation system when you develop new aircraft of spacecraft cockpits to verify that the activities required to operate the vehicle are tractable by qualified pilots.  These simulation systems are also typically good for operator training as well. The simulators let those evaluating the new tasks look for unexpected difficulties. But sometimes problems are missed.  An example can be seen in the use of side-stick controllers to operate some modern commercial jetlines.  While these new controllers have several advantages over the older, larger yokes typically used in the cockpit for primary manual flight control (climbing, diving, or turning), their typical position on the outboard sides of the cockpit, (far left and far right) and small amount of physical movement during operation has made it difficult for the two pilots to observe exactly what and when their counterpart is giving the aircraft a control input.  Normally, this is not a problem, but under conditions of significant mechanical failure, as in  the crash of Air France Flight 447, the poor visibility of each other’s control motion provided to seriously aggravate the situation.

One of the things we can learn from simulation testing is to always try to be skeptical about evidence and explanation of failures and to examine a wide variety of alternative situations to see if a conventional explanation can really stand up to scrutiny. In fact, this kind of skepticism is good in science and engineering in general as well as in life. When seeking the cause of some failure or even a simple phenomenon, we must not only consider the evidence that confirms that our conjectured cause actually does  precede an effect, but also whether other influences might additionally act like a cause.  The scientists’  use of a Control Group is one way  to exercise this caution.  But it is not necessarily the only way.

A concrete medical example can make this point clear and illustrate when it is particularly important. Consider for example the disease of Multiple Sclerosis.  This is a neurological disease without a clearly understood cause and which is indicated by a wide variety of seemingly randomly intermittent symptoms, e.g., fatigue, weakness, spasticity, balance problems, bladder and bowel problems, numbness, vision loss, tremors and depression.  Because of the intermittent nature of the symptoms, it is very difficulty to study the effectiveness of treatment solely in terms of symptomatic relief.   After any given experimental treatment during the presence of one of the symptoms, there is a very good chance that the symptom will abate within a few days or weeks giving the impression that the treatment is effective.  The key thing is to compare what happens when the treatment is tried versus when it is not tried.  This is the control.  Sometimes even in the control condition, the MS patient will improve.  This likelihood must be compared with the likelihood of improvement after actual treatment.  In essence one must compare the confirmatory information that the treatment worked with the disconfirmatory information that the patient improved despite the absence of a treatment.  This could be called the disconfirmatory evidence.

When most layman consider the results of a scientific evaluation of a drug test, they tend to focus on the positive confirmatory evidence, i.e., the probability of improvement after treatment.

But this is only half the story.  If a patient has a malady that almost always goes away, almost any treatment might appear to work if only the confirmatory information is used.  We all  seem to have a natural tendency to focus on the positive which is why the importance of examining the disconfirmatory information needs to be trained in school.  This training can be begun in an age-appropriate way with young children in primary school, with more and more sophisticated instruction introduced as they move towards high school, ultimately leading to the teaching of the meaning of Bayes Theorem.

I think learning how to balance both confirmatory and disconfirmatory information lies as the basis of the critical thinking we should universally training in our country’s  basic education system.  It is important for voters in a democracy to be able to distinguish good from poor evidence.  In teaching this balance we show that science is not just a pile of facts and a rote process of  observation-theorizing-testing and revision but is, in fact, a type of critical thinking with the goal of approximating physical truth.