Developing Sensors That Give Intelligence to Robots

Which have the better auditory perception, humans or robots?
On September 23, 2005, when the Bio-Mimetic Control Research Center (BMC) of RIKEN was opened to the public, noise was reproduced randomly from any one of six loudspeakers vertically arranged in two rows of three loudspeakers. Visitors to BMC were asked to compete with the robot that Dr. Hiromichi Nakashima and his colleagues have developed to see which was more effective in identifying which loudspeaker was on. How did the competition end up?
"You may turn round to look when someone calls to you. This is unconscious human behavior, but it is more difficult for current robots to perform than complicated mathematical calculations," says Dr. Mukai. The phrase "sound source localization" means locating the position of a sound source. The current mainstream approach in giving robots the ability of sound source localization is the "microphone array" technique with multi-microphones, which uses the differences in the sound signals picked up by multi-microphones to identify the direction of the sound.
However, what Dr. Nakashima is trying to develop is a sound source localization robot with only two microphones. The robot tries to aim its camera at the estimated direction of sound, and repeatedly learns the most effective way to estimate the direction by confirming the difference between the real and the estimated directions. "Use of a larger number of microphones is, of course, advantageous to sound source localization. Strange to say, however, many animals have only two ears. There must be some benefits, such as easier data processing or learning, in the fact that they have only two ears. We want to know what the benefits are," says Dr. Nakashima.
However, how can we identify the direction of sound with only two microphones? A sound source is localized in the lateral direction on the basis of the difference of the loudness (sound pressure) and the arriving time between the sound arriving at the left and right microphones. For example, when a sound comes from the left side, the sound arrives at the left microphone earlier than the right microphone. Furthermore, the sound pressure at the left microphone is larger than that at the right microphone. What is more difficult is how to locate the vertical direction of a sound source. For performance improvement, Dr. Nakashima and his colleagues tried to install "outer ears" (sound reflectors) around the microphones (photo on left side of cover page). With the introduction of the outer ears, direct sounds from the sound source to the microphones interfere with the sounds reflected by the outer ears, thus causing sound wave cancellation between the direct and reflected sounds. The resulting sound, picked up by the microphones, exhibits some amplitude dips in the frequency spectrum because of the sound wave-cancellation (Figure 1 B). The dip pattern varies with changes in the location of the vertical sound source, and is used to estimate the vertical direction of the sound source.
Now coming back to the result of the competition at the open house, it was not the visitors but the robot that won the game. However, the sound source that was used in the experiment was what we call "white noise," which contains an equal intensity of all wavelengths, and exhibits no dips in the frequency spectrum (Figure 1 A). When the white noise is used as a sound source, the robot can easily detect the dips caused by the interference of sound. Thus the robot won the game in its strongest field. However, a sound source generally exhibits some dips in its frequency spectrum. "Since our robot is capable of analyzing a combination of four or five dips, it can distinguish the dips that are attributed to the sound source itself from those caused by the interference of sound," says Dr. Nakashima, referring to the features of the system.
However at the open house, there was a case when the robot could not locate the sound source with accuracy. It was the moment when a lot of noise was generated as the event captured a larger and larger audience. Humans have the ability to identify a talker's voice in a noisy environment such as at a cocktail party. Thus one of the future challenges for the robot is to develop a technique that gives the robot the ability to sort out only necessary sounds from amongst other noise.
Dr. Nakashima and his colleagues are planning to conduct original experiments where auditory information and visual information captured on camera are integrated for judgment processing. When we hear a sound, and when the situation of what we see in the direction of the sound is almost in agreement, we tend to think that the sound is generated at the position of what we see. This is how ventriloquism provides the illusion of a doll speaking. However, when the ventriloquist is separated from the doll, we can easily tell that it is the ventriloquist who is actually speaking. Humans learn to tell the reasonable and appropriate distance that is necessary for matching the auditory and visual information. Dr. Nakashima says, "We will try to make the robot learn the appropriate distance." Maybe the day will come when a robot can enjoy ventriloquism.

Creating a smell identification robot
Dr. Mukai points out, "From among the five senses, the one in which robots are the most inferior to animals and humans is olfactory perception. The concentration of a known gas could be derived. However, if the kind of gas is unknown, the robot can neither tell the kind nor the concentration of the gas. The only thing the robot can detect is the existence of the gas. This is typified by the fact that some gas-leak detectors in kitchens are responsive to hot sake." "We are conducting research on how to identify the kinds and concentration of gases by using semiconductor gas sensors, which are considered to be the most durable, and used for gas-leak sensors," says Dr. Yo Kato, Research Scientist. When the semiconductor in a semiconductor gas sensor is heated to a high temperature, gases are absorbed or combusted (oxidative reaction) on the heated surface, and the semiconductor gas sensor uses the changes in electrical resistance to detect the concentration of gases. However, conventional gas sensors have been unsuccessful in identifying kinds of gases.
Why do they fail to identify the kind of gas? Because the change in electrical resistance caused by a single type-A gas molecule absorbed at the surface can be the same as that caused by two type-B molecules absorbed at the surface. This leads to an inability to separate A from B. At present, the mainstream approach for gas identification is to prepare and arrange many kinds of sensors that have different relationships between the kind of gas and the resulting change in electrical resistance. However, only about 10 kinds of sensors are available now because manufacturing itself is limited.
On the other hand, Dr. Kato, Research Scientist, and his colleagues are trying to identify the kinds and concentration of gases by applying the concept of "active sensing," which actively changes the state of a sensor, thus periodically changing the surface temperature of the semiconductor. The electrical resistance shows different time characteristics with different kinds and concentrations of gases, which enables gases to be identified (Figure 2). According to Dr. Kato, "When A burns at 80 ºC and B burns at 100 ºC, the difference in temperature can be used for identification. Our principle is based on this concept. So far, we have confirmed by experiment that just a single sensor is capable of identifying eight kinds of gases. We think it possible to identify further kinds of gases by changing the period of the temperature variation on the surface of the semiconductor sensor, or by changing the upper and lower limits of the variation."
He and his colleagues are advancing a study on how to mount this gas sensor on disaster-relief robots. For example, these robots are expected to detect gas-leaks at disaster sites. However, they need to detect gases in real time in an ever-changing environment because they are expected to move about, sometimes under a strong wind at a disaster site. Dr. Kato and his colleagues have built a gas detecting system that is capable of detecting gases in real time by providing a semiconductor surface area of 1 mm² with a heater that can change the temperature of the semiconductor from 80 to 320 per second. In the photo on the cover page, Dr. Kato (middle) is holding a robot that is equipped with this sensor. The robot has three sensors at each of the apexes of a triangle, and the robot finds its way to the region where a greater condensation of gas is detected.

Towards the birth of "RI-MAN" in 2006
At BMC, researchers are working on research into "bio-mimetic" control, which is a technology that mimics the highly-sophisticated control functions of living systems. Dr. Zhi-Wei LUO, Laboratory Head of the Environment Adaptive Robotic Systems Laboratory, is playing the leading role to integrate main achievements of the center to develop their robot, which will soon be completed. The robot is called "RI-MAN." "We aim to develop a robot that can directly contact humans, for example, a robot that is helpful for nursing care. Our short-term task with RI-MAN is to develop the capability to hold an infant of about 10 kg in its arms. This will be a world-first challenge," says Dr. Mukai.
The robot cannot hold an infant in its arms without tactile sensors. When we hold an infant, we will try to feel the position where the pressure is located and control our arms accordingly. In the same manner, when the robot tries to hold an infant, it should control its arms by feeding back the information obtained from the tactile sensors.
Without tactile sensors, the robot may hold a person in its arms so strongly that it may cause harm to the person. However, existing tactile sensors for robots can detect only simple tactile senses such as "struck" or "touched," and the accuracy of signals from the sensors are insufficient to use for feedback signals. Why has the development of tactile sensors for robots been left on the back burner? "Basically because tactile sensors have not been required for conventional robots, which have been used in stable environments in factories, and only required to accurately perform routine operations. There has been no idea of covering a robot with tactile sensors that produce information signals on which the robot does its mechanical work," says Dr. Mukai.
He made up his mind to study the structure of the human skin so that he can develop curved-surface tactile sensors that can cover the entire body and accurately detect contact points and contact strength. The human skin includes a slightly hard layer of outer skin and a soft layer of inner skin. The complicated structure of the skin allows the contact pressure to be focused on the receptor organs of tactile sensation for accurate signal detection. Dr. Mukai and his team members have completed a structure where the contact pressure is focused on high-precision semiconductor sensors by combining elastic bodies of different hardness and hard prongs (Figure 3).
However, RI-MAN would be covered all over by cables if cables were used to directly connect the signals from the numerous semiconductor sensors mounted around RI-MAN to the central computer, which plays the role as the brain of the robot. To cope with this problem, they installed a small computer at the proximity of a sheet of semiconductor sensors (8 x 8 sensors), thus building up a system that can sort out and compress the information from the sensors and send the processed information to the central computer. For example, when one of the arms or legs of the robot bumps into something, one of the computers responsible for that portion issues a directive to withdraw it. This is analogous to the reflex action in human beings, because when a person touches something very hot, they instantaneously draw back their hand. In this reflex motion, a directive is issued from the spinal cord, not from the brain.
Dr. Mukai and his team members, using an experimental robot arm wrapped with these tactile sensors, are advancing collaborative research with the Environment Adaptive Robotic Systems Laboratory on how to hold a person properly in the arms of a robot (top of the cover page, the two black sheets are tactile sensors). RI-MAN has speech recognition capabilities and is equipped with a sound source localization system and gas sensors developed by the Biologically Integrative Sensors Laboratory. When a person gives RI-MAN an order by voice to hold the child in its arms, RI-MAN turns around and looks at the person. Then RI-MAN moves toward the child and uses its tactile sensors to hold the child in its arms. In the future, RI-MAN will be able to use a gas sensor to detect when the baby has wet its diaper. The development of RI-MAN must be a step toward practical application of an intelligent robot that can live side-by-side with humans.

References: "A Learning System for Estimating the Elevation Angle of a Sound Source by Using a Feature Map of Spectrum," IEICE, D-II, Vol. J87-D-II, No.11 (2004) (in Japanese). "Mobile Robot Control Using an Active Type Semiconductor Gas Sensor,"The 5th SICE System Integration Division Annual Conference (SI2004) (in Japanese). "Development of Soft Areal Tactile Sensors for Robots with Skin," "The Frontiers of the Development of the Super-Five-Sense Sensors," NTS (2005) (in Japanese). Japanese Patent Application No. 2005-189093: Soft Tactile Sensor and Its Fabrication Method (in Japanese).

Source : Toshiharu Mukai RIKEN Frontier Research System Bio-mimetic Control Research Center Biologically Integrative Sensors Laboratory Laboratory Head, Toshiharu Mukai, Dr. Eng.