List 6 of the 10 specific sensory/perceptual systems discussed in the chapter.

1.3.3 Perceptual representation and intelligence

The perceptual system contains primarily visual, auditory, and kinesthetic input, that is, pictures, sounds, and feelings. There is also olfactorial and gustatorial input, that is, smells and tastes. Perceptual representation is a modeling approach that highlights the constructive, or generative, function of perception, or how perceptual processes construct a complete volumetric spatial world, complete with a copy of our own body at the center of that world. The representational strategy used by the brain is an analogical one; that is, objects and surfaces are represented in the brain not by an abstract symbolic code or in the activation of individual cells or groups of cells representing features detected in the visual field. Instead, objects are represented in the brain by constructing full spatial effigies of them that appear to us for all the world like the objects themselves, or at least so it seems to us only because we have never seen those objects in their raw form but only through our perceptual representations of them.

Perceptual intelligence refers to the ability to interact with the environment through various sensory organs, such as vision, hearing, touch, etc. The visual system gives the organism the ability of visual perception. The auditory system gives the organism the ability of auditory perception. Using the research results of big data and deep learning, the machine has gotten closer and closer to the human level in perception intelligence.

Visual Illusions

The human perceptual system (in fact the whole brain) is designed for efficiency, and to achieve this, it “cheats.” It cuts corners where it can. Illusions reveal the brain’s shortcuts. For example, the “Pinna–Gregory Illusion” is an optical illusion whereby concentric circles made of small squares appears to be spiraling inward when those squares are tilted (Fig. 3-12) [Pinna and Gregory 2002]. We might conjecture that by seeing the sides of the squares as tangents to a line, at each locality we see the line moving inward, and so we perceive a line always moving inward—a spiral—where in fact no such line exists. In a similar way, the Zöllner illusion (Fig. 3-13) and the Café Wall Illusion (Fig. 3-14) perceptually suggest that lines that are in fact parallel, seem to bend toward each other.

Illusions then are misperceptions; misperceptions due to misleading sensations, or at least sensations for which the brain is predisposed to interpret in a particular way. We know that by the time visual information gets to the brain it’s been preprocessed, and that some of the preprocessing involves looking for lines with particular orientations—small visual units from which an amalgamated perception can be derived when combined with many other small visual units.

Certainly, as virtual reality is about the creation of virtual worlds that we want to be (mis)perceived as being real, we have an interest in creating illusion. Of course this desire is not limited to the realm of computer-generated stimuli, and also not just to the profession of magicians (illusionists), but also to other artists. The architect Guarini, for the Santissima Sindone dome in Turin, Italy, uses geometric illusion amplified by stone color and texture to create the perception of a much taller structure than the reality [Meek 1988].

Context plays a big role in how we perceive visual images. That context may be as simple as directionality, or it may involve variations in surrounding shapes and colors. A very simple illusion can be seen with a grayscale vertical gradient inside a circle. When the darker end of the gradient is on the top, we perceive a concave impression in a surface, but when the darker end is on the bottom we see a raised convex bump (Fig. 3-15). The explanation for this is merely that we are accustomed to light shining from above, and using that “knowledge,” a concave hole would have a shadow at the top, whereas a convex bump would have a shadow beneath the protuberance. If you turn this book around while looking at Fig. 3-15, you will see that the circles that had appeared to be raised are now depressed, and vice versa.

Context can also affect our color perception. Human perception often informs us of the relationship between stimuli rather than direct values—because absolute values are hard to determine. There’s always context. Visually, there are lighting conditions that affect how we see colors (Fig. 3-16). Proprioceptively, how much force it takes to lift an object might change based on the grip of the object or the lifter’s level of exhaustion. Sonically if there is a lot of background noise we may not be able to discern a precise pitch. (In a book it’s easier to demonstrate this visually.) Fig. 3-16 shows two pair of diamonds that appear to be two shades of gray. Actually, that’s not true, Fig. 3-16 shows four diamonds all the same color, the same shade of gray, but the differing backgrounds, and the visual distractions (“noise”) allow our perception to infer that certain objects or shapes are “in shadow” and therefore must be brighter than the actual stimulus provides.

On the flip-side, color can also be the cause of illusion. In the Poggendorff illusion, an obscuring shape masks a pair of different-colored lines, one of which terminates under the mask, and the other changing shade to something closer to the other, and the single protruding segment will appear to move in a way that it is more in line with the truncated segment (Fig. 3-17). In a different vein, the shade of an object may affect our distance perception of that object. In the aforementioned dome for the Cappella della Santissima Sindone (chapel of the Shroud of Turin) Guarino Guarini selected the stones to use at each level of the dome based on their color and texture to enhance his geometric illusion of a dome with extended height. Specifically, Guarini put less dark and more polished stone lower in the dome (nearer the viewer) because they will appear to be larger, and less polished darker-appearing stone higher up, giving the impression of those stones being even farther away [Meek 1988][Evans 2000].

One final illusion related to vision to discuss is the illusion of “vection.” Vection is the false sense of motion caused by visual stimulation. Thus, this case is a cross-sensory illusion, and more accurately could be described as a vestibular illusion—and so indeed, we will discuss this further in the Vestibular Illusion section.

1 Stability and Constancy in Perception

The ability of our perceptual system to overcome the effects of changes in stimulation and maintain a veridical and stable percept is called perceptual constancy (Epstein 1977, Walsh and Kulikowski 1998). In discussing the constancies it is common to make a distinction between the distal stimulus and proximal stimulus. The distal stimulus is the actual physical stimulus, the physically objective dimensions of the viewed object. The proximal stimulus, in the case of vision, is the very image that falls on the retina. It changes with changes in position or lighting of the physical stimulus. Constancy can be defined as the ability to correctly perceive the distal stimulus; that is, the stable properties of objects and scenes, in spite of changes in the proximal stimulus.

One example of constancy is shape constancy, which refers to the ability of the visual system to ascertain the true shape of an object even when it is not viewed face on but slanted with respect to the observer. In an example of shape constancy the distal stimulus might be a door half open, while the proximal stimulus will be the trapezoid shape it projects on the retinas. Shape constancy in this case would be the perception of the distal rectangular shape of the door in spite of the fact that the proximal shape is far from rectangular. Another example of constancy is lightness constancy. Lightness refers to the perceived reflectance of an object; high reflectance is perceived as white and very low reflectance as black, with intermediate reflectances as various shades of gray. We are capable of perceiving the distal lightness of a surface in spite of the fact that the proximal amount of light reaching our eyes changes with changes in the amount of light illuminating that surface. For example, if we hold a piece of chalk in one hand and a piece of charcoal in the other, the chalk will be perceived as white and the charcoal as black. This will be true if we observe the two in a dimly lit room or in bright sunshine, in spite of the fact that the amount of light reaching the eyes from the charcoal in the sunshine might be greater than that from the chalk in the dim room lighting.

The perceptual constancy that has received the greatest amount of research attention is size constancy (Ross and Plug 1998). If the distance is not very great we usually perceive the distal size of objects in spite of the changes in their proximal size as their distance from us changes. The two theoretical approaches mentioned above, the constructivist and the direct, deal with size constancy differently. According to the constructivists, size constancy is achieved through some process whereby the perceptual system perceives the object's distance and then takes the distance into account and ‘corrects’ the proximal image to yield a true distal percept. The direct approach asserts that there is no need to involve the perception of distance in the perception of size. Instead, it claims that enough information exists in the visual environment to allow direct perception of size without the need for aid from additional ‘higher’ mental processes.

People filter information

People miss information constantly. Our perceptual system filters out more than it lets in. That isn’t a bug; it’s a feature! If we didn’t work this way, we couldn’t function in this booming, buzzing, rapidly changing world. We’d be overloaded.

Millions of years of evolution designed us to ignore most of what is going on around us and to focus our attention on what is important. When our prehistoric ancestors were hunting in the East African veldt, what was important was what was moving and what looked different from the grassy background. It might be animals they regarded as food … or animals that regarded them as food.

In modern times, when a pilot scans cockpit displays, what is important is what is abnormal, what is changing, and how it is changing. When a business executive prepares a presentation, what is important is the presentation content and anything that seems like it will help her prepare her presentation on time. Everything else is irrelevant and is ignored.

V.B.2. Reduce User's Memory Load

Capabilities and limitations of the human memory and perceptual systems were discussed earlier in this section. The interface should help users remember information while using the computer. We know that people are not good at remembering things, so programs should be designed with this in mind. Table III lists the design principles in this area.

Table III. Principles That Reduce Users' Memory Load

A sign that computer systems do not help users' memory is the use of external memory aids, such as sticky pads, calculators, reference books, frequently asked questions (FAQs), and sheets of paper. Often, application users must write information down on a piece of paper because they know they will need that information later in the program. Program elements such as undo and redo and clipboard actions such as cut, copy, and paste allow users to manipulate pieces of information needed in multiple places within an application and across applications.

Filling in on-line forms with common information such as name, address, and telephone number should be remembered by the system once a user has entered them or once a customer record has been opened.

Interfaces support long-term memory retrieval by providing users with items for them to recognize rather than having to recall information. It is much easier to browse a list to select an item rather than trying to remember the correct item to type into a blank entry field. For example, sophisticated spell-checking techniques offer users a list of possible alternatives to select from for a misspelled word, rather than just identifying that a word is spelled incorrectly.

It is critical to continuously show users where they are, what they are doing, and what they can do next. These visual feedback indicators provide the context for users to understand where they are and where they can go. Early hypertext techniques allowed users to navigate between pieces of information and documents, but users got lost and could not remember why they were at their current location and how they got there!

VII.F Perception

Humans interact with continuously flowing, diverse stimulation. Likewise, humanoids need multimodal perceptual systems that can seamlessly integrate sensors. One way to do this is to allow sensors to continually compete for dominance. At the Electro Technical Laboratory in Japan, G. Cheng and Y. Kuniyoshi have developed a humanoid with 24 dof, joint receptors with encoders and temperature sensing. The humanoid uses separate processors for control of auditory, vision, motor output, and integration. The robot itself is lightweight and flexible, allowing it to interact comfortably and safely with humans. Throughout a visual and auditory tracking task, the robot tracks a person by sight and/or sound while mimicking the upper body motion of a person. The focus of the work was to show that the robot could track people using a multiple sensory approach that is not task-specific, and does not need to switch between sensor modalities. The idea is that perceptual subsystems necessary for mimicry, tracking, vision, and auditory processing should be thought of as essential capabilities that must together contribute to high-utility human-like behavior. This said, humanoid roboticists generally agree that vision is the singlemost important sensing modality for enabling rich, human-like interactions with the environment. Of course, computer vision has long been a hard problem of itself. The main problem is that many factors are confounded into image data in a many-to-one mapping. For instance, how can a humanoid infer three-dimensional reality from a two-dimensional image? Furthermore, there is an amazing amount of data to be processed. For a long time, computer vision assumed that the goal was simply to acquire as much data about the environment as possible, and then apply “brute force” processing. This approach proved computationally intractable. Rather than view perceptual systems as passive receptors, which merely collect any and all data, we are beginning to create perceptual systems that can interact with humans and with the physical environment, actively structuring a perception of reality rather than just passively trying perceiving it.

Luiz-Marcos Garcia, Antonio A. F. Oliveira, Roderic A. Grupen, David S. Wheeler, and Andy H. Fagg use attentional mechanisms to focus a humanoid robot on visual areas of interest. On top of this capability, researchers have implemented a learning system that allows the robot to autonomously recognize and categorize the environmental elements it extracts. Robots must be equipped to exploit perceptual clues such as sound, movement, color intensity, or human body language (pointing, gazing, and so on). For rich sensor modalities such as vision, perception is as much a process of excluding input as receiving it.

Humans naturally find certain perceptual features interesting. Features such as color, motion, and face-like shapes are very likely to attract our attention. MIT has been working to create a variety of perceptual feature detectors that are particularly relevant to interacting with people and objects. These include low-level feature detectors attuned to quickly moving objects, highly saturated color, and colors representative of skin tones. The robot's attention is determined by a combination of low-level perceptual stimuli. The relative weightings of the stimuli are modulated by high-level behavior and motivational influences. For a task involving human interaction, the perceptual category “face” may be given higher priority than for a surveillance task where the robot must attend most closely to motion and color.

A sufficiently salient stimulus in any modality can supercede the robot's attention, just as a human watching a film might respond to sudden motion in the adjacent seat. MIT has implemented a number of intuitive arbitration rules into the system such as the fact that, all else being equal, larger objects are considered more salient than smaller ones. The goal is for the robot to be responsive to unexpected events, but also able to filter out superfluous ones. Otherwise the robot would become a slave to every whim of its environment. MIT has found that their attention model enables people to intuitively provide the right cues to direct the robot's attention. Actions such as shaking an object, moving closer to your intended listener, hand waving, and altering tone of voice all help the robot focus in on appropriate aspects of its environment (Fig. 21).

1 Introduction

During the past decade, researchers have explored the use of animation in many aspects of user interfaces. In 1984, the Apple Macintosh used rudimentary animation when opening and closing icons. This kind of animation was used to provide a continuous transition from one state of the interface to another, and has become increasingly common, both in research and commercial user interfaces. Users commonly report that they prefer animation, and yet there has been very little research that attempts to understand how animation affects users’ performance.

A commonly held belief is that animation helps users maintain object constancy and thus helps users to relate the two states of the system. This notion was described well by Robertson and his colleagues in their paper on “cone trees”, a 3D visualization technique that they developed.

“Interactive animation is used to shift some of the user's cognitive load to the human perceptual system…. The perceptual phenomenon of object constancy enables the user to track substructure relationships without thinking about it. When the animation is completed, no time is needed for reassimilation.” [19 p. 191].

Researchers including Robertson have demonstrated through informal usability studies that animation can improve subjective user satisfaction. However, there have been few controlled studies looking specifically at how animation affects user performance. These studies are summarized below.

User-interface design.

Humans can recognize the spatial configuration of elements in a picture and notice relationships among elements quickly.

This highly developed visual system means people can grasp the content of a picture much faster than they can scan and understand text.

Interface designers can capitalize on this by shifting some of the cognitive load of information retrieval to the perceptual system. By appropriately coding properties by size, position, shape, and color, we can greatly reduce the need for explicit selection, sorting, and scanning operations. However, our understanding of when and how to apply these methods is poor; basic research is needed. Although our initial results are encouraging, there are many unanswered user-interface design questions. How can we

design widgets to specify multiple ranges of values, such as 14 to 16 or 21 to 25?

let users express Boolean combinations of slider settings?

choose among highlighting by color, points of light, regions, and blinking?

allow varying degrees of intensity in the visual feedback?

cope with thousands of points or areas by zooming?

weight criteria?

select a set of sliders from a large set of attributes?

provide “grand tours” to automatically view all dimensions?

include sound as a redundant or unique coding?

support multidimensional input?

Habituation

A third way in which experience biases perception is called habituation. Repeated exposure to the same (or highly similar) perceptions dulls our perceptual system’s sensitivity to them. Habituation is a very low-level phenomenon of our nervous system: it occurs at a neuronal level. Even primitive animals like flatworms and ameba, with very simple nervous systems, habituate to repeated stimuli (e.g., mild electric shocks or light flashes). People, with our complex nervous systems, habituate to a range of events, from low-level ones like a continually beeping tone, to medium-level ones like a blinking ad on a website, to high-level ones like a person who tells the same jokes at every party or a politician giving a long, repetitious speech.

We experience habituation in computer usage when the same error messages or “Are you sure?” confirmation messages appear again and again. People initially notice them and perhaps respond, but eventually they click them closed reflexively without bothering to read them.

Habituation is also a factor in a recent phenomenon variously labeled “social media burnout” (Nichols, 2013), “social media fatigue,” or “Facebook vacations” (Rainie et al., 2013); newcomers to social media sites and tweeting are initially excited by the novelty of microblogging about their experiences, but sooner or later get tired of wasting time reading tweets about every little thing that their “friends” do or see—for example, “Man! Was that ever a great salmon salad I had for lunch today.”

Habituation

A third way in which experience biases perception is called habituation. Repeated exposure to the same (or highly similar) perceptions dulls our perceptual system’s sensitivity to them. Habituation is a very low-level phenomenon of our nervous system: it occurs at a neuronal level. Even primitive animals like flatworms and amoeba, with very simple nervous systems, habituate to repeated stimuli (e.g., mild electric shocks or light-flashes). People, with our complex nervous systems, habituate to a range of events, from low-level ones like a continually beeping tone, to medium-level ones like a blinking ad on a Web site, to high-level ones like a person who tells the same jokes at every party or a politician giving a long, repetitious speech.

We experience habituation in computer usage when the same error messages or “Are you sure?” confirmation messages appear again and again. People initially notice them and perhaps respond, but eventually click them closed reflexively without bothering to read them.

Habituation is also a factor in a recent phenomenon variously labeled “social media burnout” (Nichols, 2013), “social media fatigue,” or “Facebook vacations” (Rainie et al., 2013): newcomers to social media sites and tweeting are initially excited by the novelty of microblogging about their experiences, but sooner or later get tired of wasting time reading tweets about every little thing that their “friends” do or see—for example, “Man! Was that ever a great salmon salad I had for lunch today.”