Foundation Track·Core Concept

Multimodal Communication in HRI

35 min
1 / 10

The Problem with Screens

For forty years, the screen was the universal interface. If you wanted to communicate with a user, you put it on a glowing rectangle. Text, icons, modal dialogs. Designers mastered the art of managing what appeared on that surface and in what order.

Robots shatter this paradigm. The robots that actually matter (collaborative factory arms, hospital delivery bots, autonomous vehicles) operate in physical spaces where screens are useless. A worker carrying a fifty-pound load cannot stop to read a modal dialog. A surgeon cannot look away from a patient to check a robot's battery status.

Physical robots must communicate through the environment itself. They use sound, light, motion, and touch. Designing these channels deliberately is called multimodal communication. It is arguably the most technically creative area of HRI.

Multimodal Communication Channels
FIGURE 3.1: The five primary communication channels available to robots. Effective HRI design selects these based on the message, environment, and user state.
References3 selected references · updated Aug 9, 2026

These references support nonverbal robot communication through movement, orientation, color, and sound. Channel choices remain context-specific and must be validated with intended users and relevant safety engineering; communication cues do not replace safety-rated controls.

  1. 01
    Survey of Non-facial/Non-verbal Affective Expressions for Appearance-Constrained Robots

    Cindy L. Bethel and Robin R. Murphy · 2008 · IEEE Transactions on Systems, Man, and Cybernetics, Part C

    Supports: Body movement, posture, orientation, color, and sound as robot expression channels.

    Peer-reviewed research
  2. 02
    Nonverbal Sound in Human-Robot Interaction: A Systematic Review

    Brian J. Zhang and Naomi T. Fitter · 2023 · ACM Transactions on Human-Robot Interaction

    Supports: The design, use, and evaluation of nonverbal robot sound.

    Peer-reviewed research
  3. 03
    Legibility and Predictability of Robot Motion

    Anca D. Dragan, Kenton C. T. Lee, and Siddhartha S. Srinivasa · 2013 · ACM/IEEE International Conference on Human-Robot Interaction

    Supports: Motion as a channel for communicating robot goals and intent.

    Peer-reviewed research
How HRI Academy reviews content Report a content issue
← PreviousProxemics: Personal Space in HRI
Next →Anthropomorphism and the Uncanny Valley