Research on Multimodal Communication

Red Hen Lab is an international distributed cooperative of researchers in multimodal communication. We are senior professors at major research universities, senior developers in technology corporations. We also include junior professors, postdoctoral students, graduate students, undergraduate students, and even a few advanced high school students. Red Hen develops code in Natural Language Processing, audio parsing, computer vision, and joint multimodal analysis.

Red Hen's multimodal communication research involves locating, identifying, and characterizing auditory and visual elements in videos and pictures. We may provide annotated clips or images and present the challenge of developing the machine learning tools to find additional instances in a much larger dataset. Some examples are gestures, eye movements, and tone of voice. We favor projects that combine more than one modality, but have a clear communicative function—an example would be floor-holding techniques. Once a feature has been successfully identified in our full dataset of several hundred thousand hours of news videos, cognitive linguists, communication scholars, and political scientists can use this information to study higher-level phenomena in language, culture, and politics and develop a better understanding of the full spectrum of human communication. Our dataset is recorded in a large number of languages, giving Red Hen a global perspective.

For GSoC 2020, we invite proposals from students for components for a unified multimodal processing pipeline, whose aim is to extract information from text, audio, and video, and to develop integrative cross-modal feature detection tasks. Red Hen Lab is directed jointly by Francis Steen (UCLA) and Mark Turner (Case Western Reserve University).

lightbulb_outline View ideas list


  • nlp
  • asr
  • opencv
  • machine learning
  • data science


mail_outline Contact email

Red Hen Lab 2020 Projects

  • Wenyue Xi
    2020 GSoC Red Hen Lab Proposal for AI Recognizers of Frame Blends, Especially in Conversations About the Future
    For the project idea "AI Recognizers of Frame Blends, Especially in Conversations About the Future," Wenyue developed multiple approaches that can...
  • Xiaoyu Lu
    Age Group Prediction in TV news
    It has been raising industrial and research interest to analyze the age biasing in various video formats. Here I'm proposing a project to make use of...
  • Zhiqi KANG
    Hand gesture detection and recognition in news videos
    The goal is to implement a reliable pipeline which can take a raw RGB video as input and output information such as whether there is a hand gesture...
  • Himani Negi
    Image and audio clustering
    The projects aim to design a system that clusters the images and the audio from the media broadcasts and then re-orders them accordingly in the red...
    Multimodal TV Show Segmentation
    The main research question or problem is to split the videos into named show and dated show. Identify anchor/show names or recognizes what show it...
  • Frankie Robertson
    Pipeline for posture, and posture and gesture embeddings including addition of query by gesture functionality into vitrivr
    This proposal concerns the addition of pose data extraction using OpenPose and the generation of posture and gesture embeddings to Red Hen’s...
  • gulshan_kumar
    Red Hen Rapid Annotator
    With Red Hen Lab’s Rapid Annotator we try to enable researchers worldwide to annotate large chunks of data in a very short period of time with least...
  • Henry Smith
    Understanding Messages to Underrepresented Racial, Ethnic, Gender, and Sexual Groups on Social Media by Democratic Politicians and their Electoral Implications
    Social media has continued to proliferate, not only as a space for self-expression, but also for greater communication between politicians and...