Research on Multimodal Communication

Red Hen Lab is an international distributed cooperative of researchers in multimodal communication. We are senior professors at major research universities, senior developers in technology corporations. We also include junior professors, postdoctoral students, graduate students, undergraduate students, and even a few advanced high school students. Red Hen develops code in Natural Language Processing, audio parsing, computer vision, and joint multimodal analysis.

Red Hen's multimodal communication research involves locating, identifying, and characterizing auditory and visual elements in videos and pictures. We may provide annotated clips or images and present the challenge of developing the machine learning tools to find additional instances in a much larger dataset. Some examples are gestures, eye movements, and tone of voice. We favor projects that combine more than one modality, but have a clear communicative function—an example would be floor-holding techniques. Once a feature has been successfully identified in our full dataset of several hundred thousand hours of news videos, cognitive linguists, communication scholars, and political scientists can use this information to study higher-level phenomena in language, culture, and politics and develop a better understanding of the full spectrum of human communication. Our dataset is recorded in a large number of languages, giving Red Hen a global perspective.

For GSoC 2019, we invite proposals from students for components for a unified multimodal processing pipeline, whose aim is to extract information from text, audio, and video, and to develop integrative cross-modal feature detection tasks. Red Hen Lab is directed jointly by Francis Steen (UCLA) and Mark Turner (Case Western Reserve University).

lightbulb_outline View ideas list


  • nlp
  • python
  • asr
  • opencv
  • machine learning


comment IRC Channel
mail_outline Contact email

Red Hen Lab 2019 Projects

  • ajp
    Accountability Classifier from Annotated Data
    The objective of this project is to automatically detect types of accountability in news articles when describing crimes such as shootings, using a...
  • Yong Zheng Xin
    Annotating NewsScape with FrameNet 1.7 and Expanding FrameNet with BabelNet and Deep Structured Semantic Models
    This project sets out to achieve two goals. The first objective is to update the annotation system for Red Hen’s NewsScape dataset to FrameNet 1.7...
  • Aashish Agarwal
    Automatic Speech Recognition for European Language (German)
    This project aims to build an ASR pipeline for European Language (German) and it must be built as a Singularity container on the Case HPC and put...
  • Ziyi Liu
    Chinese Pipeline
    Red Hen gathers Chinese broadcasts to make data sets for NLP, OCR, audio, and video pipelines. Currently, Red Hen have a preliminary ASR pipeline but...
  • Aniruddha Mysore
    Cockpit : The Red Hen Monitoring System
    This project automates the task of sensing the health of the many Red Hen Lab remote capture stations, which are Raspberry Pi devices, and provides a...
  • Xinyu You
    Design and develop an online deep learning course for humanists
    This Project goal is to design and develops an online course, to teach deep learning for students in the humanities and social sciences. The course...
  • Swagato Chatterjee
    Feature Recognition in works of Art and Iconographic Artwork Captioning
    In this project I propose to include a tool to the Red Hen Lab's Art pipeline based on Gradient Activated Class Maps which can be used for...
  • Ayush Raj
    Gesture Recognition in works of art
    I have worked on building an annotation tool which helps to correct the extracted pose from images and build a pipeline to retrieve the extracted...
  • Amr Maghraby
    GSoC 2019 | Red Hen Lab OCR
    OCR is a very wide application which translates characters in the image to an editable format. OCR on television news shows would recognize any text...
  • Sasi Kiran Bhimavarapu
    Multimodal Show Segmentation
    The goal of the proposal is to create an algorithm that can automatically find boundaries between TV shows in unannotated recordings and also find...
  • Poulami Sarkar
    OCR for Chinese, Arabic, Hindi,Urdu,Bengali
    For this project I wish to develop OCR for television news in a tri-phased implementation model. The first phase will be consist of successfully...
  • gulshan_kumar
    Red Hen Rapid Annotator
    With Red Hen Lab’s Rapid Annotator we try to enable researchers worldwide to annotate large chunks of data in a very short period of time with least...
  • Xumeng Chen
    Semantic Art from Big Data
    In the proposal, I described and demonstrated my ideas about the visualization tasks: 1) Clusters of Event Category Over Time 2) Distribution of...
  • Shreya .
    Speaker Adapted ASR Pipeline
    The aim of the project would be to develop an​ ASR pipeline utilizing the existing news conversation dataset and audio pipeline codebase....
    Speech Recognition for Indian English & Hindi
    This project aims to build an Automated Speech Recognition engine for Indian English and Hindi using Deep Learning( RNN-CTC, TDNN, LDA-MLLT, CNN )....