Red Hen Lab

Research on Multimodal Communication

Technologies
python, tensorflow, singularity, scikit-learn, syntaxnet
Topics
machine learning, artificial intelligence, multimedia, video processing, audio processing
Research on Multimodal Communication

Red Hen Lab is a distributed consortium of researchers in multimodal communication, with participants all over the world. We are senior professors at major research universities, senior developers in technology corporations, and also junior professors, postdoctoral students, graduate students, undergraduate students, and even a few advanced high school students. Red Hen develops code in Natural Language Processing, audio parsing, computer vision, and joint multimodal analysis.

Red Hen's multimodal communication research involves locating, identifying, and characterizing auditory and visual elements in videos and pictures. We may provide annotated clips or images and present the challenge of developing the machine learning tools to find additional instances in a much larger dataset. Some examples are gestures, eye movements, and tone of voice. We favor projects that combine more than one modality, but have a clear communicative function -- an example would be floor-holding techniques. Once a feature has been successfully identified in our full dataset of several hundred thousand hours of news videos, cognitive linguists, communication scholars, and political scientists can use this information to study higher-level phenomena in language, culture, and politics and develop a better understanding of the full spectrum of human communication. Our dataset is recorded in a large number of languages, giving Red Hen a global perspective.

For GSoC 2018, we invite proposals from students for components for a unified multimodal processing pipeline, whose aim is to extract information from text, audio, and video, and to develop integrative cross-modal feature detection tasks. Red Hen Lab is directed jointly by Francis Steen (UCLA) and Mark Turner (Case Western Reserve University).

2018 Program

Successful Projects

Contributor
Awani Mishra
Mentor
Heiko Schuldt-1, Anna Bonazzi, Tim Groeling, LucaRossetto, Kai Chan
Organization
Red Hen Lab
Multimodal Television Show Segmentation
University and libraries of social science and literature department have a large collection of digitized legacy video recordings but are...
Contributor
Vikrant Goyal
Mentor
Karan Singla, shrutigullapuram, mpac, Francis Steen
Organization
Red Hen Lab
Multilingual Neural Machine Translation System
The aim of this project is to build a single Machine Translation system using Neural Networks (RNNs-LSTMs, GRUs,Bi-LSTMs) to translate between...
Contributor
Burhan Ul Tayyab
Mentor
Anna Pleshakova, Abhinav Shukla, ivan.giangreco, Weixin Li
Organization
Red Hen Lab
Russian Ticker Tape OCR
We are proposing an OCR framework for recognizing ticker text in Russian Videos. We do this by solving two main problems, improving the OCR by...
Contributor
Sumit Vohra
Mentor
Mehul Bhatt, skrish13, Jakob Suchan
Organization
Red Hen Lab
Multimodal Egocentric Perception (with video, audio, eyetracking data)
Hey, I have been in constant touch with Mehul regarding my project on Multi-modal Egocentric Perception. I have already had a skype meet with him...
Contributor
Shuwei Xu
Mentor
Huijian Lv, littleowen, Peter Uhrig, Mark Turner
Organization
Red Hen Lab
Automatic Speech Recognition for Speech-to-Text on Chinese
In this project, a Speech-to-Text conversion engine on Chinese is established, resulting in a working application. There are two leading candidates...
Contributor
Gyanesh Malhotra
Mentor
Mehul Bhatt, Rajesh Kasturirangan, Jakob Suchan
Organization
Red Hen Lab
Multi modal Egocentric Perception (with video and eye tracking data)
This project aims to tackle the problem of egocentric activity recognition based on the information available from two modalities which are video and...
Contributor
Devendra Yadav
Mentor
Mehul Bhatt, Abhinav Shukla, shrutigullapuram, skrish13, Jakob Suchan
Organization
Red Hen Lab
Emotion detection and characterization in video using CNN-RNN
This project aims to develop a pipeline for emotion detection using video frames. Specifically, we detect and analyze faces present in the video...
Contributor
Vaibhav Gupta-1
Mentor
Vera Tobin, Peter Uhrig, Peter Broadwell, Kai Chan
Organization
Red Hen Lab
Rapid Annotator
With Red Hen Lab’s Rapid Annotator we try to enable researchers worldwide to annotate large chunks of data in a very short period of time with least...
Contributor
Xu Tony
Mentor
Peter Uhrig, Mark Turner, Francis Steen, Weixin Li, Huijian Lv, littleowen, Jacek Woźny
Organization
Red Hen Lab
Chinese Pipeline
This project is roughly divided into three parts: OCR Recognition, which uses existing tools to extract captions from videos to text; Speech...
Contributor
Ahmed Ismail
Mentor
Ashry, mpac, Mark Turner
Organization
Red Hen Lab
Arabic Speech Recognition and Dialect Identification
The project proposed aims to implement an Arabic speech recognition model using training data from the MGB-3 Arabic datasets to perform speech...