Contributor
Anshuman Singh

Refactor Multi-Task BERT


Mentors
Vasily Konovalov, Anton Peganov, Dmitry Karpov
Organization
DeepPavlov

Multi-task learning shares information between related tasks, reducing the number of parameters required. State of the art results across natural language understanding tasks in the GLUE benchmark has been previously used transfer learning from a large task: unsupervised training with BERT, where a separate BERT model was fine-tuned for each task.

In the current state of DeepPavlov, multi-task BERT is implemented in Tensorflow which need to be refactored such that DeepPavlov uses new frameworks such as PyTorch. The refactored code also needs to incorporate techniques such as PAL-BERT within the DeepPavlov library by matching the results of the GLUE benchmark on the respective techniques.