Knowing about the progress and performance of a model, as we train them, could be very helpful in understanding it’s learning process and makes it easier to debug and optimize them. It could also help us affirm the results from our models or inspect them in case of counterintuitive behaviors. Hence, I aim to provide a step-by-step process of visualizing training statistics for people, who want to keep a tab on learning processes of their gensim models, and want to optimize it through experimentation with hyperparameters.
The next phase of my project would aim to introduce and build a visualization module based on Gensim models and features. This would allow the interactive exploration of applications based on Gensim and would also enable users to do a qualitative assessment of their models and analyze the results. I aim to focus on implementing a visual framework for the exploration of topic models, taking cues from TMVE, pyLDAvis, and add more options to visualize data attributes and compare across the different topic models.
This work can naturally be extended to various other features in Gensim and to the upcoming ones, to have an associated visualization.