Malayalam Visual Genome (MVG) is a multimodal dataset consisting of text and images suitable for English-to-Malayalam multimodal machine translation task and multimodal research. We have selected short English segments (captions) from Visual Genome along with associated images and automatically translated them to Malayalam with manual post-editing, taking the associated images into account.

The training set contains 29K segments. Further 1K and 1.6K segments are provided in a development and test sets, respectively. Additionally, a challenge test set of 1400 segments was prepared for the multi-modal task. This challenge test set was created by searching for (particularly) ambiguous English words based on the embedding similarity and manually selecting those where the image helps to resolve the ambiguity. 

Dataset Sample

Sample items from the randomly selected segments (train/dtest/etest)

Image Image_ID X Y Width Height English Text Malayalam Text
2323457 2323457 20 150 325 121 Many giraffes at a zoo
ഒരു മൃഗശാലയിൽ നിരവധി ജിറാഫുകൾ
2335684 2335684 61 191 437 182 Fruit stand outside market
ഫ്രൂട്ട് സ്റ്റാൻഡ് മാർക്കറ്റിന് പുറത്താണ്

Sample item from the challenge test set (chtest)

2372733 2372733 26 107 152 218 The tennis court is made up of sand and dirt
മണലും അഴുക്കും ചേർന്നതാണ് ടെന്നീസ് കോർട്ട്



Download Link

Will be updated soon.



How to cite

If you use this corpus, please cite the following paper:

  title={Hindi Visual Genome: A Dataset for Multi-Modal English to Hindi Machine Translation},
  author={Parida, Shantipriya and Bojar, Ond{\v{r}}ej and Dash, Satya Ranjan},
  journal={Computaci{\'o}n y Sistemas},