Malayalam Visual Genome

Malayalam Visual Genome (MVG) is a multimodal dataset consisting of text and images suitable for English-to-Malayalam multimodal machine translation task and multimodal research. We have selected short English segments (captions) from Visual Genome along with associated images and automatically translated them to Malayalam with manual post-editing, taking the associated images into account.

The training set contains 29K segments. Further 1K and 1.6K segments are provided in a development and test sets, respectively. Additionally, a challenge test set of 1400 segments was prepared for the multi-modal task. This challenge test set was created by searching for (particularly) ambiguous English words based on the embedding similarity and manually selecting those where the image helps to resolve the ambiguity. 

Dataset Sample

Sample items from the randomly selected segments (train/dtest/etest)
Image	Image_ID	X	Y	Width	Height	English Text	Malayalam Text
	2323457	20	150	325	121	Many giraffes at a zoo	ഒരു മൃഗശാലയിലെ നിരവധി ജിറാഫുകൾ
	2335684	61	191	437	182	Fruit stand outside market	ഫ്രൂട്ട് സ്റ്റാൻഡ് മാർക്കറ്റിന് പുറത്താണ്
Sample item from the challenge test set (chtest)
	2372733	26	107	152	218	The tennis court is made up of sand and dirt	ടെന്നീസ് കോർട്ട് മണലും അഴുക്കും ചേർന്നതാണ്

Sample ambiguous word: court (the key different meanings are court of justice vs. tennis court). As the image example illustrates, the word alone is ambiguous but already the English caption may contain sufficient information to disambiguate the word; the image is thus not always necessary for correct translation even in our challenge test set.

Download Link

•  Malayalam Visual Genome 1.0 (used in WAT 2021, WAT2022 Multi-Modal Task)