Hindi Visual Genome is a multimodal dataset consisting of text and images suitable for English-to-Hindi multimodal machine translation task and multimodal research.
The short English segments (captions) were randomly selected from Visual Genome along with associated images. We automatically translated the captions to Hindi and manually corrected the translations, taking the associated images into account.
The training set contains 30K segments and it is accompanied with a development set (dtest, 1K segments) and evaluation set (etest, 1K segments) from the same distribution.
Additionally, we prepared a challenge test set of 1400 segments (chtest). This test set was created by searching for particularly ambiguous English words based on the embedding similarity and manually selecting those where the image helps to resolve the ambiguity. The Hindi translation of the English captions were manually corrected in the same way as the main data sets.
Sample items from the randomly selected segments (train/dtest/etest)
|Image||Image_ID||X||Y||Width||Height||English Text||Hindi Text|
|2323457||20||150||325||121||Many giraffes at a zoo||एक चिड़ियाघर में कई जिराफ़|
|2335684||61||191||437||182||Fruit stand outside market||बाजार के बाहर फल स्टैंड|
Sample item from the challenge test set (chtest)
|2372733||26||107||152||218||The tennis court is made up of sand and dirt||य़ह टेनिस कोर्ट रेत और धूल से बना है|
Sample ambiguous word: court (the key different meanings are court of justice vs. tennis court). As the image example illustrates, the word alone is ambiguous but already the English caption may contain sufficient information to disambiguate the word; the image is thus not always necessary for correct translation even in our challenge test set.
Hindi Visual Genome now goes through the final verification and it will be released in Spring 2019; free of charge for non-commercial and research purposes.