SUNSPOT Dataset

We introduce a new dataset, SUN-Spot, for localizing objects using spatial referring expressions (REs). SUN-Spot is the only RE dataset which uses RGB-D images. It also contains a greater average number of spatial prepositions and more cluttered scenes than previous RE datasets. Using a simple baseline, we show that including a depth channel in RE models can improve performance on both generation and comprehension.

Paper

Preview of Pages 1 and 2 Download the paper here.

Citation

C. Mauceri, M. Palmer, and C. Heckman, “SUN-Spot: An RGB-D Dataset With Spatial Referring Expressions,” in International Conference on Computer Vision Workshop on Closing the Loop Between Vision and Language , 2019.

@inproceedings{Mauceri2019,
author = {Mauceri, Cecilia and Palmer, Martha and Heckman, Christoffer},
booktitle = {International Conference on Computer Vision Workshop on Closing the Loop Between Vision and Language},
title = {{SUN-Spot: An RGB-D Dataset With Spatial Referring Expressions}},
year = {2019}
}

Downloads

Examples

To provide a taste of the images and annotations in SUN-Spot, here are 10 randomly selected objects from the dataset with all of their referring expressions.

pitcher

  • the item sitting to the left of the sink with the water in it. used to pour water into cups and store water. sitting in front of the window.

  • the pitcher is on the counter , to the left of the sink , in front of the mirror.

  • the pitcher is to the left of the sink in front of the window

alarm

  • the white alarm is on the nightstand and close to the bed.

candle

  • the candlestick is on the left under the mirror.

cdplayer

  • cd player is next to the stacked books

bag

  • the bag is on a table in front of a mirror and near two paintings

  • the bag sits at the left end of the table that is below the mirror.

  • this bag is on the table farthest to the right. it’s red and white.

  • the red bag on the table

circuitbreaker

  • the circuit breaker is on a wall to the right of the red window shades.

plate

  • plate on the table , under and to the right of the clock on the wall

  • the plate is on a table centered between two maroon chairs with another white plate to the right.

tissue

  • the tissue is sitting on top of the toilet.

folders

  • the yellow folders are underneath the white papers on top of the counter.

  • the folders in the tray on the right side next to the other tray

lunch_box

  • the lunch box is front of the black box and to the right of the cylinder on the right hand corner of the desktop.