ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition.

Vision and touch are two of the important sensing modalities for humans andthey offer complementary information for sensing the environment. Robots arealso envisioned to be of such multi-modal sensing ability. In this paper, wepropose a new fusion method named Deep Maximum Covariance Analysis (DMCA) tolearn a joint latent space for sharing features through vision and tactilesensing. The features of camera images and tactile data acquired from aGelSight sensor are learned by deep neural networks. But the learned featuresare of a high dimensionality and are redundant due to the differences in thetwo sensing modalities, which deteriorates the perception performance. To solvethis, the learned features are paired using maximum covariance analysis.Results of the algorithm on a newly collected dataset of paired visual andtactile data relating to cloth textures show that a good recognitionperformance of greater than 90% can be achieved by using the proposed DMCAframework. In addition, we find that the perception performance of eithervision or tactile sensing can be improved by employing the sharedrepresentation space, compared to learning from unimodal data.