Sensory Substitution


The Sensory Substitution (SS) framework was initiated by Paul Bach-y-Rita in the sixties 1. The basic idea behind SS is to convey no longer accessible information about the environment through an unusual sensory channel. For example, the distance to surrounding objects through tactile feedback, when it is usually conveyed by vision (or sometimes audition).

The objective of those devices is to augment the user’s Umwelt, a word used to describe an individual’s or species specific perceptual environment. If the information provided was not innately accessible in healthy individuals, it is referred to as Sensory Augmentation.

Most existing systems focus on substituting vision by either touch or audition 2. Some well known examples are:

Type Example Summary
Vision to Audition The vOICe 3 Transcription of the camera’s feed into a soundscape of 60x60 sound sources, emitting all at once, row by row, in a 1 second left-to-right sweeping motion. Each of those sources is linked to an area of the camera’s field of view. A given source’s intensity is linked to the corresponding area’s luminosity. Its position on the horizontal axis is linked to the stereo disparity of the sound, and its elevation to the pitch of the sound.
Vision to Touch TVSS 1 Transcription of the camera’s feed into a 2D 20x20 matrix of pins placed against the users’ back. Each pin is linked to a corresponding area of the camera’s input: top-right area with top-right pin. The intensity of the pin’s vibration is linked to the average luminosity of that area.

The possibilities of SS:

Sensory Substitution has yielded interesting experimental results over the years, whether on its assistive and rehabilitative potential, or the insights it provides on the internal mechanics of how we perceive, integrate and make sense of sensory information.

Such results include:

  • Recognize and distinguish simple shapes. 4 5
  • Recognize and distinguish objects. 6
  • Localize objects in space. 7 8
  • Learn to draw. 9
  • Navigate inside a maze. 10
  • Drive a robot into a simple environment. 11

Furthermore, SS gives us an invaluable window into the still misunderstood ways our brain is able to adapt and learn to use new sensory information, and thus new interaction possibilities with its environment. Among others, studies with SS devices have shown that:

  • Some optical illusions are reproductible through SS. 12
  • Long-time users of The vOICe were able to perceive depth information through the device, despite the fact that it only provided “low-level” luminosity information. 13
  • After some training, some behavioural reflex start to appear, such as users protecting their head to brace for collision when an operator unwittingly changed the zoom level of the camera that was acting as sensor for a visuo-tactile substitution device. 14
  • Sensations become externalized only when the user can manipulate (move, rotate) the device: the stimulations start to be perceived as the product of external objects localized in the spatial surroundings of the user, and not as mere “random” stimulations when the person is able to link (through learning) their actions to the resulting changes in stimulations 15.

Sensorimotor Supplementation:

Based on the strong link between action and perception research on SS highlighted, some authors argued that SS should rather be called Sensorimotor Supplementation 7 to:

  • Better highlight the crucial role of action to disambiguate sensory information, and to externalize percepts.
  • Avoid misleading interpretation of the word substitution, since the information provided allows to supplement a set of task, but will not replace the lost modality, whether in terms of the possibilities it offers, or in the subjective feeling (qualia) it elicits in the user.

  1. Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221(5184), 963–964. ^
  2. Elmannai, W., & Elleithy, K. (2017). Sensor-Based Assistive Devices for Visually-Impaired People: Current Status, Challenges, and Future Directions. Sensors, 17(3), 565. ^
  3. Meijer, P. B. (1992). An experimental system for auditory image representations. Biomedical Engineering, IEEE Transactions On, 39(2), 112–121. ^
  4. Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221(5184), 963–964. ^
  5. Sampaio, E., Maris, S., & Bach-y-Rita, P. (2001). Brain plasticity: ‘visual’ acuity of blind persons via the tongue. Brain Research, 908(2), 204–207. ^
  6. Auvray, M., Hanneton, S., & O’Regan, J. K. (2007). Learning to perceive with a visuo – auditory substitution system: Localisation and object recognition with ‘The vOICe.’ Perception, 36(3), 416–430. ^
  7. Lenay, C., Gapenne, O., Hanneton, S., Marque, C., & Genouëlle, C. (2003). Sensory substitution: Limits and perspectives. Touching for Knowing, 275–292. ^
  8. Auvray, M., Hanneton, S., Lenay, C., & O’REGAN, K. (2005). There is something out there: distal attribution in sensory substitution, twenty years later. Journal of Integrative Neuroscience, 4(04), 505–521. ^
  9. Rovira, K., Gapenne, O., & Ammar, A. A. (2010). Learning to recognize shapes with a sensory substitution system: A longitudinal study with 4 non-sighted adolescents. In Development and Learning (ICDL), 2010 IEEE 9th International Conference on (pp. 1–6). IEEE. ^
  10. Stoll, C., Palluel-Germain, R., Fristot, V., Pellerin, D., Alleysson, D., & Graff, C. (2015). Navigating from a Depth Image Converted into Sound. Applied Bionics and Biomechanics, 2015, 1–9. ^
  11. Segond, H., Weiss, D., & Sampaio, E. (2005). Human Spatial Navigation via a Visuo-Tactile Sensory Substitution System. Perception, 34(10), 1231–1249. ^
  12. Renier, L., Collignon, O., Poirier, C., Tranduy, D., Vanlierde, A., Bol, A., … Devolder, A. (2005). Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. NeuroImage, 26(2), 573–580. ^
  13. Ward, J., & Meijer, P. (2010). Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and Cognition, 19(1), 492–500. ^
  14. Bach-y-Rita, P. (1972). Brain mechanisms in sensory substitution. New York: Academic Press. ^
  15. Auvray, M., Philipona, D., O’Regan, J. K., & Spence, C. (2007). The perception of space and form recognition in a simulated environment: The case of minimalist sensory-substitution devices. Perception, 36(12), 1736–1751. ^


comments powered by Disqus