CamIO

Camera Input-Output

Smart assitive devices providing real-time audio-feedback on objects using the smartphone’s sensors.
Research
Software Engineering
Python
Statistics
Human-Computer Interaction
Augmented Reality
Computer Vision
Consortium

Smith-Kettlewell Eye Research Institute (SKERI)

Began Around

March 1, 2018

Abstract

CamIO is a system to make physical objects (such as documents, maps, devices and 3D models) accessible to blind and visually impaired persons, by providing real-time audio feedback in response to the location on an object that the user is touching. CamIO currently works on iOS using the built-in camera and an inexpensive hand-held stylus made out of paper and cardboard or wood.

Homepage Article Code
Profile picture of Marc-Aurèle Rivière
NoteMy role in this project

1) Explore new solutions to improve the localisation & tracking capabilities of CamIO:

Their existing solution, iLocalize (Fusco & Coughlan, 2018) (Swift / iOS), used a combination of Visuo-Inertial Odometry (VIO) through Apple’s ARKit, particle filtering based on a simplified map of the environment, and drift-correction through visual identification of known landmarks (using a gradient boosting algorithm).

Screenshot showcasing the iLocalize app

I explored existing 3rd-party services for indoor localization, such as Indoor Atlas (which combines VIO, GPS data, WiFi & geomagnetic fingerprinting, dead-reckoning, and barometric readings for altitude changes), for which I made a small demo.

Illustration of the Indoor Atlas service at work
(a) Indoor Atlas’ localization
Illustration of the Indoor Atlas navigation graph
(b) Indoor Atlas’ navigation graph
Figure 1: Indoor Atlas

2) Developed a web app to send the live camera stream from a mobile phone (JavaScript / socket.io) to a backend server (Python / Flask). The goal of the application was to facilitate the exploration of new Computer Vision algorithms to process the captured video and IMU data, which would send back location or navigational information.

3) Assisted in analyzing the data from the first human experiments using CamIO (code).

4) Helped write a scientific paper on this project (Coughlan et al., 2020).

Summary

Back to top

References

Coughlan, J. M., Biggs, B., Rivière, M.-A., & Shen, H. (2020). An Audio-Based 3D Spatial Guidance AR System for Blind Users. In K. Miesenberger, R. Manduchi, M. Covarrubias Rodriguez, & P. Peňáz (Eds.), Computers Helping People with Special Needs (Vol. 12376, pp. 475–484). Springer International Publishing. https://doi.org/10.1007/978-3-030-58796-3_55
Fusco, G., & Coughlan, J. M. (2018). Indoor localization using computer vision and visual-inertial odometry (pp. 86–93). Springer International Publishing. https://doi.org/10.1007/978-3-319-94274-2_13