Please select the desired project time frame:

• July 2026
• January 2026
• July 2025
• January 2025
• July 2024
• January 2024
• July 2023
• January 2023
• July 2022
• January 2022
• July 2021
• January 2021
• July 2020
• January 2020
• July 2019
• January 2019
• July 2018
• January 2018
• July 2017
• January 2017
• July 2016
• January 2016
• July 2015
• January 2015
• July 2014
• January 2014
• July 2013
• January 2013
• July 2012
• January 2012
• July 2011
• January 2011
• July 2010
• January 2010
• July 2009
• January 2009
• July 2008
• January 2008
• July 2007
• January 2007
• July 2006
• January 2006
• July 2005
• January 2005
• July 2004
• January 2004
• July 2003
• January 2003
• July 2002
• January 2002
• July 2001
• January 2001

Start of funding 01.01.2024

Investigating the Applications of Spatial Computing for Healthcare and Accessibility

---

Augmenting the real world using spatial computing and real-time camera-based systems provides tremendous opportunities for patients and medical professionals. In particular, advances in consumer hardware allow for effective digital content augmentation of the physical world. We aim to utilize these innovations in developing software for medical-grade systems, enhancing real-time applications for healthcare and visually impaired users. Surgeons can leverage real-time AR for complex surgeries, while visually impaired patients can use features like zoom, optical character recognition, and machine learning detections for sensory enhancement. Our project aims to enhance Stanford's Spezi digital health ecosystem, incorporating spatial computing for real-time context and visual support. We will validate these new foundational modules using the latest consumer hardware in controlled studies, focusing on applications for surgeons and patients with low vision.

Final report:
Spatial computing describes a class of interactive systems that tightly integrate digital information with the physical environment by combining technologies from augmented and virtual reality. This integration enables users to perceive and manipulate virtual content directly within their physical surroundings with a spatial computing headset. Building on the recent mainstream availability of spatial computing devices, we explored how spatial computing can support accessibility use cases. We focused on individuals with low vision and examined the barriers that visual impairments introduce when accessing physical content such as books, blackboards, or posters. Through a continuous, spatially anchored accessible representation of the environment, the "Spatial Continuity" project aimed to reduce the friction introduced by digital assistive technologies when making physical objects accessible.

We developed a proof-of-concept system in which a smartphone acts as a handheld magnification device, streaming a live camera feed to a spatial computing headset. Through the headset, users can view an enlarged and resizable representation of their surroundings and capture screenshots to generate spoken descriptions using an online large language model. We evaluated the prototype in a small user study conducted in collaboration with the Bavarian Association for Blind People and People with Low Vision (Bayerischer Blinden- und Sehbehindertenbund e.V.). Participants reported that the large, spatially anchored view substantially improved interaction with physical objects, while also expressing a clear preference for more natural, voice-based interactions that directly reference content within the camera’s field of view. These findings point toward the potential of hands-free interaction paradigms enabled by privacy-preserving, front-facing camera APIs for spatial computing devices, which could allow users with low vision to access and read physical materials using a head-worn device only.

The results of this work were published at the 27th International ACM SIGACCESS Conference on Computers and Accessibility (https://doi.org/10.1145/3663547.3759711). Beyond its research contribution, the project produced several technical artifacts, including open-source contributions to Apple’s Swift OpenAPI Generator and to Stanford Spezi, an open-source ecosystem of Swift modules for digital health applications. The Spatial Continuity source code is publicly available at https://github.com/StanfordBDHG/SpatialContinuity.

We thank BaCaTeC for supporting this collaboration between California and Bavaria; the funding enabled a four-week research stay for a TUM graduate student at Stanford University and was instrumental in advancing the project and strengthening academic ties between the two institutions.