Funded Projects

Start of funding 01.01.2024

Fog Computing-Based Execution of Large Language Models for Digital Health and Education

Prof. Dr. Stefan Krusche
Technische Universität München
Informatics 1 – Chair of Distributed Systems & Operating Systems

Prof. Dr. Oliver Oppers Aalami
Stanford University
Byers Center for Biodesign

Stanford’s HealthGPT and LLMonFHIR demonstrate that Large Language Models (LLMs) are able to increase patients' health literacy by querying their personal health records via natural language and providing context around them. Given the centralized, opaque nature of cloud LLM providers and the sensitivity of personal health data, the usage of these remote services results in substantial privacy, trust, and security but also financial concerns. Therefore, the team explores shifting the LLM execution environment from cloud providers closer to the patient's device using local execution or dynamic Fog Computing architectures. Executing open-source LLMs in more trusted environments, like the patient's edge device or in the fog layer represented by the local network (e.g., within a clinic or a university in a lecture setting), is hypothesized to help mitigate privacy and resources challenges posed by cloud-based LLMs. Our main goal is to develop a reusable and privacy-preserving architecture for integrating LLMs into existing and emerging infrastructures, with a focus on enhancing education and health literacy. Additionally, we are exploring effective mechanisms to multiplex LLM-based requests within a Fog Computing-based architecture, establishing a trusted execution environment that utilizes context provided by users, specifically patients in medical settings and students in educational environments.

Final report:
Large Language Models (LLMs) offer significant potential to enhance patient health literacy by providing context for complex medical data. However, their reliance on centralized cloud providers for processing Protected Health Information (PHI) raises critical privacy, trust, and cost concerns. To address these barriers, our project explored shifting the LLM execution environment from opaque cloud services to a decentralized, dynamic fog computing architecture. By leveraging open-weight models in trusted environments, such as a patient's edge device or a local fog layer within a clinic, we aimed to mitigate the privacy and cost constraints associated with cloud-based AI. Consequently, our primary objective was to determine the necessary software mechanisms to distribute LLM inference tasks across this decentralized system to enhance privacy, trust, and cost efficiency in digital health and other sensitive domains.

To achieve this, we developed SpeziLLM (https://github.com/StanfordSpezi/SpeziLLM), an open-source framework embedded within the Stanford Spezi software ecosystem (https://github.com/StanfordSpezi). SpeziLLM serves as a unified, LLM-agnostic interface that transparently orchestrates inference tasks across three distinct layers: the Edge, the Fog, and the Cloud. Instead of burdening developers with complex infrastructure management and the computational heavy lifting of LLM inference, the framework dynamically discovers local inference services or leverages hardware-accelerated execution on mobile devices for optimized performance. We demonstrated the framework's versatility by integrating it into six digital health applications, including LLMonFHIR (https://github.com/StanfordBDHG/LLMonFHIR) and HealthGPT (https://github.com/StanfordBDHG/HealthGPT). These case studies highlighted SpeziLLM’s ability to route sensitive, lower-complexity tasks (such as summarizing records) to local or fog environments while reserving cloud resources for complex reasoning, thereby optimizing the balance between privacy, performance, and cost.

The results of this work were published in the paper "Dynamic Fog Computing for Enhanced LLM Execution in Medical Applications" at the IEEE/ACM Conference on Connected Health (CHASE 2025) (https://www.sciencedirect.com/science/article/pii/S2352648325000388). As a modular, open-source artifact, SpeziLLM is now accessible to institutions worldwide facilitating hands-on privacy-preserving AI development for students and researchers alike.

We thank BaCaTeC for supporting this collaboration between California and Bavaria; the funding was instrumental in advancing the development of SpeziLLM and strengthening the academic ties between the Technical University of Munich and Stanford University. This support enabled the exchange of expertise necessary to bridge the gap between advanced AI computing and privacy-preserving medical applications.