Private / Local / Zero Cloud LiteRT-LMOllamaDockerized

Private Pocket Vision Relay

A distributed, privacy-first medical document analysis system. Your phone captures images or live video, sends them over Tailscale WireGuard to local compute nodes, where Gemma 4 analyzes them, RxNorm looks up drug interactions, and MedGemma provides specialist interpretation. All without a single byte leaving your private network.

Krishnendu Dasgupta · AXONVERTEX AI RESEARCH  |  Dr. Julia Hitzbleck · HI10x Innovation & Transformation GmbH

Infrastructure

Four Nodes, One Private Mesh

📱

iPhone 13 (RED)

4 GB RAM512 GBA15 Bionic12 MP CameraSafari UI
Camera + UI only (4 GB too low for Gemma 4)
🐳

Docker Host

16 GB RAM16 Core CPUUbuntuLiteRT-LMFastAPIPort 6733
Gemma 4 E2B + E4B on CPU via XNNPACK
⚡

Ollama Node A

GPUgemma4:e4b-it-q8_0Video VLMChat
Tailscale: 100.82.103.57:11434
🏥

Ollama Node B

GPUmedgemma1.5:4b-it-q8_0Medical Specialist
Tailscale: 100.99.149.21:11434
Interface

Three Modes, One Unified UI

📸Image
🎥Video
💬Chat
Pipeline

Vision + RxNorm + Medical Chain

01

📸 Capture iPhone / Browser

User photographs a prescription, holds a drug packet to the camera, or types a medical question. Three input paths: image upload, live webcam, or text chat with optional photo attachment.

↓
02

🔒 Tailscale + Cloudflared WireGuard / HTTPS

Data travels over encrypted WireGuard mesh. For Video tab, Cloudflared provides HTTPS tunnel (required for getUserMedia). All inference stays on private network.

↓
03

🐳 Docker: FastAPI Port 6733

Single containerized backend orchestrates all models. Images normalized to 800px JPEG. Video frames rate-limited. Chat maintains full session context including MedGemma responses.

↓
04

👁 Gemma 4 E4B / E2B LiteRT-LM CPU Ollama GPU

Image tab: LiteRT-LM on CPU via XNNPACK. Video + Chat: Ollama on Node A GPU. All paths produce structured analysis with multilingual OCR (Hindi, German, English), medication extraction, and clinical observations streamed via SSE.

↓
05

🧬 Medical Routing Auto-Detect

~80 medical keywords scanned across user prompt and Gemma 4 response. If medical content found, the pipeline chains forward. Works identically across image, video, and chat tabs.

↓
06

💊 RxNorm Agent NLM REST API

Extracts drug names via pattern matching + brand-to-generic mapping (Trental to Pentoxifylline, Lipitor to Atorvastatin). Four-strategy lookup: exact, approximate, brand map, drugs endpoint. Retrieves RxCUI identifiers and pairwise interactions.

↓
07

🏥 MedGemma 1.5 GPU / Ollama Node B

Receives triple context: original image (base64), Gemma 4 analysis, and RxNorm drug data. Provides clinical interpretation, interaction warnings, conditions, and next steps.

Real-World Example

German Clinical Prescription with Biomarkers

14. Mai 2026
+ Klinikum St. Elisabeth
Abteilung Innere Medizin / Diabetologie
Elisabethstr. 47, 80796 Munchen | Tel: 089 / 4400-2200
PatientHans-Peter Muller
Geb.12.03.1968 (58 J.) / M
KasseAOK Bayern / 4471829003
Klinische Befunde / Biomarker
RR:155/95
Puls:92/min
HbA1c:8,2%
Krea:1,4 mg/dL
eGFR:52 mL/min
Chol:268 mg/dL
LDL:178 mg/dL
Gew:94 kg
Diagnosen
Diabetes mellitus Typ 2 (E11.9)
Art. Hypertonie Grad 2 (I11.90)
Chron. Niereninsuff. St. 3a (N18.3)
Hyperlipoproteinamie (E78.0)
Verordnung
1.Metformin 1000mg 1-0-1
2.Ramipril 5mg 1-0-0 morgens
3.Amlodipin 5mg 0-0-1 abends
4.Atorvastatin 40mg 0-0-1 abends
5.ASS 100mg 1-0-0 morgens
Hinweise
Salzarme Diat. Blutzucker-Tagebuch fuhren.
Kontrolle 4 Wochen: Blutbild + HbA1c.
Nierenwerte verschlechtert? Sofort vorstellen.
Dr. K. Weber
Dr. med. Katharina Weber
Innere Medizin / Diabetologie
KLINIKUM
ST. ELISABETH
MUNCHEN
VISIONGemma 4 E4B via LiteRT-LM / CPU
Patient: Hans-Peter Muller, 58/M, AOK Bayern
Biomarkers: BP 155/95 (Grade 2 HTN), HbA1c 8.2% (poor control), eGFR 52 (CKD 3a), Krea 1.4, Chol 268, LDL 178
Diagnoses: Diabetes mellitus Typ 2 (E11.9), Arterielle Hypertonie Grad 2 (I11.90), Chron. Niereninsuffizienz St. 3a (N18.3), Hyperlipoproteinamie (E78.0)
Medications:
• Metformin 1000mg 1-0-1 (diabetes)
• Ramipril 5mg morgens (ACE-inhibitor, BP/kidney)
• Amlodipin 5mg abends (Ca-channel blocker, BP)
• Atorvastatin 40mg abends (statin, cholesterol)
• ASS 100mg morgens (cardiovascular protection)
Instructions: Low-salt diet, blood sugar diary, follow-up 4 weeks with HbA1c. Seek immediate care if kidney function worsens.
Doctor: Dr. med. Katharina Weber, Innere Medizin / Diabetologie
RXNORMNLM API / 5 drugs resolved
Metformin RxCUI: 6809 • Biguanide antidiabetic
⚠ Caution with eGFR 52: dose adjustment needed below eGFR 45
Ramipril RxCUI: 35296 • ACE inhibitor
Amlodipine RxCUI: 17767 • Ca-channel blocker
Atorvastatin RxCUI: 83367 • HMG-CoA reductase inhibitor
Aspirin (ASS) RxCUI: 1191 • Antiplatelet
Combo check: Ramipril + Metformin: monitor renal function. No critical pairwise interactions for this regimen.
MEDGEMMAMedGemma 1.5 4B / GPU Ollama
Assessment: 58-year-old male with multi-morbidity: poorly controlled Type 2 Diabetes (HbA1c 8.2%), Grade 2 Hypertension (155/95), CKD Stage 3a (eGFR 52), and significant hyperlipidemia (LDL 178).

Medication analysis: Five-drug regimen is guideline-concordant. Metformin appropriate at current eGFR but requires monitoring as eGFR approaches 45. Ramipril provides dual benefit: antihypertensive + renoprotective (critical given CKD). Amlodipine complements for BP target. Atorvastatin 40mg appropriate for cardiovascular risk profile. ASS 100mg for primary prevention given diabetes + hypertension.

Key concern: HbA1c 8.2% despite Metformin suggests treatment escalation may be needed (consider adding SGLT2 inhibitor which also benefits renal function). eGFR 52 requires 3-monthly monitoring.

Next steps: Follow-up bloods in 4 weeks per physician instructions. If HbA1c remains above 7.5%, discuss adding Empagliflozin (renal + glycemic benefit). Annual ophthalmology screening for diabetic retinopathy.
Under the Hood

Design Decisions

🧠

CPU + GPU Split

LiteRT-LM runs Gemma 4 on CPU inside Docker. Ollama runs Gemma 4 and MedGemma on GPU in separate containers. No memory contention.

Zero cloud inference
🔗

Agentic Pipeline

Medical detection triggers a three-stage chain: Gemma 4 analysis, RxNorm drug lookup, MedGemma interpretation. Each stage feeds context to the next.

RxNorm + MedGemma chain
💊

RxNorm 4-Strategy

Exact match, approximate term, brand-to-generic mapping (20+ brands), and drugs endpoint. Pairwise interaction checks for multi-drug prescriptions.

NLM REST API
🔒

Privacy by Design

Tailscale WireGuard mesh: no port forwarding, no public IP. Images deleted after inference. GDPR-aligned architecture for EU healthcare.

GDPR-ready
🐳

Dockerized

Single docker compose up. Persistent volumes for models (survive rebuilds). Ollama containers deployed separately on GPU nodes.

One-command deploy
🎥

Live Video VLM

Webcam frames every 8s to Ollama GPU. Rate-limited: skip if busy. Medical routing on video too. Cloudflared for HTTPS camera access.

Cloudflared tunnel