On-device Medical Chatbot for Nurse-Midwives in Zanzibar

• In Zanzibar, maternal and newborn mortality remain a significant challenge, disproportionately affecting rural households.
• Nurse-midwives frequently face complex cases, but accessible guidance at the point of care is severely limited.
• High data costs and poor network connectivity make online searches unreliable.

That's why I developed MAM-AI, an on-device AI assistant for nurse-midwives in Zanzibar, providing real-time, evidence-based, locally relevant guidance.

No internet needed. Always available.

You can also try it yourself in the live web demo — a faithful browser version of the on-device app, running the same Gemma 4 generator and guideline retrieval. Ask a clinical question and you'll get an evidence-based answer with clickable citations back to the source guidelines.

Note: the demo is hosted on a free Hugging Face server just for illustration, so generation may be slow. Real on-device latency is reported further below.

The app uses Retrieval-Augmented Generation (RAG): the nurse-midwife's question is embedded with EmbeddingGemma-300M, then matched against an on-device vector store to pull the three most relevant guideline passages. Those are injected into the prompt alongside the original question, and Gemma 4 E4B (int4) generates the final answer, based on the official guidelines, with inline citations back to the source.

Every stage — embedding, vector search, and generation — runs locally on the Android device. No internet is required at any point, which is the whole point: in Zanzibar, reliable connectivity can't be assumed at the point of care.

Both building blocks are off-the-shelf open models from Google — Gemma 4 E4B (int4) for generation and EmbeddingGemma-300M for retrieval — run on-device through Google's LiteRT-LM runtime. What's mine is the system, the knowledge base, and the evaluation built around them.

A clinical assistant needs comprehensive evaluation. I evaluated MAM-AI in layers — the end-to-end system first, then the retriever and the generator on their own — and how fast it actually runs on the device. The open-ended answers are scored by an LLM judge that I validated against thousands of physician-labeled judgments first, choosing a conservative grader over higher-scoring but lenient ones.

End-to-end: the deployed system

I look at the deployed system two ways — first the scores across models and prompts, then a single case showing how those numbers play out for one patient.

Quantitative results

The biggest gain came from fixing a failure mode. The base model was evasive — on a third of questions it deflected, telling the nurse to “see a doctor” instead of answering. A prompt redesign cut that to ~3% and doubled how much correct guidance the answers actually carry.

To see the deployed choice in context, I ran a full 3×3 matrix — the two deployable on-device models (Gemma 4 E4B and Gemma 3n E4B) plus a frontier model (Qwen3.5-397B-A17B-FP8) as an unconstrained ceiling, each under three prompts: baseline, +G1 (fixes deflection), and +G1+G2 (adds consultation structure).

Each cell is scored on two open-ended benchmarks: the Kenya Clinical Vignettes — 312 nurse-written primary-care cases, close to the Zanzibar setting — and HealthBench-oss, a split of OpenAI’s physician-rubric health benchmark. I introduce both in more detail further down; here they anchor the end-to-end scores.

Prompt variants (links are the actual system prompts) — baseline (plain system prompt), +G1 (seven deflection/scope-fix levers), and +G1+G2 (G1 plus a consultation-workflow structure). Kenya columns: 312 nurse-written clinical vignettes, safety-judged on a four-level scale. Potentially harmful = a small, often-catchable harm path (e.g. imprecise dosing) — a raw judge label, not human-verified; Dangerous = would harm if followed at face value, with the deployed config’s single flag manually adjudicated to 0 genuine. HealthBench-oss: 1,209 items, where weighted_met nets completeness (positive) against active harm (penalty).

The two on-device models trade off along a single axis — usefulness versus safety. Gemma 3n is the more useful one: it answers more (higher recall) and scores about twice as high on HealthBench. But it is the less safe one — 4 to 15 dangerous answers across the prompts, and a markedly higher rate of potentially-harmful ones. Gemma 4 is the mirror image: less complete, but with near-zero dangerous answers (0–1).

I prioritized safety and deployed Gemma 4; the +G1 prompt then wins back most of the usefulness it gives up — without adding dangerous answers.

Even with the right prompt, the deployed config still trails the unconstrained Qwen ceiling on most metrics — recall, completeness, and safety alike. That on-device gap is the real limit, and the next two layers dig into where it comes from: the retriever and the generator.

Case study: when phrasing decides the answer

One concrete example shows what these end-to-end scores feel like in practice. Below are two queries describing the same patient with the same condition — they differ only in wording. One calls it “severe nausea,” the other uses the clinical term “hyperemesis gravidarum.” That single change pulls back different chunks, and because the small generator faithfully mirrors what it retrieves, it produces two very different answers. (Source documents are anonymized as A–D, as in the paper; the same letter is the same document.)

Case A — the query uses the lay term “severe nausea.” The verbatim query:

“A 20-year-old woman, Gravida 1, currently in her second trimester of pregnancy with an estimated due date of 15 September, is suffering from severe nausea. What are the recommended treatment options?”

The three retrieved passages — the actual text the model received, lightly trimmed:

[1] SOURCE A · management section
If the woman is experiencing moderate vomiting or suspected hyperemesis gravidarum, the midwife should arrange for her to be admitted to hospital without delay … A urinalysis is conducted for ketones, bilirubin, protein and glucose, and a midstream specimen of urine is sent for culture to exclude pyelonephritis. Antihistamines such as promethazine hydrochloride, prochlorperazine (Stemetil), metoclopramide (Maxolon, Primperan) or corticosteroids such as methylprednisolone are considered safe to use in pregnancy and may be utilized to control the nausea and vomiting. Blood tests, such as AST, ALT, urea and electrolytes, are required to assess renal and liver function. An ultrasound scan may also be useful … the fetal heart rate should be appropriately auscultated to monitor the health of the fetus.

[2] SOURCE D · rationale ("why the committee made the recommendations")
… this guideline only covers treatments to manage nausea and vomiting in pregnancy, and comprehensive management of hyperemesis gravidarum, which may include nutritional interventions, is not covered by this guideline …

[3] SOURCE D · recommendations
1.4.1 Reassure women that mild to moderate nausea and vomiting are common in pregnancy, and are likely to resolve before 16 to 20 weeks. … 1.4.3 For pregnant women with mild-to-moderate nausea and vomiting who prefer a non-pharmacological option, suggest that they try ginger.

The generated answer — specific and actionable, citing the management chunk throughout:

Severe nausea in pregnancy needs careful management.
• Mild to moderate nausea: Reassure the woman that this is common. Suggest trying ginger as a non-drug option [3].
• Severe nausea (Hyperemesis Gravidarum): If the nausea is severe, she needs hospital care without delay [1].
Hospital Care Steps:
1. Monitor her weight, temperature, pulse, and blood pressure [1].
2. Check her fetal heart rate regularly [1].
3. Perform a urinalysis for ketones, bilirubin, protein, glucose [1].
4. Send a midstream urine specimen for culture [1].
5. Blood tests for liver and kidney function (AST, ALT, urea, …) [1].
6. An ultrasound scan may be used to rule out other causes [1].
Medication Options:
• Antihistamines like promethazine hydrochloride, prochlorperazine (Stemetil), metoclopramide (Maxolon, Primperan), or corticosteroids like methylprednisolone may be used to control symptoms [1].

Case B — the same patient, but the query uses the clinical term “hyperemesis gravidarum.” Only that phrase changed:

“A 20-year-old woman, Gravida 1, currently in her second trimester of pregnancy with an estimated due date of 15 September, is suffering from hyperemesis gravidarum. What are the recommended treatment options?”

Now the top-3 comes back weaker — one general passage, a student exercise, and a definition:

[1] SOURCE B · general management section
The woman is usually admitted to hospital for investigations and rehydration. The cause of vomiting may not be found. An antiemetic will be given and fluids and electrolytes replaced by intravenous infusion of a solution such as Hartmann's. Vitamin B12, thiamine, vitamin C, folic acid and iron will be needed … Moran & Taylor (2002) found that weight loss of more than 5% of prepregnancy weight in women with HG was effectively treated with 10 mg of prednisolone three times a day …

[2] SOURCE C · student study-exercise (no clinical content)
Activity
Find out what is meant by hyperemesis gravidarum.
See if you can discover what is the incidence and who is most at risk.
Find out how the midwife would recognize this condition, and what the treatment is.

[3] SOURCE A · definition section (same document as Case A's [1])
Hyperemesis gravidarum is a pathological condition characterized by unremitting, severe vomiting in pregnancy. It occurs in 0.3–2.0% of pregnancies and is more common in women who are younger, non-smokers and non-Caucasian. It is diagnosed by exclusion and is a leading cause of hospital admission during pregnancy …

The answer collapses to vague guidance — the antiemetic is never named, and the workup thins out:

Hyperemesis gravidarum is severe, continuous vomiting during pregnancy. For this woman, the recommended treatment options include:
• Hospital Admission: She usually needs to be admitted for tests and rehydration.
• Fluid and Electrolyte Replacement: Give intravenous fluids, such as Hartmann's solution.
• Medication: An antiemetic will be given.
• Supplements: Give Vitamin B12, thiamine, Vitamin C, folic acid, and iron.
• Monitoring: Watch her condition closely and keep fluid balance strict …
• Diet: Start oral fluids when vomiting stops for 24 hours …
If she loses more than 5% of her pre-pregnancy weight, a doctor may consider prednisolone [2].

Queries and chunks are lightly trimmed for length (full untrimmed transcripts are in Appendix B of the MAM-AI preprint). Bracketed numbers mark retrieval order and match the citations in each answer.

Retrieval

Next I isolate the retriever. I built mamaretrieval — a benchmark of 3,185 clinical queries with graded relevance labels — and ran seven retrievers through it, scored at the top-3 depth the app actually uses. (The benchmark itself is described further down.)

mamaretrieval: 3,185 clinical queries; each retriever’s top-3 chunks judged for relevance on a 0–6 scale by a strong LLM judge (Qwen3.5-397B). HR = hit rate (at least one relevant chunk in the top-3); P = precision (share of the top-3 that are relevant); ≥3 = lenient, ≥5 = strict relevance; wHR/wP weight by the graded score. EmbeddingGemma was scored in a matched follow-up run with the same judge and pooled labels — the six original retrievers reproduce exactly, confirming no drift.

The deployed on-device retriever, EmbeddingGemma-300M, lands in the top tier — third of seven, between two cloud models and within ~8 points of the best (voyage). A 300M model running on the device competes with cloud retrieval, and it outruns the earlier on-device option, Gecko, by about 30 points of precision (0.78 vs 0.48). So on-device retrieval is largely solved; the real question is whether that quality reaches the answers — and on a matched end-to-end comparison, it doesn't:

Matched bake-off on the Kenya set (retrieval n=312, HealthBench n=1,209). Retrieval P@3 = lenient precision at top-3; Kenya recall = end-to-end key-fact recall. For HealthBench-oss, weighted_met nets completeness (Positive ↑) against harm (Penalty ↓), so ≈0 means the generator’s gains are cancelled by its error rate. Different query set and judge from the benchmark above, so Gecko’s P@3 here (0.270) is not comparable to its benchmark P@3 (0.477). The no-RAG HealthBench arm is from a parallel run (same generator and judge, retrieval off).

On the matched Kenya comparison, the same upgrade lifts retrieval precision (Gecko 0.270 → EmbeddingGemma 0.396, +12.6 pp) but answer quality doesn't follow: end-to-end key-fact recall is flat (0.125 → 0.126) and HealthBench stays essentially zero. On this small on-device generator, RAG is even net-negative on Kenya — answering with no retrieved context scores higher (0.178). Better retrieval does not convert into better answers; the binding constraint is the generator’s ability to use the context, which the next layer examines directly.

Generator faithfulness

The end-to-end layer showed Gemma 4 is the safer choice; this layer isolates why. Faithfulness strips retrieval out of the picture: I hand each model the gold guideline passages for 2,989 questions (oracle context) and ask whether its answer stays within what those passages support. A failure is a contradiction — a wrong dose or threshold — or an unsupported addition: a clinical claim that isn’t in the source. Incompleteness and refusals don’t count.

Gemma 4 is about 2× more faithful than Gemma 3n, and the gap holds at every prompt: genuine hallucination runs 2.6–3.6% for Gemma 4 versus 6.3–6.7% for Gemma 3n. The telling part is the frontier comparison — Gemma 4 grounds about as faithfully as the unconstrained Qwen (2.5–3.7%). So on faithfulness the small on-device Gemma 4 is already strong; Gemma 3n is the outlier, not the small size.

Categorized true-hallucination, computed in two passes: a detector (Patronus Lynx-70B) flags every candidate faithfulness failure, then a stronger judge (GPT-5) re-reads each flag and keeps only genuine contradictions or unsupported additions — counted over all 2,989 answers. Measured on oracle (gold) context — the model is given the correct passages — so it reflects the generator’s grounding alone; in deployment the retriever can also return wrong or missing passages, a separate source of error.

These results both explain the deployment choice and point to future work. Gemma 3n’s weaker grounding — it contradicts the provided context nearly twice as often as Gemma 4 — is a property of the model, not the prompt, so improving faithfulness depends on the generator itself rather than on further prompt engineering. A possible next step is to fine-tune the generator to use retrieved context more faithfully — which could let a more capable, more helpful model be deployed without compromising safety.

On-device latency

Finally, how fast is it in practice? I benchmarked on the project's actual device — a OnePlus OPD2413 tablet (Snapdragon 8 Elite, 16 GB RAM, Android 15) — with 54 timed runs per configuration, measuring time-to-first-token, generation speed, and the full end-to-end answer.

The app uses the device's GPU where available and falls back to CPU otherwise, so both are part of the deployment — and CPU-only devices are roughly 2–3× slower. The numbers below are at the deployed depth (top-3 retrieval).

This drove the retrieval-depth choice. On GPU, latency is essentially flat in k — both first-token and generation time barely move — so depth is nearly free. On CPU it is not: generation stays flat, but first-token time climbs steeply with the number of retrieved chunks, and total latency crosses the project's 60-second budget by k=5 (≈61 s). Because not every target device has reliable GPU support, the CPU path has to stay within budget too — so I capped retrieval at k=3, which holds the CPU case to ≈43 s.

These are flagship-device numbers. Real Zanzibar hardware — lower-to-mid-range MediaTek phones — will be slower and is not yet measured; battery drain and sustained-load thermal throttling also remain to be profiled.

The system's answers are only as good as what it can retrieve, so the knowledge base is the foundation. It is a curated corpus of 87 authoritative clinical guideline documents — covering maternal, newborn, OBGYN, and reproductive health — processed into 63,650 passages and embedded for fully-offline on-device search.

The corpus draws on authoritative international guidelines, field references designed for low-resource settings, and Zanzibar/Tanzania national materials:

On the pipeline: PDFs are converted with an ML layout model (marker-pdf, which recovers tables and headings), then chunked structure-first — headings, not page breaks, define passage boundaries, and every chunk carries a parent-section breadcrumb so it stays self-contained when retrieved. Each chunk also gets a content-hash ID (a SHA-256 of its text) that is stable across edits and doubles as a citation key. The corpus ships as a versioned bundle with a SHA-256 checksum per document, so every retrieved passage is traceable to an exact source file. The full corpus-construction pipeline is open at mamai-medical-guidelines.

Here's one real passage from the corpus. The [SOURCE | PAGE | CID] header is what lets the app cite each answer back to an exact guideline page:

[SOURCE: WHO_Complications_2017 | PAGE: 204 | CID: c5c7acd564ff3c8b]
> Hypertensive disorders of pregnancy > Magnesium sulfate maintenance dose

Withhold or delay the drug if:
- respiratory rate falls below 16 breaths per minute;
- patellar reflexes are absent;
- urinary output falls below 30 mL per hour over the preceding four hours.

To measure whether the system actually helps, I needed a benchmark fit for the task — so I built mamabench, a normalized question-answering benchmark for maternal, newborn, OBGYN, and reproductive health, released openly on Hugging Face. It pulls 25,949 questions from seven medical datasets spanning Africa, India, and the USA; the Kenya and HealthBench tracks that anchor the evaluation above come from here.

Its two open-ended tracks are the ones that anchor the end-to-end evaluation. The Kenya Clinical Vignettes are nurse-written Kenyan primary-care scenarios — maternal, neonatal, child, and reproductive health — paired with clinician reference answers, close to the Zanzibar setting. HealthBench-oss is the OBGYN slice of OpenAI's HealthBench, where each question carries weighted, physician-written rubric criteria that a judge scores the model's answer against.

Most of the work is curation. Every source is normalized into one schema with content-hash IDs and a full provenance manifest, and the open-ended sources are scope-filtered by an LLM classifier (Qwen3.6-27B) that sorts each question into one of five categories — maternal, neonatal, child health, reproductive health, or out-of-scope — and drops the off-topic ones. I validated that classifier two ways: against a much larger model (Qwen3.5-397B, 98% agreement on the HealthBench subset), and against Kenya's existing Gemini-labelled categories (87%). Its full reasoning is saved for every question, and the seven prompts it couldn't converge on are documented rather than quietly dropped.

The benchmark also bundles an OBGYN slice of HealthBench's physician-labelled grader meta-eval (6,853 judgments) — the calibration data used to validate the answer-quality judge in the evaluation above. The full construction code is open at mamabench.

The retrieval scoreboard earlier runs on mamaretrieval, a retrieval benchmark I built for medical RAG and released openly on Hugging Face. It pairs 3,185 clinical questions with 230,964 graded relevance labels across the top-20 results of six retrievers — enough to rank retrievers at the top-3 depth the app actually uses.

No human-labelled retrieval set exists for this domain, so I built one. For each clinically-useful chunk in the guideline corpus, an LLM wrote a short question that the chunk could answer; each question was run through all six retrievers, their top-20 results pooled (~72 candidate chunks per question), and every (question, chunk) pair scored for relevance by a strong LLM judge.

Relevance isn't treated as yes/no. The judge grades each pair on four dimensions, combined as score = D1 × (D2 + D3 + D4), from 0 to 6:

The judge (Qwen3.5-397B) was calibrated against Claude Opus 4.7 reference labels — 95% agreement on whether a chunk clears the relevance bar. Two honest caveats, though: it is a single LLM judge, not a human gold standard; and because the questions are themselves LLM-written from corpus chunks, they tend to flatter dense retrievers. Both mean the retriever-vs-retriever ranking is the trustworthy signal, not the absolute scores. The judge's full reasoning on every pair is shipped with the dataset for auditing.

I've been honest about the gaps throughout — here are the ones that matter most. MAM-AI is a thoroughly-evaluated research prototype, not a deployed product.

• No user testing yet. Despite five weeks on site, a lag in project funding meant the planned field test with nurse-midwives couldn't go ahead this cycle — so there's no real-world data yet on how the system performs at the point of care.
• Not fully verified by clinicians. The system was tested thoroughly and shaped by clinician input — the fix for over-deflection came directly from that feedback — but its clinical correctness has not yet been fully verified by clinicians, and the evaluation relies on LLM judges rather than a human gold standard.
• English only. The deployment context — Zanzibar and mainland Tanzania — is Swahili-speaking, but the system, knowledge base, and evaluation are entirely in English. Swahili support and evaluation remain a deployment gap.
• RAG doesn't yet pay off on the small generator. Better retrieval doesn't reach the answers, and adding retrieved context is net-neutral-to-negative — the generator can't fully use what it's given. Teaching it to (generator-side RAG-grounding) is the central open problem, not a solved one.
• Latency measured on a single flagship tablet. The cheaper MediaTek phones that make up most of the target market are untested and will be slower; battery drain and sustained thermal throttling in a warm ward are not yet profiled.

MAM-AI shows that a fully offline, source-cited medical assistant can run on an Android device — and evaluating it layer by layer shows the potential improvement directions. On-device retrieval is essentially solved: a 300M embedder rivals cloud retrievers, so the right guideline passage is usually found. The real bottleneck is the small generator. At 4B it can't be both maximally helpful and safe, so I deployed the model that grounds its answers most faithfully and won back usefulness through the prompt — a frontier model escapes that trade-off, which tells me the gap is a limit of small-model capacity, not of the task.

Several directions could close that gap. A potential next step is to finetune the generator to use retrieved context more faithfully, so that better retrieval translates into better answers. Improving the coverage and quality of the knowledge base matters too — because a small model may carry little medical knowledge of its own, the corpus is decisive: where the guidelines hold the right passage the answer is specific and safe; where they don't, it goes vague. Query rewriting before retrieval is another promising direction, especially for solving noisy queries and supporting other languages.

Evaluating it rigorously also meant building test data that didn't exist before — two open benchmarks, mamabench and mamaretrieval, which I released on their own for maternal-health RAG. MAM-AI is a thoroughly evaluated research prototype, not a fielded product. I sincerely hope this work could be tested with the local nurse-midwives and be helpful to their work in the future.

An early prototype of MAM-AI was developed to demonstrate the fully offline RAG concept using eight guideline documents and submitted to Google's Gemma 3n Impact Challenge (LiGHT MAM-AI — T. Brokowski, C. Marquard, M. Z. Moonsamy, F. Dawson, and F. Elhassan, 2025). My work turns that demonstration into an evaluated and easily deployed system grounded in a more comprehensive corpus. I expand the corpus to 87 guideline documents (63,650 passages) and version it as checksum-pinned bundles. I build and release two purpose-made benchmarks (mamabench and mamaretrieval), and use them to evaluate every layer with validated LLM judges: end to end, the retriever and generator each in isolation, and latency on device. Guided by that evidence, I upgrade each stage of the deployed pipeline — a different generator chosen for faithfulness, a stronger embedder, and a redesigned system prompt. The install is now self-serve: fully offline by sideloading for users with low connectivity, or a fully automated online download from stable sources for users with good connectivity.