I work on AI systems for African markets, and one problem keeps showing up: the people who need AI assistance the most, in local languages, in low-connectivity regions, are the least likely to have cloud access or high-end hardware. So I built a fully offline, multilingual medical assistant app that runs entirely on a mid-range Android phone, no internet connection required after install.

This tutorial walks through how I did it: quantizing a 3B parameter model, running it on-device with llama.cpp, and getting inference times low enough to actually be usable. I'll also show the real numbers from deploying this on a TECNO CAMON 40 Pro, a phone that costs a fraction of a flagship device and is representative of what most people in low-resource regions actually own.

Table of Contents

• Why offline matters more than people assume

• The hardware and model constraints

• Step 1: Picking a base model

• Step 2: Quantizing with llama.cpp

• Step 3: Fine-tuning for the target languages

• Step 4: Packaging for Android

• Real benchmarks from a $200 device

• What I'd do differently

• Wrapping up

Why Offline Matters More Than People Assume

It's easy to design AI products around the assumption of a constant connection. In much of Sub-Saharan Africa, mobile data is expensive relative to income, network coverage is inconsistent outside urban centers, and even where connectivity exists, it can be unreliable during the exact moments someone needs help, like a rural health worker trying to get a quick answer during a consultation.

An offline model removes the dependency entirely. Once it's installed, it works in a forest, on a bus, during a network outage, anywhere. For applications like basic medical guidance, that reliability isn't a nice-to-have. It's the difference between the tool being useful and the tool being decorative.

The tradeoff is that you give up the size and power of cloud-hosted models. A model that runs offline on a phone has to be small enough to fit in memory and fast enough to respond in a reasonable time, on hardware with a fraction of the compute of a server GPU. That's the engineering problem this tutorial solves.

The Hardware and Model Constraints

Before picking a model, it helps to know what you're working with. A representative low-cost Android device has:

• 4 to 8 GB of RAM, shared with the OS and other apps

• No dedicated GPU worth relying on for inference

• A CPU that will throttle under sustained load

• Storage that's often limited, so a multi-gigabyte model isn't free

Given those constraints, a full-precision 7B or larger model is out of the question. You need something small enough to quantize down to a few gigabytes, and an inference engine built for CPU execution rather than GPU.

Step 1: Picking a Base Model

I used Llama 3.2 3B as the base model. At 3 billion parameters, it's small enough to quantize into something that fits comfortably on a phone, but still large enough to hold enough general language capability to be fine-tuned effectively for a specific domain and set of languages.

The choice of base model matters more than most tutorials let on. A smaller model needs to already be reasonably strong before you quantize and fine-tune it, because both steps degrade quality somewhat, and you don't have room to spare. Llama 3.2 3B was, at the time, one of the better tradeoffs between size and capability for this kind of constrained deployment.

Step 2: Quantizing With llama.cpp

llama.cpp is the tool that makes on-device inference realistic. It's a C/C++ implementation built specifically to run LLaMA-family models efficiently on CPUs, including mobile CPUs, and it supports quantization formats that shrink a model's memory footprint significantly with a controlled loss in precision.

Here's the basic flow for converting and quantizing a fine-tuned model:

# Convert the Hugging Face model to GGUF format
python convert_hf_to_gguf.py ./my-finetuned-model \
  --outfile model-f16.gguf \
  --outtype f16

# Quantize down to a smaller format (Q4_K_M is a good balance
# of size vs quality for this kind of deployment)
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Q4_K_M quantization cuts the model down to roughly a quarter of its original size, with a quality tradeoff that's noticeable in benchmarks but often not in practical use, especially for a narrow domain task like answering medical questions in a specific set of languages.

The quantization level is the main lever you have for balancing speed, size, and quality. If your target device has more RAM to spare, a higher-precision quantization like Q5 or Q6 will preserve more quality at the cost of size and speed. If you're targeting the lowest end of the hardware spectrum, Q4 or even Q3 quantization may be necessary.

Step 3: Fine-Tuning for the Target Languages

A generic base model, even a good one, won't reliably handle Yoruba, Hausa, Igbo, or Nigerian Pidgin out of the box. These languages are underrepresented in most pretraining datasets, so getting usable output means fine-tuning on data that actually covers them.

I used a two-stage fine-tuning approach:

1. Supervised fine-tuning (SFT) on curated question-answer pairs in the target languages, covering common medical queries and appropriate responses.

2. Direct Preference Optimization (DPO) on top of the SFT model, using preference pairs to push the model toward safer, more helpful responses and away from ones that were technically fluent but medically inappropriate or unclear.

This two-stage approach mattered more than I expected going in. SFT alone gets you a model that produces grammatically correct output in the target language, but DPO is what pushes it toward output that's actually useful and appropriately cautious for a medical context.

For reference, the SFT stage converged to a loss of 1.099, and the DPO stage achieved a reward margin of 18.40, meaning the model reliably preferred the better response in each preference pair by a wide margin. If your DPO reward margin is hovering close to zero, that's usually a sign your preference data isn't clearly differentiated enough, or your SFT checkpoint wasn't strong enough going in.

Step 4: Packaging for Android

Once you have a quantized GGUF file, the last step is wrapping it in something an Android app can call. llama.cpp exposes a C API that you can bind into a Java or Kotlin app through JNI, or you can use one of the existing Android wrapper projects that expose llama.cpp as a library.

The core things to get right at this stage:

• Load the model once at app startup, not on every query, since loading a multi-gigabyte file repeatedly will kill your response times.

• Keep the context window as small as your use case allows. Every extra token of context costs memory and inference time.

• Run inference on a background thread. A 3B model on a phone CPU is not instant, and blocking the UI thread will make the app feel broken even when it's working correctly.

• Bundle the model file with the app or make it a one-time download on first launch, since redownloading a multi-gigabyte file on every session defeats the purpose of an offline app.

Real Benchmarks From a $200 Device

Here's what this setup actually achieves running on a TECNO CAMON 40 Pro, a mid-range device representative of what's widely available and affordable in the markets this app targets:

62 seconds is not fast. It's not going to compete with a cloud-hosted API response. But for the use case, a fully offline assistant that works with zero connectivity, in a local language, on hardware people actually own, that latency is an acceptable tradeoff, not a failure. The alternative for a lot of users isn't a faster online model. It's no model at all.

What I'd Do Differently

A few things I'd change if I were starting this project today:

• Test quantization levels more systematically before committing. I settled on Q4_K_M partly on convention. A structured sweep across quantization levels, measuring both latency and task-specific accuracy, would have given a more defensible choice.

• Build the preference dataset earlier in the process. DPO made a bigger difference than I expected, and I'd allocate more time to curating high-quality preference pairs from the start rather than treating it as a later refinement step.

• Profile memory usage across a wider range of devices, not just the target device. What runs fine on 6GB of RAM can behave very differently on a 4GB device that's also running background apps.

Wrapping Up

Deploying an LLM offline on a low-end phone means working backwards from your hardware constraints instead of starting with the biggest model you can find. Pick a base model small enough to quantize without falling apart, use llama.cpp to get it running efficiently on CPU, fine-tune it deliberately for your target languages and domain, and measure your results on the actual device your users will have, not a developer laptop.

The result won't be as fast as a cloud API. But it will work where cloud APIs can't reach, and for a lot of the world, that's the