DONDO is accurate but too large to deploy
DONDO is trained on Ghanaian languages, and on Twi and Ewe it does better than the big labs' multilingual models. The version I started from is a 605.8M-parameter, 24-layer Conformer (Wav2Vec2-BERT): about 2.4 GB in fp32, 1.2 GB in half precision.
That's too big for a mid-range Android phone, which is where it would get used. So the question was how much smaller it could get while keeping most of the accuracy. The model already works, so this is a compression problem, and the training is already done.
Dynamic int8 quantization does nothing here
The reflex for making a model smaller is int8 quantization, so I ran PyTorch's
quantize_dynamic on the parent. It
did almost nothing: about 1.0× speed, no real change in size or accuracy.
The reason is the architecture. DONDO is a Conformer, and dynamic quantization only touches the
nn.Linear layers. The work in this
model happens in the convolutional feature extractor, the conformer conv modules, and attention,
none of which that method quantizes. I was quantizing the parts that don't cost anything.
Naive dynamic int8 doesn't help a Conformer ASR model. Getting real int8 here needs a runtime that also quantizes conv and attention, like ONNX Runtime or ExecuTorch. More on that below.
The exercise did surface the real constraint, and it was memory.
Latency was never the problem
When I measured plain inference speed, the parent was already faster than real time on a CPU. Real-time factor is compute-seconds per audio-second, so anything under 1.0 means it transcribes faster than you can speak.
| Parent DONDO · 605.8M | GPU RTF | CPU RTF · 4 threads |
|---|---|---|
| inference speed | 0.0036 (~280×) | 0.25 (~4× realtime) |
So making it faster solved a problem I didn't have. The real constraint on a phone is memory, since a 1.2 GB model won't load comfortably in phone-class RAM. Lowering the bit-width barely moved that footprint, so the obvious move was to drop layers.
Cut depth, warm-start from the parent
Most of the parameters live in those 24 layers, so I built a shallower student and warm-started it. I initialized it from evenly-spaced parent layers (going from 24 to 6 takes layers 0, 5, 9, 14, 18, 23), copied the feature projection, adapter and CTC head straight over, then fine-tuned on WAXAL Twi and Ewe. Three depths give a clear picture.
All three students were trained with a distillation loss from the parent, and they beat the zero-shot parent (0.512) by a wide margin. Before crediting distillation for that, I checked whether the distillation term was what helped.
Does the distillation loss help?
The students were trained with two losses: CTC on the labels, and a distillation term that pulls the student's logits toward the parent's. The natural assumption is that the distillation term is doing the work. To check, I retrained the same students with CTC alone and compared.
So the credit belongs to the warm start and the in-domain fine-tuning. The teacher's logits add nothing here, and at L12 they hold the student back. When the teacher is worse on the target data than the labels themselves, matching its outputs just teaches the student the same mistakes.
A fair upper bound
The students trained on WAXAL, but the parent I compared against was zero-shot, which isn't fair. So I trained the full 24-layer parent on the same data (CTC, no distillation). Now it's an apples-to-apples comparison.
| Model | Params | Disk fp32 | CPU RTF | Blended | vs full |
|---|---|---|---|---|---|
| Fine-tuned parent | 605.8M | 2423 MB | 0.252 | 0.216 | — |
| Nano-L12 pick this | 315.6M | 1262 MB | 0.132 | 0.261 | +0.045 |
| Nano-L6 | 170.5M | 682 MB | 0.069 | 0.387 | +0.171 |
| Nano-L3 | 98.0M | 392 MB | 0.036 | ~0.57 | +0.35 |
Halving the model, from 606M to 316M, costs 0.045 blended error. Cutting to a quarter (170M) costs a lot more, and an eighth (98M) falls apart. L12 is the one I'd ship. As an external reference, Meta's MMS-1b (~965M) scores about 0.37 on these clips zero-shot. The fine-tuned 316M nano beats it, though that comparison favors the nano, which saw in-domain data while MMS did not.
The nano next to its teacher
Here is the 316M nano next to the zero-shot parent on held-out clips, about 15 seconds each. The nano was trained on WAXAL, the parent wasn't. Reference is the ground-truth transcript. The nano mostly gets the right words and spelling; the parent lands close phonetically but garbles them.
How small it gets
The disk numbers above are fp32, the format the models are saved in. What you'd ship is smaller, and since latency is already fine, the footprint is what decides whether a phone can load the model at all.
Torch's dynamic quant left the Conformer untouched, so I ran Nano-L12 through ONNX Runtime, which quantizes the conv and attention matmuls too. That shrank it from 1262 MB to 319 MB, about 4×, and cost almost nothing in accuracy (0.266 to 0.269 blended in the same runtime). It didn't run faster on this CPU, though. Dynamic int8 adds quantize and dequantize steps around each op, and the model was already faster than real time, so there was nothing to gain on speed. The footprint is what matters on device, and int8 roughly halves it again from fp16.
Is any of this new?
The method isn't new. Warm-starting a shallower student from a subset of a teacher's layers and fine-tuning is the idea behind DistilBERT and, for speech, DistilHuBERT. What's here is the applied result: a size-accuracy curve for DONDO on Ghanaian languages, a 319 MB model that runs on a phone and beats zero-shot MMS-1b on Twi and Ewe, and two documented failure modes, that naive int8 is a no-op on a Conformer and that an out-of-domain teacher makes distillation worse than plain fine-tuning.
The recipe, in one paragraph
Take a good but large Conformer ASR model. Don't quantize it naively, since that does nothing on this architecture, and don't distill from it if it's out of domain, since the soft targets are noise. Build a shallower copy, warm-start it from evenly-spaced layers of the original plus the feature and CTC heads, and fine-tune on your target-language data. You get a clear size-versus-accuracy curve. Pick the bend, which here is half depth, and you keep most of the accuracy at half the size.
What I haven't shown
On-device numbers on real ARM hardware: everything here is server CPU, and a phone SoC behaves differently. More training data: the students saw about 6k clips, the WAXAL unlabeled pool is much larger, and closing the L12-to-L24 gap (0.261 to 0.216) is probably a data problem rather than an architecture one.