Desert Ant models on a MacBook Air M5

SDK desert-ant-core 4.1.0 · Apple M5 · 32 GB · macOS 26.6.2 · public data: FLEURS, AMI (CC BY 4.0)

This independent benchmark measures seven models on public data and authored synthetic controls so the results can be inspected and reproduced. Every number below comes from results/*.json; the core stages re-run with uv run python scripts/run.py all.

Speech recognition: Voz vs Whisper large-v3

100 FLEURS test sentences per language, one recording per sentence. Both models get the language code. Whisper is whisper.cpp 1.9.1, large-v3, 8 threads, Metal, default decoding, no prompt. Voz is v0.4.0 through the Swift SDK. Each model loads once per language; load time is excluded and listed below. WER uses the same normalisation for both models (case, punctuation, ё→е); numbers are left as written, which costs both models alike.

Timing in this block was measured on AC power with the CPU 83.77% idle before the stage (gate: AC and ≥85%). Accuracy numbers are unaffected; the controlled Clear stage below shows the same speed ratio range.

set	clips	audio, min	WER Voz	WER Whisper	Voz × realtime	Whisper × realtime	Voz faster by
en	100	17.6	5.76%	5.03%	200.4×	4.8×	41.4×
ru	100	20.4	7.09%	4.92%	175.5×	3.5×	50.1×
es	100	21.8	3.93%	2.8%	155.1×	3.4×	45.7×

Model load: Voz en 465 ms, ru 483 ms, es 473 ms; Whisper en 720 ms, ru 678 ms, es 774 ms.

Read the speed column with care. Whisper pads every input to a 30 s window, so on ~10 s clips most of its encoder work is padding and the ratio above overstates the gap. The fair speed number is the continuous-audio test below.

Continuous audio: 60 s of an AMI meeting

AMI ES2004a 300-360 s, 16 kHz mono. Three fresh processes per engine, alternating, medians. Raw runs are preserved in results/continuous.json.

	Whisper, ms	Voz, ms	Voz faster by
whole call: process start + load + transcribe	5951	651	9.1×
transcription only	5317	190	28.0×

Clear on noisy speech

The 100 English clips with seeded white noise at 5 dB SNR, then Clear (studio, mastering bypass, 16 kHz requested), then both recognizers. Clean-speech WER for the same clips is in the table above.

set	clips	audio, min	WER Voz	WER Whisper	Voz × realtime	Whisper × realtime	Voz faster by
en-noisy	100	17.6	13.13%	11.68%	157.9×	3.1×	50.2×
en-noisy+clear	100	17.6	48.53%	44.61%	140.5×	3.0×	46.4×

Sample rate. Requested 16000 Hz; reported 48000 Hz; file headers: 1 ch, 48000 Hz, Int16. See finding 1.

Clear median per ~10 s clip: 32 ms.

Control: level-normalised input

Many FLEURS recordings are very quiet (RMS around −60 dBFS). To separate level from noise, the same 100 clips were normalised to −23 dBFS RMS first, then mixed with noise, then run through Voz with and without Clear.

input	Voz WER	Voz WER after Clear
clean speech	5.84%	6.06%
white noise, 15 dB SNR	7.16%	13.18%
white noise, 5 dB SNR	13.18%	36.25%

Clear leaves clean speech almost untouched, so the model is not broken. On broadband noise it costs the recognizer accuracy, and more so on quiet input. Clear is documented as a listening-quality model, not an ASR front end, and white noise is a synthetic stress test, not one of the rooms it is built for. My takeaway for an app: keep Clear on the playback path and feed the recognizer the original audio, unless a real-noise test says otherwise.

Diarization: WHO on AMI meetings

Two AMI test meetings, Mix-Headset audio, reference and scoring region from pyannote/AMI-diarization-setup (only_words). DER from pyannote.metrics, overlap scored, with no collar and with a ±250 ms collar. Three runs in one loaded process.

meeting	min	speakers found / ref	DER, no collar	DER, 250 ms collar	runs, ms	same output
ES2004a	17.5	4 / 4	16.98%	10.62%	1343, 1264, 1269	yes
IS1009a	14.0	4 / 4	22.19%	16.39%	1056, 1213, 1223	yes

Model load in this run: 215 ms (weights already cached). The first-ever load on this machine, with download and Neural Engine specialization, took 329 s in an earlier run; it is excluded.

Language ID: Ear

All 300 clean FLEURS clips (~10 s each). Top answer right: 294 / 300. Wrong but marked reliable: 3. Median 31 ms per clip.

clip	answer	confidence
ru-1689	uk	0.86
ru-1724	hr	0.585
ru-1737	uk	0.941

Text models on short controls

Hand-written probes, not a representative corpus. They check specific behaviours: negation, attribute binding, quoting an insult, counter-speech.

Title

id	input	title	description
ru-poc	Сейчас это промежуточный PoC. Набор полей зависит от источника, это не фиксированная схема продукта. До проверки на других компаниях не передаю его для утверждения требований и не обещаю срок масштабирования.	Промежуточный PoC и схема продукта	Сейчас проходит промежуточный PoC, где поле зависит от источника. Проверка на других компаниях не передаются для утверждения требований и не обещают срок масштабирования.
en-date	The architecture review has moved from Tuesday to Thursday. Please do not publish the draft before the review. The deployment date has not been agreed.	Architecture review schedule change	The architecture review has shifted from Tuesday to Thursday and requires the draft to be published before the review date. The deployment date has not been finalized.
en-condition	I can start the data migration only after security approves the access request. Approval has not arrived. This is not a commitment to deliver on Friday.	Data migration data migration process	Data migration is initiated only after security approves the access request. The process is not a commitment to deliver on Friday.
fr-date	La réunion est déplacée de mardi à jeudi. La version actuelle reste un prototype. La date de mise en production n’est pas encore décidée.	Réunion de mardi à jeudi	La réunion est déplacée de mardi à jeudi et reste un prototype actuel. La date de mise en production n'est pas encore décidée.
ru-tech	Команда разрабатывает программное обеспечение: распределённый веб-краулер, API, базу данных и автоматические тесты. Нужно измерить производительность сервера и исправить ошибки обработки данных.	Распределённый веб-краулер, API и базу данных	Команда разрабатывает программное обеспечение, включая распределённый веб-краулер, API и автоматические тесты. Нужно измерить производительность сервера и исправить ошибки обработки данных.

Gist

id	expected	top 3
ru-poc	—	finance, technology, business
ru-tech	technology	technology, business, career
en-tech	technology	technology, business, career
fr-food	food-drink	food-drink, home-garden, parenting-family
en-sports	sports	sports, gaming, history

Toxic

id	expected flags	model flags
en-clean	none	none	match
en-abuse	ABUSIVE	ABUSIVE	match
en-threat	THREAT	THREAT	match
en-hate	HATEFUL, ABUSIVE	HATEFUL, ABUSIVE, THREAT	differs
en-quote	none	none	match
en-counter-threat	none	none	match
en-positive-identity	none	none	match
fr-clean	none	none	match
fr-abuse	ABUSIVE	HATEFUL, ABUSIVE	differs
ru-clean	—	none	n/a (language unsupported)
ru-abuse	—	HATEFUL, ABUSIVE	n/a (language unsupported)

Findings

Clear writes 48 kHz on the streaming path even when 16 kHz is requested
enhance(path:to:options:) goes through enhanceStreaming, which uses ClearDSP.sampleRate for the writer and never reads options.sampleRate (Sources/Clear/Streaming.swift). The in-memory path resamples. Not touched by the open 4.2.0 release PR.
Title can drop a negation in the description
"Please do not publish the draft before the review" became "requires the draft to be published before the review date". Same prompt string as Titles.prompt, temperature 0, deterministic.
Ear marks some Russian clips as Ukrainian with isReliable = true
Confidence up to 0.94. Looks like the confident-confusion case the SDK already handles for Nordic languages.
Clear in front of ASR raises WER on broadband noise
Not a bug: a usage note worth a line in the docs. Clean speech is unaffected; see the control table above.

Not measured

p95 or SLAs: one pass per clip for ASR, three runs for WHO.
Energy, temperature and clocks during runs; only power source, CPU idle share, load average and macOS thermal warnings are logged.
Voz with vocabulary boost; Whisper with VAD or a prompt on this set.
Whether Clear helps on real (non-synthetic) noise; white noise is a stress test, not a room.
Other devices. One MacBook Air M5.