Verbatim speech-to-text: every engine we tested corrected our learners' grammar

We needed verbatim speech-to-text for an English-language-learning product, where a learner's mistakes are not noise in the recording. They are the entire signal. So we gave one 93-word learner recording to ten transcription engines, asked the four that accept a prompt to change nothing, and compared every result against a human transcript.

All ten engines repaired at least one grammar error. Two of the learner's errors were erased by all ten. The engine with the best word accuracy was fifth on the measure we actually cared about, and the engine that read closest to the human transcript had quietly deleted seven words.

The sentence that started this

Our learner, an adult studying English in Pakistan, said this:

she decided to left the wallet

Six of the ten engines gave it back that way. Three returned "she decided to leave the wallet." One dropped the clause. Those three transcripts are better English and they are useless to us, because the thing we needed to detect had been fixed before we ever saw it.

A quick note on what is not in this post: the audio. It is a real learner's voice on a real assignment, and we do not have consent to publish it, so you get the transcripts and the scoring code instead.

What the learner said, and what the engines heard

Here is the human reference transcript in full. A human evaluator listened and wrote down what was actually said, disfluencies included.

Aisha works in a busy office and Aisha works in a busy office and she often works late. At time of leaving the office, she found a wallet and with some in it and some identity with name and picture. And she has, and she decided to left the wallet. She decided to submit the wallet at the security desk, and the other morning the owner find the wallet and the owner thanks her found wallet. And after a week, she receives a written report from the company because, uh, for the honesty.

That is a rich piece of assessment data. There is a repeated opening clause, a missing article, an abandoned false start, a wrong verb form, a missing third-person "s", a dropped preposition, a filler, and a tense slip. Nine markers in ninety-three words. Each one is something a teacher would circle.

Take just the opening clause, where the learner starts the sentence, abandons it, and starts again. Here is what each engine returned for it:

Engine	What it returned for the opening
Human reference	Aisha works in a busy office and Aisha works in a busy office and she often works late.
Gemini 3.1 Pro	Aisha works in a busy office and Aisha works in a busy office and she often works late.
ElevenLabs Scribe v2	Aisha works in a busy office and Aisha works in a busy office and she often works late.
AssemblyAI 3.5 Pro (realtime)	Aisha works in a busy office and Aisha works in a busy office and she often works late.
Gladia Solaria 3	Aisha works in a busy office, and Aisha works in a busy office. Then she often works late.
smallest.ai Pulse Pro	Aisha works in a busy office and. She works in a busy office, and she often works late.
OpenAI GPT-Transcribe	Aisha works in a busy office and. Aisha works in a busy office. She often works late.
Grok STT	Ishel works in a busy office and Ishel works in a busy office and she often works late.
MAI-Transcribe 1.5	Aisha works in a busy office, and she often works late.
AssemblyAI 3.5 Pro (pre-recorded)	Aisha works in a busy office and she often works late.
OpenAI GPT-Live-Transcribe	I still works in a BC office and office memory works late.

Three engines reproduced the restart word for word: Gemini, ElevenLabs, and AssemblyAI's realtime model. Gladia kept it but inserted a "Then" the learner never said, tidying two fragments into sequenced prose. Grok kept the repetition intact and lost the name, turning Aisha into "Ishel."

The two worth staring at are MAI-Transcribe and AssemblyAI's pre-recorded model. Both returned a clean single clause. Both silently dropped seven words that told us this learner restarts sentences under pressure. Nothing in either output marks the deletion, and the MAI transcript is arguably the most readable text in the table. If you skim transcripts to pick a vendor, the one that deleted your data wins the audition.

At the bottom, GPT-Live-Transcribe produced "I still works in a BC office and office memory works late," which is a different failure entirely and at least an honest one. You can see that it is broken.

Why speech models do this

This behaviour is a design target, not a defect.

Seongjin Park at Speak wrote the clearest statement of the problem we have found, in a May 2026 post on speech recognition for language learning. He splits recognition in two. Intent-based recognition asks what the speaker meant and returns the most likely clean sentence. Surface-level recognition asks what the speaker actually produced. Almost every modern engine optimizes for the first. Most learning products quietly assume the second.

There is a whole research task built on the first assumption. Liao et al. call it ASR post-processing for readability, which "aims to correct grammar mistakes, disfluency, and readability of the ASR output." In that framing, verbatim output is the problem being solved. Their paper notes plainly that verbatim transcription "creates many problems for modern applications."

Worse for anyone hoping to switch the behaviour off: the cleanup does not have to be a separate stage. Lou and Johnson showed back in 2020 that you can train a model to map disfluent speech directly to fluent transcripts, with no distinct disfluency-removal step to disable. When the repair is baked into the acoustic model, there is no flag to find.

And the incentives point the same way. As Park observes, evaluation datasets commonly pair audio with transcripts that annotators already tidied up for captions, so the model scoring best on the benchmark "may be precisely the model that's worst for teaching." That is a familiar shape if you have ever worried about what a model-swap eval actually has to measure: the score goes up while the thing you needed goes away.

The prompt that doesn't work

Four of the ten engines accept a prompt. We gave all four this, verbatim:

Transcribe the audio exactly as it is, if it is empty return nothing, don't add anything
extra or fix any grammar/spelling/punctuation/sentence structure errors, don't even remove
repetitions.

It is about as unambiguous as an instruction gets. Three of those four engines finished in the bottom four on error retention.

We want to be careful about what that does and does not show. This is one clip, and the prompt-capable engines are not otherwise matched, so we cannot claim prompting makes fidelity worse. What we can say is narrower and still useful: on our sample, asking politely did not buy us verbatim output. If your plan for keeping learner errors is a well-worded prompt, test it before you build on it.

Measuring the right thing for verbatim speech-to-text

Our first instinct was word error rate, and it was wrong.

WER counts every word the same. An engine can transcribe eighty-five ordinary words perfectly, fix the three that carry the whole diagnosis, and post a fine score. Word accuracy answers "did it hear the words," and our question was "did it keep the mistakes."

So we scored something else. We took the nine error markers a human evaluator identified in the reference and checked, for each engine, how many survived. Call it error retention.

The two supporting columns in the results table are computed the ordinary way, on text normalised by the function below: 1−WER is one minus the Levenshtein edit distance between the reference and hypothesis word lists, divided by the reference's 93 words; character similarity is difflib.SequenceMatcher(...).ratio() over the normalised strings. No language model is involved in scoring, so you can rerun any number in this post yourself from the transcripts in the appendix.

def normalise(t):
    "lowercase, strip punctuation, collapse whitespace. disfluencies are NOT stripped."
    t = t.lower().replace("\u2014", " ").replace("--", " ").replace("-", " ")
    t = re.sub(r"[^a-z0-9\s]", " ", t)
    return re.sub(r"\s+", " ", t).strip()

MARKERS = [
    ("repeated opening clause", lambda s: s.count("works in a busy office") >= 2),
    ("missing article",         lambda s: "at time of" in s),
    ("false start",             lambda s: "she has and" in s),
    ("wrong verb form",         lambda s: "to left" in s),
    ("missing 3rd-person -s",   lambda s: "owner find the" in s),
    ("missing preposition",     lambda s: re.search(r"thanks her (?!for)", s) is not None),
    ("filler",                  lambda s: re.search(r"\buh\b", s) is not None),
    ("tense",                   lambda s: "she receives" in s),
    ("proper noun",             lambda s: "aisha" in s),
]

Before the results, one gotcha that cost us an afternoon. If you reach for difflib to compare transcripts, construct it with autojunk=False:

difflib.SequenceMatcher(None, reference, hypothesis, autojunk=False).ratio()

With the default on, once the second sequence reaches 200 elements, SequenceMatcher treats any element appearing in more than 1% of it as "popular" and skips it during matching. Compare two strings character by character and that heuristic eats most of the alphabet. We got a similarity of 9.4% for a transcript that was 84% word-accurate before we spotted it.

We are not the first to notice WER is too blunt here. The FER and DER metrics score fluent and disfluent regions separately for much the same reason. Ours is cruder and domain-specific, which is the point: the markers came from our learners, and yours should come from yours.

Fidelity against price

One clip. Ninety-three words. This is a worked example of how to measure, not a leaderboard, and the ordering below would move with a bigger sample.

Engine	Errors kept	1−WER	Char similarity	$/hr	Basis	Prompt
Google Gemini 3.1 Pro Preview (High)	6/9	93.5%	97.8%	$1.09	derived	no
Gladia Solaria 3	6/9	93.5%	97.2%	$0.61	list	no
smallest.ai Pulse Pro	6/9	93.5%	97.3%	$0.24	list	no
ElevenLabs Scribe v2	6/9	91.4%	96.9%	$0.22	list	no
AssemblyAI Universal 3.5 Pro (realtime)	5/9	94.6%	97.8%	$0.50	observed	yes
Grok Speech-to-Text	5/9	91.4%	96.7%	$0.10	list	no
Microsoft MAI-Transcribe 1.5	4/9	82.8%	93.2%	$0.36	observed	no
OpenAI GPT-Transcribe	3/9	86.0%	94.2%	$0.27	list	yes
AssemblyAI Universal 3.5 Pro (pre-recorded)	3/9	83.9%	92.9%	$0.26	observed	yes
OpenAI GPT-Live-Transcribe	0/9	67.7%	85.1%	$1.02	derived	yes

Read the basis column before you trust a price. List means it is on the vendor's public pricing page today. Observed means it is what this account was billed and you should check your own. Derived means we calculated an hourly figure from a per-token or per-minute rate. All figures were checked in August 2026 and speech pricing moves quickly. Our AssemblyAI figure of $0.26/hr includes the prompting add-on, against a $0.21/hr list rate for Universal-3.5 Pro. Gemini bills per audio token rather than per hour, so $1.09 is an effective rate we derived from our own usage on this workload. We are not publishing the arithmetic because the token-to-audio ratio shifts with configuration, which is the honest caveat: treat it as our number, not Google's, and derive your own before comparing. Gladia's $0.61 is the Starter tier and falls to $0.20 on Growth. Two figures are our own observed rates rather than published list prices: MAI-Transcribe at $0.36/hr, which is in preview and whose catalogue entry defers to Microsoft's general Speech pricing page, and AssemblyAI's realtime tier at $0.50/hr. Check both against your own account before budgeting. The rest come straight from vendor pages: Grok at $0.10/hr batch, gpt-transcribe at $0.0045/min, Pulse Pro at $0.004/min, and Scribe v2 metered at $0.22/hr, and GPT-Live-Transcribe at $0.017 per minute of session audio, which works out to $1.02. ElevenLabs quotes "from $0.40 per hour" on its own marketing page; $0.22 is the metered rate, so the basis matters when you compare.

Three things in that table are worth your attention.

No engine kept more than six of nine. The ceiling on our clip was 67%, and it was set by four engines that offer no verbatim control at all: what they returned is simply their default behaviour. The four engines we could instruct all landed below that ceiling. Either way, a third of the teaching signal is gone before your assessment logic runs.

Word accuracy did not predict fidelity. AssemblyAI's realtime model has the best 1−WER in the table at 94.6% and sits fifth on retention. AssemblyAI describes Universal-3.5 Pro as transcribing "every conversation exactly as it's heard," and on general speech we have no reason to doubt it. Our clip is not general speech, which is the whole difficulty: a claim can be true on the distribution a vendor tested and not on yours.

Fidelity does not track price. smallest.ai Pulse Pro at $0.24/hr matched Gemini 3.1 Pro at $1.09/hr, both keeping six of nine. ElevenLabs Scribe v2 did the same at $0.22. The two priciest engines here sit at opposite ends of the result: Gemini at $1.09/hr kept the most, and GPT-Live-Transcribe at $1.02/hr kept nothing at all.

The two errors nobody kept

Every engine erased the missing article in "at time of leaving the office," and every engine inserted the missing preposition in "the owner thanks her found wallet."

Both are omissions. That is the pattern. A language model with a strong prior fills a small hole almost for free, and a missing function word is the smallest hole there is. Substitutions like "decided to left" survived in six engines because the wrong word is still evidence of something. A gap is not evidence of anything until someone decides it was meant to be filled.

For a learner at this level, dropped articles and prepositions are among the most diagnostic errors there are. They were also the least likely to survive the trip through the transcript.

One vendor ships the switch we wanted

While checking pricing we found something we had missed. Microsoft's MAI-Transcribe 1.5 takes an explicit style parameter:

{
  "enhancedMode": {
    "enabled": true,
    "model": "mai-transcribe-1.5",
    "transcribeStyle": "verbatim"
  }
}

Microsoft's documentation says you can set the value to verbatim "to preserve the original spoken content, including filler words and disfluencies." A real switch, documented, in a shipping API. It is off by default, which tells you what the default is for.

We did not test it. Our MAI number above came from a run where we did not set the flag, so read that 4/9 as the default style rather than a verdict on the model. Trying it is the first thing on our list.

The same API accepts a phraseList of expected proper nouns, and Microsoft's own example uses "Rehaan." That is pointed at our other problem. Grok returned our learner's name, Aisha, as "Ishel," and eight of ten engines got it right without help. We would not generalise from one name, but it is the failure mode we see most often with South Asian names in this product, and a hint list is a cheap thing to try: most engines expose one in some form.

What we'd tell someone choosing today

We are still evaluating, so this is a method rather than a verdict.

Choosing a verbatim speech-to-text engine starts before you look at a single vendor: build your marker list first, from your own recordings. Ours has nine entries because that is what a human evaluator heard in one clip. Yours will be different, and writing it down converts a vague preference for "accurate" into something you can score.

Then test on your own speakers. Nothing on a public leaderboard covers Pakistani English, and our results moved a lot between engines that all claim high accuracy. Include the names your users actually have.

Price the fidelity rather than the hour. The cheap engines on our clip were not the bad ones, and the most expensive was the worst. Until you score retention, the price ladder tells you nothing about the thing you are buying.

If you cannot find an engine that holds enough, split the job. Park's recommendation is to keep two transcripts: a clean one for the interface, where readable text is genuinely better, and a faithful one for the system generating feedback. We think that is right, and it is where we are heading. It also means routing between providers rather than betting the product on one, since the engine that keeps the most errors and the engine that produces the nicest prose need not be the same vendor.

We build AI products where this kind of measurement decides the architecture, and we are happy to talk it through if you are working on something similar: vantaso.org.

FAQ

Can you stop a speech-to-text model from correcting grammar?

Partially, and not reliably by prompting. We gave four prompt-capable engines an explicit instruction not to fix grammar, spelling, punctuation, or repetitions, and they still normalized. Microsoft's MAI-Transcribe exposes a real transcribeStyle: "verbatim" parameter, which is the most direct control we have found. Most engines do not offer one, and where cleanup is trained into the acoustic model there may be nothing to switch off.

Is word error rate a good metric for language-learning transcription?

No. WER weights every word equally, so an engine can score well while silently repairing the handful of words that carry the diagnosis. On our clip the engine with the best 1−WER ranked fifth on how many learner errors it preserved. Score retention of the specific errors you need instead.

Which speech-to-text API is best for preserving disfluencies?

We would not trust anyone's answer to that, including ours, from a single clip. On our one sample, four engines tied at six of nine markers retained, and they ranged from $0.22 to $1.09 per hour, so the cheap ones are worth testing first. The useful move is to score candidates against your own recordings with your own marker list.

Appendix: every transcript, in full

Every number in this post comes from the eleven texts below and the scoring code above. We are publishing the complete outputs rather than excerpts so you can recompute the table, disagree with our marker list, or score the same clip a different way. Prices were checked on 23 August 2026.

What we cannot publish: the audio itself, for the consent reason given at the top. Each engine was called once, on the same file, through its default endpoint for the model named; the four engines that accept a prompt received the text in the prompt section and no other tuning, and no reference text or phrase list was supplied to any engine.

Human reference (what the learner said)

Aisha works in a busy office and Aisha works in a busy office and she often works late. At time of leaving the office, she found a wallet and with some in it and some identity with name and picture. And she has, and she decided to left the wallet. She decided to submit the wallet at the security desk, and the other morning the owner find the wallet and the owner thanks her found wallet. And after a week, she receives a written report from the company because, uh, for the honesty.

Google Gemini 3.1 Pro Preview (High)

Aisha works in a busy office and Aisha works in a busy office and she often works late. At the time of leaving the office, she found a wallet and with some cash in it and, uh, some identity with name and picture. And she has, and she decided to left the wallet, she decided to submit the wallet at the security desk. And the other morning, the owner find the wallet and, uh, the owner thanks her for found wallet. And after a week, she received a written report from the company because, uh, for the honesty.

Gladia Solaria 3

Aisha works in a busy office, and Aisha works in a busy office. Then she often works late. At the time of leaving the office, she found a wallet and with some cash in it and some identity with name and picture. And she has, and she decided to left the wallet. She decided to submit the wallet at the security desk. And the morning. The owner find the wallet, and the owner thanks her for found wallet. And after a week, she receives a written report from the company because for the honesty.

smallest.ai Pulse Pro

Aisha works in a busy office and. She works in a busy office, and she often works late. At the time of leaving the office, she found a wallet and with some cash in it and some identity with name and picture. And she has, and she decided to left the wallet. She decided to submit the wallet at the security desk. And the morning, the owner find the wallet, and the owner thanks her for found wallet. And after a week, she receives a written report from the company because for the honesty.

ElevenLabs Scribe v2

Aisha works in a busy office and Aisha works in a busy office and she often works late. At the time of her leaving the office, she found a wallet and with some cash in it and, uh, some identity with name and picture. And she has-- and she decided to left the wallet. She decided to submit the wallet at the security desk. And the other morning, the owner find the wallet and cash. Uh, the owner thanks her for found wallet. And after a week, she received a written report from the company because, uh, for the honesty

AssemblyAI Universal 3.5 Pro (realtime)

Aisha works in a busy office and Aisha works in a busy office and she often works late. At the time of leaving the office, she found a wallet and with some cash in it and some identity with name and picture. And she has— and she decided to leave the wallet. She decided to submit the wallet at the security desk, and the other morning the owner found the wallet and The owner thanks her for found wallet. And after a week, she receives a written report from the company because, uh, for the honesty.

Grok Speech-to-Text

Ishel works in a busy office and Ishel works in a busy office and she often works late. At the time of leaving the office, she found a wallet and with some gas in it and some- Identity with name and picture, and she has, and she decided to left the wallet, she decided to submit the wallet at the security desk And the morning, the owner found the wallet and, uh, the owner thanks her for found wallet, and after a week She receives a written report from the company because, uh, for the honesty.

Microsoft MAI-Transcribe 1.5

Aisha works in a busy office, and she often works late. At the time of leaving the office, she found a wallet and with some cash in it and, uh, some identity with name and picture, and she has, and she decided to left the wallet, she decided to submit the wallet at the security desk, and the other morning the owner found the wallet, and the sh- uh, the owner thanks her for found wallet, and after a week she received a written report from the company because, uh, for the honesty.

OpenAI GPT-Transcribe

Aisha works in a busy office and. Aisha works in a busy office. She often works late. At the time of leaving the office, she found a wallet with some cash in it and some identity with name and picture. And she decided to leave the wallet. She decided to submit the wallet at the security desk. And the next morning, the owner found the wallet and the owner thanked her for found wallet. And after a week, she receives a written report from the company because for the honesty.

AssemblyAI Universal 3.5 Pro (pre-recorded)

Aisha works in a busy office and she often works late. At the time of leaving the office, she found a wallet with some cash in it and some identity with name and picture. And she has— and she decided to leave the wallet. She decided to with the wallet at the security desk. And the other morning, the owner finds the wallet, and the owner thanks her for found wallet. And after a week, she receives a written report from the company because for the honesty.

OpenAI GPT-Live-Transcribe

I still works in a BC office and office memory works late. Time of leaving the office, she found a wallet with some cash in it and some identity with name and picture. And she decided to submit the wallet at the security desk. And the other morning, the owner found the wallet and the owner thanked her for found wallet. And after a week she received a little reward from the company because for the honesty