How AI Speech-to-Text Transcription Works: Acoustic & Language Models

The Automatic Speech Recognition (ASR) Evolution

For decades, speech recognition relied on rigid Hidden Markov Models (HMMs) and acoustic dictionaries that struggled with regional accents, conversational overlap, and background noise. Word error rates (WER) frequently exceeded 20-30%.

Today, end-to-end deep learning architectures (like Google's Gemini audio modality and OpenAI's Whisper) process raw audio waveforms directly through transformer-based attention layers, driving word error rates below 4%—comparable to professional human stenographers.

Step 1: Audio Preprocessing & Spectrograms

When you upload an audio or video file to MediaDocks, the server first normalizes the audio stream to a single channel (mono) at a consistent sample rate (typically 16 kHz).

The continuous sound wave is then sliced into micro-windows (usually 25 milliseconds each) and processed through a Short-Time Fourier Transform (STFT). This converts raw audio amplitude into a log-Mel spectrogram—a visual heat map that displays frequency power over time, matching how the human cochlea detects pitches.

Step 2: Acoustic Modeling & Phoneme Recognition

The neural encoder consumes the spectrogram and predicts probability distributions over phonemes (the distinct sound units that make up spoken words, like /k/, /æ/, /t/ in 'cat').

By employing multi-head self-attention, the model evaluates long-range context across the sentence rather than analyzing isolated words. This allows it to distinguish between homophones like 'their', 'there', and 'they're' based entirely on grammatical syntax.

Step 3: Language Models & Context Resolution

An acoustic model alone cannot reliably transcribe specialized terminology, acronyms, or nuanced idioms. That is where large language models (LLMs) come in.

When a speaker discusses 'Next.js App Router and Prisma ORM', the language model uses semantic probabilities to correctly spell technical nomenclature rather than outputting phonetically similar nonsense.

Step 4: Punctuation, Capitalization & Formatting

Human speech does not contain punctuation marks or capital letters. Early speech engines produced unreadable 'walls of text'.

Modern ASR pipelines include an inverse text normalization (ITN) and punctuation restoration layer. By analyzing vocal pauses, pitch inflections, and sentence boundaries, the model intelligently inserts commas, periods, question marks, and capitalizes proper nouns.

How Accuracy Is Measured (WER)

Claims that a transcription service is 95% or 99% accurate are usually quoting Word Error Rate, and the figure is worth understanding before you rely on it. WER counts three kinds of mistake — substitutions where the wrong word appears, deletions where a word is missed, and insertions where a word is added that nobody said — then divides the total by the number of words actually spoken.

Two properties of that definition matter in practice. First, WER is unweighted: mis-hearing 'a' as 'the' counts exactly the same as mis-hearing a drug name or a surname, even though one is invisible and the other could matter a great deal. Second, the figure is only meaningful against a stated recording condition. A 95% claim measured on studio-quality single-speaker audio tells you very little about a conference-room recording with four people and an air-conditioning unit.

A useful way to translate the number: at 95% WER accuracy you are looking at roughly one error every twenty words, which is about one per two sentences. That is a perfectly good first draft and it is emphatically not a publishable record. The errors also cluster rather than distributing evenly — they concentrate on names, technical terms and crosstalk, which is to say precisely the words you are most likely to be quoting.

Where Accuracy Breaks Down

Recognition quality is not a single number attached to a service; it varies enormously with the recording, and the failure modes are predictable enough to plan around.

Overlapping speech is the hardest case by a wide margin. Models are trained predominantly on one voice at a time, and when two people talk simultaneously the output typically collapses to one of them or produces a blend of both. Distance from the microphone is the next most damaging factor, because the ratio of voice to room reverberation degrades sharply with distance — a laptop microphone across a meeting table is a materially harder problem than a headset. Background music is particularly destructive, more so than steady noise like traffic, because music occupies the same frequency bands as speech and shares its rhythmic structure.

Vocabulary is the subtler issue. A model can only produce words it has representations for, so unusual proper nouns, internal project names, pharmaceutical terms and niche jargon tend to be replaced by whatever common word sounds closest. This produces the most dangerous category of error: a fluent, confident, grammatically clean sentence that says something nobody said. Accented speech varies by how well represented the accent is in training data, which means the technology works less well for some speakers than others as a straightforward matter of fact.

Speaker Diarization and Word Timings

Producing the words is only part of a useful transcript. Two further pieces of structure determine whether the output is something you can work with: who said each part, and when each part occurred.

Speaker diarization is the task of segmenting audio by speaker identity — answering 'who spoke when' without knowing in advance who anyone is. It works by extracting a voice embedding for short windows of audio and clustering windows with similar characteristics, so the output is anonymous labels like Speaker 1 and Speaker 2 rather than names. It is a genuinely difficult problem: it degrades on overlapping speech, on speakers with similar voices, and when someone's voice changes because they moved relative to the microphone. A known speaker count helps considerably.

Word-level timestamps are what make subtitles possible at all. Rather than timing only the start and end of each sentence, the model aligns individual words to positions in the audio, which allows cues to be split at arbitrary points while staying synchronised. The same alignment data powers click-to-seek transcript interfaces, where selecting a word jumps playback to that moment — a feature that turns a transcript from a document into a navigation tool for the recording itself.

Recording to Get a Better Transcript

Because recording conditions dominate accuracy, the most effective way to improve a transcript is to change how the audio is captured. These adjustments cost nothing and consistently outperform any amount of post-processing.

Microphone distance is the single biggest lever. Halving the distance between mouth and microphone roughly doubles the direct-to-reverberant ratio, and a cheap headset worn correctly beats an expensive condenser across the room. For multi-speaker recordings, a separate microphone per person transforms the problem, because it gives diarization clean separation to work from instead of asking it to untangle a single mixed channel.

Beyond that, the useful habits are mostly about the room and the conversation. Hard parallel surfaces create the reverberation that smears consonants, so soft furnishings help more than people expect. Steady mechanical noise — air conditioning, a desk fan, a laptop under load — is worth eliminating before recording rather than filtering afterwards, since filtering removes speech frequencies along with the noise. And in a group conversation, one person speaking at a time is not merely polite; it is the difference between a transcript that attributes correctly and one that does not.

• Get the microphone close. It outweighs every other factor combined.
• One microphone per speaker if you need to know who said what.
• Kill steady noise at the source; filtering it later also removes speech.
• Record music or effects on a separate track from dialogue.
• Say unusual names clearly, or note them down to fix in one pass afterwards.