Speaker Identification in Meeting Transcription: Why "Who Said It" Is the Hard Part

Play a council recording into a modern transcription model and you will get back something remarkable: near-verbatim text, punctuated, in seconds. It is easy to conclude the problem is solved.

Then you look at the speaker labels. Speaker 2 is the chair for the first forty minutes, then becomes the city engineer. A member who said exactly one word all night ("aye") never gets a label at all. Two members with similar voices are merged into one. A resident at the podium and a member on the phone are both Speaker 5.

Diarization alone can't solve this. Enrollment plus memory can: a VoicePrint for each board member, confirmed once by the clerk and kept from one meeting to the next. That's how Govably works, and the rest of this post explains why it's the approach that holds up. It's worth understanding the problem on its own terms first, because attribution fails differently than transcription does. Transcription errors are usually visible on the page. Attribution errors are invisible: the words are right, the sentences read cleanly, and only the name in front of them is wrong.

One clarification before we start. This post is about attribution specifically, not about models inventing text that was never spoken. That's a separate failure mode with a separate fix, and we covered it in AI-generated meeting minutes and the public record. Keeping the two apart matters: a transcript can be word-perfect and still put the wrong name on the motion.

Three technologies people call "speaker ID"

Most confusion in this area comes from collapsing three different things into one phrase. They have different inputs, different outputs, and wildly different reliability.

Transcription (ASR) answers what was said. Audio in, words out. No notion of people at all.

Diarization answers who spoke when, but anonymously. The standard survey of the field defines it as the task of labelling recordings "with classes that correspond to speaker identity, or in short, a task to identify 'who spoke when'." Note the word classes. A diarizer clusters the audio into groups it believes are distinct voices and numbers them arbitrarily. It has no idea who anyone is. Speaker 1 is not a person; it's a cluster.

Speaker recognition, sometimes sold as voiceprint ID, answers which known person. This is a biometric matching task, and it requires something diarization never has: enrollment. NIST's speaker recognition evaluation plan frames the task as, "given a test segment and a target individual's enrollment data, automatically determine whether the target individual is present in the test segment." The load-bearing phrase is "enrollment data." You must have captured a sample of that person's voice beforehand.

The practical consequence is the one clerks trip over: diarization alone can never produce a name. If your tool shows real names, either someone enrolled voiceprints, or a human mapped the anonymous clusters to a roster, or a language model guessed from context ("Thank you, Councilmember Ruiz"). Those three approaches have very different reliability, and it's worth knowing which one you're relying on.

How accuracy is measured, and why the numbers you've seen are unfalsifiable

The standard metric is Diarization Error Rate (DER): the percentage of reference speaker time not correctly attributed. It decomposes into three parts, which is the useful bit:

• Missed speech: someone spoke and the system didn't notice.
• False alarm: the system labelled speech where there wasn't any.
• Speaker confusion: the right words, assigned to the wrong speaker. This is the term that puts a name on a statement someone didn't make.

Now the part almost nobody discloses. DER depends enormously on how you choose to score it: specifically, whether you allow a small timing tolerance around speaker boundaries (a "collar"), and whether you score overlapping speech at all. The team behind the widely used VBx system measured their own system on identical AMI meeting-corpus audio under three common conventions:

Same system. Same recordings. A ninefold spread, produced entirely by scoring choices. Their own paper says the inconsistency "makes it practically impossible to compare results between sites," and that most published work is unaware of it.

So when a vendor page advertises a diarization accuracy figure with no collar and no overlap setting stated, that number is not a claim you can check. It isn't necessarily dishonest. It's just unfalsifiable.

This isn't a reason for despair; it's a reason to ask better questions during procurement. "What's your speaker accuracy?" invites a meaningless number. "What happens in your product when the diarizer is wrong?" invites an honest answer.

A council chamber is close to the worst case

The room you meet in is, acoustically, roughly the hardest environment this technology faces. Three reasons, in order of how much damage they do.

1. Meetings are a hard domain, not just hard audio

The DIHARD III challenge (the field's benchmark for difficult diarization) reported that median error stayed under 10% for six of its domains, mostly one- and two-party conversations, but for meeting speech, web video and restaurant audio, performance "ranges awful," with median error between roughly 35% and 45%. Aggregate leaderboard figures are dominated by the easy domains. Meetings are the tail.

There's a US-government data point on precisely this condition, too. NIST's Rich Transcription evaluations ran for years on broadcast, telephone and meeting room audio. In the 2007 evaluation, with multiple distant microphones and overlapping speech scored, the best diarization error rate was 8.5% on conference meetings and 25.8% on lecture-style meetings. That's the best of 21 research sites, on the exact acoustic setup a council chamber presents.

2. People talk over each other, and that's where the errors live

The AMI meeting corpus is roughly 19 to 20% overlapped speech as a share of total speech time. That alone would be manageable. What makes it decisive is which error it produces. In a published ablation on AMI, a baseline system's 29.7% DER broke down as 3.0 false alarm, 5.8 confusion, and 20.8 missed speech. Roughly seventy percent of all error was speech the system simply failed to attribute to anyone, overwhelmingly because two people were talking at once.

Think about when overlap happens in your meetings. A member starts speaking before the chair finishes. Someone seconds a motion on top of the motion. A brief objection lands mid-sentence. These are not throwaway moments. They're often the procedurally significant ones.

3. One microphone in the middle of the room

Far-field capture (a single omnidirectional mic on the dais rather than a lapel mic per person) costs real accuracy, though less than people assume. Across two independent systems, the distant-mic penalty on the same meetings runs a few points of DER, on the order of 15 to 20% relative. Modest but consistent.

The more startling finding concerns microphone arrays. The CHiME-7 challenge baseline noted that diarization error "between microphones belonging to the same array may vary by up to 10%." Which physical capsule in your ceiling array the system happens to read can move the result by ten points, before anything else about the meeting varies. If your chamber audio quality feels inconsistent week to week for no obvious reason, that's a plausible mechanism.

The roll-call paradox

Here is the finding that should reorganize how you think about all of this.

Diarization is worst on short utterances. Vendor guidance is candid about it: one major provider's documentation recommends that "each speaker should have at least 30 seconds of continuous speech," and warns the model "may struggle to create separate clusters for speakers who only contribute short phrases."

Thirty seconds. A member whose entire on-record contribution for the evening is the word "aye," five times, is two orders of magnitude below that threshold.

A UC Berkeley study measuring six systems on NIST evaluation data quantified it precisely, and the phrasing is worth quoting: "less than 5% of each system's DER occurs during segments between 0.50 and 0.75 seconds long despite the DERs greater than 40%."

Read that twice. Sub-second segments carry error rates above 40%, but because they occupy so little total time, they contribute almost nothing to the headline score. Over 70% of scored time sits in segments longer than 2.5 seconds, and those segments dominate the metric.

The same study found the problem is algorithmic rather than acoustic: performance degrades specifically near speaker change points, with over a third of errors occurring within half a second of one. A roll call is the densest concentration of speaker change points a meeting produces. Name, response, name, response: nothing but boundaries.

So the shape of the risk is the opposite of intuition. The three-minute staff presentation, which feels important, is the easy case. The eleven seconds of roll call, which feels mechanical, is where the machine is least trustworthy and the record is most consequential.

What the law actually asks you to attribute

Now the good news, and it reframes the whole problem: open meetings law asks for far less speaker attribution than most clerks assume.

Across the states we examined, no open meetings statute requires minutes to attribute discussion to a named speaker. Texas requires only that minutes "state the subject of each deliberation" and "indicate each vote, order, decision, or other action taken." Illinois requires "a summary of discussion on all matters proposed, deliberated, or decided, and a record of any votes taken." California's Brown Act imposes no open-session minutes-content requirement at all.

New York's Committee on Open Government has said it plainly in an advisory opinion: "there is no requirement that minutes refer to every topic discussed or identify those who may have spoken." Robert's Rules reaches the same place from a different direction: minutes are "a record of what was done at a meeting, not a record of what was said."

There is at least one real counterexample, so don't over-generalize. Arizona does require it: A.R.S. § 38-431.01(C) says minutes shall include "the names of the members who propose each motion and the names of the persons, as given, who make statements or present material to the public body." Note the hedge the legislature built in: "as given." The statute contemplates the speaker supplying the name, not the clerk deducing it. That is a procedural solution to an acoustic problem, written into statute decades before diarization existed.

What the law does consistently require is vote attribution. Michigan's OMA states that "the minutes shall include all roll call votes taken at the meeting." Georgia attaches a legal presumption to omission: "It shall be presumed that the action taken was approved by each person in attendance unless the minutes reflect the name of the persons voting against."

Put the two halves together and you get the central irony of this post. The one thing the law reliably demands, who voted how by name, is precisely the thing diarization is worst at. And the discussion attribution that diarization handles comparatively well is largely optional.

The record has always known how to handle an unknown speaker

Here's what's easy to miss while chasing better models: the official-record professions solved the unidentified-speaker problem long ago, and their answer was never "guess."

The Alaska Court System's transcript standards classify an incorrect speaker as a major verbatim error, in the same tier as transcribing "I did not commit the murder" as "I did commit the murder." The prescribed remedy is not to try harder. It's to stop: if the transcriber is uncertain who is speaking after listening for a reasonable time (one minute maximum), they type UNIDENTIFIED VOICE. The standard even specifies not to write "unidentified male" or "unidentified female," while preserving role where it's known (UNIDENTIFIED JUROR, UNIDENTIFIED DISTRICT ATTORNEY). Arizona's transcript manual uses the same convention.

Congress converged independently on the same design. The GPO Style Manual gives the Congressional Record a graceful-degradation vocabulary: A MEMBER. when an individual can't be identified, SEVERAL SENATORS. for a group. The Record never invents a name it doesn't have, and never silently drops the utterance either.

And on crosstalk, the acoustic condition that breaks diarizers, the House resolves it jurisdictionally rather than acoustically: remarks by a Member not under recognition do not appear in the Record. Interruptions render inside the recognized speaker's turn. Note that this is the exact opposite of default diarizer behavior, which treats every interjection as a new speaker turn to be labelled.

The lesson generalizes cleanly to software. A well-designed system needs a way to say "I don't know." A diarizer that must assign every segment to some speaker will confidently mislabel rather than abstain, because abstention isn't in its output vocabulary. If your tool can't surface uncertainty (if everything arrives as a confident name), you have no way to know which lines to check.

VoicePrints: the part that remembers

Enrollment-based speaker recognition genuinely helps, and it addresses two problems anonymous diarization cannot.

The first is the naming problem. The second is subtler and better documented than most people realize: diarization labels are not stable across recordings. Amazon's service card for Transcribe states it directly: labels "are randomly assigned and are not consistent across separate audio files or sessions," and the feature "cannot be used to identify who a speaker is, to match a speaker against a known voice profile, or to track the same speaker across separate audio files or sessions." Microsoft files the same limitation under the heading "Unsupported uses," noting speakers appear as Guest1, Guest2, Guest3 and "these will be randomly assigned and may not be used to identify individual speakers."

That's the case for enrollment. Speaker 3 in January's meeting and Speaker 3 in February's are unrelated. Only a voiceprint carries identity across meetings.

This is why most AI minutes tools make you re-label speakers at every meeting: underneath, they run diarization alone, so every recording starts from nothing. Govably keeps a VoicePrint for each board member across meetings. Confirm a member once and, at the next meeting, they're already named. Each time you confirm a speaker, that member's VoicePrint gets better, so recognition improves the longer you use it. Transcription handles unlimited speakers, so a large board, staff and the public at the podium all get their own labels instead of being merged together.

Memory also changes what review means. Instead of mapping Speaker 1 through Speaker 9 to names every month, the clerk checks the few lines the system isn't sure about. That is the difference between a tool that drafts and a platform that finishes.

Which raises a fair question: is enrolling your board members' voices legally risky? Based on the statutes we reviewed, the exposure for the public body itself is low, but the reasoning is worth knowing, because it's more of a drafting accident than a blessing.

• Illinois BIPA expressly names "voiceprint" as a biometric identifier, and just as expressly excludes government. 740 ILCS 14/10: "A private entity does not include a State or local government agency," nor "any court of Illinois, a clerk of the court, or a judge or justice thereof." BIPA's damages provisions run only against private entities.
• Texas CUBI covers voiceprints but is gated on capture "for a commercial purpose," and is enforceable only by the Attorney General, with no private right of action.
• Washington's RCW 19.375 excludes government twice over: its definition of "person" "does not include a government agency," and it too is limited to commercial purposes.

Every one of those carve-outs exists because these laws were written for retailers and employers. Legislatures are already closing the gap for public agencies. Washington's RCW 40.26 does bind state agencies, requiring notice that "clearly specif[ies] the purpose and use" and consent "recorded and maintained by the agency for the duration of the retention," and it shields the biometric from public-records disclosure. It reaches state agencies only, not cities and counties. But it's a preview.

Three practical notes, none of which require a lawyer to act on:

1. Get written consent anyway. RCW 40.26 is a ready-made template even where it doesn't bind you: state the purpose, state the retention period, keep the signed record. Board members are identifiable public officials being asked to enroll a biometric. A one-page consent costs nothing and defuses the political story as much as the legal one.
2. Ask your vendor, in writing, what else the voiceprints are used for. This is the sharper risk, and it's newly sharper. Illinois BIPA exempts government contractors, but in Thomas v. Cornerstone Services, 2026 IL App (3d) 240568, decided this April, the Illinois Appellate Court held that exemption applies "only to exempt a contractor from liability when acting within the scope of its government contract." VoicePrints used to serve your contract are covered. VoicePrints reused to train a general model or improve a product sold elsewhere plausibly are not. Your contract should say, in plain words, that they aren't. At Govably, VoicePrints are used only for your organization.
3. Delete voiceprints at offboarding. Every biometric statute we reviewed converges on retention limits, even the ones that don't bind you. When a member leaves, their voiceprint should leave with them. And note an unsettled question worth raising with your attorney: a voiceprint template held by a city or county is a record created by a public body, and most states have no biometric exemption to their public-records act.

A workflow that gets names right and keeps them right

Here's the structural insight, and it comes from an unexpected place. The large research corpora built from government proceedings (parliamentary speech datasets running to thousands of hours) do not derive speaker identity acoustically. They look it up in official metadata: the roster, the order of business, the recorded speaker list. Segments where the metadata is ambiguous get discarded rather than guessed.

The people with the most audio and the best models deliberately avoid solving this with sound. A clerk's office should draw the same conclusion. Attribution should be anchored to the roster and to procedure. The audio's job is to match voices against members the clerk has already confirmed, not to guess from scratch every month.

What that looks like in practice:

• Treat roll call as the anchor, not just a formality. Each member saying their own name at a known point in the meeting is the cleanest enrollment and calibration moment you will ever get. It's also the moment that lets a system map anonymous clusters to a real roster.
• Have the chair recognize speakers by name before they speak. "Councilmember Reyes." Two words, and it converts an acoustic inference into a textual fact the transcript itself carries. The Illinois Municipal League's guidance on AI minutes recommends exactly this, alongside keeping the recording device close to speakers. Arizona's statute anticipates it with "as given." It is the highest-leverage thing on this list and it costs nothing.
• Treat votes as structured data, not prose. This is the single most important design decision. Mover, seconder and each member's yea, nay or abstain should be fields the clerk can check at a glance, not words buried in a paragraph. Govably captures each motion from the recording with its mover, seconder and roll-call vote by member, using the roster and the voices it already knows, and the clerk confirms them before anything is published. The roll-call paradox stops being your problem.
• Confirm each member once, then let the system remember. Publishing "Speaker 3" in a public transcript is not a neutral act; it's an unresolved record. In Govably, confirming who a speaker is does more than fix today's transcript: it trains that member's VoicePrint. Next meeting they arrive already named, and the clerk's review shrinks to the handful of lines the system flags.
• Good audio helps everything. Given the far-field and array findings above, per-seat microphones improve attribution in any room, and they give each member's VoicePrint a cleaner sample to learn from.
• Keep public comment attribution deliberate. Commercial systems are worst on the people who speak briefly, once, from a distance, which describes every public commenter. Worth knowing too: a widely cited study across five commercial speech systems found substantially higher word error rates for Black speakers than white speakers. Public comment draws the most linguistically diverse voices in the room, and it's where most statutes require the least. Have the speaker state their name at the podium, and consider whether you need to name commenters in the minutes at all.

The through-line

Diarization is a real technology that does real work. It turns an undifferentiated wall of text into something a clerk can navigate. What it cannot do on its own is know who anyone is, or remember them next month.

The failure mode to design against is not that the machine is bad. It's that the machine is confidently wrong in the specific places the law cares most about, and its own accuracy metrics are structurally blind to those places. A system that reports 12% DER is telling you about staff presentations. It is telling you nothing about the roll call.

That takes enrollment and memory: a VoicePrint for every member, confirmed once by the clerk and sharpened by every confirmation after. Add the roster, the chair's recognition and structured votes, and the record gets names right in exactly the places the law cares about, meeting after meeting, without the clerk starting over each time. That is what Govably is built to do.

Sources: Park et al., "A Review of Speaker Diarization: Recent Advances with Deep Learning" (arXiv:2101.09624) · NIST 2024 Speaker Recognition Evaluation Plan · Landini et al., "Bayesian HMM clustering of x-vector sequences" (VBx) (arXiv:2012.14952) · Third DIHARD Diarization Challenge (arXiv:2012.01477) · NIST Rich Transcription 2007 Meeting Recognition Evaluation · Bredin & Laurent, "End-to-end speaker segmentation for overlap-aware diarization" (arXiv:1910.11646) · CHiME-7 DASR Challenge baseline (arXiv:2306.13734) · Knox, "Speaker Diarization: Current Limitations and New Directions" (UC Berkeley EECS-2013-108) · AWS AI Service Card: Amazon Transcribe · Microsoft Speech to Text Transparency Note · A.R.S. § 38-431.01 · MCL 15.269 · Georgia Open Meetings Act · Cal. Gov. Code § 54953 · NY Committee on Open Government, OML-AO-4666 · Alaska Court System transcript standards (TF-410) · GPO Style Manual, Ch. 19 · 740 ILCS 14/10 (BIPA) and 740 ILCS 14/25 · Thomas v. Cornerstone Services, Inc., 2026 IL App (3d) 240568 · RCW 40.26.020 and RCW 19.375.010 · Tex. Bus. & Com. Code ch. 503 (CUBI) · 720 ILCS 5/14-3 · Koenecke et al., "Racial disparities in automated speech recognition," PNAS (2020) · Illinois Municipal League guidance on AI-assisted minutes (Mar. 2026) · Robert's Rules of Order Newly Revised, 12th ed., § 48:2. This article is general information, not legal advice; consult your attorney or state archives for your jurisdiction's rules.