How we check our meeting summaries
The details behind every summary on our meetings page: what goes in, what the checks catch, and how well it held up when we tested it.
1. The record
Every summary starts with the meeting's official record: the approved minutes, or the clerk's list of what passed (Albuquerque's "action summary"). We download it from the government's own site and read the text. A scanned page with no text in it is read by OCR software.
Minutes can take weeks. Until they're out, the record is the meeting video. We turn its sound into text with ElevenLabs Scribe, a speech-to-text tool, which marks each new speaker and the time they spoke. If Scribe fails, we fall back to YouTube's automatic captions, and the summary says which one it used.
Transcripts get names wrong. Before the summary is written, we fix the names of board members, the mayor or superintendent, the clerk, the attorney and anyone who has presented more than once, using how the body's own records spell them. The fixer only changes a word that is very close to a listed name. We set that bar by testing it on 21 transcripts and 68 sets of minutes and agendas, where it made no wrong change.
2. The writer
An AI model (Anthropic's Claude) reads the record with every other tool turned off. It can't search the web or read anything else. It writes the summary in a fixed shape: a headline, the short version, what people said, every item on the agenda with a "For you" line, what's next, and what the record doesn't say. Residents who speak in public comment are never named; officials and presenters may be.
Every statement must come with short quotes copied word for word from the record. Those quotes are what the checker tests. When a summary comes from video, each point links to the moment in the video its quote comes from.
3. The checker
The checker is plain code, not AI. A summary is held back unless, for every statement:
- each quote appears in the record exactly;
- each dollar figure, vote count, other number, date and member's name in the statement appears in that statement's own quotes;
- each vote count matches the record's tally ("MOTION PASSED: 10-0") or a roll call that adds up to it. The checker reads the lines after the quote, so a split vote can't pass as unanimous.
A summary that fails goes back to the writer once, with the checker's list of problems. The second draft is checked from scratch. If it still fails, it is held and never published.
4. What the checks can't catch
The checker tests numbers, votes, dates and names. A statement with none of those in it ("the bonds are repaid by a tax on that district") is only as good as the writer. A transcript can also mishear a word that isn't a name. That's why a summary written from video says so, and why we rewrite it from the approved minutes once they're out, with a dated note.
5. Choosing the transcription tool
We tested seven ways to turn meeting video into text on three 15-minute stretches of real meetings: the APS board (Oct. 7, 2026), the Albuquerque City Council (Oct. 5, 2026) and the Bernalillo County Commission (Aug. 11, 2026). We counted the names, dollar figures and votes each one got right, checked against the official records.
| Tool | Names | Figures | Votes |
|---|---|---|---|
| ElevenLabs Scribe (what we use) | 23 of 34 | 18 of 18 | 14 of 14 |
| NVIDIA Canary-Qwen 2.5B | 24 of 34 | 16 of 18 | 14 of 14 |
| NVIDIA Parakeet TDT 0.6B | 24 of 34 | 16 of 18 | 13 of 14 |
| OpenAI Whisper large-v3 | 22 of 34 | 17 of 18 | 14 of 14 |
| YouTube's automatic captions | 18 of 34 | 14 of 18 | 12 of 14 |
| IBM Granite Speech 3.3 (8B) | 15 of 34 | 14 of 18 | 13 of 14 |
| IBM Granite Speech 3.3 (2B) | 14 of 34 | 14 of 18 | 13 of 14 |
Scribe was the only tool to get every figure and every vote right. Names were the hardest part for all of them, which is why the name fixer above runs before the summary is written.
6. Testing video summaries against the minutes
In October 2026 we tested the video method on six past meetings that already had official notes: the APS board on June 17, Aug. 5 and Sept. 16, and the Albuquerque City Council on Aug. 17, Sept. 21 and Oct. 19. Each was summarized from its video alone, the same way a new meeting is before its minutes are out. Transcription cost $4.26 for about 19 hours of video.
Then code, not AI, compared each summary with the meeting's official notes: the APS board's approved minutes and the City Council's action summary. Each vote count was looked up in the notes' entry for that item. Each dollar figure, other number, date and member's name was looked up anywhere in the notes.
| What we checked | Matched the notes | Contradicted | Not in the notes |
|---|---|---|---|
| Vote counts | 41 of 41 | 0 | 1 |
| Members' names | 15 of 15 | 0 | 0 |
| Dollar figures, other numbers and dates | 41 of 74 | Can't tell by code | 33 |
The code matched 40 of the vote counts on its own. We checked the 41st by hand, because a page break split that vote in the notes, and it matched. The one vote not in the notes was on a ceremonial item the clerk didn't record.
Dollar figures, numbers and dates are harder to test this way. The City Council's action summary lists each item's title and vote, not the figures said out loud at the meeting, so 33 of them have nothing to check against. Code can say a figure is in the notes. It can't say a figure is wrong just because the notes leave it out.
The checker held back three of the six summaries before they could run. Each time it was over a member's name it couldn't find in the quote backing it up. One of those three also misspelled a school's name (Duranas for Duranes). That is the kind of mistake the checks exist to stop: it would never have gone up.
What's next: we plan to run this same comparison every time the minutes replace a video summary, and show a running total here.
7. Something wrong?
If a summary gets something wrong, tell us. Say which meeting and which point, what it should say, and where you saw it. We check it against the record, fix the summary and add a dated note saying what changed.