Two supervisors listen to the same collection call. One marks the disclosure as given. The other marks it as missing, because the collector said it while the consumer was talking over them and the second supervisor does not think a sentence nobody could hear counts.
Neither is being careless. They are applying the same row of the same scorecard and reading it differently. And until somebody puts those two people in a room with that call, the floor's QA record contains two incompatible definitions of its most important category, applied at random depending on whose queue a call landed in.
Calibration is the practice of finding that out on purpose. This is how to run it, how to measure whether it is working, and why the usual way of measuring it flatters you.
What calibration is for
A QA programme is a measuring instrument, and the reviewers are part of the instrument. Engineers have a name for checking whether an instrument gives the same reading twice and whether two operators get the same reading from it: measurement system analysis. The NIST/SEMATECH e-Handbook of Statistical Methods treats it as a precondition of trusting any data, and the logic transfers to a collections floor without modification. If two reviewers disagree about the same call, the variation in your findings is partly variation in your reviewers, and you cannot tell how much until you measure it.
On most floors that matters for three concrete reasons:
- Fairness. A collector's record should not depend on which supervisor drew their calls.
- Credibility. A finding you intend to show a creditor client or rely on in a dispute is weaker if a second qualified reviewer would have dismissed it.
- Signal. A trend line built on inconsistent judgements moves when the reviewer rota changes, and you will coach to a pattern that was never in the calls.
Our earlier guide to building a scorecard that survives a dispute gives calibration four numbered steps and moves on. This piece is the long version of those steps.
The session, in the order that works
1. Pick the calls deliberately. Three to five calls, chosen by someone who is not scoring. Not the worst calls on the floor and not clean ones — pick calls where the right answer is arguable. A disclosure delivered over an interruption. A consumer who says "stop calling me at work" in the middle of a sentence about something else. A relative who answers and volunteers that they handle the bills. Easy calls produce agreement and teach nothing.
2. Score blind and alone. Every reviewer scores every call independently, in writing, before any discussion. This is the step that gets compromised first. The moment the most senior person in the room says what they thought, everyone else's score becomes an estimate of what the senior person wants, and you have measured hierarchy.
3. Lay the scores side by side, row by row. Not overall impressions — the specific row, the specific call, who marked what.
4. Argue each disagreement to a cause. There are only three: one reviewer missed something in the audio, the reviewers heard the same thing and the scorecard wording supports both readings, or the scorecard is silent on the situation. The first is a reviewer issue. The second and third are scorecard issues, and they are far more common than people expect.
5. Amend the wording, and date the amendment. If "disclosure given in the opening" allowed two readings, rewrite it so it allows one: "disclosure audible on the collector channel before any account detail, and not spoken over the consumer". Record the change with its date, because findings before and after it were made against different definitions and somebody will eventually compare them.
6. Re-score the disputed rows. Quickly, at the end. If the new wording does not produce agreement in the room where it was written, it will not produce it anywhere else.
An hour covers it once the group has done it twice. The first session always overruns, because it is the first time anybody has discovered how differently the room reads its own scorecard.
Measuring agreement without fooling yourself
Most floors that measure calibration at all report percent agreement: of all the judgements two reviewers made, how many matched. It is the obvious number and it has a serious flaw on a collections floor specifically.
Take two supervisors who each review the same fifty flagged moments and decide, for each one, whether to confirm it as a finding or dismiss it.
| Reviewer B confirms | Reviewer B dismisses | Total | |
|---|---|---|---|
| Reviewer A confirms | 12 | 5 | 17 |
| Reviewer A dismisses | 3 | 30 | 33 |
| Total | 15 | 35 | 50 |
They matched on 42 of 50. That is 84 percent agreement, and in most QA meetings the conversation would end there with everybody satisfied.
Look at where the agreement comes from. Thirty of those 42 matches are flags both reviewers dismissed — the easy ones, the false positives anybody would wave through. Now look only at the flags that at least one reviewer thought were real. There are 20 of those (12 + 5 + 3). The reviewers agreed on 12.
On the flags that might be findings, they agree 60 percent of the time. For the eight calls in the off-diagonal cells, whether a collector has a compliance finding on their record was decided by the rota.
This is the same arithmetic that makes small samples blind to rare events, pointed at a different target. On a decent floor most calls are clean and most flags are dismissible, so two reviewers who each dismissed nine flags in ten without listening closely would still agree with each other most of the time. Percent agreement rewards the base rate.
The chance-corrected version
The standard correction is Cohen's kappa, which asks how much better than chance the agreement is. It needs nothing beyond the table above.
Reviewer A confirms 17 of 50, or 34 percent. Reviewer B confirms 15 of 50, or 30 percent. If they were confirming at those rates completely independently of the calls, they would both confirm 0.34 × 0.30 = 10.2 percent of the time and both dismiss 0.66 × 0.70 = 46.2 percent of the time. So 56.4 percent agreement is what you would expect from chance alone.
Kappa is the observed agreement beyond chance, divided by the maximum possible agreement beyond chance: (0.84 − 0.564) ÷ (1 − 0.564) = 0.63.
There are published rules of thumb for what counts as a good kappa, and they disagree with each other, so we would not lean on any threshold. Two uses are safe. First, as a trend: the same pair, the same kind of calls, quarter on quarter. Second, as a corrective to complacency — 0.63 describes this pair far more honestly than 84 does.
If kappa is more statistics than your floor wants, the simpler number from above does most of the work: agreement on the items at least one reviewer confirmed. It cannot be inflated by easy dismissals, and a supervisor can compute it on the back of the session notes.
What to calibrate, and what not to pretend to
Not every row deserves the same treatment.
Binary compliance rows — disclosure present, right party confirmed before account detail, a cease or dispute request raised, prohibited content — should converge. If two trained reviewers cannot agree whether a disclosure was given, the definition is broken, and the fix is wording. The categories themselves should trace to the requirements your counsel has identified in the Fair Debt Collection Practices Act and its implementing regulation at 12 CFR Part 1006, not to a vendor template.
Graded rows — professionalism under pressure, clarity, control of the call — will never fully converge, and forcing them to produces reviewers who score everything a safe three. Calibrate these for range instead: play one call the room agrees is excellent and one it agrees is poor, so the ends of the scale mean the same thing to everyone. Then accept a band of disagreement in the middle and keep those rows out of anything consequential.
Legal conclusions are not a calibration topic at all. A session can settle "the collector continued discussing the balance for forty seconds after the consumer said to stop calling". It cannot settle whether that was a violation. That question belongs to counsel, and a reviewers' meeting that votes on it has created a document nobody will enjoy producing later. Calibrate the facts and the scorecard language; escalate the conclusions.
Calibrating against a machine
On a floor where every call is transcribed and screened automatically, calibration changes shape rather than going away. There are now two things to check.
Reviewer against reviewer, on the same flags. This is the table above. A flag arrives with a timestamp and the quoted line, two people decide independently whether it is real, and you measure the match. It is quicker than traditional calibration because nobody is hunting through a twelve-minute recording for the moment in question — what a supervisor should do with a flagged call covers the triage itself.
Reviewers against the rubric, on unflagged calls. An automated rubric has misses as well as false alarms, and you only find misses by listening to calls it passed. A small monthly set of unflagged calls, scored blind by a human, is the check. This is one of the few places where a sample is the right tool, because the question — "is the screen missing a category of thing?" — is about a pattern rather than a rare event. Why two percent sampling fails explains the distinction.
One caution applies to any automated first pass, ours included. Reviewers who see a flag are primed to confirm it. If your confirm rate drifts upward over months with no change on the floor, suspect anchoring before you suspect the collectors. Blind scoring of a few calls with the flags hidden is the test.
We thought we were calibrated because the scores were always within a few points. Then we stopped comparing totals and compared the disclosure row on its own. Four reviewers, same six calls, and on two of them we split two against two. Same scorecard we had been using for three years.
— Quality assurance manager, first-party and agency collections, 11 years, name withheld by request
Cadence, and what it costs
Monthly is a sensible default; quarterly is the floor below which drift stops being visible. Three events should trigger an extra session regardless of the calendar: a new reviewer, an amended scorecard, and a change to a client's requirements.
The cost is honest arithmetic. Four reviewers for an hour, plus perhaps half an hour each of blind scoring beforehand, is six supervisor-hours a month. Against a programme that already consumes supervisor time by the dozen hours, that is small — and the supervisor-hour maths piece works through where the rest of those hours go. For a sense of how many people are doing this work nationally, the Bureau of Labor Statistics publishes occupational data on bill and account collectors and their supervisors.
Calibration also leaves a record worth keeping: the calls used, the blind scores, the disagreements, and the dated wording changes. The Consumer Financial Protection Bureau's supervision and examination materials describe compliance management in terms of monitoring and corrective action, and a creditor client's audit tends to ask a blunter version of the same thing — how do you know your reviewers are consistent? A dated calibration log is a direct answer. What a creditor client audit asks your QA record to produce goes through the rest of that list. Nothing here is legal advice, and a calibration log does not confer compliance on anybody; it shows that the people making findings were checked against each other.
Where our product fits, and where it does not
As of October 2026, our Voice tier — $99 per monitored collector per month, supervisors and admins free — transcribes and screens every recorded call against a collections rubric and reports each dimension separately, never as one blended score. Every flag carries its timestamp and the quoted line, and every confirm or dismiss is logged against the reviewer who made it. That log is the raw material for the agreement table above. The features page describes each piece and pricing has the tiers, including Essentials at $39 for desktop activity without voice capture.
Three limits bear on calibration specifically. Review is next-morning by design, so calibration happens on yesterday's calls and nothing here supports a session run on a call still in progress. Audio comes from the collector's computer, so calls made on a desk phone that does not route through the machine are absent from the record and cannot be calibrated at all. And calls tie to collectors and times rather than to accounts in your platform, so a session that wants the account history beside the call needs it pulled from your system of record. Ask us which of those apply before you buy.
Frequently asked questions
What is a QA calibration session on a collections floor?
A recurring meeting in which everyone who reviews calls scores the same small set of calls independently and blind, then compares results row by row. The purpose is to find where reviewers read the scorecard differently and to fix the wording that allows it, so a collector's record does not depend on which reviewer happened to draw their calls.
How many calls should a calibration session use?
Three to five, chosen because the right answer is arguable. Clean calls and flagrant ones produce agreement and reveal nothing. An interrupted disclosure, an ambiguous request to stop calling, or a third party who volunteers information will expose more in one session than twenty routine calls.
Is percent agreement a good measure of reviewer consistency?
It is easy to compute and easy to be misled by. On a floor where most calls are clean, two reviewers agree on most items simply because most items are easy dismissals. In the worked example above, 84 percent overall agreement sits on top of 60 percent agreement on the flags that might be real findings. Measure agreement on the contested items, or use a chance-corrected statistic such as Cohen's kappa, and track it as a trend.
How often should reviewers be calibrated?
Monthly is a reasonable default and quarterly is the practical minimum. Run an extra session whenever a new reviewer starts, the scorecard is amended, or a creditor client changes its requirements, because each of those changes what the instrument is measuring.
Should calibration decide whether a call was a violation?
No. A session can and should settle the facts — what was said, when, by whom — and whether those facts meet the scorecard's wording. Whether the facts amount to a violation is a legal conclusion for your own counsel. Keeping the two apart also keeps the calibration record clean.
Do we still need calibration if every call is scored automatically?
Yes, in two forms. Reviewers still decide whether each flag is real, and they will disagree, so their confirm and dismiss decisions need comparing. And the automated rubric itself needs checking against human judgement on calls it did not flag, because misses are invisible unless somebody listens for them.
What should we keep from each session?
The calls used, each reviewer's blind scores, the disagreements and how each was resolved, and any wording change with its date. Findings made before and after a definition changed were made against different rules, and the dated log is the only way to compare them honestly later.
Reviewing two percent of your calls?
CollectionsQA records, transcribes and scores every call your collectors take, and sends one supervisor report each morning naming the calls that need a human.