How to compare a shortlist when the evidence is uneven

The question to start with

The debrief starts with three names and a question: “Who is strongest?”

One candidate had a detailed technical interview. Another spent most of their conversation with the founder discussing the company. The third supplied a short CV and a thoughtful written answer. Everyone has a favourite, but the team is working from three different kinds of evidence.

Ranking them now feels decisive. It also risks mistaking a well-documented candidate for a well-matched one.

Consider a hypothetical startup hiring its first engineering manager. The founder needs someone who can keep delivery moving, coach a small team, and exercise sound technical judgement. The candidates and evidence in this article are fictional. The problem is familiar enough: the hiring process has produced a shortlist before it has produced a fair comparison.

Put the role back on the table

Before discussing people, restate what this hire must do. For the fictional startup, the immediate work is:

• Make realistic delivery commitments and handle changes without losing the team
• Give engineers useful feedback and support their development
• Make technical trade-offs with the team, including when to defer or challenge a decision

These criteria should already have been agreed. If the discussion reveals that one interviewer expected a hands-on lead while another expected a people manager, resolve that role question first. Do not let the most impressive candidate silently redefine the job.

OPM's job-analysis guidance grounds assessment in the connection between tasks and the competencies needed to do them. Here, that means comparing the candidates against the same work before comparing them with each other.

Then separate the criterion from the source. A CV statement, an interview example, and an observed exercise can all be relevant, but they tell you different things. Record how you learned something as well as what you think it means.

Build a comparison with the gaps still showing

Here is the team's fictional starting point:

Scroll sideways to read the full table.

There is no total row. At this point, adding one would require deciding how much to penalise each missing answer. The table gives you something more useful: a map of the gaps that could change the decision.

Ari currently has the strongest delivery evidence. That statement is narrower and more defensible than “Ari is the strongest manager”. Dev has a developed coaching example. Ren has promising technical evidence. None has yet been assessed evenly across the job.

Avoid labels such as “weak communicator” unless you can identify relevant evidence, the standard used, and the conditions under which it was observed. An interviewer's vague discomfort should become a question to examine, not a new fact about the candidate.

Work out why the cell is empty

An unknown can mean several things. The application omitted the topic. The interviewer forgot to ask. The candidate answered but the notes lost the detail. Or the question was asked and the answer did not establish the required experience.

Keep those situations separate. They imply different next steps.

For Ren, the team never asked about coaching. It would be particularly hard to justify treating that blank as a demonstrated weakness. For Dev, the CV makes a broad delivery claim that needs a concrete example. For Ari, mentioning mentoring warrants follow-up but does not establish experience handling difficult feedback.

Return to the original notes before scheduling more interviews. Sometimes the evidence exists and only the summary is incomplete. Preserve what the candidate actually said. Do not reconstruct a favourable example from an interviewer's memory without labelling the uncertainty.

You can also discover that the team already has enough evidence to stop considering a candidate for this role. If so, record the job-related reason. Equal treatment does not require endless additional stages; it requires a coherent standard and a fair opportunity to supply relevant evidence.

Spend the next conversation on what could change your mind

For each unresolved point, ask: “Would a credible answer change our next decision?” If the answer is no, reconsider why you are asking for more time.

In this hypothetical case, coaching and technical judgement are central. The team plans a common remaining assessment rather than letting each interviewer's curiosity set a different test.

Scroll sideways to read the full table.

Keep scope, preparation information, and evaluation standards comparable. Provide a way to request appropriate accommodations. Avoid asking candidates to produce commercially useful unpaid work for the company. A bounded hypothetical problem can make the relevant judgement visible without becoming a free consulting project.

OPM's structured-interview guidance uses common questions and rating standards to support consistent assessment. When choosing an exercise, its work-sample guidance is also useful: focus on relevant tasks and capabilities the person should bring into the role.

Reconvene with independent judgements

Have interviewers write their assessments before the next debrief. Ask for the criterion, supporting evidence, uncertainty, and recommendation. Discuss substantial disagreements by going back to those records.

Suppose Dev now gives a well-reasoned technical example, while Ari's coaching discussion remains thin. The new evidence may change who advances. That is a reason to value the extra assessment, rather than defend the first ranking.

If two candidates remain credible, write the trade-off plainly. One may bring stronger delivery evidence; another may have deeper coaching evidence. Which capability is harder for the existing team to support? Which uncertainty can you investigate, and which risk would the hiring manager knowingly accept?

Use that discussion to make the decision. Avoid manufacturing a decimal-point difference just to produce a winner. Keep any final comparison tied to the agreed role, and record the limits of what you know.

The comparison should show which observations support the decision and which questions remain open. Write it so somebody who was not in the interviews can follow the trade-off.

At the next debrief, start with the gaps that could change the decision. If the team still begins with “Who is strongest?”, put the evidence table back on the screen.