If candidates can use AI, make the exercise test their judgement

Would you hire the person who wrote the nicer reply?

Two candidates reply to an upset customer. One response is warm and polished. The other is plainer. The polished one promises a recovery date that nobody has confirmed.

If your exercise mainly rewards fluent writing, you can miss the decision that mattered. Let the candidate show how they checked the facts, chose the promise and handled the missing information.

The support-lead exercise below is invented and can be adapted. It is not a validated selection instrument. OPM describes work samples as tasks resembling actual work and suited to capabilities expected on entry. Give candidates the product facts they need; do not quietly test knowledge your team would teach after hiring.[opm-work]

Give everyone this brief

“A customer reports missing records after a service incident and asks whether they will be restored by Friday. Write a short reply, identify anything you need to check and propose one internal escalation.”

Use this fictional source pack with the brief. Give every candidate the same version; none of these messages, policies or incident details describes a real customer.

“Customer message, Thursday, 10:00: “Some records are missing from our workspace since yesterday’s incident. We need them for Friday’s handover. Can you confirm they will be restored by then?””

Scroll sideways to read the full table.

Scroll sideways to read the full table.

State the time expectation, evaluation criteria, submission route and accommodation arrangements. Allow the same opportunity to ask clarifying questions. The exercise should test the work you named, not how well somebody guessed the hidden rules.

Tell candidates what assistance is allowed

For this version, AI drafting is optional and may use only the supplied materials. Ask for a short note about meaningful assistance, what the candidate checked and what they changed. Do not request their personal chat history.

If the exercise needs a particular paid tool, provide comparable access. Otherwise use a common draft everyone can critique. A subscription advantage should not become a hidden assessment criterion.

If part of the actual role must be performed without assistance, explain why and observe that part separately. Choose the tool conditions because of the job, then communicate them before candidates begin.

Put this promise in a common draft

“We understand how frustrating this is. Your records will be restored by Friday, and we’ll keep you updated.”

The Friday sentence is the problem. The source pack does not support it. NIST describes confabulation as confidently false generated content. A useful assessment asks whether the candidate spots the unsupported promise in a draft, whoever wrote it.[nist-profile]

Ask candidates to revise the reply and identify the source for any assurance they keep. A good answer can explain the investigation, avoid inventing a deadline and propose checking the next confirmed update with the incident team.

In our example, Leon keeps the Friday promise. Priya removes it and asks when an approved update will be available. Do not turn those observations into permanent character labels. They show how these two submissions handled the supplied facts.

Ask the same follow-up, then read the rubric

“Which statement needed the most checking? What would you change if the incident team supplied a confirmed recovery date?”

Give each candidate time to examine the materials. Record the submitted answer and any revision during discussion. Leon may notice the problem only after the follow-up; Priya may explain that she caught it before submitting. Those are different observations.

Scroll sideways to read the full table.

Write these anchors before opening submissions. Review the dimensions independently before discussing an overall recommendation. A long account of careful checking deserves little credit if the final reply still makes the unsupported promise.

Try the brief with colleagues first to catch missing instructions and unreasonable scope. That can improve clarity; it does not establish predictive validity. The hiring note should name the observed behaviour: “removed an unsupported deadline after checking the policy sheet”. That is enough to discuss. “AI expert” or “always trustworthy” says more than the exercise showed.