Automated candidate shortlist built from real interviews

It is a shortlist assembled by software: applicants scored on some basis, sorted, and handed to a recruiter with the strongest at the top. The concept is uncontroversial and the implementations vary enormously in quality, because the sorting is only as good as the signal underneath it. A tool that ranks 400 applicants by resume keyword density has automated a shortlist, and it has also automated a bad one.

So the question worth asking any vendor is not whether they automate shortlisting but what the ranking is computed from. If the answer is the application, you are getting a faster version of resume screening with its weaknesses intact. If the answer is a structured interview the software conducted, you are ranking on what candidates said about their own work, which is a different class of evidence. That is what an AI agent for interviews produces before anything gets ranked.

The sequence that produces a shortlist worth reading has four steps, and the order matters. Define the criteria and the rating anchors before the req opens. Collect the same evidence from every applicant, which for us means a structured screening interview conducted by the agent. Score each answer against the anchors so a score is a rating of a specific response rather than an overall impression. Then rank, and hand the recruiter the list with the transcripts attached.

The step teams most often skip is the first, and skipping it is what makes automated shortlists feel arbitrary. If nobody wrote down what a strong answer to question three sounds like, the ranking is a black box even to the people who bought it. Doing it in this order also means the shortlist arrives with its own justification, so a recruiter can disagree with position four and see immediately why the software put it there. The mechanics of the round that feeds it are on first round interview screening.

For a first-round handoff, enough that the hiring manager has a real choice and few enough that every one gets a proper conversation. Three to five per opening is the range most teams settle into for final interviews, and eight to twelve is common for the stage right after screening. The number should come from your interview capacity for the next stage rather than from a percentage of applicants, because a shortlist longer than you can interview well is just a queue with better branding.

A ranked list with scores is more useful than a fixed cutoff, which is why we return the ranking rather than a pass or fail. It lets you take the top eight this week and go deeper into the list if two decline, without rerunning anything. It also means the boundary is a decision your team makes and can revisit, rather than a threshold buried in a vendor's configuration.

Yes, with duties attached, and shortlisting is precisely the activity those duties were written for. Under NYC Local Law 144 an automated employment decision tool is a computational process that issues a simplified output such as a score, classification or recommendation and substantially assists or replaces discretionary decision making. A ranked shortlist is a simplified output that substantially assists the decision, so if the role is located in New York City you are almost certainly in scope.

That means an independent bias audit within the prior 12 months, published summary, and candidate notice at least 10 business days before the tool is used. The duty follows the job location rather than your headquarters, and the audit cannot be performed by you or by the vendor. Illinois requires notice, explanation and consent under AIVIA, and its amended Human Rights Act took effect on 1 January 2026, naming zip code as an impermissible proxy. Whether a given tool counts is covered in what is an AEDT, and the employer-side duties are on AI hiring compliance.

It does in some products, and it does not here, and the distinction is worth pinning down in writing before you buy anything in this category. A tool that applies a threshold and auto-rejects everyone below it has replaced a discretionary decision. A tool that ranks and advances, leaving the reject decision to a person, has assisted one. Both are regulated, but they carry very different exposure and they produce very different candidate experiences.

InterviewAgent.ai ranks candidates and advances the strongest into recruiter review. Nobody is rejected by the agent. Everyone below the line stays visible in the ranking with their transcript, which matters practically as well as legally: the most common reason to go back into a pool is that the top of the shortlist declined, and a tool that already discarded the rest makes that expensive. How the scores that drive the ranking are produced is on candidate interview scoring.

Spot-check the top and the bottom, not the middle. Read the transcripts for the top three and confirm the scores match what you would have given, then read two from just below the cutoff and check that nothing obvious was missed. Disagreements cluster at the boundary, and they usually reveal a rubric problem rather than a scoring problem: a criterion that was weighted too heavily, or an anchor that was written vaguely enough that two answers of different quality both land on a 3.

Do that for the first two reqs and the calibration converges quickly. It is also the habit that keeps a human genuinely in the loop rather than nominally, which is the difference that matters if anyone ever asks how your shortlist was produced. Teams running large pools should read how to screen 500 applicants without adding recruiters alongside this.