When we started this ATS build, the CHRO's first ask was, "we want AI to rank candidates for us". Fair — every ATS launch deck in 2026 says that. But the more we dug in, the more it became clear that leading with an LLM would create three problems that a keyword matcher wouldn't.
The three problems with LLM-first scoring
1. Explainability
When an LLM ranks candidates, you get a number. When HR asks "why did this candidate rank at 72%?", you get some paragraph the model wrote. When a candidate later files a complaint under Pakistan's PDPA (or the EU's AI Act, if you have EU applicants), you need to say exactly why they were scored where they were. "The model thought so" is not a defence.
A weighted keyword matcher, by contrast, produces an audit trail like this:
Score: 69.9. Sources: title + JD + scorecard. Matched (29/46 terms): industrial, engineer, garment, production, SMV, capacity, planning, kaizen, ISO, quality... Missing: value stream mapping, kanban, MOST, autonomation, TQM.
That's a defensible answer. Every candidate can be told exactly why. There's no black box.
2. Cost and latency
At our client's scale (~500 applications/month), running every CV through Claude at 15,000 input tokens each is $75-150/month in LLM costs alone. Small money, sure. But if that goes to $5,000-15,000/month at 10× the volume with no meaningful ranking improvement, it's just a subsidy to Anthropic.
Latency matters too. An LLM round-trip is 5-15 seconds. A keyword score runs in under 100 ms. If you're calling the scorer on every apply-form submission (as we are), that's a very noticeable UX difference.
3. Regressions between model versions
When Anthropic ships Claude 5, your scoring distribution changes overnight. If HR has been using a threshold ("shortlist if score ≥ 60%"), that threshold now means something different. You have to re-tune. Every model update.
A weighted keyword matcher is deterministic. Its score distribution is stable across years. HR's threshold means the same thing today as it did last quarter.
The keyword design that actually works
We iterated a lot on what "weighted keyword scoring" means. The current design (call it v2) does roughly this:
- Tokenise the JD, uploaded JD PDF, and scorecard. Remove stop words. Drop very-common recruiter fluff ("responsibilities", "requirements", "candidate").
- Assign source weights:
title: 5.0,department: 2.0,typed JD: 3.0,uploaded JD PDF: 3.5,scorecard: 4.0. Terms in higher-weight sources count more. - Boost per-frequency modestly: a term appearing 3× in the source is worth ~1.36× a term appearing once. But cap the boost — otherwise boilerplate ("company" × 15) dominates.
- Keep the top 60 reference terms. Score the CV as
matched_weight / total_weight × 100. - Emit a full audit line: matched terms, missing terms, source list, mode ("keyword" vs. "ai"). Written to
applications.score_breakdown.
Result: an Industrial Engineer with 7 years of textile experience scores ~70% against a real IE JD. An automotive IE (transferable but not textile) scores ~35%. A retail manager or a chef scores < 5%. Defensible ranking, zero LLM cost, zero latency.
Where the LLM is the right tool
We didn't throw out AI scoring — we just moved it. Two places it's genuinely useful:
As a fallback for empty JDs
If a requisition has a title and department but no typed JD or uploaded JD (a common mid-cycle case), the keyword scorer has almost nothing to work with. In that scenario we fall through to Claude with the CV plus whatever role text we do have. It's not great, but it's better than scoring against 5 keywords.
As an optional "second opinion" mode
For high-stakes roles (senior hires, sensitive plants), the recruiter can hit a "get AI opinion" button on any candidate. The LLM output is stored separately from the keyword score, both are shown, and the recruiter chooses which weighs more. That way the LLM is advisory, not the ranker.
Design principle: the LLM should assist a human's judgment, not replace it — especially where the human is legally accountable.
The compliance checklist
Anyone building CV-scoring anywhere is about to run into some form of AI regulation. Pakistan's PDPA is coming. The EU AI Act is here. Even without regulation, one lawyer's letter from a rejected candidate can ruin a quarter.
What we make sure every ATS ships with:
- Explainable score. Every score has a written rationale attached — which terms matched, which missed, which sources contributed.
- Score is not the decision. HR always has to click something to advance or reject a candidate. The score is a signal, not an auto-decider.
- Complete audit trail. Every score, every status change, every human decision, timestamped and attributed. Immutable append-only.
- Data retention policy. CVs are auto-deleted after N days for non-hired candidates. Configurable per-tenant.
- Right of access. When a candidate exercises their PDPA rights, we can produce every piece of data we hold about them within 48 hours — including the score breakdown.
- Consent capture. Explicit opt-in with a timestamp when the candidate submits the form. Not a checkbox they don't notice.
The takeaway
"AI" in your ATS is a feature, not a foundation. Build a boring, deterministic, explainable scorer first. Layer LLM assistance on top for the edge cases where it's genuinely useful. Keep humans on the actual decision.
Your compliance team will thank you. Your candidates will get honest feedback. And your Anthropic bill will stay four digits instead of six.
If you're building an ATS and thinking about how to layer AI in, let's talk — we've done the design work already.