Table of Contents
- Abstract
- 1. The bottleneck moved
- 2. What 60 years of research says about predicting performance
- 3. The productivity paradox nobody screens for
- 4. The five things a coding test can't see
- 5. What I built, and why
- 6. How you grade judgment
- 7. Honest limitations
- 8. Conclusion
- References
Abstract
For most of the last forty years, the technical interview rested on one assumption: typing correct code under pressure is a good proxy for being a good developer. In 2026 that assumption is broken. AI tools now produce working code faster, and often more correctly, than most candidates can write it on a whiteboard. Yet the cost of a bad hire hasn't fallen. Arguably it has risen, because a developer who can't read a brief, can't reason about a system, and can't catch a confidently wrong AI now ships those failures at machine speed.
This piece makes a research-backed argument for agency owners and hiring managers: stop testing the part the machine already does, and start testing the part it can't. I'll connect six decades of selection-psychology research to recent studies on AI-assisted productivity, lay out the five human capabilities a coding test is blind to, and describe an assessment I built for my own team to measure them directly.
1. The bottleneck moved
A short, honest story first, because it's where this all started.
A few days ago I had a short, strange window of access to Fable 5, Anthropic's Mythos-tier model. It didn't last. On June 12, the U.S. government issued an export-control directive requiring Anthropic to suspend access to Fable 5 and Mythos 5 for all foreign nationals, and to comply, Anthropic disabled the models for every customer worldwide. While that window was still open, instead of doing what everyone else seemed to be doing (generating reams of code), I used it to build something for myself: a hiring assessment for Titan Dev, my agency. The irony wasn't lost on me. I was using the most capable code-writing model I'd ever touched to build a test that contains zero coding questions.
That choice wasn't a gimmick. It was the conclusion of a problem I'd been stuck on for a year. Every time I interviewed a web developer, I was testing a skill that had quietly stopped being scarce. I'd watch a candidate struggle to reverse a linked list on a call while, in another tab, an AI agent would have written, tested, and documented the same thing in seconds. I was measuring the cheapest input in the entire production process.
The bottleneck moved. When code is abundant, the constraint on shipping good work is no longer can you produce code. It's everything wrapped around the code:
- Did you read the brief and hold every constraint in your head at once?
- Can you reason about why the bill is 4× higher when traffic only doubled?
- Did you notice the spec contradicts itself on page two?
- Are you curious enough to poke at the thing nobody's complaining about?
- And, the new core skill: can you direct an AI and catch it when it's confidently wrong?
None of those show up on a LeetCode score. All of them show up in the first month of a real engagement, usually as a missed requirement, a silent data bug, or a feature that technically works but solves the wrong problem.
The code is the cheapest part of the job now. We're still interviewing as if it's the expensive part.
Andrej Karpathy captured the shift in a February 2, 2025 tweet that coined the term "vibe coding," describing a workflow where you "fully give in to the vibes," accept all suggestions, and stop reading diffs because the models have become good enough. A year later, in his 2026 Sequoia Ascent talk, he drew the distinction that matters: vibe coding raises the floor for everyone; agentic engineering (the discipline of overseeing fallible agents with rigorous verification, taste, and accountability) raises the ceiling for professionals. The people who will win are not the fastest prompt engineers, but the ones who treat AI output as something that still needs human judgment.
2. What 60 years of research says about predicting performance
Here's the part most "rethink your hiring" posts skip: we already have a mountain of evidence on what actually predicts job performance. We've just been ignoring the parts that are inconvenient.
The foundational reference is Schmidt and Hunter's 1998 meta-analysis in Psychological Bulletin, which synthesized 85 years of personnel-selection research. Its central finding: the single best predictors of on-the-job performance are general mental ability (GMA), work-sample tests, and structured interviews. Just as important, combining a cognitive measure with a structured behavioral one predicts far better than either alone (Schmidt & Hunter, 1998).
That ordering got an important update. In 2022, Sackett, Zhang, Berry, and Lievens re-examined the same literature and showed that decades of estimates had over-corrected for range restriction, inflating the apparent power of standalone cognitive tests. Their corrected ranking pushed structured interviews (mean operational validity .42) to the top and moved raw GMA down to .31 (Sackett et al., 2022).
Read those two findings together and a coding whiteboard looks worse than we pretend:
- A LeetCode round is a work sample, but a sample of the wrong work. It tests algorithmic recall under artificial pressure, not the work an agency developer actually does.
- The thing with the strongest, most durable validity, the structured interview with consistent rubrics, is the thing most agencies do least rigorously, replacing it with vibes and gut feel.
There's a parallel lesson from inside one of the most data-driven hiring machines on earth. Google's Project Oxygen and Project Aristotle both concluded that the differentiators for high-performing engineers and teams were overwhelmingly behavioral (communication, judgment, psychological safety), not technical brilliance in isolation (Google re:Work). Google also found that brain-teaser interview questions predicted nothing at all (Bock, 2015).
So the research consensus, put plainly: structured, job-relevant, rubric-graded assessment of reasoning and behavior beats an unstructured technical gauntlet. AI didn't create that conclusion. It just removed our last excuse for avoiding it.
3. The productivity paradox nobody screens for
If you only read the vendor headlines, AI made every developer faster. The truth is messier, and it's the reason verification became a hireable skill.
The optimistic data is real. In a 2023 controlled experiment, developers using GitHub Copilot completed an HTTP server implementation task 55.8% faster than the control group (Peng, Kalliamvakou, Cihon & Demirer, 2023). That's a genuine, measurable speedup on well-scoped tasks.
But then came the counter-evidence. In July 2025, METR published a randomized controlled trial with 16 experienced open-source developers working on real issues in repositories they had maintained for years. The developers expected AI to speed them up by about 24%, and afterward believed they had been about 20% faster. In reality they were 19% slower when using AI tools (primarily Cursor Pro with frontier models) than when working without them (METR, 2025).
Both results can be true at once, and the gap between them is the hiring signal. AI accelerates people who can frame the task, spot wrong output, and reject it fast. It quietly decelerates people who accept plausible-looking answers, then spend their afternoon debugging code they never understood. The decisive variable isn't whether someone uses AI. It's whether they verify it.
Psychology has a name for the failure mode: automation bias, the tendency to over-trust automated output and stop checking it independently (Parasuraman & Manzey, 2010). Pair it with the Dunning–Kruger effect (Kruger & Dunning, 1999), where the people least equipped to judge an answer are also the least able to notice they can't, and you get a precise description of the developer who'll hurt your agency in the AI era: confident, fast, and wrong, with an AI to make the confidence look earned.
Surveys confirm the tools are now everywhere. The 2025 Stack Overflow Developer Survey found that 84% of developers use or plan to use AI tools, yet more developers actively distrust the accuracy of AI output (46%) than trust it (33%), and only 3% say they "highly trust" the results (Stack Overflow, 2025).
The takeaway for hiring is blunt: AI usage is table stakes; AI skepticism is the differentiator. You are no longer hiring a person to write code. You're hiring an editor-in-chief for a fast, confident, occasionally-lying junior.
4. The five things a coding test can't see
If we accept that the constraint moved, the assessment has to move with it. When I designed the Titan Dev test, I reduced "everything around the code" to five measurable capabilities. Here's the reasoning behind each, and how I make it visible.
4.1 Instruction-following
This is the most underrated predictor in agency work and, in my experience, the strongest single signal of whether someone reads carefully. Client briefs are multi-constraint by nature: use this exact date format, don't exceed this length, hit these eight requirements, in this order. Most candidates can satisfy three constraints. The expensive mistakes come from dropping the fourth silently.
So I test it the way reality tests it: a task with eight simultaneous constraints (a fixed date string, an exact price format, a forbidden-words list, a "second paragraph must be exactly two sentences" trap) where dropping one ball is detectable. I also weave hidden compliance checks through the instructions (a buried "write the word meridian in the header," a required answer order, hard word limits) that are scored silently and only revealed in the report. The candidate who skims will never see them. The candidate who reads will catch every one.
4.2 Reasoning
Code can be generated; the decision about what to build and whether the output makes sense cannot. I grade reasoning on four small tasks: a deduction puzzle (graded on the steps shown, not just the answer), spotting a logical flaw (the classic "every project that used Framework X shipped late, so avoiding X guarantees on-time," which denies the antecedent and dresses it up as strategy), explaining an anomaly (traffic doubled, cost 4×'d, so give two mechanisms and the one data point that would confirm each), and a Fermi estimate where the number is irrelevant and the assumptions are everything.
This maps directly to the reasoning predictors in the research, but framed in the language of the actual job, not the abstract puzzles Google already proved useless (Bock, 2015).
4.3 Reading comprehension
I give candidates a realistic internal policy document (a deployment policy with rules, exceptions, and an exception-to-the-exception) containing exactly one planted internal contradiction. Almost everyone can answer the factual questions ("what's required for a Friday deploy?"). Very few notice when the document disagrees with itself ("hotfixes require two approvals" vs. "hotfixes may deploy at any time without additional sign-off").
That noticing, the realization that the spec is internally inconsistent and should be flagged before any work starts, is exactly the skill that prevents a week of work in the wrong direction. It's invisible on a coding test and obvious in a comprehension test.
4.4 Curiosity
Curiosity is the hardest to fake and the easiest to spot once you stop rewarding polish. I ask open questions where specific, honest answers score well and polished-but-hollow ones score poorly: the last rabbit hole you actually went down; three questions you'd want answered about why HTTP status 418 "I'm a teapot" exists in an official-looking standard (it really does: RFC 2324, the Hyper Text Coffee Pot Control Protocol, an April Fools' RFC from 1998 that the community later fought to keep); something you believed a year ago and no longer do.
The teapot question is a curiosity check dressed up as trivia. The incurious answer treats it as a fact to memorize. The curious one immediately asks who proposed it, what process let a joke into a standard, and why it survived, which is exactly how a good developer reacts to anything strange in a codebase. When the status code was nearly dropped from some libraries in 2017, developers rallied behind a "Save 418" campaign to keep it, which tells you how much engineers value a little play in their standards.
4.5 Working with AI (verification)
This is the only section where AI use is required, because pretending people won't use it is theater. I test three things: can you find planted errors in an AI-written explanation (for example, "localStorage is encrypted by default and synced across devices and sent with every HTTP request," which mixes three confident falsehoods with two true facts); can you write a real prompt with the constraints, context, and jurisdiction a task actually needs; and can you describe a time AI was confidently wrong and how you caught it.
The last one has a deliberate trap. "I've never caught an AI being wrong" is not a flex. It's a red flag about verification habits, and it's scored as one. As the METR result suggests, the developer who never catches the AI is usually the one being slowed down by it without noticing.
5. What I built, and why
The artifact that came out of the Fable 5 window is a single self-contained assessment: a 90-minute, browser-based "exam" with six sections mapped to the five capabilities above, plus a layer of hidden instruction-following checks. It's styled deliberately like a paper exam booklet, because the format primes a different mode of attention than a code editor does.
The design principles, for anyone wanting to build their own:
- Test the job, not the trivia. Every task is a stand-in for real agency work (a client email with constraints, a messy data list to clean by rule, a policy doc to interpret), not an abstract puzzle.
- Make failure visible. Multi-constraint tasks and hidden checks mean a careless candidate produces a detectably careless artifact, rather than a plausible one.
- Grade against a written rubric, every time. This is the "structured" in structured assessment, the highest-validity factor in the research literature (Schmidt & Hunter, 1998; Sackett et al., 2022). Deterministic checks (formats, word limits, the multiple-choice item, the hidden compliance rules) are scored mechanically; open answers are graded by an AI model against the same rubric a human evaluator would use.
- Separate red flags from the score. Quietly proposing to hide a client-affecting mistake isn't a "minus two points" event. It's surfaced on its own, because integrity failures matter more than the total.
- Treat it as a first read, not a verdict. The output is a structured report that tells you who to interview and what to probe live, never an automated hire/no-hire.
I want to be precise about what this is: a proof of concept for a hiring philosophy, not a SaaS product and not something I'm selling. It runs entirely client-side, it's built for my own funnel at Titan Dev, and the whole point is that the philosophy is forkable even if the code isn't. Swap my questions for yours; keep the principle: hire for judgment, reading, and verification, because the code is no longer the test.
6. How you grade judgment
"But judgment is subjective" is the usual objection, and it's wrong in the same way "interviews are subjective" is wrong. Interviews are subjective when they're unstructured. The research is unambiguous: structure is what converts a soft signal into a valid predictor (Sackett et al., 2022). You operationalize it like this:
- Write the rubric before you see any answers. For each question, define what a full-mark answer contains. Example: the anomaly question earns points for each plausible mechanism paired with a specific confirming data request. An explanation without a data request caps low, by rule, not by mood.
- Score behavior, not personality. Situational-judgment items ask what someone would do (you find a flaw at 17:30 the night before launch and the PM has gone home; what's your order of operations?). You grade the observable proposed action, not an inferred trait. This keeps it a structured behavioral interview, which is exactly the high-validity instrument, rather than a covert personality test, which isn't.
- Make the deterministic stuff deterministic. Anything with a right answer (formats, dedup rules, the multiple-choice item, whether a word limit was respected, whether the hidden instructions were followed) should be scored by code, not by a tired human at 6pm.
- Use AI to grade open answers against the rubric, with feedback. A consistent model applying one written rubric across every candidate is more even-handed than a human panel drifting over a long day, provided you treat it as assistive and audit it. Early evidence on LLM-as-grader reliability is promising but still emerging, so the prudent stance remains "consistent first read, human second look."
- Let red flags override the number. A 78/100 with "proposed silently fixing a client-affecting bug without telling anyone" is not a pass. Encode that explicitly so a strong test-taker can't average their way past an integrity problem.
The output you want isn't a leaderboard. It's a marked report per candidate: section scores, the hidden checks revealed, per-question feedback, and any flags. It's a document you walk into the live interview holding, knowing exactly where to push.
7. Honest limitations
A research-style argument owes you its weaknesses, so here they are.
- AI grading of open answers is consistent but not infallible. It's a structured first read, not a final judgment. Always pair it with a live conversation.
- Client-side anything is inspectable. A determined candidate could read the page source. For real rounds, rotate the specifics (names, numbers, the planted contradiction) between hiring cycles and never reuse a question set you've published.
- Situational judgment measures stated behavior, not proven behavior. Someone telling you they'd disclose the mistake is a signal, not a guarantee. It earns them the interview where you find out for real.
- This isn't a substitute for a work trial. The highest-validity approach remains structured assessment plus a paid, realistic work sample on actual agency-style work, ideally watching how they collaborate with AI on it.
- Self-report bias is real in the curiosity and working-style sections. The hidden compliance checks exist partly as an objective counterweight to the parts that rely on what people say about themselves.
None of these undo the core argument. They just keep it honest: this replaces the wrong test with a better one, not with a perfect one.
8. Conclusion
The hard truth for anyone hiring developers in 2026 is that we spent forty years getting good at measuring the one thing that just got automated. The whiteboard, the algorithm round, the speed-of-typing proxy: they were always rough stand-ins for judgment, and the stand-in finally broke.
The research was pointing this way long before the models arrived. Structured, job-relevant assessment of reasoning and behavior has out-predicted technical gauntlets since at least 1998; we just preferred the gauntlet because it felt rigorous. AI took away the comfort of that illusion. The developer who'll make your agency money now is the one who reads the brief completely, reasons about the system, notices the contradiction, stays curious about the weird thing, and, above all, never quite trusts the machine that's making them fast.
So test for that. Not because it's fashionable, but because it's what the evidence said all along and what the next month of every engagement will actually demand.
I built my version of this test in a borrowed window with a model that may never be publicly available again. The model was extraordinary. But the most valuable thing it produced wasn't code. It was a 90-minute argument, in the form of an exam, that code is no longer the test.
If you run an agency and you're rethinking your hiring funnel, I'm always up for comparing notes. Reach out here.
References
- Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274.
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068.
- Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590.
- METR. (2025, July 10). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
- Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410.
- Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134.
- Google re:Work. Project Oxygen and Project Aristotle. https://rework.withgoogle.com/
- Bock, L. (2015). Work Rules!
- Stack Overflow. (2025). Developer Survey: AI section. https://survey.stackoverflow.co/2025/AI
- Anthropic. (2026, June 12). Statement on the US government directive to suspend access to Fable 5 and Mythos 5. https://www.anthropic.com/news/fable-mythos-access
- Masinter, L. (1998). RFC 2324: Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0). IETF.
