Decision Models Judge. They Don't Forecast.
Most of the conversation about AI is about models that write. A newer kind writes nothing at all: you hand it some text and a narrow question, and it hands back an answer with a probability attached. We spent three days putting one of these decision models to work on three jobs. It earned two of them, and why it didn't earn the third is the useful part.
By Ivaylo Tsvetkov, Co-Founder
A model that doesn't write
The model was Jev, from TypeSafe, available through OpenRouter. It answers three kinds of question: pick one of these options, place this on a scale, or say whether a condition holds. Every answer comes back as a structured value with a probability for each possible outcome, and nothing else: no prose, no explanation, no reasoning trace. It is built for the spot where you would otherwise ask a chat model a narrow question and parse a label out of its reply: sorting a ticket, checking a rule, ranking a list. It is also priced like infrastructure, at about four cents per million input tokens, with output free. At that price the cost of trying it is engineering time, not the bill.
Jobs one and two: putting the right result first
Search is the obvious fit. We run two search systems: one over a shared store of short team notes, and one over several hundred pages of technical documentation. Both already had a first pass that blends keyword matching with semantic search, and both returned a reasonable top ten. The problem was order. The right answer was usually in the list but not always at the top, and people and AI agents mostly read the top. So we added a second pass. Jev reads the question next to each of the ten candidates, grades how well that candidate answers it, and the list is reordered by grade. How we asked mattered: grading each candidate on its own, with all ten calls in parallel, beat asking the model to pick the best of the ten in one call. Every call has a deadline, and when the grades don't arrive in time the search keeps the order it already had.
What the second pass bought
We tested it the way we test anything we might ship. We used questions the system had never seen, and wrote down what would count as a win before the first run. On the notes search, which was already good, the share of searches with the right note in first place rose from 92 to about 96 percent. On the documentation search, the right page came first on 92 of 120 test questions, against 61 without the second pass. On the notes search we had tried the textbook alternative first: a small open-source reranking model running on our own hardware. It cost nothing per call, but it did not clear the same bar, and it took around six seconds per search. Jev adds between half a second and a second and a half per search, and about four hundredths of a cent. Both second passes are now live.
Job three: choosing the model for each task
The third job looked like the same kind of decision. Like many teams, we run AI coding agents on a fixed default: a strong model at a high reasoning setting for all coding work. A router that read each task and sent the easy ones to a cheaper model, or to less reasoning, would save money on every easy task. Routing is on Jev's own list of uses, so we built one. It had guardrails: a person's explicit choice always wins, the most expensive settings are never picked automatically, and any error or unsure answer falls back to the default. Then we ran it over about 150 real task descriptions from our own history, three times each. It passed every safety rule we had written, and it almost never chose the cheaper option. For more than a quarter of the coding tasks it asked for the highest reasoning setting on offer, and for most of the rest it agreed with the default. Switched on, it would have raised the bill.
Does the setting even matter?
A backtest shows what a router picks, not whether the picks are right. So we replayed six finished coding tasks from their exact original inputs at three settings: the default, the same model with more reasoning, and a cheaper model. Each replay was scored by the automated tests the original work had been accepted against. The extra reasoning took 1.5 to 2.2 times as long and bought nothing measurable. The cheaper model matched the default on three tasks and fell short on the other three. The router's confidence that a task needed the stronger model was right on three of the six, which is chance. Even the same setting on the same task moved by up to nine points of test pass rate between replays, so only large, consistent differences count. The router stays switched off. The replay setup stays in use: it is how we will measure the next model release before we change a default.
Judging versus forecasting
The two jobs Jev won have one thing in common: the evidence was in the input. Whether a note answers a question is a judgment about two pieces of text the model can see. The job it did not win asks for something else, a forecast of how a different model will perform on work nobody has attempted yet. Nothing in a task description says that. Ticket routing, the textbook use, fits the first shape: which team should handle a ticket is written in the ticket. Which model will succeed at a task is not written in the task. That is the line we now draw. A decision model is a good judge of what is in front of it. It is not a forecaster, and a narrow question with a probability attached can make a forecast look like a judgment.
What to take from this
Use a decision model where the answer is in the input: relevance, classification, policy checks, whether an output meets a rule. Be wary wherever the answer depends on something the input does not contain, such as difficulty, future behaviour or outcome. Give it a shortlist rather than the whole collection, and ask one narrow question per candidate. Make it fail open: every call needs a deadline and a fallback to what you had before, which is what made switching it on a non-event for us. Measure value separately from safety, because a router can pass every safety check and still save nothing, and only replaying real work will show which. Expect the top result to stay stable while the rest of the order shuffles from run to run. And decide deliberately what leaves your systems, because every call sends the text being judged to an outside service.
Want to discuss how this applies to your business? Book a free call.
Ready to add AI to your business?
We help businesses identify, design, and deploy AI systems that actually work. Book a free discovery call and see what's possible.
Book a free call →Related Posts
View all posts →AI Chatbots for E-commerce Support: What to Automate First
A practical framework for e-commerce teams deciding what an AI chatbot should handle first: repetitive questions, product discovery, policies, and human handoff.
AI Security Is Now a Buying Issue
Anthropic's Mythos preview and an OpenAI tooling incident say the same thing: buying AI means vetting permissions, connectors, and blast radius, not just the model.
Why a Local Dev Studio Beats a Freelancer for Custom Software
Freelancers are cheap. Agencies are expensive. Boutique dev studios offer something better: senior expertise, direct accountability, and no outsourcing.