scoreComing soon

Model routing

A chat product answers easy requests with a small, cheap model and hard ones with a large one. A hard request sent small comes back as a bad answer; an easy one sent large costs money on every call.

“How hard is this request?”

The request · POST /v1/systemone

{
  "model": "jev-latest",
  "state": {
    "message": "…",
    "history_turns": 3
  },
  "questions": {
    "difficulty": {
      "type": "score",
      "instructions": "How hard is this request?",
      "criteria": ["Trivial", "Easy", "Hard", "Expert"]
    }
  }
}

The policy

P(level ≥ 2) < tACCEPTthe small model answers
otherwiseESCALATEthe large model answers

target

The lowest expected cost per 1,000 requests, with a large-model call at $0.012 and a bad small-model answer at $0.20 (illustrative).

t = [X]. A cost target finds the cheapest point, not a guarantee, and the policy says so.

When the truth arrives

nowThe small or the large model answers.proxy
minutesA thumbs-down or a retry within 10 minutes of a small-model answer.downstream
minutes to hoursAn LLM judge grades a 2% sample with the true level.downstream

In your app · Python SDK, gate() from M3

d = ts.gate("difficulty", answer=resp["answers"]["difficulty"],
            model_served=resp["model"], decision_id=ids[0])
model = small_model if d.action == "ACCEPT" else large_model
reply = model.answer(message)

[ dashboard screenshot ] Expected cost per 1,000 requests across thresholds, from the demo.

This use case is on the way. The roadmap shows where trimset is.

See the roadmap