How it works
A decision model answers in milliseconds. Whether it was right shows up hours or months later. trimset keeps those two together and turns them into thresholds you can defend.
01 · Capture
Point your app's base URL at trimset's proxy, or send each decision to its logging API. trimset forwards the request unchanged, returns the model's answer, and records the probabilities, the model version that really answered, latency and cost. The state itself is kept only as a keyed hash.
02 · Join
Send outcomes when you learn them: a chargeback, a reviewer's verdict, the team that finally closed a ticket. Name the decision by its id or by your own reference, such as the refund id. An outcome that arrives before its decision waits for it, a correction keeps the history, and a rule like "no chargeback in 120 days means not fraud" fills in the silence.
03 · Refit
Give each question a target: "at most 2% false passes with 95% confidence", or a cost for each kind of mistake. trimset recalibrates the model's probabilities on the labels you have, per question wording and model version, and proposes thresholds with an escalation band. When the labels aren't enough for the target, it says how many more it needs instead of guessing.
04 · Enforce
A person approves the proposed policy, with the trade-off in front of them. Your app's SDK polls for it and gates locally, with no network call on the request path. If the SDK can't get a policy, it falls back to the default you chose. Rolling back is one command, and trimset keeps watching for drift.
A model is calibrated when "0.8" comes true 8 times in 10. Most aren't. A reliability diagram plots what the model said against what happened; the diagonal is honest.
Two cut points split the probability axis. Below lo the app acts on its own, above hi it acts the other way, and the uncertain middle goes to a bigger model or a person.
Traffic changes, and models change under the same name. trimset compares this week's probabilities with the ones the policy was fit on, with a significance test, so a small sample doesn't raise an alarm. A new model version behind an alias like jev-latest is flagged on its first answer.
"False passes ≤ 2% with 95% confidence" means: on traffic like the labeled data the policy was fit on, the chance that its real false-pass rate is above 2% is at most 5%. trimset gets there with exact binomial bounds, not a rule of thumb.
The promise only covers traffic like that data. From M4, when traffic shifts, trimset says "no guarantee" on the policy instead of keeping a number it can no longer back.
refund_fraud · policy v7
false passes ≤ 2%, 95% confidence holds
after a drift alert no guarantee