OPEN TOKEN MODEL MIGRATION LAB
Before you switch models, test the candidate on your production workload.
Compare your current model with a candidate using real cases, the same evaluation criteria, and the cost of each alternative. Make the migration decision with evidence before moving production traffic.
- 01Capture
- 02Dataset
- 03Criteria
- 04Candidate
- 05Decide
The best model on a public benchmark may be worse for your product.
Public benchmarks don't know your prompts, users, output constraints, or edge cases. A model migration should answer three questions. Open Token compares both alternatives on the same cases using the same evaluation criteria.
Does quality hold?
Where do regressions appear?
What happens to the actual cost of the workload?
Turn real production traffic into your private benchmark.
Select normal cases, failures, and edge cases from captured traces to build a dataset that represents how your product is actually used.
A model migration needs evidence, not a demo.
The candidate runs through the same gateway that already measures your production. The comparison is about your workload, not prompts written for a demo.
Capture
Capture real cases
Select a representative production sample, including normal cases, errors, and edge cases.
PRODUCTION TRACES
Base sampleErrors, loops, and outliersSAMPLE DATA — FICTITIOUS
Dataset
Build the dataset
Turn those traces into a reproducible set of cases for the evaluation.
DEMO DATASET · 240 CASES
case_0142normalcase_0187failurecase_0203edge casecase_0219normalSAMPLE DATA — FICTITIOUS
Criteria
Define what “better” means
Combine deterministic rules with criteria scored by a judge.
DETERMINISTIC CHECKS + JUDGE RUBRICV3
JSON · schemacheckLanguage · lengthcheckConfigured rulescheckVersioned rubricjudge · thresholdSAMPLE DATA — FICTITIOUS
Candidate
Run the candidate
Open Token replays the main model call against a candidate supported by your gateway configuration.
MAIN MODEL CALL
Same casesSame criteriaDifferent modelSAMPLE DATA — FICTITIOUS
Decide
Make the decision
Compare quality and cost case by case, along with sample size, errors, and how uncertain the difference is.
Decide
Quality and costSample sizeErrors and uncertaintySAMPLE DATA — FICTITIOUS
Is the improvement enough to justify the migration?
You don't get a single opaque score. See which model wins case by case, how cost changes, and how much confidence there is in the observed quality difference.
Current model
US$0.0124
Candidate
US$0.0087Cost change −29.8 %
Hard rules where rules are hard. A judge where judgment matters.
Deterministic checks
JSON, schema, format, language, length, and rules you configure.
Judge with a rubric
Versioned criteria for attributes a deterministic check cannot score.
Calibration
Compare the judge’s score with numeric or boolean feedback.
Control coverage and spend.
How many traces you include and how many cases a judge scores are separate controls. You can widen coverage without the evaluation cost growing at the same rate. Errors and relevant cases can be kept in the sample on purpose.
Trace sampling
Judge sampling
- Errors
- Loops
- Outliers
Content remains opt-in.
Prompt and response capture is off by default. When you enable it for a project, configured redaction, retention, and access policies apply before those traces are used in an evaluation. When traces are stored, redaction removes valid CUIT, CUIL, and CBU values. That is storage redaction, not the PII evaluator.
Opt in per project
Redaction before storage
Configurable retention
Audited access
SAMPLE DATA — FICTITIOUS
Design the pilot before touching production.
Custom · Talk to us
A migration evaluation is scoped around four things: your workload, the candidate, the quality criteria, and the evaluation budget. Storage, candidate calls, and judge calls stay separate.
INCLUDES
- Sample design
- Rubric and checks
- Current/candidate run
- Quality and cost report
- Results review
What the pilot covers
What runs again?
The evaluation replays the main model call for the selected case.
What is compared?
The current model's output and the candidate's, using the same evaluation criteria.
Is this available as a self-serve product?
No. Model Migration Lab is a pilot with an agreed scope, run with Open Token's team.
Which models can be evaluated?
Models from your own provider connections (BYOK) that your gateway configuration already supports.
Decide the migration with evidence from your production.
Tell us what you run today, which candidate you're considering, and what it would need to achieve for the migration to make sense.
Model Migration Lab is a pilot run with Open Token's team, on your own traces and your BYOK models through the gateway.