Skip to content

OPEN TOKEN MODEL MIGRATION LAB

Before you switch models, test the candidate on your production workload.

Compare your current model with a candidate using real cases, the same evaluation criteria, and the cost of each alternative. Make the migration decision with evidence before moving production traffic.

  1. 01Capture
  2. 02Dataset
  3. 03Criteria
  4. 04Candidate
  5. 05Decide

The best model on a public benchmark may be worse for your product.

Public benchmarks don't know your prompts, users, output constraints, or edge cases. A model migration should answer three questions. Open Token compares both alternatives on the same cases using the same evaluation criteria.

  • Does quality hold?

  • Where do regressions appear?

  • What happens to the actual cost of the workload?

Turn real production traffic into your private benchmark.

Select normal cases, failures, and edge cases from captured traces to build a dataset that represents how your product is actually used.

A model migration needs evidence, not a demo.

The candidate runs through the same gateway that already measures your production. The comparison is about your workload, not prompts written for a demo.

  1. Capture

    Capture real cases

    Select a representative production sample, including normal cases, errors, and edge cases.

    PRODUCTION TRACES

    Base sampleErrors, loops, and outliers

    SAMPLE DATA — FICTITIOUS

  2. Dataset

    Build the dataset

    Turn those traces into a reproducible set of cases for the evaluation.

    DEMO DATASET · 240 CASES

    case_0142normal
    case_0187failure
    case_0203edge case
    case_0219normal

    SAMPLE DATA — FICTITIOUS

  3. Criteria

    Define what “better” means

    Combine deterministic rules with criteria scored by a judge.

    DETERMINISTIC CHECKS + JUDGE RUBRICV3

    JSON · schemacheck
    Language · lengthcheck
    Configured rulescheck
    Versioned rubricjudge · threshold

    SAMPLE DATA — FICTITIOUS

  4. Candidate

    Run the candidate

    Open Token replays the main model call against a candidate supported by your gateway configuration.

    MAIN MODEL CALL

    Same cases
    Same criteria
    Different model

    SAMPLE DATA — FICTITIOUS

  5. Decide

    Make the decision

    Compare quality and cost case by case, along with sample size, errors, and how uncertain the difference is.

    Decide

    Quality and cost
    Sample size
    Errors and uncertainty

    SAMPLE DATA — FICTITIOUS

Is the improvement enough to justify the migration?

You don't get a single opaque score. See which model wins case by case, how cost changes, and how much confidence there is in the observed quality difference.

Evaluation report · Demo dataset · 240 casesSAMPLE DATA — FICTITIOUS

Current model

Average score0.81
Average cost per case

US$0.0124

Candidate

Average score0.84
Average cost per case

US$0.0087Cost change −29.8 %

Wins per case
Current model 61Ties 67Candidate 112
95% CI for the quality difference+0.01 to +0.05
Skipped cases6
Errors2
Sample size240

Hard rules where rules are hard. A judge where judgment matters.

  • Deterministic checks

    JSON, schema, format, language, length, and rules you configure.

  • Judge with a rubric

    Versioned criteria for attributes a deterministic check cannot score.

  • Calibration

    Compare the judge’s score with numeric or boolean feedback.

Control coverage and spend.

How many traces you include and how many cases a judge scores are separate controls. You can widen coverage without the evaluation cost growing at the same rate. Errors and relevant cases can be kept in the sample on purpose.

Trace sampling

LESS COVERAGEMORE COVERAGE

Judge sampling

LESS COVERAGEMORE COVERAGE
ALWAYS RETAINED
  • Errors
  • Loops
  • Outliers

Content remains opt-in.

Prompt and response capture is off by default. When you enable it for a project, configured redaction, retention, and access policies apply before those traces are used in an evaluation. When traces are stored, redaction removes valid CUIT, CUIL, and CBU values. That is storage redaction, not the PII evaluator.

  • Opt in per project

  • Redaction before storage

  • Configurable retention

  • Audited access

CUIT 20-•••••••-•[CUIT_REDACTED]

SAMPLE DATA — FICTITIOUS

Design the pilot before touching production.

Custom · Talk to us

A migration evaluation is scoped around four things: your workload, the candidate, the quality criteria, and the evaluation budget. Storage, candidate calls, and judge calls stay separate.

INCLUDES

  • Sample design
  • Rubric and checks
  • Current/candidate run
  • Quality and cost report
  • Results review

What the pilot covers

  • What runs again?

    The evaluation replays the main model call for the selected case.

  • What is compared?

    The current model's output and the candidate's, using the same evaluation criteria.

  • Is this available as a self-serve product?

    No. Model Migration Lab is a pilot with an agreed scope, run with Open Token's team.

  • Which models can be evaluated?

    Models from your own provider connections (BYOK) that your gateway configuration already supports.

Decide the migration with evidence from your production.

Tell us what you run today, which candidate you're considering, and what it would need to achieve for the migration to make sense.

Model Migration Lab is a pilot run with Open Token's team, on your own traces and your BYOK models through the gateway.