Skip to content
Ascenda

How do we test whether switching AI model or vendor helped our engineers?

Short answer

Don't switch everyone at once. Roll the change out to groups one after another, so the groups still on the old setup act as the comparison at each step. Then compare a measure you chose in advance. Ascenda runs this as a staggered trial and reports whether the difference is real or too small to tell.

Why it matters

When everyone switches on the same day, anything else that changed that month gets mixed into the result: a release crunch, a new hire, the end of a quarter. A staggered rollout separates the switch from the calendar.

How Ascenda helps

  1. Declare the switch, the groups, the order and the measure before starting. Failed tool calls per active hour is one of the measures you can choose.
  2. Roll out to one group, then the next.
  3. Read out: Ascenda compares each group that has switched with the groups that haven't yet. Because of the staggered design, the readout can say whether the switch caused the difference.

When the difference is too small to tell apart from noise, the readout says so instead of picking a winner. Each group is reported as an aggregate of ten or more, and a team result is never read as any one engineer's.

Next step

Start with a 30-day Report

The Report works from your engineers' imported history for a cohort of ten or more, with no integration, and ends in a written deliverable. The 90-day Study is the next stage and runs live.

Book a StudyGet Flow for Mac

Related questions

How do I know whether rolling out Claude Code (or another AI agent) helped my team?

Declare the change, the measure and the window before you roll out. Ascenda freezes the baseline and reports the readout whichever way it goes.

How much of our team's AI agent work goes through the approved model or vendor?

Declare your approved model vendors and routes; Ascenda reports what share of the cohort's agent time went through them, as an aggregate of ten or more.

Does a different model or effort level change how my AI coding goes?

Flow can assign which model or effort level you use on each working day, in pairs, then lay the failed tool calls side by side, so you compare on your own work.