Before
- Changes to the agent setup were judged from isolated observations.
- Recurring mistakes were found during manual review.
- The team could not compare configurations consistently.
Back to case studies
Evaluation and benchmarking
Agent success reached 87%
2 teams · 18 developers
Two teams improved coding-agent correctness using a benchmark built from their own development work. Repeatable evaluation showed which changes helped before the teams rolled them out.
Impact
- Verified agent success
- 87%
- Improvement in success rate
- 29 points
- Fewer failed tasks
- 69%
Repeatable evaluation helped the teams turn failures into targeted improvements.
The challenge
Agents produced convincing changes that could miss edge cases or break related behavior. Recurring mistakes were often found during manual review.
Developers adjusted prompts and switched models based on isolated observations. They lacked a consistent way to assess whether a change improved overall reliability or created a new regression.
What we changed
Build a representative task dataset
Cortiscale assembled tasks from historical bugs, small features, and refactors. Each task had a reproducible starting state, a clear request, and checks for the expected behavior. Sensitive data was removed. The teams used a development set to improve the setup and reserved unseen tasks for the final comparison.
Make evaluation repeatable
Agents ran in isolated environments that recorded patches, test results, time, and cost. A correct solution had to pass task-specific behavior and regression checks. Environment failures and flaky tests were flagged separately, and developers reviewed ambiguous outcomes.
Use failures to guide controlled improvements
A dashboard grouped failures by task type and cause, including missing context and incorrect assumptions. The teams improved instructions, tool behavior, and verification, then reran the benchmark. New configurations had to pass regression checks before rollout.
Before and after
After
- A shared benchmark showed which changes improved correctness.
- Repeatable checks exposed failure patterns before rollout.
- The same held-out tasks, checks, and run budget supported the final comparison.
Results
Verified success reached 87% on the held-out benchmark, improving by 29 percentage points. Failed tasks fell by about 69%. The team used the results to improve instructions, tools, and verification.
What changed for the team
The teams gained a repeatable way to improve agents already in use. Quality decisions moved from isolated impressions to measured task performance, with regression checks and developer review before rollout.
Start a conversation
What would you like your team to achieve with AI?
Tell us how your software team uses AI today and what you’d like to improve. We’ll discuss whether tailored training or an ongoing collaboration fits.