Developing a mature AI-first software development (AIFSD) practice involves shaping how AI participates in engineering work. Models provide capabilities, while agent harnesses support tool use, context management and execution. Developers also define their engineering practices: how requirements are understood, knowledge is captured, implementation proceeds and results are verified. We call these practices workflows.

As workflows become part of engineering practice, they need a way to evolve through evidence. How can developers establish whether their workflows deliver the intended benefits, understand their shortcomings, and evaluate improvements?

Workflow challenges

Workflows express expectations about how engineering work should proceed. Writing a specification may help clarify requirements. Planning may help organize implementation. Reviewing intermediate results may help catch mistakes earlier. Established workflow approaches organize these practices in different ways. Whether they improve delivery, and whether their benefits justify the additional effort, requires evaluation.

We built SpecBench, an evaluation platform, to support this. It runs configured workflows on defined tasks, independently assesses their outputs and records the process and resources used to produce them. Developers can use this evidence to compare approaches, investigate failures and test revisions.

The purpose is to make workflow development measurable. A team can state what a workflow should improve, examine whether that improvement appears, and use the results to guide its next change.

Our methodology: Controlled evaluation from task input to evidence

SpecBench evaluates workflows through execution, independent assessment and recorded evidence. We use four scenarios, frontend, backend, fullstack, command line application, to represent different kinds of everyday software development. 

Comparison controls. Let W denote a configured workflow and x a task input. Executing the workflow produces an artifact, a, an execution record, τ and resource measurements: u:

(a, τ, u) = Run(x, W; M, H, C)    (1)

Here, M is the generation model and variant, H is the agent runtime and C contains the execution conditions, including the accepting threshold and retry policy.

A controlled workflow comparison varies W while holding x, M, H, C constant. This compares complete workflow configurations. Identifying the contribution of an individual practice requires a further comparison that changes that practice while preserving the others.

Independent evaluation. SpecBench assesses the resulting artifact and execution record using a fixed evaluation configuration E:

rW = Evaluate(a, τ, u; E) = (QE(a), PE(τ), u)    (2)

Here, QE measures artifact quality, PE summarizes the execution process where supported, and u reports resource use.

SpecBench retains the configuration, artifacts, execution records and measurement status so that results can be inspected and failures investigated. Independent repetitions are needed to estimate variation; retries within one evaluation serve a different purpose and must be reported separately.

Using SpecBench evidence to improve workflows

SpecBench results help developers examine three questions: 

• What a workflow achieved.

   

• What effort it required.

   

• What should be investigated next.

   

The main results should therefore present artifact quality alongside resource use, with scenario-level results showing where outcomes differ.

Using historical Spec-to-Code evaluation data across several test runs, the tables below demonstrate how this methodology functions in practice. W0 is the result of a sample run using the direct implementation baseline. W1 is another sample run that uses specification, planning, task decomposition, implementation and convergence stages. Both use the recorded generation model gpt-5.6-luna[max]. These historical results provide descriptive evidence; matching the model label alone does not establish identical execution conditions.

W1 records higher behavioral correctness across all four scenarios, with a 2.75 percentage-point increase in the mean. Its overall score increases by 0.71 points, while mean execution time nearly doubles and token consumption increases substantially. The result raises a practical question: whether the additional correctness is worth the additional effort for the intended task.

Developers can use this evidence to identify the next experiment. Failed behaviors and execution records can guide investigation into where requirements were lost, implementation went wrong, or verification fell short. A revision can then target a specific weakness and be evaluated under matched conditions, checking both improvements and regressions.

For workflow authors, this supports validation and revision. For adopting teams, it helps assess quality and resource trade-offs. Shared evaluation protocols could extend these benefits across the industry by making workflow claims easier to reproduce and compare.

Observations, current coverage and outlook

As models develop, we notice excessive behavioral workflows or harnesses may interfere with the performance of frontier models, especially when compared to the improvements they bring to lower-level models.

SpecBench currently evaluates four scenarios through Spec-to-Code and Code-to-Spec. It does not yet evaluate workflows for modifying, extending, or maintaining brownfield projects.

The test cases and experimental data are limited, and many frontier models are not covered. The current scoring algorithms may also be insufficiently sensitive to distinguish some models or workflows in particular scenarios. These limits leave room to improve both the breadth of evaluation and the measurements themselves.