We tested Claude Opus 5.5 on the task of creating integration workflows. We gave it a specification and documentation, and let it run.

By one metric, Opus scored about the same as our new “skeleton builder,” which primarily uses Sonnet and Haiku (0.72 against 0.70). But that metric, LLM-as-a-judge scoring, is imperfect. In a broader view, two of its 21 workflows had invalid schemas; it made twice as many data-flow errors; it took nearly three times as long per case, and it cost nearly seven times as much as our system.

Opus is a horizontal model, built for breadth. The skeleton builder is a vertical system. It wraps horizontal models in what we’ve learned about integration. That system, which replaced a previous module, raised the share of generated workflows that pass our acceptance criteria from 78% to 96.8% while cutting build time by 69% and costs by 72%.

Building our vertical AI system took hundreds of experiments, more than 350 batch runs over almost ninety builder revisions, and weeks spent learning where an LLM is useful in our domain, and where it is just expensive.

Horizontal and vertical

Horizontal AI is driven by model providers and general-purpose platforms: Google, Anthropic, OpenAI, other players, and the open-weight community. Their job is breadth. A model has to reason about tax law, Python and poetry with the same weights, and each new generation raises the floor for everyone.

Vertical AI is built by companies that put those models in systems crafted from domain knowledge. This buys efficiency and reliability, not raw capability. These companies determine which decisions in their domain require judgment and which can be handled mechanically. They learn which errors the platform's validators miss and which facts about the runtime were never documented.

Bessemer Venture Partners sees vertical AI as a larger opportunity than vertical SaaS ever was. Two of their ten principles match our experience closely.

• The eighth says competitive edge depends on "combining data types and workflow integrations, not proprietary models alone.”
• The ninth says to build systems that can "flexibly incorporate the best models as AI evolves."

The latter applies directly to our approach. We can replace the models in most of our components as needed. We’ve already done so. A version of our system that predates the ones discussed in this piece used Claude Sonnet 4.6 to build workflows and Haiku 4.5 to test them. Now it uses Sonnet 5.5 and Haiku 5.5. This flexibility allows us to pair each task with the most efficient model available for it.

A good model may solve the problem, inefficiently

A good model can solve a meaningful share of most problems. It gets there inefficiently, and less reliably as the work gets longer, as reported by CompWoB and TheAgentCompany. The problem is not necessarily LLMs’ ability to solve individual steps; it is the size of the search space. The more steps a task has, the more choices an agent has to get right, and a general-purpose agent has few tools to narrow them.

The research community has started to measure this.

• Agents can cost more for no accuracy gain. In AI Agents That Matter, Princeton researchers show that, on HumanEval, several state-of-the-art agent architectures offer no significant accuracy advantage over simple baselines, while incurring substantially higher inference costs.
• A fixed pipeline can beat the agent. Agentless replaced an autonomous coding agent with a fixed localize-then-repair pipeline. It solved 32% of SWE-bench Lite problems, which was comparable to top performers, often at a fraction of their cost.
• The providers say the same. Anthropic's own guidance, Building effective agents, recommends predefined workflows when a task is well understood. It warns that agentic systems trade latency and cost for flexibility.

The test: a frontier model, with (almost) nothing of ours

In October, we tested Claude Opus 5.5, Anthropic's frontier model, as a general-purpose agent through the Claude Agent SDK, on the 21 cases of our main dataset.

We gave Opus what a capable engineer would get: the specification, the connector documentation, and the same validation and test tools our builders call. It had up to 80 turns to build, without our planner, normalizers, data-flow lint, or a definition of “done.” 

We evaluated Opus the same way we evaluated the skeleton builder: on a range of factors, including the average rating given by three independent LLM judges. Opus basically tied the skeleton builder’s performance according to judge ratings, but fell short everywhere else. 

The biggest outcome of this experiment is that Opus costs 6.8 times as much as our builder and takes 2.9 times as long for inferior performance. It produced an invalid schema in two of the 21 cases and twice as many data-flow defects, such as references to fields that don’t exist. It ran tests, but on modified pieces of its pipelines, not on the pipelines it delivered, so those tests can’t tell us whether the delivered pipelines work. This was one run on 21 cases, and our run-to-run variance is about one case, so read the 0.02 judge gap as a tie. 

Where the efficiency actually came from

Our skeleton builder went through several distinct phases. Most of the gains came from decisions that only make sense if you know the domain.

Prefer code over prompts for anything mechanical. Prompt instructions failed to fix structural defects. Normalizers, sanitizers, and acceptance gates in code worked consistently. Our rule became simple: if a check can be expressed in code, it does not belong in a prompt.

Order the work by data dependency. Data-flow defects were our largest source of failures: steps reading fields that previous steps had replaced or never produced. We now fill the steps that produce data before the steps that use it, in dependency-aware waves. That cut first-fill defects by 65%

Use external feedback, not self-correction. Research from Google DeepMind shows that models struggle to correct their own reasoning without external signals. Our builder relies on platform validation, static data-flow linting and instrumented engine tests. LLM-generated corrections are accepted only when these deterministic checks do not regress.

Route models based on task complexity. Once the architecture was made per-step filling mechanical, switching from Sonnet to Haiku cut costs by a further 39% with no measurable quality loss. Using Haiku for correction or credential assignment doubled builder failures. This is consistent with Stanford's FrugalGPT, but our results add an important condition: model routing pays off when domain engineering has first made the task sufficiently predictable.

Specialization moved from the weights to the system

There was a period when domain specialization meant training a domain model. Bloomberg trained BloombergGPT on financial data. A few months later, an EMNLP study found that general-purpose GPT-4 beat it on four out of five financial benchmarks tested. Microsoft's Medprompt work showed something similar in medicine: careful prompting and orchestration lifted a generalist model past models fine-tuned for the domain.

The conclusion is not that domain knowledge stopped mattering. It moved. It now lives in what Andrej Karpathy describes as an emerging thick layer of software that coordinates individual LLM calls: what goes into each context, in what order, under what constraints, and how each result is checked. That layer is where a vertical company accumulates advantage.

It is also where the hardest-won knowledge sits. Much of how a platform behaves is undocumented, and therefore not in any model’s training data. What does a step return when it fails but the flow keeps going? What does a template write for a field that does not exist? A minimal pipeline run on the real engine settles each of these questions in minutes.

One of those answers alone changed our builder. A failing transformation does not stop the flow by default, so a duplicate check that had silently failed was sending every record down the “create” path, and our tests still passed. None of our LLM judges caught it, because none of them knew how the engine behaves. When documentation runs out, an experiment is the only reliable source of truth.

Experimentation is the product

The hundreds of experiments we conducted were not overhead. They were an essential part of the engineering process: how we discovered which assumptions were wrong, which approaches actually worked, and, surprisingly often, where our own measurement instruments were misleading us.

Over the course of the project we found more than twenty defects in our instruments:

• Token counts that accounted for only one model.
• Missing pricing data for newly introduced models.
• An LLM judge whose evaluation rubric changed midway through the study.
• Metric denominators that systematically favored one builder. 

Many of these discoveries were subtle. Our test pass rate counted only cases that reached the testing stage, meaning that a builder’s worst failures (the ones that didn’t pass validation) never entered the average. This made the previous version of our builder look better because it failed earlier.

LLM judges also proved insufficient for detecting mechanical defects. A pipeline with a broken deduplication step received the same score as its corrected version.

Every domain has its own failure modes, sources of noise, and measurement blind spots. Identifying them requires continuous experimentation with real-world cases, domain knowledge, and engineering judgment to distinguish genuine improvements from measurement artifacts.

Experimentation is not merely how we evaluate the product. It is part of how we build it.

Why not let an agent do the optimizing?

A natural objection: if the problem is a large search space, automate the search. Let an optimizer, or a meta-agent, try configurations until it finds the best one.

The numbers don’t work. Choosing which of five models to use in each of six phases of our builder, for example, would mean testing 15,625 configurations. That’s before changing a single prompt, gate, documentation mode, or retry policy.

A full evaluation run across our two datasets costs about $30. Run-to-run variance is roughly one case per run, meaning we need at least five replicas per configuration to separate differences from noise. A brute-force sweep across that single dimension would cost around $2.3 million. Smarter search strategies could reduce that cost, but not enough. Moreover, the tests would still optimize against evaluation instruments that we later discovered were flawed.

Recent research points the same way.

• Automated agent design is expensive. Automated Design of Agentic Systems (2025) reports that a single meta-agent search costs hundreds of dollars on small academic benchmarks.
• It rarely pays for itself. A follow-up study, Inefficiencies of Meta Agents for Agent Design, found that the cost of designing an agent this way was recovered on only two datasets, and only after deploying it on more than 15,000 examples. On every other dataset, the gains never covered the design cost at any scale.

Domain knowledge is what prunes the search. One run with Haiku showed nested-function errors that a small model cannot repair. An engineer who understands the platform knew why, and we moved on. Combinatorial search needs a prior. In a vertical domain, that prior is expertise.

What this means

For companies adopting AI, access to increasingly capable and cheaper models will become a commodity. The differentiator will be how effectively those models are applied to real business problems.

This requires deep domain knowledge, integration with real workflows, reliable validation, and disciplined, continuous experimentation.

These are not temporary workarounds until models become better. They are the engineering assets that turn general-purpose intelligence into reliable business outcomes.

[Get access to the first AI-native integration platform →]