Most AI pilots never scale. Here's how operators are turning one-off AI demos into production systems their teams can actually depend on.
This piece is adapted from an Operators Guild Focus Session, Demo to Production: Making AI Builds You Can Depend On, the first session in the OG AI Playbook series. The session was led by Geetha Rajan, who leads AI adoption and strategic operations at Freshworks, Justin McCarty, co-founder of Bungalow and most recently Fort Mason, and Paul Seufert, VP of portfolio operations at a private equity firm.
Focus Sessions are small-group, member-only conversations where operators compare notes on decisions in flight, pressure-test tradeoffs, and surface the operational realities that rarely show up in polished frameworks.
If you want access to sessions like this, including the recordings and the community behind them, you can apply to join OG.
Almost every company has an AI win by now. Research shared in the session found that 78% of enterprises have deployed AI in at least one function. Only 22% report tangible value from it, and only 4% are scaling it across the organization. Meanwhile, pilot abandonment jumped from 17% to 42% in a single year. Teams aren't learning from failure. They're repeating it faster.
Adoption is no longer the hard part. The hard part is what happens after the demo works. Getting from the first successful run to the thousandth is an execution problem, and it breaks in predictable places.
Find your binding constraint before you launch anything
When senior leaders were asked where scaling breaks, 78% pointed to the middle of the organization. Strategy exists at the top. Willingness exists at the front line. The translation layer in between is where execution stalls.
Before launching a pilot, diagnose which constraint is actually holding you back:
- The black box. High ambition, but nobody can explain how the work gets done. The process lives in tribal knowledge, so AI has nothing to attach to. Map the logic first
- The invisible break. Leadership is bought in, but middle managers quietly resist because their influence is measured in headcount. Realign incentives before buying more seats
- The broken pipe. Good tools that don't talk to each other. If a human is copy-pasting AI output from one system into the next, you don't have a production workflow. You have a demo on life support
For workflow automation, the broken pipe is usually the killer. Every manual handoff is a point of failure. The fix is orchestration, with data moving as infrastructure.
Segment your workforce, because rollouts fail as one-size-fits-all
Most rollouts treat the whole workforce as a single population: universal training, an org-wide tool launch, the same message to everyone. That fails because people start from very different places. The research surfaced four cohorts:
- Champions (~20%). Already using AI, often without permission. Deploy them as infrastructure rather than celebrating them as mascots
- Skeptics (~25%). Risk aware, and they've watched initiatives fail before. Evidence moves them, enthusiasm won't
- Experts (~15%). Deep capability. Point them at your hardest integration problems
- Explorers (~40%). Curious, willing, and genuinely unsure where to start. This is the swing vote. Give them structured experiments, guardrails, and protected time, and adoption spreads. Ignore them and the pilot stalls with whoever built it
Get the data layer in order before an agent touches anything
One operator in the session ran a property management business where a simple owner question could burn ten hours of team time, bouncing across email, phone logs, a leasing channel, Slack, and property management software. Six systems, no shared context.
The first build was a single context layer that pulls every system into one queryable database, serving both the team and the agents. Agents perform surprisingly well when you ask "what would you do next?" and they can see everything. They perform poorly when the context is scattered.
A few practical notes from that build: it started in Airtable and later moved to Supabase, and any Postgres database works. Enrichment automations matter, like auto-tagging summaries with the relevant owner or property, since models are reluctant to search long bodies of raw text on their own.
Treat your internal tool like a production system
The context layer took a week to vibe code. Then it broke every day. The team couldn't count on it, duplicate work crept back in, and the person who built it became a part-time QA engineer.
The fix was borrowing standard engineering reliability practice:
- Freshness monitoring. A dashboard and Slack feed showing whether every pipeline is online and current
- Canaries. Hourly test runs that mimic what a real user or agent would do, like "this owner is calling, what will they ask about?"
- Auto-logged tickets. Anything red creates a ticket automatically, in the same queue an engineering team would use
- Runbooks. Any issue seen twice gets a runbook: what went wrong, what to verify, how to fix it. The runbook attaches to every matching ticket
The runbook system is what changed the economics. With a playbook attached, agents resolve 90 to 95% of tickets on their own, follow the process instead of improvising, and can run on cheaper models. After a few hundred tickets, the issue types converged to about 19, with no new ones appearing.
The results: team usage up 100x, customer resolution times down from days to minutes, and retention moving from middle of the pack to top decile.
Match the architecture to the task
Most agents on the market are React agents: a loop that reasons about the problem, takes an action, and reasons again. They're flexible, fast to launch, and great for ambiguous, unstructured problems.
The trouble shows up on long, structured processes. A React agent that's 95% accurate per step sounds good until you chain ten steps and watch total reliability collapse. They're also expensive, since they need frontier-level intelligence and burn tokens re-deciding what to do next, even on processes they've run twenty times.
The alternative is an agentic state machine: a predefined flowchart of steps, with a specialized agent persona owning each one. One session speaker built a diligence system this way that ingests confidential information memoranda and produces a 60-page deck plus a fully built Excel model in the firm's exact templates, consistently, run after run.
The state machine tradeoff is real. Speed to launch is meaningfully slower, since you're building a production line where every station is purpose-built. In exchange you get precision, predictability, clean handoffs to deterministic code, and debugability that feels like normal software. When an output was wrong, the builder traced it to a single data compression step, fixed it, and the whole system healed. You can also right-size the model per step, using lightweight models for mechanical work and reserving frontier intelligence for the steps that need it.
Two patterns worth stealing:
- The adversarial reviewer. A fresh model call, with no conversation history, reviews each output against explicit guidelines and sends it back until it passes. A focused reviewer catches what a loaded-up prompt misses
- The SOP as north star. The production diligence system is anchored to a 40-page, human-readable process document owned by the deal team. The technical build implements that document, node by node. One team built theirs by recording a training session, having Claude turn the transcripts into an SOP, and letting the team edit it
This is also where operators have a real edge. Disaggregating a process into well-scoped steps depends on domain knowledge, knowing how the work actually flows, what good looks like, and where the nuance hides. That's operating experience applied directly to a technical system, and it's an advantage no outside builder starts with.
Measure production, and watch for the efficiency illusion
Demo metrics and production metrics are different animals. In production, the numbers that matter flip:
- Task volume becomes zero-touch resolution: did it finish without a human?
- Handle time becomes resolution quality: did it actually solve the problem?
- Logins and training completion become voluntary adoption: does usage climb when nobody's watching?
- Compliance becomes escalation ratio: humans should handle the 10 to 15% of cases that are genuinely ambiguous
One trap to watch: revenue per employee rises, everyone declares victory, and true productivity stays flat because the cost just moved from the labor column to the software column. The number that signals real production scale is zero-touch completion holding steady as volume climbs.
The teams reaching that thousandth successful run aren't there because they picked a better model. They diagnosed their constraint, segmented their rollout, built reliability infrastructure around their tools, and measured outcomes over activity. The technology has been ready for a while. Execution is the gap.
Join the conversations operators actually need
This session was one example of the work happening inside OG every day.
If you want access to sessions like this one, including the recording, the discussion, and the operators comparing notes on what's working across their companies, consider becoming a member.
OG is where senior operators go to learn from peers, pressure-test decisions, and trade the practical insights that help teams and companies move faster.
If you're ready to be part of a community built for operators, you can apply to join OG today.