The best AI agents are secretly teams

Ben Tannyhill is a product manager at LangChain, where he's building LangSmith Engine—an agent that finds and fixes your agent's failures. Engine continuously analyzes your production traces, clusters them into actionable issues, and opens pull requests to fix them. Engine's architecture is a lot like an org chart: a main model delegating to a team of cheaper, faster sub-agents. It launched in public beta at Interrupt 2026, and in this conversation, Ben unpacks why it uses a sandbox as a tool, how the team turned it into a self-improving agent that learns from its own traces, and the hard problem of testing a fix before it ships. We also discuss: • Why Engine is "the agent for agent engineers" • Making LangSmith agent-native with condensed trace views • Why the team keeps handing more control to the agent • Inside Engine's four sub-agents: the screener, verifier, and more • Giving Engine memory with an agent overview document • How to keep an always-on agent from blowing the inference budget • Where Insights, Polly, and Engine are converging Timestamps: 00:00 Introduction 01:25 LangSmith 101 02:22 Why Engine is "the agent for agent engineers" 03:49 Under the hood: Engine is a deep agent 06:08 Clustering millions of traces with condensed views 10:10 Why the team keeps handing more control to the agent 13:21 Why Engine uses a sandbox as a tool 14:11 Engine's four sub-agents and the org-chart analogy 16:51 Evals for Engine: IssueBench, Harbor, and synthetic environments 23:05 How Engine evolved: from noisy PRs to an issue inbox 25:56 Inside Engine's memory: the agent overview document 29:25 How to keep an always-on agent from blowing the inference budget 30:52 What models Engine uses 31:30 How Engine was rolled out: from Forge to public beta at Interrupt 34:18 Inside the two teams building Engine 35:53 Where Insights, Polly, and Engine are converging 40:06 The missing piece: testing a fix before it ships 42:22 Running a branched agent, and the write-access eval problem 46:35 Using Engine as long-term memory 47:39 Pointing Engine at coding-agent traces 48:49 Running Engine on Engine: the meta self-improvement loop References: • Anthropic: https://www.anthropic.com/ • Chat LangChain: https://chat.langchain.com/ • Claude Code: https://www.anthropic.com/claude-code • Claude Haiku: https://www.anthropic.com/claude/haiku • Claude Opus: https://www.anthropic.com/claude/opus • Codex: https://openai.com/codex/ • Context Hub: https://docs.langchain.com/langsmith/use-the-context-hub • Credit Genie: https://www.creditgenie.com/ • Deep Agents: https://docs.langchain.com/oss/python/deepagents/overview • Gemini: https://gemini.google.com/ • GPT-5.5: https://openai.com/index/introducing-gpt-5-5/ • Harbor: https://www.harborframework.com/ • Hex: https://hex.tech/ • Insights: https://docs.langchain.com/langsmith/insights • Interrupt: https://interrupt.langchain.com/ • LangGraph: https://www.langchain.com/langgraph • LangSmith: https://smith.langchain.com/ • LangSmith Chat (formerly Polly): https://docs.langchain.com/langsmith/chat • LangSmith Engine: https://www.langchain.com/langsmith/engine • LangSmith Observability: https://www.langchain.com/langsmith/observability • Mintlify: https://mintlify.com/ • OpenAI: https://openai.com/ • Palash Shah: https://www.linkedin.com/in/palash-sh/ • Terminal-Bench: https://www.tbench.ai/ • Unify: https://www.unifygtm.com/ Where to find Ben: • LinkedIn: https://www.linkedin.com/in/benjamintannyhill/ • Twitter/X: https://x.com/bentannyhill Where to find Harrison: • LinkedIn: https://www.linkedin.com/in/harrison-chase-961287118/ • Twitter/X: https://x.com/hwchase17 Where to find LangChain: • Website: https://langchain.com • Docs: https://docs.langchain.com/ Send feedback or questions to [email protected]