Lessons Learned Building Rippling AI | Interrupt 26

Senthil Velu Sundaram and Akash Ashok from Rippling's AI team walk through the hard lessons from building and launching Rippling AI — a natural language assistant over the company's employee graph that can answer payroll, HR, and benefits questions across every system of record. They cover three major shifts: why they abandoned a multi-agent sub-agent architecture in favor of a single flat agent with declarative skills; how they replaced a large catalog of bespoke tools with LLM-generated SQL over a cached schema; and how they built an eval-driven development process modeled on statistical confidence intervals (Wilson's interval) to know exactly how many eval repetitions to run before shipping. Chapters: 0:00 The problem: an HR leader's Monday morning inbox 1:36 Demo: Rippling AI and the employee graph 2:37 Architecture overview: LangGraph, entity resolution, and the flat agent 4:54 Why the multi-agent sub-agent model failed 5:33 The flat agent: declarative skills and SOPs instead of sub-agents 6:12 Generic composable tools: one SQL-powered get-data method 7:31 The hallucination problem and why LLM-generated SQL solves it 8:14 SQL is more powerful than bespoke tools: a real example 9:38 Caching data for iterative follow-up queries 10:17 Evals first: what eval-driven development actually means 11:50 EDD is like TDD but harder: the stochasticity problem 12:37 Wilson's confidence interval: how many reps do you actually need? 13:40 The cost-uncertainty-lag triangle: you can only pick two 14:17 Smoke evals on every commit vs. health evals pre-prod 15:00 Building custom tooling and synthetic test environments in production 16:00 Three takeaways: flat agents, composable tools, evals first Resources: → LangGraph: https://www.langchain.com/langgraph → LangSmith: https://www.langchain.com/langsmith → LangChain Academy: https://academy.langchain.com