Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

A hands-on workshop on evaluating AI applications where mistakes are expensive.

Annabell Schäfer and Doneyli De Jesus from Langfuse use a banking assistant to walk through the whole AI engineering loop. They cover tracing with PII masking, building datasets, turning domain-expert comments into failure modes, LLM-as-judge and code evaluators, and CI that blocks a prompt change when scores drop. Chapters: - 00:00 Why evals are hard - 05:01 The AI engineering loop - 12:47 Demo: a banking assistant with tracing and PII masking - 20:35 Building datasets - 25:18 Expert review and failure modes - 30:07 Choosing evaluators - 35:00 Running evals automatically on every prompt change - 39:44 Dashboards for stakeholders - 45:58 Working with your coding agent - 52:13 Q&A Doneyli De Jesus — https://langfuse.com Annabell Schäfer — https://langfuse.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!

Mer att läsa

Lumaril, which develops AI software for optimizing industrial production by connecting to existing control systems, raised a $66M Series A at a $400M valuation

Techmeme · för 2 tim sedan

DeepSeek open sources Super-Agent as a research prototype, a proactive coding agent it says can monitor group chats for coding issues and suggest solutions

Techmeme · för 2 tim sedan

Insurance software maker Guidewire agrees to acquire Hyperexponential, which makes AI-powered insurance underwriting and pricing software, for $1.7B in cash

Techmeme · för 2 tim sedan

One word for AI agents in 2027

Nate Herk · för 6 tim sedan