Open Source Summit Korea 2026

Building AI agents is easy, but ensuring their reliability is hard. Because agents operate in open-ended, non-deterministic environments, traditional Test-Driven Development (TDD) falls short. The solution is Eval-Driven Development (EDD), a paradigm that embeds continuous evaluation throughout the agent lifecycle.

This talk explores how to operationalize EDD. First, we examine the critical role of observability. Evaluating just a final output is insufficient for multi-step workflows. We show how OpenTelemetry (OTel) compatible tracing exposes internal reasoning and tool usage for granular debugging.

Next, we tackle scaling evaluation via LLM-as-a-Judge. Since uncalibrated judges often miss domain-specific nuances, we demonstrate aligning custom judges with human experts. Using open-source MLflow, attendees will learn to capture traces, collect expert feedback, and use alignment optimizers to create expert-aligned evaluators.