Open Source Summit Korea 2026

As vector search becomes foundational to AI workloads, teams need reliable ways to evaluate engine performance before committing to one. We extended OpenSearch Benchmark (OSB) to support Milvus and Vespa alongside OpenSearch, making it possible to compare throughput, latency, and recall across engines using the same datasets and query patterns.

In this talk, we'll walk through how OSB's new engine-as-module architecture lets you benchmark any search engine with a single CLI command. We'll show how customizable workload parameters control everything from HNSW graph construction to bulk ingestion strategies, and demonstrate how to visualize comparative results in OpenSearch Dashboards, turning raw benchmark data into actionable performance insights.

Whether you're evaluating engines for a new project or optimizing an existing deployment, you'll leave with practical knowledge of how to run your own benchmarks and how to extend OSB to support additional engines by implementing five functions.