Open Source Summit Korea 2026

Large language models are getting faster GPUs every year, yet users still notice the pause before the first word appears. That pause has a name: Time To First Token (TTFT). And in production LLM systems, shaving even a few hundred milliseconds from it can dramatically change how responsive an application feels.

This talk tells the story of where those milliseconds go.

We will walk through the lifecycle of a request in modern LLM serving systems and explore the practical techniques engineers use to reduce TTFT in real deployments. Using examples from open source stacks like vLLM, TensorRT-LLM, and Hugging Face TGI, we will examine four powerful optimization levers: KV cache strategies, speculative decoding, model quantization, and batching policies.

Instead of focusing only on theory, the session highlights the tradeoffs practitioners face. When does speculative decoding actually help? When does batching hurt latency? When does quantization reduce memory pressure enough to speed up the first token?

Attendees will leave with a practical playbook for diagnosing TTFT bottlenecks and choosing the right optimization strategy for their model, infrastructure, and workload.

As AI matures from a novelty into a strategic asset, 79% of organisations now prioritise sovereignty to mitigate vendor lock-in and secure critical data. Despite this, the market remains paralysed by a paradox: while open models have achieved performance parity with proprietary systems at a fraction of the cost, they remain massively under-utilised due to perceived friction and trust gaps.

This session dissects this market inefficiency, with insights from LF Research revealing how closed-source dominance is often driven by inertia rather than superior capability. We will explore how to operationalise true independence by rejecting "open-washing" in favour of rigorous frameworks that demand full model completeness: verifying everything from training data to weights. Join us to learn how to transition from renting black-box APIs to architecting a transparent, reproducible, and economically superior AI stack.

Machine learning is central to high-energy physics,.These environments impose strict latency, throughput, and memory constraints, requiring data processing at unprecedented rates. Addressing these demands requires efficient, hardware-aware inference across heterogeneous architectures.

The ML4EP team at CERN is developing an integrated ecosystem for this purpose. We present recent advances in efficient ML deployment. First, aie4ml ports trained models to next-generation AMD FPGAs for ultra-low-latency inference. PQuantML complements this with hardware-aware pruning and quantization, reducing model size and compute cost while preserving performance.

For CPU and GPU inference, SOFIE translates trained models into optimized C++ for heterogeneous systems. Using alpaka, it enables backend-agnostic execution while minimizing data movement. It integrates with PQuantML to support quantized models and is deployable in both online systems and offline workflows.

"Let’s be real: My AI has dementia."
"Vibe-coding in real: Me and My AI is stupid"
Welcome to the "Vibe Coding" era, where anyone can prompt their way to complex code. But as a 17-year-old developer who "bullied" LLMs to build a custom Linux Based OS (MaruxOS) from scratch, I’ve seen the ugly truth: AI is a brilliant assistant, but a chronic liar with a 5-minute memory.
In this session, I expose the messy reality of AI-native development. I’ll share how I battled "Contextual Dementia"—where AI forgets my ARM64 architecture mid-patch—and survived "Mindless Yes-Clicking Syndrome," where a single trusted prompt almost nuked my entire glibc.
Key Takeaways:
The Hallucination Hunter: Sniffing out AI’s "confident bullshit."
The Dementia Doctor: Managing AI’s memory loss to maintain architectural integrity.
The Final Decider: Moving beyond the "Yes-Clicking" trap to take true technical ownership.
AI might be writing the code, but only a real engineer can stop it from burning the house down.

Most open-source AI demos stop at a chatbot, but real users want agents that can work with private files, call tools, and run close to their important data without sending everything to the public cloud.

This talk shows a practical reference architecture for private agent workflows on a single-node K3s-based personal cloud, including local model serving, private knowledge bases, app sandboxing, secure remote access, and multiple GPU allocation modes for competing AI workloads.

Using an open-source personal cloud OS as a case study, I will share what works, what breaks, and which design choices matter when you try to make local AI usable by developers, creators, and small teams rather than just homelab experts.

As organizations explore the Model Context Protocol (MCP), many of the early conversations naturally happen around AI tooling, integrations, and experimentation. At the same time, MCP may also raise questions that are familiar to Open Source Program Offices (OSPOs) and related teams or professionals with open source management experience, including governance, contribution strategy, standards engagement, and cross-functional coordination

This talk reflects on what organizations might gain by bringing those perspectives into MCP discussions earlier, as well as into their broader Agentic AI strategy. Rather than treating MCP only as a technical AI topic, the session will explore it as an area where open source process knowledge may also add value. Attendees will leave with ideas for how OSPOs can contribute to MCP adoption in practical, collaborative, and organization-aware ways

As vector search becomes foundational to AI workloads, teams need reliable ways to evaluate engine performance before committing to one. We extended OpenSearch Benchmark (OSB) to support Milvus and Vespa alongside OpenSearch, making it possible to compare throughput, latency, and recall across engines using the same datasets and query patterns.

In this talk, we'll walk through how OSB's new engine-as-module architecture lets you benchmark any search engine with a single CLI command. We'll show how customizable workload parameters control everything from HNSW graph construction to bulk ingestion strategies, and demonstrate how to visualize comparative results in OpenSearch Dashboards, turning raw benchmark data into actionable performance insights.

Whether you're evaluating engines for a new project or optimizing an existing deployment, you'll leave with practical knowledge of how to run your own benchmarks and how to extend OSB to support additional engines by implementing five functions.

Agentic search lets you ask in Natural Language and have OpenSearch plan and execute retrieval. The Query Planner re-writes Natural Language to OpenSearch DSL using SOTA LLMs. This works amazingly well but not without important limitations:

1/ The biggest factor is added latency for remote inference. eCommerce Search demands sub-100ms response times. Even the best results when slow lead to search abandonment.
2/ We'll demo overcoming latency by hosting SLMs locally within OpenSearch nodes using vLLM/Ollama. This is faster & cheaper, but SLMs suffer quality decline in query re-writes. We'll explore fine-tuning with domain-specific data—proving fine-tuned SLMs beat SOTA LLMs.
3/ Relevance remains generic even with Agentic Search. We'll show how to improve it using user and business contexts with hybrid search and reranking.

This hands-on talk covers:
1/ Search Latency with Remote Inference
2/ Locally hosting SLMs using vLLM/Ollama
3/ Agentic search with SLM Local Inference
4/ Improving relevance with user and business contexts
5/ Fine-tuning SLMs for domain-specific query re-writing via Axolotl/LLaMA-Factory