Open Source Summit Korea 2026

Tech blogs usually talk about huge Ceph clusters with thousands of disks. But in reality, many of us run smaller on-prem setups. I will share my real experience of debugging a small Ceph cluster (10 nodes, 10 NVMe, 15 normal SSDs) and how tracking a slow app led me to write an upstream C++ patch.

It started when our Valkey (Redis) pods suffered from severe write latency. We checked CephFS metadata and Istio network metrics, but they were fine. So we dug into the storage layer using ceph osd perf. We saw huge latency on one specific node. Looking at the kernel logs (dmesg -k), we found a failing NVMe disk. I will explain the "Slow OSD" issue—how one broken disk can freeze a 3-replication cluster.

During this outage, reading ceph osd perf was very frustrating because the OSD IDs were completely unordered. Since it made debugging harder, I decided to fix it. I looked into Ceph's C++ code, changed the unordered hash map to a sorted vector (std::sort), and opened PR #67915 (https://github.com/ceph/ceph/pull/67915). I will share my experience discussing the fix with Ceph maintainers and why I believe engineers should fix the open-source tools they use.