Cephalocon 2022

Bloomberg's Ceph clusters are the backbone of our internal S3 cloud storage systems, which handle billions of requests per day. In order to ensure the smooth and stable operation of our Ceph clusters, we must have a reliable and stable telemetry system. Ceph’s Prometheus module provides performance counter metrics via the ceph-mgr component. While this paradigm works well for smaller installations, it can be problematic to put metric workloads into ceph-mgr at scale. To make this even more challenging, Ceph is just one component of our internal S3 product. We also need to gather telemetry data about consumables like space, objects per bucket, buckets per tenancy, etc., as well as telemetry from a software-defined distributed quality of service (QoS) system which is not natively supported by Ceph. Therefore, we built a holistic telemetry system to collect and monitor various aspects of our system, including Ceph clusters, usage, and QoS to present a unified view of our product to our internal users in a single pane of glass. In this presentation, we will talk about why we built a custom message-driven telemetry monitoring system and how we made it scalable, extensible, fault-tolerant, and able to support S3 and block storage clusters.