Cephalocon 2022

Bloomberg's Ceph clusters are the backbone of our internal S3 cloud storage systems, which handle billions of requests per day. This traffic comes in many shapes and sizes based on unpredictable instantaneous client workloads. Our workload is an ever-changing combination of reads, writes, small and large objects, bursty and consistent loads. Regardless, we must ensure the overall performance of our Ceph clusters, as well as guarantee the performance of Bloomberg's applications, which deliver the accurate and market-moving information that our clients use to help make financial decisions. We built a distributed software-defined Quality of Service (QoS) product that is running on Layer 7 of the OSI model, which sits in front of our Ceph clusters. This allows us to do aggregated QoS and traffic shaping using multiple dimensions. These flows are controlled through policies generated based on instantaneous usage patterns and clients’ system needs. These policies can be applied on various combinations of IP addresses, Ceph users, S3 buckets, number of simultaneous connections, as well as request type. To productionize this for our internal cloud users, dashboards and alarms are provided to enable users to monitor their system’s adherence to, and violations of, their QoS policy.