Cephalocon 2022

The Storage Engineering team is responsible for the oversight of Bloomberg’s Ceph infrastructure. These clusters range in scale from hundreds of terabytes, to more than eight petabytes. Bloomberg is facing many unique challenges in terms of our ability to continue scaling live clusters without impacting our clients, while having no offline maintenance windows or lowering our performance standards. The Bloomberg Storage Engineering team successfully completed the live addition of more than 6 petabytes of raw capacity to two of our Ceph-based object stores. We added 50% of raw new capacity to an online cluster without making it unavailable for clients -- in under a single hour. This presentation will document the origins, planning, execution, and bugs Bloomberg faced during this massive undertaking. This project touched on multiple facets of the Ceph project, including a deep dive into the inner workings of the CRUSH algorithm, fundamentals of placement group peering, manipulating data recovery operations, as well as automated scaling of placement groups to scale with the newly-added raw capacity.