Cephalocon 2022 has ended
July 11 - 13, 2022 | Portland, Oregon + Virtual
View More Details & Registration

Please note: This schedule is automatically displayed in Pacific Daylight Time (PDT). To view the schedule at your preferred time, please choose your location on the right-hand navigation panel under ’Timezone.’
The schedule is subject to change.
Back To Schedule
Wednesday, July 13 • 12:15pm - 12:55pm
We Added 6 Petabytes of Ceph Storage and No Clients Noticed! Here’s How We Did It. - Abhijeet Agrawal & Matthew Leonard, Bloomberg LP

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
The Storage Engineering team is responsible for the oversight of Bloomberg’s Ceph infrastructure. These clusters range in scale from hundreds of terabytes, to more than eight petabytes. Bloomberg is facing many unique challenges in terms of our ability to continue scaling live clusters without impacting our clients, while having no offline maintenance windows or lowering our performance standards. The Bloomberg Storage Engineering team successfully completed the live addition of more than 6 petabytes of raw capacity to two of our Ceph-based object stores. We added 50% of raw new capacity to an online cluster without making it unavailable for clients -- in under a single hour. This presentation will document the origins, planning, execution, and bugs Bloomberg faced during this massive undertaking. This project touched on multiple facets of the Ceph project, including a deep dive into the inner workings of the CRUSH algorithm, fundamentals of placement group peering, manipulating data recovery operations, as well as automated scaling of placement groups to scale with the newly-added raw capacity.

avatar for Abhijeet Agrawal

Abhijeet Agrawal

Senior Software Developer, Bloomberg LP
avatar for Matthew Leonard

Matthew Leonard

Storage Engineering Manager, Bloomberg LP
Matthew Leonard leads the Storage Engineering organization in Bloomberg Engineering’s Technology Infrastructure department. Matthew worked on software for fighter planes, and now leverages his “slow is smooth, smooth is fast” mentality from the aerospace industry to direct Bloomberg’s... Read More →

Wednesday July 13, 2022 12:15pm - 12:55pm PDT
Regency Ballroom B