Cephalocon 2022

When working with production Ceph clusters, making changes to your CRUSH map (e.g. adding storage) can be highly disruptive. These changes can result in days to weeks of backfill that's hard to control due to the way that Ceph's backfill scheduling works, holding your cluster hostage by blocking other maintenance activities and slowing down recovery from failure events. In 2018, Dan van der Ster of CERN IT presented techniques ("Mastering Ceph Operations: Upmap and the Mgr Balancer") that they had developed to improve the safety of CRUSH changes. DigitalOcean's storage team has built on these techniques with an open-source tool called pgremapper. In this talk, Joshua Baergen will discuss the problems that operators encounter with CRUSH changes at scale and how DigitalOcean built pgremapper to control and speed up CRUSH-induced backfill.