Cephalocon 2022: Full Schedule

July 11 - 13, 2022 | Portland, Oregon + Virtual
View More Details & Registration

Please note: This schedule is automatically displayed in Pacific Daylight Time (PDT). To view the schedule at your preferred time, please choose your location on the right-hand navigation panel under ’Timezone.’
The schedule is subject to change.

8:00am PDT

Registration & Badge Pick-up

Monday July 11, 2022 8:00am - 5:00pm PDT
Regency Ballroom Foyer

Breaks & Networking

9:00am PDT

Developer Summit Discussions Talks To Be Announced

Monday July 11, 2022 9:00am - 5:00pm PDT

Developer Discussions

8:00am PDT

Registration & Badge Pick-up

Tuesday July 12, 2022 8:00am - 9:00am PDT
Regency Ballroom Foyer

Breaks & Networking

9:00am PDT

Welcome & Opening Remarks - Mike Perez, Ceph Community Manager, Red Hat

Speakers

Mike Perez

Ceph Community Manager & Acting Director/Ceph Foundation, Red Hat

Mike is currently the community manager and acting director for the Ceph Foundation. Being a contributing member of OpenStack since 2010, he has served as a core developer for the OpenStack block storage project Cinder and as a project technical lead for the Kilo and Liberty releases... Read More →

Tuesday July 12, 2022 9:00am - 9:10am PDT
Regency Ballroom B

Keynote Sessions

9:10am PDT

SoftIron Sponsored Keynote: Ceph, SoftIron and the Edge - A Vision for Ceph's Future Role - Phil Straw, SoftIron

The shift to Edge Computing is happening, and with it will come dramatic changes in the way we build Data Infrastructure as a whole. In his keynote presentation, Van Alstyne will map out some of the changes he foresees in the coming years, what he believes that will mean for the Ceph and the crucial role he believes it can play, and what - at a practical level - SoftIron intends to do to support that vision and the broader opensource community as a whole.

Speakers

Phil Straw

CEO, SoftIron

The technical visionary co-founder behind SoftIron, Phil Straw, initially served as the company’s CTO before stepping into the role as CEO. Previously Phil served as CEO of Heliox Technologies, co-founder and CTO of dotFX, VP of Engineering at Securify and worked in both technical... Read More →

Tuesday July 12, 2022 9:10am - 9:25am PDT
Regency Ballroom B

Keynote Sessions

9:30am PDT

Intel Sponsored Keynote: The Future of Cloud Storage with Faster Interconnects, Storage Class Memory and Platform Innovations - Anjaneya 'Reddy' Chagam, Intel

Emerging workloads such as Edge, 5G, AI, and High-Performance Computing are driving explosive data growth and processing needs. Innovations in memory technologies such as Intel® Optane™ Technology, open interconnect standards such as Compute Express Link™ (CXL™), programmable infrastructure processing unit (IPU) and platform offload capabilities such as compression/encryption are driving storage infrastructure innovations to meet these explosive data processing requirements. In this talk, we will present how Intel technologies are enabling next-generation storage infrastructure innovations and how Ceph can exploit these capabilities to deliver highly scalable cloud storage solutions.

Speakers

Reddy Chagam

Senior Principal Engineer and Lead Cloud Storage Architect, Intel

Anjaneya “Reddy” Chagam is a Senior Principal Engineer and Lead Cloud Storage Architect in Intel’s Cloud and Enterprise Solutions Group. He is responsible for developing software-defined storage strategy, architecture, and platform technology initiatives. He is a board member... Read More →

Tuesday July 12, 2022 9:30am - 9:45am PDT
Regency Ballroom B

Keynote Sessions

9:50am PDT

Keynote Session: State of the Cephalopod - Josh Durgin; Venky Shankar; Neha Ojha; Ernesto Puerta; Casey Bodley; Adam King; Ilya Dryomov, Red Hat & Dan van der Ster, CERN

An update from the leaders of the Ceph project about recent developments, current priorities, and other activity in the Ceph community.

Speakers

Josh Durgin

Manager, Red Hat

Venky Shankar

Ceph Filesystem Tech Lead, IBM

I am the tech lead for the Ceph File System project. I have worked on distributed file systems for over a decade and started contributing to Ceph File System from mid of 2018. Over these years, I have worked on a number of really interesting parts of Ceph File System - metadata journaling... Read More →

Dan van der Ster

Computing Engineer, CERN

Dan is the Ceph Service Manager at CERN, where he has worked the past 11 years as a Distributed Systems and now Storage Engineer. Dan is an experienced Ceph operator, and founding member of the Ceph Board, representing the Associate Members. Dan holds a PhD in Computer Engineering... Read More →

Neha Ojha

Software Engineering Manager, IBM

Neha manages a diverse, globally distributed team of software engineers who work on Ceph Core RADOS. She is the Project Tech Lead of the Ceph Core RADOS Team. She is also an elected member of the Ceph Executive Council in-charge of leading the Ceph project's open source community... Read More →

Ernesto Puerta

Principal Software Engineer, Red Hat

Ernesto is the Ceph Dashboard component lead. He previously worked at Telefonica R&D, Alcatel-Lucent, Bell Labs, and Nokia, where he first came to know about Ceph, for a Cloud Video Storage project back in 2015. After that stimulating experience, he joined Red Hat in 2018 and has... Read More →

Casey Bodley

Ceph Developer, Red Hat

Casey Bodley works for Red Hat, and is the current maintainer of RGW, the Ceph object gateway. He has contributed to the development of the RGW multi-site sync system, and also worked on other Ceph and RGW projects, such as the introduction of the beast frontend, and related work... Read More →

Adam King

Red Hat

Ilya Dryomov

Software Engineer, Red Hat

Tuesday July 12, 2022 9:50am - 10:30am PDT
Regency Ballroom B

Keynote Sessions

10:30am PDT

Break & Networking

Tuesday July 12, 2022 10:30am - 11:00am PDT
Regency Ballroom Foyer

Breaks & Networking

11:00am PDT

Optimizing RGW Object Storage Mixed Media through Storage Classes and Lua Scripting - Curt Bruns & Anthony D'Atri, Intel

Ceph enables flexible and scalable object storage of unstructured data for a wide variety of workloads. RGW (RADOS GateWay) deployments experience a wide variety of object sizes and must balance workload, cost, and performance requirements. S3 storage classes are an established way to steer data onto underlying media that meet specific resilience, cost, and performance requirements. One might for example define RGW back end storage classes for SSD or HDD media, non-redundant vs replicated vs erasure coding pools, etc. Diversion of individual objects or entire buckets into a non-default storage class usually requires specific client action. Compliance however can be awkward to request and impossible to enforce, especially in multi-tenant deployments that may include paying customers as well as internal users. This work enables the RGW back end to enforce storage class on uploaded objects based on specific criteria without requiring client actions. For example one might define a default storage class on performance TLC or Optane media for resource-intensive small S3 objects while assigning larger objects to dense and cost-effective QLC SSD media.

Speakers

Curt Bruns

Staff Engineer, Solidigm Technology

Curt Bruns is a Staff Engineer working on optimizing Open Source Storage software, such as Ceph and SPDK, to take advantage of features inside Solidigm's Solid State Drives (SSDs). Curt has previously worked on the OpenStack Cinder and Manila projects. Curt has a Master’s Degree... Read More →

Anthony D'Atri

Solutions Architect, Intel

Anthony D'Atri's career in system administration, ops, and architecture has spanned laptops to vector supercomputers. He brings his passion for effective fleet management and server components to bear on a holistic yet, detailed approach to deployment and operations. Experience with... Read More →

Tuesday July 12, 2022 11:00am - 11:40am PDT
Regency Ballroom B

Track 1

11:00am PDT

What's New with Crimson and Seastore? - Samuel Just, Red Hat

Next generation storage devices require a change in strategy, so the community has been developing Crimson, an eventual replacement for ceph-osd intended to minimize cpu overhead and improve throughput and latency. Seastore is a new backing store for crimson-osd targeted at emerging storage technologies including persistent memory and ZNS devices. This talk will explain recent developments in the Crimson project and Seastore.

Speakers

Samuel Just

Engineer, IBM

Sam began working on the Ceph project in 2011. Most of his time currently is spent working on crimson, the next generation ceph-osd implementation.

Tuesday July 12, 2022 11:00am - 11:40am PDT
Regency Ballroom C

Track 2

11:00am PDT

Improved Business Continuity for an Existing Large Scale Ceph Infrastructure: A Story from Practical Experience - Enrico Bocch & Arthur Outhenin-Chalandre, CERN

The IT Department at CERN (European Organization for Nuclear Research) operates a large-scale computing and storage infrastructure for processing scientific data and providing IT services to its user community. Ceph is a critical part of this picture as it provides: 1. Block storage for the OpenStack infrastructure (440k cores - 25 PB), 2. S3 object storage for cloud-native applications, HTTP-based software distribution, and backup needs (16 PB), 3. CephFS for shared filesystems in HPC clusters and storage persistency in OpenShift and Kubernetes (7 PB). In the past year, our Ceph infrastructure has been largely restructured with the goal of offering storage solutions for High(er) Availability and Disaster Recovery / Business Continuity. In this presentation we will detail how we transitioned from a single RBD zone to multiple Storage AZs; how we hardened and optimized RBD snapshot mirroring for OpenStack; how we integrated a restic-based CephFS backup orchestrator with Manila; and our experience merging two independent S3 clusters into a single multi-region Zonegroup, as well as experimentation with Maglev load balancing.

Speakers

Enrico Bocchi

Computing Engineer, CERN (European Organization for Nuclear Research)

Enrico is a Computing Engineer at CERN, where he has worked in the past 5 years Distributed Storage Systems. He is responsible for the operating and evolving critical production services at the scale of 10's of PBs including Ceph block and object storage. Enrico holds a joint-PhD... Read More →

Arthur Outhenin-Chalandre

Computing Fellow, CERN

Arthur is a Computer Engineer at CERN where he started to work on Business Continuity for Ceph in early 2021. He is an active contributor to the Ceph project, especially in the context of RBD mirroring features, and shares responsability for operating production Ceph clusters at CERN... Read More →

Tuesday July 12, 2022 11:00am - 11:40am PDT
Regency Ballroom D

Track 3

11:45am PDT

DisTRaC: Accelerating High-Performance Compute Processing for Temporary Data Storage - Gabryel Mason-Williams, Rosalind Franklin Institute

There is a growing desire within scientific and research communities to start using object stores to store and process their data in high performance (HPC) clusters. However, object stores are not necessarily designed for performance and are better suited for long term storage. Therefore, users often use a High-Performance File system when processing data. However, network filesystems have issues where one user could potentially thrash the network and affect the performance of everyone else's data processing jobs in the cluster. This talk presents a solution to this problem DisTRaC - (Dis)tributed (T)raisent (Ra)m (C)eph. DisTRaC offers a solution to this problem by providing a method for users to deploy Ceph onto their HPC clusters using RAM. Their intermediate data processing can now be done in RAM, taking the pressure off the networked filesystem by using the node interconnect to transfer data. In addition, all the data is localized, creating a hyper-converged HPC cluster for the duration of the job. DisTRaC reduces the I/O overhead of the networked filesystem and offers a potential data processing performance increase.

Speakers

Gabryel Mason-Williams

Junior Research Software Engineer, Rosalind Franklin Institute

Gabryel Mason-Williams is a junior research software engineer focusing on HPC, cloud and storage. They studied computer science at the University of Plymouth, where they did a year in industry at Diamond Light Source researching high-performance object stores for tomography data... Read More →

Tuesday July 12, 2022 11:45am - 12:25pm PDT
Regency Ballroom B

Track 1

11:45am PDT

[Virtual] Understanding SeaStore Through Profiling - Yingxin Cheng & Tushar Gohad, Intel

SeaStore is the new ObjectStore designed to complement Crimson OSD to support new generation of storage interfaces/technologies (NVMe, ZNS, Persistent Memory, etc). As SeaStore matures, profiling becomes increasingly critical to understand comprehensive performance impact of design choices and to set direction moving forward as the backend moves to mainstream. Profiling infrastructure will also aid new contributors understand the inner workings of SeaStore. In this session, we will talk about SeaStore support for performance profiling, optimizations made based on the initial analysis, the current status or gaps vs BlueStore along with performance data.

Speakers

Tushar Gohad

Principal Engineer, Cloud SW Architecture, Intel

Tushar is a Principal Engineer - Software Architect with Intel's Data Center Group. He has been working on the open-source networking and storage-related technologies for over a decade now – his recent contributions have been to Ceph, CDN software stacks, SPDK and networking in... Read More →

Yingxin Cheng

Software Engineer, Intel

Yingxin has been contributing to the Ceph Crimson project since it was launched. Over the 3 years, he majorly worked on implementing Crimson messenger and SeaStore components. And recently he was focused on profiling and optimizing SeaStore.

Tuesday July 12, 2022 11:45am - 12:25pm PDT
Regency Ballroom C

Track 2

11:45am PDT

How we Operate Ceph at Scale - Matt Vandermeulen, Digital Ocean

As clusters grow in both size and quantity, operator effort should not grow at the same pace. In this talk, Matt Vandermeulen will discuss strategies and challenges for operating clusters of varying sizes in a rapidly growing environment for both RBD and object storage workloads based on DigitalOcean's experiences.

Speakers

Matt Vandermeulen

Senior Storage Engineer, Digital Ocean

Matt Vandermeulen is a senior storage engineer at DigitalOcean who works on the storage systems team that maintains the persistence layer of storage services at DO.

Tuesday July 12, 2022 11:45am - 12:25pm PDT
Regency Ballroom D

Track 3

12:30pm PDT

[Virtual] Ceph QoS Refinements for Background Operations Using mClock - Sridhar Seshasayee & Aishwarya Mathuria, Red Hat

In a distributed storage environment like Ceph, client operations (foreground) and background operations such as recovery, scrubs, snap trim etc., compete for resources and depending on the situation, one can overwhelm the other without any QoS or other throttling mechanisms in place. Ceph's use of mClock to achieve QoS is based on an implementation of the dmClock algorithm. Recent refinements made to the mClock scheduler implementation in the Ceph code base leverages the full potential of the algorithm. This talk presents the results of a study that compared the existing 'weighted priority queue' scheduler with the 'mclock scheduler' and the refinements that were made to the mClock scheduler in Ceph. The study performed tests with client operations and background operations like recoveries and scrubs. The results show that mClock scheduler is able to provide better QoS to the different classes of operations when compared to the 'weighted priority queue'. The talk also introduces config profiles that are classified based on whether preference is to be given to client operations, background operations or provide a balance between all the operations. When applied, the profile automatically sets the QoS parameters in the mClock server thus reducing complexity from users.

Speakers

Aishwarya Mathuria

Senior Software Engineer, IBM

Sridhar Seshasayee

Software Architect, IBM India Private Limited

I have been working for Red Hat India since 2018 and recently moved to IBM as part of the acquisition. I am a part of the core engineering team working on Ceph and more specifically on the RADOS component. I have been primarily involved in developing solutions to provide QoS for various... Read More →

Tuesday July 12, 2022 12:30pm - 1:10pm PDT
Regency Ballroom B

Track 1

12:30pm PDT

[Virtual] Dashboard: Exploring Centralized Logging with Ceph Storage - Gaurav Sitlani, Koor Technologies, Inc, & Aashish Sharma, Red Hat

This talk focuses and explores about Centralized Logging in Ceph Storage with the help of three leading Open Source solutions:

ELK Stack
Graylog
Grafana stack with Loki (https://github.com/grafana/loki)

The objective is to highlight the various aspects and importance of Observability and Centralized Logging which can be very useful to view and manage the logs in a Dashboard view.
A good log management solution would focus on the following key capabilities for logs :

Aggregation
Processing
Log storage and retention policies
Visualization and analysis of logs

Towards the end of the talk we would like to show the results of the POC conducted by us which can help in deciding the framework or the technology stack from the above mentioned technologies which could be a best solution for centralized logging with Ceph.
We’ll also highlight the importance of pattern identification in logs which can be simplified,categorized and visualized based on the solution chosen for Centralized logging.
Among the results we’ll highlight some of the Advantages and Limitations of the various frameworks assessed by us along with some real world use cases.

Speakers

Gaurav Sitlani

Cloud Storage Engineer, Koor Technologies, Inc.

Gaurav Sitlani, joined Red Hat as an Intern where he started working on Ceph in 2018 supporting Enterprise customers.He graduated from University of Pune with a Bachelor's degree in Computer Engineering in 2018.Currently he's working as a Cloud Storage engineer at Koor Technologies... Read More →

Aashish Sharma

Associate Software Engineer, Red Hat

Aashish Sharma currently working as a Software engineer in the Ceph Dashboard team.

Tuesday July 12, 2022 12:30pm - 1:10pm PDT
Regency Ballroom C

Track 2

12:30pm PDT

Over A Billion Requests Served Per Day: Ensuring Everyone is Happy with Our Ceph Clusters’ Performance - Jane Zhu & Matthew Leonard, Bloomberg LP

Bloomberg's Ceph clusters are the backbone of our internal S3 cloud storage systems, which handle billions of requests per day. This traffic comes in many shapes and sizes based on unpredictable instantaneous client workloads. Our workload is an ever-changing combination of reads, writes, small and large objects, bursty and consistent loads. Regardless, we must ensure the overall performance of our Ceph clusters, as well as guarantee the performance of Bloomberg's applications, which deliver the accurate and market-moving information that our clients use to help make financial decisions. We built a distributed software-defined Quality of Service (QoS) product that is running on Layer 7 of the OSI model, which sits in front of our Ceph clusters. This allows us to do aggregated QoS and traffic shaping using multiple dimensions. These flows are controlled through policies generated based on instantaneous usage patterns and clients’ system needs. These policies can be applied on various combinations of IP addresses, Ceph users, S3 buckets, number of simultaneous connections, as well as request type. To productionize this for our internal cloud users, dashboards and alarms are provided to enable users to monitor their system’s adherence to, and violations of, their QoS policy.

Speakers

Jane Zhu

Senior Software Engineer, Bloomberg LP

Dr. Jane Zhu is a Senior Software Engineer at Bloomberg. She is part of the Storage Engineering team in the company’s Technology Infrastructure department. Jane designed and built a highly available, scalable, and durable software-defined cloud storage platform. In addition to shepherding... Read More →

Matthew Leonard

Storage Engineering Manager, Bloomberg LP

Matthew Leonard leads the Storage Engineering organization in Bloomberg Engineering’s Technology Infrastructure department. Matthew worked on software for fighter planes, and now leverages his “slow is smooth, smooth is fast” mentality from the aerospace industry to direct Bloomberg’s... Read More →

Tuesday July 12, 2022 12:30pm - 1:10pm PDT
Regency Ballroom D

Track 3

1:10pm PDT

Lunch Break

Tuesday July 12, 2022 1:10pm - 2:10pm PDT
Regency Ballroom A

Breaks & Networking

2:10pm PDT

Sponsored Lightning Talk: Ceph in a Model-driven World - Philip Williams, Canonical

In this talk, Phil Williams will briefly describe how Ceph can be deployed and managed as a model-driven application with Juju and Charms, and how this approach reduces administrative effort and operational risk.

Speakers

Philip Williams

Product Manager, Canonical

Phil Williams is a Product Leader at Canonical responsible for Ceph and other storage solutions. He has over 15 years experience in the storage industry, having previously been responsible for storage infrastructure and products at a number of leading technology companies.

Tuesday July 12, 2022 2:10pm - 2:25pm PDT
Regency Ballroom D

Lightning Talks

2:10pm PDT

Modifying Ceph for Better HPC Performance - Darren Soothill, CROIT

With the ever-increasing use of Ceph for HPC workloads, we take a look at what can be done to modify Ceph to make it perform better in these use cases. This includes looking at modifications that can be done to the OSD code to improve its performance. Along with performance metrics of the sort of improvements that can be achieved.

Speakers

Darren Soothill

VP of Presales, CROIT

Place Holder

Tuesday July 12, 2022 2:10pm - 2:50pm PDT
Regency Ballroom B

Track 1

2:10pm PDT

Introduction to Container Object Storage Interface aka COSI for ceph RGW - Jiffin Tony Thottan, Red Hat

For applications in Kubernetes, CSI provides a way to consume file/block storage for their workloads. The main motivation behind the Container Object Storage Interface is to provide a similar experience for Object Store. Basic idea is to provide a generic, dynamic provisioning API to consume the object store and the app pods can access the bucket in the underlying object-store like a PVC. The major challenge for this implementation there is no standard protocol defined for object and the COSI project need to be vendor agonistic. It won't handle the orchestration/management of object store, rather it will be another client and provide bucket access on behalf of applications running in Kubernetes. The initial version of the ceph-cosi driver can be found at https://github.com/ceph/ceph-cosi.

Speakers

Jiffin Tony Thottan

Senior Software Engineer, Red Hat

Jiffin Tony Thottan is part of the Red Hat Storage Team working as a Senior Software Engineer. Initially part of the NFS team and contributed to GlusterFS , NFS-Ganesha projects. He was an active contributor for both projects worked on implementing different features like nfsv4acls... Read More →

Tuesday July 12, 2022 2:10pm - 2:50pm PDT
Regency Ballroom C

Track 2

2:50pm PDT

[Virtual] Lightning Talk: Installing Ceph on Kubernetes Using the Rook Operator and Helm - Mike Petersen, Platform9

This presentation will cover a basic install of Ceph using the Rook Operator and the App Catalog in Platform9 Managed Kubernetes. Mike will go over some basic installation options, how to setup the toolbox for Ceph troubleshooting, and prerequisites for a working cluster. If you are interested in adding storage to your on-prem cluster then this presentation will get you one step closer, or at least show you one option.

Speakers

Mike Petersen

Technical Marketing Engineer, Platform9

Mike Petersen has been working in the Open Source world for 15+ years. He started getting more involved in open source communities around the Grizzly release of OpenStack. Since then he has contributed to Knative, Tekton, and other projects. He is currently a Technical Marketing Engineer... Read More →

Tuesday July 12, 2022 2:50pm - 3:05pm PDT
Regency Ballroom D

Lightning Talks

2:55pm PDT

Lightning Talk: Ceph and 6G: Are We Ready for zettabytes? - Babar Khan, Technical University Darmstadt

Ceph has been a promising solution for the 5G storage requirements and the edge data center. But what's next? This talk will give the audience a brief yet comprehensive overview of the future 6G technology challenges and how Ceph should embrace itself for these challenges.

Speakers

Babar Khan

Researcher and Software Engineer, Technical University Darmstadt

Babar Khan is a software engineer. Currently he is working as a researcher at Technical University of Darmstadt, Germany. Since his research is based on software defined storages, he is an ardent Ceph user and developer. Therefore, he understands Ceph ecosystem to a good extent and... Read More →

Tuesday July 12, 2022 2:55pm - 3:10pm PDT
Regency Ballroom D

Lightning Talks

2:55pm PDT

[Virtual] Introducing the New RBD Image Encryption Feature - Or Ozeri & Danny Harnik, IBM

A new encryption capability in librbd was introduced in the Ceph Pacific release. This feature allows users to secure their RBD images by setting per-image passphrases, which will be used to unlock a new data-at-rest encryption layer. Greater flexibility, ease-of-use and minimal performance overhead are the main advantages of this feature over alternatives of using external encryption components, such as dm-crypt and qemu LUKS. Furthermore, this feature will soon allow a unique capability that is not be available by other means: setting RBD cloned images with an encryption key different from the encryption key of its parent image. The talk will describe the new feature, outline various use-cases, present performance evaluation results, and show a quick demo.

Speakers

Or Ozeri

Research Staff Member, IBM

Or Ozeri is a research staff member at the Cloud Infrastructure group at IBM Haifa Research Lab, located in Tel-Aviv, Israel. Since joining the group in late 2015, Or has been mainly working on R&D projects involving Object Storage. He has contributed to open source projects such... Read More →

Danny Harnik

Senior Technical Staff Member, IBM Research

Danny Harnik is a senior technical staff member at IBM Research, and part of the cloud infrastructure research group in Haifa, Israel . His current research interests include cloud storage and large scale storage systems in general, compression and deduplication, as well as security... Read More →

Tuesday July 12, 2022 2:55pm - 3:35pm PDT
Regency Ballroom B

Track 1

2:55pm PDT

[Virtual] S3select: Computational Storage in S3 - Gal Salomon & Girjesh Rajoria, Red Hat

S3 Select is an S3 operation (introduced by Amazon in 2018) that implements a pushdown paradigm that pulls out only the data you need from an object, which can dramatically improve the performance and reduce the cost of applications that need to access data in S3. The talk will introduce s3select operation and architecture. It will describe what the pushdown technique is, why and where it is beneficial for the user. It will cover s3select supported features and their integration with analytic applications. It will discuss the main differences between columnar and non-columnar formats (CSV vs Parquet). We’ll also discuss recent developments for ceph/s3select. The presentation will show how easy it is to use ceph/s3select.

Speakers

Girjesh Rajoria

Software Engineer, Red Hat

Girjesh Rajoria is a Software Engineer working at Red Hat. She holds a Master’s Degree in Computer Applications. She is a contributor to Ceph, RGW, NFS-Ganesha and glusterfs.

Gal Salomon

Senior SW Engineer, Red Hat

gal salomon (gsalomon@redhat.com) is a senior SW engineer in Rados Gatway team. his main work is to introduce S3-select operation into CEPH, the s3-select operation is actually an SQL dialect and has a spec(maintained by AWS). so far, the great majority of the spec has been implemented... Read More →

Tuesday July 12, 2022 2:55pm - 3:35pm PDT
Regency Ballroom C

Track 2

3:15pm PDT

Lightning Talk: Introduction to Cephadm - Melissa Li, Red Hat

Many tools have existed for deploying Ceph, however Cephadm seeks to make the deployment of Ceph easier in non-Kubernetes environments, and make clusters easier to manage through integration with the Ceph CLI and orchestration API. This talk will showcase a brief overview of how Cephadm works, followed by a demo of how to use Cephadm to quickly set up and expand a Ceph cluster, interact with it, and add storage.

Speakers

Melissa Li

Software Engineer Intern, Red Hat

Melissa Li is a Software Engineer Intern at Red Hat working on Ceph distributed storage. She started contributing to Cephadm last year through open source with Outreachy, and continues to contribute as part of the orchestrator team. She is currently working while studying for her... Read More →

Tuesday July 12, 2022 3:15pm - 3:30pm PDT
Regency Ballroom D

Lightning Talks

3:35pm PDT

Break & Networking

Tuesday July 12, 2022 3:35pm - 4:05pm PDT
Regency Ballroom Foyer

Breaks & Networking

4:05pm PDT

Why We Built A “Message-Driven Telemetry System At Scale” Ceph Cluster - Xiaolin Lin & Matthew Leonard, Bloomberg LP

Bloomberg's Ceph clusters are the backbone of our internal S3 cloud storage systems, which handle billions of requests per day. In order to ensure the smooth and stable operation of our Ceph clusters, we must have a reliable and stable telemetry system. Ceph’s Prometheus module provides performance counter metrics via the ceph-mgr component. While this paradigm works well for smaller installations, it can be problematic to put metric workloads into ceph-mgr at scale. To make this even more challenging, Ceph is just one component of our internal S3 product. We also need to gather telemetry data about consumables like space, objects per bucket, buckets per tenancy, etc., as well as telemetry from a software-defined distributed quality of service (QoS) system which is not natively supported by Ceph. Therefore, we built a holistic telemetry system to collect and monitor various aspects of our system, including Ceph clusters, usage, and QoS to present a unified view of our product to our internal users in a single pane of glass. In this presentation, we will talk about why we built a custom message-driven telemetry monitoring system and how we made it scalable, extensible, fault-tolerant, and able to support S3 and block storage clusters.

Speakers

Xiaolin Lin

Senior Software Engineer, Bloomberg LP

Xiaolin Lin is a Senior Software Engineer at Bloomberg. He is part of the Storage Engineering team in the company's Technology Infrastructure department. Prior to current role, he has worked on Charting Platform to deal with real-time time-series data. He currently leads metric system... Read More →

Matthew Leonard

Storage Engineering Manager, Bloomberg LP

Tuesday July 12, 2022 4:05pm - 4:45pm PDT
Regency Ballroom B

Track 1

4:05pm PDT

[Virtual] CephFS At-Rest Encryption with fscrypt - Jeffrey Layton, Red Hat

The fscrypt facility was added to the Linux kernel in 2015 which added the ability for unprivileged users to store files on certain local filesystems in an encrypted fashion using their own keys. The CephFS team has been engaged in a multi-year project to extend this support to the kernel CephFS client, allowing users to transparently work with encrypted files on CephFS. This talk will cover an overview of the effort to add this support and information on how to use this feature.

Speakers

Jeff Layton

Sr. Principal Software Engineer, Red Hat

Jeff Layton is a longtime Linux kernel and Ceph developer who specializes in network filesystems. He maintains the Linux kernel CephFS client and is a contributor to many other open source projects.

Tuesday July 12, 2022 4:05pm - 4:45pm PDT
Regency Ballroom D

Track 3

4:50pm PDT

[Virtual] You Should’ve Used kURL: Solving the Storage Struggle for Kubernetes with Ceph + Rook - Treva Williams, Replicated

At Replicated we like to believe that we’ve succeeded in creating a way for our users to quickly launch an enterprise-ready Kubernetes cluster with a single command, but the road from creation to shipment & beyond was long & filled with obstacles, with special emphasis on the “beyond.” Never one to let a crisis go to waste, every issue became a learning opportunity for our engineers, who rapidly developed an entire ecosystem of tools that were in no small way inspired by the many, many learning opportunities presented from shipping an Enterprise product with Ceph+Rook. Please join our presentation to hear more of our war stories & the innovation that resulted.

Speakers

Treva Williams

Technical Community Manager, Open Infrastructure Foundation

Treva is technical community manager for the Kata Containers project at Open Infrastructure foundation. Before Joining OIF, Treva was developer advocate for Replicated.com, & was a Cloud virtualization administrator at RackSpace. They intermittently hold several certifications from... Read More →

Tuesday July 12, 2022 4:50pm - 5:30pm PDT
Regency Ballroom B

Track 1

4:50pm PDT

RGW – An Ultimate S3 Frontend for MultipleBackends: An Implementation Story - Gregory Touretsky & Basavaraj Kirunge, Seagate

RGW is the production-proven S3-compatible interface implementation running in front of RADOS in thousands of clusters. But can it be used with other backends? In this talk we’ll share our experience with RGW integration as a frontend to Motr, Seagate-driven open-source mass capacity storage platform. We’ll describe how we were able to get first functional integrated solution up and running within just 3 weeks. We’ll share results of our analysis – including gaps identified in the current RGW feature set. An experience of joining a vibrant RGW development community and adjusting our development methodologies will be covered as well. We’ll talk about our plans around RGW and Motr integration, as well as will cover our proposed improvements to RGW.

Speakers

Basavaraj Kirunge

Senior Staff Engineer, Seagate

Basavaraj Kirunge has over 17 years of experience in Developing software for Distributed Systems, Cloud, Security, Storage, Business Intelligence Analytics and Middleware products. He is the CORTX S3 Architect and has lead the development of data access interface for object storage... Read More →

Gregory Touretsky

Principal Product Manager, Seagate

Gregory Touretsky is a Principal Product Manager at Seagate. He drives the company’s roadmap around object storage and Kubernetes. He has over twenty years of practical experience with distributed computing and storage as an architect, product manager, and systems engineer. Gregory... Read More →

Tuesday July 12, 2022 4:50pm - 5:30pm PDT
Regency Ballroom C

Track 2

4:50pm PDT

Ceph for Public Cloud Workloads - Philip Williams, Canonical

Public clouds were initially popularized under the premise that workloads are dynamic, and that you could easily match available compute resources to the peaks and troughs in your consumption, rather than having to maintain mostly idle buffer capacity to meet peak user demand. However, what has become more apparent is that this isn't necessarily true when it comes to storage. Typically what is observed in production environments is a continual growth of all data sets, across those that are actively used for decision making or transactional processing, those maintained as training data for AI/ML, or kept for archival purposes, and simply just backups of critical data. During this talk, we will discuss how Ceph can be deployed in a cost-effective manner adjacent to public clouds, and investigate the financial implications of both approaches.

Speakers

Philip Williams

Product Manager, Canonical

Tuesday July 12, 2022 4:50pm - 5:30pm PDT
Regency Ballroom D

Track 3

5:35pm PDT

BoF Session: Ceph in Scientific Computing and Large Clusters - Kevin Hrpcek, Space Science & Engineering Center, University of Wisconsin - Madison

Ceph has found its place in supporting many scientific projects throughout the world and it is also used as a backend in many large clusters for companies. High throughput/performance computing introduces its own challenges and these groups are often pushing the limits of Ceph whether it be cluster size, throughput, or clients. Join this BoF session for a chance to connect with people who use Ceph to support science and research or at the multi petabyte scale.

Speakers

Kevin Hrpcek

Software Developer, University of Wisconsin - Madison

Kevin is a technology and science enthusiast who is a DevOps Engineer for the Space Science and Engineering Center at the University of Wisconsin - Madison. He joined SSEC in 2015 at the beginning of the NASA Atmosphere SIPS contract. Working with a small team of developers he has... Read More →

Tuesday July 12, 2022 5:35pm - 6:15pm PDT
Regency Ballroom D

BoF Sessions

5:35pm PDT

BoF Session: To Be Announced

Tuesday July 12, 2022 5:35pm - 6:15pm PDT
Regency Ballroom C

BoF Sessions

5:35pm PDT

BoF Session: To Be Announced

Tuesday July 12, 2022 5:35pm - 6:15pm PDT
Regency Ballroom B

BoF Sessions

6:15pm PDT

Sponsor Showcase & Booth Crawl

Tuesday July 12, 2022 6:15pm - 7:15pm PDT
Regency Ballroom Foyer

Breaks & Networking

8:00am PDT

Registration & Badge Pick-up

Wednesday July 13, 2022 8:00am - 9:00am PDT
Regency Ballroom Foyer

Breaks & Networking

9:00am PDT

Welcome Back & Remarks

Wednesday July 13, 2022 9:00am - 9:10am PDT
Regency Ballroom B

Keynote Sessions

9:10am PDT

Red Hat Sponsored Keynote: Reaching Deeper - Vision and Scale of Ceph's Future - Federico Lucifredi & Kyle Bader, Red Hat

Large enterprises, public cloud operators, and governments rely on Ceph technology to provide users with the scale-out dynamics and economics that cloud workloads require — at petabyte scale capacities. Ceph achieves these milestones while also delivering consistent and scalable performance. In this keynote session, you will first learn how Red Hat envisions the future of the datacenter as a software-defined entity spanning from compute all the way to storage, and what our five year vision looks like. We touch on what our two-year investment roadmap in Ceph technology looks like. We then show how Ceph addresses orthogonal performance dimensions by delivering consistent performance for thousands of Kubernetes persistent volumes backed by a managed Rook cluster, and doing so while volumes are constantly being created and destroyed in the background.
Ten Billion Objects
We share the result of joint work with Seagate and Intel to store more than a billion objects in RGW. The lessons learned observing the cluster’s performance while ingesting more than a billion objects into a relatively small cluster are enlightening.
Twenty Thousand PVs
Rook-managed clusters face impressive rates of volume churn as container lifecycles are fast and furious: a container could be instantiated, tested and decommissioned all within minutes. We share the inside track on Red Hat’s Storage testing of Rook on Amazon AWS Kubernetes clusters.

Speakers

Federico Lucifredi

Product Management Director, Red Hat

Federico Lucifredi is the Product Management Director for Ceph Storage at Red Hat and a co-author of O'Reilly's "Peccary Book" on AWS System Administration. Previously, he was the Ubuntu Server product manager at Canonical, where he oversaw a broad portfolio and the rise of Ubuntu... Read More →

Kyle Bader

Data Foundation Architect, Red Hat

Kyle is the Data Foundation Architect covering both OpenShift Data Foundation and Red Hat Ceph Storage products at Red Hat. His focus is at the intersection of open source, distributed storage systems, data engineering, and machine learning.

Wednesday July 13, 2022 9:10am - 9:25am PDT
Regency Ballroom B

Keynote Sessions

9:30am PDT

Bloomberg Sponsored Keynote Sessions - Matthew Leonard, Storage Engineering Manager, Bloomberg LP

Speakers

Matthew Leonard

Storage Engineering Manager, Bloomberg LP

Wednesday July 13, 2022 9:30am - 9:35am PDT
Regency Ballroom B

Keynote Sessions

9:35am PDT

Lead Developers Town Hall (Submit Questions Prior to Start - Link in Description) - Venky Shankar, Neha Ojha, Ernesto Puerta, Casey Bodley, Adam King, Ilya Dryomov & Sam Just, Red Hat

This will be a town hall panel with the Ceph component leads. Please submit questions ahead of time to the etherpad, or ask them during the session: https://pad.ceph.com/p/cephalocon-2022-town-hall

Speakers

Samuel Just

Engineer, IBM

Sam began working on the Ceph project in 2011. Most of his time currently is spent working on crimson, the next generation ceph-osd implementation.

Venky Shankar

Ceph Filesystem Tech Lead, IBM

Neha Ojha

Software Engineering Manager, IBM

Ernesto Puerta

Principal Software Engineer, Red Hat

Casey Bodley

Ceph Developer, Red Hat

Adam King

Red Hat

Ilya Dryomov

Software Engineer, Red Hat

Wednesday July 13, 2022 9:35am - 10:15am PDT
Regency Ballroom B

Keynote Sessions

10:15am PDT

Break & Networking

Wednesday July 13, 2022 10:15am - 10:45am PDT
Regency Ballroom Foyer

Breaks & Networking

10:45am PDT

Ceph Crash Telemetry - Observability in Action - Yaarit Hatuka, Red Hat

To increase product observability and robustness, Ceph’s telemetry module allows users to automatically report anonymized crash dumps. Ceph’s telemetry backend runs tools that detect similarities among these reported crash events, then feed them to Ceph’s bug tracking system. In this session we will explore Ceph crash telemetry end-to-end, and how it helps the developer community to detect emerging and frequent issues encountered by production systems in the wild. We will share our insights so far, and learn how users benefit from this module, and how they can contribute.

Speakers

Yaarit Hatuka

Software Engineer, Red Hat

Software engineer at Red Hat working on Ceph telemetry and observability

Wednesday July 13, 2022 10:45am - 11:25am PDT
Regency Ballroom B

Track 1

10:45am PDT

Putting the Compute in your Storage - Federico Lucifredi & Brad Hubbard, Red Hat

This presentation walks through what is required to create an environment that will enable you to get started programming with librados and examines in detail its core capabilities, including key/value storage, atomic transactions, object cloning and snapshot support. We will then explore how to to extend the object interface using arbitrary code executed on the storage nodes themselves. We aim to show you how to bring your code to the data for high value, high volume, and background data-munging tasks, inverting the industry-standard path of retrieving the data and bringing it to the code. We will demo performing operations automatically on newly uploaded data, in truly serverless fashion.

Speakers

Federico Lucifredi

Product Management Director, Red Hat

Brad Hubbard

Principal Software Engineer, Red Hat

Brad Hubbard is a Senior Software Engineer at Red Hat and works on RADOS, the open source, distributed object storage system at the heart of Ceph.

Wednesday July 13, 2022 10:45am - 11:25am PDT
Regency Ballroom C

Track 2

10:45am PDT

Operating Ceph from the Ceph Dashboard: Past, Present and Future - Ernesto Puerta, Red Hat

Ceph Dashboard was born in 2018 as a joint effort of the Ceph Community: initially driven by SUSE (reborn from the ashes of the OpenATTIC project) and also from Red Hat, Fujitsu and individual contributors. Many challenges laid ahead: there was a long trail of unsuccessful attempts to wrap the complexities of Ceph behind an easy-to-use graphical interface. However, Ceph Dashboard built on the strengths of a relatively fresh Ceph service (Ceph Manager) and its plugin-like Python API. Leveraging this, Ceph Dashboard became another first-class citizen in the Ceph world, enabling seamless, lightweight and responsive operation of a Ceph cluster. The next major milestone in this story came 2 years ago: building on the successful experience of joint collaboration beyond vendor interests, the advent of Cephadm meant for the Dashboard not only the ability to manage a running Ceph cluster, but to create one from the ground up and control and monitor the underlying infrastructure. Now the Ceph Dashboard is turning 4 and a lot has happened since then (not all good though), and from this stage of maturity it's time to review the present and future challenges of the Ceph Dashboard and the experience of managing Ceph.

Speakers

Ernesto Puerta

Principal Software Engineer, Red Hat

Wednesday July 13, 2022 10:45am - 11:25am PDT
Regency Ballroom D

Track 3

11:30am PDT

[Virtual] Lightning Talk: Introduction to Crimson from a Newbie - Joseph Sawaya, Red Hat

Crimson is the next-generation OSD for Ceph with a new disk backend, SeaStore. Starting development for Crimson/SeaStore can be challenging, as it makes use of modern C++ and futures in the Seastar C++ library. SeaStore is also designed to accommodate ZNS devices, a new class of storage device. This talk will introduce some of those concepts and offer Joseph's perspective as an inexperienced developer being introduced to Crimson.

Speakers

Joseph Sawaya

Open Source Software Developer Intern, Red Hat Inc.

I'm a Computer Engineering Student at the University of Toronto, currently doing a 16-month internship at Red Hat. I am part of the Emerging Technologies department and I've been working on Ceph since the beginning of my internship, first on the Rook orchestrator and now on Crims... Read More →

Wednesday July 13, 2022 11:30am - 11:40am PDT
Regency Ballroom C

Lightning Talks

11:30am PDT

RGW: Sync What? Sync Info Provider: Early Peek - Yehuda Sadeh-Weinraub, Red Hat

The RGW multi-site subsystem evolved over the years from primary-replica for DR purposes to a more elaborate data replication and distribution system (as well as other extended features such as metadata indexing). The system's cloud sync feature allows sending data to an external cloud storage system, however, the system still lacks the ability to pull data from external sources. The Sync Info Provider work generalizes the sync process by defining a generic API that makes it possible to sync data from external data sources into RGW. In this presentation we will describe the RGW multisite sync capabilities, the Sync Info Provider work, and the first external module that allows syncing object data from AWS.

Speakers

Yehuda Sadeh-Weinraub

Ceph Dev, Red Hat

Ceph developer since 2008; original author of RGW, and some other ceph modules.

Wednesday July 13, 2022 11:30am - 12:10pm PDT
Regency Ballroom B

Track 1

11:30am PDT

Ceph on Windows - Alessandro Pilotti, Cloudbase Solutions

We are thrilled to present the results of the porting of Ceph RADOS, RBD and CephFS on Microsoft Windows, a community effort led by SUSE and Cloudbase Solutions. The goal consisted in porting librados and librdb on Windows Server, providing a kernel driver for exposing RBD devices natively as Windows volumes, support for Hyper-V VMs and last but not least, even CephFS. Through this work, Ceph becomes the most widely adopted cross platform Open Source distributed storage solution, allowing operators to deploy a cluster across heterogeneous operating systems and leveraging a unified distributed storage strategy on any commodity hardware. During this session we will talk about the architectural differences between Windows and Linux from a storage standpoint and how we retained the same CLI so that long time Ceph users will feel at home regardless of the underlying operating system. Performance is a key aspect of this porting, with Ceph on Windows significantly outperforming the iSCSI gateway, previously the main option for accessing RBD images from Windows nodes. There will be no lack of live demos, including automating the installation of the Windows binaries, setting up and managing a Ceph cluster across Windows and Linux nodes, spinning up Hyper-V VMs from RBD, and CephFS.

Speakers

Alessandro Pilotti

CEO, Cloudbase Solutions

Alessandro Pilotti is the CEO of Cloudbase Solutions, a company focused on cloud computing interoperability, including key contributions for porting Kubernetes on Windows, OpenStack on Hyper-V components, Open vSwitch and last but not least, porting Ceph on Windows. Alessandro lives... Read More →

Wednesday July 13, 2022 11:30am - 12:10pm PDT
Regency Ballroom D

Track 3

11:40am PDT

Lightning Talk: Improving Ceph Build and Backport Automations Using Github Actions - Deepika Upadhyay, Red Hat

Build fast, ship faster, iterate better! Learn to use GitHub Action for optimizing release and more! It is said that if your code push to release is taking more than 10 minutes, you are already at a disadvantage, not once, but with each change! The faster the build, the smoother the CI/CD pipeline, the faster the feedback and more robust your product(also more forgiving!). The research and findings were done as part of improvements in the release process done for Ceph and offer an insight into the challenges, tooling that goes into managing Ceph. Come learn about the build system of Ceph and how we are working on improving it.

Speakers

Deepika Upadhyay

Cloud Storage engineer, Koor Technologies Inc.

Deepika is currently working as a Cloud Storage Engineer at Koor Technologies and is a contributor to Rook project, she has worked as an Outreachy intern for Ceph with focus on adding tracing to Ceph OSD. She has also worked as a Ceph Storage Engineer in RADOS and RBD(Block based... Read More →

Wednesday July 13, 2022 11:40am - 11:50am PDT
Regency Ballroom C

Lightning Talks

11:50am PDT

[Virtual] Lightning Talk: Ceph and QCOW2 a Match Made in Heaven: From Live Migration to Differential Snapshots - Effi Ofer, IBM

QCOW2 is a sparse disk image format made popular by the QEMU emulator and virtualizer. Ceph RBD has its own native storage format and supports exporting its own native disk image format. However, many image ecosystems are built around the QCOW2 format, making it advantageous to use the QCOW2 image format in an external image repository for compatibility as well as advanced features. In this talk we discuss how to enable export and live migration in Ceph of sparse QCOW2 differential images using the Network Block Device, NBD, protocol. As well, we describe future changes to enable native support of the NBD protocol in Ceph in order to facilitate easier adoption of the QCOW2 format with Ceph.

Speakers

Effi Ofer

Researcher, IBM

Effi Ofer is a researcher at IBM with interests in data analytics and storage. He is currently building a next generation cloud scale block storage service. Prior to joining research, Effi was a lead developer on IBM Db2 where he lead various projects in the areas of transaction management... Read More →

Wednesday July 13, 2022 11:50am - 12:00pm PDT
Regency Ballroom C

Lightning Talks

12:00pm PDT

[Virtual] Lightning Talk: Ceph Messenger DPDkstack Development and Debugging - Chunsong Feng, Huawei

1. The NIC SR-IOV is used. Each OSD uses an exclusive VF NIC. 2. The DPDK interrupt mode is added. 3. The single-CPU core and multiple NIC queues are implemented to improve performance. 4. The admin socket command is added to obtain the NIC status, collect statistics, and locate faults. 5. Adjust the CEPH throttling parameters, TCP, and DPDK packet sending and receiving buffer sizes to prevent packet loss and retransmission. 6. The Crimson message component uses the Seastar DPDK.

Speakers

Chunsong Feng

senior software engineer, huawei

7 years in distributed storage development

Wednesday July 13, 2022 12:00pm - 12:10pm PDT
Regency Ballroom C

Lightning Talks

12:15pm PDT

We Added 6 Petabytes of Ceph Storage and No Clients Noticed! Here’s How We Did It. - Abhijeet Agrawal & Matthew Leonard, Bloomberg LP

The Storage Engineering team is responsible for the oversight of Bloomberg’s Ceph infrastructure. These clusters range in scale from hundreds of terabytes, to more than eight petabytes. Bloomberg is facing many unique challenges in terms of our ability to continue scaling live clusters without impacting our clients, while having no offline maintenance windows or lowering our performance standards. The Bloomberg Storage Engineering team successfully completed the live addition of more than 6 petabytes of raw capacity to two of our Ceph-based object stores. We added 50% of raw new capacity to an online cluster without making it unavailable for clients -- in under a single hour. This presentation will document the origins, planning, execution, and bugs Bloomberg faced during this massive undertaking. This project touched on multiple facets of the Ceph project, including a deep dive into the inner workings of the CRUSH algorithm, fundamentals of placement group peering, manipulating data recovery operations, as well as automated scaling of placement groups to scale with the newly-added raw capacity.

Speakers

Abhijeet Agrawal

Senior Software Developer, Bloomberg LP

Matthew Leonard

Storage Engineering Manager, Bloomberg LP

Wednesday July 13, 2022 12:15pm - 12:55pm PDT
Regency Ballroom B

Track 1

12:15pm PDT

Stretch Clusters in Ceph: Algorithms, Use Cases, and Improvements - Gregory Farnum, Red Hat

While Ceph is designed for use in a single data center, users have deployed “stretch” clusters across multiple data centers for many years, and deploying Ceph to back Red Hat’s OpenShift Data Foundation product required us to support that workload explicitly and well — in particular, in the face of netsplits. At FOSDEM 2020, I reported on improvements to our monitor leader elections and plans for the OSD peering process to keep data available without breaking our data integrity guarantees. This talk presents the initial completed version of that work from an algorithm and programmer perspective: new monitor election algorithms and OSD peering changes to support a 2-site cluster, which is all present in the Pacific point releases. Lastly, we’ll discuss what still needs to be done to expand stretch functionality into more environments and use cases.

Speakers

Gregory Farnum

Software Engineering Manager, IBM

Greg Farnum has been in the core Ceph development group since 2009. Greg has done major work on all components of the Ceph ecosystem, previously served as the CephFS tech lead, and manages IBM’s CephFS engineering team.

Wednesday July 13, 2022 12:15pm - 12:55pm PDT
Regency Ballroom C

Track 2

12:55pm PDT

Lunch Break

Wednesday July 13, 2022 12:55pm - 1:55pm PDT
Regency Ballroom A

Breaks & Networking

1:55pm PDT

NVMe-over-Fabrics Support for Ceph - Jonas Pfefferle, IBM & Scott Peterson, Intel

NVMe-over-Fabrics (NVMeoF) is an open, widely adopted, defacto standard in high performance remote block storage access. More and more storage vendors are introducing NVMeoF target support, with hardware offloads both for NVMeoF targets and initiators. Ceph does not support the NVMeoF protocol for block storage access; its clients use the Ceph RADOS protocol to access RBD images for good reason: RADOS is a distributed m-to-n protocol that provides reliable object access to sharded and replicated (or erasure coded) Ceph storage. However, there are good reasons to enable NVMeoF for Ceph: to enable its use in datacenters that are already utilizing storage hardware that offload NVMeoF capabilities, and to allow existing NVMeoF storage users to easily migrate to Ceph. In this talk we present our effort to integrate a native NVMeoF target for Ceph RBD. We discuss some of the challenges of implementing such a support for Ceph including subsystem/namespace discovery, multi-pathing for fault tolerance and performance, authentication and access control (e.g., namespace masking). Furthermore, we describe how the NVMeoF target design can be extended to allow reducing additional network hops by leveraging the Ceph CRUSH algorithm (ADNN).

Speakers

Jonas Pfefferle

Software Engineer, IBM

Jonas Pfefferle is a software engineer at IBM Research. He holds a Master's degree in Computer Science from ETH Zurich. Jonas's research interests are virtualized distributed systems and data centers, specifically in state-of-the-art network and storage technologies. Currently, he... Read More →

Scott Peterson

Intel

Wednesday July 13, 2022 1:55pm - 2:35pm PDT
Regency Ballroom C

Track 2

1:55pm PDT

Introducing Sibench: A New Open Source Benchmarking Optimized for Ceph - Danny Abukalam, SoftIron

Benchmarking Ceph has always been a complex task - there are lots of tools but many have drawbacks and are written for more general-purpose use. For Ceph we need to benchmark Librados, RBD, CephFS, and RGW and each of these protocols has unique challenges and typical deployment scenarios. Not only that, Ceph works better at scale and so we need to ensure that we can build a benchmarking system that will also scale and be able to generate an adequate load at large scale. When sizing clusters it’s useful not just to sweep over typical things like object size and count, but also being able to sweep over workers to identify the correct benchmarking cluster size, and over OSD count to identify a suitable cluster size for our performance requirements is also useful. Sibench is an in-house tool that we’ve written at SoftIron to help us benchmark our solution and do a lot of the heavy-lifting for us. In this talk we’ll go over our benchmarking journey, what prompted us to write Sibench, how it works, and maybe also show a demo of the project in action.

Speakers

Danny Abukalam

Director, Product Engineering, SoftIron

Danny runs product engineering at SoftIron, working to turn raw and untamed free software projects into commercially supported, turn-key infrastructure products. Most of his time these days is spent taming Ceph and SONiC, and supporting customers in new and interesting environments... Read More →

Wednesday July 13, 2022 1:55pm - 2:35pm PDT
Regency Ballroom D

Track 3

2:40pm PDT

[Virtual] Revealing BlueStore Corruption Bugs in Containerized Ceph Clusters - Satoru Takeuchi, Cybozu

Cybozu has been running and testing their Rook/Ceph clusters for two years. During this time, they have suffered from a bunch of BlueStore corruption (e.g. #51034 and #53184). Most corruptions happened just after OSD creation or on restarting OSDs. They have been able to detect these problems because the nodes in their clusters are restarted frequently and lots of OSD creation happens for each integration test. These scenarios are not so popular in traditional Ceph clusters but are common in containerized Ceph clusters. They will share what the known problems are in detail and how they have overcome these problems with the Ceph community. In addition, they will also propose improvements to the QA process to prevent similar problems in the future.

Speakers

Satoru Takeuchi

Storage Architect, Cybozu

Satoru is a software developer of an on-premise Kubernetes cluster and Rook/Ceph cluster at Cybozu. He is a maintainer of Rook/Ceph. He made a presentation named "Revealing BlueStore Corruption Bugs in Containerized Ceph Clusters" at Ceph Virtual 2022 and also made some Rook maintainer... Read More →

Wednesday July 13, 2022 2:40pm - 3:20pm PDT
Regency Ballroom B

Track 1

2:40pm PDT

[Virtual] The Effort to Exploit Modern SSDs on Ceph - Myoungwon Oh, Samsung Electronics

Modern flash storage devices have evolved in many ways in the past few years. Z-SSD provides higher IOPS and throughput than existing NVMe SSD. ZNS SSD significantly reduces write amplification and expands SSD life span at a low cost. In addition, NVMeOF technology is about to replace existing storage network with the protocol that exploits NVMe command over RDMA. In this talk, I will introduce four topics with regard to changes on modern SSDs I currently working on: 1) Seastore development for fast NVMe device including design proposal for ZNS SSD’s append command, 2) Tiering/Dedup status, 3) NVMeOF on Ceph. Firstly, I will explain a write strategy based on circular bounded journal and random block manager on Seastore and propose how the append command, which has been newly added in ZNS spec.,can be used in Seastore. Next, tiering/dedup feature development status will be introduced. Finally, I will give some thoughts on NVMeOF and Ceph.

Speakers

Myoungwon Oh

Staff Engineer, Samsung Electronics

Myoungwon has over 10+ years of experience in distributed system and storage systems including distributed ledger. His focus is on resolving performance issues such as Linux kernel, distributed storage system, consensus protocol, and KVM. He has been working on developing Ceph as... Read More →

Wednesday July 13, 2022 2:40pm - 3:20pm PDT
Regency Ballroom C

Track 2

2:40pm PDT

Data Security and Storage Hardening in Rook and Ceph - Federico Lucifredi & Michael Hackett, Red Hat

We explore the security model exposed by Rook with Ceph, the leading software-defined storage platform of the Open Source world. Digging increasingly deeper in the stack, we examine hardening options for Ceph storage appropriate for a variety of threat profiles. Options include defining a threat model, limiting the blast radius of an attack by implementing separate security zones, the use of encryption at rest and in-flight and FIPS 140-2 validated ciphers, hardened builds and default configuration, as well as user access controls and key management. Data retention and secure deletion are also addressed. The very process of containerization creates additional security benefits with lightweight separation of domains. Rook makes the process of applying hardening options easier, as this becomes a matter of simply modifying a .yaml file with the appropriate security context upon creation, making it a snap to apply the standard hardening options of Ceph to a container-based storage system.

Speakers

Federico Lucifredi

Product Management Director, Red Hat

Michael Hackett

Senior Principal Product Experience Engineer, Red Hat

Principal Product Experience Engineer at Red Hat | IBM Michael Hackett is a Principal Product Experience Engineer at Red Hat with excellent technical, analytical and communication skills demonstrated by certifications and over 13 years of experience. He has co-published two books... Read More →

Wednesday July 13, 2022 2:40pm - 3:20pm PDT
Regency Ballroom D

Track 3

3:20pm PDT

Break & Networking

Wednesday July 13, 2022 3:20pm - 3:50pm PDT
Regency Ballroom Foyer

Breaks & Networking

3:50pm PDT

Bringing emails@ceph Into the Field - Danny Al-Gaaf, Deutsche Telekom AG

Deutsche Telekom operates a growing email system with several million accounts and billions of emails stored on traditional NFS. Four years ago we introduced librmb (librados mailbox) to the community, a universal open source library used to store emails in a Ceph cluster. Librmb uses RADOS to store email directly in Ceph to achieve maximum performance through parallel access from many email gateways simultaneously, for millions of active customers. Email systems are much to complex to be simulated in a way which would allow to verify if the switch to librmb will work for a large number of users, even after intensive testing. Therefore a field test with real and active customers is necessary to provide an educated guess on the behavior of the final setup. This presentation will cover the current project status, how we run all relevant test cases, the field test setup and a progress report onto the first results and experiences from the field test with around a million customers. Previous presentations have been about the development around librmb, this talk covers bringing the project into production.

Speakers

Danny Al-Gaaf

Senior Cloud Technologist, Deutsche Telekom AG

Danny Al-Gaaf is a Senior Cloud Technologist working for Deutsche Telekom. As a Ceph upstream developer he is a driver for using Ceph at Deutsche Telekom. For the last 15 years his professional focus has been on Linux and open source. He works actively in several upstream communities... Read More →

Wednesday July 13, 2022 3:50pm - 4:30pm PDT
Regency Ballroom B

Track 1

3:50pm PDT

Lessons Learned from Hardware Acceleration Initiatives for Ceph-specific Workloads - Harry Richardson & Lionel Corbet, SoftIron

For a number of years SoftIron has been researching a number of approaches for accelerating workloads in Ceph. This has entailed building an underlying framework for hardware acceleration and dedicated communication protocols. Designed to be generic, it gives us a flexible platform to run applications or workloads in hardware with the appropriate design - in an entirely transparent way to a Ceph user. Our first workload was erasure coding, and we’ve now moved on to looking at compression algorithms with specific Ceph workloads. In this talk we’ll cover the original design goals for our project, walk through our architecture and design, and demonstrate how it has evolved over time.

Speakers

Harry Richardson

Chief Scientist, SoftIron

Harry is Chief Scientist at SoftIron. His background is in security, cryptography and high-performance computing. His primary obsessions are creating elegant APIs and ensuring that every network protocol has a version field.

Lionel Corbet

Principal FPGA Engineer, SoftIron

Lionel is Principal FPGA Engineer at SoftIron. He has a background in logic synthesis and static timing analysis; both as application engineer and R&D engineer. Lionel previously worked at Altera and has been involved in taking multiple FPGAs to market.

Wednesday July 13, 2022 3:50pm - 4:30pm PDT
Regency Ballroom C

Track 2

3:50pm PDT

[Virtual] Secure Token Service in Ceph - Pritha Srivastava, Red Hat

This talk will outline how implementation of Secure Token Service in Ceph has increased its usability. This will cover the basic concepts of STS (roles and IAM policies) and then will talk about details of APIs that help in cross account access and web identity federation. This talk will also cover STSLite which was implemented to aid in reducing the traffic that every s3 request generates for Keystone. Advanced topics like attribute based access control that helps in simplifying IAM policies and their maintenance will be covered. And lastly this talk will cover details about a custom JAVA module that has been developed to provide temporary credentials (and refresh them under the covers), called Custom Credentials Provider. Introduction to STS in RGW Basic concepts like (Role, IAM policies etc) STS APIs that have been supported (AssumeRole, AssumeRoleWithWebIdentity) STS Lite (GetSessionToken) Attribute based access control in STS Custom Credentials Provider Ongoing work (STS metadata replication in multisite)

Speakers

Pritha Srivastava

IBM

I am a Software Developer with IBM and have been working on Ceph RGW for the last six and a half years. Prior to that I worked with HP for StoreVirtual. I have completed my MS from University of Colorado at Boulder.

Wednesday July 13, 2022 3:50pm - 4:30pm PDT
Regency Ballroom D

Track 3

4:35pm PDT

New Workload Balancer in Ceph - Josh Salomon & Laura Flores, Red Hat

One of the new features in the Quincy release is the introduction of a new workload balancer (aka primary balancer). While capacity balancing exists and works well since the introduction of the upmap balancer, the issue of primary balancing in order to even the load on all the OSDs was never handled. This proves to be a performance problem, especially in small clusters and in pools with less PGs. In this presentation we will discuss the difference (and sometimes the contradiction) between capacity balancing and workload balancing, explain what we did for Quincy, and outline future plans to further improve the Ceph balancing process.

Speakers

Josh Salomon

Senior Principal Software Engineer, Red Hat

Josh has more than 30 years of experience in software development, more than 15 years in architecture positions and 7 years of experience as an architect in the storage development industry (in ScaleIO and Red Hat).

Laura Flores

Software Engineer, IBM

Laura is an Software Engineer at IBM. She is a member of the core RADOS team. Her work primarily encompasses the Ceph read balancer, the Telemetry performance channel, and upstream lab testing and maintenance. Laura holds a bachelor's degree in Computer Science & English Literature... Read More →

Wednesday July 13, 2022 4:35pm - 5:15pm PDT
Regency Ballroom B

Track 1

4:35pm PDT

[Virtual] Accelerating PMEM Device Operations in BlueStore with Hardware Based Memory Offloading Technique - Ziye Yang, Intel

With more and more fast devices (especially persistent memory) equipped in data center, there is great pressure on CPU to drive those devices (e.g., Intel Optane DC persistent memory) for persistency purpose under heavy workloads. Because there is no DMA related capability provided by persistent memory compared with those hdds and SSDs. And the same issue also exists in Ceph while using persistent memory. We would like to address such pain points by leveraging memory offloading devices (e.g., DSA). So generally in this talk, we will talk: 1) While persistent memory integration is not very successful in Ceph due to the high CPU overhead while performing I/O operations on persistency device; 2) We introduce the memory offloading devices (e.g., DSA) in order to offload the CPU pressure while doing I/Os; 3) We will describe the main change in pmem device (i.e., src/blk/pmemdevice.cc) and state how we can achieve the offloading including the challenges. 4) We would like to have some early performance results if Intel's SPR platform is available in public.

Speakers

Ziye Yang

Staff Cloud software engineer, Intel

Ziye Yang is a staff software engineer at Intel and is currently involved in cloud native related projects. Before that, Ziye worked at EMC for 4.5 years. Ziye is interested in system virtualization, file system and storage related research and development work. Ziye currently has... Read More →

Wednesday July 13, 2022 4:35pm - 5:15pm PDT
Regency Ballroom C

Track 2

4:35pm PDT

[Virtual] Chasing Bad Checksums: A Journey through Ceph, TCMalloc, and the Linux kernel - Mauricio Faria de Oliveira & Dan Hill, Canonical

In this talk Mauricio and Dan will discuss the steps for root cause analysis of transient checksum errors seen with Ceph on Ubuntu 18.04 LTS (probably issue #22464 [1]), covering components such as Ceph, TCMalloc, and the Linux kernel. They will describe the problem symptoms, approaches used for analysis, instrumentation, and synthetic reproducers. Drilling down to the root cause, and covering the proposed fix. There will be some (light) code walkthrough, changes for instrumentation, and usage of tracing tools in the Linux kernel (eg, ftrace, perf) thus experience with development, instrumentation, tracing, is helpful, but not strictly required. [1] Issue #22464 – https://tracker.ceph.com/issues/22464#note-50

Speakers

Mauricio Faria de Oliveira

Sustaining Engineer, Canonical

Mauricio works in the Sustaining Engineering team at Canonical, handling problem reports from Ubuntu Advantage support customers primarily in the Linux kernel, some userspace packages, and more recently, a bit of Ceph.

Dan Hill

Sustaining Engineer, Canonical

Dan is a Sustaining Engineer working in TechOps at Canonical. He has been working with distributed storage solutions since 2016, and has over 15 years of experience working with enterprise storage customers. Recently, Dan has been focused on Ceph performance tuning for application-specific... Read More →

Wednesday July 13, 2022 4:35pm - 5:15pm PDT
Regency Ballroom D

Track 3

5:20pm PDT

pgremapper: CRUSHing Cluster OperationaComplexity - Joshua Baergen, DigitalOcean

When working with production Ceph clusters, making changes to your CRUSH map (e.g. adding storage) can be highly disruptive. These changes can result in days to weeks of backfill that's hard to control due to the way that Ceph's backfill scheduling works, holding your cluster hostage by blocking other maintenance activities and slowing down recovery from failure events. In 2018, Dan van der Ster of CERN IT presented techniques ("Mastering Ceph Operations: Upmap and the Mgr Balancer") that they had developed to improve the safety of CRUSH changes. DigitalOcean's storage team has built on these techniques with an open-source tool called pgremapper. In this talk, Joshua Baergen will discuss the problems that operators encounter with CRUSH changes at scale and how DigitalOcean built pgremapper to control and speed up CRUSH-induced backfill.

Speakers

Joshua Baergen

Senior Engineer II, DigitalOcean

Joshua Baergen is the Technical Lead of the Storage Systems team at DigitalOcean. This team is responsible for designing and operating the persistence layers for the Volumes and Spaces products (which are built on Ceph) as well as for Droplet images, snapshots, and backups. He has... Read More →

Wednesday July 13, 2022 5:20pm - 6:00pm PDT
Regency Ballroom B

Track 1

5:20pm PDT

RGW Zipper - Daniel Gryniewicz & Soumya Koduri, Red Hat

RGW was developed to provide object access (S3/Swift) to a Ceph cluster. The Zipper abstraction API divides the RGW into an upper half containing the Operations (Ops) for S3 and Swift, and a lower half, called a Store, containing the details of how to store data and metadata. This allows the same Ops code to provide correct S3 and Swift semantics via a variety of storage platforms. The primary Store is the current RadosStore, which provides access to a Ceph cluster via RADOS. However, new Stores are possible that store the data in any desired platform. One such Store, called DBStore, has been developed that stores data in SQL, and specifically in a local SQLite database. Additional Stores, such as S3, are planned to provide additional flexibility. Zipper also allows intermediate Filter layers that can transform Ops, perform policy (such as directing different objects to different Stores), or perform caching for data and metadata. The first planned Filter is a LuaFilter, which will allow rapid prototyping and testing of other filters. An individual instance of RGW will consist of a stack of Filters, along with one or more Stores providing actual data. This presentation will cover information about Zipper, about the existing DBStore, and plans for the future.

Speakers

Soumya Koduri

Principal Software Engineer, Red Hat

Soumya has been working on Ceph and RGW for more than 5 years at Red Hat, and is the primary developer of the DBStore backend for Zipper. Before working on Ceph, she worked on Gluster for Red Hat.

Daniel Gryniewicz

Manager, Software Engineering, Red Hat

Daniel has been working on Ceph and RGW for 9 years, first with CohortFS and later with Red Hat. He is the primary developer for the Zipper project. Before Ceph, he worked in networking on routing and switching software and hardware.

Wednesday July 13, 2022 5:20pm - 6:00pm PDT
Regency Ballroom C

Track 2

5:20pm PDT

Connecting The Dots: Benchmarking Ceph at Scale - Shon Paz & Ido Pal, Red Hat

We have a customer that required to know what are Ceph's limitations top to bottom. In the benchmark process, we've tested Ceph at all levels, starting from the underlying hardware, to the S3 object storage service exposed by Ceph RGW. The main cause of this benchmark was to create the connection between all layers: Hardware (RAM, Disk, CPU, Network) --> OS (sysctl tunning) --> Ceph RADOS --> Ceph RGW --> User experience. The artifact from this benchmark should be the SLA and SLO our customer can expose to his own customers within the organization. This customer has pushed Ceph's architecture to the edge, while allowing his customers to consume Amazon-like storage classes (on for fast processing and low latency response time, and the other for archival, quite similar to Amazon S3 storage classes) while dividing the cluster to performance domains to prevent noisy neighbors (using Realms, Zones, and different root buckets). Having this architecture pushed the benchmark complexity even further, and we'd like to share this with the Ceph community to ease their understanding on how to manage Ceph at scale.

Speakers

Shon Paz

Sr. Solution Architect, Red Hat

A Data Solutions Architect, experienced in developing and maintaining a cloud-native approach based on the agile methodology for all modern IT supply chains such as Distributed Cloud, Data & AI/ML, Automation, Cloud Native Development, Telco.

Ido Pal

Solutions Architect, Red Hat

Solutions architect with expertise in designing and implementing cutting-edge technology solutions. Currently, holding the position of defensive sector solution architect at Red Hat-Israel, Greece, and Cyprus regions. Prior to SA role, I worked as a storage and data services consultant... Read More →

Wednesday July 13, 2022 5:20pm - 6:00pm PDT
Regency Ballroom D

Track 3