How and Where Ceph stores

Ceph provides object, block and file storage in one unified system. The object in this sentence means that in Amazon S3 and MinIO. When Ceph is called as an object storage as whole, the word object in that context refers RADOS object. The RADOS objects are the foundation of that unified system where all the replication and dynamic data redistribution happens and scalability are provided.

This article explains how data are stored, make a link from data to RADOS objects. Let’s start from a surface: a block storage. Ceph has a block storage interface called RBD. Let’s create a RBD pool and put some data in it.

sudo ceph osd pool create test-rbd-pool
sudo rbd pool init test-rbd-pool
sudo rbd create --size 5G test-rbd-pool/test-image
sudo rbd map test-image -p test-rbd-pool
sudo mkfs.ext4 /dev/rbd/test-rbd-pool/test-image
sudo mkdir /mnt/test-image
sudo mount /dev/rbd/test-rbd-pool/test-image /mnt/test-image
sudo cp ubuntu-24.04-live-server-amd64.iso /mnt/test-image/

Here, we create an RBD pool (test-rbd-pool), create a block disk (test-image), mount the disk on a directory, and put ubuntu-24.04 server installer iso file (~2.6 GiB). Where is this file stored in the Ceph cluster?

Everything will be stored as RADOS objects. The object default size is 4 MiB. So, the file is split into many RADOS objects and the objects are placed into OSDs.

To see the objects for the file and related information, we can run

sudo rados -p test-rbd-pool ls
sudo rbd…