Ceph Experience


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Ceph Experience
# 1  
Old 03-16-2020
Ceph Experience

Hi,

I got asked if I can list some additional informations about ceph "in production" at my site.

Well, the small hyperconverged cluster here with ceph is not really in production by the customer. It's handed over to the customer. The customer creates his new setup on the hardware, is still running tests and minor applications and we are responsible for monitoring, backup and updating the system. Therefore the virtual machines get automatically moved around the cluster while staying live once every week in an uncritical time window and the host systems get their updates.

Terminology
  • Node

    A node is a physical servers which serves for different storage units(="OSDs").
  • OSD

    An OSD is a storage unit in ceph. It's short for Object storage Daemon. The process which manages one storage unit.
  • Hyperconverged

    Hyperconverged means the nodes have the function to serve the ceph cluster as well as serving as compute nodes for virtual machines.

The facts
  • Node count 3
  • OSD count 12
  • Locations 1 (everything on the same location)
  • Type of intended usage Java Applications connected to external Postgres Database servers
  • Network connection 2 x 10 Gb on each Node (Bonding: active/passive)
  • Used Switches Huawei CE6810-32T16S4Q-LI(redundant setup, lacp connected active/passive)
  • Used OSD Disks INTEL SSDSC2KG019T8 2 TB
  • Used Virtualization Environment Proxmox PVE(based on Debian Linux 10 - Buster)
  • Used Guests Various(Debian, Ubuntu, BSD)
  • Ceph Redundancy Factor 3 - (in a healthy system there are 3 copies of each data block)

Further Questions
  • What happens if an OSD fails?

    When an OSD fails up to a configured time value requests are served without doing anything. When the timeout happened the full redundancy is being restored by balancing out the data on the remaining OSDs. At any time you may replace an OSD(SATA-SSD) and reintegrate it with a simple ceph command.
  • What happens if Node fails?

    When a Node fails 4 OSDs are going down with it. Since the redundancy factor is 3 every node holds a copy of the data. So the remaining redundancy will be 2 in this setup. The overall speed capacity will go down, but the cluster is still fully functional and still redundant.
  • How is the system monitored?

    The system is monitored via our in-house monitoring system(check_mk). The monitoring includes SSD SMART status, Ceph Health and Ceph OSD health. Ceph has it's own internal health check system which can be queried on all nodes and wich is queried from our monitoring system.
  • How long is the system running?

    The system is running for 12 month now without any issue so far.
  • Anything else to report?

    Due to a firmware issue, a new firmware had to be installed on all ssd devices. The servers were needed to be powered off and on again for the new firmware to become active. That was no problem to do while the cluster was running.

    Ceph is like many HA-Setups sensitive to clock synchronisation. The server clocks should be fully synchronized at any time.

Other notes:

Ceph is not the ideal system for databases, since it adds network latency to the storage. If you need high speed databases, ceph might not be the best option. Every Database provides its own replication method. I would prefer that to any external replication method. But well, if you through a lot of money onto the hardware with high performance components, that will allow that too.

Why and when Ceph is a good choice

Ceph advantages

Ceph is a scalable, thin provisioned, self healing, redundant network storage. From what I've experienced so far the management is very easy. No complex cluster configurations. (Well, you can setup you own replication maps with different crash zones at any complexity you like, so for example if you span your ceph accross different data centers(DC) your redundancy is still the best way possible when one data center goes offline in contrast to the worst case scenario to loose all replicas because those are all located in the offline DC. The process of figuring out the location for an address is called crush map in ceph.). If your redundancy allows it, just reboot a node when needed for maintenance - bring it back online - everything syncs automatically.

Ceph use cases

Probably there are a lot. The one that I use is the here mentioned use as storage for virtualization. It allows thin provisioning, data sharing between cluster nodes and snapshotting of virtual machines.

Last edited by stomp; 03-17-2020 at 09:11 AM..
These 5 Users Gave Thanks to stomp For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. AIX

Looking for AIX Practical Experience

I have been in IT for around 7 years now and have benefited greatly from Open Source as well as free commercial offerings like Vmware Server that allow me to setup virtual environments and get some hands on time with different OS's and software. I am starting a new job in a few weeks which has... (5 Replies)
Discussion started by: supercrazy1
5 Replies

2. UNIX for Advanced & Expert Users

Comparing experience with AIX, HP-UX, and Solaris

I'm investigating AIX/HP-UX/Solaris for use in a research environment. Although there is plenty of technical documentation online, some important questions can only be answered from long user experience. I'd like to hear whatever you can contribute if you can compare at least two of these. To... (16 Replies)
Discussion started by: wenp
16 Replies

3. UNIX for Dummies Questions & Answers

Anyone have experience with Quantum DLT8000

I know this is off subject, but I have been having fits with a Quantum M1500 library with DTL8000 drives, and figured that some of you have had some experience with them. Thanks, Chuck (1 Reply)
Discussion started by: 98_1LE
1 Replies

4. UNIX for Advanced & Expert Users

How to get more experience???

Hi everybody, I'm kinda fed up with all the theoritical part of programming. I wonder if there is any online project or something where i can participate to practice my skills? Any other suggestions how to build up practical side of Unix C programming? Thank you all. (1 Reply)
Discussion started by: solvman
1 Replies

5. Programming

How to get more experience???

I'm kinda fed up with all the theoretical side of the programming. now i'd love to participate in some project dealing with C network programming under Unix. My question is: is there any online programs or projects where i can really participate? Any other suggestions how to build up practical... (1 Reply)
Discussion started by: solvman
1 Replies
Login or Register to Ask a Question