Today, CNCF is publishing the third of our quarterly CNCF End User Technology Radars; the topic for this Technology Radar is database storage.
In June, we launched the CNCF End User Technology Radar, a new initiative from the CNCF End User Community. This is a group of more than 140 top companies and startups who meet regularly to discuss challenges and best practices when adopting cloud native technologies. The goal of the CNCF End User Technology Radar is to share what tools are actively being used by end users, the tools they would recommend, and their patterns of usage. More information about the methodology can be found here.
Go to radar.cncf.io to find other Radars, the votes, and industries represented.
Be sure to tune in to KubeCon + CloudNativeCon North America Virtual on Friday, November 20, at 4:00 pm ET for the TechRadar session featuring a live Q&A with Cheryl Hung and the Radar editors to hear more of their thoughts on the results and what they learned.
The survey on database storage
During October 2020, the members of the End User Community were asked which database storage solutions they had assessed, trialed, and subsequently adopted. A total of 273 data points were sorted and reviewed to determine the final positions.
This may be read as:
- The six tools in the “Adopt” ring are widely adopted and recommended by the respondents.
- The technologies in “Trial” were recommended by a number of end users, but they either did not have enough total responses or only received a few “Adopt” votes.
- Projects in “Assess” lack clear consensus. MariaDB, CockroachDB, and Vitess have some awareness, but only a few users recommend adoption. Organizations that are looking for new database storage solutions should take into account their own requirements when considering those in “Assess.”
The themes
The themes describe interesting patterns and editor observations:
1. Companies are cautious with their data and slow to adopt newer technologies.
The newer technologies, like CockroachDB, TiDB, and Vitess, have not been widely studied by many of the companies that responded. CockroachDB and Vitess ended up in “Assess.”
A few different factors drive organizations to take caution with their data, but the main reason is that it is difficult to manage. There is a lot of overhead involved when moving massive amounts – sometimes terabytes or petabytes – of data from one data storage technology to another. The benefits have to outweigh the costs for a move to make sense. Even when transitioning from legacy solutions to the cloud, some companies consider integrating the tools they already have in place.
Another factor may be that it is harder to hire developers with expertise in these newer technologies. All of the projects in “Assess” (CockroachDB, MariaDB, and Vitess) are API-compatible with the technologies in “Adopt,” so organizations can integrate elements without having to transition to a new tool.
Interestingly, etcd did not make it onto the radar. etcd usage is mostly driven by Kubernetes, as it’s the only supported backend. Companies rarely use etcd as a standalone choice for hosting their data, which means companies transitioning from legacy infrastructures are less likely to have experience with it.
2. Choosing a managed database service depends heavily on use cases.
We were surprised to see low adoption of cloud-managed services. This led us to realize that the use of managed database services can vary widely depending on the use case – where the application is deployed, the amount of data being stored, if a cloud provider is already being used. If a company has a large amount of data, for instance, there can be significant cost overhead to using a managed database solution.
The use of cloud-managed databases is likely impacted by whether or not a company is already using a particular cloud provider. For instance, if a company exclusively uses AWS for its other cloud services, chances are they will also be using related AWS database technologies. If they host on premises, they most likely won’t host just their databases in the cloud.
In other instances, decisions may be driven by data security and protection. Companies working with more sensitive data are more likely to host databases in-house and may even be required to.
While we did ask about RDS, it didn’t end up in the final radar. We removed it as its usage was vague, and it was not clear what particular technology is being used.
3. Keep an open mind!
We found that database storage is still an evolving space. Some projects have been around for quite a long time, which has likely driven up their adoption rate, especially considering usage in large companies. Many of these legacy technologies have a good reputation, given they are stable and proven to work.
New cloud native projects are emerging, and many of these are better suited for new use cases. There are several new technologies with specialized use cases that did not make it into the radar; we did not see any graph databases, long-term storage for metrics, or serverless databases.
Ultimately, you have to choose the right technology for you, your team, and your organization. Does it make more sense to use a technology you can drop in and replace vs. burdening engineers with forcing something to fit? Is there a thriving community behind the open source project you are considering? Do your research and go with what makes sense, but don’t be afraid to try something new!
The editors
Jackie Fong is an Engineering Leader in the Platform organization at Ticketmaster responsible for Container Orchestration, CI/CD, Observability, and Developer Experience. At the beginning of 2020, Jackie started a Service Mesh End User Group at CNCF and serves as a co-chair.
Smaïne Kahlouch is a DevOps Team Leader at Dailymotion. He leads a team in charge of building a reliable and scalable platform, as well as the release management. He is the organizer of the Cloud Native Computing Foundation meetup in Paris and a CNCF Ambassador in France. Twitter: _smana_
Mya Pitzeruse was the Principal Engineer for the Service Platforms Group at Indeed, where she designed and guided the development of their cloud native platform across compute, storage, and observability. Twitter: myajpitz
Read more
Case studies: Read how JD.com, Slack, and Square are handling database storage using CNCF technologies.
What’s next
The next CNCF End User Technology Radar is targeted for February 2021, focusing on a different topic in cloud native. Vote to help decide the topic for the next CNCF End User Technology Radar.
Join the CNCF End User Community to:
- Find out who exactly is using each project and read their comments.
- Contribute to and edit future CNCF End User Technology Radars.
We are excited to provide this report to the community, and we’d love to hear what you think. Email feedback to info@cncf.io.
About the methodology
In October 2020, the 140 companies in the CNCF End User Community were asked to describe what their companies recommended for different solutions: Hold, Assess, Trial, or Adopt. They could also give more detailed comments. As the answers were submitted via a Google spreadsheet, they were neither private nor anonymized within the group.
A total of 29 companies submitted 273 data points on 36 solutions. These were sorted in order to determine the final positions. Finally, the themes were written to reflect broader patterns, in the opinion of the editors.