Oct 31, 2023
Innovating databases is hard
Our industry is undergoing massive and fast-paced changes every year: new programming languages, ways to deploy the infrastructure and apps, cloud, serverless, everything-as-a-code, and every second year a massive disruption hits this or other layer of the application stack. There’s one piece however that remains relatively conservative - and that’s our databases.
During our discussions with market experts and practitioners and also looking at our own experiences from building a big data platforms as well as aiding our past clients in managing application performance and reliability, we’ve learned a few interesting observations:
Developers hate to change databases: schema changes, changes to indexing, forget about whole engine swap - sleepless nights and lots of stress guaranteed.
Tech leaders (CTOs, Chief architects) are afraid of innovation in this realm: as we heard, none of them lost their position due to lack of innovation around databases, but there were quite a few who lost it as a result of a failed DB refactoring that affected end customers.
New DB engines need many years to get solid traction: while there’s quite a few of them that started over the past few years, it is always a very long process.
Classic DB engines stick around despite lack of innovation: can you envision a web browser that doesn't change for years? Or a programming language that retains decade-old libraries? For some reason people are reluctant to change and very often feel locked-in to their incumbent DB that is inferior compared to new offerings on the market.
Just look at the chart above and compare it to an equivalent for web browsers, programming languages or web server engines
How does it speak about our abilities to make applications better and our customers happier? How well are we as an industry in general, but specifically as every single organization prepared for the upcoming AI boom that has its own requirements regarding how data is stored and accessed?
Challenges of change
That doesn't mean people don’t change DBs at all. But it is always hard and painful, unless you are still a small company either in terms of data stored or complexity of the application architecture and teams. With enough critical mass, this sooner or later becomes a ticking bomb. If you are a CTO and want your applications to run on the best and most modern DB platform, or maybe because you are being pressed by your CFO for budget and maybe also having problems expanding your team because nobody wants to work with legacy tacks, how do you convince ALL your teams (and internal customers) to re-write ALL their queries at once? Who’s going to test it and ensure everything is going to work after the switch?
This is hard. People sometimes spend years preparing for this moment and quite often have to go back because something was not ready or failed to work. How many times can you do the roll-back before your credibility as CTO is gone?
New needs ahead…
The proliferation of generative AI-driven applications has all the hallmarks of something that could become a massive disruption to how we do things every day, comparable only to the inventions of the World Wide Web or social networking. Many companies are already implementing AI-driven functionality in their apps, and almost everyone is looking into the topic to understand how to ride this new wave to benefit their customers.
One of the most important things these people learn is that AI, as great as it sounds, among many other challenges, also requires a very specific approach to storing and analyzing data. Very few existing classic databases are immediately prepared to handle this task; very often, a dedicated database platform is required to be successful in this area.
That means the need for innovation in the DB realm will be (or already is) bigger than ever before. The amount of changes, experimentation, and quick iterations without risking existing business will be crucial for those who want to achieve satisfactory results faster than the competition.
I’ll leave you, dear reader, with these thoughts to contemplate on the problem. If you navigate to the same conclusions as the vast majority of experts we talked to, you will enjoy the following blogs a lot! Till then…
