Best tool for the job

by Pawel Brzoska

TL;DR: If your team loves Kibana and the rest of Elastic stack but your AWS/Azure/GCP bill for running Elasticsearch/OpenSearch infrastructure makes your jaw drop - handle it the Uber way with Quesma. Read on or get in touch to learn more.

Struggle of adoption

In recent years many great technologies have debuted in the database market. We have new databases that provide new functionality (e.g. vector search), more convenience (e.g. DucksDB), better performance (e.g. ClickHouse) or scalability (e.g. CockroachDB) . This proliferation is not new and what is also not new is unfortunately a slow pace of market adoption, despite the fact a lot of these technologies are cutting-edge and much better than classic vendors.

Still, the IT organizations are reluctant to try these new technologies. They already struggle with using the current database technologies efficiently, e.g. schema changes are hard and tweaking them is risky. New database stack is a long time commitment that tends to outlive the generation of engineers that support it. Engineering teams and architects are frequently conservative and stick to a handful of choices. That approach has its cons, however it frequently ends up in bad compromises and increased operational cost.

Why is that? Often it is a question of skills, learning new toys, sometimes manageability. Having multiple DB stacks means more difficult governance, backward compatibility problems, duplication of the code, harder tests and longer roll outs.

Difficult compromises

Still while the company grows, engineering teams expand, new use cases arrive and question arises how do we address them from technology support perspective. Very often, rather than using the best available technology for the job, we decide to stretch an already adopted platform.

For example Elasticsearch was invented as a document search database and is still used as such - with great success - to drive the product catalogs for e-commerce sites as one of example use cases. Then, people realized that search is search and started dumping logs into it in order to search through them for exceptions and other errors, then aggregations came, then dashboards, metrics, APM, real user monitoring and all of the sudden we have one of the main DYI (do-it-yourself) Observability stacks built on document search technology. That is great success of Elastic stack while it is most likely a distraction of a product direction from the original company's mission. People started to look for improvements, especially in the era of market-wide cost cuts. High performance OLAP optimized DBs are just better for such analytical log queries.

On the other hand, according to ClickHouse, their input and output compression allows to save a lot of disk space, driving the cost of the hardware down. According to their tests CH used for log aggregation is around 20 more cost effective than keeping data in Elastic. Great, if you are just starting as a new Observability vendor or for whatever reason you didn't do any log management before. But how does it help you if you want to keep the rest of great Elastic stack?

Existence of such cheap and fast DB platforms has caused many new observability vendors (Betterstack, SigNoz, Qryn, Highlight - just to name a few) to bet their chips on CH as a provider of the back-end to host their data. Also many companies with DYI observability are looking to save some money by leveraging CH and other similar DB vendors. The "only" problem is that Observability/Security is not just about storing data but also (or maybe first of all) an analytics layer - equivalent of an application stack. You would need to migrate all of it and sometimes it will turn out that it is not compatible at all. New DB vendors typically don't have time to distract themselves (and they are right in this) to build the rich ecosystem of monitoring integrations and dashboarding/analytical tools like Elastic (aka ELK) stack. Moreover, designing SQL schema and query patterns to support interactive troubleshooting log sessions is challenging and often requires iterative evolution. Sometimes this turns into interesting ideas like "let's use SQL for everything", but it often means more time spent on a crucial DevOps and Security Analysts tasks and typically requires a painful migration process to be planned.

Work smarter - not harder

Migrations are obviously possible - they take time and resources, but they are manageable things. The only question is how many times we think this is going to be our last migration? You just pull this off now and have peace of mind for some time and could focus on what's important for YOUR customers and roadmap. What if even more appealing database technology appears in a few years making another migration a tempting scenario?

We want to be smarter this time… That's why we invented Quesma: an interoperability, security and connectivity layer at the front of all your databases. Not only allowing you to easier manage and secure it, but also help interconnecting different technologies together. Do you want to keep your data in cheap and efficient data storage like ClickHouse, Snowflake or even AWS Athena? That's great! But you also want to use your favorite Elastic stack Observability (Kibana, Logstash/Beats)? That's cool as well - just use our translation layer and connect these two worlds together. Companies like Uber have come to the same conclusion and implemented such approach in-house. We want to democratize this idea and give more people freedom of choice of the right tool for the right job.