by Antoni Olendzki

As organizations grow, managing and analyzing massive amounts of data becomes increasingly critical. Kibana, as a part of the Elastic stack (or equivalently, OpenSearch Dashboards for OpenSearch), is a favorite for many teams due to its powerful data visualization and exploration capabilities. However, the rising costs associated with Elasticsearch storage, driven by ever-growing amounts of data and increasing user and regulatory/compliance demands, can become a significant burden. This is where Quesma steps in, offering a seamless transition to a more cost-effective data solution without sacrificing the familiar Kibana interface.

Understanding the Cost Challenge

Elasticsearch can become prohibitively expensive as data volumes grow, especially with regulatory demands for detailed and prolonged data storage. While a data tiering feature is available to alleviate the pain of storage cost, it comes with maintenance overhead and limitations to data querying. This has led many DevOps, SecOps, and SRE teams to seek more affordable alternatives. However, they often face resistance from teams that rely heavily on Kibana for daily operations.

The Dilemma: Staying with Elasticsearch for Kibana

Kibana is integral to the Elastic stack, known for its ease of use and powerful visualization capabilities. Many organizations hesitate to move away from Elasticsearch because they want to maintain Kibana's benefits. This creates a dilemma: continue bearing the high costs of Elasticsearch or switch to a different stack and risk disrupting workflows. The fact that Kibana can only visualize data from Elasticsearch and OpenSearch Dashboards can visualize data only from Opensearch has persisted until Quesma pulled up.

Enter Quesma: The Best of Both Worlds

Quesma offers a database gateway that enables organizations to retain their beloved Kibana interface while switching the underlying data source to more cost and performance-effective columnar SQL databases like ClickHouse or Hydrolix.

Here’s how it works:

1. Seamless Integration: Quesma provides a lightweight component, deployable as a Docker container, that connects Kibana to SQL database (e.g. ClickHouse or Hydrolix). This smooth integration ensures that users can continue using Kibana without any noticeable changes in their workflow.

2. Dual-Write/Dual-Read Mode: Quesma supports a dual-write/dual-read mode, empowering you to compare data performance and integrity between Elasticsearch and ClickHouse. This feature enables a gradual transition, giving you full control over how much data to migrate based on your needs and enhancing your decision-making capabilities. 

3. Cost Efficiency: ClickHouse is known for its high performance and cost efficiency, particularly for OLAP workloads. Moving your data storage to ClickHouse can significantly reduce storage and scaling costs while maintaining high query performance.

4. Schema Translation and Query Compatibility: Quesma handles the schema translation and ensures query compatibility between Elasticsearch and ClickHouse SQL. This means you don’t have to worry about rewriting queries or changing your data schema.

Why Choose Quesma?

1. Cost Reduction: Offloading data storage to ClickHouse can drastically reduce the expenses associated with Elasticsearch. ClickHouse's columnar storage and efficient data compression make it an ideal choice for large-scale data analytics.

2. Performance: Elasticsearch is an outstanding full-text search database. However, it can fall short with OLAP scenarios run on bulk data loads. ClickHouse offers superior performance for analytical queries, often outperforming Elasticsearch in query speed and efficiency.

3. Flexibility: Quesma’s solution allows for data storage and retrieval flexibility, enabling you to choose the best tools for your specific needs without being locked into one ecosystem.

Generating Interest and Early Adoption

As an early-stage startup, Quesma is eager to engage with forward-thinking organizations that are willing to explore innovative solutions to reduce Elasticsearch costs while retaining Kibana's benefits. By leveraging our database gateway, you can be at the forefront of a cost-saving revolution in data management.

Get Involved with Quesma

We are about to wrap up our early adopters list to collaborate with us and refine our solution. By partnering with Quesma, you'll have the opportunity to influence the development of a tool designed to meet your specific needs and ensure it delivers maximum value to your organization.

Conclusion

For organizations grappling with the high costs of Elasticsearch/OpenSearch but reluctant to give up Kibana/OSDashboards, Quesma offers a compelling solution. By enabling Kibana/OSDashboards to work with SQL columnar databases like ClickHouse or Hydrolix, Quesma provides a cost-effective, high-performance alternative that allows you to continue using your favorite data visualization tools without compromise.

Call to Action: Ready to reduce your Elasticsearch costs without losing Kibana? Contact us to learn more about how Quesma can help.