Kibana on Clickhouse data with Quesma - first working prototype

Kibana on Clickhouse data with Quesma - first working prototype

Feb 26, 2024

Kibana on Clickhouse data with Quesma

by Pawel Brzoska

Quesma has a great pleasure to unveil first working prototype of Quesma translation gateway [now in Preview] for connecting two best of breed worlds:

  • Kibana front end (including OpenSearch Dashboards) and

  • ClickHouse database

We've already invested time in explaining why this combination makes sense both technically and economically. Now, it's time to translate words into action.

Quesma is provided as a lightweight component, deployable as a docker container, also in Kubernetes, that accepts connections from Kibana and/or Beats/Logstash on one end and communicates with Elasticsearch and/or ClickHouse on another end.

It can work in dual-write/dual-read mode where you can compare performance and integrity of returned data and decide how much of your bulk load you want to move to alternative storage (here: ClickHouse) for better ROI and performance. Quesma takes care of the rest - schema, translation and query compatibility. Here's the result:

You can easily navigate your logs in Kibana Log Explorer, change the time window, check and navigate the log histogram picking most interesting time to inspect. All required filtering is provided - both free text filters as well as field filters from drop-down lists. You can also pick any field and display/hide as existing column.

It is our goal at Quesma to give every developer, DevOps, SecOps and their teams a freedom of choice of the best tool for the job. We've heard from many people that they love Kibana and would like to continue using it, but they express discomfort with its tight coupling to the Elastic ecosystem, which, by design, is intended to be open and flexible. At the other hand OLAP use cases are much better to be realized in databases like ClickHouse - therefore the needed connection piece that we have a pleasure to present above.

If you have any questions or would like to try it - let us know.