by Antoni Olendzki and Przemysław Hejman

If you’ve ever tried to pin down the cost of using the Elastic Stack, you’ve likely found it frustratingly elusive. Many users have voiced their concerns about Elastic's lack of transparent, predictable pricing, with some even turning to community forums to voice their concerns. Whether you’re a small business or an enterprise looking to leverage the Elastic Stack (aka ELK), understanding the total cost of ownership is crucial for budgeting and decision-making. This post aims to clarify Elasticsearch's pricing, address common pain points, and uncover the hidden costs that could catch you by surprise.

The Challenge of Elasticsearch Pricing

Whether considering Elastic Cloud or a self-hosted setup, costs can quickly become complex and unpredictable, especially as your data needs grow. As many organizations have discovered, deploying the “free” Elastic Stack into a production environment can lead to significant unplanned expenses. These can include hardware, infrastructure, consulting, security, training, and ongoing maintenance costs.

Managed Service vs. Self-Hosted: Understanding the Trade-offs

Choosing between using a managed service like Elastic Cloud and self-hosting your Elastic Stack is one of the most critical decisions you’ll make, with significant implications for both cost and complexity.

1. Managed Service (Elastic Cloud):

• Simplicity and convenience: Elastic Cloud offers a fully managed service where Elastic handles the infrastructure, scaling, and maintenance. This is ideal for organizations that want to offload the operational burden and focus on building their solutions on top of the stack.

• Predictable (to some extent) costs: Elastic Cloud pricing is based on resource consumption, data ingestion, and the specific features you use. However, Data Transfer Service (DTS) charges, API calls, and snapshotting of large clusters in production environments can lead to unexpected costs. Additionally, ongoing maintenance, updates, and the need for advanced features can further complicate cost estimation. While the pricing structure is more straightforward than self-hosting, surprises can still arise, especially as data volumes grow or advanced features are needed. Elastic’s pricing calculator can help, but estimating costs fully requires careful consideration.

• Support and features: As a managed service user, you get access to support services provided by the same people who build the whole product, which can be beneficial. There are currently four subscription levels to choose from. Higher-tier subscriptions include advanced features like machine learning or advanced security, which can be vital for enterprise use cases.

2.Self-Hosted Elastic Stack:

• Flexibility with trade-offs: Hosting your Elastic Stack gives you complete control over the infrastructure, leading to lower costs if managed effectively. However, as users on Reddit have pointed out, the hidden costs of self-hosting—such as infrastructure, maintenance, and staffing—can quickly add up. Beyond the initial setup, ongoing costs for maintaining a production-ready environment can escalate, particularly as you scale. Finding the people with the right expertise can be especially difficult. 

• Infrastructure and operational costs: While you might save on managed service fees, you’ll need to account for the price of servers, storage, and network infrastructure. Additionally, stack maintenance requires skilled personnel, which can add significant operational overhead.

• Less predictable scaling costs: As your data grows, so does the complexity of scaling your infrastructure. Managing multi-tier clusters with dedicated ingest/master nodes while ensuring suitable capacity becomes more difficult. Given that native autoscaling is unavailable (unless we’re considering ECK), capacity planning becomes a real challenge.    

It’s worth mentioning that there are additional “hybrid” options if you still prefer to bring your hardware while alleviating the pain of cluster management. Elastic Cloud Enterprise (ECE) is essentially Elasticsearch Cloud installed at your premises. A dedicated control plane will handle the deployment creation and management. It’s a paid, non-open-source service, allowing you to create as many clusters as your capacity permits.

For those who already adopted Kubernetes in their organizations, there’s ECK - an operator supplied by Elastic. This also takes some cluster management tasks off your shoulders, but the learning curve is steeper than ECE. However, if you and your team are already comfortable with k8s, it’s an option worth exploring. The default distribution of ECK comes with a basic license, so features here are also limited (e.g., there’s no autoscaling support). 

Real-World Pricing Examples and Scenarios

Let’s explore some hypothetical scenarios to provide a clearer picture of what you might expect regarding costs. The prices can deviate depending on your chosen underlying cloud provider - currently, Elastic Cloud supports AWS, Azure, and GCP.

• A minimal Elasticsearch deployment with Kibana

  Small cluster, consisting of a tiny (1GB RAM) cluster, will cost around $25-30 monthly. This is a single-instance cluster, so there is no HA. It is enough to start with API, load sample data, and create dashboards. DTS charges are negligible in this scenario.
  It’s worth mentioning that a 14-day free trial is offered for such cases, but it is limited to only one cluster per trial account.

  
  

• Small Elasticsearch deployment with HA, Kibana included.

  A 2-node, 8GB RAM Elasticsearch deployment with Kibana will cost around $500 monthly. This assumes a Standard subscription; however, the cost can grow 50% or even double if you move to higher tiers. Such a deployment will be perfect for small, production-facing apps, using ES as a full-text search engine backing up a website.

  
  

• Elasticsearch deployments with larger storage

  Suppose we were to scale the deployment, specifically for storage. Our dataset grows and instead of ~0.5TB of data, we’d need about 1.5TB of storage. Of course, compute power will have to be adjusted accordingly, as it’s improbable that a 2x8GB RAM cluster can handle it.
  Depending on the specific setup, 2x30GB RAM or 3x16GB cluster will increase our monthly bill to ~$2000/month, but we are still considering a Standard subscription.
  The good news is that we can leverage Data Tiers. Instead of scaling up the cluster horizontally, as mentioned earlier, we just add “warm” nodes to hold the data. In this case, our monthly cost will be ~$800/month (plus, we’ll even get more storage).

  
  

• Larger scale deployments

  For clusters larger than 6 data nodes, Elastic Cloud will require to add dedicated master nodes. At a certain scale, separating the master from data nodes is considered a best practice, so you might also want to do this on a self-hosted setup. Add separate coordinating/ingest nodes to your cluster to balance the workload. Elastic Cloud also offers this option. Each node, however, comes with a cost, so now we’re talking about $2k, $5k, $7k per month clusters. We could safely assume that the DTS charges can substantially affect the total bill at this scale. 

The calculations above only discuss a single cluster price. In your organization, however, you might want to consider having more clusters—for testing/pre-production environments or because you need to keep specific datasets on separate clusters.

Self-hosting costs are even more challenging to assess. Considering Cloud - while one can just visit providers' pricing, like the AWS EC2 pricing page, the hardware cost is merely the tip of the iceberg. Efforts to set up Elasticsearch cluster properly with snapshotting, fault-tolerance, and ensuring everything is securely exposed to the outside world barely scratch the surface when providing enterprise-grade Elasticsearch service to your organization. As mentioned, there are options to help with that, but they are either paid (ECE) or require k8s experience in the organization (ECK).

Alternatives to Elasticsearch: Is There a Better Option?

If you’re looking for hosted Elasticsearch alternatives, managed OpenSearch on AWS could be an option. It doesn’t have all the premium features Elastic offers, but depending on your use case, it might meet your needs. AWS publishes the pricing calculator for the hosted OpenSearch service, which you can check out here - but similarly to Elastic, it does not consider DTS costs. Another option, especially if your use case is around analytical (OLAP) queries like Security or Observability, is to completely shift to a different database engine and use columnar databases like ClickHouse or Hydrolix, ready to perform analytical queries on a petabyte-scale at a fraction of Elasticsearch/OpenSearch cost. The biggest challenge here, of course, is that the interface for data access is no longer HTTP and Query DSL but good old SQL (with few exceptions). The compatibility with existing applications - e.g., Kibana with its dashboards - breaks down. While it seems to be a deal breaker, it doesn’t have to be when Quesma comes to the rescue!

How Quesma Can Help You Optimize Your Elasticsearch Costs

Imagine an example infrastructure for a simple e-commerce website with Elasticsearch used for full-text search and analytical data. By putting Quesma between the application and database layer, you could seamlessly offload your analytical data from Elastic to ClickHouse (and enjoy up to 23x in cost savings), while being able to use your beloved Kibana analytical dashboards (more resources on how ClickHouse can be a game changer here). See diagram below:

Reach out here to get started.

Tips for Optimizing Costs and Performance

We wrote a whole blog post on this — check it out here :)

Conclusion

Understanding the costs of hosting/managing the Elasticsearch can be challenging, but it’s essential for making informed decisions. Whether you opt for Elastic Cloud, a self-managed stack, or even consider alternatives, knowing where the costs come from can help you better plan your budget and avoid surprises.

At Quesma, we’re dedicated to helping you navigate these complexities easily. If you’d like to discuss your options and what our gateway enables you to do, please reach out via our contact form.