Nov 24, 2024
Quesma database gateway
This is the third and last of the micro-blog series about what we do and why at Quesma. If you haven't - I highly recommend reading the first and second parts first.
In previous blogs we’ve discussed how hard it is to innovate around the database layer of application stack. How immensely risky it is to introduce even the smallest change and how impactful any potential failures in this area can become to your business, customers, brand reputation and bottom-line.
DB monolith hypothesis
We have also come up with a hypothesis of the reason for such a situation. In our opinion that was confirmed by many industry insiders we had a pleasure to talk to, there’s a lot of similarities around the symptoms of the problem in how difficult it is to improve databases and how generally difficult it was some time ago to innovate, build and release monolithic applications. Let’s quickly summarize them:
Long release cycles
Lots of dependencies
Lack of common communication standard
Any change in one area (DB layer in our case) affects many other components
As we all well know the above problems have been addressed many years ago by splitting (breaking down) the classical application monolith code-base to a smaller set of interconnected, independent entities - micro-services. That helped to achieve the following outcomes:
Ability to pick the right technology (e.g. programming language) in one point of the system without affecting others
Independent, often very fast, but still safe release cycles, enabled by:
Ability to make quick, safe, iterative changes without affecting other components (microservices)
Micro-services, API and gateways
All of the achievements mentioned above have become possible due to the adoption of a standardized approach for integrating microservices (that at the end of the day are parts of a single ecosystem - the application - that needs to realize its business objectives). Instead of tight in-code couplings and calls, we are nowadays using a set of well defined APIs. As long as the API format stays the same and is backward compatible between releases/versions of the microservice being developed, the others are safe - everything will work. Easy to experiment, easy to test, even if something fails - the roll-back is also very easy.
These APIs created a substance between microservices that similarly how the mortar sticks the bricks together in the building, plays a role of interoperability and backward-compatibility layer.
In addition to the above outcomes, this intermediate layer also introduced other interesting opportunities to make our applications even better. They are quickly becoming essential parts of modern application architecture best practices and are often developed and supported by a variety of vendors in the space. For example:
Performance load-balancing
Safe-release blue-green deployment technique
Authentication and authorization control
Data leak, fraud and bad actor detection and prevention
Observability and performance diagnostics
Very often these use cases are integrated together into a dedicated microservice that plays the role of API gateway, especially for connecting from/to outside world or used in a distributed way, as sidecars for example, forming a concept of a service-mesh.
Database gateways
What if we could apply the same paradigms to our databases? Let’s imagine we have a backward-compatibility and interoperability layer that works similarly to APIs between other microservices. What if we could just treat a DB as another microservice connected to such a layer?
Obviously that’s not so easy - there’s no standard protocol like http to access databases. For those of you who work with SQL, it’s well-known that despite the existence of a standard for this language, practical implementations (dialects) can vary significantly. Not mentioning noSQL DBs and other, even more difficult, use cases. An equivalent of an API gateway for DBs would need to have some sort of translation functionality to allow for DB interoperability. As a matter of fact many DB vendors are adding similar concepts at the top of their products - to help with migration and provide backward compatibility with their typical target customer base incumbent platforms.
But once we build this, we would have an architectural component (think gateway) that we could place at the front of both our old and new databases. The process of migrating your workloads to the new database or changing the data structures within a single database would then appear more straightforward. Do you agree?
As a matter of fact, many IT organizations (typically of bigger size), implemented similar component as an ad-hoc lever helping them to pull off a particularly hard migration project to a new DB technology. Certainly many of them have removed this component from the critical path “till next time”, as there were no appetite nor funds to develop and invest in it any further, however let’s imagine that such component actually stays at the front of the new database and it can interconnect client side of the application stack with our databases. We are closer and closer to having what we have discussed above: a layer covering the role of APIs in the microservices world. This component, in addition to providing translation logic, could also serve use cases similar to ones we listed above for APIs:
Performance load-balancing
Safe-release blue-green deployment technique
Authentication and authorization control
Data leak, fraud and bad actor detection and prevention
Observability and performance diagnostics
And (last but not least) a DB specific use case: query performance optimization and safe-guarding
Our mission
We, at Quesma, believe that our industry needs to start thinking differently about how we all architect our applications and how we connect them to their source of data. We are committed to help the industry and lead an open community of developers to build an equivalent of API gateway for databases.
That’s the mission and goal we have set for ourselves for years to come. And as the saying goes, every journey starts with the first step, so now it is the time for it!
