CQRS : A quick read before implementation.

I don’t want to write another story on CQRS pattern, describing how good it is, how it can solve all your problems, making your system scalable, etc. This writing's Intention is mainly to highlight areas you should consider while implementing CQRS pattern into your system. Over the last ten years, I have worked extensively on CQRS pattern, as it is very common to most of the high volume low latency systems.

I am trying to break it down under the following headings.

• What is CQRS?
• When to use
• The common way of implementation
• Consistency issues
• Potential functional problems
• Ways to overcome them
• How to Monitor CQRS systems

Skip sections if you are familiar with the concept.

What is CQRS

CQRS stands for Command and Query Responsibility Segregation. This pattern segregates, write and read responsibilities. On easy terms, data will be written at one system, and then it will be replicated to another system; all kinds of reads will happen from the next system. Better to highlight that your writing data store and read data store can be completely different, e.g., write data store can be SQL data store, whereas read data store can be NoSQL or the other way, as per the business need. This is a quick way to visualize it.

Intentionally I am keeping this section short; if you are not familiar with CQRS, please read this article on Azure docs.

When to use

These are widespread uses of CQRS.

• To reduce contention between high volume read and write. As we separate the destination of writing and the source of reading, contention gets reduced, and the system becomes more scalable.
• If the shape and form of data are different between the way it is written and how it is read, this is a perfect way to implement it.
• To keep a physical separation between write and read applications, e.g., systems updating transaction data vs. systems using transaction data for reporting purposes.

How to Implement

There are several ways to implement CQRS in your platform. An implementation combining CQRS and Event Sourcing is commonly used.

Through Code : A decade back, CQRS pattern used to be implemented through code. The write API / Service used to take the responsibility of publishing the message in a messaging system (mostly JMS based); on the other end, there can be multiple listeners, which are picking up the message and, based on the use case changing the form and shape of the data and storing it in various data stores.

Bad Implementation

We passed that era, if you are reading this in 2021 or beyond, you should not implement the whole flow by yourself from scratch. You will end up coding a lot of scenarios to make it robust. Most of those are out of the box feature if you are using standard products.

Using a combination of Products : The data flow or replication flow can be implemented through various sets of tools.

Change Data Stream + Kafka : Most new-gen databases, be it SQL or NoSQL, comes with a change data stream option, which is nothing but a mechanism to publish all changes at the data level. If you are a DB guy, it is a little bit different than pre-historic triggers. The idea is to publish a data stream to Kafka and use Kafka as an event store. From the event store, the next targetted data store can be populated as per the need.

The use of a standard product makes the system resilient from common failure scenarios.

Cloud Data Store : Cloud Data Stores such as AWS RDS or Google Cloud SQL comes with the provision of “Read Replica”, through which you can create an exact copy of the data store in a different zone or region. This will not give you a provision to change the shape of the data, but your readers will have no contentions with your writers. This can be achieved just from the configuration, with zero development effort.

Quick Cloud Implementation

Consistency Issues

As CQRS solves most of the contention problems and makes systems scalable, it comes with many functional problems. It introduces a lag between source and destination data. A decisive read which is time-sensitive will not able to cope with this consistency issue. Let me give an example to explain it better.

Here my horizontal axis is time. If a system is making a decision based on the read data store, there are chances it will end up allowing a user to spend more than his / her buying power.

Can we solve it?

As you can see, this consistency issue can be reduced by throwing hardware and providing a premium network, but it cannot be fixed; hence a write choice of a data source is important while implementing any use case on a CQRS based platform.

• If your implementation, taking a decision based on the state of the data, it is important to read the state from the write data store. It is definitely an anti-pattern, but with no choice.
• Your write data store can expose a very light version of API, through which the version number of any data can be read. Transaction systems can compare reading and write data store version before picking up the data. This is very helpful if you are reading a huge chunk of the data, as most of the time, you’ll end up using the read data store, hence less pressure on the write data store.

Monitoring of CQRS Platform

The monitoring of the CQRS platform should be robust because things can go south very fast as it has multiple integration points.

These are areas you should have proactive monitoring...

• Monitor backlog in the messaging system. In case of slow message update to read store, there will be more lag between write and read stores.
• If messages are failing to update read stores, that should trigger the alarm. There should be a provision to replay those failed messages or update the read store correctly.
• There should be an alarm if the read store's update frequency gets dropped below a threshold value. This check can be implemented like a circuit breaker.

I hope these help. As you can see, replication speed plays a big role in CQRS implementation. I am planning to cover replication in detail in one of my next stories.