How to Partition Your Data Between Microservices

By Christian Horsdal

This article was excerpted from the book Microservices in .NET.

When deciding where to store data in a microservice system, there are some competing forces at play. The two main forces are data ownership and locality:

• Owning data means being responsible for keeping it up to date.
• Locality means that the data a microservice needs often should be “nearby”—preferably within the microservice itself.

These two forces may be at odds, and in order to satisfy both you will often have to store data in several places. That is OK, but it is important that only one of those places be considered the authoritative source. Figure 1 illustrates that while one microservice stores the authoritative copy of a piece of data, other microservices can mirror that data in their own data stores if they want.

Figure 1: The top and bottom microservices collaborate with the middle microservice. The top and bottom microservices can store mirrors of the data owned by the middle microservice, but the authoritative copy is stored in the middle microservice’s own data store.

Rule 1: Data Ownership Follows Business Capabilities

The first rule when deciding where a piece of data belongs in a microservices system is that data ownership follows business capabilities. The primary driver in deciding on the responsibility of a microservice is that it should handle a business capability. The business capability defines the boundaries of the microservice—everything belonging to the capability should be implemented in the microservice. This includes storing the data that falls under the business capability.

Domain Driven Design teaches us that some concepts can appear in several business capabilities and that the meaning of the concepts might very well differ slightly. Several microservices may have the concept of a customer, and they will work on and store customer entities. There may be some overlap between the data stored in the different microservices, but it’s important to be clear about which microservice is in charge of what.

For instance, only one microservice should own the home address of a customer. Another microservice could own the customer’s purchase history, and a third the customer’s notification preferences. The way to decide which microservice is responsible for a given piece of data—the customer’s home address, for instance—is to figure out which business process keeps that data up to date. The microservice responsible for the business capability is responsible for storing the data and keeping it up to date.

Figure 2 shows an overview of how a commerce system handles user requests for adding an item to a shopping cart. Most of the microservices in figure 2 are grayed out, to put the focus on three microservices: the Shopping Cart Microservice, the Product Catalog Microservice, and the Recommendations Microservice.

Figure 2: In this ecommerce example, we’ll focus on the partitioning of data between the Shopping Cart Microservice, the Product Catalog Microservice, and the Recommendations Microservice.

The highlighted microservices in figure 2 each handle a business capability: the Shopping Cart Microservice is responsible for keeping track of users’ shopping carts, the Product Catalog Microservice is responsible for the giving the rest of the system access to information from the product catalog, and the Recommendation Microservice is responsible for calculating and giving product recommendations to users of the ecommerce site. There is data associated with each of these business capabilities, and each microservice owns and is responsible for the data associated with its capability.

Figure 3 shows the data each of the three microservices owns.

Figure 3: Each microservice owns the data belonging to the business capability it implements

That a microservice owns a piece of data means that it must store that data and that it is the source of truth for that piece of data.

Rule 2: Replicate for Speed and Robustness

The second force at play when deciding where a piece of data should be stored in a microservices system is locality. There’s a big difference between a microservice querying its own database for data and a microservice querying another microservice for that same data. Querying its own database is likely both faster and more reliable than querying another microservice.

Once you have decided on the ownership of data, you will likely discover that your microservices will often need to query data from each other. This type of collaboration creates a certain coupling: one microservice querying another means that the first is coupled to the other. If the second microservice is down or slow, the first microservice will suffer.

To loosen this coupling, you can cache query responses. Sometimes you’ll cache the responses as they are, but other times you can store a read model based on query responses. In both cases, you must decide when and how a cached piece of data becomes invalid. The microservice owning the data is in the best position to decide when a piece of data is still valid and when it has become invalid. Therefore, endpoints responding to queries about data owned by the microservice should include cache headers in the response telling the caller how long it should cache the response data.

USE HTTP CACHE HEADERS TO CONTROL CACHING

HTTP defines a number of headers that can be used to control how HTTP responses can be cached. The purpose of the HTTP caching mechanisms is twofold:

• To eliminate the need, in many cases, for requesting information the caller already has
• To eliminate the need, in many other situations, to send full HTTP responses

In order to eliminate the need to make requests for information the caller already has, the server can add a cache-control header to responses. The HTTP specification defines a range of controls that can be set in the cache-control header. The most common are probably the private|public and the max-age directives. The first indicates whether only the caller—private—may cache the response or if intermediaries—proxy servers for instance—may cache the response too. The max-age directive indicates the number of seconds the response may be cached. For example, the following cache-control header indicates that the caller, and only the caller, can cache the response for 3600 seconds:

cache-control: private, max-age:3600

That is, the caller, may reuse the response any time it wants to make an HTTP request to the same URL with the same method—GET POST PUT DELETE, , ,—and the same body within 3600 seconds. It is worth noting that the query string is part of the URL, so caching takes query strings into account.

In order to eliminate the need for sending a full response in cases where the caller has a cached but stale response, the server can add an etag header to responses. This is an identifier for the response. When the caller makes a later request to the same URL using the same method and the same body, it can include the etag in a request header. The server can read the etag, and through it know which response the caller already has cached. If the server decides that the response is still valid, it can return a response with the 304 Not Modified status code to tell the client to use the already cached response. Furthermore, the server can include add a cache-control header to the 304 response to prolong the period the response may be cached. Note that the etag is set by the server and later read again by the same server.

Let’s consider the microservices in figure 3 again. The Shopping Cart Microservice uses product information that it gets by querying the Product Catalog Microservice. How long the product catalog information for any given product is likely to be correct is best decided by the Product Catalog Microservice, which owns the product catalog information. Therefore, the Product Catalog Microservice should add cache headers to its responses, and the Shopping Cart Microservice should use them to decide how long it can cache a response. Figure 4 shows a sequence of requests to the Product Catalog Microservice that the Shopping Cart Microservice wants to make.

Figure 4: The Product Catalog Microservice can allow its collaborators to cache responses by including cache headers in its HTTP responses. In this example, it sets the max-age to indicate how long responses may be cached, and it also includes an etag built from the product IDs and versions.

In figure 4, the cache headers on the response to the first request tell the Shopping Cart Microservice that it can cache the response for 3600 seconds. The second time the Shopping Cart Microservice wants to make the same request, the cached response is reused because less than 3600 seconds have passed since it was received. The third time, the request to the Product Catalog Microservice is made because more than 3600 seconds have passed. That request includes the etag from the first response. The Product Catalog Microservice uses the etag to decide that the response would still be the same, so it sends back the shorter 304 Not Modified response instead of a full response. The 304 response includes a new set of cache headers that allows the Shopping Cart Microservice to cache the already cached response for an additional 1800 seconds.

USING READ MODELS TO MIRROR DATA OWNED BY OTHER MICROSERVICES

It’s normal for a microservice to query its own database for data it owns, but querying its database for data it doesn’t own might not seem as natural. The natural way to get data owned by another microservice would be to query that microservice. However, it’s often possible to replace a query to another microservice with a query to the microservice’s own database by creating a read model. A read model is a data model that can be queried easily and efficiently. This is in contrast with the model used to store the data owned by the microservice, where the purpose is to store an authoritative copy of the data and to be easily update it when necessary.

Data is, of course, also written to read models—otherwise they would be empty—but the data is written as a consequence of changes somewhere else. You trade some additional complexity at write time for less complexity at read time.

Read models are often based on events from other microservices. One microservice subscribes to events from another microservice and updates its own model of the event data as events arrive.

Read models can also be built from responses to queries to other microservices. In this case, the lifetime of the data in the read model is decided by the cache headers on those responses, just as in a straight cache of the responses. The difference between a straight cache and a read model is that to build a read model, the data in the responses is transformed and possibly enriched to make later reads easy and efficient. This means the shape of the data is determined by the scenarios in which it will be read instead of the scenario is which it is written.

Let’s consider an example. The Shopping Cart Microservice publishes events every time an item is added to or removed from a shopping cart. Figure 5 shows a Shopper Tracking Microservice that subscribes to those events and updates a read model based on the events. The Shopper Tracking Microservice allows business users to query how many times specific items are added to or removed from shopping carts.

Figure 5: The Shopper Tracker Microservice subscribes to events from the Shopping Cart Microservice and keeps track of how many times products are added to and removed from shopping carts.

The events published from the Shopping Cart Microservice are not in themselves an efficient model to query when you want to find out how often a product has been added to or removed from shopping carts. The events are, however, a good source to build such a model from. The Shopper Tracking Microservice keeps two counters for every product: one for how many times the product has been added to a shopping cart, and one for how many times it has been removed. Every time an event is received from the Shopping Cart Microservice, one of the counters is updated, and every time a query is made about a product, the two counters for that product are read.

Where Does a Microservice Store its Data?

A microservice can use one database, two, or more. Some of the data stored by the microservice might fit well into one type of database, and other data might fit better into another type. There are many viable database technologies available today, and I won’t get into a comparison here. There are, however, some broad database categories that you can consider when you are making a choice, including relational databases, key/value stores, document databases, column stores, and graph databases.

The choice of database technology, or technologies, for a microservice can be influenced by many factors, including these:

• What is the shape of your data? Does it fit well into a relational model, a document model, a key/value store, or is it a graph?
• What are the write scenarios? How much data is written, and do the writes come in bursts or are they evenly distributed over time?
• What are the read scenarios? How much data is read at a time? How much is read all-in-all? Do the reads come in bursts or not?
• How much is written compared to how much is read?
• Which databases do the team already know how to develop against and run in production?

Asking yourself these questions—and finding the answers—will not only help you decide on a suitable database but will also likely deepen your understanding of the non-functional qualities expected from the microservice. That is, you’ll deepen your understanding of how reliable the microservice must be, how much load it must handle, what the load looks like, how much latency is acceptable, and so on.

Gaining that deeper understanding is valuable, but note that I’m not recommending that you undertake a big analysis of the pros and cons of different databases each time you spin up a new microservice. You should be able to get a new microservice going and deployed to production quickly. The goal is not to find a database technology that is perfect for the job—just to find one that is suitable given your answers to the previous questions. You may be faced with a situation where a document base seems like a good choice, and where you are confident that both CouchBASE and MongoDB would be well suited. In that case, just choose one of them. It’s better to get the microservice to production with one of them quickly, and then at a later stage possibly replace the microservice with an implementation that uses the other one, than it is to delay getting the first version of the microservice to production because you’re analyzing CouchBASE and MongoDB in detail.

This article was excerpted from the book Microservices in .NET. For source code and other resources, go to https://www.manning.com/books/microservices-in-net