I recently had a conversation about ACID, I don’t think it would surprise anyone that I’m a big proponent of ACID. After all, RavenDB was an ACID database from the first release.
When working with distributed systems, on the other hand, it is far harder to get ACID guarantees at a reasonable cost. Pretty much all the 1st generation NoSQL databases left ACID on the sidelines, because it is a hard problem. That was one of the primary reasons why RavenDB even exists. I couldn’t imagine living without transactions. This is a post from 2011, talking about just that topic.
Consistency in a distributed system is a hard problem, mostly because it has an impact on the design and performance of the system. It is also common to think about ACID as a binary property, which is sort of true (A for Atomic ). However, it turns out that the real world is a lot more nuanced than that.
I want to discuss the consistency model for RavenDB as it applies to running in a distributed cluster. It is ACID with eventual consistency, which doesn’t sound like it makes sense, right?
I found a good example to explain the importance of ACID operations from your database even in the presence of eventual consistency.
Consider the following scenario, we have a married couple with a shared bank account. Both husband and wife have a checkbook for the account and primarily use checks to pay for things in their day to day life.
Checks are anachronistic for some people, who are used to instant payments and wire transfers. The great thing about checks is that they are (by definition) a way to work in a distributed system. You hand someone a check and at some future point in time they will deposit that and get the money from your account.
One of the most important aspects of using checks was managing that delay. The amount of money you had in the account didn’t necessarily represent how much money you had available. If your rent check wasn’t deposited yet, you still had to consider the rest money “gone”, even if you could still see it in the bank statement.
Because of checks’ eventual consistency, a really important part of using checks was to keep track of all the outstanding checks that weren’t deposited yet. You did that by filling in the stub of the check in the checkbook whenever you wrote a check. In other words, you never gave a check before you properly filled the stub for that.
That brings us back to ACID. The act of filling the stub and writing the check is a transaction, composed of two separate actions. That action isn’t a global transaction. The husband and wife in our example do not have to coordinate with one another whenever they write a check. But they do need to ensure that no check would be handed off without a proper stub (and vice versa, if we want to be exact). If the act of writing a check and filling the stub isn’t atomic, you may have a check unexpectedly hit your account, which is… exciting (in the Chinese proverb manner).
On the other side, the entity that you handed the check to also needs a transaction. They need to fill out an invoice for the check (even though it hasn’t been deposited yet). Having a check with no invoice or an invoice with no check is… bad (as in, IRS agents having shots and high fives during an audit).
The idea is that at the local level, you have to use transactions, otherwise, you cannot be sure about the consistency of your own data. If you don’t have transactions at the persistence layer, you’ll have to build it on top of that, which is… not ideal, really hard and usually not going to work in all cases.
With local transactions, you can then start pushing consistent data out and resolve all the distributed states you have.
Going back to our husband and wife example, for the most part, they can act completely independently of one another, and they’ll reconcile their account status with each other at a later date (weekly budget meeting). At the same time, there are certain transactions (pun intended) where they won’t act independently. A great example is buying a car, that sort of amount requires that both will be consulted on the purchase. That is a high value operation, so it is worth the additional cost of distributed consistency.
With RavenDB, we have the notion of local node transactions, which are then sent out to the rest of the nodes in the cluster in the background (async replication) as well as support for cluster wide transactions, which requires the consent of a majority of the nodes in the cluster. You can choose for each scenario exactly what level of transactions and consistency you want to have, local or global.
I got a great question about using RavenDB from Serverless applications:
DocumentStore should be created as a singleton. For Serverless applications, there are no long-running instances that would be able to satisfy this condition. DocumentStore is listed as being "heavy weight", which would likely cause issues every time a new insurance is created.
RavenDB’s documentation explicitly calls out that the DocumentStore should be a singleton in your application:
We recommend that your Document Store implement the Singleton Pattern as demonstrated in the example code below. Creating more than one Document Store may be resource intensive, and one instance is sufficient for most use cases.
But the point of Serverless systems is that there is no such thing as a long running instance. As the question points out, that is likely going to cause problems, no? On the one hand we have RavenDB’s DocumentStore, which is optimized for long running processes and on the other hand we have Serverless systems, which focus on minimal invocations. Is this really a problem?
The answer is that there is no real contradiction between those two desires, because while the Serverless model is all about a single function invocation, the actual manner in which it runs means that there exists a backing process that is reused between invocations. Taking AWS Lambda as an example, you can define a function that will be invoked for SQS (Simple Queuing Service), the signature for the function will look something like this:
async Task HandleSQSEvent(SQSEvent sqsEvent, ILambdaContext context);
The Serverless infrastructure will invoke this function for messages arriving on the SQS queue. Depending on its settings, the load and defined policies, the Serverless infrastructure may invoke many parallel instances of this function.
What is important about Serverless infrastructure is that a single function instance will be reused to process multiple events. It is the Serverless infrastructure's responsibility to decide how many instances it will spawn, but it will usually not spawn a separate instance per message in the queue. It will let an instance handle the messages and spawn more as they are needed to handle the ingest load. I’m using SQS as an example here, but the same applies for handling HTTP requests, S3 events, etc.
Note that this is relevant for AWS Lambda, Azure Functions, GCP Cloud Functions, etc. A single instance is reused across multiple invocations. This ensure far more efficient processing (you avoid startup costs) and can make use of caching patterns and common optimizations.
When it comes to RavenDB usage, the same thing applies. We need to make sure that we won’t be creating a separate DocumentStore for each invocation, but once per instance. Here is a simplified example of how you can do this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
using Amazon.Lambda.Core;
using Amazon.Lambda.RuntimeSupport;
using Amazon.Lambda.Serialization.SystemTextJson;
using Amazon.Lambda.SQSEvents;
using Raven.Client.Documents;
using System.Security.Cryptography.X509Certificates;
using var documentStore = new documentStore
{
Urls = new string[] { /* urls */},
Certificate = new X509Certificate2(/*path*/)
};
documentStore.Initialize();
var handler = async (SQSEvent sqsEvent, ILambdaContext context) =>
{
using var session = documentStore.OpenAsyncSession();
foreach (var record in sqsEvent.Records)
{
// use the session to process the records
}
}
await LambdaBootstrapBuilder.Create(handler, new DefaultLambdaJsonSerializer())
.Build()
.RunAsync();
view raw
lambda.cs
hosted with ❤ by GitHub
We define the DocumentStore when we initialize the instance, then we reuse the same DocumentStore for each invocation of the lambda (the handler code).
We can now satisfy both RavenDB’s desire to use a singleton DocumentStore for best performance and the Serverless programming model that abstracts how we actually run the code, without really needing to think about it.
I was interviewed recently by Michael Shpilt about RavenDB and building a database in C#.I think it went very well, you can read / listen to that in the following link. As always, I would love your feedback.
By default, classes are not sealed. This means that you can inherit from them. I think this is not the right default. Indeed, unless a class is designed to be inherited from, it should be sealed. You can still remove the sealed modifier later if there is a need. In addition to not be the best defau
We are looking to hire another Developer Advocate for RavenDB. The position involves talking to customers and users, help them build and design RavenDB based solutions. It requires excellent written and oral communications with good presentation skills in English, good familiarity with software architecture, DevOps and of course, RavenDB itself.Responsibilities:Developing and growing long-term relationships with technology leaders within prospect organizationsUnderstanding the customer’s requirements and assessing the product fitMaking technical presentations and demonstrating how a product meets clients needsInvolvement in the completion of technical questions on RFPsAttending conferences, meetups and trade showsIf you are interested, or know someone who is, please ping us at: jobs@ravendb.net
When you have a distributed system, one of the key issues that you have to deal with is the notion of data ownership. The problem is that it can be a pretty hard issue to explain properly, given the required background material. I recently came up with an explanation for the issue that I think is both clear and shows the problem in a way that doesn’t require a lot of prerequisites.
First, let’s consider two types of distributed systems:
A distributed system that is strongly consistent – such a system requires coordination between at least a majority of the nodes in the system to do anything meaningful. Such systems are inherently limited in their ability to scale out, since the number of nodes that you need for a consensus will become unrealistic quite quickly.
A distributed system that is eventually consistent – such a system allows individual components to perform operations on their own, which will be replicated to the other nodes in due time. Such systems are easier to scale, but there is no true global state here.
A strongly consistent system with ten nodes requires each operation to reach at least 6 members before it can proceed. With 100 nodes, you’ll need 51 members to act, etc. There is a limit to how many nodes you can add to such a system before it is untenable. The advantage here, of course, is that you have a globally consistent state. An eventually consistent system has no such limit and can grow without bound. The downside is that it is possible for different parts of the system to make decisions that each make sense individually, but are not taken together. The classic example is the notion of a unique username, a new username that is added in two distinct portions of the system can be stored, but we’ll later find out that we have a duplicate.
A strongly consistent system will prevent that, of course, but has its limititations. A common way to handle that is to split the strongly consistent system in some manner. For example, we may have 100 servers, but we split it into 20 groups, of 5 servers each. Now each username belongs to one of those groups. We can now have our cake and eat it too, we have 100 servers in the system, but we can make strongly consistent operations with a majority of 3 nodes out of 5 for each username. That is great, unless you need to do a strongly consistent operation on two usernames that belong in different groups.
I mentioned that distributed system can be tough, right? And that you may need some background to understand how to solve that.
Instead of trying to treat all the data in the same manner, we can define data ownership rules. Let’s consider a real world example, we have a company that has three branches, in London, New York City and Brisbane. The company needs to issue invoices to customers and it has a requirement that the invoice numbers will be consecutive numbers. I used World Clock Planner to pull the intersection of availability of those offices, which you can see below:
Given the requirement for consecutive numbers, what do we know?
Each time that we need to generate a new invoice number, each office will need to coordinate with at least another office (2 out of 3 majority). For London, that is easy, there are swaths of times where both London and New York business hours are overlapping.
For Brisbane, not so much. Maybe if someone is staying late in the New York office, but Brisbane will not be able to issue any invoices on Friday past 11 AM. I think you’ll agree that being able to issue an invoice on Friday’s noon is not an unreasonable requirement.
The problem here is that we are dealing with a requirement that we cannot fulfill. We cannot issue globally consecutive numbers for invoices with this sort of setup.
I’m using business hours for availability here, but the exact same problem occurs if we are using servers located around the world. If we have to have a consensus, then the cost of getting it will escalate significantly as the system becomes more distributed.
What can we do, then? We can change the requirement. There are two ways to do so. The first is to assign a range of numbers to each office, which they are able to allocate without needing to coordinate with anyone else. The second is to declare that the invoice numbers are local to their office and use the following scheme:
LDN-2921
NYC-1023
BNE-3483
This is making the notion of data ownership explicit. Each office owns its set of invoice numbers and can generate them completely independently. Branch offices may get an invoice from another office, but it is clear that it is not something that they can generate.
In a distributed system, defining the data ownership rules can drastically simplify the overall architecture and complexity that you have to deal with.
As a simple example, assume that I need a particular shirt from a store. The branch that I’m currently at doesn’t have the particular shirt I need. They are able to lookup inventory in other stores and direct me to them. However, they aren’t able to reserve that shirt for me.
The ownership on the shirt is in another branch, changing the data in the local database (even if it is quickly reflected in the other store) isn’t sufficient. Consider the following sequence of events:
Branch A is “reserving” the shirt for me on Branch B’s inventory
At Branch B, the shirt is being sold at the same time
What do you think will be the outcome of that? And how much time and headache do you think you’ll need to resolve this sort of distributed race condition.
On the other hand, a phone call to the other store and a request to hold the shirt until I arrive is a perfect solution to the issue, isn’t it? If we are talking in terms of data ownership, we aren’t trying to modify the other store’s inventory directly. Instead we are calling them and asking them to hold that. The data ownership is respected (and if I can’t get a hold of them, it is clear that there was no reservation).
Note that in the real world it is often easier to just ignore such race conditions since they are rare and “sorry” is usually good enough, but if we are talking about building a distributed system architecture, race conditions are something that happens yesterday, today and tomorrow, but not necessarily in that order.
Dealing with them properly can be a huge hassle, or negligible cost, depending on how you setup your system. I find that proper data ownership rules can be a huge help here.
The internet is a hostile place. Publicly accessible machines will be attacked within minutes of being connected, and any unencrypted data in transit is likely to be intercepted and modified. Every day there are successful attacks on applications and databases that are insufficiently protected, resulting in data leaks and ransom demands. In this webinar, RavenDB CEO Oren Eini will go over the threat model for RavenDB, the security aspects involved in deploying in production, and the assumptions involved in working with trusted parties and byzantine partners.
Laurent Kempé asked on DevApps, a French .NET community, about how to validate a form on the first render in ASP.NET Core Blazor. This could be useful, for instance, when you load draft data, and you want to immediately show errors. The default behavior in Blazor is to validate fields when the valu
We use cookies to analyze our website traffic and provide a better browsing experience. By
continuing to use our site, you agree to our use of cookies.
