RavenDB offers both single node transactions as well as cluster wide transactions. You are free to use either one or even mix them together. That level of freedom, on the other hand, brings with it its own set of challenges. How do you know what to use? What are the scenarios and implications for each operation?Remember, RavenDB is a distributed database that can allow you to make modification on any node in the cluster.
In essence, this boil down to a simple concept, how important is the write that you are making. In detail, this gets complex. It’s easy to say that if for low importance writes, you’ll use single node transactions, and for high value items, you’ll use cluster wide transactions. But that isn’t correct. The primary issue is what you are trying to achieve. I’m afraid that I have no choice but to dig into this topic.
Let’s consider the following scenario: A user clicked on “Add to Cart” in the application. How should we record this fact? There is a “shopping-carts/ayende” document for this user, which represent their current shopping cart. But how should we save it?
Obviously, we never want to lose an item from the shopping cart, right? We can use a cluster wide transaction here to ensure maximum safety! Except… a cluster wide transaction will fail if the node that we reached cannot access the majority of the nodes in the cluster. Going back to the business, I asked them about it. The answer I got was “never lose an item from the shopping cart”. That means that we need to process the write even if we can reach no other node.
That leads us to single node transactions, which will do just that. However, now we have to deal with another issue, what happened if two concurrent transactions modified the same document on different nodes at the same time? Now we have a conflict, and when the nodes will replicate the data to one another, we’ll need to resolve it somehow. RavenDB will default to resolving to latest, meaning that some of the changes will be lost. However, we can setup a resolution script that can merge our changes between multiple versions of a document.
This is confusing, I’m aware. The rule of thumb goes like this:
Use single node transactions by default – if there are errors / conflicts / issues, let RavenDB resolve them to the latest version (a revision exists so you can recover anything lost).
Use single node transactions + conflict resolver script if you actually care about applying any sort of logic to the merging of conflicts. This is rare, the scenario is usually when we have something that can be modified and merged together. Shopping cart is an excellent example of this.
Use a cluster wide transaction when you would rather fail than go forward if you cannot ensure the operation is successful. This is also rare, usually reserved for things such as selling limited amount of some item.
The default recommendation, let RavenDB manage that and accept that it may select the latest version is not something that I make lightly. It is based on quite a bit of experience in how users are actually using RavenDB. In almost any business context, you are going to have large parts of the model that have only a single reason to change, even in the worst case assumption. A customer changing its billing address, for example, can be reasonably assumed to want to keep the latest version they put in. There is also no real meaning to concurrency in this scenario, the modification to a particular document is done by the relevant customer directly. Failures are rare (but they do happen, so you have to account for them), so you need to consider what the impact you’ll have. If this is something that doesn’t have multiple concurrent operations going on it normally (and proper document modeling will suggest that this isn’t the case), you can just ignore the problem.
I’m saying ignore the problem because there is the question on what is the meaning of not ignoring the issue? You can try to write your conflict resolution script, but even with knowledge of your model, what are you expected to do with two conflicting versions of a customer, with different billing addresses in each?
And trying to do something generic doesn’t work. It will fail, but because this is rare, it will happen only a year after deployment, when no one recalls what exactly the behavior is and an error on such a case will cause hard failure in production.
For some cases, like the shopping cart, you can meaningful write merge code, and the scenario make sense, I may click on two “Add to Cart” buttons at the same time from different locations and I don’t want to lose any of that.
The last scenario, using cluster wide transaction, is actually the reserve. Usually RavenDB will jump through all sorts of hoops to ensure that it won’t lose a write, but cluster wide transactions are actually going the other way. They need to fail if they can’t ensure that they went through. In this case, you’ll usually be working on something very specific. The classic example is ensuring a unique user name in the system, we want to fail if we can’t absolutely ensure that this username is unique. But that isn’t something that we want to do all the time, updating the LastLogin time on the user’s document is not something that you need to ensure will be consistent (and in this case, selecting the latest is also by definition the right thing to do).
I like to say that you should use a single node transaction to record that you purchased a lottery ticket, and a cluster wide transaction to record who won the lottery. That gives the right mindset about the stakes involved. I never want to lose the record of a sale, but I want to ensure that once the win is awarded, I get it absolutely right.
Working in professional software development teams requires a lot of writing. As you advance in your career, frequently the percentage of…Keep Reading →
Assume that you have a service that needs to accept some data from a user. Let’s say that the scenario in question is that the user wants to upload a photo that you’ll later process / aggregate / do stuff with. How would you approach such a system? One way to do this is to do something similar to this:The user will upload the function to your code (in the case above, a Lambda function, but can be an EC2 instance, etc) which will then push the data to its final location (S3, in this case). This is simple, and quite obvious to do. It is also wrong.There is no need to involve your code in the middle. What you should do, instead, is to have the user talk directly to the end location (S3, Azure Blob Storage, Backblaze, etc). A better option would be:In this model, we have:User ping your code to generate a secured upload link. You can also setup an “upload only area” in storage that a user can upload files to ahead of time, removing this step.User upload directly to S3 (or equivalent).S3 will then ping your code when the upload is done.Why use this approach rather than the first one?Quite simply, because in the first example, you are paying for the processing / upload / bandwidth for the work. In the second option, it is on the cloud / storage provider to actually provision enough resources to handle this. You are paying for the storage, but nothing else.Consider the case of a user that uploads a 5 MB image over 5 seconds, if you are using the first option, you’ll pay for the full 5 seconds of compute time if you are using something like Lambda. If you are using EC2, your machine is busy and consume resources. This is most noticeable if you also have to handle spikes / load. If you have 100 concurrent users, the first option will likely cost quite a lot just in the compute resources you use (either server less or provisioned machines). In the second option, it is the cloud provider that needs to have the machines ready to accept the data, and we don’t pay for any of that.In fact, a much better solution is shown here. Again, the user gets the upload link in some manner and then upload directly to S3. At that point, instead of S3 calling you, it will push the notification to a queue (SQS) and then your code can handle this.Here is what this looks like:Note that in this case, you are in control of how fast or slow you want to process the data on the queue. You can set a maximum number of concurrent workers / lambdas and let the cloud infrastructure manage that for you. At this point, you can smooth any peaks that you have in the process. A lot of this is just setting up the orchestration properly so you aren’t in the way, that you utilize the cloud infrastructure instead of writing your code.
GUIDs are commonly used as a key in a database. If you want to create a URL to a specific record, you may want to use the GUID in the URL. Now you have a problem: how to display the GUID in the URL? The easy solution is to use ToString("N"), but this will return the GUID in the format 0000000000000
In RavenDB 5.2 we made a small, but significant, change to how RavenDB is processing cluster wide transactions. A cluster wide transaction in RavenDB prior to the 5.2 release would look like this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
using var session = store.OpenAsyncSession(new SessionOptions
{
TransactionMode = TransactionMode.ClusterWide
});
session.Advanced.ClusterTransaction.CreateCompareExchangeValue("usernames/ayende", "users/1-A");
await session.StoreAsync(new User{Name = "Ayende"}, "users/1-A");
await session.SaveChangesAsync();
view raw
old_tx.cs
hosted with ❤ by GitHub
A cluster wide transaction in RavenDB allows you to mix writing documents as well as compare exchange values. The difference between those two options is that documents are stored at the database level and compare exchange values are stored at the cluster level. As such, a cluster wide transaction can only fail if the compare exchange values aren’t matching the expected values. To be rather more blunt, optimistic concurrency on a document with a cluster wide transaction is not going to work (in fact, we’ll throw an error if you’ll try). Unfortunately, it can be tricky to understand that and we have seen users in the field that made the wrong assumptions about how cluster wide transactions interact with documents.The most common mistake that we saw was that people expected changes to documents to be validated as part of the cluster wide transaction. However, RavenDB is a multi master database, so changes can happen across cluster wide transactions, normal transactions as well as replication from isolated locations. That make it challenging to match the expected behavior. Leaving the behavior as it, however, means that you have a potential pitfall for users to stumble into. That isn’t where we want to be, so we fixed it.Let’s consider the following piece of code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
using var session = store.OpenAsyncSession(new SessionOptions
{
TransactionMode = TransactionMode.ClusterWide
});
var user = new User{Name = "Ayende"};
await session.StoreAsync(user);
await session.StoreAsync(new {ReservedFor = user.Id}, "usernames/" + user.Name);
await session.SaveChangesAsync();
view raw
new_tx.cs
hosted with ❤ by GitHub
Notice the code in line 8, we are creating a reservation for the username by allocating a document id for the new document. However, that is the kind of code that would work when running in a single server, not in a cluster. The change we made in RavenDB 5.2 means that we are now able to make code this simple happen across the cluster in a consistent and usable manner.Behind the scenes, RavenDB creates these guard compare exchange values. Those are used to validate the state of the transaction across the cluster.The change vectors of the documents are also modified, like so: "RAFT:2-XiNix+yo0E+sArbTyoLLyQ,TRXN:559-bgXYYHaYiEWKR2mpjurQ1A".The idea is that the TRXN notation gives us the relevant compare exchange value index, which we can use to ensure that the cluster wide state of the document matches the expected state at the time of the read.In short, you can skip using compare exchange values directly in cluster wide transactions. RavenDB will understand what you mean and create the relevant entries and manage them for you. That would work also when you are working with both cluster wide and regular transactions.A common scenario for this code above is that you’ll include the usernames/ayende reservation document whenever you modify the user’s name, and that will always be using a cluster wide transactions. However, when modifying the user and not the user name, you can skip the cost of a cluster wide transaction.
I think that it was the Pragmatic Programmer that recommend that you should learn a new language a year. For me, in 2020 that was Rust. I read a bunch of books about the language, I read significant amount of code and wrote some non trivial amount of code in Rust. That was sufficient to get me to grok the language, I’m not a Rust developer by any mean, but I walked with Rusty shoes for long enough to get the feeling.This year, I decided to look into Zig. Both Zig and Rust are more or less in the same space, replacing C. They couldn’t be more different, however. Zig is a small language. I spent a couple of evenings going through the language reference and that was it, I had a pretty good idea about how to do things.The learning curve is mostly flat, and that is pretty huge. This is especially because I can’t help but compare Zig to Rust. I spent a lot of effort understanding Rust, but I had spent almost no cycles trying to grok Zig. It was simple, obvious and quite clear. In terms of power, mind, I would rate both languages on roughly the same spot. You can write really nice code in Zig, it is just that you don’t need to bend your head into funny shapes to get the compiler to accept your code. One of the key features for Zig is its comptime feature. This is a feature that allow Zig to run code at compilation time. That isn’t a new or exciting feature, to be honest, C++ had it for many years. The key difference is that Zig can use this feature for code generation. For example, to create a generic list, you’ll write the following code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
fn LinkedList(comptime T: type) type {
return struct {
pub const Node = struct {
next: ?*Node,
data: T,
};
first: ?*Node,
len: usize,
pub const Self = @This(),
fn push(self: *Self, val: T, allocator: *std.mem.allocator) error{OutOfMemory}!void {
var node = try allocator.create(Node);
node.next = self.first;
node.data = val;
self.first = node;
self.len +=1 ;
}
};
}
view raw
list.zig
hosted with ❤ by GitHub
Note that we are writing here a function that returns a type, which you can then use. That approach is incredibly powerful, but at the same time, this is simple, it is obvious. Zig is easy to learn, because there isn’t a whole lot more that is hidden from you behind the scenes.That actually leads to another really important aspect in the design of Zig. There isn’t anything behind the scenes. For example, you cannot allocate memory in Zig, there is no global function that will do that for you. You need to do so using an allocator. That means that the fact that memory allocations can fail is pervasive throughout the API, standard library and your code. There isn’t a "this may fail on rare occasions” scenario that you somehow need to handle, this is clear and in your face.At the same time, Zig does a lot more to make things easier than C. I want to focus on a few important points:Zig has the concept of Errors. In the code above, the function push() may fail because of an allocation failure. In this case, the function will fail with a return code. That is handled by the try keyword, which will abort the current function and return the error. Note that errors and regular values are separate channels in Zig (there is a union mark with ! at the function declaration).Zig has support for defer and errdefer keyword. The defer keyword works just as you would expect it to, at the function exit, it will run all the deferred statement in reverse order. The errdefer is a lot more interesting, because that will only run if the function exits with an error. This seemingly simple change has a huge impact on the code quality and the complexity that a developer need to keep in their head.Zig has built-in testing, to the point where test is a keyword in the language.To give you some context, when I was writing C code, I literally wrote the exact same thing (manually, with macros and nastiness) in order to help me get things done. In the same manner, the fact that allocation are explicit and managed from the top (all types that needs to allocate gets the allocator from their parents) means that you get to do some really cool things with memory. It is easy to say something like “this piece of code gets 10MB of memory only” and let it run like that. It also end up creating a more robust software system, I think, so memory allocations happen aren’t a rare occurrence, they happen all the time.In general, Zig feel like a lot better C, no additional mental overhead. Compared to Rust, you can get working almost immediately and the compilation speed is excellent, to the point where you don’t really need to think about it. Rust makes you feel the slow compilation cost from the get go, basically, which is noticeable as your system grows bigger. Thinking about this, I actually feel that we should compare Zig to Go, because it is closer in concept to what I think Go wanted to be. In fact, looking at the most common complaints people has against Go, Zig answers them all.If you haven’t noticed, I’m quite enjoying working with Zig.And as an aside, the fact that a language can implement a language server and get automatic IDE support is freaking amazing. You can also debug Zig code inside VS Code, for example, pretty much with no more issues than you would for native code. Zig is implemented on top of LLVM and gains a lot of the benefits from it. One thing that kept going through my mind when I looked at all that I got out of the package is: standing on the shoulders of giants.
In an ASP.NET Core application, you can configure the JSON serializer options used by controllers using the AddJsonOptions method:C#copypublic void ConfigureServices(IServiceCollection services)
{
services.AddControllers()
.AddJsonOptions(options =>
options.JsonSer
In this episode of Coffee with Pros , we have special guest speaker Oren Eini. He is the CEO of RavenDB, a NoSQL Distributed Database that's Fully Transactional (ACID) both across your database and throughout your database cluster. RavenDB Cloud is the Managed Cloud Service (DBaaS) for easy use.
I’m teaching a course in university, which gives me some interesting perspective into the mind of new people who join our profession. One of the biggest differences that I noticed was with the approach to software architecture and maintenance concerns. Frankly, some of the the exercises that I had to review made my eyes bleed a little (the students got full marks, because the point was getting things done, not code quality). I talked with the students about the topic and I realized that I have a very different perspective on software development and architecture than they have.The codebase that I work with the most is RavenDB, I have been working on the project for the past 12 years, with some pieces of code going back closer to two decades. In contrast, my rule for giving tasks for students is that I can complete the task in under two hours from an empty slate. Part and parcel of the way that I’m thinking about software is the realization that any piece of code that I’ll write is going to be maintained for a long period of time. A student writing code for a course doesn’t have that approach, in fact, it is rare that they use the same code across semesters. That lead to seeing a lot of practices as unnecessary or superfluous. Even some of the things that I consider as the very basic (source control, tests, build scripts) are things that the students didn’t even encounter up to this point (years 2 and 3 for most of the people I interact with) and they may very well get a degree with no real exposure for those concerns.Most tasks in university are well scoped, clear and they are known to be feasible within the given time frame. Most of the tasks outside of university are anything but. That got me thinking about how you can get a student to realize the value inherent in industry best practices, and the only real way to do that is to immerse them in a big project, something that has been around for at least 3 – 5 years. Ideally, you could have some project that the students will do throughout the degree, but that requires a massive amount of coordination upfront. It is likely not feasible outside of specific fields. If you are learning to be a programmer in the gaming industry, maybe you can do something like produce a game throughout the degree, but my guess is that this is still not possible. A better alternative would be to give students the chance to work with a large project, maybe even contributing code to it. The problem there is that having a whole class start randomly sending pull requests to a project is likely to cause some heartburn to the maintenance staff. What was your experience when moving from single use, transient projects to projects that are expected to run for decades? Not as a single running instance, just a project that is going to be kept alive for a long while…
