Next week is Black Friday, which has reached a global phenomenon status. It is a fun day for shoppers, and a nervous wreck for IT admins everywhere. It is not uncommon to see traffic doubles or triples and the actual load (processing more heavyweight requests) can go up an order of magnitude. Preparing for Black Friday can be a harrowing issue since you have a narrow window of opportunity and it is hard to know exactly where the stress points are.
This year, I decided to make your life easier, and RavenDB is offering a Black Friday Surge to all our customers. No, we aren’t offering you 50% off and everything must go. What we do instead is try to be of help.
This Black Friday (and Cyber Monday as well), we are offering all our customers double what they paid for. When running RavenDB on premise, if you purchased a RavenDB license for a 12 cores cluster (running on 3 nodes of 4 cores each), we’ll offer you 30 days of double the core count. In other words, you can scale your system to be twice as powerful, and it won’t cost you a cent.
On the cloud, as well, we will provide users with credits to upgrade their clusters to the next level up (doubling their power) for a full week during the next 30 days. Again, there is no extra cost here.
You can register for the Surge here to request the upgrade and you’ll get twice as much power to handle the increased load.
Enjoy the power up!
A really nice feature that we have in RavenDB 5.3 is support for wire protocol compatibility with PostgreSQL. That opens up RavenDB to the entire PostgreSQL ecosystem. You are now able to connect to a RavenDB instance using the tools such as psql, Npgsql, etc. This feature is both surprisingly simple and incredibly complex at the same time.
The actual wire protocol from Postgres is well documented and pretty clean. Doing a clean room implementation of that is a straightforward process. Adding that to RavenDB, on the other hand, led to a number of interesting challenges. To start with, the protocol assumes, at the wire level, that all the rows that you return for a query have the same structure. This is a very reasonable assumption to make for a relation database protocol, but it doesn’t hold true when you are talking about a schema-less database such as RavenDB.
Then there is the fact that clients will generate all sort of pretty scary queries before you even get to running a user’s query. For example, take a look at how Npgsql is detecting the capabilities of the database that it connects to. Just supporting the wire protocol isn’t sufficient, you also need to support quite a bit of additional behavior.
When we implemented this feature, we decided that we’ll support the wire protocol, so you’ll be able to connect, issue queries and get results. However, the query language itself is going to be RQL. We aren’t attempting to pretend that we are a PostgreSQL instance to the outside world, only implement enough to make integration and compatibility work.
Here is an example of running a query through the Postgres integration.
This is an experimental feature, mind you. It is showing a lot of promise, but we want to get some more feedback from our users about which ways we should take it. The feature opens up many doors, but it is also bringing with it a non trivial amount of complexity.
This feature requires that we’ll open up another port to the world, this is something that we require the user to explicitly allow. To enable this feature, you’ll need to set the following options in the settings.json configuration file:
"Integrations.PostgreSQL.Enabled": true"Features.Availability" : "Experimental"
You can also control which port it will use using the Integrations.PostgreSQL.Port configuration option. We default to 5433 if none is specified.
At the current time, we only allow to issue queries and not modify data using the Postgres integration. This is something that we would very much like more feedback on, what kind of scenarios would you like to have where write scenario is supported? What kind of writes do you expect to have at that point?
Finally, a word about security. The PostgreSQL protocol supports using TLS for encryption. When running in insecure mode, RavenDB will reject SSL/TLS connections from Postgres client. When running in a secured mode (the default), the same server certificate that is used inside of RavenDB will also be used for the Postgres connection. However, while we usually require that the other side also authenticates using a client certificate, in the case of Postgres connection, we run into a problem. There are quite a few scenarios where we found out that while the Postgres protocol supports mutual authentication using client certificates, clients aren’t supporting it.
For that reason, we are allowing user & password authentication (on top of TLS connection, obviously) for the Postgres connections at this time. Note that there is no correlation between the Postgres login and the access to any other RavenDB features (where client certificates is the only option).
This is part of the RavenDB 5.3 release (expected in mid November) and will be available in the Professional and Enterprise editions.
RavenDB tries to be a good neighbor in your systems. RavenDB is typically used in polyglot solutions and we are often brought in to existing ecosystems. One of the things that we do to make it easier to use RavenDB is to have a full suite of built-in tools to make pushing data to other destinations.
For example, you can define an ETL process that will push document changes from RavenDB (potentially transforming & filtering them) to a relational database, another RavenDB instance, a data lake / OLAP system and much more.
In RavenDB 5.3 we have added Elasticsearch as an ETL target for RavenDB. If you are familiar with RavenDB ETL processes, the behavior is pretty much the same as you would expect. You select which collections you want to push to Elasticsearch, you provide a script that filters and transform the data and then you are done. From that point on, it is RavenDB’s responsibility to keep the Elasticsearch target up to date with any changes that are happening inside of RavenDB.
I’ll discuss the exact details on how to make it work shortly, but first I want to talk a bit about the usage scenario for this. Elasticsearch, just like RavenDB, it using Lucene behind the scenes to implement indexes. Unlike RavenDB, however, Elasticsearch is all about… well, searching. In that context, there is a pretty big overlap between RavenDB and Elasticsearch. In fact, one of the primary reasons we see people selecting RavenDB is that they now don’t need to maintain multiple environments (one to store the data and an Elasticsearch cluster for searching on that), RavenDB is able to undertake both needs in a single highly integrated and performant package.
The most common scenario for Elasticsearch ETL is when you already have an existing investment in Elasticsearch. RavenDB will naturally integrate into your environment, without needing to make any significant changes. That can enable you to start running queries, Kibana dashboard, etc on your RavenDB documents.
Here is the transformation script:
And the configuration telling RavenDB where to go:
You can push multiple collections to multiple Elasticsearch indexes. It is important to note that you must include the RavenDB document Id as a property in the script and also set it in the destination index configuration. If the Elasticsearch index doesn't already exist, RavenDB will create it for you on the fly.
This is… pretty much it. The actual feature is fully fledged, of course. You get monitoring and tracking, it will run in high availability mode and will be assigned an owner node in the cluster, etc. If there is a failure on Elasticsearch, there is no data loss, RavenDB will wait for the target to come back up and push all the data that was changed in the meantime. The ETL process is an online process, which means that you can expect to see changes in RavenDB reflected in Elasticsearch index within a few milliseconds of the transaction commit.
This feature is available in the Professional and Enterprise editions of RavenDB and will be included in the RavenDB 5.3 released, scheduled for mid November.
About a year and a half ago Kamran wrote an article showing how you can implement rate limits in RavenDB. He used both counters and expiration to handle this scenario. I think that incremental time series make it so much easier to work with, it’s not even funny.Consider the following code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
using var session = GetDocumentStore().OpenAsyncSession();
var now = DateTime.UtcNow;
var result = await session.Query<Watch>()
.Where(x => x.Id == "api/stock-exchange")
.Select(w =>
RavenQuery.TimeSeries(w, "INC:Usage", now.AddSeconds(-30), now) // past 30 seconds
.GroupBy(x => x.Days(1)) // only single result
.Select(x => x.Sum())
.ToList()
).SingleOrDefaultAsync();
if (result?.Results[0]?.Sum[0] > 50)
{
throw new RateLimitedExceededException("More than 50 requests in the last 30 seconds, try again later");
}
// do work here....
session.IncrementalTimeSeriesFor("api/stock-exchange","INC:Usage")
.Increment(1);
await session.SaveChangesAsync();
view raw
rate-limit.cs
hosted with ❤ by GitHub
Those are ~20 lines of code and that is all you need to handle a sliding window for rate limits. This will work quite nicely even with high levels of concurrency, you can also establish more interesting scenario, such as maximum 50 requests in the past 30 seconds, but also maximum of 200 requests in the past 5 minutes, etc. If you ever tried to make sense of the way Let’s Encrypt is handling rate limits, for example, you can see how you can get some really crazy complexity.That is now all handled for you. We have the part where we record the number of operations under the limit, and the rest is limited only by your imagination.For fun, you can now decide what you want to do with this data. If you are only interested in that for rate limiting purposes, you can tell RavenDB to just delete the old data once it is no longer of use using time series retention. You can also just roll them up to a 5 minute boundary and keep that information (for example, monitoring, audits and billing purposes come to mind).The whole thing is far more cohesive and easier to work with.
It's typical for API endpoints to call application or domain services. In the case of success, the API can simply return Ok and the result…Keep Reading →
Everyone is on the cloud these days, and one of the things that I keep seeing pushed is the notion of usage based billing. Basically, the idea that you are paying for what you use.
Let’s assume that we are building a software as a service where users can submit an image and you’ll do some computation on that. The actual details aren’t relevant. What matters is that your pricing model is based around how much time processing each image takes and how much memory is used. You are running this on many machines and need to figure out how to do billing at the end of the month. It turns out that this can be quite a challenge. With incremental time series, a lot of the details around that just go away.
Here is how you can implement this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
async Task Process(ProcessImageCmd cmd)
{
var sp = Stopwatch.StartNew();
var size = cmd.Image.SizeInBytes * GetMemMultiplierFor(cmd.Image.Format);
await ProcessImage(cmd.Image.GetStream()); // do actual work
sp.Stop();
using var session = store.OpenAsyncSession();
await session.StoreAsync(new ProcessingRecord(cmd.AccountId, cmd.Image.Name,sp.ElapsedMilliseconds));
session.IncrementalTimeSeriesFor(cmd.AccountId, "INC:Processing")
// record the details for billing
.Increment(new[]{ sp.ElapsedMilliseconds, size });
await session.SaveChangesAsync();
}
view raw
process.cs
hosted with ❤ by GitHub
You count the required memory as well as the actual runtime and record that in an incremental time series. We are also storing the details in a separate document for that particular run in the same transaction (if the user cares about that level of detail). The interesting bit about how this can be used is that the data is now immediately available for the user to see how much they are going to be billed.
Typically, a lot of time is spent in figuring out how to record those details efficiently and then how to query and aggregate those. We tested time series in RavenDB to billions of data points, and the internal format lends itself very well to aggregated queries.
Now you can take the code above, run it on 100s of machines, and it will all end up giving you the proper result in the end.
Table Of ContentsWhat is OpenTelemetry?Code instrumentationLoggingTracingMetricsExporting dataStarting the collector and back-endsConfiguring the application to export data to the collectorMonitoring multiple servicesEnriching the request activity created by ASP.NET CoreAdditional resourcesNoteThe
In RavenDB 5.0 we had a major new feature, native time series support. Using this feature, you can store values over time, query and aggregate them, store them efficiently, produce rollups, etc.
The classic example for time series data in RavenDB is when you have data coming from sensors. For example a Fitbit monitoring heartrate, a stock exchange feed giving you stock values. You don’t care about a particular value, you care about the value over time. It turns out that there are quite a lot of use cases for those kind of details. We have seen a major pick up in IoT related fields in particular.
However, the API we provided for users to insert data for time series had a limitation, have a look:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
// add one value
void Append(DateTime timestamp, double value, string tag = null);
// add multiple values (up to 32)
void Append(DateTime timestamp, IEnumerable<double> values, string tag = null);
view raw
api-v5.cs
hosted with ❤ by GitHub
The API gives you the ability to record a value (or a set of values) at a particular point in time, with an optional tag for additional meaning. What is the problem with this API, then?
Well, it works great if you are processing data from a singular source (the stock exchange feed, or a medical device), but it fails to do its job if you may need to record multiple values for the same timestamp.
Huh? What does that even mean? If we a are storing a value per timestamp, obviously there should be a value for that timestamp. How can there be multiple values? Note that here I’m not talking about something like location (with latitude and longitude coordinates), those are covered under storing an array of values on the same timestamp.
The issue happens when you have the need to record multiple different values at the same timestamp. Typical time series are things like Heartrate, Location, StockPrice, etc. Having multiple values for the same thing at the same time frame doesn’t really work. In the Location time series, if I’m both here and there, you can expect trouble (if only because the paradox cops will show up). A stock may have different prices at the same time in different exchanges, sure, but that is not the same value, by its very nature.
There is a common scenario where this will happen. When what I’m recording is not the full value, but part of that value. The classic example for that is tracking page views. Let’s say that I want to know how many people are looking at this blog post, I cannot use the Append() API for that purpose. Each individual operation is going to belong to a particular timestamp. What happens if I have two views on this post at the exact same millisecond? For that matter, what happens in the more “interesting” case of having writes to the same millisecond on two different nodes in the cluster?
With timeseries as we envisioned them for the 5.0 release, that wasn’t an issue, a timeseries had a value in a particular timestamp. But supporting a scenario such as tracking views, or any scenario where we want to record partial data and have RavenDB take care of everything else isn’t served well by this model.
Note that RavenDB already has the notion of distributed counters, they are intended specifically for doing such things. It is trivial in RavenDB to implement a counter that would track the overall views on a post. It will also handle concurrency, distributing data between nodes, everything that needs to be handled. So why can’t I use that?
It turns out that I typically want to know more than just the total number of views on the post, I want to know when they happened. Counters are only a partial answer for that.
That is why incremental time series were created. They are here to marry the ability of time series to track a value over time and the distributed counters ability to aggregate information concurrently and in a safe distributed manner. Here is the new API for incremental time series:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
// Increment at the current time, single value
void Increment(double value);
// Increment at the current time, up to 32 values
void Increment(IEnumerable<double> values);
// Increment at a given point in time (UTC only), single value
void Increment(DateTime timestamp, double value);
// Increment at a given point in time (UTC only), up to 32 values
void Increment(DateTime timestamp, IEnumerable<double> values);
view raw
api-v53.cs
hosted with ❤ by GitHub
The changes are apparent at the API level, the Increment() is not setting the value, it is incrementing it with a delta value. So two increments on the same timestamp will give you the right result. Note that we don’t have a way to tag the entry any longer. That is no longer meaningful, because a single timestamp may have multiple different values. The method is called increment, but note that you can also pass negative values, if you want to reduce the amount.
You can see in the image on the right how this looks like in the studio. An incremental time series is one that has the “INC:” prefix in the name. Such a time series is able to accept only increment operations, it will reject attempts to append values to it. In the same sense, a non incremental time series will not allow you to increment a value, only append new entries. We wanted to have a strong separation between the two time series modes because mixing them up resulted in a huge mess of edge cases that are really hard to solve.
I probably should explain the terminology here, because it reflects an important distinction:
Append – add a new timestamp and the value(s) for that time. This appends to the time series a new entry. Appending an entry to a time that is already in the timeseries will overwrite that time.
Increment – add a new timestamp and its values. If there is already value for that time in the time series, we’ll add the new value and existing value together, writing their sum as the new value.
That isn’t actually how it works internally, but that is the conceptual model.
Aside from using increment to set the values, incremental time series behave just like any other time series. You can query over them, aggregate, index, etc. They can create rollups (a rolled up incremental time series is a normal time series, not an incremental one), apply retention polices, and everything else that you can do with a time series, the special behavior of incremental time series does not extend to its rolled-up versions.
Here is a full example of how you can use this feature:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
using var session = store.OpenAsyncSession();
// this is safe to run concurrently
session.IncrementalTimeSeriesFor("posts/2931-A", "INC:Views").Increment(1);
await Task.Delay(1000); // simulate work
// here we commit the transaction, the incremented timestamp
// will be the recording time, not the tx commit time
await session.SaveChangesAsync();
view raw
usage.cs
hosted with ❤ by GitHub
As usual, this is transactional with any other operation you may want to do, so you can increment a time series along side uploading an attachment and modifying a document, as a single atomic transaction.
And now we can ask about view counts on an hourly basis for the last week, like so:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
from Posts
where id() = 'posts/2931-A'
select timeseries(
from 'INC:Views' between $weekAgo and $now
group by 1h
select sum()
)
view raw
stats.sql
hosted with ❤ by GitHub
This feature is going to be available in all editions of RavenDB 5.3, expected for release in mid November. I got so many ideas about what you can use this for .
Almost as soon as we introduced concurrent subscriptions, we ran into a serious problem in their use. The desire was to do things in a serial fashion. That was quite infuriating, because we spent to much time working on making things concurrent, and now we had to deal with making them serial again? What the hell?
Before I dive any further, it will probably be for the best if I explained a bit more about the context of this very strange feature request.
Consider a system where the subscription is used to process commands, which may relationships between one another. For example, consider the following commands (all of them belonging to the same “Commands” collection):
EmployeePayroll – commands/40-A
EmployeeBankAccountChange – commands/34-A
EmployeeContractUpdate – commands/49-C
For each one of those commands (and many more), we want to run some logic. Some of this requires us to touch third party services, which means that we are likely to be slow / stalled on some cases. That is the exact case for using concurrent subscriptions.
The developers quickly jumped on the new system, setting the mode of the subscription as concurrent and running multiple workers. Things worked, latency was down and everyone was happy. Everyone, that is, except for George. The problem was George had gotten married recently. Well, that wasn’t the actual problem. George is happily married. The problem is that George and his wife have a new joint bank account. George let the HR department know about the new bank account in advance, which resulted in the EmployeeBankAccountChange command being generated. Then payroll day hit, and we have an EmployeePayroll command as well.
This is where things started to get iffy. In terms of timing, the EmployeeBankAccountChange happened before the EmployeePayroll command. When the subscription was running in serial mode, it was guaranteed that it will always process the commands in the order that they were modified. That meant that handling things like changing the bank account and actually paying had a very natural order. If you made the change before payroll, it got processed before hand, otherwise, it was processed afterward.
With concurrent subscriptions, this is no longer the situation. We are still working roughly in the order of modification, but we are no longer guaranteeing it. And it is possible to process documents out of order.
RavenDB’s concurrent subscriptions will ensure that you’ll not have to worry about concurrent processing of a single document, but in this case, there are different documents, so they can be processed concurrently. An EmployeeBankAccountChange may take a long time (verifying accounts, etc) while EmployeePayroll is just adding a line to a ACH file, so it is very likely that we’ll process the payroll before the account change. And that makes George very sad. Let’s see how we can avoid depressing the newlywed.
One option is to make use of another RavenDB feature, the compare exchange support. This allows you to use strongly consistent, cluster-wide, values which are suitable for distributed locks. I looked into what it will take to build this and quailed in fear. I don’t want to let things become this complicated.
The key issue here is that we want both concurrency and serial work. An interesting observation is that there is a scope for such things. Commands on the same employee should run in the same order they were issued, commands on different employees are free to run in whatever order they like. How can we make this work without diving head first into complexity the like of which will keep you up at night?
For the most part, we can assume that concurrent operations for the same employee is rare. Even when we have multiple commands for the same employee, we can expect that there won’t be many of them. Given that, we can change the way we model the commands themselves. Instead of creating a document per command, we’ll have a document per employee.
Where before we had this model:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
public record class EmployeePayrollCommand(string EmployeeId, decimal Amount);
public record class EmployeeBankAccountChange(string EmployeeId, BankInfo bankInfo);
public record class EmployeeContractUpdateCommand(string EmployeeId, HourlyRate newRate);
view raw
before.cs
hosted with ❤ by GitHub
We’ll now have the following model:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
public record class EmployeeCmd();
public record class EmployeePayrollCommand(decimal Amount) : EmployeeCmd();
public record class EmployeeBankAccountChange(BankInfo bankInfo) : EmployeeCmd();
public record class EmployeeContractUpdateCommand(HourlyRate newRate) : EmployeeCmd();
// this is the document, containing an array of commands in order
public record class EmployeeCommands(string EmployeeId, List<EmployeeCmd> Commands);
view raw
after.cs
hosted with ❤ by GitHub
What does this give us? We now have a commands/employees/1-A for the first employee, all operations on the employee and handled as a single unit, guaranteed by the concurrent subscription. Let’s explore further how that works, okay?
With the previous model/modeling, to register a command, we need to just call:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
using var session = store.OpenAsyncSession();
await session.StoreAsync(new EmployeePayroll("employees/1-A", 500));
await session.SaveChangesAsync();
view raw
before_register.cs
hosted with ❤ by GitHub
All the commands were using the Commands collection, so the subscription worker will look like::
from Commands
And if we process this concurrently, we may process the commands for the same employee at the same time, leading to sadness in the household of George. Instead, with the new model/modeling, we can use the patching API to handle this. Here is what this looks like:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
using var session = store.OpenAsyncSession();
var cmd = new EmployeePayroll(500);
session.Advanced.AddOrPatch(
"commands/employees/1-A",
new EmployeeCommands("employees/1-A", new [] { cmd} ), // create new instance
// run a patch operation on existing instance
x => x.Commands, u => u.Add(cmd) );
await session.SaveChangesAsync();
view raw
after_register.cs
hosted with ❤ by GitHub
The idea in this case is that all commands for the same employee use the same document. If there isn’t already such a value, we’ll create a new instance, otherwise, we’ll apply the patch script and add to it. The end result is that we can have multiple concurrent operations and they will all be added to the same document in order of execution. However, so far this has nothing to do with concurrent subscriptions. What do we do from here? Here is what the subscription worker looks like after these changes:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show hidden characters
worker.Run(async batch =>
{
using var session = batch.OpenAsyncSession();
foreach (var item = batch.Items)
{
foreach (var cmd in item.Result.Commands)
{
await cmd.ExecuteAsync();
}
// now clear the commands we executed / delete the collection if needed
session.Advanced.Defer(new PatchCommandData(_docId, null, new PatchRequest
{
Script = @"
this.Commands = this.Commands.splice(0, args.CmdCount);
if(this.Commands.length == 0) // no more commands
del(id(this));
",
Values = { ["CmdCount"] = item.Result.Commands.Length}
}));
}
await session.SaveChangesAsync();
});
view raw
process.cs
hosted with ❤ by GitHub
The idea is that when we enqueue a command, we register them in the document specifically for the employee (the scope for serial work in a concurrent subscription) and when we process the command in the subscription worker we patch out all the commands that we already executed.
This behavior will guarantee that we can process commands serially within a concurrent worker. All commands for the same employee will be processed serially in the order they were submitted, while different employees will be processed concurrently.We even support adding additional commands to the employee document while the worker is processing commands, we’ll simply handle them in the next batch after the employee commands are all done.
One thing that I’m not discussing here is what to do in case we have concurrent modifications on the commands document in multiple nodes? That would generate a conflict and RavenDB defaults to selecting the latest version. You can configure RavenDB to resolve this property, I talk about this at length here.
Aside from leaning on the new concurrent subscriptions feature, all the rest of the things that we have been using in this post to solve the problem are long standing features of RavenDB and both conceptually and in practice this gives us a great deal of simplicity to handle a non trivial issue.
As usual, I would very much welcome your feedback.
