by Steve Gordon

posted on: September 18, 2024

I’ve recently been building some hobby code to dogfood the various observability tooling we develop at Elastic. Additionally, I’ve been interested in identifying the pain points of using our products as well as the .NET instrumentation libraries (from System.Diagnostics) used to instrument code in an OpenTelemetry-compatible way. Recording Activities in .NET In today’s short post, […]

by Oren Eini

posted on: September 18, 2024

It has been almost a year since the release of RavenDB 6.0. The highlights of the 6.0 release were Corax (a new blazing-fast indexing engine) and Sharding (server-side and simple to operate at scale). We made 10 stable releases in the 6.0.x line since then, mostly focused on performance, stability, and minor features.The new RavenDB 6.2 release is now out and it has a bunch of new features for you to play with and explore. The team has been working on a wide range of new features, from enabling serverless triggers to quality-of-life improvements for operations teams. RavenDB 6.2 is a Long Term Support (LTS) releaseRavenDB 6.2 is a Long Term Support release, replacing the current 5.4 LTS (released in 2022). That means that we’ll support RavenDB 5.4 until Oct 2025, and we strongly encourage all users to upgrade to RavenDB 6.2 at their earliest convenience.You can get the new RavenDB 6.2 bits on the download page. If you are running in the cloud, you can open a support request and ask to be upgraded to the new release. Data sovereignty and geo-distribution via Prefixed ShardingIn RavenDB 6.2 we introduced a seemingly simple change to the way RavenDB handles sharding, with profound implications for what you can do with it. Prefixed sharding allows you to define which shards a particular set of documents will go to. Here is a simple example:In this case, data for users in the US will reside in shards 0 & 1, while the EU data is limited to shards 2 & 3. The data from Asia is spread over shards 0, 2, & 4.  You can then assign those shards to specific nodes in a particular geographic region, and with that, you are done. RavenDB will ensure that documents will stay only in their assigned location, handling data sovereignty issues for you. In the same manner, you get to geographically split the data so you can have a single world-spanning database while issuing mostly local queries.You can read more about this feature and its impact in the documentation.Actors architecture with Akka.NETNew in RavenDB 6.2 is the integration of RavenDB with Akka.NET. The idea is to allow you to easily manage state persistence of distributed actors in RavenDB. You’ll get both the benefit of the actor model via Akka.NET, simplifying parallelism and concurrency, while at the same time freeing yourself from persistence and high availability concerns thanks to RavenDB.We have an article out discussing how you use RavenDB & Akka.NET, and if you are into that sort of thing, there is also a detailed set of notes covering the actual implementation and the challenges involved.Azure Functions integration with ETL to Azure QueuesThis is the sort of feature with hidden depths. ETL to Azure Queue Storage is fairly simple on the surface, it allows you to push data using RavenDB’s usual ETL mechanisms to Azure Queues. At a glance, this looks like a simple extension of our already existing capabilities with queues (ETL to Kafka or RabbitMQ). The reason that this is a top-line feature is that it also enables a very interesting scenario. You can now seamlessly integrate Azure Functions into your RavenDB data pipeline using this feature. We have an article out that walks you through setting up Azure Functions to process data from RavenDB.OpenTelemetry integrationIn RavenDB 6.2 we have added support for the OpenTelemetry framework. This allows your operations team to more easily integrate RavenDB into your infrastructure. You can read more about how to set up OpenTelemetry for your RavenDB cluster in the documentation.OpenTelemetry integration is in addition to Prometheus, Telegraf, and SNMP telemetry solutions that are already in RavenDB. You can pick any of them to monitor and inspect the state of RavenDB. Studio Omni-SearchWe made some nice improvements to RavenDB Studio as well, and probably the most visible of those is the Omni-Search feature.  You can now hit Ctrl+K in the Studio and just search across everything:Commands in the StudioDocumentsIndexesThis feature greatly enhances the discoverability of features in RavenDB as well as makes it a joy for those of us (myself included) who love to keep our hands on the keyboard.SummaryI’m really happy about this release. It follows a predictable and stable release cadence since the release of 6.0 a year ago. The new release adds a whole bunch of new features and capabilities, and it can be upgraded in place (including cross-version clusters) and deployed to production with no hassles.Looking forward, we have already started work on the next version of RavenDB, tentatively meant to be 7.0. We have some cool ideas about what will go into that release (check the roadmap), but the key feature is likely to make RavenDB a more intelligent database, one might even say, artificially so.

by Oren Eini

posted on: September 17, 2024

I wanted to test low-level file-system behavior in preparation for a new feature for RavenDB. Specifically, I wanted to look into hole punching - where you can give low-level instructions to the file system to indicate that you’re giving up disk space, but without actually reducing the size of the file.This can be very helpful in space management. If I have a section in the file that is full of zeroes, I can just tell the file system that, and it can skip storing that range of zeros on the disk entirely. This is an advanced feature for file systems. I haven't actually used that in the past, so I needed to gain some expertise with it.I wrote the following code for Linux:int fd = open("test.file", O_CREAT | O_WRONLY, 0644); lseek(fd, 128 * 1024 * 1024 - 1, SEEK_SET); // 128MB file write(fd, "", 1); fallocate(fd, // 32 MB hole from the 16MB..48MB range FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 16 * 1024 * 1024, 32 * 1024 * 1024); close(fd);The code for Windows is here if you want to see it. I tested the feature on both Windows & Linux, and it worked. I could see that while the file size was 128MB, I was able to give back 16MB to the operating system without any issues. I turned the code above into a test and called it a day. And then the CI build broke. But that wasn’t possible since I tested that. And there had been CI runs that did work on Linux. So I did the obvious thing and started running the code above in a loop. I found something really annoying. This code worked, sometimes. And sometimes it just didn’t. In order to get the size, I need to run this code:struct stat st; fstat(fd, &st); printf("Total size: %lld bytes\n", (long long)st.st_size); printf("Actual size on disk: %lld bytes\n", (long long)st.st_blocks * 512);I’m used to weirdness from file systems at this point, but this is really simple. All the data is 4KB aligned (in fact, all the data is 16MB aligned). There shouldn’t be any weirdness here.As you can see, I’m already working at the level of Linux syscalls, but I used strace to check if there is something funky going on. Nope, there was a 1:1 mapping between the code and the actual system calls issued.That means that I have to debug deeper if I want to understand what is going on. This involves debugging the Linux Kernel, which is a Big Task. Take a look at the code in the relevant link. I’m fairly certain that the issue is in those lines. The problem is that this cannot be, since both offset & length are aligned to 4KB.I got out my crystal ball and thinking hat and meditated on this. If you’ll note, the difference between the expected and actual values is exactly 4KB. It almost looks like the file itself is not aligned on a 4KB boundary, but the holes must be. Given that I just want to release this space to the operating system and 4KB is really small, I can adjust that as a fudge factor for the test. I would love to understand exactly what is going on, but so far the “file itself is not 4KB aligned, but holes are” is a good working hypothesis (even though my gut tells me it might be wrong). If you know the actual reason for this, I would love to hear it. And don't get me started on what happened with sparse files in macOS. There, the OS will randomly decide to mark some parts of your file as holes, making any deterministic testing really hard.

by Andrew Lock

posted on: September 17, 2024

In this post I show how you can enable Git's long-path support in a GitLab job to override Window's MAX_PATH limitations which can cause git clone failures…

by

posted on: September 16, 2024

ManualResetEventSlim represents a thread synchronization event that, when signaled, must be reset manually. This is a common synchronization primitive. However, it doesn't expose a method to wait asynchronously. Hopefully, it's not too complicated to create an extension method using ThreadPool.Regi

by Andrew Lock

posted on: September 10, 2024

In this post I show how you can use the PublicApiGenerator project to document your public API and build a snapshot unit test to prevent breaking changes…

by

posted on: September 09, 2024

Microsoft Graph is a REST API that allows you to interact with Microsoft 365 services. You can use it to automate tasks such as creating users, updating their properties, or exporting data. In this post, I'll show you how to export user email addresses and aliases from Microsoft Entra using PowerSh

by Andrew Lock

posted on: September 03, 2024

In this post I show how to create, extract, and list tar files using the built-in tar support introduced in .NET 7, and discuss some of the limitations…

by Gérald Barré

posted on: September 02, 2024

Someone asked me why I write blog posts. So, let's write a blog post about why I write blog posts!#Reason 1: I forget thingsAs a developer, you learn lots of new things every day. But my memory has limited capacity, so I also forget a lot of things. One solution is to dump part of my knowledge into

by Maddy Montaquila

posted on: August 29, 2024

.NET Aspire 8.2 is here with some nice updates for components... we mean integrations! Learn more about this rename, what it means for you, and what the teams has been adding to testing in .NET Aspire!