by Andrew Lock

posted on: May 13, 2025

In this post I use the new Microsoft's new .NET AI template to ingest the contents of a website and create a chatbot that can answer questions with citations…

by Oren Eini

posted on: May 12, 2025

Let’s say I want to count the number of reachable nodes for each node in the graph. I can do that using the following code:void DFS(Node start, HashSet<Node> visited) { if (start == null || visited.Contains(start)) return; visited.Add(start); foreach (var neighbor in start.Neighbors) { DFS(neighbor, visited); } } void MarkReachableCount(Graph g) { foreach(var node in g.Nodes) { HashSet<Node> visited = []; DFS(node, visisted); node.ReachableGraph = visited.Count; } }A major performance cost for this sort of operation is the allocation cost. We allocate a separate hash set for each node in the graph, and then allocate whatever backing store is needed for it. If you have a big graph with many connections, that is expensive. A simple fix for that would be to use:void MarkReachableCount(Graph g) { HashSet<Node> visited = []; foreach(var node in g.Nodes) { visited.Clear(); DFS(node, visisted); node.ReachableGraph = visited.Count; } }This means that we have almost no allocations for the entire operation, yay!This function also performs significantly worse than the previous one, even though it barely allocates. The reason for that? The call to Clear() is expensive. Take a look at the implementation - this method needs to zero out two arrays, and it will end up being as large as the node with the most reachable nodes. Let’s say we have a node that can access 10,000 nodes. That means that for each node, we’ll have to clear an array of about 14,000 items, as well as another array that is as big as the number of nodes we just visited. No surprise that the allocating version was actually cheaper. We use the visited set for a short while, then discard it and get a new one. That means no expensive Clear() calls. The question is, can we do better? Before I answer that, let’s try to go a bit deeper in this analysis. Some of the main costs in HashSet<Node> are the calls to GetHashCode() and Equals(). For that matter, let’s look at the cost of the Neighbors array on the Node.Take a look at the following options:public record Node1(List<Node> Neighbors); public record Node2(List<int> NeighborIndexes);Let’s assume each node has about 10 - 20 neighbors. What is the cost in memory for each option? Node1 uses references (pointers), and will take 256 bytes just for the Neighbors backing array (32-capacity array x 8 bytes). However, the Node2 version uses half of that memory.This is an example of data-oriented design, and saving 50% of our memory costs is quite nice. HashSet<int> is also going to benefit quite nicely from JIT optimizations (no need to call GetHashCode(), etc. - everything is inlined).We still have the problem of allocations vs. Clear(), though. Can we win?Now that we have re-framed the problem using int indexes, there is a very obvious optimization opportunity: use a bit map (such as BitsArray). We know upfront how many items we have, right? So we can allocate a single array and set the corresponding bit to mark that a node (by its index) is visited.That dramatically reduces the costs of tracking whether we visited a node or not, but it does not address the costs of clearing the bitmap.Here is how you can handle this scenario cheaply:public class Bitmap { private ulong[] _data; private ushort[] _versions; private int _version; public Bitmap(int size) { _data = new ulong[(size + 63) / 64]; _versions = new ushort[_data.Length]; } public void Clear() { if(_version++ < ushort.MaxValue) return; Array.Clear(_data); Array.Clear(_versions); } public bool Add(int index) { int arrayIndex = index >> 6; if(_versions[arrayIndex] != _version) { _versions[arrayIndex] = _version; _data[arrayIndex] = 0; } int bitIndex = index & 63; ulong mask = 1UL << bitIndex; ulong old = _data[arrayIndex]; _data[arrayIndex] |= mask; return (old & mask) == 0; } }The idea is pretty simple, in addition to the bitmap - we also have another array that marks the version of each 64-bit range. To clear the array, we increment the version. That would mean that when adding to the bitmap, we reset the underlying array element if it doesn’t match the current version. Once every 64K items, we’ll need to pay the cost of actually resetting the backing stores, but that ends up being very cheap overall (and worth the space savings to handle the overflow).The code is tight, requires no allocations, and performs very quickly.

by Gérald Barré

posted on: May 12, 2025

In .NET, many types provide a static Parse method to convert strings into their respective types. For example:C#copyint.Parse("123"); double.Parse("123.45"); DateTime.Parse("2023-01-01"); IPAddress.Parse("192.168.0.1");However, enums require the use of the Enum.Parse method:C#copyEnum.Parse<MyEn

by Oren Eini

posted on: May 09, 2025

I ran into an interesting post, "Sharding Pgvector," in which PgDog (provider of scaling solutions for Postgres) discusses scaling pgvector indexes (HNSW and IVFFlat) across multiple machines to manage large-scale embeddings efficiently. This approach speeds up searches and improves recall by distributing vector data, addressing the limitations of fitting large indexes into memory on a single machine.That was interesting to me because they specifically mention this Wikipedia dataset, consisting of 35.1 million vectors. That… is not really enough to justify sharding, in my eyes. The dataset is about 120GB of Parquet files, so I threw that into RavenDB using the following format:Each vector has 768 dimensions in this dataset. 33 minutes later, I had the full dataset in RavenDB, taking 163 GB of storage space. The next step was to define a vector search index, like so:from a in docs.Articles select new { Vector = CreateVector(a.Embedding) }That index (using the HNSW algorithm) is all that is required to start doing proper vector searches in RavenDB.Here is what this looks like - we have 163GB for the raw data, and the index itself is 119 GB. RavenDB (and PgVector) actually need to store the vectors twice - once in the data itself and once in the index. Given the size of the dataset, I used a machine with 192 GB of RAM to create the index. Note that this still means the total data size is about ⅓ bigger than the available memory, meaning we cannot compute it all in memory. This deserves a proper explanation  - HNSW is a graph algorithm that assumes you can cheaply access any part of the graph during the indexing process. Indeed, this is effectively doing pure random reads on the entire dataset. You would generally run this on a machine with at least 192 GB of RAM.  I assume this is why pgvector required sharding for this dataset.I decided to test it out on several different machines. The key aspect here is the size of memory, I’m ignoring CPU counts and type, they aren’t the bottleneck for this scenario. As a reminder, we are talking about a total data size that is close to 300 GB.RAMRavenDB indexing time:192 GB2 hours, 20 minutes64 GB14 hours, 8 minutes32 GB37 hours, 40 minutesNote that all of those were run on a single machine, all using local NVMe disk. And yes, that is less than two days to index that much data on a machine that is grossly inadequate for it.I should note that on the smaller machines, query times are typically ~5ms, so even with a lot of data indexed, the actual search doesn’t need to run on a machine with a lot of memory. In short, I don’t see a reason why you would need to use sharding for that amount of data. It can comfortably fit inside a reasonably sized machine, with room to spare. I should also note that the original post talks about using the IVFFlat algorithm instead of HNSW. Pgvector supports both, but RavenDB only uses HNSW. From a technical perspective, I would love to be able to use IVFFlat, since it is a much more traditional algorithm for databases. You run k-means over your data to find the centroids (so you can split the data into reasonably sized chunks), and then just do an efficient linear search on that small chunk as needed. It fits much more nicely into the way databases typically work. However, it also has some significant drawbacks:You have to have the data upfront, you cannot build it incrementally.The effectiveness of the IVFFlat index degrades over time with inserts & deletes, because the original centroids are no longer as accurate. Because of those reasons, we didn’t implement that. HNSW is a far more complex algorithm, both in terms of the actual approach and the number of hoops we had to go through to implement that efficiently, but as you can see, it is able to provide good results even on large datasets, can be built incrementally and doesn’t degrade over time.Head-to-head comparisonI decided to run pgvector and RavenDB on the same dataset to get some concrete ideas about their performance. Because I didn’t feel like waiting for hours, I decided to use this dataset. It has 485,859 vectors and about 1.6 GB of data.RavenDB indexed that in 1 minute and 17 seconds. My first attempt with pgvector took over 7 minutes when setting maintenance_work_mem = '512MB'. I had to increase it to 2GB to get more reasonable results (and then it was 1 minute and 49 seconds).RavenDB is able to handle it a lot better when there isn’t enough memory to keep it all in RAM, while pgvector seems to degrade badly. SummaryIn short, I don’t think that you should need to go for sharding (and its associated complexity) for that amount of data. And I say that as someone whose database has native sharding capabilities. For best performance, you should run large vector indexes on machines with plenty of RAM, but even without that, RavenDB does an okay job of keeping things ticking.

by Oren Eini

posted on: May 08, 2025

We recently tackled performance improvements for vector search in RavenDB. The core challenge was identifying performance bottlenecks. Details of specific changes are covered in a separate post. The post is already written, but will be published next week, here is the direct link to that.In this post, I don’t want to talk about the actual changes we made, but the approach we took to figure out where the cost is. Take a look at the following flame graph, showing where our code is spending the most time. As you can see, almost the entire time is spent computing cosine similarity. That would be the best target for optimization, right?I spent a bunch of time writing increasingly complicated ways to optimize the cosine similarity function. And it worked, I was able to reduce the cost by about 1.5%!That is something that we would generally celebrate, but it was far from where we wanted to go. The problem was elsewhere, but we couldn’t see it in the profiler output because the cost was spread around too much. Our first task was to restructure the code so we could actually see where the costs were. For instance, loading the vectors from disk was embedded within the algorithm. By extracting and isolating this process, we could accurately profile and measure its performance impact. This restructuring also eliminated the "death by a thousand cuts" issue, making hotspots evident in profiling results. With clear targets identified, we can now focus optimization efforts effectively.That major refactoring had two primary goals. The first was to actually extract the costs into highly visible locations, the second had to do with how you address them. Here is a small example that scans a social graph for friends, assuming the data is in a file.def read_user_friends(file, user_id: int) -> List[int]: """Read friends for a single user ID starting at indexed offset.""" pass # redacted def social_graph(user_id: int, max_depth: int) -> Set[int]: if max_depth < 1: return set() all_friends = set() visited = {user_id} work_list = deque([(user_id, max_depth)]) with open("relations.dat", "rb") as file: while work_list: curr_id, depth = work_list.popleft() if depth <= 0: continue for friend_id in read_user_friends(file, curr_id): if friend_id not in visited: all_friends.add(friend_id) visited.add(friend_id) work_list.append((friend_id, depth - 1)) return all_friendsIf you consider this code, you can likely see that there is an expensive part of it, reading from the file. But the way the code is structured, there isn’t really much that you can do about it. Let’s refactor the code a bit to expose the actual costs. def social_graph(user_id: int, max_depth: int) -> Set[int]: if max_depth < 1: return set() all_friends = set() visited = {user_id} with open("relations.dat", "rb") as file: work_list = read_user_friends(file, [user_id]) while work_list and max_depth >= 0: to_scan = set() for friend_id in work_list: # gather all the items to read if friend_id in visited: continue all_friends.add(friend_id) visited.add(friend_id) to_scan.add(curr_id) # read them all in one shot work_list = read_users_friends(file, to_scan) # reduce for next call max_depth = max_depth - 1 return all_friendsNow, instead of scattering the reads whenever we process an item, we gather them all and then send a list of items to read all at once. The costs are far clearer in this model, and more importantly, we have a chance to actually do something about it.Optimizing a lot of calls to read_user_friends(file, user_id) is really hard, but optimizing read_users_friends(file, users) is a far simpler task. Note that the actual costs didn’t change because of this refactoring, but the ability to actually make the change is now far easier. Going back to the flame graph above, the actual cost profile differs dramatically as the size of the data rose, even if the profiler output remained the same. Refactoring the code allowed us to see where the costs were and address them effectively.Here is the end result as a flame graph. You can clearly see the preload section that takes a significant portion of the time. The key here is that the change allowed us to address this cost directly and in an optimal manner.The end result for our benchmark was:Before: 3 minutes, 6 secondsAfter: 2 minutes, 4 secondsSo almost exactly ⅓ of the cost was removed because of the different approach we took, which is quite nice. This technique, refactoring the code to make the costs obvious, is a really powerful one. Mostly because it is likely the first step to take anyway in many performance optimizations (batching, concurrency, etc.).

by Kathleen Dollard

posted on: May 08, 2025

C# 14 introduces extension members. See how the `extension` syntax offers flexibility for extension authors and continuity for developers using extensions

by Oren Eini

posted on: May 07, 2025

Join Our Community Discussion: Exploring the Power of AI Search in Modern ApplicationsWe're excited to announce our second Community Open Discussion, focusing on a transformative feature in today's applications: AI search.This technology is rapidly becoming the new standard for delivering intelligent and intuitive search experiences.Join Dejan from our DevRel team for an open and engaging discussion.Whether you're eager to learn, contribute your insights, or simply listen in, everyone is welcome!We’ll talk about:The growing popularity and importance of AI search.A deep dive into the technical aspects, including embeddings generation, query term caching, and quantization techniques.An open forum to discuss best practices and various approaches to implementing AI search.A live showcase demonstrating how RavenDB AI Integration allows you to implement AI Search in just 5 minutes, with the same simplicity as our regular search API.Event Details:Date: Wednesday, May 7th, 19:00 CETLocation:RavenDB Developers Community DiscordBring your questions and your enthusiasm – we look forward to seeing you there!

by Sam Cox

posted on: May 07, 2025

Tracebit built their B2B SaaS security product using C#.

by Andrew Lock

posted on: May 06, 2025

In this post I explore the new .NET AI Chat Web App template (currently in preview) and take a brief look at the implementation it provides…

by Oren Eini

posted on: May 05, 2025

Orleans is a distributed computing framework for .NET. It allows you to build distributed systems with ease, taking upon itself all the state management, persistence, distribution, and concurrency.The core aspect in Orleans is the notion of a “grain” - a lightweight unit of computation & state. You can read more about it in Microsoft’s documentation, but I assume that if you are reading this post, you are already at least somewhat familiar with it.We now support using RavenDB as the backing store for grain persistence, reminders, and clustering. You can read the official announcement about the release here, and the docs covering how to use RavenDB & Microsoft Orleans.You can use RavenDB to persist and retrieve Orleans grain states, store Orleans timers and reminders, as well as manage Orleans cluster membership. RavenDB is well suited for this task because of its asynchronous nature, schema-less design, and the ability to automatically adjust itself to different loads on the fly.If you are using Orleans, or even just considering it, give it a spin with RavenDB. We would love your feedback.