skip to content
Relatively General .NET

Recording

by Oren Eini

posted on: April 11, 2022

Last week I did a webinar about Clean Architecture, and it run about twice as long as I expected it to be. Mostly because I got some really interesting questions and I think we had a great discussion.You can watch all of it here, as usual, comments are very welcome:

Uploading multiple files using InputFile in Blazor

by Gérald Barré

posted on: April 11, 2022

Blazor provides a component to upload files. You can use the <InputFile> component and its OnChange event to get the selected files. It works well for single-file uploads. But, if the user selects new files while the first files are still uploading, you may get an error. You can reproduce thi

Surprising behavior when roundtripping JSON documents

by Oren Eini

posted on: April 08, 2022

We run into a really interesting bug. For some reason, the system was behaving in a totally unexpected manner for some parts of the data. For pretty much the same input, we would get the wrong result, and we couldn’t figure out why. Here is our source data: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters { "Server": "PRD-392", "CpuLoad": { "2": "94%", "1": "49%", "0": "32%", "3": "10%" } } view raw metrics.json hosted with ❤ by GitHub This is some metrics data about servers, and you’ll note that we report the CPU load for each core on the instance and that the results are sorted based on the actual load. Here is what this looks like, the image on the left is wrong while the image on the right is right (pun intended). Why do we have this behavior? Well, let’s look at the actual data, shall we? They are… the same. Exactly the same, in fact. We can throw that into diff engine and they will tell me that they are identical (except for the document id). What is going on here? Well, here is the issue, what you see is not what you get. Look at the JSON text that I have above, and compare that to the documents we see in the images. RavenDB shows the documents in a nicely formatted manner, and along the way, it messed up something pretty important. In our case, we used an object to hold the various details about the instances. And we relied that the insertion sort order for the properties would stay the same when reading the document. That is actually the case, and RavenDB goes to great lengths to ensure that this is the case. However… In order to prettify the document, we call to JSON.parse() and JSON.stringify() (on the client side), which give us nicely formatted output. Along the way, however, we run into JavaScript and its “ideas” about how things should work. In particular, JavaScript threats properties whose key is a number in a different way than other values. All the numeric properties will be sorted according to their integer value, while non numeric values will be sorted using insertion order. That only applies to documents that were modified in the studio, however. The RavenDB server and client API are keeping the properties in insertion order. Only if you modified the document using the Studio will you get this. But because we always show the documents in the same manner, it was invisible to us for a long while. For that matter, it took an embarrassingly long time of debugging this problem, because (naturally) whenever we viewed the data, we did that with formatting, which meant that we never actually saw the differences between the raw versions of the documents.

Answer

by Oren Eini

posted on: April 07, 2022

I asked why this code is broken, and now is the time to dig into this. The issue is in this block of code. Take a look at that for a moment, if you please: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters tasks.Add(Task.Factory.StartNew(async () => { using var data = File.OpenRead(file); var url = "/file?=" + Uri.EscapeDataString(Path.GetFileName(file)); await httpClient.PostAsync(url, new StreamContent(data)); })); view raw problem.cs hosted with ❤ by GitHub The code is trying to gather the async upload of all the files, and then it will await them. This code compile and runs successfully, but it will not do what you expect it to do. Let’s break it up a bit to understand what is going on: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters var task = Task.Factory.StartNew(async () => { // redacted }) tasks.Add(task); view raw one.cs hosted with ❤ by GitHub We are passing the variable task to the list of tasks we have. We just extract a variable, nothing much going on here. Let’s explore further, what is the type of task? We know it must be a subtype of Task, since that is what the tasks collection will take. It turns out that this isn’t that simple: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters Task<Task> task = Task.Factory.StartNew(async () => { // redacted }) tasks.Add(task); view raw two.cs hosted with ❤ by GitHub What is that, Task<Task> thingie? Well, let’s look at the signature of Task.Factory.StartNew(), shall we? public Task<TResult> StartNew<TResult>(Func<TResult> function); Just from the signature, we can tell what is going on. StartNew() accepts a function that returns a value and will then return a task for the eventual result of this function. However, the function we are actually passing to the StartNew() doesn’t produce a value. Except that it does… Let’s explore that thing for a bit: var func = async () => { }; What is the type of func in this case? Func<Task> func = async() => {}; The idea is that when the compiler sees the async keyword, it transforms the function to one that returns a Task. Combining both those features together means that our original code actually registers the start of the asynchronous process to happen and will return as soon as it got started. Basically, we’ll only wait for the actual opening of the file, not for the actual network work that has to happen here. The right way to express what we want here is: Task.Run(async () => {}); The signature for this is: public static Task Run<TResult>(Func<Task> function); You can see here that we get a function that returns a task, but we aren’t actually wrapping that in another Task instance. The task that will be returned will be completed once the full work has been completed. It is an interesting pitfall in the API, and can be quite hard to figure out exactly what is going on. Mostly because there are several different things happening all at once.

Challenge

by Oren Eini

posted on: April 06, 2022

The following code looks straightforward, but it has a really subtle issue. This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters public async Task SendFiles(params string[] files) { var httpClient = new HttpClient { BaseAddress = new Uri("https://my.backup.server") }; var tasks = new List<Task>(); foreach (var f in files) { var file = f; tasks.Add(Task.Factory.StartNew(async () => { using var data = File.OpenRead(file); var url = "/file?=" + Uri.EscapeDataString(Path.GetFileName(file)); await httpClient.PostAsync(url, new StreamContent(data)); })); } await Task.WhenAll(tasks); Console.WriteLine("All you data has been backed up!"); } view raw subtle.cs hosted with ❤ by GitHub Can you spot what is going on?You can ignore the error handling here, by the way, the issue isn’t related to handling unexpected errors.

On fixing a bug (and all its siblings) with a forward looking view

by Oren Eini

posted on: April 05, 2022

We run into a strange situation deep in the guts of RavenDB. A cluster command (the backbone of how RavenDB is coordinating action in a distributed cluster) failed because of an allocation failure. That is something that we are ready for, since RavenDB is a robust system that handles such memory allocation failures. The problem was that this was a persistent allocation failure. Looking at the actual error explained what was going on. We allocate memory in units that are powers of two, and we had an allocation request that would overflow a 32 bits integer. Let me reiterate that, we have a single cluster command that would need more memory than can fit in 32 bits. A cluster command isn’t strictly limited, but a 1MB cluster command is huge, as far as we are concerned. Seeing something that exceeds the GB mark was horrifying. The actual issue here was somewhere completely different, there was a bug that caused quadratic growth in the size of a database record. This post isn’t about that problem, it is about the fix. We believe in defense in depth for such issues. So aside from fixing the actual cause for this problem, the issue was how we can prevent similar issues in the future. We decided that we’ll place a reasonable size limit on the cluster commands, and we chose 128MB as the limit (this is far higher than any expected value, mind). We chose that value since it is both big enough to be outside anyone's actual usage, but at the same time, it is small enough that we can increase this if we need to. That means that this needs to be a configuration value, so the user can modify that in place if needed. The idea is that we’ll stop the generation of a command of this size, before it hits the actual cluster and poison it. Which brings me to this piece of code, which was the reason for this blog post: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters private void ThrowTooLargeRaftCommand(BlittableJsonReaderObject cmd) { var type = RachisLogHistory.GetTypeFromCommand(cmd); throw new ArgumentOutOfRangeException( $"The command '{type}' size of {new Size(cmd.Size, SizeUnit.Bytes)} exceed the max allowed size."); } view raw impr.cs hosted with ❤ by GitHub This is where we are actually throwing the error if we found a command that is too big (the check is done by the caller, not important here). Looking at the code, it does what is needed, but it is missing a couple of really important features: We mention the size of the command, but not the actual size limit. We don’t mention that this isn’t a hard coded limit. The fix here would be to include both those details in the message. The idea is that the user will not only be informed about what the problem is, but also be made aware of how they can fix it themselves. No need to contact support (and if support is called, we can tell right away what is going on). This idea, the notion that we should be quite explicit about not only what the problem is but also how to fix it, is very important to the overall design of RavenDB. It allows us to produce software that is self supporting, instead of ErrorCode: 413, you get not only the full details, but how you can fix it. Admittedly, I fully expect to never ever hear about this issue again in my lifetime. But in case I’m wrong, we’ll be in a much better position to respond to it.

Learning by Consumption and Creation

by Ardalis

posted on: April 05, 2022

Preamble After a conversation on our devBetter Discord server, I published a short twitter thread about learning. It resonated a bit and as…Keep Reading →