by Oren Eini

posted on: February 07, 2022

My talk at the Carnegie Mellon Database Group about the internals of Voron and how we build a transactional storage engine.

by Gérald Barré

posted on: February 07, 2022

HttpClient allows to send HTTP requests. HttpClient is intended to be instantiated once and re-used throughout the life of an application. Indeed, the HttpClient has a connection pool to reuse connections and reduce the number of TCP connections. So, if you send multiple requests to the same host,

by Andrew Lock

posted on: February 01, 2022

Creating a source generator - Part 8

by Gérald Barré

posted on: January 31, 2022

If you have read my introduction to xUnit.NET, you know that tests are executed sequentially inside a collection. By default, a collection is created per each class. So, all tests in a class are executed sequentially. You may want to execute all tests in parallel to reduce the execution time. A wor

by Oren Eini

posted on: January 28, 2022

Next week I’ll be talking at CMU about how to build storage engines and what you need to consider when building them. The talk is open to the public (don’t have to be at CMU to be there).Here are the details:Event Date: Monday January 31, 2022Event Time: 04:30pm ESTLocation: https://cmu.zoom.us/j/95002789605?pwd=eEtJRnRaNnQ1bFZmTnUwbDRMaXBRQT09In this talk, Oren Eini, founder of RavenDB, will discuss the design decisons and the manner in which RavenDB deals with storing data on disk.Achieving highly concurrent and transactional system can be a challenging task. RavenDB solves this issue using a storage engine called Voron. We’ll go over the design of Voron and how it is able to achieve both high performance and maintain ACID integrity.This talk is part of the Vaccination Database (Booster) Tech Talk Seminar Series.Zoom Link: https://cmu.zoom.us/j/95002789605 (Passcode 982149)

by Ardalis

posted on: January 25, 2022

When designing web APIs, it's important to think about how the data being passed to and from the endpoint will be structured. How important…Keep Reading →

by Andrew Lock

posted on: January 25, 2022

Creating a source generator - Part 7

by Oren Eini

posted on: January 24, 2022

Databases will typically divide their data files into pages, and manage all data in pages. As you can imagine, the notion of pages is pretty important to a database. Up to this point, we worked with pages, but not really. A page was just a buffer that we could read and write to. It took me 13 posts to manage to read and write pages from disk, but that is just the beginning. We need to add some structure for the pages. The usual way to do that is to have a page header, which contains some metadata about the page. That leads to an interesting question, how do we manage the page metadata? One way to do that is to reserve a part of the page for this purpose, like so:   This is probably the simplest option (and what we use in Voron), but I don’t like it very much. It makes certain operations harder to deal with, because the usable size of the page is no longer a power of 2. Assuming we have 8KB pages and a page header of 64 bytes, that means that we can use just 8,128 bytes in the page. A lot of data structures are far easier to use if we can assume that the size is a power of 2. Another thing to remember is that the underlying pager that we use operates on the level of 2 MB chunks. In other words, reading a value from the same 2MB range is as cheap as reading a value from the same 8KB page. For that reason, we can utilize the following approach, we’ll reserve the first section of the chunk for all the page headers. Here is what this will look like:   A chunk is 2MB in size, and pages are 8KB. That gives us 256 pages in a chunk. If we reserve the first page for page headers, that gives us 32 bytes for each page header. That is quite a lot, to be honest, which is great, because having more space on the header gives us a lot of options down the road. An important consideration to pay attention to is that we read from the disk at 2MB chunks, but when we write, the writes happen at the page boundary. A change on Page #3, for example, means that we have to change both the header page and then page #3 as separate operations. If we were using an embedded page header, we’ll have just a single write to deal with. In practice, I don’t think this is a major issue. We are doing asynchronous writes and write coalescing anyway. For that matter, writes are actually done lazily (and the critical path goes to the Write Ahead Log anyway) so I don’t think this aspect matters so much. At any rate, let’s get some idea about what the page header will look like: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub const PageTag = enum(u8) { none = 0, freeSpaceBitmap = 1, metadata = 2, multiPage = 3, }; pub const PageHeader = extern struct { tag: PageTag, reservedAlignment: [3]u8, data: extern union { fresSpaceBitmap: extern struct { freePages: u32 }, multiPage: extern struct { pagesCount: u32 }, reserved: [20]u8, }, checksum: [8]u8, pub fn getPageCount(self: *PageHeader) u32 { return switch (self.tag) { .none, .metadata => 1, .freeSpaceBitmap => pager.PagesInFile / @bitSizeOf(u8) / pager.PageSize, // 128 .multiPage => self.data.multiPage.pagesCount, }; } }; comptime { if (@sizeOf(PageHeader) != 32) { @compileError(std.fmt.comptimePrint("The PageHeader must be exactly 32 bytes in size but was {}", .{@sizeOf(PageHeader)})); } } view raw PageHeader.zig hosted with ❤ by GitHub I should note that this is merely a skeleton of the page header, just showing its structure. Over time, we’ll add a lot more details, but this is sufficient to understand the direction I’m going. A page header is denoted by a tag, which determines what type of page this is. Based on the type of the page, we can set various fields inside the header. You can also see the getPageCount(), which is a good demonstration of how we work with the page header. I need to have fine-grained control over the bytes in the header, so I’m setting them up in this manner. Zig currently has a number of bugs related to packed / extern structs and unions, which make this a bit harder than expected, sadly. I’m waiting for the self hosted compiler, which will hopefully fix those issues. With the PageHeader struct in place, we can start making use of this, like so: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub const Page = struct { page: u64, header: *PageHeader, buffer: []u8, }; fn getPage(p: *pager.Pager, loadedChunks: *pager.Pager.ChunksSet, page: u64, options: pager.GetPageOptions) !Page { const headerPage = (page / pager.PagesInChunk) * pager.PagesInChunk; const headerBytes = try p.getPage(loadedChunks, headerPage, 1, options); const pageIndexInChunk = page % pager.PagesInChunk; const headerEntries: []PageHeader = std.mem.bytesAsSlice(PageHeader, headerBytes); const pageHeader: *PageHeader = &headerEntries[pageIndexInChunk]; const pageCount = pageHeader.getPageCount(); const buffer = try p.getPage(loadedChunks, page, pageCount, options); return Page{ .page = page, .header = pageHeader, .buffer = buffer }; } view raw getPage.zig hosted with ❤ by GitHub There isn’t much going on here, which is great, but it does deserve some explanation. We first get the first page in the chunk for the page that was requested. For example, if we wanted to get page # 4, we get page #0, if we wanted to get page # 300, we’ll get page #256.  That page is the metadata page for the chunk, the only thing that it contains is the headers for all the pages in the chunk in question (including itself, of course). The reason that this matters is that the header for the page is used to compute the number of pages that we need to return. As you’ll note, the API we have here does not provide a way for the caller to say how many pages should be returned. We need to figure it out on our own. To do that, however, we need to store that information somewhere. That is where the page header comes into place. We get the page header of the page we care about from the metadata page of the chunk and compute the number of pages to get from the pager. Then we return to the caller the page header and the page buffer as a single unit. Of course, this presupposes that the values got to the pages somehow. That leads us to other directions, however, dealing with page allocations. You can already see some hints of that in the code above, where we use freeSpaceBitmap. I’ll dedicate my next post to this topic.

by Gérald Barré

posted on: January 24, 2022

async void methods are not a good way to define async methods. You should return a Task or ValueTask instead. The main point is to be able to await the method. But, is it possible to await an async void method? I don't say you should use async void in your code, I'm just questioning the point of no

by Oren Eini

posted on: January 21, 2022

In the last post, we figured out how we can find what pages we should flush to disk. In this one, I want to talk about how we can actually do the flushing. Once a page is marked as ready for flushing, we need to start an async process in which we write it to the disk. While that process is running, we need to ensure that there are no further writes happening on that page. We want to be able to handle flushing writes in the background with as little fuss as we possibly can, while at the same time maximizing parallelism. That turns out to be a significant challenge. Our previous write strategy simply gathered all the writes in memory and then flushed them to disk. Now we need to have an ongoing process. In addition, we also want to handle the scenario where we flush everything in memory for certain scenarios, such as checkpoints or orderly shutdown. In short, the approach we previously took isn’t going to be sufficient for our needs. For that reason, I want to clear the board and start from scratch. As a reminder, here is the API that we provide: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters { var writer = pager.acquireWriter(); defer writer.release(); writer.write(pageNum, pageBuffer); } view raw usage.zig hosted with ❤ by GitHub The rules for this API are fairly simple. You acquire the writer and then may call the write() method any number of times. The writer will aggregate the writes in memory, and will use write behind to write the data to disk in the background. In other words, we aren’t necessarily going to write to the disk when write() is called, only to our own buffer. Note that the writer is working using pages, not chunks, even though the buffer pool we use operates on chunks. In addition to the write() method, we have a flush() which pushes everything that is currently in the writer to the disk. That one is meant to be called when we want to create a checkpoint or when we do an orderly shutdown. The flush() call starts an asynchronous process that we can wait upon, so we aren’t going to stall on checkpoints.  While we write a page to disk, we’ll reject any writes to that page. This is important, because to do otherwise will mean that we may send messed up data to disk. We also need to handle any and all sequences of writes. For example, consider: write(2, [1 page]);  // write a single page #2 write(4, [4 pages]);// write 4 pages from page #4 (so pages: 4,5,6,7) Now, we may actually get additional writes as well: write(6, [3 pages]); write(4, [1 page]); In other words, there are no requirements from the caller about what should be called and in what order. The caller may write a set of pages and then repurpose them again. In this case, we expect that the end result is that we’ll issue the following writes to the disk: write(2, 1 page); write(4, 1 page); write(6, 3 pages); Note that we have page #5, it was written in the first call to page #4 (when it used 4 pages), however, with the update of page #4 to be a single page, the data in page #5 is no longer relevant and we can skip writing it to disk. This is an effect of working with pages. The data doesn’t make sense outside of a page boundary, after all, so we should only write meaningful data. After spending a lot of time thinking about it, I decided to go with the following behavior: The Pager’s writer will manage the write buffers, flushing to the disk when it is time. Callers can decide to let writes happen in the background or to wait for them. Here is the new state of the Writer: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub const Writer = struct { lock: std.Thread.Mutex, cache: Lru, parent: *Pager, inflight: collections.SortedList(*WriteState, sortWriteState), completed: ?*WriteState, backgroundWrites: std.atomic.Atomic(u32), }; view raw Writer.zig hosted with ❤ by GitHub There are a few interesting fields here: The lock is there to ensure that we have only a single writer. As a reminder, the actual I/O is asynchronous. The cache implements an Lru with a twist, I improved on that since the last post, more on this later. The parent field is the owning pager, obviously. The last three fields (inflight, completed and backgroundWrites) are all about handling the asynchronous writes for us. Let’s take a look at how we handle the state for a particular write and then we’ll be able to discuss the implementation in depth: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters const WriteState = struct { page: u64, count: u32, writer: *Writer, chunks: ChunksSet = .{}, err: ?anyerror = null, list: *Lru.List = undefined, next: ?*WriteState = null, }; view raw WriteState.zig hosted with ❤ by GitHub On the surface, the WriteState is pretty simple. It contains the page we are writing and how many pages are involved in a particular write, but there are also quite a few additional fields that are… strange. The writer field is used when we are returning from a callback, the chunks are used to record which chunks are relevant for this particular write (remember, the Writer operates at the page level, while the Pager operates at the chunk level). Because we deal with individual pages, we have to ensure that each one of them has its own reference for the chunk in question. That also has the effect of pinning any dirty pages in memory until they are flushed to disk. The err field holds whether there was an error (and what was it) when writing to the disk. The list field is used in conjunction with the write behind policy and will be discussed shortly. We use the next field to create a single linked list with the head being the completed field on the Writer. With that in mind, let’s look at how the write() method is implemented: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub fn write(self: *Writer, page: u64, buffer: []u8) !void { std.debug.assert(buffer.len % FileChunks.PageSize == 0); const allocator = self.parent.allocator; const count = @intCast(u32, buffer.len / FileChunks.PageSize); _ = try self.drainCompleted(); try self.ensureNotWrittenTo(page, count); if (self.cache.get(page)) |ws| { // can just update the buffer... if (ws.count == count) { var cachedBuffer = try self.parent.getPage(&ws.chunks, page, count, .{ .forWrites = true }); std.mem.copy(u8, cachedBuffer, buffer); return; } if (self.cache.remove(allocator, page)) |removed| { // previous was the wrong size, so can just discard try removed.deinit(self); } } { const state = try allocator.create(WriteState); errdefer allocator.destroy(state); state.* = .{ .writer = self, .page = page, .count = count }; var cachedBuffer = try self.parent.getPage(&state.chunks, page, count, .{ .forWrites = true }); std.mem.copy(u8, cachedBuffer, buffer); try self.cache.push(allocator, state); } while (self.cache.evictOverflow(allocator)) |ws| { try self.writePageToDisk(ws); } } view raw write.zig hosted with ❤ by GitHub We start by draining any completed writes. It is important to detect any errors and to release the references from the chunks we locked during the write portion. We treat any error during the write process as fatal and abort immediately. At this point, we don’t know what the state of the system is. The only safe way to handle this is to shutdown and run recovery. The next step is ensureNotWrittenTo() call. This is required because we may be in the process of writing a page to the disk, and we cannot modify the memory for that page (or pages) while they are being written to the disk. This is a recoverable error, by the way. The caller should wait until the pending writes are complete and retry. We then check in the cache if the page is already there. If it is, we have to consider the potential for a mismatch in the sizes (described earlier in this post), when we have a previous write that spanned three pages but the current one only applies to a single page. Since the page boundary is important, we can just discard the previous write and continue. For the common case of a write to a page that was previously written to, however, we can simply copy the data from the provided buffer to the right page in memory and move on. Most of the hot pages should fit into this code path. We then allocate the state for the current write, get the page (or pages) and copy the data from the provided buffer to our own memory. We also, crucially, push the new state into the writer’s cache. In the previous post that action would evict a page that we need to handle. In the current code, that is handled at the epilog of the method. We iterate over the cache and evict any pages that don’t fit into the cache any longer. This is because a single write may evict multiple pages. If we have space for 32KB, and we currently have 4 separate pages in the cache, we are full. Adding a write for 2 pages, however, will need to evict two items out of the cache. Another aspect that I changed from the last post is that I’m no longer requiring that the current page will not be evicted immediately. I mentioned that I changed the Lru’s implementation significantly.  Previously it was fairly generic, but it turns out that it is better to specialize it specifically for our needs. Here is the current version: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters const Lru = struct { const List = struct { list: collections.DoublyLinkedList(*WriteState) = .{}, used: usize = 0, max: usize, }; young: List, old: List, map: std.AutoHashMapUnmanaged(u64, *collections.DoublyLinkedList(*WriteState).Node) = .{}, }; view raw Lru.zig hosted with ❤ by GitHub This Lru is implemented using a generational approach. The idea is that we can specify the sizes of the young and old generations independently. A new page that is added to the cache is written to the young generation. If there is another write to the page, it will be moved to the old generation. If the size of the data in each generation is too big, we use a simple Least Recently Used to evict items. An item that expired on the old generation will be moved to the young generation, while an item that expires from the young generation will be evicted completely. Here is how this looks like in code: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub fn get(self: *Lru, page: u64) ?*WriteState { if (self.map.get(page)) |existing| { const item = existing.item; if (item.list != &self.old) { item.list.used -= item.count * FileChunks.PageSize; item.list.list.evict(existing); item.list = &self.old; self.old.used += item.count * FileChunks.PageSize; } self.old.list.moveFirst(existing); return existing.item; } return null; } pub fn push(self: *Lru, allocator: std.mem.Allocator, item: *WriteState) !void { std.debug.assert(self.map.get(item.page) == null); var node = try self.young.list.prepend(allocator, item); item.list = &self.young; self.young.used += item.count * FileChunks.PageSize; try self.map.put(allocator, item.page, node); } pub fn evictOverflow(self: *Lru, allocator: std.mem.Allocator) ?*WriteState { if (self.old.used > self.old.max) { if (self.old.list.evictLast()) |e| { self.young.list.moveFirst(e); self.young.used += e.item.count * FileChunks.PageSize; self.old.used -= e.item.count * FileChunks.PageSize; e.item.list = &self.young; } } if (self.young.used > self.young.max) { if (self.young.list.popLast(allocator)) |e| { self.young.used -= e.count * FileChunks.PageSize; return e; } } return null; } view raw Lru.imp.zig hosted with ❤ by GitHub This is the same rough algorithm used by MySQL. The advantage here is that there is O(1) cost for the entire process, but at the same time, busy pages actually have a good chance of not being written to the disk (which is ideal, since if they are busy, they keep changing). To give some context to the sizes involved, we may decide that the old generation contains 192 MB and the young generation is 64 MB. A page write will remain in the young generation until we have another 64 MB (8,192 individual page writes), if there were any writes to it again in that time frame, it will go to the old generation. Only if we didn't have any writes to it in that time frame will we flush it to disk. Once a page was moved to the old generation, it has to go through a buffer of 256 MB (32K page writes) before it will go to the disk. Hot pages should be safe from spurious writes, especially since we’ll write them to the disk anyway whenever we do a checkpoint. When a page is evicted from the cache, the writePageToDisk() method is called on it, there is a bit of work going on here: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters fn writePageToDisk(self: *Writer, ws: *WriteState) !void { // merge backward while (ws.page > 0 and try self.tryMergePages(ws, ws.page - 1)) {} // merge forward while (try self.tryMergePages(ws, ws.page + ws.count)) {} try self.inflight.add(self.parent.allocator, ws); var pageBuffer = try self.parent.getPage(&ws.chunks, ws.page, ws.count, .{}); _ = self.backgroundWrites.fetchAdd(1, .Release); var fileIdx = ws.page / FileChunks.PagesInFile; var file = self.parent.files[fileIdx] orelse return error.FileIsNotLoaded; self.parent.ring.submit(.{ .tag = .Write, .fd = file.fd, .buffer = pageBuffer, .offset = (ws.page % FileChunks.PagesInFile), .context = @ptrToInt(ws), .callback = completeFlushCallback, }) catch |e| { // we aren't removing the inflight record, an error here is probably // fatal, and we already merged pages, so we can't revert the state self.backgroundWrites.fetchSub(1, .Release); return e; }; } fn tryMergePages(self: *Writer, cur: *WriteState, page: u64) !bool { var curFileIdx = cur.page / FileChunks.PagesInFile; var fileIdx = page / FileChunks.PagesInFile; if (curFileIdx != fileIdx) return false; if (self.cache.remove(self.parent.allocator, page)) |p| { cur.page = std.math.min(cur.page, page); cur.count += p.count; // move chunk ownership var it = p.chunks.keyIterator(); while (it.next()) |c| { try cur.chunks.put(self.parent.allocator, c.*, {}); } try p.deinit(self); return true; } return false; } view raw writePageToDisk.zig hosted with ❤ by GitHub An important observation about writing to the disk is that coalescing writes can have a huge impact on performance. Instead of having to go to the disk multiple times, we can schlepp the data there just once. The amount of the data we write is actually far less significant than what you may expect. You can see that the first thing that we do is to try to merge the pages (both forward and backward). If we find a candidate, we expand our own WriteState and release the one that subsumed. This is an important aspect, because the writes we send to the disk don’t necessarily match the old / young generations division. If we evicted a page that is adjacent to another modified page, we’ll write them both to the disk. That is even if the second page is in the old generation (and that will evict the page from the write cache). In practice, databases exhibit a lot of locality, so I don’t expect that to be an actual issue. The last thing we do before actually submitting the I/O operation is to register the write in the inflight record. This is basically a sorted list (by the first page for that write) which we’ll check on any future writes. We also increment the number of background writes we have so we’ll know to wait for them to complete. Note that if we fail to submit the I/O, we decrement the number of backgroundWrites, we use this value to wait for async operations (for example, during shutdown). And since such errors are probably fatal, we are heading that way soon. The actual I/O happens in the background and we’ll call the callback function when that is done (either successfully or with an error). So far, all the code we saw was single threaded and protected by the lock mutex. The callback, on the other hand, is running without any such protections. Furthermore, the callback thread is running all the notifications we have. We want to do as little as possible there. Here is what I ended up with: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters fn completeFlushCallback(work: *PagerRing.Work) void { // this is called from background thread, without the lock! // we need to do minimal amount of work here var ws = @intToPtr(*WriteState, work.context); var self = ws.writer; while (true) { ws.next = @atomicLoad(?*WriteState, &self.completed, .Acquire); const result = @cmpxchgWeak(?*WriteState, &self.completed, ws.next, ws, .AcqRel, .Acquire); if (result == null) { break; } // failed to update, need to retry } // now can wake any waiters... _ = self.backgroundWrites.fetchSub(1, .Release); std.Thread.Futex.wake(&self.backgroundWrites, std.math.maxInt(u32)); } view raw completeFlushCallback.zig hosted with ❤ by GitHub We register the completed write in the completed linked list and wake any pending waiters. Any actual processing, of course, is moved to the part of the process where we actually run under lock and don’t have to worry about concurrency and multi threading. The actual behavior around completing writes is implemented in the drainCompleted() call that is part of the write() call. This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters fn drainCompleted(self: *Writer) !bool { var hasAny = false; while (@atomicLoad(?*WriteState, &self.completed, .Acquire)) |done| { if (@cmpxchgWeak(?*WriteState, &self.completed, done, done.next, .AcqRel, .Acquire)) |_| { // failed to update, need to retry continue; } hasAny = true; _ = self.inflight.remove(done); if (done.err) |err| { done.deinit(self) catch { // return the original error, if it happened }; return err; // we abort on the first error } try done.deinit(self); } return hasAny; } view raw drainCompleted.zig hosted with ❤ by GitHub Even though we are under lock, the I/O completion may compete with us on the completed list, so we’re using lock free methods to iterate through the list of completed actions. The actual behavior is straightforward, we remove the write from the list of inflight writes and return an error if the write had a problem. Note that we have to deal with potentially nested errors as well.  Aside from calling this as part of the write call(), we may also want to explicitly wait for all the pending operations to complete. This is done using the waitForWrites() call: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub fn waitForWrites(self: *Writer, timeout: ?u64) !bool { var hasAny = false; while (true) { if (try self.drainCompleted()) { hasAny = true; } var pending = self.backgroundWrites.load(.Acquire); if (pending == 0) return hasAny; try std.Thread.Futex.wait(&self.backgroundWrites, pending, timeout); } } view raw waitForWrites.zig hosted with ❤ by GitHub We drain all the completed writes and then wait for any pending ones to complete as well. We repeat this until there are no more backgroundWrites. Note that we run this under the lock, so we know that there can never be additional writes happening. In special circumstances, we may want to ask the writer to push everything to the disk, we do that by calling the checkpoint() method: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub fn checkpoint(self: *Writer, timeout: ?u64) !void { // first, clear all pending writes to disk... while (self.cache.evictYoungest()) |ws| { try self.writePageToDisk(ws); } // now wait for them to complete _ = try self.waitForWrites(timeout); } view raw checkpoint.zig hosted with ❤ by GitHub There is almost nothing there, I’m happy to say, we are almost done. We evict pages from the cache from the youngest to the oldest, using the same write coalescing as before. Given write locality, that is likely to produce good results. We complete the process by waiting for all those writes to complete, and we are done. There is just one last topic that I want to cover: Shutting  down: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters pub fn deinit(self: *Writer, allocator: std.mem.Allocator) void { while (true) { self.waitForWrites(null) catch { // if we are here, the waitForWrites returned an error // however, we still need to drain the whole queue of async // operations, otherwise, we may have a memory leak. // an error at this stage on the write isn't something that we // *can* handle, and we are tearing down the writer anyway continue; }; break; } // we just discard the pending writes at this point, caller should either // call checkpoint() or will run recovery on startup while (self.cache.evictYoungest(allocator)) |e| { e.deinit(self) catch { // nothing that we can do about this }; } self.cache.deinit(allocator); self.inflight.deinit(allocator); } view raw deinit.zig hosted with ❤ by GitHub Shutting down turns out to be a fairly involved process. This is especially the case if we are shutting down because of an error. We need to wait until all the pending writes have been completed (to do otherwise is to invite Use-After-Free bugs). That is why we call the waitForWrites() until it completes successfully. At worst, each time this is called will process a single write. On shutdown, we can’t really start writing to the disk (after all, we may be shutting down because of the disk). We just discard the data in this case. That is something that we are okay to do, because we assume that any important data can be restored when we run recovery. This is one of those cases where the fact that we are building a database makes our life easier. We don’t need to try very hard to persist to disk, we can assume that as a given. I’m pretty happy with how all this turned out. The full code for the pager is now at around 1,500 lines of code. I think that at this point we are pretty much at or near what you’ll need to replace the mmap() based pager that I demonstrated earlier in this series. This is the sort of code that needs a metric ton of tests, of course, and a lot of actual experience throwing stuff at it. The nice thing about this approach is that this is entirely in our hands. We can now implement whatever policies we want. The actual behavior is more or less done, but we can play with the exact mechanics and what will trigger what at will. In some cases, that can give us a great deal of power and flexibility.