Introduction

In many database builds, backups pose a very large problem. Most backup systems require an exclusive table lock and don’t have any support for incremental backups; they require a full backup every time. When database sizes grow to several terabytes, this becomes a huge problem. The normal solution to this is to rely on snapshots. In the cloud this is quite easy, since the cloud platform can take snapshots while still guaranteeing a certain level of performance. In the datacenter, few good solutions exist. One method frequently used is utilizing LVM on Linux to perform the snapshot at the block device layer.

LVM snapshots

LVM is a Linux technology that allows for advanced block device manipulation including splitting block devices into many smaller ones, and combining smaller block devices into larger ones through either concatenation or striping methods, which include redundant striping commonly referred to as RAID. In addition to this, it also supports a copy on write (CoW) feature that allows for snapshots. The method used to implement this is to allocate a section of the underlying physical volumes that the original data is copied to before updating the main logical volume.

BTRFS

According to the Btrfs kernel wiki: “Btrfs is a modern copy-on-write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration.” It is an inherently CoW filesystem, which means it supports snapshotting at the filesystem level in addition to many more advanced features.

Experiment 1: a simple benchmark

The hypothesis

Since both LVM with snapshotting and Btrfs are CoW, it would stand to reason that the solution providing the features at a higher layer will be more performant and provide more flexibility compared to one at a lower layer that has less information to work with for optimization. Because of this, Btrfs should perform better, or at least similarly, and provide more flexibility and simplify management.

The experiment

The experiment consisted of a custom-written script that would allocate a large block of data, pause to allow for a snapshot to be taken, then randomly update sections of the large block of data. A custom script was chosen because there are few benchmarks that allow for one to pause between initialization and testing stages. LVM had an EXT4 filesystem on top of it created using the following flags: -E lazy_itable_init=0,lazy_journal_init=0. Btrfs was created using the default options. The script is produced below:

This was tested on a dedicated server to remove as much abstraction that could lead to measurement errors as possible. It was also performed on a single spinning disk, meaning that the random seeking caused by CoW should be amplified compared to SSDs. In addition, fragmentation was collected via the filefrag utility both before the update test and after.

The results

The results are tabulated below:

Btrfs took slightly longer to do an initial create, which is expected since CoW was not in place at this time for LVM, meaning Btrfs has more overhead. For the core part of the hypothesis, Btrfs was 75% faster than LVM in an update workload, which aligns with the hypothesis. What is more surprising is that Btrfs was 342% faster than LVM on a single threaded linear read after the update test. This could be explained by Btrfs having more aggressive readahead policies than either EXT4 or LVM. Another surprising find was that after updating, Btrfs had 9.5% less fragmented extents than EXT4, which could explain part of the slowdown. If fragmentation was solely responsible for the slowdown, then using Ahmdal’s law, an operation on a fragmented extent would have to be on average 3,600% slower than on an unfragmented extent.

Experiment 2: a real world benchmark

With the success of the previous experiment, a more real world benchmark was warranted.

The hypothesis

The hypothesis is that the previous findings would be maintained with the more mainstream benchmarking tool.

The experiment

I chose blogbench as the test platform since it provides a good mix of both linear and random writes and reads.  I targeted 10GB of space being used, which equated to 136 iterations. Blogbench 1.1 was used for the benchmark. The following script was utilized to automate the testing process:

The results

The results are tabulated below:

In this test, Btrfs outperformed LVM in every benchmark. Higher scores are better. Btrfs was 107% faster in initial read scores and 24% faster in initial write scores. It was also 164% faster in post-snapshot reads and 17% faster in post-snapshot writes. This correlates with the previous experiment and the hypothesis. Another thing to note is that LVM post-snapshot suffered greatly from locking issues where for several iterations nothing happened, as shown in the below output:

Reasoning and final notes

Both of these benchmarks show that Btrfs outperforms LVM in terms of performance in the presence of snapshots. The reason for this is actually fairly intuitive and has to do with the method of implementing CoW in the systems. In LVM, CoW is achieved by first copying the block from the main logical volume to the snapshot logical volume, then updating the main logical volume. This operation requires one read and two writes to update a single block! Btrfs does this better by utilizing a log-structured data structure for writes, which means that an update requires only a single linear write. This explains why the initial create time in Experiment 1 was so similar overall: the overhead was not in CoW but in data checksuming and other features. It also explains why Btrfs was so much faster than LVM in CoW mode. Using a CoW system when one isn’t necessary leads to severe performance degradation, especially in database workloads. But if you will be implementing CoW anyway, it would stand to reason to use a CoW system that operates on the filesystem layer or higher. An example of a higher than filesystem form of CoW would be one that utilizes CoW in the database engine to create snapshots. A sort of named, persistent transaction that can be referenced.

Further work

I would like to perform a similar benchmark on MySQL. Initial work was done on this, but due to time limitations, I could not complete a benchmark using SysBench and MySQL for this. It would be interesting to see the results from a real database, which traditionally has seen terrible performance on top of a CoW filesystem.