J@ArangoDB

{ "subject" : "ArangoDB", "tags": [ "multi-model", "nosql", "database" ] }

Benchmarking ArangoDB's Write-ahead Log

Motivation

One of the major changes in ArangoDB 2.2 was the introduction of the write-ahead log (abbreviated WAL).

The introduction of the WAL changed how documents are stored internally in ArangoDB. A lot of things have been changed for it under the hood, and it has been a lot of work to implement it.

During the implementation, we refactored some code parts and made them considerably faster. From these changes we expected a positive effect on the database performance. But due to the fact that shape information is now also saved in the write-ahead log, there may also be some negative effect.

We developers were of course very interested in seeing the net effects, so we ran some tests for a few use cases. We compared ArangoDB 2.1.2 (still without the WAL) with ArangoDB 2.2.1 (with the WAL). The results are interesting.

Test setup

To get a broad overview of performance changes, we ran a few different test cases:

  • document: inserts a document
  • crud: inserts a document, fetches it, updates it, and deletes it
  • crud-append: inserts a document, fetches it, updates it, and fetches it again
  • multi-collection: transactionally save two documents in two collections
  • random-shapes: save documents with completely different structures/shapes

All tests were run with the arangob benchmark tool, with various concurrency levels, complexity settings and repeated several times. arangob was invoked like this:

1
2
3
4
5

arangob                         \
  --test-case $case             \
  --request $requests           \
  --concurrency $concurrency    \
  --complexity $complexity

Both ArangoDB servers and arangob were located on the same server. Only one ArangoDB server was active during each test run, and the other was shut down so it didn’t compete for system resources.

Test results

Following are the results for the different test cases.

In the result tables, the columns have the following meanings:

  • Complexity: the number of attributes for the test documents, used as parameter --complexity for arangob
  • Requests: the number of operations executed, used as parameter --requests for arangob
  • Concurrency: the number of client threads started by arangob, used a parameter concurrency for arangob
  • Time 2.1: test execution time (in seconds) for ArangoDB 2.1
  • Time 2.2: test execution time (in seconds) for ArangoDB 2.2
  • T2.1/T2.2: relative performance of ArangoDB 2.1 compared to ArangoDB 2.2. Values less than one indicate that ArangoDB 2.1 was faster than ArangoDB 2.2. Values greater than one indicate that ArangoDB 2.1 was slower than ArangoDB 2.2. A value of one means that both versions had equal speed.

Please note that the absolute execution times aren’t too interesting in the results shown here. We haven’t used the most powerful server on earth for running these tests. We were most interested in how ArangoDB 2.2 compared to ArangoDB 2.1.

Inserting documents

The document test appends new documents (with an identical structure) to a collection. Here are the results for inserting a million documents with 100 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
       100       1000000               1       97.452       94.839         1.02    
       100       1000000               2       61.146       53.928         1.13   
       100       1000000               4       54.368       31.379         1.73

Following are the results for 100,000 documents with 1,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
      1000        100000               1       86.883       80.669         1.07 
      1000        100000               2       59.381       48.599         1.22 
      1000        100000               4       53.784       27.494         1.95

The results for 5,000 documents with 10,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
     10000          5000               1       43.991       40.577         1.08 
     10000          5000               2       31.403       26.117         1.20 
     10000          5000               4       28.713       20.044         1.43

As we can see in the results above, ArangoDB 2.2 is faster than ArangoDB 2.1 for all tested configurations. The difference is negligible if the test client is single-threaded (one insert thread), but ArangoDB 2.2 is considerably faster than ArangoDB 2.1 with more concurrent clients.

CRUD operations (I)

The crud test case inserts a document, fetches it, updates it, fetches it again and finally deletes it. Executing 1 million crud operations on documents with 100 attributes each results in the following execution times:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
       100       1000000               1       66.856       67.072         0.99       
       100       1000000               2       47.907       47.043         1.01        
       100       1000000               4       42.089       42.047         1.00

Running 100,000 crud operations on documents with 1,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
      1000        100000               1      160.228      158.312         1.01 
      1000        100000               2      147.413      147.990         0.99  
      1000        100000               4      143.575      143.233         1.00

And here are the test results for running 5,000 crud operations on documents with 10,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
     10000          5000               1      645.859      643.924         1.00        
     10000          5000               2      640.414      640.767         0.99       
     10000          5000               4      637.438      637.132         1.00

The results of all these tests show that performance for the tested workload hasn’t changed between ArangoDB 2.1 and 2.2. This is somewhat expected, as the WAL should not affect the performance of read and delete operations much: reading a document does not require writing to the WAL at all, and removing a document only requires writing a very small remove marker to the WAL.

CRUD operations (II)

The crud-append test case inserts a document, fetches it, updates it, and fetches it again. Executing 1 million crud operations on documents with 100 attributes each results in the following execution times:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
       100       1000000               1       81.187       79.079         1.02 
       100       1000000               2       59.544       56.934         1.04  
       100       1000000               4       53.845       52.098         1.03

Running 100,000 crud operations on documents with 1,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
      1000        100000               1      200.057      196.722         1.01 
      1000        100000               2      182.436      181.633         1.00 
      1000        100000               4      181.233      180.631         1.00

And here are the test results for running 5,000 crud operations on documents with 10,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
     10000          5000               1      804.781      801.903         1.00 
     10000          5000               2      796.589      797.693         0.99  
     10000          5000               4      796.664      795.823         1.00

Again, ArangoDB 2.1 and 2.2 are pretty much the same speed for the tested operations.

Multi-collection write transactions

The multi-collection test stores two documents in two different collections transactionally. Here are the results for executing 100,000 transactions for documents with 100 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
       100        100000               1     6936.895       30.736       225.69   
       100        100000               2     6946.524       25.577       271.59    
       100        100000               4     7720.682       24.334       317.27

Executing 10,000 transactions on documents with 1,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
      1000         10000               1      949.974       21.368        44.45     
      1000         10000               2      953.672       19.322        49.35    
      1000         10000               4      941.645       20.486        45.96

And finally, the results for executing 500 transactions on documents with 10,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
     10000           500               1       46.776       10.413         4.49 
     10000           500               2       45.102       10.173         4.43 
     10000           500               4       44.172        9.192         4.80

The above results show that ArangoDB 2.2 is much faster than ArangoDB 2.1 for executing transactions that write to two collections.

We expected that! Multi-collection (write) transactions in ArangoDB 2.2 require far less calls to msync than in ArangoDB 2.1. ArangoDB 2.2 can sync operations of multiple transactions together in one call to msync. ArangoDB 2.1 needed to synchronize each transaction separately.

(Fully) heterogenous documents

The random-shapes test inserts documents that have different structures each. For each inserted document a new shape will need to be stored. Inserting one million documents with 100 attributes each results in the following figures:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
       100       1000000               1      664.006       87.181         7.61        
       100       1000000               2      433.570       60.613         7.15       
       100       1000000               4      313.666       45.327         6.92

The results show that ArangoDB 2.2 is considerably faster than ArangoDB 2.1 for these cases – even with the storage overhead of writing the shapes to the WAL.

Now we inserted 100,000 documents with 1,000 document attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
      1000        100000              1        61.709       66.954         0.92    
      1000        100000              2        39.294       46.458         0.84   
      1000        100000              4        36.866       40.910         0.90

In these cases, the storage overhead of writing all shapes to the WAL seems to start to matter, and ArangoDB 2.2 gets slower than ArangoDB 2.1 in this test case.

The same was true when we inserted 5,000 documents with 10,000 attributes each:

1
2
3
4
5

Complexity      Requests     Concurrency     Time 2.1     Time 2.2    T2.1/T2.2
-------------------------------------------------------------------------------
     10000          5000               1       25.337       30.412         0.83    
     10000          5000               2       20.021       22.128         0.90     
     10000          5000               4       14.800       18.872         0.78

Note that the random-shapes test case is an extreme test case. We do not consider it realistic that all documents in a collection have completely different attribute names. Still we included it in our tests because we were sure that this would be a case in which the overhead of the WAL would be clearly measurable.

Note that there is a way to make ArangoDB 2.2 faster than ArangoDB 2.1 even for this test case: setting the option --wal.suppress-shape-information to true will make ArangoDB not write shape information to the WAL, making document write operations much faster in case all documents have heterogenous structures.

The option won’t help much if a shapes repeat a lot. In this case, the WAL overhead shouldn’t matter too much already, or ArangoDB 2.2 should already be faster than 2.1 (as shown in the results above).

Summary

The tests revealed that the overhead of the WAL in ArangoDB 2.2 seems to be negligible for most of the tested workloads. In many cases, ArangoDB 2.2 with the WAL is even faster than ArangoDB 2.1 that did not have a WAL at all.

There is one notable exception: when documents have fully heterogenous structures, the overhead of the WAL is measurable and significant. Though we consider this to be a rather hypothetical case, we have added the configurtion option --wal.suppress-shape-information. This can be used to turn off storing shape information in the WAL. This is not safe when the server is to be used as a replication master, but should work in cases when replication is not used.

The option should not have a big effect if documents are greatly or at least somewhat homogenous. For these cases, ArangoDB 2.2 with its WAL should already be as fast as 2.1 (or even faster).