A while ago I wrote a blog post about handling binary data in Foxx applications. That blog post provided a solution for retrieving and serving non-UTF-8 data from a Foxx application. It was written for the 2.x series of ArangoDB.

With the release of ArangoDB 3.0 that solution needs some adaption, and this blog post provides the source code for the 3.0 version of the Foxx application.

The rawBodyBuffer() function of the request object is gone in 3.0, and to retrieve the body as a blob we now need to use req.rawBody. And as there are no controllers and applicationContexts in Foxx 3.0 anymore, the original route code needs to be changed from

1
2
3
4
5
6
7
8

controller.post('/receive-binary', function (req, res) {
  // fetch request body into the buffer
  var body = req.rawBodyBuffer();
  // create an absolute filename, local to the Foxx application directory
  var filename = applicationContext.foxxFilename("body");

  require("fs").write(filename, body);
});

to the following code:

1
2
3
4
5
6
7
8

router.post('/receive-binary', function (req, res) {
  // fetch request body into a Buffer
  var body = req.rawBody;
  // create an absolute filename, local to the Foxx application directory
  var filename = module.context.filename("body");

  require("fs").write(filename, body);
});

To serve binary data from a Foxx action, the original route code needs to be changed from

1
2
3
4
5
6

controller.get('/provide-binary-file', function (req, res) {
  // create an absolute filename, local to the Foxx application directory
  var filename = applicationContext.foxxFilename("body");
  // send the contents, this will also set mime type "application/octet-stream"
  res.sendFile(filename);
});

1
2
3
4
5
6

router.get('/provide-binary-file', function (req, res) {
  // create an absolute filename, local to the Foxx application directory
  var filename = module.context.fileName("body");
  // send the contents, this will also set mime type "application/octet-stream"
  res.sendFile(filename);
});

The adjusted Foxx application can be downloaded here with full source code.

Here is the answer for some frequently used containers types containing uint64_t values (an 8 byte type). For the associative containers I also used uint64_t as the key type.

All results are for the STL implementation I had ready at hand (g++5.3 on Ubuntu 16.04, x86_64, libstdc++.so.6.0.22, with default allocator). The results may be completely different for other platforms and/or other data types.

The following containers are compared:

• std::vector<uint64_t>
• std::map<uint64, uint64_t>
• std::set<uint64_t>
• std::unordered_map<uint64, uint64_t>
• std::unordered_set<uint64_t>
• std::deque<uint64_t>

The containers themselves are creared on the stack. To store the elements, the containers will need to use heap memory. The number of elements in the containers (n) is increased exponentially from 0 (empty) to 512 in the tests. The memory usage pattern should be quite clear by then.

Reported are the amount of heap memory allocated by the container after all elements have been inserted, the total number of heap memory allocated by the container during its whole lifetime (i.e. the sum of all its mallocs), plus the number of calls to malloc and free the container issued.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108

n =     0
---------
vector         =>     0 bytes allocd at end,  total:     0 bytes mallocd,    0 malloc(s), 0 free(s)
map            =>     0 bytes allocd at end,  total:     0 bytes mallocd,    0 malloc(s), 0 free(s)
set            =>     0 bytes allocd at end,  total:     0 bytes mallocd,    0 malloc(s), 0 free(s)
unordered_map  =>     0 bytes allocd at end,  total:     0 bytes mallocd,    0 malloc(s), 0 free(s)
unordered_set  =>     0 bytes allocd at end,  total:     0 bytes mallocd,    0 malloc(s), 0 free(s)
deque          =>   576 bytes allocd at end,  total:   576 bytes mallocd,    2 malloc(s), 0 free(s)


n =     1
---------
vector         =>     8 bytes allocd at end,  total:     8 bytes mallocd,    1 malloc(s), 0 free(s)
map            =>    48 bytes allocd at end,  total:    48 bytes mallocd,    1 malloc(s), 0 free(s)
set            =>    40 bytes allocd at end,  total:    40 bytes mallocd,    1 malloc(s), 0 free(s)
unordered_map  =>    40 bytes allocd at end,  total:    40 bytes mallocd,    2 malloc(s), 0 free(s)
unordered_set  =>    32 bytes allocd at end,  total:    32 bytes mallocd,    2 malloc(s), 0 free(s)
deque          =>   576 bytes allocd at end,  total:   576 bytes mallocd,    2 malloc(s), 0 free(s)


n =     2
---------
vector         =>    16 bytes allocd at end,  total:    24 bytes mallocd,    2 malloc(s), 1 free(s)
map            =>    96 bytes allocd at end,  total:    96 bytes mallocd,    2 malloc(s), 0 free(s)
set            =>    80 bytes allocd at end,  total:    80 bytes mallocd,    2 malloc(s), 0 free(s)
unordered_map  =>    88 bytes allocd at end,  total:   104 bytes mallocd,    4 malloc(s), 1 free(s)
unordered_set  =>    72 bytes allocd at end,  total:    88 bytes mallocd,    4 malloc(s), 1 free(s)
deque          =>   576 bytes allocd at end,  total:   576 bytes mallocd,    2 malloc(s), 0 free(s)


n =     4
---------
vector         =>    32 bytes allocd at end,  total:    56 bytes mallocd,    3 malloc(s), 2 free(s)
map            =>   192 bytes allocd at end,  total:   192 bytes mallocd,    4 malloc(s), 0 free(s)
set            =>   160 bytes allocd at end,  total:   160 bytes mallocd,    4 malloc(s), 0 free(s)
unordered_map  =>   136 bytes allocd at end,  total:   152 bytes mallocd,    6 malloc(s), 1 free(s)
unordered_set  =>   104 bytes allocd at end,  total:   120 bytes mallocd,    6 malloc(s), 1 free(s)
deque          =>   576 bytes allocd at end,  total:   576 bytes mallocd,    2 malloc(s), 0 free(s)


n =     8
---------
vector         =>    64 bytes allocd at end,  total:   120 bytes mallocd,    4 malloc(s), 3 free(s)
map            =>   384 bytes allocd at end,  total:   384 bytes mallocd,    8 malloc(s), 0 free(s)
set            =>   320 bytes allocd at end,  total:   320 bytes mallocd,    8 malloc(s), 0 free(s)
unordered_map  =>   280 bytes allocd at end,  total:   336 bytes mallocd,   11 malloc(s), 2 free(s)
unordered_set  =>   216 bytes allocd at end,  total:   272 bytes mallocd,   11 malloc(s), 2 free(s)
deque          =>   576 bytes allocd at end,  total:   576 bytes mallocd,    2 malloc(s), 0 free(s)


n =    16
---------
vector         =>   128 bytes allocd at end,  total:   248 bytes mallocd,    5 malloc(s), 4 free(s)
map            =>   768 bytes allocd at end,  total:   768 bytes mallocd,   16 malloc(s), 0 free(s)
set            =>   640 bytes allocd at end,  total:   640 bytes mallocd,   16 malloc(s), 0 free(s)
unordered_map  =>   568 bytes allocd at end,  total:   712 bytes mallocd,   20 malloc(s), 3 free(s)
unordered_set  =>   440 bytes allocd at end,  total:   584 bytes mallocd,   20 malloc(s), 3 free(s)
deque          =>   576 bytes allocd at end,  total:   576 bytes mallocd,    2 malloc(s), 0 free(s)


n =    32
---------
vector         =>   256 bytes allocd at end,  total:   504 bytes mallocd,    6 malloc(s), 5 free(s)
map            =>  1536 bytes allocd at end,  total:  1536 bytes mallocd,   32 malloc(s), 0 free(s)
set            =>  1280 bytes allocd at end,  total:  1280 bytes mallocd,   32 malloc(s), 0 free(s)
unordered_map  =>  1144 bytes allocd at end,  total:  1472 bytes mallocd,   37 malloc(s), 4 free(s)
unordered_set  =>   888 bytes allocd at end,  total:  1216 bytes mallocd,   37 malloc(s), 4 free(s)
deque          =>   576 bytes allocd at end,  total:   576 bytes mallocd,    2 malloc(s), 0 free(s)


n =    64
---------
vector         =>   512 bytes allocd at end,  total:  1016 bytes mallocd,    7 malloc(s), 6 free(s)
map            =>  3072 bytes allocd at end,  total:  3072 bytes mallocd,   64 malloc(s), 0 free(s)
set            =>  2560 bytes allocd at end,  total:  2560 bytes mallocd,   64 malloc(s), 0 free(s)
unordered_map  =>  2312 bytes allocd at end,  total:  3016 bytes mallocd,   70 malloc(s), 5 free(s)
unordered_set  =>  1800 bytes allocd at end,  total:  2504 bytes mallocd,   70 malloc(s), 5 free(s)
deque          =>  1088 bytes allocd at end,  total:  1088 bytes mallocd,    3 malloc(s), 0 free(s)


n =   128
---------
vector         =>  1024 bytes allocd at end,  total:  2040 bytes mallocd,    8 malloc(s), 7 free(s)
map            =>  6144 bytes allocd at end,  total:  6144 bytes mallocd,  128 malloc(s), 0 free(s)
set            =>  5120 bytes allocd at end,  total:  5120 bytes mallocd,  128 malloc(s), 0 free(s)
unordered_map  =>  4664 bytes allocd at end,  total:  6144 bytes mallocd,  135 malloc(s), 6 free(s)
unordered_set  =>  3640 bytes allocd at end,  total:  5120 bytes mallocd,  135 malloc(s), 6 free(s)
deque          =>  1600 bytes allocd at end,  total:  1600 bytes mallocd,    4 malloc(s), 0 free(s)


n =   256
---------
vector         =>  2048 bytes allocd at end,  total:  4088 bytes mallocd,    9 malloc(s), 8 free(s)
map            => 12288 bytes allocd at end,  total: 12288 bytes mallocd,  256 malloc(s), 0 free(s)
set            => 10240 bytes allocd at end,  total: 10240 bytes mallocd,  256 malloc(s), 0 free(s)
unordered_map  =>  9416 bytes allocd at end,  total: 12488 bytes mallocd,  264 malloc(s), 7 free(s)
unordered_set  =>  7368 bytes allocd at end,  total: 10440 bytes mallocd,  264 malloc(s), 7 free(s)
deque          =>  2624 bytes allocd at end,  total:  2624 bytes mallocd,    6 malloc(s), 0 free(s)


n =   512
---------
vector         =>  4096 bytes allocd at end,  total:  8184 bytes mallocd,   10 malloc(s), 9 free(s)
map            => 24576 bytes allocd at end,  total: 24576 bytes mallocd,  512 malloc(s), 0 free(s)
set            => 20480 bytes allocd at end,  total: 20480 bytes mallocd,  512 malloc(s), 0 free(s)
unordered_map  => 18872 bytes allocd at end,  total: 25216 bytes mallocd,  521 malloc(s), 8 free(s)
unordered_set  => 14776 bytes allocd at end,  total: 21120 bytes mallocd,  521 malloc(s), 8 free(s)
deque          =>  4752 bytes allocd at end,  total:  4816 bytes mallocd,   11 malloc(s), 1 free(s)

I have intentionally not done any optimizations such as calling reserve() on the containers before inserting the individual elements, in order to reflect some real-word scenarios in which it is unclear how much elements will be added to the containers during program execution.

The test program for reproduction is here. Compile and run it with:

1
2

g++ -std=c++11 -O0 containers.cpp -o containers
for i in 0 1 2 4 8 16 32 64 128 256 512; do ./containers "$i"; done

There are three main reasons for compiling on your own:

• as a developer you want to make changes to the ArangoDB C++ source code. Then your only option obviously is to compile on your own. Please consult the compiling a debug version of ArangoDB page for more information.

• you are trying to get meaningful stack traces from core dumps produced by ArangoDB and need an ArangoDB binary that comes with enough debug information (debug symbols, probably also assertions turned on). In this case, please also consult the compiling a debug version of ArangoDB blog post for how to get this done.

• you want to use an ArangoDB binary that is optimized for your specific target architecture.

The latter reason is relevant because the official release packages that are provided by ArangoDB cannot make too many assumptions about the environment in which they will be used. In the general release packages there is not so much room for platform-specific optimizations as there would be if you are compiling just for the local machine.

For example, all relevant CPU offer SIMD instructions that a compiler can exploit when generating code. But different generations of CPUs offer different levels of SIMD instructions. Not every CPU in use today provides SSE4, not to talk about AVX.

To make our release packages compatible with most environments, we have had to make some conservative assumptions about the CPU abilities, which effectively disables many optimizations that would have been possible when creating a build that only needs to run on a specific architecture.

To fully exploit the capabilities of a specific target environment, it’s required to build executables for that specific architecture. Most compilers offer an option -march for that. You normally want to set this to native when compiling an optimized version. There are also lots of compiler options for enabling or disabling specific CPU features such as -msse, -msse2, -msse3, -mssse3, -msse4.1, -msse4.2, -msse4, -mavx, -mavx2, to name just a few.

The good news is that there is no need to deal with such compiler-specific optimization options in order to get an optimized build. The cmake-based ArangoDB 3.0 builds will automatically test the local environment’s capabilities and set the compiler options based on which CPU abilities were detected.

For example, a mere (cd build && cmake -DCMAKE_BUILD_TYPE=Release ..) will run the CPU ability detection and configure the build to use the features supported by the local architecture:

1

(mkdir -p build; cd build && cmake -DCMAKE_BUILD_TYPE=Release ..)

For example, on my laptop this prints:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

-- The CXX compiler identification is GNU 5.3.1
-- The C compiler identification is GNU 5.3.1
...
-- target changed from "" to "auto"
-- Detected CPU: haswell
-- Performing Test check_cxx_compiler_flag__march_core_avx2
-- Performing Test check_cxx_compiler_flag__march_core_avx2 - Success
-- Performing Test check_cxx_compiler_flag__msse2
-- Performing Test check_cxx_compiler_flag__msse2 - Success
-- Performing Test check_cxx_compiler_flag__msse3
-- Performing Test check_cxx_compiler_flag__msse3 - Success
-- Looking for pmmintrin.h
-- Looking for pmmintrin.h - found
-- Performing Test check_cxx_compiler_flag__mssse3
-- Performing Test check_cxx_compiler_flag__mssse3 - Success
-- Looking for tmmintrin.h
-- Looking for tmmintrin.h - found
-- Performing Test check_cxx_compiler_flag__msse4_1
-- Performing Test check_cxx_compiler_flag__msse4_1 - Success
-- Looking for smmintrin.h
-- Looking for smmintrin.h - found
-- Performing Test check_cxx_compiler_flag__msse4_2
-- Performing Test check_cxx_compiler_flag__msse4_2 - Success
-- Performing Test check_cxx_compiler_flag__mavx
-- Performing Test check_cxx_compiler_flag__mavx - Success
-- Looking for immintrin.h
-- Looking for immintrin.h - found
-- Performing Test check_cxx_compiler_flag__msse2avx
-- Performing Test check_cxx_compiler_flag__msse2avx - Success
-- Performing Test check_cxx_compiler_flag__mavx2
-- Performing Test check_cxx_compiler_flag__mavx2 - Success
-- Performing Test check_cxx_compiler_flag__mno_sse4a
-- Performing Test check_cxx_compiler_flag__mno_sse4a - Success
-- Performing Test check_cxx_compiler_flag__mno_xop
-- Performing Test check_cxx_compiler_flag__mno_xop - Success
-- Performing Test check_cxx_compiler_flag__mno_fma4
-- Performing Test check_cxx_compiler_flag__mno_fma4 - Success
...

The detected options will end up in the CMakeCache.txt file in the build directory. The Makefile generated by cmake will automatically make use of these options when invoking the C++ compiler.

The compiler options are not shown by default, but they can be made visible by compiling with the option VERBOSE=1, e.g.

1
2

(mkdir -p build; cd build && cmake -DCMAKE_BUILD_TYPE=Release ..)
(cd build && make -j4 VERBOSE=1)

Note that this will be very verbose, so you only want to set the VERBOSE=1 option to check that the compiler options were picked correctly.

On my local laptop, the architecture-specific compiler options that were automatically detected and used were

1

-march=core-avx2 -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -msse2avx -mavx2 -mno-sse4a -mno-xop -mno-fma4

The build has detected core-avx2, which in my case is good – and a lot more specific than the official packages which for example cannot assume the presence of either SSE4 or AVX instructions.

And now that we can rely on the presence of specific CPU instructions, some code parts such as JSON parsing can make use of SSE4.2 instructions, or the compiler can use some optimized SIMD variants for operations such as memcpy, strlen etc.

These sanitizers provide runtime instrumentation for executables and check for common C++ programming errors such as buffer overflows, use-after-free bugs and memory leaks. The sanitizers supported natively by recent versions of g++ and clang++.

The general runtime overhead of the sanitizers is neglectable compared to other instrumentation tools such as Valgrind, so it can well be used in the day-to-day development process.

Configuring the build to use ASAN

To build with ASAN (Address Sanitizer) and UBSAN (Undefined Behavior Sanitizer), the following environment variables need to be set when invoking cmake:

1

CXXFLAGS="-fsanitize=address -fsanitize=undefined -fno-sanitize=alignment -fno-sanitize=vptr"

The full command to configure a debug build with ASAN and UBSAN support is:

1

(cd build && CXXFLAGS="-fsanitize=address -fsanitize=undefined -fno-sanitize=alignment -fno-sanitize=vptr cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_MAINTAINER_MODE=On ..)

This will configure the build so it will pass the sanitizer options to the C++ compiler. Note that you’ll need a recent version of g++ or clang++ for this to work.

After that, build arangod normally. The binaries produced will be instrumented by the sanitizers, allowing many nasty memory errors to be detected very early.

Checking if an arangod binary is using ASAN

Whether or not an ArangoDB binary is instrumented can be found out at any time by calling the binary with the --version option. This will print some version information for ArangoDB itself and the other required libraries, and it will also print in the line starting with asan whether the binary was compiled with ASAN support or not:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

> build/bin/arangod --version

3.0.x-devel

architecture: 64bit
asan: true                       # !!!!
asm-crc32: true
boost-version: 1.61.0b1
build-date: 2016-06-09 12:07:07
compiler: gcc
cplusplus: 201103
endianness: little
fd-client-event-handler: poll
fd-setsize: 1024
icu-version: 54.1
jemalloc: false
libev-version: 4.22
maintainer-mode: true
openssl-version: OpenSSL 1.0.2g-fips  1 Mar 2016
rocksdb-version: 4.8.0
server-version: 3.0.x-devel
sizeof int: 4
sizeof void*: 8
sse42: true
tcmalloc: false
v8-version: 5.0.71.39
vpack-version: 0.1.30
zlib-version: 1.2.8

Controlling ASAN runtime behavior

ASAN behavior can also be controlled at runtime, after the binaries have been produced, by adjusting the environment variable ASAN_OPTIONS. The setting I use for this is:

1

ASAN_OPTIONS="handle_ioctl=true:check_initialization_order=true:detect_container_overflow=1:detect_stack_use_after_return=false:detect_odr_violation=1:allow_addr2line=true:strict_init_order=true"

How to do this is different in the 2.8 and 3.0 branches of ArangoDB, but luckily it’s really easy to achieve in both branches.

ArangoDB 2.8

For the 2.8 branch of ArangoDB, clone the 2.8 version of the ArangoDB repository first:

1
2

git clone -b 2.8 https://github.com/arangodb/arangodb
cd ArangoDB

And in that directory execute the following commands:

1
2
3
4
5

make setup
export CFLAGS="-g -Og"
export CXXFLAGS="-g -Og"
./configure --enable-relative --enable-maintainer-mode
make -j4

(note that you’ll need a working C++11 compiler and some prerequisites such as the OpenSSL library, GNU Bison and Flex installed for this to work).

This will compile ArangoDB and all of its dependencies, and finally make them available in the bin subdirectory of the current directory. There is no need to install ArangoDB using make install. arangod and the client tools can be run locally using the following commands:

1
2

bin/arangod --console mydb # start arangod in console mode
bin/arangosh # starts an ArangoShell

Now, after any modification to the ArangoDB C++ source code you can re-compile using make. This will only rebuild the things that need to be rebuilt. If the build is successful, the changes you have made should be visible when restarting the binaries.

As the -g option used above will have configured a build with debug symbols, it’s also possible to use a debugger such as gdb for stepping through the executables, attach to the while they are running, or to obtain stack traces in case any of the executables crashed.

ArangoDB 3.0

For the 3.0 branch of ArangoDB, it’s required to once clone that version of the ArangoDB repository into a local directory:

1
2

git clone -b 3.0 https://github.com/arangodb/arangodb
cd ArangoDB

In that directory execute the following commands to configure the build for debugging:

1

(mkdir -p build; cd build && cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_MAINTAINER_MODE=On ..)

(again you’ll need a working C++11 compiler and some prerequisites such as the OpenSSL library installed for this to work).

This will configure the build to be a debug build, with debug symbols but most optimizations disabled. To build ArangoDB, just execute:

1

(cd build && make -j4)

To execute one of the binaries, run them from the checkout directory as follows:

1
2

build/bin/arangod --console mydb # start arangod in console mode
build/bin/arangosh # starts an ArangoShell

After any modification to the ArangoDB C++ source code, build again using (cd build && make -j4) to see the changes in effect. The build is a debug build, meaning you can use a debugger and get meaningful stack traces for core dumps produced by the binaries.

If you got, then I strongly recommend to also enable the Address Sanitizer (ASAN) for your debug build. This can greatly help finding common memory usage errors during development. There is another
blog post about using ASAN in ArangoDB development.

std::stoull and std::strtoull

The natural solution for this kind of conversion is the standard library’s std::stoull function. On the one hand, that solution is potentially optimized and definitely robust. It performs validation of the input characters and is also battle-tested by millions of users.

On the other hand, std::stoull has a few “features” that some would consider “anti”-features, and that sound like they could negatively influence performance:

• it may behave locale-dependent
• it can parse and will let pass negative input values, which is often not desired for a result of unsigned type
• it can perform base conversion

In C++11 there is also the alternative std::strtoull. It should behave about the same as std::stoull except in case of error. std::stoull will throw and std::strtoull will not.

That’s what we get from the standard library for long unsigned integer types.

Alternative implementations (w/o error checking)

Another alternative is to roll a string to number conversion function ourselves. If we hard-code it to base 10 and do not care about error checking, a naive implementation may look like this:

1
2
3
4
5
6
7
8
9
10

inline uint64_t naive(std::string const& value) {
  uint64_t result = 0;
  char const* p = value.c_str();
  char const* q = p + value.size();
  while (p < q) {
    result *= 10;
    result += *(p++) - '0';
  }
  return result;
}

Obviously the above will produce wrong results for invalid input data, but for “trusted” (known to be valid) input it may be just fine.

Here’s just another implementation for the problem at hand. This one doesn’t perform the times 10 operation, but splits it into two bitshifting operations:

1
2
3
4
5
6
7
8
9
10

inline uint64_t bitshift(std::string const& value) {
  uint64_t result = 0;

  char const* p = value.c_str();
  char const* q = p + value.size();
  while (p < q) {
    result = (result << 1) + (result << 3) + *(p++) - '0';
  }
  return result;
}

Again no error checking is present in the above function, but it should be ok for trusted inputs.

By manually unrolling the while loop and converting it into a switch statement, we can also come up with a conversion function that has minimal branching:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

inline uint64_t unrolled(std::string const& value) {
  uint64_t result = 0;

  size_t const length = value.size();
  switch (length) {
    case 20:    result += (value[length - 20] - '0') * 10000000000000000000ULL;
    case 19:    result += (value[length - 19] - '0') * 1000000000000000000ULL;
    case 18:    result += (value[length - 18] - '0') * 100000000000000000ULL;
    case 17:    result += (value[length - 17] - '0') * 10000000000000000ULL;
    case 16:    result += (value[length - 16] - '0') * 1000000000000000ULL;
    case 15:    result += (value[length - 15] - '0') * 100000000000000ULL;
    case 14:    result += (value[length - 14] - '0') * 10000000000000ULL;
    case 13:    result += (value[length - 13] - '0') * 1000000000000ULL;
    case 12:    result += (value[length - 12] - '0') * 100000000000ULL;
    case 11:    result += (value[length - 11] - '0') * 10000000000ULL;
    case 10:    result += (value[length - 10] - '0') * 1000000000ULL;
    case  9:    result += (value[length -  9] - '0') * 100000000ULL;
    case  8:    result += (value[length -  8] - '0') * 10000000ULL;
    case  7:    result += (value[length -  7] - '0') * 1000000ULL;
    case  6:    result += (value[length -  6] - '0') * 100000ULL;
    case  5:    result += (value[length -  5] - '0') * 10000ULL;
    case  4:    result += (value[length -  4] - '0') * 1000ULL;
    case  3:    result += (value[length -  3] - '0') * 100ULL;
    case  2:    result += (value[length -  2] - '0') * 10ULL;
    case  1:    result += (value[length -  1] - '0');
  }
  return result;
}

Performance testing

To check out how all these alternatives perform, I put them into a small test driver program. To compile it with g++ and run it, I used this command:

1

g++ -Wall -Wextra -march=native -std=c++11 -O3 stdoull-test.cpp && ./a.out

The test program will convert input strings of different lengths to uint64_t using the above (and some other) implementations, going up from 10,000 iterations per string up to 100,000,000. It also prints the wall-clock time spent in each run. The most interesting output it prints is in the “ms” column. The “result” column can be fully ignored. It’s only there so the compiler won’t optimize away the calls to the conversion functions.

The timings from my local laptop (Intel Core i7-4710HQ CPU @ 2.50GHz, Ubuntu 16.04, g++ 5.3.1 – your mileage may vary!) for the 100,000,000 conversions are:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

test 'std::stoull'       string '1'                           1209 ms
test 'std::stoull'       string '99'                          1382 ms
test 'std::stoull'       string '1234'                        1725 ms
test 'std::stoull'       string '1234567'                     2257 ms
test 'std::stoull'       string '1234567891'                  2764 ms
test 'std::stoull'       string '12345678901234'              3899 ms
test 'std::stoull'       string '12345678901234678901'        7391 ms

test 'std::strtoull'     string '1'                           1104 ms
test 'std::strtoull'     string '99'                          1300 ms
test 'std::strtoull'     string '1234'                        1628 ms
test 'std::strtoull'     string '1234567'                     2428 ms
test 'std::strtoull'     string '1234567891'                  2662 ms
test 'std::strtoull'     string '12345678901234'              3705 ms
test 'std::strtoull'     string '12345678901234678901'        6631 ms

test 'naive'             string '1'                            202 ms
test 'naive'             string '99'                           314 ms
test 'naive'             string '1234'                         475 ms
test 'naive'             string '1234567'                      732 ms
test 'naive'             string '1234567891'                   987 ms
test 'naive'             string '12345678901234'              1343 ms
test 'naive'             string '12345678901234678901'        1862 ms

test 'bitshift'          string '1'                            188 ms
test 'bitshift'          string '99'                           245 ms
test 'bitshift'          string '1234'                         397 ms
test 'bitshift'          string '1234567'                      462 ms
test 'bitshift'          string '1234567891'                   666 ms
test 'bitshift'          string '12345678901234'               888 ms
test 'bitshift'          string '12345678901234678901'        1277 ms

test 'unrolled'          string '1'                            289 ms
test 'unrolled'          string '99'                           289 ms
test 'unrolled'          string '1234'                         351 ms
test 'unrolled'          string '1234567'                      408 ms
test 'unrolled'          string '1234567891'                   547 ms
test 'unrolled'          string '12345678901234'               778 ms
test 'unrolled'          string '12345678901234678901'        1068 ms

It’s no surprise the standard library functions with their generality and error checking features are slower than the specialized functions that took the liberty to ignore all that.

Which method is fastest now?

As can be seen above, the “bitshift” variant was fastest for short and medium length inputs in my environment. At some point when input values get longer, the “bitshift” methods gets overtaken by the “unrolled” variant. The “naive” variant was slower than the two in most cases, but still performs surprisingly well.

Conclusion

The take-away: even though string-to-number conversion routines are present in the standard library, it can still be beneficial to hand-craft specialized variants of them for maximum performance. This is especially true when most of the generality that the standard library routines provide is not required in your use case.

For example, if you know that the conversion functions will only operate on “trusted” input (only numeric digit input characters, length of input string won’t exceed the maximum length of a uint64_t number etc.) then error checking is not required.

Additionally, if you can limit yourself to numbers of a specific base and don’t negative base conversion nor the handling of negative values, a lot of the generality and safety overhead of std::stoull may be avoided.

This post explains how to use CMake to build ArangoDB 3.0 on a recent Ubuntu Linux. For the impatient there’s a command summary at the end of this post.

Installing prerequisites

Here’s how to build ArangoDB 3.0 with cmake on a recent Ubuntu Linux.

Ubuntu 15.x and 16.x should have recent enough packages so any missing prerequisites can be installed via a simple sudo apt-get install command:

1

sudo apt-get install cmake make build-essential openssl python2.7 g++ gcc

Older versions of Ubuntu can be convinced to work too, but this requires a g++ version of at least 4.9. Ubuntu 14 ships with older g++ versions by default, so you will first need to install a newer g++ version (see here or here for some external instructions). Once the g++ and gcc compilers are recent enough, install the other prerequisites.

Cloning ArangoDB

After having installed the prerequisites, clone the ArangoDB repository from Github and then cd into the directory arangodb that the cloning will have created:

1
2

git clone https://github.com/arangodb/arangodb
cd arangodb

Then check out the 3.0 branch and pull the latest changes:

1
2

git checkout 3.0
git pull

Building ArangoDB

A convention when using CMake is to not build directly in the source directory, but use a separate build directory instead. The benefit of this is that building in a separate directory won’t change the source directory, and the build directory can be disposed easily.

To create an initial build directory named build and cd into it, just type:

1
2

mkdir -p build
cd build

It’s now time to invoke CMake. CMake should be executed from the build directory, but needs to know where it’s build instructions file (named CMakeLists.txt is). This file is located in the source directory. To run CMake without any specific options, we can use:

1

cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..

This will run CMake from inside the build directory and tell it to look for CMakeLists.txt in the source directory. The -DCMAKE_BUILD_TYPE=RelWithDebInfo part will tell CMake to build binaries with optimizations and debug symbols. Other useful build types are:

• -DCMAKE_BUILD_TYPE=Debug: without optimizations, for debugging only
• -DCMAKE_BUILD_TYPE=Release: with optimizations, no debug symbols, not useful for debugging

Invoking CMake will perform some checks for required components, compiler and platform features. If anything fails, it should print some error message. If everything works well, the command should have created a Makefile in the build directory.

side note: CMake on Linux will by default generate UNIX Makefiles, but it can also be told to generate ninja files instead. On Windows it can generate solution files for Visual Studio.

Now that the Makefile is there, we can run make as usual (note that the -j4 means to build with 4 parallel processes – adjust as needed!):

1

make -j4

Starting arangod and arangosh

And that’s already it for a normal build. Note that the build artefacts and thus the binaries will also be created inside the build directory. arangod and arangosh will be located in the bin subdirectory of the build directory. There is no need to install these binaries, they can be run without installing them first. However, starting them from inside the build directory will not work, because they will complain about missing files. It is better to cd one level up first and then start them:

1
2

cd ..
build/bin/arangod ~/testdata --server.authentication false

The former will start arangod with default configuration, with a fresh database directory located in ~/testdata.

If you ran all the commands on your local machine (127.0.0.1) then you can now open a browser and point it to http://127.0.0.1:8529/. This should bring up ArangoDB’s web interface.

You can alternatively start an ArangoShell to connect to the ArangoDB server. In another terminal window, cd into the arangodb directory and start an ArangoShell as follows:

1
2

cd arangodb
build/bin/arangosh

You should now see the ArangoShell prompt and be able to type some commands!

Command summary

Note: in contrast to the above step-by-step instructions, some commands here have been put together on a single line to make developer’s life easier.

1

sudo apt-get install cmake make build-essential openssl python2.7 g++ gcc

1
2
3
4
5
6

git clone https://github.com/arangodb/arangodb
cd arangodb
git checkout 3.0
git pull
mkdir -p build
(cd build && cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..)

1
2
3

git pull
(cd build && make -j4)
build/bin/arangod ~/testdata --server.authentication false

The problem is that when compiling ArangoDB with g++6 with default options, arangod starts and almost immediately segfaults.

A backtrace of the crashed arangod process shows that the segfaults originate from the V8 JavaScript engine that ArangoDB uses internally. The problem seems to have affected other users of V8 as well, indicated by the following error reports:

• node.js: https://github.com/nodejs/node/issues/6724
• chromium: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68853

The reason for the error now popping up is a change in the g++ 6 optimizer as stated in the gcc6 release notes:

Value range propagation now assumes that the this pointer of C++ member functions is non-null. This eliminates common null pointer checks but also breaks some non-conforming code-bases (such as Qt-5, Chromium, KDevelop). As a temporary work-around -fno-delete-null-pointer-checks can be used. Wrong code can be identified by using -fsanitize=undefined.

That means if compiling ArangoDB 2.8 from source with g++ 6 or higher, please be sure to set the environment variables CFLAGS="-fno-delete-null-pointer-checks" and
CXXFLAGS="-fno-delete-null-pointer-checks" before invoking the configure command:

1
2
3

make setup
CFLAGS="-fno-delete-null-pointer-checks" CXXFLAGS="-fno-delete-null-pointer-checks" ./configure
make

The options will then be passed to the sub-make that builds the V8 engine. If you have already built V8 without these options, try removing the file .v8-build-64 from the build directory and run configure and make again.

As a reminder, ArangoDB 2.8 will also emit this big notice at the end of its configure output:

1
2
3
4
5
6
7
8
9

configure: --------------------------------------------------------------------------------
configure:                                                                                 
configure: NOTE: if you are compiling ArangoDB with g++ 6.0 or higher, please make sure to 
configure: set the following environment variables when compiling ArangoDB:                
configure:                                                                                 
configure:   CFLAGS="-fno-delete-null-pointer-checks"                                      
configure:   CXXFLAGS="-fno-delete-null-pointer-checks"                                    
configure:                                                                                 
configure: --------------------------------------------------------------------------------

Users of g++ 5.x do not need to set these flags, neither need users of clang++. And in the ArangoDB 3.0 build the options will be set automatically when compiling V8 if the compiler is g++. They will not be set for compiling any other parts of ArangoDB as it’s only required for building V8.

If a connection to the server can still be established, the easiest is to use the ArangoShell to fetch the list of currently executing AQL queries and send a kill command to the server for the correct query.

To start, we can fetch the list of all running queries and print their ids, query strings and runtimes. This is only inspection and does not abort any query:

1
2

var queries = require("org/arangodb/aql/queries");
queries.current();

Here’s an example result for the list of running queries:

1
2
3
4
5
6
7
8

[
  {
    "id" : "190",
    "query" : "RETURN SLEEP(1000)",
    "started" : "2016-01-26T22:41:24Z",
    "runTime" : 218.49146389961243
  }
]

To now kill a query from the list, we can pass the query’s id to kill:

1
2

var queries = require("org/arangodb/aql/queries");
queries.kill("190");  /* insert actual query id here */

If a query was actually killed on the server, that call should return without an error, and the server should have logged a warning in addition.

If we wanted to abort one or many queries from the list solely by looking at query string patterns or query runtime, we could iterate over the list of current queries and kill each one that matches a predicate.

For example, the following snippet will abort all currently running queries that contain the string SLEEP anywhere inside their query string:

1
2
3
4
5
6
7
8

var queries = require("org/arangodb/aql/queries");

queries.current().filter(function(query) {
  return query.query.match(/SLEEP/);  /* predicate based on query string */
}).forEach(function(query) {
  print("killing query: ", query);    /* print what we're killing */
  queries.kill(query.id);             /* actually kill query */
});

Filtering based on current query runtime is also simple, by adjusting the predicate. To abort all queries that ran longer than 30 seconds use:

1
2
3
4
5
6
7
8

var queries = require("org/arangodb/aql/queries");

queries.current().filter(function(query) {
  return query.runTime > 30;          /* predicate based on query runtime */
}).forEach(function(query) {
  print("killing query: ", query);    /* print what we're killing */
  queries.kill(query.id);             /* actually kill query */
});

Please make sure the predicates are correct so only the actually intended queries get aborted!

To test a predicate without killing a query, use the above code without the forEach part that did the killing.

While previous versions required writing a hard-to-memoize command like

1

require("org/arangodb/aql/explainer").explain(query);

to explain an AQL query from the ArangoShell, 2.8 reduces this task to a mere

1

db._explain(query);

Apart from that, explain in 2.8 is smarter when confronted with very lengthy query strings, and with queries that contain huge hard-coded string, array, or object values.

For example, when creating an array bind variable with 1,000 values and using them in an explained query, 2.7 would print the entire 1,000 array values in the explain output:

1
2
3
4
5
6
7
8
9
10
11
12

var keys = [];
for (var i = 0; i < 1000; ++i) {
  keys.push("test" + i);
}

var query = "FOR i IN @keys RETURN i";
require("org/arangodb/aql/explainer").explain({
  query: query,
  bindVars: {
    keys: keys
  }
});

2.8 will instead truncate longer arrays and objects in the explain output for much improved readability:

Automatic value truncation will occur for array and object values with more than 20 elements or for string values longer than 1,024 characters. The truncation for explain will occur if these values are hard-coded into the query or are passed via bind parameters.

Truncation only happens inside the explain results processing and thus cannot affect the actual query results.

With 2.8 it becomes as simple as RETURN POW(2, 16) to raise 2 to the power of 16 from inside AQL.

In order to create a missing target collection arangoimp for long has provided the option --create-collection true:

1
2
3
4

arangoimp                       \
  --file users.json             \
  --collection users            \
  --create-collection true

However there hasn’t been a way of specifying the type for the target collection, so the new collection was always created as document collection.

To import data into an edge collection, the target collection needed to be created by another means, e.g. by using the ArangoShell. It would have been more handy if arangoimp were able to create edge collections too.

2.8 finally adds that feature, and it’s simple to use: to create an edge collection if the target collection does not exist, append the --create-collection-type edge option when invoking arangoimp:

1
2
3
4
5

arangoimp                       \
  --file users.json             \
  --collection users            \
  --create-collection true      \
  --create-collection-type edge

Array indexes

2.8 allows creating hash and skiplist indexes on attributes which are arrays. Creating such index works similar to creating a non-array index, with the exception that the name of the array attribute needs to be followed by a [*] in the index fields definition:

1
2

db._create("posts");
db.posts.ensureIndex({ type: "hash", fields: [ "tags[*]" ] });

Now if the tags attribute of a document in the posts collection is an array, each array member will be inserted into the index:

1
2
3

db.posts.insert({ tags: [ "arangodb", "database", "aql" ] });
db.posts.insert({ tags: [ "arangodb", "v8", "javascript" ] });
db.posts.insert({ tags: [ "javascript", "v8", "nodejs" ] });

The index on tags[*] will now contain the values arangodb, database, aql and nosql for the first document, arangodb, v8 and javascript for the second, and javascript, v8 and nodejs for the third.

The following AQL will find any documents that have a value of javascript contained in their tags value:

1
2
3

FOR doc IN posts
  FILTER 'javascript' IN doc.tags[*]
  RETURN doc

This will use the array index on tags[*].

The array index works by inserting all members from an array into the index separately. Duplicates are removed automatically when populating the index.

An array index can also be created on a sub-attribute of array members. For example, the following definition will make sure the name sub-attributes of the tags array values will make it into the index:

1

db.posts.ensureIndex({ type: "hash", fields: [ "tags[*].name" ] });

That will allow storing data as follows:

1
2
3

db.posts.insert({ tags: [ { name: "arangodb" }, { name: "database" }, { name: "aql" } ] });
db.posts.insert({ tags: [ { name: "arangodb" }, { name: "v8" }, { name: "javascript" } ] });
db.posts.insert({ tags: [ { name: "javascript" }, { name: "v8" }, { name: "nodejs" } ] });

The (index-based) selection query for this data structure then becomes

1
2
3

FOR doc IN posts
  FILTER 'javascript' IN doc.tags[*].name
  RETURN doc

Contrary to MongoDB, there is no automatic conversion to array values when inserting non-array values in ArangoDB. For example, the following plain strings will not be inserted into an array index, simply because the value of the index attribute is not an array:

1
2
3

db.posts.insert({ tags: "arangodb" });
db.posts.insert({ tags: "javascript" });
db.posts.insert({ tags: "nodejs" });

Note that in this case a non-array index can still be used.

Use of multiple indexes per collection

The query optimizer can now make use of multiple indexes if multiple filter conditions are combined with logical ORs, and all of them are covered by indexes of the same collection.

Provided there are separate indexes present on name and status, the following query can make use of index scans in 2.8, as opposed to full collection scans in 2.7:

1
2
3

FOR doc IN users 
  FILTER doc.name == 'root' || doc.status == 'active' 
  RETURN doc

If multiple filter conditions match for the same document, the result will automatically be deduplicated, so each document is still returned at most once.

Sorted IN comparison

Another improvement for the optimizer is to pre-sort comparison values for IN and NOT IN so these operators can use a (much faster) binary search instead of a linear search.

The optimization will be applied automatically for IN / NOT IN comparison values used in filters, which are used inside of a FOR loop, and depend on runtime values. For example, the optimization will be applied for the following query:

1
2
3
4

LET values = /* some runtime expression here */
FOR doc IN collection
  FILTER doc.value IN values
  RETURN doc

The optimization will not be applied for IN comparison values that are value literals and those that are used in index lookups. For these cases the comparison values were already deduplicated and sorted.

“sort-in-values” will appear in the list of applied optimizer rules if the optimizer could apply the optimization:

Optimization for LENGTH(collection)

There are multiple ways for counting the total number of documents in a collection from inside an AQL query. One obvious way is to use RETURN LENGTH(collection).

That variant however was inefficient as it fully materialized the documents before counting them. In 2.8 calling LENGTH() for a collection will get automatically replaced by a call to a special function that can efficiently determine the number of documents. For larger collections, this can be several thousand times faster than the naive 2.7 solution.

C++ implementation for many AQL functions

Many existing AQL functions have been backed with a C++ implementation that removes the need for some data conversion that would otherwise happen if the function were implemented in V8/JavaScript only. More than 30+ functions have been changed, including several that may produce bigger result sets (such as EDGES(), FULLTEXT(), WITHIN(), NEAR()) and that will hugely benefit from this.

Improved skip performance

2.8 improves the performance of skipping over many documents in case no indexes and no filters are used. This might sound like an edge case, but it is quite common when the task is to fetch documents from a big collection in chunks and there is certainty that there will be no parallel modifications.

For example, the following query runs about 3 to 5 times faster in 2.8, and this improvements can easily sum up to notable speedups if the query is called repeatedly with increasing offset values for LIMIT:

1
2
3

FOR doc IN collection
  LIMIT 1000000, 10
  RETURN doc

The following list shows the AQL functions for which a C++ implementation has been added in 2.8. The other C++-based AQL function implementations added since ArangoDB 2.5 are also still available. Here’s the list of functions added in 2.8:

• document-related functions: DOCUMENT, EDGES, PARSE_IDENTIFIER
• numerical functions: ABS, FLOOR, RAND, ROUND, SQRT
• statistical functions: MEDIAN, PERCENTILE, STDDEV_POPULATION, STDDEV_SAMPLE, VARIANCE_POPULATION, VARIANCE_SAMPLE
• geo functions: NEAR, WITHIN
• array functions: APPEND, FIRST, FLATTEN, LAST, MINUS, NTH, POP, POSITION, PUSH, REMOVE_NTH, REMOVE_VALUE, REMOVE_VALUES, SHIFT, UNSHIFT
• informational functions: COLLECTIONS, CURRENT_DATABASE, FIRST_DOCUMENT, FIRST_LIST, NOT_NULL
• object-related functions: MERGE_RECURSIVE, ZIP

Following are a few example queries that benefit from using the C++ variants of some of the above functions:

Fetching documents programmatically using the DOCUMENT function:

• query: FOR i IN 1..10000 RETURN DOCUMENT(test, CONCAT('test', i))
• 2.7: 0.3005 s
• 2.8: 0.1050 s

Fetching edges programmatically using the EDGES function:

• query: FOR i IN 1..100000 RETURN EDGES(edges, CONCAT('test/test', i), 'outbound'):
• 2.7: 4.3590 s
• 2.8: 1.4469 s

Fetching many documents from a geo index, post-filtering most of them:

• query: FOR doc IN WITHIN(locations, 0, 0, 100000) FILTER doc.value2 == 'test1001' LIMIT 1 RETURN doc
• 2.7: 2.9876 s
• 2.8: 0.4087 s

Generating random numbers:

• query: FOR value IN 1..100000 RETURN RAND() * 50
• 2.7: 0.1743 s
• 2.8: 0.1364 s

Please note that not in every case there will be a tremendous speedup. As usual, it depends on how often a function is called inside a query and what other constructs are used. Your mileage may vary.

This is much better in 2.8. Now the query optimizer can use multiple indexes on the same collection for FILTER conditions that are combined with a logical OR.

For all following queries, I have set up a collection named test, which has two isolated hash indexes on the attributes value1 and value2, and a skiplist index on attribute value3.

Let’s first try an AQL queries that uses a logical OR on two different attributes of the collection:

1
2
3

FOR doc IN test
  FILTER doc.value1 == 11 || doc.value2 == 19 
  RETURN doc

The execution plan for this query in 2.7 reveals that query will perform a full collection scan and cannot use indexes because of the logical OR on two different attributes:

1
2
3
4
5
6
7
8
9
10
11
12
13

Execution plan:
 Id   NodeType                  Est.   Comment
  1   SingletonNode                1   * ROOT
  2   EnumerateCollectionNode      0     - FOR doc IN test   /* full collection scan */
  3   CalculationNode              0       - LET #1 = doc.`value1` == 11 || doc.`value2` == 19
  4   FilterNode                   0       - FILTER #1
  5   ReturnNode                   0       - RETURN doc

Indexes used:
 none

Optimization rules applied:
 none

Running the same query in 2.8 / devel will produce a much better execution plan:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Execution plan:
 Id   NodeType          Est.   Comment
  1   SingletonNode        1   * ROOT
  6   IndexNode            2     - FOR doc IN test   /* hash index scan, hash index scan */
  3   CalculationNode      2       - LET #1 = doc.`value1` == 11 || doc.`value2` == 19  
  4   FilterNode           2       - FILTER #1
  5   ReturnNode           2       - RETURN doc

Indexes used:
 By   Type   Collection   Unique   Sparse   Selectivity   Fields         Ranges
  6   hash   test         false    false       100.00 %   [ `value1` ]   doc.`value1` == 11
  6   hash   test         false    false       100.00 %   [ `value2` ]   doc.`value2` == 19

Optimization rules applied:
 Id   RuleName
  1   use-indexes

Multiple indexes will also be used if different index types are accessed, or for non-equality filter conditions. For example, the following query will make use of the two hash indexes and also the skiplist index:

1
2
3

FOR doc IN test 
  FILTER doc.value1 == 11 || doc.value2 == 19 || doc.value3 > 42 
  RETURN doc

Here is its execution plan from 2.8:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Execution plan:
 Id   NodeType          Est.   Comment
  1   SingletonNode        1   * ROOT
  6   IndexNode            3     - FOR doc IN test   /* hash index scan, hash index scan, skiplist index scan */
  3   CalculationNode      3       - LET #1 = doc.`value1` == 11 || doc.`value2` == 19 || doc.`value3` > 42 
  4   FilterNode           3       - FILTER #1
  5   ReturnNode           3       - RETURN doc

Indexes used:
 By   Type       Collection   Unique   Sparse   Selectivity   Fields         Ranges
  6   hash       test         false    false       100.00 %   [ `value1` ]   doc.`value1` == 11
  6   hash       test         false    false       100.00 %   [ `value2` ]   doc.`value2` == 19
  6   skiplist   test         false    false            n/a   [ `value3` ]   doc.`value3` > 42

Optimization rules applied:
 Id   RuleName
  1   use-indexes

For comparison, here is the non-optimized plan from 2.7 for the same query:

1
2
3
4
5
6
7
8
9
10
11
12
13

Execution plan:
 Id   NodeType                  Est.   Comment
  1   SingletonNode                1   * ROOT
  2   EnumerateCollectionNode      0     - FOR doc IN test   /* full collection scan */
  3   CalculationNode              0       - LET #1 = doc.`value1` == 11 || doc.`value2` == 19 || doc.`value3` > 42
  4   FilterNode                   0       - FILTER #1
  5   ReturnNode                   0       - RETURN doc

Indexes used:
 none

Optimization rules applied:
 none

Still the query optimizer will not be able to use any indexes on a collection when there are multiple FILTER conditions combined with logical OR and at least one of them is not satisfisable by an index of the collection. In this case it has no other choice but to do a full collection scan.

For queries that combine multiple FILTER conditions with a logical AND, the optimizer will still try to pick the most selective index for the query and use it for the collection.

First of all, hash and skiplist indexes can now index individual array values. A dedicated post on this will follow shortly. Second, the query optimizer can make use multiple indexes per collection for queries with OR-combined filter conditions. This again is a subject for another post. Third, there have been some speed improvements due to changes in the general index handling code. This is what this post is about.

In order to assess the speedups in 2.8, I have run some already existing performance tests that I initially ran when comparing ArangoDB 2.5 with 2.6. The test cases and methodology are detailed in this earlier blog post.

For measuring the index-related performance improvements, I simply re-ran the index related tests in 2.7 and in 2.8 / devel. I did not bother re-running all tests from the original blog article because only some are index-related. In particular, I only ran these tests again:

• join-key: for each document in the collection, perform a join on the _key attribute on the collection itself (i.e. FOR c1 IN @@c FOR c2 IN @@c FILTER c1._key == c2._key RETURN c1)
• join-id: ditto, but perform the join using the _id attribute
• join-hash-number and join-hash-string: ditto, but join using a hash index on a numeric or string attribute
• join-skiplist-number and join-skiplist-string: ditto, but join using a skiplist index on a numeric or string attribute
• lookup-key, lookup-hash-number, lookup-hash-string, lookup-skiplist-number, lookup-skiplist-string: compile an IN-list of 10,000 lookup values and search these 10,000 documents in the collection using either the primary index (_key attribute), a hash index or a skiplist index. The latter two are tested on numeric and string attributes.

The test queries were run 5 times each on collections containing 10,000, 100,000 and 1,000,000 documents.

Here are the query execution times from 2.7 and 2.8 for the individual tests, in seconds (less is better):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

test name                collection     avg 2.7 (s)      avg 2.8 (s)
--------------------------------------------------------------------
join-key                 10k                 0.0997           0.0612
join-key                 100k                1.0611           0.6538
join-key                 1000k              10.2975           6.2507
join-id                  10k                 0.1088           0.0606
join-id                  100k                1.1126           0.6694
join-id                  1000k              10.8180           6.5336
join-hash-number         10k                 0.1044           0.0679
join-hash-number         100k                1.1137           0.7430
join-hash-number         1000k              10.7923           7.1534
join-hash-string         10k                 0.1193           0.0712
join-hash-string         100k                1.2656           0.7915
join-hash-string         1000k              12.3075           7.6667
join-skiplist-number     10k                 0.1387           0.1030
join-skiplist-number     100k                1.6693           1.3062
join-skiplist-number     1000k              18.0406          15.5203
join-skiplist-string     10k                 0.1928           0.1469
join-skiplist-string     100k                2.3166           1.8997
join-skiplist-string     1000k              27.1513          23.2058
lookup-key               10k                 1.4996           1.5245
lookup-key               100k                1.4951           1.5189
lookup-key               1000k               1.5545           1.5662
lookup-hash-number       10k                 1.5572           1.5526
lookup-hash-number       100k                1.5595           1.5435
lookup-hash-number       1000k               1.5436           1.5648
lookup-hash-string       10k                 1.6023           1.5623
lookup-hash-string       100k                1.5892           1.5741
lookup-hash-string       1000k               1.5841           1.5770
lookup-skiplist-number   10k                 1.5978           1.6145
lookup-skiplist-number   100k                1.5782           1.6269
lookup-skiplist-number   1000k               1.5891           1.6258
lookup-skiplist-string   10k                 1.6443           1.6840
lookup-skiplist-string   100k                1.6787           1.6985
lookup-skiplist-string   1000k               1.7319           1.7562
in-key                   10k                 0.1076           0.0754
in-key                   100k                0.1083           0.0775
in-key                   1000k               0.1104           0.0773
in-hash-number           10k                 0.0898           0.0696
in-hash-number           100k                0.0889           0.0674
in-hash-number           1000k               0.0877           0.0675
in-hash-string           10k                 0.1174           0.0867
in-hash-string           100k                0.1188           0.0878
in-hash-string           1000k               0.1174           0.0853
in-skiplist-number       10k                 0.1095           0.0849
in-skiplist-number       100k                0.1110           0.0873
in-skiplist-number       1000k               0.1106           0.0910
in-skiplist-string       10k                 0.1744           0.1315
in-skiplist-string       100k                0.1990           0.1594
in-skiplist-string       1000k               0.2369           0.1968

It looks like joins and IN list lookups got significantly faster in 2.8, whereas the performance for point lookups is more or less the same as in 2.7.

Note that the changes to the index code in 2.8 only affect how indexes are accessed from within AQL queries and how filtering works. No changes have been made for other index operations such as insert, updates, and removals.

The new deadlock detection mechanism will kick in automatically when it detects operations that are mutually waiting for each other. In case it finds such deadlock, it will abort one of the operations so that the others can continue and overall progress can be made.

In previous versions of ArangoDB, deadlocks could make operations wait forever, requiring the server to be stopped and restarted.

How deadlocks can occur

Here’s a simple example for getting into a deadlock state:

Transaction A wants to write to collection c1 and to read from collection c2. In parallel, transaction B wants to write to collection c2 and read from collection c1. If the sequence of operations is interleaved as follows, then the two transactions prevent each other from making progress:

• transaction A successfully acquires write-lock on c1
• transaction B sucessfull acquires write-lock on c2
• transaction A tries to acquire read-lock on c2 (and must wait for B)
• transaction B tries to acquire read-lock on c1 (and must wait for A)

Here’s these such two transactions being started from two ArangoShell instances in parallel (left is A, right is B):

(note that this screenshot is from 2.8 and the automatic deadlock detection had already detected the deadlock and aborted one of the transactions)

In general, deadlocks can occur only when multiple operations (AQL queries or other transactions) try to access the same resources (collections) at the same time, and only if the operations already have already acquired some locks on these resources. And finally each operation needs to involve more than one collection, so there is the potential for already having acquired some locks but having to wait for others.

Dynamically added collections

Most operations will just work fine and will not cause any deadlocks. This is especially true for all operations that involve only a single collection. This leaves multi-collection AQL queries and multi-collection userland transactions.

Normally these will also work fine. This is because when a query or transaction starts, it will tell the transaction manager about the resources (collections) it will need. The transaction manager can then acquire the required resources in a deterministic fashion that prevents deadlocks. If all queries and transactions properly announce upfront which collections they will access, there will also be no deadlocks.

But for some operations its hard to predict at transaction start which collections will be accessed. This includes some AQL functions that can dynamically access collection data without having to specify the collection name anywhere in the query.

A good example for this is the GRAPH_EDGES AQL function, which will get a graph name as its first input parameter, but not the names of the underlying edge collection(s). When this function is used in an AQL query, the query parser will just find a function parameter containing a graph name but doesn’t know it’s a collection name.

1

GRAPH_EDGES("myEdges", [ { type: "friend" } ])

The "myEdges" graph name will look like any other string to the parser. It does not know about the contexts in which strings may have special meanings.

Note that even if this would be fixed, the problem won’t go away entirely: a function call parameter in a query isn’t necessarily a constant but can be an arbitrary expression:

1
2

FOR doc IN collection
  RETURN GRAPH_EDGES(CONCAT(doc.graphName, '-test'), [ doc.example ])

At least in this case the AQL query parser won’t find a collection name, so when the AQL query starts it is yet unknown which collections will be accessed. Only at runtime when the function is actually executed, the collection names will be looked up by finding the graph description in the _graphssystem collection. Then the edge collections participating in the graph will be added to the query dynamically. Only this dynamic addition adds the potential for deadlock.

This dynamic addition of collections in unavoidable for conveniently querying data from collections whose names are unknown when the query starts.

Deadlock detection

Whenever transaction manager detects a deadlock in ArangoDB 2.8, it will automatically abort one of the blocking transactions. The transaction will be rolled back and all modifications it has made will be reverted. The operation will fail with error code 29 (deadlock detected) and raise an exception that the user can handle in the calling code.

Deadlocks will be found if two transactions mutually lock each other as seen in the screenshot above, but also for more complex setups. The following screenshot shows four parallel transactions that block each other indirectly.

The top left window (transaction 1) will block the one in the top right (transaction 2), and is itself blocked by the transaction in the bottom left (transaction 3).

The transaction in the top right window (transaction 2) blocks the one in the bottom left (transaction 3), and is itself blocked by the one in the top left (transaction 1).

Transaction 3 (bottom left) is blocked by transaction 2 (top right). Transaction 4 (bottom right) does exactly the same as transaction 3.

With these transactions, we end up in this waiting state:

• T1 waits for T3 and T4
• T2 waits for T1
• T3 waits for T2
• T4 waits for T2

This waiting state is cyclic (T1 < T3 < T2 < T1) and therefore no progress can be made. This is exactly a situation in which the transaction manager will abort one of the transactions.

No configuration is required for the deadlock detection mechanism. It will always be active and cannot be configured or turned off.

So far the AQL editor only supported using query string literals, but it lacked support for bind parameters. Queries issued by application code however often will use bind parameters for security reasons. Often enough this prevented copying & pasting queries from the application code into the AQL editor and vice versa without making manual adjustments.

This has been fixed in the upcoming ArangoDB version 2.8 (currently in development). Bind parameters can now be used in the AQL editor as well. Bind parameters can be entered as JSON values, the same format that is used for bind parameters in the HTTP REST API and in (JavaScript) application code.

The queries can also be saved in the AQL editor along with their bind parameter values for later reuse.

Screenshot from the feature in 2.8:

The easiest way to prevent memory leaks is to create all objects on the stack and not using dynamic memory at all. However, often this is not possible, for example because stack size is limited or objects need to outlive the caller’s scope.

Another way to prevent memory leaks and leaks of other resources is obviously to employ the RAII pattern. How can it be used safely and easily in practice, so memory leaks can be avoided?

This post will start with a few seemingly working but subtly ill-formed techniques that a few common pitfalls. Later on it will provide a few very simple solutions for getting it right.

None of the solutions here are new or original.

I took some inspiration from the excellent constructor failures GotW post. That doesn’t cover smart pointers and is not explicitly about preventing preventing memory leak, so I put together this overview myself.

Naive implementation

Let’s pretend we have a simple test program main.cpp, which creates an object of class MyClass on the stack like this:

1
2
3
4
5
6
7
8
9
10
11
12

#include <iostream>
#include "MyClass.h"

int main () {
  try {
    MyClass myClass;
    std::cout << "NO EXCEPTION" << std::endl;
  }
  catch (...) {
    std::cout << "CAUGHT EXCEPTION" << std::endl;
  }
}

The above code creates the myClass instance on the stack, so itself will not leak any memory. When the creating of the myClass instance fails for whatever reason, the instance newer existed so the memory for holding a MyClass object will be freed automatically. If object creation succeeds and the object goes out of scope at the end of the try block, then the object’s destructor will be called and resources can be freed, too.

Obviously this is already good, so let’s keep it as it is and have a look at the implementation of MyClass now. This class will manage two heap objects of type A, which are created using the helper function createInstance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#include <iostream>
#include "A.h"

struct MyClass {
  A* a1;
  A* a2;

  MyClass ()
    : a1(createInstance()),
      a2(createInstance()) {

    std::cout << "CTOR MYCLASS" << std::endl;
  }

  ~MyClass () {
    std::cout << "DTOR MYCLASS" << std::endl;
    delete a1;
    delete a2;
  }
};

For completeness, here is class A. It won’t manage any resources itself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#include <iostream>

struct A {
  A () {
    std::cout << "CTOR A" << std::endl;
  }
  ~A () {
    std::cout << "DTOR A" << std::endl;
  }
};

// helper method for creating an instance of A
A* createInstance (bool shouldThrow = false) {
  if (shouldThrow) {
    throw "THROWING AN EXCEPTION";
  }
  return new A;
}

During this complete post, the code of A.h will remain unchanged.

Compiling and running the initial version of main.cpp will produce the following output:

1
2
3
4
5
6
7

CTOR A
CTOR A
CTOR MYCLASS
NO EXCEPTION
DTOR MYCLASS
DTOR A
DTOR A

Valgrind also reports no memory leaks. Are we done already?

Introducing exceptions

No, because everything still went well. Let’s introduce exceptions into the picture and check what happens then.

Let’s first introduce an exception in the constructor of MyClass. We’ll make the createInstance function throw on second invocation (we do this by passing a value of true to it):

1
2
3
4
5
6

MyClass ()
  : a1(createInstance()),
    a2(createInstance(true)) {

  std::cout << "CTOR MYCLASS" << std::endl;
}

Running the program will now emit the following:

1
2

CTOR A
CAUGHT EXCEPTION

As we’re throwing in the initializer list already, we don’t even reach the constructor body. This is no problem, but worse is that the destructor for class MyClass is not being called at all. Valgrind therefore reports the memory for first A instance as leaked.

By the way, the destructor for the MyClass instance is intentionally not being called as the object hasn’t been fully constructed and logically never existed.

Will it help if we move the heap allocations from the initializer list into the constructor body like this?

1
2
3
4
5

MyClass () {
  std::cout << "CTOR MYCLASS" << std::endl;
  a1 = createInstance();
  a2 = createInstance(true);
}

Unfortunately not. Still no destructor invocations:

1
2
3

CTOR MYCLASS
CTOR A
CAUGHT EXCEPTION

Remember: an object’s destructor won’t be called if its constructor threw and the exception wasn’t caught. That also means releasing an object’s resources solely via the destructor as in implementation above will not be sufficient if resources are allocated in the constructor and the constructor can throw.

What can be done about that?

Obviously all resource allocations can be moved into the constructor body so exceptions can be caught there:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

MyClass () {
  std::cout << "CTOR MYCLASS" << std::endl;
  a1 = createInstance();

  try {
    a2 = createInstance(true);
  }
  catch (...) {
    // must clean up a1 to prevent a leak
    delete a1;
    // and re-throw the exception
    throw;
  }
}

While the above will work, it’s clumsy, verbose and error-prone. If more objects need to be managed this will make us end up in deeply nested try…catch blocks.

try…catch for the initializer list

But wait, wasn’t there a try…catch feature especially for initializer list code? Sounds like it could be useful. Maybe we can use this instead so we can catch exceptions during initialization?

There is indeed something like that: exceptions thrown from the initializer list can be caught using the following special syntax:

1
2
3
4
5
6
7
8
9
10

MyClass ()
  try : a1(createInstance()),
        a2(createInstance(true)) {

    std::cout << "CTOR MYCLASS" << std::endl;
  }
  catch (...) {  // catch block for initializer list code
    std::cout << "CATCH BLOCK MYCLASS" << std::endl;
    delete a1;
  }

Running the program with the above MyClass constructor will also do what is expected: when creating the second A instance, the initializer list code will throw, invoking its catch block. Again code execution won’t make it into the constructor body, and we don’t see the destructor code in action.

The output of the program is:

1
2
3
4

CTOR A
CATCH BLOCK MYCLASS
DTOR A
CAUGHT EXCEPTION

Valgrind does not report a leak, so are we done now?

No, as the above code has a severe problem. It worked only because we knew the second invocation of createInstance would fail.

But in the general case, either the first call or the second call can fail. If the first call fails, then the initializer hasn’t initialized any of the object’s members, and it would be unsafe to delete any object members in the initializer’s catch block. If the second createInstance call fails, then the initializer has created a1 but not a2. To prevent a leak in this case, we should delete a1, but we better don’t delete a2 yet.

But how do we tell in the catch block at what stage the initializer list had thrown? There is no natural way to do this correctly without introducing more state. And without that, we have the choice between undefined behavior when deleting the not-yet-initialized object members, and memory leaks when ignoring them.

Not using pointers at all

Note that if we wouldn’t have used pointers for our managed A objects, then we could have used the fact that destructors for all initialized object members are actually called when object construction fails.

However, simple pointers don’t have a destructor, so the objects they point to remain and the memory is lost.

So one obvious solution for preventing memory leaks is to not use pointers, and get rid of all new and delete statements.

In some situations we can probably get away with making the managed objects regular class members of the class that manages them:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

struct MyClass {
  A a1; // no pointer anymore!
  A a2; // no pointer anymore!

  MyClass ()
    : a1(),
      a2() {

    std::cout << "CTOR MYCLASS" << std::endl;
  }

  ~MyClass () {
    std::cout << "DTOR MYCLASS" << std::endl;
    // no delete statements needed anymore!
  }
};

Now if any of the A constructors will throw an exception during initialization, everything will be cleaned up properly. Now we can make use of the destructor of A. If A instances are not pointers but regular objects, the destructors for already created instances will be called normally, and no destructors will be called for the not-yet-initialized A instances. That’s how it should be. We don’t get this benefit with regular pointers, which don’t have a destructor.

As an aside, we got rid of the delete statements in the destructor and may even get away with the default destructor.

Obviously this is an easy and safe solution, but it also has a few downsides. Here are a few (incomplete list):

• when compiling MyClass, the compiler will now need to know the definition for class A. You can’t get away with a simple forward declaration for class A anymore as in the case when the class only contained pointers to A. So this solution increases the source code dependencies and coupling.
• instances of managed objects (e.g. A) will need to be created when the managing object (e.g. MyClass) is created. There is no way to postpone the object creation as in the case of when using pointers.
• in general, the lifetime of the managed objects is tied to the lifetime of the managing object. This may or may not be ok, depending on requirements.

Using smart pointers (e.g. std::unique_ptr)

In many cases the superior alternative to all the above is using one of the available smart pointer classes for managing resources.

The promise of smart pointers is that resource management becomes easier, safer and more flexible with them.

Really useful smart pointers (this excludes std::auto_ptr) are part of standard C++ since C++11, and to my knowledge they can be used in all C++11-compatible compilers and even in some older ones. Apart from that, smart pointers are available in Boost for a long time already.

In the following snippets, I’ll be using smart pointers of type std::unique_ptr as it is the perfect fit for this particular problem. I won’t cover shared_ptr, weak_ptr or other types of smart pointers here.

When using an std::unique_ptr for managing the resources of MyClass, the MyClass code becomes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#include <memory>

struct MyClass {
  std::unique_ptr<A> a1;
  std::unique_ptr<A> a2;

  MyClass () :
    a1(createInstance()),
    a2(createInstance(true)) {

    std::cout << "CTOR MYCLASS" << std::endl;
  }

  ~MyClass () {
    std::cout << "DTOR MYCLASS" << std::endl;
  }
};

With a unique_ptr, we can still create resources when needed, either in the initializer list, the constructor or even later. The resources can still be created dynamically using new (as is still done by function createInstance). When we’re not taking the resources away from the unique_ptrs, then they will free their managed objects automatically and safely. We don’t need to bother with delete.

And we don’t need to bother with nested try…catch blocks either. If anything goes wrong during object creation, any already assigned unique_ptrs will happily release the resources they manage in their own destructors.

It does not matter if the above code throws an exception in the first invocation of createInstance, in the second or not at all: in every case any allocated resources are released properly, and still there is no need for any explicit exception handling or cleanup code. This is what a smart pointer will do for us, behind the scenes.

Simply compare the following two code snippets, which both create three instances of A while making sure no memory will be leaked if the initialization goes wrong:

1
2
3
4
5
6
7
8
9

std::unique_ptr<A> a1(createInstance());
std::unique_ptr<A> a2(createInstance());
std::unique_ptr<A> a3(createInstance());

// now do something with a1, a2, a3
// managed objects will be released automatically when
// the unique_ptrs go out of scope
// note: they may go out of scope unintentionally if
// some code below will throw an exception...

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

A* a1 = nullptr;
A* a2 = nullptr;
A* a3 = nullptr;

a1 = new A;
try {
  a2 = new A;
  try {
    a3 = new A;
  }
  catch (...) {
    delete a2;
    throw;
  }
}
catch (...) {
  delete a1;
  throw;
}

// now do something with a1, a2, a3
// objects a1, a2, a3 will not be released automatically
// when a1, a2, a3 go out of scope. any user of a1, a2, a3
// below must make sure to release the objects when they
// go out of scope or when an exception is thrown...

Obviously the smart pointer-based solution is less verbose, but it is also safer and hard to get wrong. It is especially useful for initializing and managing dynamically allocated object members, because as we’ve seen most of the other ways to do this are either subtly broken or much more complex.

Apart from that, we can take the managed object from out of a unique_ptr and take over responsibility for managing its lifetime.

Further on the plus side, a class definition that contains unique_ptrs can be compiled with only forward declarations for the managed types. However, when the unique_ptr is a regular object member, at least the class destructor implementation will need to know the size of the managed type so it can call delete properly.

The downside of using smart pointers is that they may impose minimal overhead when compared to the pure pointer-based solution. However in most cases this overhead should be absolutely negligible or even be optimized away by the compiler. It may make a difference though when compiling without any optimizations, but this shouldn’t matter too much in reality.

The 2.7 version of the PHP driver now supports the AQL query results cache. The cache can be turned on or off globally, or be set to demand mode. The demand mode will allow controlling caching on a per-AQL-query basis.

Additionally, the HTTP transport layer in the PHP driver was improved. Some internal string handling methods were optimized so that the transport part becomes cheaper. All driver operations that communicate with the ArangoDB server will benefit from this.

For a demonstration of the improvements, here is a script that creates 100,000 documents in a local ArangoDB database via the PHP driver. As we’re interested in assessing the HTTP layer improvements, the script intentionally issues 100,000 HTTP requests instead of using the specialized import method provided by the driver.

The script code can be found here.

The baseline for the improvments is the (non-optimized) 2.6 version of the PHP driver. Here are the results for issuing 100,000 requests with the 2.6 driver (script was run twice to see if there are variations in execution time):

1
2
3
4
5

creating 100000 documents
creating documents took 55.144556999207 s

creating 100000 documents
creating documents took 54.476955890656 s

Running it with the 2.7 version of the PHP driver now shows the improvements. Execution time for the same script goes down from 54 seconds to 42 seconds:

1
2
3
4
5

creating 100000 documents
creating documents took 42.886090040207 s

creating 100000 documents
creating documents took 42.578990936279 s

The PHP version used here was:

1
2
3
4

PHP 5.5.12-2ubuntu4.6 (cli) (built: Jul  2 2015 15:27:14) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies

Following are the results from a different machine, this time using PHP 5.6:

1
2
3
4
5

creating 100000 documents
creating documents took 48.394731044769 s

creating 100000 documents
creating documents took 47.618598937988 s

1
2
3
4
5

creating 100000 documents
creating documents took 40.535583972931 s

creating 100000 documents
creating documents took 40.041265010834 s

The PHP version details for this machine were:

1
2
3
4

PHP 5.6.4-4ubuntu6.2 (cli) (built: Jul  2 2015 15:29:28) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies

The actual improvements depend on many factors, so your exact mileage may vary. The improvements may not be noticable for applications that issue only a few requests with the driver, but they will be significant when performing lots of requests, as in the above examples.