Benchmarking and evaluating with ups_bench

upscaledb comes with a handy tool “ups_bench” which is used for benchmarking and testing a myriad of configurations and data patterns. If you wonder which database configuration is the fastest for you then simply fire up ups_bench, describe your data and your operations through a few command line parameters and look at the results (or the generated image files).

upscaledb’s performance depends on its configuration, the different APIs that you use, whether your application mainly concentrates on reading or writing (or deleting) data, and what kind of data you store. This document describes the various options that you can use to run the tests.

ups_bench usage

Unix and Windows users find ups_bench in the tools/ups_bench directory; the executable file is called ups_bench (or ups_bench.exe). Simply execute it and it will insert 1 million random keys, each key with a size of 16 bytes and each record 1024 bytes large. A progress bar is shown, and at the end a few statistical results are printed to screen:

upscaledb 2.1.11 - Copyright (C) 2005-2015 Christoph Rupp (chris@crupp.de).

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Configuration: --seed=1381063435 --keysize=16 --recsize=1024
--distribution=random 

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************

[OK] 
total elapsed time (sec)                 11.239185
upscaledb elapsed time (sec)             9.453202
upscaledb total_#ops                     1000001
upscaledb insert_#ops                    999999 (105784.155906/sec)
upscaledb insert_throughput              95224190.254546/sec
upscaledb insert_latency (min, avg, max) 0.000000, 0.000009, 0.255820
upscaledb filesize                       1016430592
                

Depending on your input, the data is created with a Random Number Generator (RNG). To get reproducible results you need to specify a “seed” value for the RNG. If you do not specify a seed then upscaledb picks one for you and prints it to stdout (see above). If you tune the options then just specify this seed and upscaledb will use the identical data as in the previous run.

A word of warning – never use the Debug build of upscaledb if you want to run benchmarks. The debug build is slower by magnitudes and will scale terribly.

Try running ./ups_bench --help to get a full overview about the various options!

Data layout

upscaledb 2.2.x supports basic schema information, and ups_bench exposes these parameters. The following options describe the key’s type:

--key=binary                       binary data (this is the default)
--key=string                       concatenated ascii strings from /usr/share/dict
--key=custom                       binary data, but uses a user-supplied callback function to compare keys
--key=uint8                        unsigned integer, 8 bits (1 byte)
--key=uint16                       unsigned integer, 16 bits (2 byte)
--key=uint32                       unsigned integer, 32 bits (4 byte)
--key=uint64                       unsigned integer, 64 bits (8 byte)
                

For custom, string and binary key types you can specify the additional key size with the --keysize parameter.

The record size can be specified with --recsize. The default is 1024 bytes.

Data distribution

The ups_bench tool supports several data patterns – random data, ascending or descending data. The default data distribution is “random”.

--distribution=random              random data is generated (the default)
--distribution=ascending           data is ascending (0, 1, 2, ...)
--distribution=descending          data is descending (999, 998, 997, ...)
                

Data access patterns

Applications have different patterns when accessing data; some mostly write data, but never delete any. Others insert some data but often look up keys or run full table scans. Others even might have different stages where they bulk load, and then look up or scan exclusively.

By default, upscaledb only inserts data. Here are the options to control this behavior:

--erase-pct=<n>                    <n> % of all operations are deletes/erase operations
--find-pct=<n>                     <n> % of all operations are lookup operations
--table-scan-pct=<n>               <n> % of all operations are full table scans
                

If you want to simulate separate application runs with a bulk load phase and a lookup phase, then first generate the file by running ups_bench, then run it again and specify --reopen:

./ups_bench --keysize=32 # generates the file by bulk loading with data
./ups_bench --keysize=32 --find-pct=50 --erase-pct=10 --reopen  # opens the generated file; 50% of all operations are lookups, 10% are deletes, the remaining 40% are additional inserts
                

Limiting the test data

By default, ups_bench stops after 1 million database operations. You can change this limit, or use a different limit, either by specifying the amount of inserted bytes or by setting a time limit:

--stop-ops=<n>                     stops after <n> database operations (inserts, erase, lookups, scans)
--stop-seconds=<n>                 stops after <n> seconds
--stop-bytes=<n>                   stops after inserting <n> bytes (key data + record data)
                

Additional configuration settings

--duplicate=first                  enables duplicate
keys and inserts duplicates at the *beginning* of the duplicate list. Requires
`--use-cursors`!
--duplicate=last                   enables duplicate keys and inserts duplicates at the *end* of the duplicate list; this is the default duplicate behavior.
--overwrite                        overwrites existing keys
--no-mmap                          disables use of mmap
--pagesize=<n>                     sets the pagesize (in bytes)
--cache=<n>                        sets the cache size (in bytes)
--cache=unlimited                  uses an unlimited cache size
--direct-access                    uses the flag UPS_DIRECT_ACCESS. Requires `--inmemorydb`
--use-transactions=tmp             enables transactions and runs each database operation in a temporary transaction
--use-transactions=all             enables transactions and runs *all* database operations in one single transaction
--use-transactions=n               enables transactions and groups <n> database operations in a transaction
--flush-txn-immediately            does not buffer Transactions before they are committed (`UPS_FLUSH_WHEN_COMMITTED`)
--disable-recovery                 disables recovery; used in combination with Transactions
--inmemorydb                       runs upscaledb in memory only
--use-fsync                        uses fsync after transactions are committed
--use-recovery                     uses recovery, even if transactions are disabled
--use-remote                       enables the remote client/server
--use-cursors                      uses a database cursor for all operations
                

Metrics

You can also specify output parameters which track the performance and other various metrics. Internally, ups_bench measures the wall-clock time of each database operation, and prints a summary about throughput and latency when done:

total elapsed time (sec)                 0.747461
upscaledb elapsed time (sec)             0.099174
upscaledb total_#ops                     10001
upscaledb insert_#ops                    4046 (70337.495855/sec)
upscaledb insert_throughput              70224427.374380/sec
upscaledb insert_latency (min, avg, max) 0.000002, 0.000014, 0.000124
upscaledb find_#ops                      2972 (140719.583701/sec)
upscaledb find_throughput                4121208.803951/sec
upscaledb find_latency (min, avg, max)   0.000002, 0.000007, 0.000093
upscaledb erase_#ops                     2981 (145190.417647/sec)
upscaledb erase_latency (min, avg, max)  0.000002, 0.000007, 0.000049
upscaledb filesize                       4358144
                

If you’re interested in even more details then run with --metrics=all and you will receive a myriad of information including cache hits, cache misses, but also two .png graphic files with a visualization of the latencies and the throughput (the images are only generated if gnuplot is installed).

Examples

Simulating an application with the following data access patterns: inserts = 60%, lookups = 30%, deletes=10%; keys are 32 bytes random binary data, records are 128 bytes:

ups_bench --find-pct=30 --erase-pct=10 --keysize=32 --recsize=128
                

Same as above, but runs the test in memory:

ups_bench --find-pct=30 --erase-pct=10 --keysize=32 --recsize=128 --inmemorydb
                

… or with a cache size of 512 mb:

ups_bench --find-pct=30 --erase-pct=10 --keysize=32 --recsize=128 --cache=536870912
                

Use 64bit numeric keys and 8 byte records with an ascending distribution (this is more or less equal to a UPS_RECORD_NUMBER database):

ups_bench --key=uint64 --recsize=8 --distribution=ascending
                

Bulk load the data, groups 10 inserts into a single transaction; enables duplicate keys:

ups_bench --use-transactions=10 --duplicate=last