.TH "doc_tutorials_benchmarking_md" 3elektra "Fri Aug 4 2023" "Version 0.11.0" "Elektra" \" -*- nroff -*-
.ad l
.nh
.SH NAME
doc_tutorials_benchmarking_md \- Benchmarking 
 
.SH "Execution Time"
.PP
One of the usual questions for a standard benchmark is, how much more or less time two or more different programs take to execute on the same hardware\&. This tutorial will introduce some tools and techniques that will help you to answer this question\&. For that purpose we compare the time it takes for certain YAML \fBstorage plugins\fP to translate YAML data into Elektra’s key set structure\&. Most of the techniques we describe here should be applicable too, if you want to compare the run-time of other parts of Elektra\&. If you want to know why a certain part of Elektra takes a long time to execute, then you might also be interested in the \fBprofiling tutorial\fP\&.
.SS "Translating Elektra"
If you have never translated the code base of Elektra before, then please take a look \fBhere\fP before you continue\&.
.PP
Usually you want to compare the execution time of the fastest version of a compiled binary\&. For that purpose it makes sense to change the CMake build type to \fCRelease\fP, which means that the generated build system will optimize the code and strip debug symbols\&. You should also disable the logger and debug code\&. An example CMake command that uses \fCNinja\fP as build tool could look like this:
.PP
.PP
.nf
mkdir build
cd build
cmake -GNinja \&.\&.                 \
      -DCMAKE_BUILD_TYPE=Release \
      -DENABLE_LOGGER=OFF        \
      -DENABLE_DEBUG=OFF         \
      -DPLUGINS=ALL
ninja
cd \&.\&. # Change working directory back to the root of repository
.fi
.PP
.SS "Using the Plugin Benchmark Helper Tool"
Elektra already includes a tool that helps you to benchmark the \fCget\fP and \fCset\fP methods of a certain \fBplugin\fP called \fB`benchmark_plugingetset`\fP\&. To show you how to use \fCbenchmark_plugingetset\fP, we create a file named \fCtest\&.yamlcpp\&.in\fP with the following content:
.PP
.PP
.nf
- You,
- Me, &
- The Violence
.fi
.PP
.PP
and save it in the folder \fCbenchmarks/data\fP:
.PP
.PP
.nf
mkdir -p benchmarks/data
printf -- '- You,\n'       >  benchmarks/data/test\&.yamlcpp\&.in
printf -- '- Me, &\n'      >> benchmarks/data/test\&.yamlcpp\&.in
printf -- '- The Violence' >> benchmarks/data/test\&.yamlcpp\&.in
.fi
.PP
.PP
\&. As you can see the filename has to use the pattern:
.PP
.PP
.nf
test\&.$plugin\&.in
.fi
.PP
.PP
, where \fC$plugin\fP specifies the name of the plugin the benchmark tool should call\&. We can now call the \fCget\fP method of the plugin \fBYAML CPP\fP using the following shell command
.PP
.PP
.nf
build/bin/benchmark_plugingetset benchmarks/data      user    yamlcpp       get
#                                       ↑              ↑        ↑           ↑
#                                parent directory  namespace  plugin   only use `get`
#                                 of config file                       plugin method
.fi
.PP
.PP
\&. If you can want you can also use the \fCtime\fP utility to measure the execution time of the last command:
.PP
.PP
.nf
time build/bin/benchmark_plugingetset benchmarks/data user yamlcpp get
#>       0\&.00 real         0\&.00 user         0\&.00 sys
.fi
.PP
.PP
\&. As you can see in the output above a real configuration file that tests the performance of the \fBYAML CPP\fP plugin should be much larger\&.
.SS "Comparing Execution Times"
Now that you know how to execute \fCbenchmark_plugingetset\fP, you can use it to compare the performance of different plugins\&. Since you usually want
.PP
.IP "\(bu" 2
to run \fCbenchmark_plugingetset\fP multiple times, and
.IP "\(bu" 2
compare different plugins
.PP
.PP
it makes sense to use a benchmarking tool such as \fChyperfine\fP for that task\&. For our tutorial we assume that you copied the file \fC\fCkeyframes\&.yaml\fP\fP to the locations
.PP
.IP "\(bu" 2
\fCbenchmarks/data/test\&.yamlcpp\&.in\fP, and
.IP "\(bu" 2
\fCbenchmarks/data/test\&.yanlr\&.in\fP
.PP
.PP
\&. You can do that using the following commands:
.PP
.PP
.nf
mkdir -p benchmarks/data
curl -L https://github\&.com/ElektraInitiative/rawdata/raw/master/YAML/Input/keyframes\&.yaml -o benchmarks/data/test\&.yamlcpp\&.in
cp benchmarks/data/test\&.yamlcpp\&.in benchmarks/data/test\&.yanlr\&.in
.fi
.PP
.PP
\&. Afterwards you can use:
.PP
.PP
.nf
hyperfine --warmup 3 'build/bin/benchmark_plugingetset benchmarks/data user yamlcpp get' \
                     'build/bin/benchmark_plugingetset benchmarks/data user yanlr get'
.fi
.PP
.PP
to compare the performance of the plugins\&. The output of this benchmark would look something like this:
.PP
.PP
.nf
Benchmark #1: build/bin/benchmark_plugingetset benchmarks/data user yamlcpp get
  Time (mean ± σ):      18\&.3 ms ±   0\&.9 ms    [User: 15\&.2 ms, System: 1\&.4 ms]
  Range (min … max):    17\&.1 ms …  21\&.2 ms    136 runs

Benchmark #2: build/bin/benchmark_plugingetset benchmarks/data user yanlr get
  Time (mean ± σ):      16\&.2 ms ±   0\&.9 ms    [User: 12\&.7 ms, System: 1\&.7 ms]
  Range (min … max):    14\&.8 ms …  20\&.0 ms    161 runs

Summary
  'build/bin/benchmark_plugingetset benchmarks/data user yanlr get' ran
    1\&.13 ± 0\&.08 times faster than 'build/bin/benchmark_plugingetset benchmarks/data user yamlcpp get'
.fi
.PP
.PP
\&. You can now remove the input files and the folder \fCbenchmarks/data\fP:
.PP
.PP
.nf
rm benchmarks/data/test\&.yamlcpp\&.in
rm benchmarks/data/test\&.yanlr\&.in
rmdir benchmarks/data
.fi
.PP