\label{doc_tutorials_benchmarking_md_md_doc_tutorials_benchmarking}%
\Hypertarget{doc_tutorials_benchmarking_md_md_doc_tutorials_benchmarking}%
 \hypertarget{doc_tutorials_benchmarking_md_autotoc_md4208}{}\doxysection{Execution Time}\label{doc_tutorials_benchmarking_md_autotoc_md4208}
One of the usual questions for a standard benchmark is, how much more or less time two or more different programs take to execute on the same hardware. This tutorial will introduce some tools and techniques that will help you to answer this question. For that purpose we compare the time it takes for certain YAML \mbox{\hyperlink{doc_tutorials_storage-plugins_md}{storage plugins}} to translate YAML data into Elektra’s key set structure. Most of the techniques we describe here should be applicable too, if you want to compare the run-\/time of other parts of Elektra. If you want to know why a certain part of Elektra takes a long time to execute, then you might also be interested in the \mbox{\hyperlink{doc_tutorials_profiling_md}{profiling tutorial}}.\hypertarget{doc_tutorials_benchmarking_md_autotoc_md4209}{}\doxysubsection{Translating Elektra}\label{doc_tutorials_benchmarking_md_autotoc_md4209}
If you have never translated the code base of Elektra before, then please take a look \mbox{\hyperlink{doc_COMPILE_md}{here}} before you continue.

Usually you want to compare the execution time of the fastest version of a compiled binary. For that purpose it makes sense to change the CMake build type to {\ttfamily Release}, which means that the generated build system will optimize the code and strip debug symbols. You should also disable the logger and debug code. An example CMake command that uses \href{https://ninja-build.org}{\texttt{ Ninja}} as build tool could look like this\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{mkdir build}
\DoxyCodeLine{cd build}
\DoxyCodeLine{cmake -\/GNinja ..                 \(\backslash\)}
\DoxyCodeLine{      -\/DCMAKE\_BUILD\_TYPE=Release \(\backslash\)}
\DoxyCodeLine{      -\/DENABLE\_LOGGER=OFF        \(\backslash\)}
\DoxyCodeLine{      -\/DENABLE\_DEBUG=OFF         \(\backslash\)}
\DoxyCodeLine{      -\/DPLUGINS=ALL}
\DoxyCodeLine{ninja}
\DoxyCodeLine{cd .. \# Change working directory back to the root of repository}

\end{DoxyCode}
\hypertarget{doc_tutorials_benchmarking_md_autotoc_md4210}{}\doxysubsection{Using the Plugin Benchmark Helper Tool}\label{doc_tutorials_benchmarking_md_autotoc_md4210}
Elektra already includes a tool that helps you to benchmark the {\ttfamily get} and {\ttfamily set} methods of a certain \mbox{\hyperlink{doc_tutorials_plugins_md}{plugin}} called \`{}benchmark\+\_\+plugingetset\`{}. To show you how to use {\ttfamily benchmark\+\_\+plugingetset}, we create a file named {\ttfamily test.\+yamlcpp.\+in} with the following content\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{-\/ You,}
\DoxyCodeLine{-\/ Me, \&}
\DoxyCodeLine{-\/ The Violence}

\end{DoxyCode}


and save it in the folder {\ttfamily benchmarks/data}\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{mkdir -\/p benchmarks/data}
\DoxyCodeLine{printf -\/-\/ '-\/ You,\(\backslash\)n'       >  benchmarks/data/test.yamlcpp.in}
\DoxyCodeLine{printf -\/-\/ '-\/ Me, \&\(\backslash\)n'      >> benchmarks/data/test.yamlcpp.in}
\DoxyCodeLine{printf -\/-\/ '-\/ The Violence' >> benchmarks/data/test.yamlcpp.in}

\end{DoxyCode}


. As you can see the filename has to use the pattern\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{test.\$plugin.in}

\end{DoxyCode}


, where {\ttfamily \$plugin} specifies the name of the plugin the benchmark tool should call. We can now call the {\ttfamily get} method of the plugin \mbox{\hyperlink{md_src_plugins_yamlcpp_README}{YAML CPP}} using the following shell command


\begin{DoxyCode}{0}
\DoxyCodeLine{build/bin/benchmark\_plugingetset benchmarks/data      user    yamlcpp       get}
\DoxyCodeLine{\#                                       ↑              ↑        ↑           ↑}
\DoxyCodeLine{\#                                parent directory  namespace  plugin   only use `get`}
\DoxyCodeLine{\#                                 of config file                       plugin method}

\end{DoxyCode}


. If you can want you can also use the {\ttfamily time} utility to measure the execution time of the last command\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{time build/bin/benchmark\_plugingetset benchmarks/data user yamlcpp get}
\DoxyCodeLine{\#>       0.00 real         0.00 user         0.00 sys}

\end{DoxyCode}


. As you can see in the output above a real configuration file that tests the performance of the \mbox{\hyperlink{md_src_plugins_yamlcpp_README}{YAML CPP}} plugin should be much larger.\hypertarget{doc_tutorials_benchmarking_md_autotoc_md4211}{}\doxysubsection{Comparing Execution Times}\label{doc_tutorials_benchmarking_md_autotoc_md4211}
Now that you know how to execute {\ttfamily benchmark\+\_\+plugingetset}, you can use it to compare the performance of different plugins. Since you usually want


\begin{DoxyItemize}
\item to run {\ttfamily benchmark\+\_\+plugingetset} multiple times, and
\item compare different plugins
\end{DoxyItemize}

it makes sense to use a benchmarking tool such as \href{https://github.com/sharkdp/hyperfine}{\texttt{ hyperfine}} for that task. For our tutorial we assume that you copied the file \href{https://github.com/ElektraInitiative/rawdata/blob/master/YAML/Input/keyframes.yaml}{\texttt{ {\ttfamily keyframes.\+yaml}}} to the locations


\begin{DoxyItemize}
\item {\ttfamily benchmarks/data/test.\+yamlcpp.\+in}, and
\item {\ttfamily benchmarks/data/test.\+yanlr.\+in}
\end{DoxyItemize}

. You can do that using the following commands\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{mkdir -\/p benchmarks/data}
\DoxyCodeLine{curl -\/L https://github.com/ElektraInitiative/rawdata/raw/master/YAML/Input/keyframes.yaml -\/o benchmarks/data/test.yamlcpp.in}
\DoxyCodeLine{cp benchmarks/data/test.yamlcpp.in benchmarks/data/test.yanlr.in}

\end{DoxyCode}


. Afterwards you can use\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{hyperfine -\/-\/warmup 3 'build/bin/benchmark\_plugingetset benchmarks/data user yamlcpp get' \(\backslash\)}
\DoxyCodeLine{                     'build/bin/benchmark\_plugingetset benchmarks/data user yanlr get'}

\end{DoxyCode}


to compare the performance of the plugins. The output of this benchmark would look something like this\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{Benchmark \#1: build/bin/benchmark\_plugingetset benchmarks/data user yamlcpp get}
\DoxyCodeLine{  Time (mean ± σ):      18.3 ms ±   0.9 ms    [User: 15.2 ms, System: 1.4 ms]}
\DoxyCodeLine{  Range (min … max):    17.1 ms …  21.2 ms    136 runs}
\DoxyCodeLine{}
\DoxyCodeLine{Benchmark \#2: build/bin/benchmark\_plugingetset benchmarks/data user yanlr get}
\DoxyCodeLine{  Time (mean ± σ):      16.2 ms ±   0.9 ms    [User: 12.7 ms, System: 1.7 ms]}
\DoxyCodeLine{  Range (min … max):    14.8 ms …  20.0 ms    161 runs}
\DoxyCodeLine{}
\DoxyCodeLine{Summary}
\DoxyCodeLine{  'build/bin/benchmark\_plugingetset benchmarks/data user yanlr get' ran}
\DoxyCodeLine{    1.13 ± 0.08 times faster than 'build/bin/benchmark\_plugingetset benchmarks/data user yamlcpp get'}

\end{DoxyCode}


. You can now remove the input files and the folder {\ttfamily benchmarks/data}\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{rm benchmarks/data/test.yamlcpp.in}
\DoxyCodeLine{rm benchmarks/data/test.yanlr.in}
\DoxyCodeLine{rmdir benchmarks/data}

\end{DoxyCode}
