\subsection*{Introduction}

Libraries need a pervasive testing for continuous improvement. Any problem found and behaviour described must be written down as test and integrated so that after running all tests in the build directory\+: \begin{DoxyVerb}make run_all
\end{DoxyVerb}


and on the target (installed) system\+: \begin{DoxyVerb}kdb run_all
\end{DoxyVerb}


it is assured that no new regressions were added. To run memcheck tests run in the build directory\+: \begin{DoxyVerb}make run_memcheck
\end{DoxyVerb}


For tests in the build directory ctest is used, so you alternatively also can call ctest with its options. On the target (installed) system our own scripts drive the tests.

Some tests write into system paths. If the access is denied, several tests will fail. You have some options to avoid running them as root\+:


\begin{DoxyEnumerate}
\item To void running the problematic test cases (reduces the test coverage!) run ctest without tests that have the label {\ttfamily kdbtests}\+: {\ttfamily ctest -\/-\/output-\/on-\/failure -\/\+L\+E kdbtests}
\item To give your user the permissions to the relevant paths, i.\+e. (once as root)\+: ``` kdb mount-\/info chown -\/\+R {\ttfamily whoami} {\ttfamily kdb get system/info/constants/cmake/\+C\+M\+A\+K\+E\+\_\+\+I\+N\+S\+T\+A\+L\+L\+\_\+\+P\+R\+E\+F\+I\+X}/{\ttfamily kdb get system/info/constants/cmake/\+K\+D\+B\+\_\+\+D\+B\+\_\+\+S\+P\+E\+C} chown -\/\+R {\ttfamily whoami} {\ttfamily kdb get system/info/constants/cmake/\+K\+D\+B\+\_\+\+D\+B\+\_\+\+S\+Y\+S\+T\+E\+M} ``` After that all test cases should run successfully as described above.
\item Compile Elektra so that system paths are not actual system paths, e.\+g. to write everything into the home directory ({\ttfamily $\sim$}) use cmake options\+: {\ttfamily -\/\+D\+K\+D\+B\+\_\+\+D\+B\+\_\+\+S\+Y\+S\+T\+E\+M=\char`\"{}$\sim$/.\+config/kdb/system\char`\"{} -\/\+D\+K\+D\+B\+\_\+\+D\+B\+\_\+\+S\+P\+E\+C=\char`\"{}$\sim$/.\+config/kdb/spec\char`\"{}} (for another example with ini see {\ttfamily scripts/configure-\/home})
\item Use the X\+D\+G resolver (see {\ttfamily scripts/configure-\/xdg}) and set the environment variable {\ttfamily X\+D\+G\+\_\+\+C\+O\+N\+F\+I\+G\+\_\+\+D\+I\+R\+S}, currently lacks spec namespaces, see \#734.
\end{DoxyEnumerate}

\subsection*{Conventions}


\begin{DoxyItemize}
\item All names of the test must start with test (needed by test driver for installed tests).
\item No tests should run if E\+N\+A\+B\+L\+E\+\_\+\+T\+E\+S\+T\+I\+N\+G is O\+F\+F.
\item All tests that access system/spec namespaces (e.\+g. mount something)\+:
\begin{DoxyItemize}
\item should be tagged with kdbtests\+: \begin{DoxyVerb} set_property(TEST testname PROPERTY LABELS kdbtests)
\end{DoxyVerb}

\item should not run, if {\ttfamily E\+N\+A\+B\+L\+E\+\_\+\+K\+D\+B\+\_\+\+T\+E\+S\+T\+I\+N\+G} is O\+F\+F.
\item should only write below
\begin{DoxyItemize}
\item {\ttfamily /tests/$<$testname$>$} (e.\+g. {\ttfamily /tests/ruby}) and
\item {\ttfamily system/elektra} (e.\+g. for mounts or globalplugins).
\end{DoxyItemize}
\end{DoxyItemize}
\item If your test has memleaks, e.\+g. because the library used leaks and that cannot be fixed, give them the label memleak with following command\+:

set\+\_\+property(\+T\+E\+S\+T testname P\+R\+O\+P\+E\+R\+T\+Y L\+A\+B\+E\+L\+S memleak)
\end{DoxyItemize}

\subsection*{Strategy}

The testing must happen on every level of the software to achieve a maximum coverage with the available time. In the rest of the document we describe the different levels and where these tests are.

\subsubsection*{C\+Framework}

This is basically a bunch of assertion macros and some output facilities. It is written in pure C and very lightweight.

It is located here.

\subsubsection*{A\+B\+I Tests}

C A\+B\+I Tests are written in plain C with the help of cframework.

The main purpose of these tests are, that the binaries of old versions can be used against new versions as A\+B\+I tests.

So lets say we compile Elektra 0.\+8.\+8 (at this version dedicated A\+B\+I tests were introduced) in the {\ttfamily -\/full} variant. But when we run the tests, we use {\ttfamily libelektra-\/full.\+so.\+0.\+8.\+9} (either by installing it or by setting {\ttfamily L\+D\+\_\+\+L\+I\+B\+R\+A\+R\+Y\+\_\+\+P\+A\+T\+H}). You can check with ldd which version is used.

The tests are located here.

\subsubsection*{C Unit Tests}

C Unit Tests are written in plain C with the help of cframework.

It is used to test internal data structures of libelektra that are not A\+B\+I relevant.

A\+B\+I tests can be done on theses tests, too. But by nature from time to time these tests will fail.

They are located here.

\paragraph*{Internal Functions}

According to {\ttfamily src/libs/elektra/libelektra-\/symbols.\+map}, all functions starting with\+:


\begin{DoxyItemize}
\item libelektra
\item elektra
\item kdb
\item key
\item ks
\end{DoxyItemize}

get exported. Functions not starting with this prefix are internal only and therefore not visible in the test cases. Test internal functionality by including the corresponding c file.

\subsubsection*{Module Tests}

The modules, which are typically used as plugins in Elektra (but can also be available statically or in the {\ttfamily -\/full} variant), should have their own tests.

Use the Cmake macro {\ttfamily add\+\_\+plugintest} for adding these tests.

\subsubsection*{C++ Unit Tests}

C++ Unit tests are done using the gtest framework. See \hyperlink{doc_decisions_unit_testing_md}{architectural decision}.

Use the C\+Make macro {\ttfamily add\+\_\+gtest} for adding these tests.

\subsubsection*{Script Tests}

Script test are done using P\+O\+S\+I\+X shell + C\+Make. See \hyperlink{doc_decisions_script_testing_md}{architectural decision}.

The script tests have different purposes\+:
\begin{DoxyItemize}
\item End to End tests (usage of tools as a user would do)
\item External Compilation tests (compile and run programs as a user would do)
\item Conventions tests (do internal checks that check for common problems)
\item Meta Test Suites (run other test suites)
\end{DoxyItemize}

\subsubsection*{Fuzz Testing}

Copy some import files to testcase\+\_\+dir and run\+: \begin{DoxyVerb}/usr/src/afl/afl-1.46b/./afl-fuzz -i testcase_dir -o findings_dir bin/kdb import user/tests
\end{DoxyVerb}


\subsubsection*{A\+S\+A\+N}

To enable sanitize checks use {\ttfamily E\+N\+A\+B\+L\+E\+\_\+\+A\+S\+A\+N} via cmake.

Then, to use A\+S\+A\+N, run {\ttfamily run\+\_\+asan} in the build directory, which simply does\+: \begin{DoxyVerb}    ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=$(shell which llvm-symbolizer) make run_all
\end{DoxyVerb}


It could also happen that you need to preload A\+S\+A\+N library, e.\+g.\+: \begin{DoxyVerb}    LD_PRELOAD=/usr/lib/clang/3.8.0/lib/linux/libclang_rt.asan-x86_64.so run_asan
\end{DoxyVerb}


or on Debian\+: \begin{DoxyVerb}    LD_PRELOAD=/usr/lib/llvm-3.8/lib/clang/3.8.1/lib/linux/libclang_rt.asan-x86_64.so run_asan
\end{DoxyVerb}


See also build server jobs\+:


\begin{DoxyItemize}
\item \href{http://build.libelektra.org:8080/job/elektra-clang-asan/}{\tt clang-\/asan}
\item \href{http://build.libelektra.org:8080/job/elektra-gcc-asan/}{\tt gcc-\/asan}
\end{DoxyItemize}

\subsubsection*{cbmc}

For bounded model checking tests, see {\ttfamily scripts/cbmc}.

\subsubsection*{Code Coverage}

Run\+: \begin{DoxyVerb}    make coverage-start
    # now run all tests! E.g.:
    make run_all
    make coverage-stop
    make coverage-genhtml
\end{DoxyVerb}


The htmls can be found in the build directory in the folder {\ttfamily coverage}.

See also the build server job\+:


\begin{DoxyItemize}
\item \href{http://build.libelektra.org:8080/job/elektra-incremental/}{\tt gcc-\/asan}
\end{DoxyItemize}

\subsection*{See also}


\begin{DoxyItemize}
\item \hyperlink{doc_COMPILE_md}{C\+O\+M\+P\+I\+L\+E}.
\item \hyperlink{doc_INSTALL_md}{I\+N\+S\+T\+A\+L\+L}. 
\end{DoxyItemize}