\label{doc_decisions_0_drafts_change_tracking_md_md_doc_decisions_0_drafts_change_tracking}%
\Hypertarget{doc_decisions_0_drafts_change_tracking_md_md_doc_decisions_0_drafts_change_tracking}%
 \hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1716}{}\doxysection{Problem}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1716}
Currently, a range of Elektra plugins are implementing some sort of change tracking for configuration data. This includes, but is not limited to, the \mbox{\hyperlink{md_src_plugins_internalnotification_README_src_plugins_internalnotification_README_md}{internalnotification}} and \mbox{\hyperlink{md_src_plugins_dbus_README_src_plugins_dbus_README_md}{dbus}} plugins. In the near future, Elektra shall also be extended with session recording.

KDB itself also has some rudimentary change tracking (via the {\ttfamily key\+Needs\+Sync} flag) to determine whether {\ttfamily kdb\+Set} needs to actually do something.

These competing change tracking implementations create multiple problems\+:


\begin{DoxyEnumerate}
\item pointless waste of resources, as data is duplicated in each plugin,
\item multiplication of code, generating a maintenance burden.
\item various subtle differences in change tracking behavior, e.\+g., {\ttfamily kdb\+Set} might write a config file but notification is not sent out
\item the current approach to change tracking in plugins is fragile, which is outlined in \mbox{\hyperlink{doc_decisions_0_drafts_operation_sequences_md}{a separate decision about valid kdb\+Get/kdb\+Set sequences}}.
\end{DoxyEnumerate}

For {\ttfamily Key\+Set} we need to track which of the keys\+:


\begin{DoxyItemize}
\item have been removed
\item have been added
\end{DoxyItemize}

For {\ttfamily Key} we need to track\+:


\begin{DoxyItemize}
\item original value of the key
\item size of the original value (for binary keys)
\item metadata, which is a combination of the tracking of keysets and keys
\item tracking should only be done on the following namespaces\+:
\begin{DoxyItemize}
\item {\ttfamily system\+:/}
\item {\ttfamily user\+:/}
\item {\ttfamily dir\+:/}
\item {\ttfamily meta\+:/}
\item {\ttfamily spec\+:/}
\end{DoxyItemize}
\end{DoxyItemize}\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1717}{}\doxysection{Constraints}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1717}
Change tracking must\+:


\begin{DoxyItemize}
\item be transparent for applications using the public Elektra API
\item be transparent to users changing configuration data
\item if overhead is not negligible\+: only do tracking as required, i.\+e., a plugin specifically requests it
\item have negligible overhead if disabled
\item not duplicate data for each plugin that wants change tracking
\item work with all allowed sequences of {\ttfamily kdb\+Get} and {\ttfamily kdb\+Set} as \mbox{\hyperlink{doc_decisions_0_drafts_operation_sequences_md}{per this decision}}
\end{DoxyItemize}

The library {\ttfamily libelektra-\/core} must be kept minimal.\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1718}{}\doxysection{Assumptions}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1718}

\begin{DoxyItemize}
\item It is possible to do change tracking with reasonable memory and computation overhead
\item It is possible to design a single change tracking API that is useful for all existing and future plugins
\end{DoxyItemize}\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1719}{}\doxysection{Considered Alternatives}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1719}
\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1720}{}\doxysubsection{Alternative 1 -\/ Per-\/parent key tracking within $<$tt$>$libelektra-\/kdb$<$/tt$>$}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1720}
Do the per-\/parent-\/key tracking within {\ttfamily libelektra-\/kdb}, within the {\ttfamily kdb\+Get} and {\ttfamily kdb\+Set} operations.

Essentially, we\textquotesingle{}d have a \textquotesingle{}giant\textquotesingle{} hashmap in the form of (parent key)-\/$>$(Key\+Set) for all parent keys that were used for a {\ttfamily kdb\+Get} operation in the lifetime of the {\ttfamily kdb} instance. We also need to update it on {\ttfamily kdb\+Set} so that a future {\ttfamily kdb\+Set} operation without a {\ttfamily kdb\+Get} will also work.

When change tracking is enabled, this one will have the most memory overhead, as we need to deep-\/dup every key we have read or stored.\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1721}{}\doxysubsection{Alternative 2 -\/ Per-\/parent key tracking with meta keys}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1721}
Do the per-\/parent-\/key tracking within {\ttfamily libelektra-\/kdb}, but with meta keys.

Essentially the same approach as above, but instead of deep-\/duping, we add the original value as a metakey to every key. Not yet clear how we handle changes to metadata then.\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1722}{}\doxysubsection{Alternative 3 -\/ Change tracking within a separate plugin}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1722}
Outsource the change tracking into a separate plugin.

Essentially the same as (1), just that it is implemented within a plugin and not {\ttfamily libelektra-\/kdb}. This will be a hook plugin, and will be called within {\ttfamily kdb\+Get} and {\ttfamily kdb\+Set} accordingly. It will also need to export a hook-\/method to get the changeset.

The following hooks will be needed\+:


\begin{DoxyItemize}
\item {\ttfamily tracking/get}\+: wil be called at the end of {\ttfamily kdb\+Get}, directly before the result is returned.
\item {\ttfamily tracking/set/preliminary}\+: will be called at the beginning of {\ttfamily kdb\+Set} and after every step/phase in {\ttfamily kdb\+Set}. We need this to have change tracking information as soon as possible, so that plugins in any step of the process can use this information. ~\newline
 Also, plugins in every step of the process might change data, so it is important to call it after every step.
\item {\ttfamily tracking/set/final}\+: will be called in the {\ttfamily post-\/commit} phase, after all changes have been written to disk, but before the {\ttfamily notification/send} hook. Represents the final, real changes to the KDB.
\item {\ttfamily tracking/changeset}\+: compute the changeset for the requested parent key and return it.
\end{DoxyItemize}\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1723}{}\doxysubsection{Alternative 4 -\/ Copy-\/on-\/write change tracking within $<$tt$>$libelektra-\/core$<$/tt$>$}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1723}
The idea here is that we extend the {\ttfamily Key\+Set} and {\ttfamily Key} structs with additional fields.

We need to extend {\ttfamily Key\+Set} with the following info\+:


\begin{DoxyItemize}
\item What keys have been removed
\item What keys have been added
\item Whether tracking is enabled for this Key\+Set
\end{DoxyItemize}

We need to extend {\ttfamily Key} with the following info\+:


\begin{DoxyItemize}
\item Original value of the key
\item Size of the original value (for binary keys)
\item Whether tracking is enabled for this key
\end{DoxyItemize}

The tracking itself would be done within the {\ttfamily ks$\ast$} and {\ttfamily key$\ast$} methods, after checking if it is enabled. It would also transparently work for metadata, as metadata itself is implemented as a keyset with keys.

Downsides of this approach\+:


\begin{DoxyItemize}
\item It adds functionality to {\ttfamily libelektra-\/core} which may violate the constraint above. It may, however, be debatable whatever \textquotesingle{}minimal\textquotesingle{} means in this context.
\item Adding fields to the structs causes a slight memory overhead, even with tracking turned off. While negligible for {\ttfamily Key\+Set} due to the low amount of keysets in typical applications, it may be noticable for {\ttfamily Key}. On a 64-\/bit system we\textquotesingle{}d add 16 bytes to it. 8 bytes for the pointer to the original value, 8 bytes for the size of the original value. To put this in perspective, the current size of the {\ttfamily Key} struct is 64 bytes, so we\textquotesingle{}d add 25\% overhead to an empty key. However, this percentage will be much lower in a real-\/world application, as the usefulness of an empty key is very low.
\item Another downside here is that it is not so easy to determine what the \char`\"{}original\char`\"{} value is. Some part of {\ttfamily libelektra-\/kdb} would need to mark the keys as original (after transformations etc.)
\end{DoxyItemize}\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1724}{}\doxysubsection{Alternative 5 -\/ Use $<$tt$>$backend\+Data-\/$>$keys$<$/tt$>$ for change tracking}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1724}
Use the {\ttfamily backend\+Data-\/$>$keys} for change tracking

We already store which keys have been returned by {\ttfamily kdb\+Get} for each backend within KDB. Currently, however, this is not a deep duplication, as we are returning the keys from {\ttfamily backend\+Data-\/$>$keys} directly. This means we can not detect changes to the values or metadata of keys right now. We can, however, rely on this for detecting removed and added keys in its current form.

If we don\textquotesingle{}t want to deep-\/dup it, we\textquotesingle{}d need to do something different to detect which keys have been modified. One possibility would be to add metadata within the {\ttfamily key} functions (e.\+g. {\ttfamily meta\+:/elektra/original}), but that would violate the {\ttfamily libelektra-\/core} must be minimal constraint.

There is already \mbox{\hyperlink{doc_decisions_2_in_progress_copy_on_write_md}{another decision}} which discusses adding general copy-\/on-\/write semantics to Elektra. We could use this together with {\ttfamily backend\+Data-\/$>$keys} deep-\/duped to do memory efficient change tracking.\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1725}{}\doxysection{Decision}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1725}
\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1726}{}\doxysection{Rationale}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1726}
\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1727}{}\doxysection{Implications}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1727}

\begin{DoxyItemize}
\item 
\item 
\item 
\end{DoxyItemize}\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1728}{}\doxysection{Related Decisions}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1728}

\begin{DoxyItemize}
\item \mbox{\hyperlink{doc_decisions_0_drafts_operation_sequences_md}{Valid kdb\+Get/kdb\+Set sequences}} from \href{https://github.com/ElektraInitiative/libelektra/pull/4574}{\texttt{ \#4574}}.
\item \mbox{\hyperlink{doc_decisions_2_in_progress_copy_on_write_md}{Copy On Write}}
\end{DoxyItemize}\hypertarget{doc_decisions_0_drafts_change_tracking_md_autotoc_md1729}{}\doxysection{Notes}\label{doc_decisions_0_drafts_change_tracking_md_autotoc_md1729}

\begin{DoxyItemize}
\item Issue \href{https://issues.libelektra.org/4514}{\texttt{ \#4514}} uncovered a problem in the current change tracking approach
\item Issue \href{https://issues.libelektra.org/4520}{\texttt{ \#4520}} already explored some of the considered alternatives 
\end{DoxyItemize}