.TH "md_src_plugins_toml_README" 3elektra "Thu Sep 7 2023" "Version 0.11.0" "Elektra" \" -*- nroff -*-
.ad l
.nh
.SH NAME
md_src_plugins_toml_README \- Plugin: toml 

.IP "\(bu" 2
infos = Information about the toml plugin is in keys below
.IP "\(bu" 2
infos/author = Jakob Fischer jakobfischer93@gmail.com
.IP "\(bu" 2
infos/licence = BSD
.IP "\(bu" 2
infos/provides = storage/toml
.IP "\(bu" 2
infos/needs = base64
.IP "\(bu" 2
infos/recommends = type
.IP "\(bu" 2
infos/placements = getstorage setstorage
.IP "\(bu" 2
infos/status = unittest experimental unfinished
.IP "\(bu" 2
infos/metadata = order comment/# comment/#/start comment/#/space type tomltype origvalue
.IP "\(bu" 2
infos/description = This storage plugin reads and writes TOML files using Flex and Bison\&.
.PP
.SH "Introduction"
.PP
This plugin is a storage plugin for reading and writing TOML files\&. The plugin retains most of the file structure of a read TOML file such as comments, empty lines and TOML tables\&. It supports all kinds of TOML specific types and tables, including nested inline tables and multiline strings\&.
.SH "Requirements"
.PP
The plugin needs Flex (>=2\&.6\&.2) and Bison (>=3) for parsing TOML files\&.
.SH "Types"
.PP
.SS "Reading"
On reading, the plugin will set the \fCtype\fP metakey for strings, integers, floats and boolean values, if applicable\&. For decimal integers, the metakey is set to \fClong_long\fP\&. For binary/octal/hexadecimal integers, the metakey is set to \fCunsigned_long_long\fP\&. For floats, the metakey will be set to \fCdouble\fP\&. These types are chosen to conform with the TOML format as stated on the offical \fCTOML\fP page\&.
.PP
On reading, non-decimal integers will get converted to decimal\&. The non-decimal representation will be stored in the \fCorigvalue\fP metakey of the key\&. When writing a key, the value of that metakey will be written instead, in order to retain the original format\&. Note that the \fCorigvalue\fP metakey gets removed if the value of the key changes\&.
.PP
.PP
.nf
sudo kdb mount test\&.toml user:/tests/storage/types toml type

# Create a TOML file with 4 keys
mkdir -p $(dirname $(kdb file user:/tests/storage/types))
echo 'plain_decimal = 1000' >> `kdb file user:/tests/storage/types`
echo 'file_permissions = 0o777' >> `kdb file user:/tests/storage/types`
echo 'pi = 3\&.1415' >> `kdb file user:/tests/storage/types`
echo 'division_gone_wrong = -inf' >> `kdb file user:/tests/storage/types`

# Print the content of the toml file
cat `kdb file user:/tests/storage/types`
# > plain_decimal = 1000
# > file_permissions = 0o777
# > pi = 3\&.1415
# > division_gone_wrong = -inf

# Print types and values of the keys with `kdb`

kdb meta-get 'user:/tests/storage/types/plain_decimal' 'type'
# > long_long
kdb get 'user:/tests/storage/types/plain_decimal'
# > 1000

kdb meta-get 'user:/tests/storage/types/file_permissions' 'type'
# > unsigned_long_long
# The octal value will be converted to decimal
kdb get 'user:/tests/storage/types/file_permissions'
# > 511

kdb meta-get 'user:/tests/storage/types/pi' 'type'
# > double
kdb get 'user:/tests/storage/types/pi'
# > 3\&.1415

kdb meta-get 'user:/tests/storage/types/division_gone_wrong' 'type'
# > double
kdb get 'user:/tests/storage/types/division_gone_wrong'
# > -inf

# Cleanup
kdb rm -r user:/tests/storage/types
sudo kdb umount user:/tests/storage/types
.fi
.PP
.SS "Writing"
On writing, for most values, the plugin will infer the appropriate type and will write them accordingly\&. This means, values, that match a TOML float, integer or date will be written without any quotes around them\&. If a value does not match any of these types, it will be written as a string\&.
.PP
The plugin uses an existing \fCtype\fP metakey only to check if it should write a value as a \fCstring\fP or a \fCboolean\fP\&.
.PP
With this functionality, numbers can be written as a string to the file, if wanted\&.
.PP
To write a boolean value, the \fCtype\fP metakey must be set to \fCboolean\fP for the key\&. Otherwise, no conversion to the TOML boolean values will take place\&. Per default, Elektra uses 0/1 to represent boolean values\&.
.PP
.PP
.nf
sudo kdb mount test\&.toml user:/tests/storage/types toml type

# Create a key, which may be a integer, boolean or string
kdb set 'user:/tests/storage/types/value' '1'

# The plugin infers `long_long` for this value
kdb meta-get 'user:/tests/storage/types/value' 'type'
# > long_long

# The value is written as an integer
cat `kdb file user:/tests/storage/types`
# > value = 1

# Manually set the `type` metakey to boolean
kdb meta-set 'user:/tests/storage/types/value' 'type' 'boolean'

# The value is written as a boolean
cat `kdb file user:/tests/storage/types`
# > value = true

# Manually set the `type` metakey to string
kdb meta-set 'user:/tests/storage/types/value' 'type' 'string'

# The value is written as a string
cat `kdb file user:/tests/storage/types`
# > value = "1"

# Cleanup
kdb rm -r user:/tests/storage/types
sudo kdb umount user:/tests/storage/types
.fi
.PP
.SH "Numbers"
.PP
The plugin supports reading and writing of any kind of number supported by the TOML format, such as floating point numbers and binary/octal/decimal/hexadecimal integers\&. To write a non-decimal integer, add the corresponding prefix to the number (\fC0b\fP for binary, \fC0o\fP for octal, \fC0x\fP for hexadecimal)\&. The value will be written in the given base to the file, but converted to decimal within Elektra (see \fCReading\fP)\&. Note that the plugin doesn't warn about invalid prefix/digit combinations\&. If the combination is not valid, it will be written as a string instead\&.
.PP
If the \fCtype\fP plugin is enabled and you want to change the value of an existing number key which needs conversion (all keys which have a \fCorigvalue\fP), you have to change the value of \fCorigvalue\fP instead of the key value\&. Otherwise the \fCtype\fP plugin will give an error\&.
.PP
.PP
.nf
# Mount a new TOML file
sudo kdb mount test\&.toml user:/tests/storage/numbers toml type

# Write an octal value
kdb set 'user:/tests/storage/numbers/a' '0o777'

# Get the converted decimal value
kdb get 'user:/tests/storage/numbers/a'
# > 511

# Get the original octal value
kdb meta-get 'user:/tests/storage/numbers/a' 'origvalue'
# > 0o777

# Get the type of the number; since it's originally octal, we get `unsigned_long_long`
kdb meta-get 'user:/tests/storage/numbers/a' 'type'
# > unsigned_long_long

# Change the value by changing the `origvalue` metakey
kdb meta-set 'user:/tests/storage/numbers/a' 'origvalue' '0o666'

# Get the new value as decimal
kdb get 'user:/tests/storage/numbers/a'
# > 438

# Change the value to an invalid octal value\&.
kdb meta-set 'user:/tests/storage/numbers/a' 'origvalue' '0o888'

# The key value is no longer considered a number
kdb meta-get 'user:/tests/storage/numbers/a' 'type'
# > string

# Cleanup
kdb rm -r user:/tests/storage/numbers
sudo kdb umount user:/tests/storage/numbers
.fi
.PP
.SH "Strings"
.PP
The plugin can read any kind of TOML string: basic, literal, basic multiline and literal multiline\&.
.PP
Similar to the TOML-specific values (table, array) discussed below, we use the \fCtomltype\fP metakey to store the kind of string\&. The values used are: \fCstring_basic\fP, \fCstring_literal\fP, \fCstring_ml_basic\fP and \fCstring_ml_literal\fP\&.
.PP
When writing a file, the value of \fCtomltype\fP will be respected\&. The plugin will therefore preserve the kind of string throughout a kdbGet/kdbSet cycle\&. You can also set \fCtomltype\fP manually to transform the string into a different type\&. If \fCtomltype\fP is set to a kind of string that is incompatible with the current value of the key, the plugin will report an error\&.
.PP
If \fCtomltype\fP is not set, the plugin defaults to a basic string, or if there are at least two \fC\\n\fP to a basic multiline string\&. This way a \fCkdb set\fP will always work correctly, as long as the given string is valid UTF-8\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_strings\&.toml user:/tests/storage toml type

# setting a string containing a newline escape sequence
kdb set 'user:/tests/storage/string' 'I am a basic string\not a literal one\&.'

kdb get 'user:/tests/storage/string'
# > I am a basic string
# > ot a literal one

# setting the string again, but escape the backslash with another backslash
kdb set 'user:/tests/storage/string' 'I am a basic string\\not a literal one\&.'

kdb get 'user:/tests/storage/string'
# > I am a basic string\not a literal one

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.PP
The plugin supports all kinds of escape sequences used by TOML in basic and basic multiline strings, like \fC\\n\fP, \fC\\r\fP, \fC\\t\fP and even \fC\\u\fP/\fC\\U\fP for Unicode escape sequences\&. When reading a file, all these escape sequences will be translated into valid UTF-8 byte sequences that will be used as the key values\&.
.PP
When writing a file, the plugin uses the escape sequences, whenever a part of the key value cannot be placed literally into the file\&. If a key contains a non-UTF-8 string value, the plugin replaces any invalid sections with the Replacement Character U+FFFD and reports a warning\&.
.PP
Finally, the plugin also uses \fCorigvalue\fP to remember the original formatting of the string\&. This is particularly useful for escape sequences\&. Without this mechanism, a \fC\\u\fP escape sequence would always be written literally (not as an escape sequence), because when reading the file it is translated into UTF-8 bytes\&. For multiline strings \fCorigvalue\fP also includes any leading or escaped newlines\&.
.SH "Binary/NULL values"
.PP
The plugin handles binary values by using the \fBbase64\fP plugin\&. As a result, binary values get written as base64 encoded strings, which start with the special prefix \fC@BASE64\fP\&. \fCNULL\fP key values are written as special strings of value \fC@NULL\fP\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_binary\&.toml user:/tests/storage toml type

# Creating a key with a NULL value
kdb set 'user:/tests/storage/nullkey'
# > Create a new key user:/test/nullkey with null value

# Print file content
cat `kdb file user:/tests/storage`
# > nullkey = '@NULL'

# Write base64 encoded data to the file
echo "base64 = '@BASE64SSBhbSBiYXNlIDY0IGVuY29kZWQgZm9yIG5vIHJlYXNvbi4='" > `kdb file user:/test`

# Print the value of the key, which is a binary value
kdb get 'user:/test/base64'
#> \x49\x20\x61\x6d\x20\x62\x61\x73\x65\x20\x36\x34\x20\x65\x6e\x63\x6f\x64\x65\x64\x20\x66\x6f\x72\x20\x6e\x6f\x20\x72\x65\x61\x73\x6f\x6e\x2e

# Print the value again, but apply the escape codes
echo -e `kdb get 'user:/test/base64'`
#> I am base 64 encoded for no reason\&.

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.SH "TOML specific structures"
.PP
TOML specific structures are represented by the metakey \fCtomltype\fP on a certain key\&. It will be set when the TOML plugin reads a TOML structure from a file\&. Additionally, this metakey can be set by the user, if they want a certain TOML structure to be written\&. No automatic inference of this metakey is done on writing\&.
.SS "Simple Tables"
TOML's simple tables are represented by setting the \fCtomltype\fP metakey to \fCsimpletable\fP\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_table\&.toml user:/tests/storage toml type

# Create three keys, which are all a subkey of 'common',
# but we have no 'common' simple table key yet
kdb set 'user:/tests/storage/common/a' '0'
kdb set 'user:/tests/storage/common/b' '1'
kdb set 'user:/tests/storage/common/c' '2'

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> common\&.a = 0
#> common\&.b = 1
#> common\&.c = 2

# Create a simple table key
kdb meta-set 'user:/tests/storage/common' 'tomltype' 'simpletable'

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> [common]
#> a = 0
#> b = 1
#> c = 2

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.SS "Table Arrays"
Table arrays are represented by setting the \fCtomltype\fP metakey to \fCtablearray\fP\&. It is not required to also set the array metakey, since the plugin will set the metakey, if it is missing\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_table_array\&.toml user:/tests/storage toml type

# Create a table array containing two entries, each with a key 'a' and 'b'
kdb meta-set 'user:/tests/storage/tablearray' 'tomltype' 'tablearray'
kdb set 'user:/tests/storage/tablearray/#0/a' '1'
kdb set 'user:/tests/storage/tablearray/#0/b' '2'

kdb set 'user:/tests/storage/tablearray/#1/a' '3'
kdb set 'user:/tests/storage/tablearray/#1/b' '4'

# Print the highest index of the table array
kdb meta-get 'user:/tests/storage/tablearray' 'array'
#> #1

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> [[tablearray]]
#> a = 1
#> b = 2
#> [[tablearray]]
#> a = 3
#> b = 4

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.SS "Inline Tables"
Inline tables are represented by setting the \fCtomltype\fP metakey to \fCinlinetable\fP\&. The plugin also supports reading/writing nested inline tables\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_inline_table\&.toml user:/tests/storage toml type

# Create a table array containing two entries, each with a key 'a' and 'b'
kdb meta-set 'user:/tests/storage/inlinetable' 'tomltype' 'inlinetable'
kdb set 'user:/tests/storage/inlinetable/a' '1'
kdb set 'user:/tests/storage/inlinetable/b' '2'
kdb meta-set 'user:/tests/storage/inlinetable/nested' 'tomltype' 'inlinetable'
kdb set 'user:/tests/storage/inlinetable/nested/x' '3'
kdb set 'user:/tests/storage/inlinetable/nested/y' '4'

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> inlinetable = { a = 1, b = 2, nested = { x = 3, y = 4 } }

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.SS "Arrays"
Arrays are recognized by the \fCarray\fP metakey\&. On writing, the plugin will detect arrays automatically and set the appropriate metakey if it is missing\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_array\&.toml user:/tests/storage toml type

# Create array elements
kdb set 'user:/tests/storage/array/#0' '1'
kdb set 'user:/tests/storage/array/#1' '2'
kdb set 'user:/tests/storage/array/#2' '3'

# Print the highest index of the array
kdb meta-get 'user:/tests/storage/array' 'array'
#> #2

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> array = [1, 2, 3]


# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.SH "Comments and Empty Lines"
.PP
The plugin preserves all comments with only one limitation for arrays\&. The whitespace in front of a comment is also saved\&.
.PP
Comments can also be created by assigning meta keys to a key\&. The meta keys must be of the form \fCcomment/#n\fP, where \fCn\fP is a positive number, indicating the position of the comment relative to the key\&. An index of 0 is always the inline comment of the key\&. Indices greater than zero are for comments preceding the given key, where 1 is the top-most comment and the highest index comment is right above the key\&.
.PP
Preceding whitespace can be added to a comment by creating a \fCcomment/#n/space\fP metakey\&. This metakey can contain any number of space or tab characters, which will be placed before the \fC#\fP starting the comment\&. Any whitespace \fIafter\fP the \fC#\fP is considered part of the comment and will therefore be part of \fCcomment/#n\fP\&.
.PP
File ending comments must be assigned to the file root key\&.
.PP
Empty lines in front of a key can be created by adding an empty \fCcomment/#n/start\fP entry to it\&. In this case, no \fCcomment/#n\fP key is needed\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_comments\&.toml user:/tests/storage toml type

# create a key-value pair, ready for comment decoration

kdb set 'user:/tests/storage/key' '1'

# add an inline comment with 4 leading spaces
kdb meta-set 'user:/tests/storage/key' 'comment/#0' ' This value is very interesting'
kdb meta-set 'user:/tests/storage/key' 'comment/#0/space' '    '

# add some comments preceding the key
kdb meta-set 'user:/tests/storage/key' 'comment/#1' ' I am the top-most comment relative to my key\&.'
kdb meta-set 'user:/tests/storage/key' 'comment/#2' ' I am in the middle\&. Just boring\&.'
kdb meta-set 'user:/tests/storage/key' 'comment/#3' ' I am in the line right above my key\&.'

# add file ending comments and empty lines
kdb meta-set 'user:/tests/storage' 'comment/#1' ' First file-ending comment'
kdb meta-set 'user:/tests/storage' 'comment/#2/start' ''
kdb meta-set 'user:/tests/storage' 'comment/#3' ' Second file-ending comment\&. I am the last line of the file\&.'

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> # I am the top-most comment relative to my key\&.
#> # I am in the middle\&. Just boring\&.
#> # I am in the line right above my key\&.
#> key = 1    # This value is very interesting
#> # First file-ending comment
#>
#> # Second file-ending comment\&. I am the last line of the file\&.

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.SS "Comments in Arrays"
Any amount of comments can be placed between array elements or between the first element and the opening brackets\&.
.PP
However, only one comment - an inline comment - can be placed after the last element and the closing brackets\&. On reading, the plugin discards any non-inline comments between the last element and the closing brackets\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_array_comments\&.toml user:/tests/storage toml type

# Create array elements
kdb set 'user:/tests/storage/array/#0' '1'
kdb set 'user:/tests/storage/array/#1' '2'
kdb set 'user:/tests/storage/array/#2' '3'

# Add inline comment after the array
kdb meta-set 'user:/tests/storage/array' 'comment/#0' ' Inline comment after the array'
kdb meta-set 'user:/tests/storage/array' 'comment/#0/start' '#'
kdb meta-set 'user:/tests/storage/array' 'comment/#0/space' '     '

# Add comments for array elements
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#0' ' Inline comment of first element'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#0/start' '#'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#0/space' '    '

kdb meta-set 'user:/tests/storage/array/#0' 'comment/#1' ' Comment preceding the first element'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#1/space' '    '

kdb meta-set 'user:/tests/storage/array/#0' 'comment/#2' ' Another comment preceding the first element'
kdb meta-set 'user:/tests/storage/array/#0' 'comment/#2/space' '      '

kdb meta-set 'user:/tests/storage/array/#1' 'comment/#0' ' Inline comment of second element'
kdb meta-set 'user:/tests/storage/array/#1' 'comment/#0/space' '    '

kdb meta-set 'user:/tests/storage/array/#1' 'comment/#1' ' Comment preceding the second element'
kdb meta-set 'user:/tests/storage/array/#1' 'comment/#1/space' '      '

kdb meta-set 'user:/tests/storage/array/#2' 'comment/#0' ' Inline comment of the last element'
kdb meta-set 'user:/tests/storage/array/#2' 'comment/#0/space' '     '

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> array = [    # Comment preceding the first element
#>       # Another comment preceding the first element
#> 1,    # Inline comment of first element
#>       # Comment preceding the second element
#> 2,    # Inline comment of second element
#> 3     # Inline comment of the last element
#> ]     # Inline comment after the array

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.SH "Order"
.PP
The plugin preserves the file order by the usage of the metakey \fCorder\fP\&. When reading a file, the order metakey will be set according to the order as read in the file\&. If new keys are added, eg\&. via \fCkdb set\fP, the order of the set key will be set to the next-to-highest order value present in the existing key set\&.
.PP
However, the order is only relevant between elements with the same TOML-parent\&. For example keys of a simple table are only sorted with respect to each other, not with any keys outside that table\&. If that table has it's order changed and moves to another position in the file, so will it's subkeys\&.
.PP
When sorting elements under the same TOML-parent, tables (simple and array) will always be sorted after non-table elements, regardless of their order\&. With this limitation, we prevent that a newly set key, that is not part of a certain table array/simple table, would be placed after the table declaration, making it a member of that table on a subsequent read\&.
.PP
.PP
.nf
# Mount TOML file
sudo kdb mount test_order\&.toml user:/tests/storage toml type

# Create three keys in reverse alphabetical order under the subkey common
# Additionally, create one key not in the common subkey space

kdb set 'user:/tests/storage/common/c' '0'
kdb set 'user:/tests/storage/common/b' '1'
kdb set 'user:/tests/storage/common/a' '2'
kdb set 'user:/tests/storage/d' '3'

# Print the content of the resulting TOML file
# The keys are ordered as they were set

cat `kdb file user:/tests/storage`
#> common\&.c = 0
#> common\&.b = 1
#> common\&.a = 2
#> d = 3

# Create a simple table for the three keys under `common`
kdb meta-set 'user:/tests/storage/common' 'tomltype' 'simpletable'

# Print the content of the resulting TOML file
cat `kdb file user:/tests/storage`
#> d = 3
#> [common]
#> c = 0
#> b = 1
#> a = 2

# Cleanup
kdb rm -r user:/tests/storage
sudo kdb umount user:/tests/storage
.fi
.PP
.PP
In this example, \fCd\fP and \fCcommon\fP have the same parent, the file root\&. This means, they need to be sorted with each other\&. \fCd\fP would be placed before \fCcommon\fP by it's order, since it was set before, and thus, has lesser order\&. However, their order never gets compared, since \fCcommon\fP is a simple table and \fCd\fP is not, so \fCd\fP will get sorted before the table regardless of order\&.
.SH "Limitations"
.PP
While the plugin has good capabilities in handling the TOML file format, it currently lacks some features possible with Elektra:
.PP
.IP "\(bu" 2
Sparse arrays are not preserved on writing, they get a continuous array without index holes\&.
.IP "\(bu" 2
Values on non-leaf keys are currently not supported, they get discarded\&. This applies especially to the parent key of the mountpoint\&.
.IP "\(bu" 2
Custom metakeys cannot be written by the plugin, so they get discarded\&.
.PP
.PP
Additionally, there are some minor limitations related to the TOML file format, mostly related to the preservation of the original file structure:
.PP
.IP "\(bu" 2
On writing, each string is written as a TOML basic string (single or multiline)\&. See \fCStrings\fP
.IP "\(bu" 2
Comments and newlines between the last array element and closing brackets are discarded\&.
.IP "\(bu" 2
Trailing commas in arrays and inline tables are discarded
.IP "\(bu" 2
Only spaces in front of comments are preserved\&. 
.PP