Formalized Plant Model

Status: Draft

Assignee: Lea Birringer

Problem

Currently, there is no centralized formal model of how plant attributes are validated and how information about them is stored. This results in the following sub-problems:

  1. Specifications and our schema exist as comments in backend/src/model/entity.rs as database constraints and as constraints within the scraper. We tried establishing a "single source of truth" in backend/src/model/entity.rs, but we are still far off.
  2. When adding or modifying a plant attribute in our database, multiple files/directories need to be adjusted with the same or similar information and could drift apart:
    1. The following feature the attributes themselves and should fully match (except created_at and updated_at in entity.rs):
      • backend/src/model/entity.rs contains the attributes defined for Rust
      • scraper/src/helpers/dp_plants_columns.js contains the attributes in the db
      • backend/migrations for new attributes that have not been in a migration so far, a migration needs to be added
    2. The following should be subsets of the above attributes:
      • scraper/src/helpers/inheritable_attributes.js contains attributes specified as inheritable between ranks in the scraper
      • scraper/src/insert_plants.js contains the formatting applied to the inserted plants in the database
      • backend/migrations for new attributes that need an enum, a migration needs to be added for the enum
      • backend/src/model/enum contains the enums used for Diesel and Typeshare. The enums for plant attributes must match the database enums. If the enum is used on the frontend, a definition has to be added here.
    3. For frontend implementation (display of relation lines, display of plant attributes in the plant summary), there are multiple related files. These are more difficult to automatically generate and strictly enforce.
  3. There is no full end-to-end validation of the data in the plant attributes. Mechanisms already present are Diesel for DB<->backend and Typeshare for backend<->frontend. A mechanism for scraper and database constraints is still missing. Centrally defined constraints as validation rules could enable implementing validation in the following files:
    • tools/update-fill-error-ratio/error_ratio/src/main.rs currently validates only against enums and datatypes
    • backend/migrations for new attributes, constraints on the database columns for validation need to be added via a migration
    • Files in the scraper pipeline could utilize validation constraints (and enum definitions) for validation in between steps, to validate input data and to provide better error messages.

Some kind of code generation tool based on a formalism (model-driven development) would therefore be helpful. The formalism should be able to describe constraints which can be implemented using code generation.

This decision covers how the model is implemented.

Constraints

  1. The solution should be able to model or describe all information currently in backend/src/model/entity.rs comments, database constraints and constraints within the scraper.
  2. The solution should be able to serve as a basis for efficiently automatically generating attribute lists, enums and migration code for Problem 2i and 2ii.
  3. The solution should provide machine-readable representation of constraints and enums to use for validation between scraper and database or provide tools for validation directly.
  4. Language agnosticism: Modified / generated code is written in different languages, as such, a formalism applicable regardless of language is needed.
  5. Shared schema: The modeled attributes should themselves conform to an automatically checked schema to make automated code generation possible.
  6. The solution should be extensible in case of future features.

Assumptions

  1. This decision does not affect the user experience, only development.
  2. Development regarding adding/modifying plant attributes should be faster.
  3. Risk: The formalism could become too specific to be applicable to multiple code chunks.
  4. Problem: There could still be many manual adjustments necessary after the code generation based on the formalism is executed.

Solutions

Alternative A: UML

UML serves as a universal language for modeling and has widely supported code generation tools. Additionally, the model can be visualized. [1] However, industry practitioners report standard UML to be insufficient for their purposes, resulting in either not using UML or a heavily customized version of it. [2]

Pros:

  • Supported by many enterprise tools

Cons:

  • Supported tools aren't very useful for e.g. javascript validation code generation
  • Practical application needs to be highly customized anyway

Alternative B: DSL (e.g. CUE)

In contrast to UML, DSL are domain-specific. Especially for our purpose of implementing validation constraints, CUE offers built-in support. If needed, the CUE schema can also be converted to raw JSON. The following pros and cons are based on CUE:

Pros:

  • Expressive constraints (RegEx, cross-field validation) [3]
  • Could, for example, validate intermediate files in the scraper data without writing extra code
  • Automatically checks logical impossibilities on constraints
  • Built-in templating support [4]
  • Can be exported as JSON Schema compliant JSON [5]

Cons:

  • Separate language not in the project so far
  • JSON output from CUE can be nested and have many lines compared to custom JS schema

Alternative C: Standardized models using semi-structured data (e.g. JSON)

Semi-structured data, e.g. raw JSON, can be used to store information and constraints about attributes. Since these can be arbitrary values, they can be basically anything, but need additional parsing. To validate the plant attribute schemas themselves to a standard-compliant form, JSON Schema [6] can be utilized.

Pros:

  • Easily machine-readable and usable for manually writing scripts generating code
  • JSON Schema can be used to validate the models of the plant attributes

Cons:

  • Constraints can be verbose

Decision

Rationale

Implications

Notes