Scraper Team - Meeting 21.07.2026

Meeting notes by Christoph Schreiner

Attendees

  • Johannes
  • Lea
  • Markus
  • Vseslav
  • Christoph

Architecture / Roles Discussion

flowchart TD
    CSV[("CSV files")]
    DB[("Database")]
    BE["Backend<br/>Vseslav (with support from Johannes)"]
    FE["Frontend"]
    CSV -->|"Scraper pipeline inserts data into DB<br/>(many steps)<br/><i>Vseslav + Lea</i>"| DB
    DB -->|"plant attributes (1d): Vseslav<br>search (1w): Johannes<br>heatmap (1m): open (maybe Lea/Johannes)"| BE
    BE --> FE
  • Backend: Vseslav (with support from Johannes)
  • Database: open
  • Scraper code: Vseslav and Lea
  • Scraper data validation: maybe Lea
  • Heatmap: open
  • Plant attributes: Vseslav
  • Plant search: Johannes
  • Frontend: open

Open Issues (Christoph)

  • Temporal family/genus relations -> can be reassigned; will be refined by Christoph first
  • Ambivalent temporal relations -> can be reassigned
  • Plant hierarchy: Add genus-to-family mapping via CSV -> can be reassigned
  • Guidelines for data -> will be finished by Christoph

Questions & Decisions

  • Rename scraper-data to plant-data repo? -> Should be renamed
  • Code in scraper-data repo as well? -> Not so important, moves the problem of having to create tandem-merge-requests to backend but remains
  • Making data available via Jenkins, so that people don't need to check out scraper-data? How to do access-control for this? -> We will not continue with this. It is not that much improvement
  • Where to put further tools (statistical propagation etc.)? -> Statistical propagation should stay in the pipeline.
  • What is to do for the scraper? -> Check format, check if plants are in the dataset, check if a column is missing from the CSV file. Currently written in JavaScript, so everything is untyped; consider libraries for better validation. Goal: see issues early, so that if one step makes a bad transformation the next step does not just reject the data. In the future, non-technical people will edit the CSV files and should get good error messages. Huge topic; a good starting point is validating the manually generated CSV files and producing good error messages

Tasks

  • Christoph: wrap up with documentation, babysit last MRs
  • Johannes: investigate SQL improvement / parallel code from Christoph
  • Lea: new scraper-data issues
  • Vseslav: no new tasks

Outlook

  • Rename of repo
  • Testing of scraper-data (reproducibility, regressions, ...)
  • Improvements of search