Updating manual tests with scraper changes

Status: Solutions Clear

Assignee: @e12231202, @e01129489, @lea.birringer, @e12422776

Problem

Manual updating of the manual test cases according the scraper data seems to be very error-prone, the most frequent errors being heatmap related (wrong color expected, non-existing plant relations, inconsistent time periods in temporal relations, etc.).

It also implicates work for any scraper update/fix, i.e. anything that changes finalPlants.csv.

Constraints

  1. We want to avoid any regressions of currently correct scraper data, e.g. from programming errors in the scraper or accidental changes in scraper data.
  2. We want to detect changes of manual tests without executing manual tests.

Assumptions

Solutions

Alternative A

The manual test would no longer contain concrete expected data. Instead, the tester would inspect finalPlants.csv and determine the relevant plants, relations, hierarchies, time periods, etc. from the current scraper output.

Pros

  • No need for maintenance of data in the manual tests.

Cons

  • Large CSV files are very hard to navigate even with various visualizations and extensions.
  • Extra steps for testing that would require maintenance anyway if e.g. data structures change.

Decision

Rejected because:

  • Testers would have to inspect the current finalPlants.csv to determine the expected data, making manual tests significantly harder and more time consuming to execute.
  • Changes to the CSV would not be detected as regressions; the tests would simply use the (potentially incorrectly) changed data.

Alternative B

Generate the manual test cases from the scraper data.

A generator would read the scraper output from finalPlants.csv, identify the data required by the manual tests, and generate or update the corresponding manual test cases.

The implementation could be a Python, Javascript or bash script. It would need to parse the CSV, map scraper records to the corresponding manual test cases, and either generate the complete Markdown document or update the relevant sections.

The generator would need a stable mapping between scraper data and test cases. For example, a test could identify a plant or relation by unique name and the generator could then insert its name, spatial & temporal relations, and other values into the test case.

The workflow would be approximately:

  • Scraper data is generated.
  • The generator reads the resulting CSV.
  • Relevant rows and columns are selected for each manual test.
  • Expected values in the manual tests are generated from those columns.
  • The generated document is committed and/or checked for differences.

A data change therefore results in a change to the generated manual tests.

Pros

  • Automated test generation is almost always a plus when possible.

Cons

  • Potentially difficult and complex implementation (parsing CSV data and generating/updating a Markdown document).
  • May require changes if test case structure changes or some other decisions are made regarding them.

Decision

Rejected because:

  • Requires a relatively complex generator to parse scraper data, map it to manual tests, and generate/update the Markdown document.
  • Regenerating tests from changed data would not distinguish intentional scraper changes from regressions.
  • Additional mechanisms would therefore still be needed to detect unexpected changes.

Alternative C

Creating a bash script that must be run before tests and that inserts special test fixtures (and then another script for removal) that the relevant tests are executed on.

Currently implemented in MR !2254, but requires an update due to the new plant unique name database constraints.

The workflow would be approximately:

  • The fixture script adds a predefined set of plants and relations with known values.
  • The manual tests use these fixtures instead of relying on arbitrary scraped plants.
  • The tester executes the relevant manual tests.
  • The fixtures are removed afterwards.

Changes to normal scraper data therefore do not affect the expected data of these tests.

The fixture data would have to be chosen so that it covers the features being tested, e.g. spatial & temporal relations, hierarchies, heatmap, etc.

Pros

  • Guaranteed reliable data that tests the relevant features and doesn't need to be maintained unless data structure changes significantly (e.g. new mandatory field is added).

Cons

  • Can require major updates if new constraints (e.g. for plant unique names) are added.
  • Extra steps for the tester and potential extra maintenance for the script itself.

Decision

Rejected because:

  • The fixture naming approach conflicts with the current unique-name constraints, requiring either potentially confusing names or exceptions for test fixtures.
  • Requires an additional fixture setup/removal workflow for testers.
  • The same data isolation can be achieved more simply by testing the relevant scraper data directly.

Alternative D

Simply maintain manual test cases & guided tour by hand, with unit tests (using the Vitest framework) failing in scraper with reference to the test if these plant values change.

This generally keeps the current workflow. Manual tests continue to contain concrete expected values. The values that are known to be relevant to manual tests are additionally checked by automated tests and will fail the pipeline if they don't pass.

Tests have been previously manually updated in now merged MR !2231.

Pros

  • Easy implementation.
  • Can be easily integrated into workflow and/or guidelines.

Cons

  • Manual test data still needs to be manually maintained.
  • Extra work during intentional scraper data changes/updates.

Decision

Rejected because:

  • Missing some implementation specifics and concrete data correctness guarantees.

Improved and made as a new alternative solution (H).

Alternative E

Build a consistency checker between scraper data and test data in manual tests.

The checker would parse the manual test Markdown and extract the concrete plant names, spatial & temporal relations, or other values referenced by the tests. It would then compare these values with the corresponding records in finalPlants.csv.

The workflow would be approximately:

  • The scraper generates finalPlants.csv.
  • The checker parses the manual test document.
  • It extracts all scraper-dependent values from the manual tests.
  • It looks up the corresponding values in finalPlants.csv.
  • It reports missing or changed values.
  • The tests are updated if the data change was intentional.

The checker could be implemented as a standalone script (likely Python, bash or Javascript) and run as part of the pipeline.

Pros

  • Easier to implement than full automatic generation of the manual tests.

Cons

  • Still a need for manual maintenance, doesn't really reduce the workload.

Decision

Rejected because:

  • The same information would still need to be maintained in both the scraper data and manual tests.
  • The checker would require additional complex logic to parse the Markdown and match its values against finalPlants.csv.
  • It would detect inconsistencies but would not significantly reduce the maintenance work.

Alternative F

Add automated test cases that check the resulting scraper CSV, i.e. finalPlants.csv, for important properties such as hierarchy data, relations, inherited attributes, and other values required by the manual tests.

These tests should run against fixed input fixtures rather than live scraped data. The fixtures would represent a known scraper input, and the automated tests would verify that the scraper produces the expected output for that input.

For example, a test could verify that an attribute inherited through a hierarchy is present in the resulting CSV, preventing a regression such as the one described in issue #2625.

The manual test cases that use the same behavior/data can be linked from the automated scraper tests.

The workflow would be approximately:

  • A fixed scraper input fixture is defined.
  • Automated tests inspect finalPlants.csv.
  • Assertions verify important output properties.
  • If an assertion fails, the scraper change must be reviewed.
  • If the data change is intentional, the automated test and the dependent manual tests are updated.

See also: #1949.

Pros

  • Can catch regressions in data generation, inheritance, relations, and hierarchy handling.
  • Fixed input fixtures make the tests deterministic and avoid depending on live scraped data.

Cons

  • The fixture problem is similar to Alternative C: the fixture data itself must remain representative of the behavior being tested.
  • There is still a need to update dependent manual tests when an intentionally changed scraper output affects them.

Decision

Rejected because:

  • Needs more specifics in regards to what is being tested and when.
  • Fixed input fixtures introduce an additional fixture maintenance mechanism.

Alternative G

Make some manual tests less dependent on concrete scraper data by making their assertions data-independent.

Instead of asserting a specific value, the test would assert the behavior of the feature. For example, rather than requiring a specific plant to have a specific heatmap color, the test could create or select a known previous planting with a known relation and verify that the heatmap is colored.

The workflow would be approximately:

  • The test establishes or selects a known state satisfying the precondition.
  • The feature under test is executed.
  • The test verifies the relevant behavior rather than a specific scraped value.

Only the behavior of the heatmap or other application feature is therefore tested.

Pros

  • Isolated testing of the heatmap feature itself.

Cons

  • Unchecked potential bug/regression where the heatmap itself works and the data is correct but the color gets calculated incorrectly.
  • Potential case where a plant gets removed from the database or completely changes relations data, resulting in yet more manual test maintenance.

Decision

Rejected because:

  • Data-independent assertions verify application behavior but no longer verify that the underlying scraper data is correct.
  • A heatmap test could pass even if the relevant plant relations, hierarchies, or time periods had changed incorrectly.
  • It can be useful for individual tests, but cannot replace data correctness checks.

Alternative H

Unit tests in the scraper for the specific data that is present in manual tests currently marked with Can be broken on scraper data change.

Like existing scraper unit tests, they would be made using the Vitest framework.

These markers should be retained, but rephrased to make their purpose clearer, e.g. that the test depends on specific scraper data and that duplicating the test also requires a corresponding scraper unit test.

For every such manual test, a scraper unit test checks the relevant values in the scraper output. The unit test should verify the concrete data required by the manual test, while the manual test itself continues to verify the actual application behavior.

The workflow would be approximately:

  • A manual test is created or identified as depending on specific scraper data.
  • The manual test is marked as dependent on scraper data.
  • A corresponding scraper unit test is added.
  • The scraper unit test verifies the specific plants, their spatial & temporal relations, hierarchies, or other values required by the manual test.
  • The pipeline runs these tests automatically.
  • If the scraper data changes, the scraper test fails and produces a warning/error referring to the affected manual test.
  • If the data change is intentional, both the scraper unit test and the manual test are updated manually.

The scraper tests should use data that is extremely unlikely to change as part of normal scraping where possible, reducing unnecessary maintenance.

The scraper should log a clear warning or error when a value expected by a manual test changes, explicitly explaining that the associated manual test needs to be reviewed. These can use warning-level logs once the logging level in #2821 is implemented.

This approach makes the scraper unit test the source of truth for detecting unexpected changes to data that is intentionally relied upon by manual tests, while the manual test remains responsible for testing the application's behavior.

Pros

  • Easy implementation
  • Guarantees correctness of data in manual tests
  • No extra work for the tester

Cons

  • Needs to be maintained additionally to the manual tests, and potentially updated when scraper data gets updated

Decision

Accepted because:

  • Implementation is straight forward.
  • Specific scraper data required by manual tests is verified automatically, so unexpected changes are detected before the manual tests are executed.
  • It explicitly documents the dependency between manual tests and scraper data and defines how this dependency is maintained.

Decision

Rationale

Implications

Notes

  • Importance of the problem: We should spend time on this because it is important that data changes are isolated from heatmap code changes as much as possible, to improve the efficiency of manual testing and detection of possible regressions.
  • Scope: Manual Tests/Scraper
  • Relation to other problems: It is at least indirectly related to any issue that has to do with scraper data changes and heatmap code changes