Collection of test data during digital testing
Summary
Collection of test data during digital testing
S00183
Version: 1.10
- arable farming
- horticulture
- viticulture
- collection of test data; provision of datasets
- data; other (including: nothing)
- Location: remote
- Offered by: POLIMI; UNIMI
Description
One of the key activities during digital testing is the collection of data concerning the evolution and the outcomes of the tests that enable the evaluation of system performance (e.g., via Service S00184). This service manages the collection of relevant data during testing.
The scope of data collected can range from data collected within a virtual environment to simulate sensor data collection in physical environments, statistics about AI model performance in the test and deployment phase (e.g., occupied memory, number of trainable parameters, training/optimization loss, etc.), to the collection of specific labels and annotations to use as ground truth for evaluating the system. The minimum set of data that are collected by S00183 is defined by the evaluation metrics chosen (e.g., via Service S00178) to process them, but generally a larger set of data is defined by AgrifoodTEF together with the customer to provide a richer view of the system’s performance and to enable the application of other metrics in the future, if needed.
In addition to the raw collected data, we also provide customers with a complete documentation describing the logged features, conditions of the testing environment at the time of testing, as well as any parameter values, variation ranges and specifics required for reproducibility purposes. This allows the customer to leverage the capability (that only digital testing has) of allowing perfect reproducibility/repeatability of tests: perfect control over the test environment is in fact available, so the problem of reproducibility/repeatability is reduced to possessing full information about the environment and the tests performed in it.
Example service: To avoid damaging plants during robotic weeding operations, the customer wants to test an intra-row navigation solution in simulation before deploying the system in field, so a suitable simulation (based on Gazebo) of robot and environment has been prepared. To help the customer thoroughly validate their navigation software, we run the simulation and collect point cloud data within a Gazebo simulator. The collected data represent plant rows of different length (3, 5, up to 10 metres), intra-row width (from 3 down to 1 metre), and plant density, to enable system testing under scenarios of varying difficulty. During these activities, we collect datasets describing the interaction of the robot with the simulated environment; additionally, we collect the full (simulated) sensor data streams, so that the customer can use them later (e.g., for training of AI models). We also provide the customer with a report describing the different data fields included, their frequency of collection, as well as the full configuration files of the simulation environment, for reproducibility.