Copy the page URI to the clipboard
Summers, Joshua D.; Eckert, Claudia and Goel, Ashok
(2017).
DOI: https://doi.org/10.1017/S0890060417000476
Abstract
This paper presents the requirements and needs for establishing a benchmarking protocol that considers representation characteristics, supported cognitive criteria, and enabled reasoning activities for the systematic comparison of function modeling representations. Problem types are defined as reverse engineering, familiar products, novel products, and single-component systems. As different modelling approaches share elements, a comparison of modelling approaches on multiple levels was also undertaken. It is recommended that researchers and developers of function modeling representations collaborate to define a canonically acceptable set of benchmark tests and evaluations so that clear benefits and weaknesses for the disparate collection of approaches can be compared. This paper is written as a call to action for the research community to begin establishing a benchmarking standard protocol for function modeling comparison purposes. This protocol should be refined with input from developers of the competing approaches in an academically open environment. At the same time the benchmarking criteria identified should also serve as a guide for validating a modelling approach or analyzing its failure.