An Empirical Comparison of Subjective Evaluation and Metrics in the Maintenance of COBOL software

Gorman, Richard (2006). An Empirical Comparison of Subjective Evaluation and Metrics in the Maintenance of COBOL software. Student dissertation for The Open University module M801 MSc in Software Development Research Dissertation.

Please note that this student dissertation is made available in the format that it was submitted for examination, thus the author has not been able to correct errors and/or departures from academic standards in areas such as referencing.

DOI: https://doi.org/10.21954/ou.ro.00016043

Abstract

The cost of software maintenance and in particular the maintenance of legacy software such as COBOL has been widely reported. It is therefore important to be able to measure the maintainability of such software. This study investigates the two primary methods of measuring the maintainability of software; subjective evaluation using software developers, and the more formal and objective approach of software metrics. In addition to these primary methods, a taxonomy of COBOL code smells is developed for potential use in determining software maintainability. The two methods of measurement were applied to a sample of nine COBOL programs. A group of six developers gave a subjective evaluation of the software by answering a questionnaire which assessed their opinion of the software and of the taxonomy of smells, and how these might influence their views. In addition, a total of seven software metrics were used to gain an alternative view of the same software. These metrics were lines of code, McCabe's Cyclomatic Complexity and Halstead's Software Science Indicators. The results show good correlation between the individual metrics evaluated and an equally strong correlation between the metrics and the results of the subjective evaluation. Only one metric, Halstead's Difficulty showed little relationship to the other metrics or to the results of the subjective evaluation. Interestingly, the subjective evaluation showed greater variation in the results than did the software metrics and the subjective evaluation using the taxonomy of smells showed a good correlation with the initial subjective view, yet it did cause a change in opinion in 42.6% of the evaluations. Additionally, the results showed that whatever the method of measuring maintainability, the age of the software is important, and the older the software, the harder it is likely to be to maintain. The results also showed that complexity and size both contribute to maintainability and also that developer experience leads to a better view of the maintainability of the software. This study concludes that software maintainability is influenced by a number of factors, including the source code, program age and developer experience. As a result, the subjective element of developer intuition is required to be able to include all of the factors relevant to measuring software maintainability. Metrics alone cannot easily capture all of these factors, and therefore a metric, or combination of metrics should be used to corroborate the opinions of developers, especially where inexperienced developers are involved in the evaluation.