Software reliability estimation on the basis of
the operational experience of pre-developed components
Project - Component based Reliability Estimation
In safety-critical application areas with high reliability demands, a quantitative assessment of software reliability is crucial for licensing the use of software systems. For the purpose of estimating software reliability, it is not the number of software faults which is significant, but rather the frequency of their manifestation during runtime. Therefore, a sound reliability assessment has to be carried out by anticipating as far as possible the expected operational profile during the testing phase.
As described in former annual research reports a preliminary sub-project investigated techniques for logging usage data and visualise corresponding operational profiles in order to document past operational experience. To do so, a tool was implemented allowing the logging of all external method calls of a predefined Java component during runtime. This technique for logging runtime calls is intended to support systematic decision-making concerning the need for additional tests of functionalities to be re-used in a new context.
In another former sub-project a method was developed permitting to estimate the future operational profile of a software program. This method was applied to a real-world medical system for the calibration of hearing aids. Based on this information the test process can be adjusted to future software usage by selecting single test cases with operationally representative frequencies. In such a way, test results can be converted into reliability estimates by means of reliability growth models.
In the past, the application of statistical hypothesis testing for the purpose of quantitative reliability estimation was investigated. Based on a preliminary testing or usage period, an upper bound of the failure probability can be determined at any given confidence level. In the case of pre-developed software components, component-specific reliability estimates can be composed into a single system reliability assessment. For this purpose, approaches were developed which allow for the architecture-based combination of component-specific reliability estimates. Moreover, techniques for the sensitivity analysis and optimisation of additional testing were derived.
Apart from these theoretical considerations, more practical questions concerning the recording, the analysis and the statistical evaluation of past operational usage were considered. For this purpose, a general guideline was developed, supporting the statistical analysis of operational data. Within a student thesis, a tool was designed and implemented, supporting the extraction of independent subsets of operational data by application of genetic algorithms.
In order to investigate the practical applicability of this guideline, an industrial research cooperation with the automotive supplier ZF Friedrichshafen AG was accomplished: a software-based gearbox controller was analysed in the light of the applicability of statistical testing.
This project was finally finished within the last annual report period by a PhD-Thesis considering these topics, which was published in a series of the working reports of the department of computer science.
- Sven Söhnlein: "Quantitative Bewertung der Softwarezuverlässigkeit komponentenbasierter Systeme durch statistische Auswertung der Betriebserfahrung", Arbeitsberichte des Department Informatik, Friedrich-Alexander-Universität Erlangen-Nürnberg. Band 43, Nummer 1, Oktober 2010, Dissertation, 200 Seiten. ISSN 1611-4205.
- Sven Söhnlein, Francesca Saglietti, Franz Bitzer, Matthias Meitner: "Bewertung der Zuverlässigkeit von Software - Betriebserfahrung an einer Getriebesteuerung nutzen", Automatisierungstechnische Praxis (atp EDITION), 52. Jahrgang, 6/2010, 32-39, Oldenbourg Industrieverlag, 2010. ISSN 2190-4111.
- Sven Söhnlein, Francesca Saglietti, Franz Bitzer, Matthias Meitner, Siegfried Baryschew: "Software Reliability Assessment Based on the Evaluation of Operational Experience", Proc. 15th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems and Dependability and Fault Tolerance (MMB & DFT 2010), Lecture Notes in Computer Science, Vol. LNCS 5987, S. 24-38, 2010.
- Sven Söhnlein, Francesca Saglietti, Franz Bitzer, Siegfried Baryschew: "Software Reliability Assessment for a Gearbox Controller by Analysis of Operating Experience", Supplemental Proc. 20th International Symposium on Software Reliability Engineering (ISSRE), 2009.
- Sven Söhnlein, Francesca Saglietti, Franz Bitzer, Siegfried Baryschew: "Zuverlässigkeitsbewertung einer Getriebesteuerungs-Software durch Auswertung der Betriebserfahrung", Softwaretechnik-Trends, Gesellschaft für Informatik (GI) e. V., 29 (3), S. 41-46, 2009.
- Francesca Saglietti, Florin Pinte, Sven Söhnlein: "Integration and Reliability Testing for Component-based Software Systems", Proc. 35th EUROMICRO SEAA 2009, IEEE Computer Society Press, 2009.
- S. Söhnlein, F. Saglietti: " Auswertung der Betriebserfahrung zum Zuverlässigkeitsnachweis
sicherheitskritischer Softwaresysteme", Automotive - Safety & Security 2008,
Sicherheit und Zuverlässigkeit für automobile Informationstechnik,
H. Klenk, H. Keller, E. Plödereder, P. Dencker editors, Schriftenreihe Softwaretechnik, ISBN 978-3-8322-7681-2, S. 179 - 188,
Shaker Verlag, Aachen, 2008
- S. Söhnlein, F. Saglietti: " Software Reliability Assessment by Statistical Analysis of Operational
Experience", ERCIM News No. 75, p. 17, European Research Consortium for Informatics and Mathematics, 2008
- S. Söhnlein, F. Saglietti: " Software Reliability Estimation for Component-based Systems
by Evaluation of Operational Experience gained with Components", 34th Euromicro Conference on
Software Engineering and Advanced Applications, WIP, Università di Pavia, 2008
- F. Saglietti, N. Oster, S. Söhnlein: " Qualität und Zuverlässigkeit
im Software Engineering", Zeitschrift für wirtschaftlichen Fabrikbetrieb (ZWF), 103, Nr. 6, S. 407 - 412,
Carl Hanser Verlag, 2008
- S. Söhnlein, F. Saglietti: " Nachweis hoher Softwarezuverlässigkeit auf der Basis von
Test- und Betriebserfahrung mit wiederverwendbaren Komponenten", Sicherheit - Schutz und Zuverlässigkeit 2008,
A. Alkassar, J. Siekmann editor, Lecture Notes in Informatics, Band P - 128, S. 303 - 314, ISBN 978-3-88579-222-2,
Gesellschaft für Informatik, Bonn, 2008
Contact Person: Dr. Sven Söhnlein