Detection of Rare Software Failures by Load Testing of Communication Software

LoTCoS - Load Testing of Communication Software

Testing software of communication networks is a very complex task; in particular, it is extremely difficult to identify failures occurring only under very rare circumstances. On the one hand they may occur too seldom in small networks to be detected during testing; on the other hand, large networks in which such failures are likely to occur more often, in general are not available for extensive testing phases due to their cost. In order to support the timely identification of such software bugs new methods are required, aiming at maximizing the reproducibility of real failure scenarios in a cost-effective manner.

Based on this need the primary goal of the research project presented is to check the effectiveness of load testing for communication software. This research project is based on current experiences in industry.

The following goals are part of the project:

• Definition of the system under test and of the investigation goals
  

  At first communication systems are defined which are nodes of a network. Different nets of communication systems are analysed in the project. Fault classes manifesting themselves quite rarely - e.g. concurrency faults occurring only under very special race conditions - are identified and characterized. Load testing strategies are introduced and the goals of the research project are defined.

  Definition of the communication systems:
  
  • attributes of the systems (number of processes, priority of the processes, length of the queues of the system, strategy of the queues, duration of the timeguards of connection setup and teardown, number of entries in the database)

  Definitions of the network:
  
  • attributes of the network (topology, number of nodes, maximal number of connections between two nodes)
    
  • attributes of the connections (routing algorithm, average path length)
    

  Identification of the fault classes:
  • characterization of the faults
    
  • schematic description of possible consequences
    

  Definition of the strategy of load tests:
  
  • attributes of the load (distribution of the connection requests, distribution of the holding time of the connection, distribution of the write access, size of the strings when accessing the database)

  Definition of the goals:
  • results of the analytical model in terms of probabilities of failure occurrence
  • results of the simulation model in terms of frequencies of occurrence
  • variance and standard deviation of the results of both analytical and simulation models with respect to different load parameters

• Derivation of an analytical model
  

  Based on the definitions of the network an analytical model is derived.

• Application of the analytical model
  

  Using stochastic methods and based under realistic assumptions the probability of the different fault manifestations are analytically determined with respect to different load test strategies.

• Development of a simulation model
  

  A simulation model is developed to validate the behaviour of the network of communication systems. This model allows connections through the network to be created and deleted, as well as the execution of predefined processes (like the storage of data) for existing connections. Measurement attributes, e.g. duration and number, are defined.

• Measurements using the simulation model
  

  Faults of specified classes are injected into the system. Their frequency of manifestation is measured with respect to various load testing parameters (e.g. frequency of simulated events).

• Validation of the analytical model
  

  For each fault class the probability of failure occurrence derived from the analytical model is compared with the frequency of failure occurrence obtained by the simulation model. The comparison yields a range of load parameters, within which the analytical model can be validated by simulation.

• Guidelines for the industrial use of the investigation results
  

  Based on the validated results recommendations to optimize load testing in real world applications are derived. This guidance is meant to support the systematic definition of load testing parameters in practice for the purpose of reproducing rare software failures when testing communication systems.