Evolution and Fragilities in Scripted GUI Testing of Android applications

In literature there is evidence that Android applications are not rigorously tested as their desktop counterparts. However – especially for what concerns the graphical User Interface of mobile apps – a thorough testing should be advisable for developers. Some peculiarities of Android applications discourage developers from performing automated testing. Among them, we recognize fragility, i.e. test classes failing because of modifications in the GUI only, without the application functionalities being modified. The aim of this study is to provide a preliminary characterization of the fragility issue for Android apps, dentifying some of its causes and estimating its frequency among Android open-source projects. We defined a set of metrics to quantify the amount of fragility of any testing suite, and measured them automatically for a set of repositories hosted on GitHub. We found that, for projects featuring GUI tests, the incidence of fragility is around 10% for test classes, and around 5% for test methods. This means that a significant effort has to be put by developers in fixing their test suites because of the occurrence of fragilities.