Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning

Abstract

In the industrial setting, mobile apps undergo frequent updates to catch up with the changing real-world requirements. It leads to the strong practical demands of continuous testing, i.e., obtaining quick feedback on app quality during development. However, existing automated GUI testing techniques fall short in this scenario as they simply run an app version from scratch and do not reuse the knowledge from previous testing runs to accelerate the testing cycle. To fill this important gap, we introduce a reusable automated model-based GUI testing technique. Our key insight is that the knowledge of event-activity transitions from the previous testing runs, i.e., executing which events can reach which activities, is valuable for guiding the follow-up testing runs to quickly cover major app functionalities. To this end, we propose (1) a probabilistic model to memorize and leverage this knowledge during testing, and (2) design a model-based guided testing strategy (enhanced by a reinforcement learning algorithm), to achieve faster-and-higher coverage testing. We implemented our technique as an automated testing tool named Fastbot2. Our evaluation on the two popular industrial apps (with billions of user installations) from ByteDance, Douyin and Toutiao, shows that Fastbot2 outperforms the state-of-the-art testing tools (Monkey, APE and Stoat) in both activity coverage and fault detection in the context of continuous testing. To date, Fastbot2 has been deployed in the CI pipeline at ByteDance for nearly two years, and 50.8% of the developer-fixed crash bugs were reported by Fastbot2, which significantly improves app quality. Fastbot2 has been made publicly available to benefit the community at: https://github.com/bytedance/Fastbot_Android. To date, it has received 500+ stars on GitHub and been used by many app vendors and individual developers to test their apps.