Hawkeye: Change-targeted Testing for Android Apps based on Deep Reinforcement Learning

Abstract

Android Apps are frequently updated to keep up with changing user, hardware, and business demands. Ensuring the correctness of App updates through extensive testing is crucial to avoid potential bugs reaching the end user. Existing Android testing tools generate GUI events that focus on improving the test coverage of the entire App rather than prioritising updates and its impacted elements. Recent research has proposed change-focused testing but relies on random exploration to exercise the updates and impacted GUI elements that is ineffective and slow for large complex Apps with a huge input exploration space. At ByteDance, our established model-based GUI testing tool, Fastbot2, has been in successful deployment for nearly three years. Fastbot2 leverages event-activity transition models derived from past App explorations to achieve enhanced test coverage efficiently. A pivotal insight we gained is that the knowledge of event-activity transitions is equally valuable in effectively targeting changes introduced by App updates. This insight propelled our proposal for directed testing of App updates with Hawkeye. Hawkeye excels in prioritizing GUI actions associated with code changes through deep reinforcement learning from historical exploration data. In our empirical evaluation, we rigorously compared Hawkeye with state-of-the-art tools like Fastbot2 and ARES. We utilized 10 popular open-source Apps and a notable commercial App for this evaluation. The results showcased that Hawkeye consistently outperforms Fastbot2 and ARES in generating GUI event sequences that effectively target changed functions, both in open-source and commercial App contexts. In real-world industrial deployment, Hawkeye is seamlessly integrated into our development pipeline, performing smoke testing for merge requests in a complex commercial App. The positive feedback received from our App development teams further affirmed Hawkeye’s ability in testing App updates effectively.