Software and Programmer Efficiency Research Group

GUI performance test automation has two key requirements that go beyond traditional GUI test automation: (1) the need to replay realistically complex interactive sessions, and (2) the minimal perturbation of the measured performance by the tool. First, many existing GUI test automation approaches and tools primarily focus on functional testing, and thus do not need to support the capturing and replaying of realistically long interactive sessions. However, for performance testing, the use of realistic interaction sequences is essential.

The reasons for this are based on the problem that applications interact with the underlying platform in non-functional ways, and that these interactions can significantly affect performance. For example, excessive object allocations in one part of an application may indirectly trigger garbage collection during the execution of a different part; or the first use of a class may trigger dynamic class loading, may cause the language runtime to just-in-time compile and optimize application code, and may even cause previously optimized code in a different class to be deoptimized.

Finally, the size of data structures (e.g. the documents edited by the user) directly affects performance (the runtime of algorithms depends on data size), but it can also indirectly affect performance (processing large data structures decreases memory locality, and thus performance). To observe these direct and indirect effects, which can be significant when programs are run by real users, we need an approach that replays realistically complex interaction sequences. Second, existing GUI test automation tools are not constrained in their impact on performance. However, to allow GUI performance test automation, a tool must not significantly perturb the application's performance.

Record and replay tools can cause perturbation due to additional code being executed (e.g. to parse the file containing a recorded session, or to find the component that is the target of an event), or additional memory being allocated (e.g. to store the recorded session in memory). Thus, we need a capture and replay approach that incurs little overhead while still being able to replay realistically complex sessions. Hereby, we evaluate capture and replay approaches as implemented in a set of open-source Java GUI testing tools.

For more information, check out our publications:

• AST'10 - Automating Performance Testing of Interactive Java Applications
• TESTBEDS'10 - Performance Testing of GUI Applications