Android App Performance Testing: An End-to-end Approach

Android application performance tests need to follow an end-to-end application delivery model that tests for all the varying scenarios on each component involved. That way, android app developers can gain a true understanding of how their Android apps will perform on a given device as the end users would perceive them.

In a recent DevX article, we proposed such an end-to-end approach for mobile application performance testing. In this article we validate that approach for Android developers by will walking through an Android application performance test using a sample banking app.

Android App Performance Test Components

The Android app performance test involves the following components and tools.

Android Application

The sample Android app is a hybrid application for processing banking transactions. Hybrid mobile applications are installed on mobile devices just like native mobile apps but users can access them via either an installed app or the device browser. The aim is to test this application for performance and find different performance viewpoints.

Server Component

The server component (see details below) handles the business logic for the banking transactions initiated from either a mobile thick/thin client or a desktop Web browser.

• Operating System: Windows Server 2000
• JDK: 1.6
• Web Server: Apache Tomcat
• Database Server: MySQL

Network Emulator

A network emulator recreates a wide variety of network conditions such as latency, jitter, packet loss/error/reordering, and bandwidth restrictions, so that developers can simulate environments such as Wide Area Networks (WANs), Wireless LANs, GPRS, 3G, 2G, IP over Radio / Radio over IP (RoIP), Satellite or MPLS networks. The network emulator determines the traffic load generated by the application under different network conditions. We used Infosys WindTunnel, an in-house tool developed by Infosys Labs, for this performance test.

Load Generator

A load generator tool generates load on the application under test, and it will measure the load times. It allows configuration changes to ramp up the load, increase concurrent users, increase transactions per second, enable parameterization of the URLs, and name each URL. We used JMeter to recreate the load from different delivery modes for the sample application.

Android Device

We used both a high-end and a low-end device for testing to because performance can vary and we wanted to prove the importance of variability in device configurations on the same platform.

Performance Monitor

Perfmon is a performance monitoring tool that is mapped with windows. The required performance counters are configured before running the test. After you start the configured data collector set, it automatically collects the results data during the scenario run.

Android Debug Bridge

The Android Debug Bridge (adb), a tool that comes with the Android SDK, allows us to control and interface with our Android device. We used various adb commands to monitor the sample hybrid Android app.

• adb shell top — Returns the CPU info for all the Process IDs (PID)
• adb shell dumpsys meminfo <PID> — Returns the memory info for the particular Process ID
• adb shell procrank — Returns the PSS, RSS, VSS and USS for all the process IDs (PID)
• adb shell cat /proc/cpuinfo — Returns the processor information of the emulator
• adb shell cat /proc/meminfo — Returns the memory Info for all the process IDs (PID)
• adb logcat — Returns the log messages, including GC

Dalvik Debug Monitor Server

DDMS is an Android debugging tool that provides port-forwarding services, screen capture on the device, thread and heap information on the device, logcat, process, and radio state information, incoming call and SMS spoofing, location data spoofing, and more.

DDMS is integrated into Eclipse and is also shipped in the tools/ directory of the SDK. To launch the tool, follow these instructions.

• From Eclipse: Click Window > Open Perspective > Other... > DDMS.
• From the command line: Type ddms (or ./ddms on Mac/Linux) from the tools/ directory.

To set up representative Android app performance testing scenarios, follow the instructions for each of the following settings.

Workload Type

1. Script the necessary flows for different workload types using the load generation tool for the server.
   • Thin client — Install the hybrid application in your application server. Record the necessary flows using JMeter.
   • Thick client — Install the hybrid apk file in the emulator. Record the necessary flows using JMeter.

Workload Mix

2. Load the server with the different workload scenarios:
   • 100% from the thin client
   • 100% from the thick client
   • Combination of both thick and thin client (For example, 40% load from the thin client and 60% load from the thick client)

Network types

3. Identify the different network profiles for which you will test. Set up the network emulation tool for the following profiles.
   • Airtel* Average 3G
   • Airtel* Best 3G
   • Airtel* Worst 3G
   • Airtel* Average 2G
   • Airtel* Best 2G
   • Airtel* Worst 2G
   
   
   Airtel is an Indian telecom operator. Note that the speed emulated for the profiles do not represent the actual values, but they should during actual tests.

Device types

4. Install the Android hybrid apk in both the high-end and low-end devices.
   • High-end: Samsung Galaxy S
   • Low-end: Samsung Galaxy Y

During your actual performance tests, be sure to recreate all the scenarios applicable to the test application's context.

Android App Performance Test Execution

Here are the steps for actually executing the Android app performance test.

1. Load the server: The server was loaded with the scenarios identified in the previous section using JMeter. The server was monitored for performance while loaded.
   
   
2. Select a network type and a device: With the server loaded, the identified network profiles were recreated using the network emulator tool. In the tool, we edited the bandwidth, jitter, packet drop parameters based on our requirements such as 3G or 2G. The device was configured to use the emulated network. The POC MAC address of the device and the server were configured in the network emulator tool to filter the relevant information.
   
   
3. Access the application on a real device: The hybrid application was installed on the device to be tested. The Android command for installing the application (apk) is:

   adb install <apk>

   Once installed, the transaction was carried out on the real device with server and network conditions emulated. In addition, the transaction was also carried out on the on-device browser.
   
   
4. Monitor the components involved: The components were monitored during the transaction execution on the device.

These steps were repeated to cover all the test scenarios mentioned earlier. The following data was captured on the device using adb.

Android App Performance Test Results

Here are a few sample results that can be obtained by testing. Note that these results are only the tip of the iceberg of what you can do with end-to-end performance testing.

Device Performance

The graph below is the device result which is monitored by the adb commands. We observed no huge difference in PSS, memory allocated, heap size or used memory with the increase in load.

Click here for larger image
Figure 3. Device Performance Results

Thick and Thin Client Performance Comparison

The graph below shows the difference between thick client and thin client. This allows us to understand how the rendering of responses affects performance from the end user's point of view.

Click here for larger image
Figure 4. Thick and Thin Client Performance Results

Network Performance Comparison

The graph below presents the network speed for 3G and 2G networks. Load is generated via the load runner tool. We observed that the 2G response time rose higher than that of the 3G as we increased the load.

Click here for larger image
Figure 5. Comparison of Network Performance Results

As you can see, extensive analysis can be performed with the end-to-end data available for the different components involved in mobile application delivery. Such analysis would not be possible if the device were tested in isolation with no control over server/network conditions.

Conclusion

This article presented a proof of concept for end-to-end mobile application performance testing using a sample Android app. It detailed every component to be tested, the method for testing as well as the challenges involved.

When adopting this approach, be sure your Android application performance tests are under these conditions:

• On multiple devices
• With native applications as well as browser-based applications
• Under production like delivery mode-based workload mix on server
• Under production like geography-based workload mix on server
• Under varying network profiles
• Under varying device states