Feeling Good About Your Performance Test Coverage?

Do you feel confident your Mobile App performs well enough to not only meet your company’s expectations but also your customers’ expectations?

Not all Mobile Apps have the same architectural design despite what some believe. Then, does it make sense to provide the same test coverage to all types of Mobile Apps? If you think it does make sense, ask yourself, “am I providing maximum test coverage efficiently?” For example, would you test a communication Mobile App like LinkedIn, Twitter, or Facebook the same way as you would an eCommerce app like a food delivery service? Would the same set of tests for a video streaming app be applicable to a game app or an eCommerce app? To be efficient in your test coverage, I suspect you will need to understand what types of performances are important to your company and to your customers/users. Each Mobile App type I refer to is architecturally defined as a hybrid app, which means usage of a Web server as well as the device’s features like a camera, audio own storage.

Performance testing is an umbrella type of testing. I break down Performance Testing into 4 concepts:

1. Speed/timing

2. Load

3. Stress

4. Endurance

These concepts can be defined in different ways depending on the application under test. Define your needs of performance based on these concepts. Your Mobile App may display behavior concerns for your company but may not be seen as a problem by your users. Testers need to decide to balance out the type of tests applied to successfully launch the app for usage and meet your company’s mission.

Let’s examine each type of Mobile App and then discuss the types of most appropriate tests which provide efficient and maximum test coverage.

Speed/Timing or Response Tests

Communications apps rely on transactions that primarily fall on the responsibility of the webserver. When Mobile Apps utilize the device’s features like a camera, audio, notifications, storage, then speed becomes a factor. Depending on the criticality of speed/timing is to the user response of these features existing on the device to communicate with the app, a speed test becomes a vital test where device features are utilized interacting with the communication app. For example, being able to take a photo you took is not as timingdependent to post on LinkedIn or Twitter as it would be to loading that photo to a medical app where you are sharing the photo which will be sent to your doctor’s office. How fast can the app respond to loading a photo into the app to be able to be sent would be the performance test? Setting up the appropriate condition to test that response time may or may not be important to your app’s usage. Testers need to make decisions based on what is most relevant to the c o m p a n y ’ s mission for the Mobile App’s intended use as well as how r e l e v a n t performance is to customers using the app. If the Mobile App is communicating statuses in real-time where users rely on those statuses, timing then becomes time-dependent therefore, the speed or timing tests will become more significant. Speed/Timing tests materialize based on context which is why testers need to recognize both the company’s goals and the users’ goals for the app and prioritize their testing activities accordingly.

E-commerce apps like a food delivery service have a few areas which require speed/timing tests. In addition, there might be some areas that require fast communicating, the speed at which search features exhibit results, calculating the total cost, or verifying payment method are all accessing a webserver performing those functions. Therefore, the Mobile App tester will rely on the web server tester to test those functionalities. A Mobile App tester can also conduct web server performance testing although performance testers typically concentrate on webserver testing only. If the Mobile App takes advantage of the audio on the device, does the Mobile App respond to the audio commands within an expected rate? Does the Mobile App utilize Notifications? A food delivery service can include notifications like expected delivery time, the shopper is communicating with the user within the app itself, confirmation of the order placed which are all important to the customer. Is the notification timely or is that notification lagging behind? What is the expectation by the customer and by the Mobile App company?

Gaming apps pose a lesser requirement when talking about speed/ timing. Game apps are typically played by users accessing a web server and are not reliant on the mobile device to provide transactional data. Speed tests will be centered on the transactional speed of the webserver which means testing the webserver and not the Mobile App. Typically, web server testing is done by a different type of tester possessing certain skills while a performance tester has other specific skills to conduct performance testing on a webserver very similar to performance engineering. To learn more about webserver testing, I suggest following Performance Test Academy on Twitter and PerfBytes podcasts site are great places to start.

Video streaming apps are a combination of E-Commerce and Game Apps when talking speed and/or timing testing depending on the app. All of the content available to stream originates from the webserver. Does the video streaming app send notifications about available or changed content based on the user signing into an account? Does the app intend to have a functionality where downloading content is a primary or even a secondary function for the app? Paying close attention to the app’s intended usage as well as meeting the company’s mission for the app can dictate the types of tests prior to release.

Load Testing

Communications apps and testing the load can have different meanings depending on the Mobile App’s objective. When performance testers talk about “load,” they are concentrating on the load or number of concurrent transactions accessing the webserver. But the load on a mobile device can refer to concurrent transactions the mobile device’s CPU is accessed. If the user has too many Mobile Apps in use at the same time and running in the background, the load on the CPU can create annoying problems or worse, crashes. Knowing the limitations of your app’s thresholds is a vital test. Developers can provide the limitation of the CPU load on a mobile device but they also may not know what that threshold is, due to the many different types of devices. However, with a typical communication-type app, the load tends to be on the webserver and not on the device CPU. If the app is a safety-critical app, the tester should try to explore the threshold value to avoid a disastrous event. Checking with stakeholders might be a solid approach to assist in the planning of appropriate test decisions. To be efficient, project teams need to communicate early in the project regarding the intended design of the app and testers need to be involved as part of the project team.

E-commerce apps and Gaming Apps: Load Testing puts a very little load on the device’s CPU. Unless the Mobile App requires downloading data, images, video directly saved onto the device, this type of test is not applicable to Mobile App testing.

Video Streaming apps pose an interesting challenge for Mobile App testers. Many apps we currently use for streaming require connectivity to a webserver. However, some video streaming apps allow the user to download the video content, making use of the device’s CPU. If the video content infringes on the CPU to the point of not allowing other apps to run in the background, the functionality of that Mobile App will prove to be unsatisfactory to the user. Does the Mobile App allow notifications from other apps while a video is playing? Can the video be paused and not place an unforeseen load on the CPU? These tests might be considered important information affecting the use of the Mobile App.

Stress Testing

Communications app: Stress testing is testing the reliability and stability under normal and abnormal conditions. Having a known set of thresholds after setting a baseline of values although developers can set thresholds and communicate those values through documentation or verbal communication. Because communication apps themselves access data to and from a web server, there is little stress testing required. However, some conditions may be required to be tested. Can the app send only a certain-sized image taken by the device’s camera? If the image is too large, how does the app communicate the problem? Does the app crash or recover? Is there a limitation of notifications it can display either from within the app or from other apps? Do the Mobile App’s notifications step on other Mobile Apps? (Note: this was an actual bug I found with one communication Mobile App stepping on other apps which is why this question needs to be asked.) Depending on the criticality of the purpose of the Mobile App, these tests might all be vital testing.

E-commerce and Gaming apps: Stress testing can involve downloading the app to be saved onto the device. If the app is too large to be stored on the device, how does the app recover? Was there a warning to let the user know they need more free space on the device? Gaming apps in particular can be large with heavy graphics so the app should have a built-in check during the installation of the Mobile App and warn the user if the app is too large. If downloading is started but cannot complete, how does the Mobile App respond to the adverse condition? Will it crash the device with no explanation of the circumstances of the crash? Another type of test to consider, when the connection to the webserver has been lost, how does the Mobile App react? These types of apps tend to have advertisements pop up during use especially the game apps. But those transactions, along with storing the place where the user is while playing the game (or placing an order), are done through the webserver. When there is a loss of connectivity, does the app maintain the location where the user is, in using the app? Example: While playing the game, is the score saved locally on the device, and then when connectivity is returned, does the app update the data? Or does the app become unusable? With games, stoppage of game playing might anger gamers not being able to continue. These tests should be a standard stress test for any Mobile App and can be a fairly quick test to set up.

Video streaming apps can only allow downloading the content, allowing the user to view it completely locally on the device. This is the only type of video content viewing app that would require stress testing. Streaming content would involve access to the webserver and not have any impact on the device. Stress testing for the video content storage and playing type apps requires pushing the limits of the storage capacity, the type of image files to be played, and testing if that content file type is supported or not within the app. Will the app recognize if the image file type is playable, provide an error message if it’s not compatible, or simply crash the app? Evaluate whether the impact of these tests would be important for the intended use of the app.

Endurance Testing

Endurance performance testing of Mobile Apps doesn’t come much into play for hybrid apps and definitely not with mobile web apps. This is because these types of Mobile Apps rely on transactions between the device to a web server. Endurance type tests need to be done on the webserver and not on the downloaded Mobile App.

E-commerce apps do not force the user to download inventory on their devices as this would be considered an outdated design type. Endurance tests for E-commerce testing will be done on the webserver and not on the Mobile App on the device itself.

Gaming apps have the ability to sustain without connectivity and then once connectivity is reestablished. Did the gameplay data endure reconnecting to the webserver where the gamer could continue to play the game with no interruption is apparent? An example of this type of game is a casino-type app where the saved coins earned from play when not connected are updated appropriately to the web server once reconnected. Losing their winnings would not make the gamer happy once there is a reestablished connection with the webserver. Specifically, can the user play the casino app for hours, stop play, exit the app and then start the app again without connectivity? Can the game then be updated once connectivity is reestablished? Would there be any loss of data on either the app itself or once the data is transferred?

Video streaming apps also rely on webservers to provide content. If the app allows the downloading of the content, testers should then test the ability to download content and set up some endurance-type tests. One test could be: play content, replay content, maybe on repeat to ensure there are no crashes. If the app stops playing after one completed content but is not available for further playing, it could create a negative reaction to the company’s brand. Testers and stakeholders need to examine the risk of a poor result the conditional test is not tested. How extensive would the impact of not testing the conditions of endurance to the Mobile App be?

Mobile App performance testing requires testers to discern risk for a poor performance observed by the user. Testers need to understand what types of performance tests are appropriate for their type of Mobile App, the purpose of that Mobile App, and the Impact of not conducting certain tests. Do you test only to meet the company’s intended design or do you concentrate on tests meeting the user’s expectation? Testers and stakeholders need to prioritize what is the greatest impact overall, but testers need to evaluate a user impact as well as their company’s brand. Each impact is a vital discussion point with stakeholders within the project.