Mobile has changed the world, and that includes the world of functional testing.

Because of mobile, the testing matrix has become complex. Because of mobile, the testing process has been compressed. Because of mobile, the testing process has required a strategic use of automation and has put a bigger emphasis on testing strategy and test planning. And because of mobile, there is now a laser-focus on the testing of non-functional requirements and performance testing.

I touched on this subject in last week’s post, and today I want to call it out on its own in the first post of a seven-part series about creating mobile tests that make sense in the technological world we live in today.

5 critical factors have reshaped the testing process

Previously, test engineers have executed suites of tests and produced pass/fail results that merely evaluate the performance of the software. But there are five important changes that are dramatically reshaping the testing process…

1. Expanding test plans

As software gets more and more sophisticated, many developers respond by creating exponentially larger test plans to cover larger sets of functionality and performance issues. If you work on a large team, you may be able to keep pace with this growth — but if you don’t, then this expansion is outstripping the capacities and capabilities of your team.

2. Shorter product cycles

Gone are the days of multi-year software product lifecycles (or lengthy software development lifecycles). Today, development teams – and, by extension, their counterparts on testing teams – are under more pressure than ever to release software far more frequently.

3. More device platforms

With the emerging dominance of tablet and smartphone platforms, testers must have the expertise to test more devices – and accomplish it all in shrinking windows of time. Translation: more tests in less time.

(According to OpenSignal’s data, the number of distinct devices that downloaded their app grew by almost 60% in just one year — from 11,868 in 2013 to 18,769 in 2014. The graphic at the top of this post perfectly (if painfully) illustrates the massive challenge faced by mobile testers.)

4. A changing focus

Traditionally, software testing has focused on validating, verifying, and measuring the performance of a software program, module, or function. Today, that’s not enough. Instead, forward-thinking development teams are emphasizing development and testing frameworks that evaluate software from the user’s perspective and that account for the user’s experience, regardless of underlying technical metrics and performance stats.

5. An emphasis on analysis

With all these changes, development teams want more than a list of test scores and stats. They want analyses of the results and their implications on performance and on changes in development or infrastructure strategies. The impact of these changes are felt particularly in mobile testing, which often present large user volumes, massive numbers of devices, low profit margins, and fast cycles.

Over the next few weeks, I’m going to cover a number of strategies — from understanding your testing matrix to identifying your edge cases — that can help you create faster, yet more comprehensive, testing cycles that are more aligned with your product strategies and market goals

For today, let’s start with the most fundamental thing you need to know:

Know your user

If you don’t understand who your users are and how they use your site, then you’re missing out on creating truly meaningful tests. That’s because a good test plan is about much more than software requirements or functional specs. It starts with a thorough, comprehensive understanding of the user.

Unfortunately, test cycles can be so compressed that it’s too easy to forget that all of the individual functions that are being tested ultimately roll up into what really matters: the user experience. You need to take the time to immerse yourself into the user’s world to fully understand the use cases and usage contexts for the application. Taking the one-size-fits-all approach doesn’t work, because performance requirements and capabilities can vary hugely from user to user.

To properly test for the real user experience, you must invest in fully understanding the people who come to your site. For instance, before you test, it can be helpful to consider questions like:

• What time of day does a user typically access the app?
• How long is the typical session?
• Is the user in a hurry-up mode looking for quick answers and information – or is he browsing in “couch potato” mode?
• Is your app providing a “second screen” experience?
• Will the user likely be simultaneously accessing multiple applications?
• How long do items sit in a shopping cart before they are purchased?
• Is there a correlation between the performance of the app and revenue?

Takeaway

To gain the most complete understanding of the intended user, talk to your product manager or read the marketing requirements document or product requirements document. For high-profile apps, you may even consider investing in market research or focus-group testing to identify preferences and behavior patterns that can inform your testing plans further.

Download the report: The 7 Steps to Pragmatic Mobile Testing