Whether your background is in quality control, telecommunications, or product management, a project that has you testing mobile devices and their applications will surprise you. Here’s what you need to know to make it a success:
- Mobile devices are consumer devices;
- You need to test on emulators and real devices;
- Touch matters;
- Networking often works badly; and
- Usability testing will probably be more important for you than on any project you’ve done before.
To understand how these mantras apply in quality assurance, think through some of the implications:
Mobile devices are consumer devices
Mobile markets–cellular handsets, tablets, and so on–are different. Many of your end users will be people who don’t use desktop computers, or do so only rarely. They know nothing about keyboard shortcuts that your cells have memorized. They’re more likely to have chosen a device based on the color of its case than its CPU specifications. “Hardware refresh” cycles are around a year and determined by physical loss and service contract renewal rather than technology. Desktop computers might resemble furniture or dishwashers in their production and marketing; mobiles are more like shoes.
This has consequences both in technology and use. Mobiles incorporate rather remarkable engineering that can be updated even within a product release; software inevitably struggles to keep up, and often ships with substantial errors. Two examples of the same telephone handset–same brand and model–can easily differ enough in the details of their operating system (OS) releases that one simply fails in an application programming interface (API) that the other handles without a hitch.
You need to test on emulators and real devices
One of the consequences for your test plan is that you’ll need to schedule phases both for testing on emulators, and on model devices. You must take advantage of the emulators (and, occasionally, simulators) that are available for nearly all mobile devices to exploit the opportunities they offer for automation. Any “continuous testing” and comprehensive regression testing probably need to start on emulators.
It’s indispensable, though, to test thoroughly on real devices. With all the engineering variations in what reaches your end users, it’s unlikely that any emulator can keep up with what you truly require of it. However tedious it is to maintain a mobile inventory and tap away at it, there’s no alternative for “shaking out” the faults sure to turn up on your first releases.
It’s not just that emulators don’t track subtle differences; they still largely fail to manage touch. It’s taken years for the testing community to figure out “best practices” for validation of all the behaviors mouse pointers elicit from applications. We simply haven’t advanced far enough yet to have touch interfaces right.
That doesn’t mean touch shouldn’t be tested–far from it! The point is that you’ll want to invest extra time in crafting your test plan to figure out how to address touch. Your end users certainly will be counting on gestural interfaces; in fact, many of them will regard gestures as the natural door to applications. Did your application originate on the desktop, with the conventions and idioms expected there? That doesn’t matter at all to your new mobile audience. Adjust your plans accordingly.
Mobile networking is erratic
What use does your application make of Internet networking? Maybe none. Maybe it’s purely a local application. Mobile devices–often endowed with accelerometers, GPS (global positioning systems), and an abundance of other features–certainly have the potential to do wonderful things on their own.
It might surprise you how often your applications “calls home,” though. Consumers expect applications to tell them the traffic conditions five miles ahead, which of their friends are currently visiting the local block, and whether it will rain tomorrow. Typically all those reports, along with a lot of internal “housekeeping” functions, rely on Web connections.
Those Web connections are better than what science fiction imagined just a few years ago. The engineering underlying them is quite marvelous. It’s also rather immature. In particular, the way the Internet Protocol (IP) and especially the Hypertext Transport Protocol (HTTP) are packaged for wireless 3G transmission leave them vulnerable to delays that simply have a different shape than even the most congested wired environment. It’s not enough to estimate that your application either can acquire a strong signal, or it will switch gracefully to a disconnected mode; you need to prepare and test carefully for long, long delays while nominally connected.
Usability testing will probably be more important for you than on any project you’ve done before
At one level, mobile devices are “just computers,” like the ones you’ve already managed and tested. Many of the details are different, though, and many of these different details will be more apparent to end users than they are to you. You must stage “usability tests” or otherwise watch and learn what your real-world audience does with your audience.
Even if your application is an unusual one, in that all the end users also use a desktop version of the application, you’ll find they have different habits with mobile devices. There’s no substitute for sessions with new real-world users.