As smart phones move into the consumer and enterprise space, there is a growing need to test the suite of applications that reside on these devices. Test automation has always been an attractive alternative to expensive, time consuming and inconsistent manual testing – the question now becomes “How do we leverage test automation in this new, ill-defined space?”
Let’s try and understand test automation and test execution that will yield the highest return given the maturity of the smart phone space. Let’s also talk about the factors we should consider when implementing a test architecture to support this space.
Consider the Challenges :
Before we address the question of smart phone automation, we have to accept the unique set of challenges smart phone development and testing represent and how these challenges impact your overall mobility testing program. Smart phones, when combined with other mobile devices, represent the fastest growing segment in IT terms of market share, OS deployments, device deployments and software deployments, with a constantly growing set of basic functionality. At the same time, these devices represent one of the least secure deployment architectures IT, as a whole, has ever been asked to address. This means any test automation program needs to be pragmatic, supported by a flexible supporting architecture and focused on high-yield opportunities – your test automation yield being defined in terms of full-time equivalents – high being defined as at least eight to one. You need these types of yields to help offset the impacts of a constantly changing device and application landscape.
In terms of a “smart phone test automation program,” the target needs to include high-yield elements of your application space that the current set of automation tools can address. How much test automation to apply towards a program must align with these goals. Otherwise, the long-term success of the test automation effort will be in jeopardy and will almost certainly fail.
Though the focus of this article is on the implementation of a smart phone test automation program, it must be noted that test automation is not a silver bullet – treating it as a “cure all” will certainly lead to disappointment.
Test architecture: Leverage “soft” solutions that work across multiple devices and multiple OS’s
The early adopters in the smart phone testing space implemented architectures with both logical (i.e. test automation tools) and physical (i.e. cradles) components. As smart phone vendors, developers, consumers, and enterprises begin to treat smart phones as business appliances, test architectures need to become cheaper, faster and more nimble. History has shown us that this leads to software-driven solutions that leverage OS provisioned “hooks” and on-device agents.
New “players” in the smart phone test automation space are introducing promising software-driven approaches to smart phone test automation; approaches that require minimal hardware – often just an IP address on your existing network – while leveraging existing test management toolsets. This enables you to implement a cost effective on-site smart phone test automation solution – a must have for many financial institutions.
Test design: Use keyword-driven automated tests, manual look-and-feel tests
A keyword-driven test design and automation approach yields a significant return in the smart phone testing space. The difference between the smart phone space and more traditional interfaces is the focus of your test automation design. To date, harvesting the highest yield from your keyword-driven test case designs involves focusing only on business functionality – look-and-feel testing appears to be best left in the hands of manual exploratory testers. This split supports a sustainable test automation solution while preventing look-and-feel issues from escaping into production undetected. It also enables automation solutions that work across more than one device or one operating system.
In summary, strive for:
- Business events captured in keyword driven automated tests.
- Look and feel captured by skilled manual exploratory testers.
Build: Focus on building what you need and refactoring when additional ROI is available
Constructing a test automation solution involves a set of choices that automation engineers – especially more disciplined automation engineers – may find difficult to accept. Smart phones represent an evolving architectural space so investing in a sophisticated test automation framework may be premature. Opportunities to create modular, maintainable toolsets should not be ignored but try to determine if the investment is going to harvest an eight to one return before proceeding. In this space, you have to be willing to drop current solution sets and invest in newer solutions – if the return on investment changes to favor a different approach. An Agile approach to both test automation and the supporting test automation framework yields the highest return – build what you need when you need it and re-factor/rebuild when the return on investment dictates you should.
Execution and reporting: Test management solutions that support mechanized execution
The sheer volume of mobility test cases requires a test management solution that enables mechanized execution and reporting across a large number of devices on a regular execution schedule. Several commercial solutions provide this functionality – look for test management solutions that support software-based automation solutions. This will allow you to harvest the return from the next generation of tooling that leverage approaches that require minimal hardware – OS-provisioned “hooks” and on-device agents.
As has been stated before, test automation is not a silver-bullet – it is a powerful test enabler when used appropriately. Test automation is becoming an absolute necessity with the explosive growth of smart phones and the applications that reside on those devices. Appropriate investment in the test automation will yield returns – but as we have already discussed, this return must be higher than traditional test automation (“8 to 1” vs. “4 to 1”). As the test automation tools, devices and OS’s mature your organization should be willing to move into more effective solutions – especially when there is a significant difference in ROI. If the history of IT and testing in IT shows us anything, it is that you do not want to be behind the curve in a quickly evolving space – your costs will quickly become prohibitive if you do not adapt to changes within the mobile and mobile testing space.