Building a test suite might seem like a simple task at first sight, but unfortunately it is not like that. First of all you need to decide what properties you want to test, and since we are talking about mobile devices there are so many things to test that it becomes hard to make a decision.
For instance, is it more important to test the application component or the hardware component? What about network connections? Mobile phones and mobile devices in general are small and often fit in your pocket, but this does not mean they are simple. Mobile phones are devices made primarily for making phone calls and optionally for other services. Nonetheless most modern mobile devices share a lot of functionality with a Nintendo DS that was born as a gaming console, but also lets you browse the internet with some optional software and none of these are very different from a Nokia N800 with the exception that the N800 is not made for making calls (and this is no small exception!).
Let’s say that we want to focus on the mobile web then, and I guess that this is a good choice for people like me who work for a company like dotMobi. Making the decision of focusing on browsing we have already restricted our field a lot, but this does not mean that the task of building a test suite has suddenly become a simple task.
Before even starting to create your tests you need to decide how the test suite will be organized and there are a few options to be considered. You might want to take a safe approach and create atomic tests (one per page) or try to optimize for speed and combine multiple tests in a single page. Both approaches have a few advantages and other disadvantages. If you start creating atomic tests you will have to create a single page or media file for every possible test that you want to have and this means a lot of pages. The advantage of this approach is that you will be confident that there will be no confusion when running the test and there will be no possibility of a test element in a page causing the failure of other test elements in the same page. Let’s say that you want to do tests on CSS, if the browser is not able to interpret or parse correctly some parts of the test CSS, you might have a misleading rendering of other parts of the style sheet that alone would have been rendered correctly. A combination of styles might provide different results and when you are testing functionalities and support you want to be sure that a test will not interfere with another. On the other hand, if you create a single page for every test, you end up with hundreds of pages.
If you decide to go for speed, then you can probably group a number of tests, for example about font effects or image formats (GIF, animated GIF, JPEG, etc) and make the testing process more compact. This approach is definitely better for speed, but introduces a few issues such as the possibility of interference of the tested elements, as mentioned. Also, what about the collection of results? If each test element is self-contained, users can report if supported or not, but if you have one page with 3 or more tests and some work and some others do not, how do you reliably collect this information? You will probably need a more complex form, but this might interfere with the rest of the test case, while a simple pair of links declaring support or not with a few parameters in the query string will be handled by any browser (if the browser does not even support GET requests, I wonder where it could browse).
Another thing that I haven’t mentioned, but should have popped up in your mind, is that tests need to be accurate. You don’t want your users to report that a certain content was not supported or a feature was not working as expected and the test was wrong in the first place. Tests need to be tested!
If you are a purist you might even suggest that each test page should be self-contained, static and not prone to errors generated by third parties such as a CMS serving the content and that the results should be collected using a different interface so that not even links need to be provided within the test page, but only the test itself.
I think that a successful test suite needs to make some trade-offs, start from the simple approach of atomic tests and then build on top of them trying to optimize navigation and speed without losing in reliability. If you can’t trust your tests and the collection of the results, building a test suite is meaningless.
At dotMobi we want to build a test suite that can be used by anyone around the world and this means that it must be first of all simple to use and solid. Anyone who is willing to share their experience or test their mobile browser should be able to go through it and report the results in an easy and trusted way.
