Robust QA is an integral part of mobile/embedded software development. Mobile QA can, at times, be quite daunting even for experienced embedded professionals especially given the permutations and combinations of devices, carriers, features, platforms, content, security, performance and much more. Though we can dedicate a complete book to write about the best practices on mobile QA, let me try to highlight a few key areas that should be covered:
Mantra: Nimble process, Domain specific coverage!
1. Device Capabilities: The QA team needs to understand device features really well. If the application needs to be tested on various devices, it is virtually impossible to keep track of device feature sets unless the team has a feature/model/platform matrix that is used as a reference for developing test cases. The most common mistake I have seen across apps is inability of QA/Dev teams to gauge device display resolution and relate the display related nuances to assets/resources of their applications. SD Card and dependency on storage is another one.
2. Platform Interaction: Lets put it in layman’s terms. Device send signals to the Operating System (platform) which sends signals to the runtime (virtual platform) which sends signals to your application. Further, access to device/platform resources is governed by policies set by/on the platform. If you don’t understand this, writing test cases around interaction with OS/Runtime is virtually impossible. Memory management is wrongly considered to be an in-process imperative by a large number of QA professionals, another common mistake that requires early attention. It is important to understand bidirectional communication between your app and the platform and via platform with other apps. Get this and your architectural thought process will improve, hence better QA around Platform Interaction
3. Network Interaction: A number of platforms put policies/set restrictions on how you access network resources (edge/3G/wifi), concurrency around network connections and at times the data formats that are allowed. If you don’t understand how your app communicates over transports with your servers (beyond the device) and how your app feature set is dependent on this communication, you will certainly have a few ugly hiccups related to performance, no data access and/or your app being blocked from distribution channels. QA team needs to account for these scenarios. Again, understanding of this imperative and comprehension of these scenarios is the key before good test cases can be written around these.
4. Application Features: Functional testing. What else can I say? If you don’t understand the primary, secondary and tertiary features of your application. Your test coverage will always be below par. Usability, User Experience, Screens, Data, Ads, Metering … all of it is part of app features. Get the features right, you will get the functional test cases right. The most common mistake: What to do when a feature misbehaves? Simple: Graceful recovery or alternate scenario.
5. Location of Consumer: If you cannot think like a consumer, your app will misbehave in the New York Subway, In London Tube, Under water, Around Tall buildings, near bad vibes and during the demos. On a serious note: Understand the usage scenarios from consumer’s perspective.
6. Service Provider: Plan for edge, make the experience rich on 3G, even better on 4G and the best on Wifi. Understanding data formats and how they behave across these carrier network access levels is the key to writing great test cases around service provider scenarios. Another one is carrier policies: data limits, usage plans, internet tethering policies, get your provider matrix ready. By the AT&T’s network is ….. than Verizon …. than Sprint …. to I left it at my mommy’s place is not a valid excuse for a good QA team.
I can go on and on and give 100s of examples around these scenarios. My take on QA is simple:
a). Understand the domain (write test cases that are domain aware)
b). Keep the processes nimble
c). Automate last (imagine having automated tests running in a huge environment covering 5% of the functionality)
Tune in for more ….
Advertisement
Like this:
3 bloggers like this post.
I completely agree.
Understanding the platform highly essential in writing good test cases. Similarly, if our understanding of the app itself is not correct i.e. we do not get the feature priorities right, it would end up affecting the quality of test cases.
I personally believe, Test Driven Programing is the way to go about. As a developer, if I can start by writing a test case around the feature and ultimately aim to pass the test case, it effectively helps my tester. It also helps in eliminating lots of problems right in the bud which could have escalated later. A simple 1-10-100 rule.
Steps involved in TDD are :
1. Write a test case around the feature.
2. Run the test case. it would invariably fail as we do not have the feature implemented yet.
3. Write the code so that the test passes.
4. Run the tests again .
5. Clean the code.
And this process is repeated.
Again, it should be noted that if we do not have a sound understanding of the features at hand, we would end up writing either incorrect test cases or incomplete test cases.
My proposed solution would be :
1. Understand the feature specifications.
2. Write user stories and use cases for the features keeping the recovery scenarios in mind.
3. At the same time write test cases around the use cases .
4. Implement a feature keeping the test cases in mind.
In addition to all of the great points mentioned above I would like to add my 2 cents.
Start testing early – Teams often tend to delay testing right till the very end of the project cycle. Then obviously nothing works as it is supposed to, giving lot of stress to developers and testers. If we start testing early, the QA teams can catch a lot of defects early on.
Test everyday – Each developer should make it a habit to spend an hour each day to ensure that the code he wrote, did not break any other functionality. Its much more easy to fix a broken function early on rather than realizing the mistake after having written 50 functions depending on the output of the wrong function.
Rigorous Testing – Quite often people testing the app don’t “rough it up”. They are very gentle with the application. They wait for the data for that screen to get downloaded (loaded from the database) before tapping on any button on the screen. Gentle testing does not help the application development effort at all. When anybody is testing the application, their aim should always be to find flaws in the application.
Testing Prioritization – It is very important to prioritize the test effort . During the early stages of the application all of the testing effort should be towards ensuring that the developers have not made any mistakes in the basic architecture (and functionality) of the app. Once these aspects of the app has been verified then the testing effort should shift towards other finer points of the app.
I totally agree with the key areas mentioned!
QA, a tearm associated with an individual, running the app and trying to go crazy with it. Crazy, to crash it, to make it behave the way it is not supposed to, or just to find if places where things are not according to specifications.
Some people feel, QA is the easiest process in a company. Its a process where a person sits with a phone, or many phones and devices, and just plays around with the app.
But this is only the surface. In reality, QA is the process that can control the destiny of a product, in front of of the clients and in front of the consumer. QA person has to think and understand the details of how the devices work, how their OS functions, how and what are the mood sets in which the consumer might interact with the app, and finally what will be the impact of the app on the consumer.
He has to understand the device, understand its short comings, and then check the app against these short comings. He has to not only check the user interface, but also keep in mind the guidelines that can beautify the app. Beautification can be as small as the application is according to requirements, or as through as all the error scenarios have been covered.
According to me, QA should not be considered as a job, but a learning process. It can really help a person to learn a platform in more depth. It can help a person to understand the advantages of one platform over other.
For doing QA, one has to think beyond the current beauty of the application, and try beautify it a but more. He has to think about the consumer, and more specifically about the brand value of the application.
For me the QA process is the best way to get insight into all the aspects of the application. It starts with understanding the requirements, is followed by writing test scripts that cover all the features to be tested, and is concluded by running multiple test passes around the app. In these phases, QA team has to be active when the app is being developed, TDD, when its dev complete, and even after the launch, trying to find errors reported by customers, and trying to get in place for the next version of the app.
While QAing an app, one has to keep following things in mind:
1. Features of an application: Don’t make assumptions. Making an assumption, might result in killing a specification.
2. Understand all the functional aspects in advance. If in doubt, its good to ask questions.
3. Test cases should be feature specific. Look out for edge cases, but don’t forget the base cases.
4. Consider all the scenarios, by keeping the consumer in mind. If we avoid this, we loose the customer.
5. Check the UI throughly. An app with good features but bad UI can be a big failure.
6. Start automation when you have achieved majority test coverage.
7. Try to find a balance between functional testing and User experience testing.
8. Don’t ever ever ever forget User Experience testing. Good UI design, great feature set, but a frustrated consumer is a bad mix.
Many think that testing is easier than development. In reality it is not. One can definitely say that development is 100% complete (if not atleast to a certain percent) but the same cannot said with respect to QA. Even with best efforts from the QA team they tend to miss certain test case scenarios. The world has seen this when windows crashed in front of bill gates on the demo he gave and recently iPhone 4 and its issue with the antenna. In the end QA team always comes under the hammer when the product goes live. Though QA cannot guarantee a 100% efficiency on any release its quality can certainly be improved if we follow the above steps. Automation as mentioned above can reduce the load on QA team since it increases the efficiency and also saves valuable time. Apart from that the QA team can have a bullet point checklist that they need to verify before sending each build for respective platforms. This would help them not to miss any high priority test case scenarios before delivering a product.
Quality Assurance in simpler words is a process to determine whether product or services that we create, meet or exceed customer expectations.
Quality Assurance is a very important process as development. If a proper QA is not done on an application, the application is sure to fail. And no one buys a fail product or application.
It is always thought developer is a better designation than tester. And those who are not so smart go into testing. But I think a good tester can become a very good developer, as he knows all the areas where a developer goes wrong, and developing he will try to avoid those mistake.
Moreover it is the QA team and not the developer who is responsible with the product is launched in the market, as it was his responsibility to catch all the bugs before the application is launched.
Few things that a QA team needs to take care of
1) Requirement from the customer is clear. Don’t make any assumptions
2) For each user stories/functionality there should be at least one test case.
3) Other than the main scenarios, each test case should have an edge case scenario in it, which describes what should happen if any thing goes wrong.
4) Get the test cases verified from the client.
5) In case of change of functionality, make sure to update the test case document.
In case of testing mobile application, always check for the following scenarios.
1) Check for memory leaks scenarios if possible
2) Check for scenario where is a memory card in the device when the application is launched.
3) Check for false network condition
4) Network connection gets switched between 3G or wifi
5) If a third party is invoked in case of communication between the client (mobile) and server, make sure that the connection between the client and the third party is happening properly. As we recently found out in case of BES network, the entire request made to the content server was being routed via BES servers.
6) UI of the application is very important. Make sure that once of the images in the application is looking blured. If possible have different set of resources for each set of devices (like one of iphone 3 and one of iphone 4)
In addition to all of the great points mentioned above I would like to add the following
Start testing early – Teams often tend to delay testing right till the very end of the project cycle. Then obviously nothing works as it is supposed to, giving lot of stress to developers and testers. If we start testing early, the QA teams can catch a lot of defects early on.
Test everyday – Each developer should make it a habit to spend an hour each day to ensure that the code he wrote, did not break any other functionality. Its much more easy to fix a broken function early on rather than realizing the mistake after having written 50 functions depending on the O/p of the wrong function.
Rigorous Testing – Quite often people testing the app don’t “rough it up”. They are very gentle with the application. They wait for the data for that page to get downloaded (loaded from the database) before tapping on any button on the screen. Gentle testing does not help the application development effort at all. When anybody is testing the application, their aim should always be to find flaws in the application
Testing Prioritization – It is very important to prioritize the test effort . During the early stages of the application all of the testing effort should be towards ensuring that the developers have not made any mistakes in the basic architecture (and functionality) of the app. Once these aspects of the app has been verified then the testing effort should shift towards other finer points of the app
Yes,QA process is very important for any software development. At last QA team has to show the green signal for product launch.Before product launch we need to make sure that our app will behave normally in all the devices(considering all the device features,screen sizes and data plan),in all the best and worst case scenarios.The first step of QA process is to check each and every functionality and screen layouts are as per application requirements and user stories. Assuming something in dev phase and testing phase must be avoided.To make any product development lifecyle as fast and complete we need test cases build around the app.A test case should give clear explanation about “How our app functionality should behave in all the possible scenarios”,”How our app will behave in upgrade scenarios” etc.
“Discovering the unexpected is more important than confirming the known”
This quote will not only suit for QA team.It’s for every team member in the project.
QA is an important phase in software development. QA Team shoulders a greater responsibility. In addition to analyzing the application physically, for certain functionality/scenarios, the team must go through the code. Some examples of these type of scenarios will be i) to ensure that old data from database/file-system is removed periodically, this functionality is present in many news/feed reader type applications, ii) to ensure that random content from a list such as tip of the day is displayed correctly.
Team must also validate and make sure that excess network calls are not made. Tools such as fiddler, charles are of great help in this.
Test cases play a major role in reducing manual/physical QA and also to validate edge case scenarios which occurs rarely or never occurs at all during manual QA.
Another important thing is to benchmark every functionality. Doing this periodically and comparing the results ensures that the build is not regressed.
Team should also have a sound knowledge in platform usability/UI and also validate it against Software’s UI requirements.