Essential app tests

One of the things I’ve learned from my involvement in the development of the Foodpanda apps is that you simply cannot test enough. Although we have apps in 26 countries, the same learnings apply to everybody out there just starting out. QA is key.

Good QA is even more important on iOS than on Android. The weeklong waiting time until your bug fixes and improvements are finally approved, is just too painful to bear knowing about bugs.

Being used to pushing fresh code live and having it available to every visitor right away, makes the long feedback cycle imposed by Apple very hard to endure.

With public ratings in place, bad ratings stay alive, even if you solve the problems over time. Therefore, starting with an untested app is an unnecessary risk that you shouldn’t take. Our goal is always to launch reliable and seamless apps that users love to use.

Simulator & Emulator Tests

Tests inside the Simulator and Emulator are clearly a necessary and widely practiced QA step. When testing your app before releasing it, make sure that you cover the following tests:

• Connection Throttling
  • Test your app under the worst possible connection using tools like
    • SpeedLimit for iOS
    • Network Delay and Network Speed settings in the Emulator for Android
• Low Battery
  • Test your app under low battery conditions
    • Emulate low battery on Android using the power options
    • While showing the low battery UIAlertView can be done in the simulator, you should test under low battery conditions in a real device
• Low Memory
  • The phone operating systems monitor overall memory usage for you and notify your app when memory runs low. You have to factor in for those situations by reacting to these warning by, e.g. clearing the cache.
    • Use the “Simulate memory warnings” option in Xcode and understand what it does
    • Set the total memory size via the “ulimit” command and then test using the “Immediately destroy activities” settings
• Double-high Status Bar
  • In some situations (e.g. audio recording, while using the personal hotspot) on iOS the system displays another status bar. Your app needs to be prepared for these situations and behave well. The status bar should not push the UI down or cover it. You can activate it in the Simulator by choosing Hardware > Toggle In-Call Status Bar.

Real device testing

Simulating alone is not enough, though. Many important things are different on real devices and can’t be properly accessed using only the simulator. The processor is different, the graphics power is worse and other factors, like memory capacity and real network connectivity come into play. So testing on the target devices is an unavoidable step towards releasing your app.

• Incoming calls
  • Test how your app behaves when, while in the foreground, the device receives a call
• Incoming text
  • Test how your app behaves when, while in the foreground, the device receives a text message
• Bringing your app to foreground and background
• Test how app behaves with connecting to public WiFi without access to internet
  • On iOS you might need to adjust the connectivity framework and do something like pinging the backend first to determine whether wifi and internet connection are present and do not rely only on the framework
• Geolocation
  • Actually use geolocation of your app in different places (inside buildings, while driving, in the metro) and see how it behaves and how accurate it is
• Low battery
  • Test how your app behaves when the battery runs low on an actual device. You should test with 5% of battery remaining.

Crash Reporting

The last point has nothing to do with testing. But of course, I wouldn’t assume to get it all right the first time(s). In software, there is always more room for improvement. This is why it is instrumental to include a good crash reporting tool into your app.

Although the app stores provide crash reporting, we found it to be too unreliable. In fact we track 10x more crashes with the built in tools. Also a tool like the HockeyApps Crash Reporter provides direct indication of where the crash occurred and makes it really easy for the developer to find the source of evil.

Xcode Instruments

This last paragraph is about Xcode Instruments. Before really shipping the app we had to make sure that it is nice to use. This means that it is fast, table views scroll like butter, etc. In order to analyze parts of the code that could be improved, Apple provides a great toolbox called instruments that is integrated with Xcode. There’s a great intro tutorial of Instruments over at Ray Wenderlich’s blog.

Do you have some other essential tests that you are doing? I’d love to hear to comments and suggestions