---
# System prepended metadata

title: Mobile Testing Strategy

---

## Types
* Mobile Web
* Hybrid/Native

## SDLC


|                 | Build            | Test | Deploy | Operate/Feedback |
| --------        | --------         | -------- | -------- | -------- |
| Mobile Web      | Fast             | Fast     | Daily/Hourly     | Easy     |
| Native/Hybrid   | Medium - Slow    | 2-6 times slower     | Review: 1-7 days, Release: up to 24H     | Hard without proper tools (ie. TestFairy)     |

## Specific Mobile Issues
* **Can't Revert Installed Apps** - Unlike with Web Apps, a Mobile Native can't be reverted / removed from a user's phone. This can result in having more business impact when a Native App contains bugs.
* **Mobile is remote** - An issue with your app is always happening somewhere else, not at your desk or your screen. It happens in the hand of a person that you cannot easily communicate with to understand why and how the issue happened.
* **Fragmented Market** - Massive variety of Android and iOS devices across hardware and software configurations, different levels of OS versions, and locals. And this is before we get to the mobile device itself that needs reception, WiFi, battery, and more.

## Debugging Options

### Mobile Web

#### Using Mobile Simulation
##### Pros
* **Fast Execution** - matter of seconds to launch a browser, also headless is possible.
* **Scalable** - Easily setup 10+ browser instances.
* **Cross Platform** - Ability to use browsers on different operating systems.

##### Cons
* It's a desktop browser - rendering is different, so no visual testing.
* No Native integration - No native keyboards, incoming calls, etc.
* Just not a device - Incredibly fast, but still not a real device.

##### What to focus on
* **System Testing** - Isolated browser test performing full functional validations.
* **Cross browser** - use equivalent desktop browsers.
* **Responsive design** - Resizing browsers and toggling user agents.
* **Overall Visual Layout** - No extensive visual checks because the rendering is different than devices.

### Emulators/Simulators

#### Closer to the Mobile Experience
##### Pros
* **Easy to set up** - Just download, install and run.
* **Scalable** - Virtualisation means scalable and runs in parallel.
* **Native (API) Integration** - Test keyboard impact, biometrics, incoming calls, GPS, native elements.

##### Cons
* **Vanilla Android Only** - hardly no manufacturers skins, so stock.
* **No Real Usage** - CPU/Memory of the host.
* **No Real Interoperability** - No camera, Bluetooth, network conditions, etc.

##### What to focus on
* **Functional flow** - click paths through the app.
* **Visuals (Vanilla only)** - Emulators are limited to vanilla versions only.
* **Native (API) Integration** - Incoming calls, GPS, native dropdown, keyboard.
* **Touch interactions** - touch interactions like swipe and tap come closer to the real experience.

### Real Devices
#### The Real Thing

##### Pros
* **Native (API) in Real Conditions** - ability to test native APIs not only with injections for automation, like camera or biometrics.
* **Faster than Emulators** - some real devices are just faster than emulators due to the simulation of hardware.
* **Just the real thing** - Get device vitals or the impact of, manufacturer's sauce on top of the OS.

##### Cons
* **Cost** - real devices come with a price.
* **Complexity of Procurement** - when new phone comes out, it's not always available immediately to procure. In the meantime, the iOS simulator would already be avaiable.
* **Specific iOS build/Certificates** - iOS apps need to be signed with Development Distribution Certificate and Provisioning Profile for automation.

##### What to focus on
* **Usability** - validating usability such as actual, click areas, touch actions, native dropdown, etc.
* **Specific Browsers** - specific manufacturer browsers.
* **Visuals** - the real rendering, the leftovers that could not be checked on emalators/simulators.
* **Native API Integration** - incoming calls, GPS, text, etc.
* **Error, Hang and Crashes** - research why you have errors/ hanging screens and crashes.
* **Performance** - CPU/Memory usage, battery, network.

## Testing

### Alpha testing

* Done by an expert team of develpers/testers.
* Manual and or automated testing.
* Narrow testing focus
* Limited set of possible configurations.

### Beta testing
* Done by real world users.
* Manual testing.
* Try different flows to validate your ideas
* Environmental conditions.

> Get real-world exposure and feedback where early users can try our your app and let you know what needs fixing and what needs ditching.

![](https://hackmd.io/_uploads/Sy0VKcgjn.png)


## Reporting Bugs
Should be:
* **Easy** - prevent having complex menus, flows to find filling a bug.
* **3-5 seconds** - the longer it takes to report a bug the less likely it is that people will report a new one.
* **Zero Friction** - don't ask what is already known like brand, model, OS version, etc.
* **Non intrusive** - don't push people to find bugs, let them focus on using the app and what they see is missing / wrong.

#### Why people don't report bugs
* "I'm too busy, now is not a good time"
* "It takes too much time, our bug system is not friendly and requires lots of mandatory info".
* "Someone probably reported that bug"
* "Last time I reported a bug I had to attend a conference call the next day, I won't do it again."
* "It's late and I'm on my way to bed, I'll do it in the morning..."
* "I restarted my app and it didn't happen again, probably not a bug. My fault.."
* "I'm actuall not sure it is a bug"
* "I forgot."

### Bug reporting tool
* TestFairy - https://youtu.be/ZXMFqJPG0LI
