Automated iOS Performance Benchmarking

The system should be able to perform automated performance benchmarking tests with minimal manual steps.

The two applications currently needs testing are:

1. MockTaoBao: to test scrolling performance
2. comp_demo: to test performance of different UI animation effects

Data Points

The performance tests should collect the following data metrics:

• CPU Cycles
• CPU Frequency (Avg GHz)
• CPU Frequency (Max GHz)
• CPU Memory (Avg MB)
• CPU Memory (Max MB)
• GPU Memory (Avg MB)
• GPU Memory (Max MB)

Power Related:

• Foreground consumption
• Background consumption

Automation

For each of the application's graphics features, the same data metrics should be collected.

A minimum of 5~10 runs of the test suite should be performed for each application. Each run should collect all of the metrics listed above. Metric results from all runs can then be summed and averaged for a balanced final result.

One key requirement is to create a consistent test environment in order to minimize variances to the result.

Scripts to launch and collect

This requires a way to automate the test process and steps instead of manually performing them.

A script should be launched to automatically launch the test applications (in this case MockTaoBao and comp_demo).

The performance measuring process should also be setup and launched at the same time in order to collect data metrics.

Programmatic action vs. human action

The automated script should then invoke the "actions" which will stress the specific application features, while the instrumenting tool is collecting the data, and then complete the "actions", and stop the data collection process.

The actual actions involving human touch controls for scrolling, etc. should be done programmatically if possible, in order to minimize human variances. However, if this is not possible, then care should be taken so that the human activities should vary as little as possilbe during each run.

The collected data should then be processed and analyzed for the data metrics listed above.

Scrolling Test

For testing the Swift UI version of the TaoBao application, the system should perform "frequent scrolling" to stress the system and gather the data points.

This simulates fast scrolling by user of the applications' main screen. An automated script to quickly scroll the main product display at least 5-10 times downward (by swiping up) and then reversing the scroll upward (by swiping down).

Repeating the actions above, within a predetermined time window (such as 10 seconds) should yield enough data points for analysis.

Procedure

1. Clear device of all other non-essential running applications
2. Setup device with proper instrumentation and data collection
3. Launch the test application, MockTaoBao or comp_demo
4. Start instrumentation and data collection
5. Start the continous scrolling test and run for 10 seconds
6. Stop the test after 10 seconds
7. Stop the instrumentation and data collection
8. Export collected data for processing and analysis