Introduction
Testing is a common practice to ensure that code logic is not easily broken during development and refactoring. Having tests running as part of Continuous Integration (CI) infrastructure is essential, especially with a large codebase contributed by many engineers. However, the more tests we add, the longer it takes to execute. In the context of iOS development, the execution time of the whole test suite might be significantly affected by the increasing number of tests written. Running CI pre-merge pipelines against a change, would cost us more time. Therefore, reducing test execution time is a long term epic we have to tackle in order to build a good CI infrastructure.
Apart from splitting tests into subsets and running each of them in a CI job, we can also make use of the Xcode parallel testing feature to achieve parallelism within one single CI job. However, due to platform-specific implementations, there are some constraints that prevent parallel testing from working efficiently. One constraint we found is that tests of the same Swift class run on the same simulator. In this post, we will discuss this constraint in detail and introduce a tip to overcome it.
Background
Xcode Parallel Testing
The parallel testing feature was shipped as part of the Xcode 10 release. This support enables us to easily configure test setup:
- There is no need to care about how to split a given test suite.
- The number of workers (i.e. parallel runners/instances) is configurable. We can pass this value in the
xcodebuild
CLI via the-parallel-testing-worker-count
option. - Xcode takes care of cloning and starts simulators accordingly.
However, the distribution logic under the hood is a black-box. We do not really know how tests are assigned to each worker or simulator, and in which order.
It is worth mentioning that even without the Xcode parallel testing support, we can still achieve similar improvements by running subsets of tests in different child processes. But it takes more effort to dispatch tests to each child process in an efficient way, and to handle the output from each test process appropriately.
Test Time Imbalance
Generally, a parallel execution system is at its best efficiency if each parallel task executes in roughly the same duration and ends at roughly the same time.
If the time spent on each parallel task is significantly different, it will take more time than expected to execute all tasks. For example, in the following image, it takes the system on the left 13 mins to finish 3 tasks. Whereas, the one on the right takes only 10.5 mins to finish those 3 tasks.
Assume there are N workers. The ith worker executes its tasks in ti seconds/minutes. In the left plot, t1 = 10 mins, t2 = 7 mins, t3 = 13 mins.
We define the test time imbalance metric as the difference between the min and max end time:
max(ti) – min(ti)
For the example above, the test time imbalance is 13 mins – 7 mins = 6 mins.
Contributing Factors in Test Time Imbalance
There are several factors causing test time imbalance. The top two prominent factors are:
- Tests vary in execution time.
- Tests of the same class run on the same simulator.
An example of the first factor is that in our project, around 50% of tests execute in a range of 20-40 secs. Some tests take under 15 secs to run while several take up to 2 minutes. Sometimes tests taking longer execution time is inevitable since those tests usually touch many flows, which cannot be split. If such tests run last, the test time imbalance may increase.
However, this issue, in general, does not matter that much because long-time-execution tests do not always run last.
Regarding the second factor, there is no official Apple documentation that explicitly states this constraint. When Apple first introduced parallel testing support in Xcode 10, they only mentioned that test classes are distributed across runner processes:
“Test parallelisation occurs by distributing the test classes in a target across multiple runner processes. Use the test log to see how your test classes were parallelised. You will see an entry in the log for each runner process that was launched, and below each runner you will see the list of classes that it executed.”
For example, we have a test class JobFlowTests
that includes five tests and another test class TutorialTests
that has only one single test.
final class JobFlowTests: BaseXCTestCase {
func testHappyFlow() { ... }
func testRecoverFlow() { ... }
func testJobIgnoreByDax() { ... }
func testJobIgnoreByTimer() { ... }
func testForceClearBooking() { ... }
}
...
final class TutorialTests: BaseXCTestCase {
func testOnboardingFlow() { ... }
}
When executing the two tests with two simulators running in parallel, the actual run is like the one shown on the left side of the following image, but ideally it should work like the one on the right side.
Diving Deep into Xcode Parallel Testing
Demystifying Xcode Scheduling Log
As mentioned above, Xcode distributes tests to simulators/workers in a black-box manner. However, by looking at the scheduling log generated when running tests, we can understand how Xcode parallel testing works.
When running UI tests via the xcodebuild
command:
$ xcodebuild -workspace Driver/Driver.xcworkspace \
-scheme Driver \
-configuration Debug \
-sdk 'iphonesimulator' \
-destination 'platform=iOS Simulator,id=EEE06943-7D7B-4E76-A3E0-B9A5C1470DBE' \
-derivedDataPath './DerivedData' \
-parallel-testing-enabled YES \
-parallel-testing-worker-count 2 \
-only-testing:DriverUITests/JobFlowTests \ # 👈👈👈👈👈
-only-testing:DriverUITests/TutorialTests \
test-without-building
The log can be found inside the *.xcresult
folder under DerivedData/Logs/Test
. For example: DerivedData/Logs/Test/Test-Driver-2019.11.04\_23-31-34-+0800.xcresult/1\_Test/Diagnostics/DriverUITests-144D9549-FD53-437B-BE97-8A288855E259/scheduling.log
2019-11-05 03:55:00 +0000: