Skip to content

Commit 6884a11

Browse files
maartenemaartene
authored
ST-0016 is ready for review
[ST-0016] Test cancellation
2 parents aa4a7ca + d6e913c commit 6884a11

File tree

1 file changed

+333
-0
lines changed

1 file changed

+333
-0
lines changed

‎proposals/testing/0016-test-cancellation.md‎

Lines changed: 333 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,333 @@
1+
# Test cancellation
2+
3+
* Proposal: [ST-0016](0016-test-cancellation.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Review Manager: [Maarten Engels](https://github.com/maartene)
6+
* Status: **Active review (October 23...November 5, 2025)**
7+
* Bug: [swiftlang/swift-testing#120](https://github.com/swiftlang/swift-testing/issues/120)
8+
* Implementation: [swiftlang/swift-testing#1284](https://github.com/swiftlang/swift-testing/pull/1284)
9+
* Review: ([pitch](https://forums.swift.org/t/pitch-test-cancellation/81847))
10+
11+
## Introduction
12+
13+
Swift Testing provides the ability to conditionally skip a test before it runs
14+
using the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:)),
15+
[`.disabled(if:)`](https://developer.apple.com/documentation/testing/trait/disabled(if:_:sourcelocation:)),
16+
etc. family of traits:
17+
18+
```swift
19+
@Test(.disabled(if: Species.all(in: .dinosauria).isEmpty)
20+
func `Are all dinosaurs extinct?`() {
21+
// ...
22+
}
23+
```
24+
25+
This proposal extends that feature to allow cancelling a test after it has
26+
started but before it has ended.
27+
28+
## Motivation
29+
30+
We have received feedback from a number of developers indicating that their
31+
tests have constraints that can only be checked after a test has started, and
32+
they would like the ability to end a test early and see that state change
33+
reflected in their development tools.
34+
35+
To date, we have not provided an API for ending a test's execution early because
36+
we want to encourage developers to use the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:))
37+
_et al._ trait. This trait can be evaluated early and lets Swift Testing plan a
38+
test run more efficiently. However, we recognize that these traits aren't
39+
sufficient. Some test constraints are dependent on data that isn't available
40+
until the test starts, while others only apply to specific test cases in a
41+
parameterized test function.
42+
43+
## Proposed solution
44+
45+
A static `cancel()` function is added to the [`Test`](https://developer.apple.com/documentation/testing/test)
46+
type. When a test author calls this function from within the body of a test (or
47+
from within the implementation of a trait, e.g. from [`prepare(for:)`](https://developer.apple.com/documentation/testing/trait/prepare(for:))),
48+
Swift Testing cancels the currently-running test.
49+
50+
Parameterized tests are special-cased: if the currently-running test is
51+
parameterized and you call `cancel()`, only the current test case is cancelled
52+
and other test cases in the same test continue to run.
53+
54+
### Relationship between tasks and tests
55+
56+
Each test runs in its own task during a test run, and each test case in a test
57+
also runs in its own task. Cancelling the current task from within the body of a
58+
test will, therefore, cancel the current test case, but not the current test:
59+
60+
```swift
61+
@Test(arguments: Species.all(in: .dinosauria))
62+
func `Are all dinosaurs extinct?`(_ species: Species) {
63+
if species.in(.aves) {
64+
// Birds aren't extinct (I hope)
65+
withUnsafeCurrentTask { $0?.cancel() }
66+
return
67+
}
68+
// ...
69+
}
70+
```
71+
72+
Using [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl)
73+
here is not ideal. It's not clear that the intent is to cancel the test case,
74+
and [`UnsafeCurrentTask`](https://developer.apple.com/documentation/swift/unsafecurrenttask)
75+
is, unsurprisingly, an unsafe interface.
76+
77+
> [!NOTE]
78+
> The version of Swift Testing included with Swift 6.2 does not correctly handle
79+
> task cancellation under all conditions. See [swiftlang/swift-testing#1289](https://github.com/swiftlang/swift-testing/issues/1289).
80+
81+
## Detailed design
82+
83+
A new static function is added to [`Test`](https://developer.apple.com/documentation/testing/test):
84+
85+
```swift
86+
extension Test {
87+
/// Cancel the current test or test case.
88+
///
89+
/// - Parameters:
90+
/// - comment: A comment describing why you are cancelling the test or test
91+
/// case.
92+
/// - sourceLocation: The source location to which the testing library will
93+
/// attribute the cancellation.
94+
///
95+
/// - Throws: An error indicating that the current test or test case has been
96+
/// cancelled.
97+
///
98+
/// The testing library runs each test and each test case in its own task.
99+
/// When you call this function, the testing library cancels the task
100+
/// associated with the current test:
101+
///
102+
/// ```swift
103+
/// @Test func `Food truck is well-stocked`() throws {
104+
/// guard businessHours.contains(.now) else {
105+
/// try Test.cancel("We're off the clock.")
106+
/// }
107+
/// // ...
108+
/// }
109+
/// ```
110+
///
111+
/// If the current test is a parameterized test function, this function
112+
/// instead cancels the current test case. Other test cases in the test
113+
/// function are not affected.
114+
///
115+
/// If the current test is a suite, the testing library cancels all of its
116+
/// pending and running tests.
117+
///
118+
/// If you have already cancelled the current test or if it has already
119+
/// finished running, this function throws an error to indicate that the
120+
/// current test has been cancelled, but does not attempt to cancel the test a
121+
/// second time.
122+
///
123+
/// - Important: If the current task is not associated with a test (for
124+
/// example, because it was created with [`Task.detached(name:priority:operation:)`](https://developer.apple.com/documentation/swift/task/detached(name:priority:operation:)-795w1))
125+
/// this function records an issue and cancels the current task.
126+
public static func cancel(_ comment: Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation) throws -> Never
127+
}
128+
```
129+
130+
Cancelling a test or test case implicitly cancels its associated task (and any
131+
child tasks thereof) as if [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel())
132+
were called on that task.
133+
134+
### Throwing semantics
135+
136+
Unlike [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()),
137+
this function always throws an error instead of returning. This simplifies
138+
control flow when a test is cancelled; instead of having to write:
139+
140+
```swift
141+
if condition {
142+
theTask.cancel()
143+
return
144+
}
145+
```
146+
147+
A test author need only write:
148+
149+
```swift
150+
if condition {
151+
try Test.cancel()
152+
}
153+
```
154+
155+
The errors this function throws are of a type internal to Swift Testing that is
156+
semantically similar to [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
157+
but carries additional information (namely the `comment` and `sourceLocation`
158+
arguments to `cancel(_:sourceLocation:)`) that Swift Testing can present to the
159+
user. When Swift Testing catches an error of this type[^cancellationErrorToo],
160+
it does not record an issue for the current test or test case.
161+
162+
[^cancellationErrorToo]: Swift Testing also catches errors of type
163+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
164+
if the current task has been cancelled. If the current task has not been
165+
cancelled, errors of this type are still recorded as issues.
166+
167+
Suppressing these errors with `do`/`catch` or `try?` does not uncancel a test,
168+
test case, or task, but can be useful if you have additional local work you need
169+
to do before the test or test case ends.
170+
171+
### Support for CancellationError
172+
173+
Cancelling a test's or test case's associated task is equivalent to cancelling
174+
the test or test case. Hence, if a test or test case throws an instance of
175+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
176+
_and_ the current task has been cancelled, it is treated as if the test or test
177+
case were cancelled.
178+
179+
### Interaction with recorded issues
180+
181+
If you cancel a test or test case that has previously recorded an issue, that
182+
issue is not overridden or nullified. In particular, if the test or test case
183+
has already recorded an issue of severity **error** when you call
184+
`cancel(_:sourceLocation:)`, the test or test case will still fail.
185+
186+
### Example usage
187+
188+
To cancel the current test case and let other test cases run:
189+
190+
```swift
191+
@Test(arguments: Species.all(in: .dinosauria))
192+
func `Are all dinosaurs extinct?`(_ species: Species) throws {
193+
if species.in(.aves) {
194+
try Test.cancel("\(species) is birds!")
195+
}
196+
// ...
197+
}
198+
```
199+
200+
## Source compatibility
201+
202+
This change is additive only.
203+
204+
## Integration with supporting tools
205+
206+
The JSON event stream Swift Testing provides is updated to include two new event
207+
kinds:
208+
209+
```diff
210+
<event-kind> ::= "runStarted" | "testStarted" | "testCaseStarted" |
211+
"issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" |
212+
- "runEnded" | "valueAttached"
213+
+ "runEnded" | "valueAttached" | "testCancelled" | "testCaseCancelled"
214+
```
215+
216+
And new fields are added to event records to represent the comment and source
217+
location passed to `cancel(_:sourceLocation:)`:
218+
219+
```diff
220+
<event> ::= {
221+
"kind": <event-kind>,
222+
"instant": <instant>, ; when the event occurred
223+
["issue": <issue>,] ; the recorded issue (if "kind" is "issueRecorded")
224+
["attachment": <attachment>,] ; the attachment (if kind is "valueAttached")
225+
"messages": <array:message>,
226+
["testID": <test-id>,]
227+
+ ["comments": <array:string>,]
228+
+ ["sourceLocation": <source-location>,]
229+
}
230+
```
231+
232+
These new fields are populated for the new event kinds as well as other event
233+
kinds that can populate them.
234+
235+
An event of kind `"testCancelled"` is posted any time an entire test function or
236+
test suite is cancelled. An event of kind `"testCaseCancelled"` is posted any
237+
time a single test case is cancelled.
238+
239+
These new event kinds and fields will be included in the next revision of the
240+
JSON schema (currently expected to be schema version `"6.3"`).
241+
242+
## Future directions
243+
244+
- Adding a corresponding `Test.checkCancellation()` function and/or
245+
`Test.isCancelled` static property. These are beyond the scope of this
246+
proposal, primarily because [`Task.isCancelled`](https://developer.apple.com/documentation/swift/task/iscancelled-swift.type.property)
247+
and [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation())
248+
already work in a test.
249+
250+
- Adding a `Test.Case.cancelAll()` interface that explicitly cancels all test
251+
cases in a test function. We want to further evaluate the use cases and
252+
semantics for such a function before we commit to introducing it as API.
253+
254+
## Alternatives considered
255+
256+
- Doing nothing. While we do want test authors to use [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:))
257+
_et al._ trait, we recognize it does not provide the full set of functionality
258+
that test authors need.
259+
260+
- Ignoring task cancellation or treating [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
261+
as a normal error even when the current task has been cancelled. It is not
262+
possible for Swift Testing to outright ignore task cancellation, and a
263+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
264+
instance thrown from [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation())
265+
is not really a test issue but rather a manifestation of control flow.
266+
267+
- Using the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
268+
type from XCTest. Interoperation with XCTest is an area of exploration for us,
269+
but core functionality of Swift Testing needs to be usable without also
270+
importing XCTest.
271+
272+
- Spelling the function `static func cancel(_:sourceLocation:) -> some Error`
273+
and requiring it be called as `throw Test.cancel()`. This is closer to how
274+
the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
275+
type is used in XCTest. We have received indirect feedback about [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
276+
indicating its usage is unclear, and sometimes need to help developers who
277+
have written:
278+
279+
```swift
280+
if x {
281+
XCTSkip()
282+
}
283+
```
284+
285+
And don't understand why it has failed to stop the test. More broadly, it is
286+
not common practice in Swift for a function to return an error that the caller
287+
is then responsible for throwing.
288+
289+
- Providing additional `cancel(if:)` and `cancel(unless:)` functions. In
290+
Objective-C, XCTest provides the [`XCTSkipIf()`](https://developer.apple.com/documentation/xctest/xctskipif)
291+
and [`XCTSkipUnless()`](https://developer.apple.com/documentation/xctest/xctskipunless)
292+
macros which capture their condition arguments as strings for display to the
293+
test author. This functionality is not available in Swift, but XCTest's Swift
294+
interface provides equivalent throwing functions as conveniences. We could
295+
provide these functions (without any sort of string-capturing ability) too,
296+
but they provide little additional clarity above an `if` or `guard` statement.
297+
298+
- Implementing cancellation using Swift macros so we can capture an `if` or
299+
`unless` argument as a string. A macro for this feature is probably the wrong
300+
tradeoff between compile-time magic and technical debt.
301+
302+
- Relying solely on [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()).
303+
Ignoring the interplay between tests and test cases, this approach is
304+
difficult for test authors to use because the current [`Task`](https://developer.apple.com/documentation/swift/task)
305+
instance isn't visible _within_ that task. Instead, a test author would need
306+
to use [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl)
307+
to get a temporary reference to the task and cancel _that_ value. We would
308+
also not have the ability to include a comment and source location information
309+
in the test's console output or an IDE's test result interface.
310+
311+
With that said, [`UnsafeCurrentTask.cancel()`](https://developer.apple.com/documentation/swift/unsafecurrenttask/cancel())
312+
_does_ cancel the test or test case associated with the current task.
313+
314+
- Providing both `Test.cancel()` and `Test.Case.cancel()`, with `Test.cancel()`
315+
_always_ cancelling the current test in its entirety and `Test.Case.cancel()`
316+
_always_ cancelling the current test _case_ and leaving other test cases
317+
alone.
318+
319+
We have received pitch feedback from multiple test authors indicating that
320+
they could introduce subtle bugs while refactoring test functions into
321+
parameterized test functions. If they had written `Test.cancel()` and forgot
322+
to change the call to `Test.Case.cancel()` when refactoring, they could
323+
introduce a bug causing none of their test cases to run (because the entire
324+
test is cancelled instead of just the current test case).
325+
326+
## Acknowledgments
327+
328+
Thanks team!
329+
330+
Thanks Arthur! That's right, dinosaurs _do_ say "roar!"
331+
332+
And thanks to [@allevato](https://github.com/allevato) for nerd-sniping me into
333+
writing this proposal.

0 commit comments

Comments
 (0)