Skip to content

Commit 6023a5f

Browse files
authored
Introduce issue handling trait (as SPI) (#1080)
This introduces a new trait type named `IssueHandlingTrait` as SPI. It allows observing, transforming, or filtering the issue(s) recorded during the test it's applied to. Here's a contrived example: ```swift @test(.transformIssues { issue in var issue = issue issue.comments.append("A comparison of two literals") return issue // Or, return `nil` to suppress the issue }) func example() { #expect(1 == 2) } ``` ### Motivation: Sometimes it can be useful to customize an issue recorded during a test. For example, you might wish to add supplemental information to it, such by adding comments. Another example of this could be adding an attachment to an issue, which was a capability mentioned as a [future direction](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0009-attachments.md#future-directions) in [ST-0009 Attachments](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0009-attachments.md). Other times, you might wish to suppress an issue which is later determined to be irrelevant or cannot be marked as a [known issue](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/known-issues). Or, you may simply want to be notified that an issue was recorded, to react to it in some other way while still recording the issue normally. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated. Resolves rdar://140144041
1 parent 2f45665 commit 6023a5f

File tree

5 files changed

+426
-5
lines changed

5 files changed

+426
-5
lines changed

Sources/Testing/CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ add_library(Testing
9797
Traits/ConditionTrait.swift
9898
Traits/ConditionTrait+Macro.swift
9999
Traits/HiddenTrait.swift
100+
Traits/IssueHandlingTrait.swift
100101
Traits/ParallelizationTrait.swift
101102
Traits/Tags/Tag.Color.swift
102103
Traits/Tags/Tag.Color+Loading.swift

Sources/Testing/Running/Runner.swift

Lines changed: 53 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -91,10 +91,11 @@ extension Runner {
9191
return try await body()
9292
}
9393

94-
// Construct a recursive function that invokes each trait's ``execute(_:for:testCase:)``
95-
// function. The order of the sequence is reversed so that the last trait is
96-
// the one that invokes body, then the second-to-last invokes the last, etc.
97-
// and ultimately the first trait is the first one to be invoked.
94+
// Construct a recursive function that invokes each scope provider's
95+
// `provideScope(for:testCase:performing:)` function. The order of the
96+
// sequence is reversed so that the last trait is the one that invokes body,
97+
// then the second-to-last invokes the last, etc. and ultimately the first
98+
// trait is the first one to be invoked.
9899
let executeAllTraits = test.traits.lazy
99100
.reversed()
100101
.compactMap { $0.scopeProvider(for: test, testCase: testCase) }
@@ -108,6 +109,41 @@ extension Runner {
108109
try await executeAllTraits()
109110
}
110111

112+
/// Apply the custom scope from any issue handling traits for the specified
113+
/// test.
114+
///
115+
/// - Parameters:
116+
/// - test: The test being run, for which to apply its issue handling traits.
117+
/// - body: A function to execute within the scope provided by the test's
118+
/// issue handling traits.
119+
///
120+
/// - Throws: Whatever is thrown by `body` or by any of the traits' provide
121+
/// scope function calls.
122+
private static func _applyIssueHandlingTraits(for test: Test, _ body: @escaping @Sendable () async throws -> Void) async throws {
123+
// If the test does not have any traits, exit early to avoid unnecessary
124+
// heap allocations below.
125+
if test.traits.isEmpty {
126+
return try await body()
127+
}
128+
129+
// Construct a recursive function that invokes each issue handling trait's
130+
// `provideScope(performing:)` function. The order of the sequence is
131+
// reversed so that the last trait is the one that invokes body, then the
132+
// second-to-last invokes the last, etc. and ultimately the first trait is
133+
// the first one to be invoked.
134+
let executeAllTraits = test.traits.lazy
135+
.compactMap { $0 as? IssueHandlingTrait }
136+
.reversed()
137+
.map { $0.provideScope(performing:) }
138+
.reduce(body) { executeAllTraits, provideScope in
139+
{
140+
try await provideScope(executeAllTraits)
141+
}
142+
}
143+
144+
try await executeAllTraits()
145+
}
146+
111147
/// Enumerate the elements of a sequence, parallelizing enumeration in a task
112148
/// group if a given plan step has parallelization enabled.
113149
///
@@ -177,7 +213,19 @@ extension Runner {
177213
Event.post(.testSkipped(skipInfo), for: (step.test, nil), configuration: configuration)
178214
shouldSendTestEnded = false
179215
case let .recordIssue(issue):
180-
Event.post(.issueRecorded(issue), for: (step.test, nil), configuration: configuration)
216+
// Scope posting the issue recorded event such that issue handling
217+
// traits have the opportunity to handle it. This ensures that if a test
218+
// has an issue handling trait _and_ some other trait which caused an
219+
// issue to be recorded, the issue handling trait can process the issue
220+
// even though it wasn't recorded by the test function.
221+
try await Test.withCurrent(step.test) {
222+
try await _applyIssueHandlingTraits(for: step.test) {
223+
// Don't specify `configuration` when posting this issue so that
224+
// traits can provide scope and potentially customize the
225+
// configuration.
226+
Event.post(.issueRecorded(issue), for: (step.test, nil))
227+
}
228+
}
181229
shouldSendTestEnded = false
182230
}
183231
} else {

Sources/Testing/Testing.docc/Traits.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,13 @@ types that customize the behavior of your tests.
4848
- ``Trait/bug(_:id:_:)-10yf5``
4949
- ``Trait/bug(_:id:_:)-3vtpl``
5050

51+
<!--
52+
### Handling issues
53+
54+
- ``Trait/transformIssues(_:)``
55+
- ``Trait/filterIssues(_:)``
56+
-->
57+
5158
### Creating custom traits
5259

5360
- ``Trait``
@@ -64,3 +71,4 @@ types that customize the behavior of your tests.
6471
- ``Tag``
6572
- ``Tag/List``
6673
- ``TimeLimitTrait``
74+
<!--- ``IssueHandlingTrait``-->
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2025 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
/// A type that allows transforming or filtering the issues recorded by a test.
12+
///
13+
/// Use this type to observe or customize the issue(s) recorded by the test this
14+
/// trait is applied to. You can transform a recorded issue by copying it,
15+
/// modifying one or more of its properties, and returning the copy. You can
16+
/// observe recorded issues by returning them unmodified. Or you can suppress an
17+
/// issue by either filtering it using ``Trait/filterIssues(_:)`` or returning
18+
/// `nil` from the closure passed to ``Trait/transformIssues(_:)``.
19+
///
20+
/// When an instance of this trait is applied to a suite, it is recursively
21+
/// inherited by all child suites and tests.
22+
///
23+
/// To add this trait to a test, use one of the following functions:
24+
///
25+
/// - ``Trait/transformIssues(_:)``
26+
/// - ``Trait/filterIssues(_:)``
27+
@_spi(Experimental)
28+
public struct IssueHandlingTrait: TestTrait, SuiteTrait {
29+
/// A function which transforms an issue and returns an optional replacement.
30+
///
31+
/// - Parameters:
32+
/// - issue: The issue to transform.
33+
///
34+
/// - Returns: An issue to replace `issue`, or else `nil` if the issue should
35+
/// not be recorded.
36+
fileprivate typealias Transformer = @Sendable (_ issue: Issue) -> Issue?
37+
38+
/// This trait's transformer function.
39+
private var _transformer: Transformer
40+
41+
fileprivate init(transformer: @escaping Transformer) {
42+
_transformer = transformer
43+
}
44+
45+
public var isRecursive: Bool {
46+
true
47+
}
48+
}
49+
50+
extension IssueHandlingTrait: TestScoping {
51+
public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self? {
52+
// Provide scope for tests at both the suite and test case levels, but not
53+
// for the test function level. This avoids redundantly invoking the closure
54+
// twice, and potentially double-processing, issues recorded by test
55+
// functions.
56+
test.isSuite || testCase != nil ? self : nil
57+
}
58+
59+
public func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws {
60+
try await provideScope(performing: function)
61+
}
62+
63+
/// Provide scope for a specified function.
64+
///
65+
/// - Parameters:
66+
/// - function: The function to perform.
67+
///
68+
/// This is a simplified version of ``provideScope(for:testCase:performing:)``
69+
/// which doesn't accept test or test case parameters. It's included so that
70+
/// a runner can invoke this trait's closure even when there is no test case,
71+
/// such as if a trait on a test function threw an error during `prepare(for:)`
72+
/// and caused an issue to be recorded for the test function. In that scenario,
73+
/// this trait still needs to be invoked, but its `scopeProvider(for:testCase:)`
74+
/// intentionally returns `nil` (see the comment in that method), so this
75+
/// function can be called instead to ensure this trait can still handle that
76+
/// issue.
77+
func provideScope(performing function: @Sendable () async throws -> Void) async throws {
78+
guard var configuration = Configuration.current else {
79+
preconditionFailure("Configuration.current is nil when calling \(#function). Please file a bug report at https://github.com/swiftlang/swift-testing/issues/new")
80+
}
81+
82+
configuration.eventHandler = { [oldConfiguration = configuration] event, context in
83+
guard case let .issueRecorded(issue) = event.kind else {
84+
oldConfiguration.eventHandler(event, context)
85+
return
86+
}
87+
88+
// Use the original configuration's event handler when invoking the
89+
// transformer to avoid infinite recursion if the transformer itself
90+
// records new issues. This means only issue handling traits whose scope
91+
// is outside this one will be allowed to handle such issues.
92+
let newIssue = Configuration.withCurrent(oldConfiguration) {
93+
_transformer(issue)
94+
}
95+
96+
if let newIssue {
97+
var event = event
98+
event.kind = .issueRecorded(newIssue)
99+
oldConfiguration.eventHandler(event, context)
100+
}
101+
}
102+
103+
try await Configuration.withCurrent(configuration, perform: function)
104+
}
105+
}
106+
107+
@_spi(Experimental)
108+
extension Trait where Self == IssueHandlingTrait {
109+
/// Constructs an trait that transforms issues recorded by a test.
110+
///
111+
/// - Parameters:
112+
/// - transformer: The closure called for each issue recorded by the test
113+
/// this trait is applied to. It is passed a recorded issue, and returns
114+
/// an optional issue to replace the passed-in one.
115+
///
116+
/// The `transformer` closure is called synchronously each time an issue is
117+
/// recorded by the test this trait is applied to. The closure is passed the
118+
/// recorded issue, and if it returns a non-`nil` value, that will be recorded
119+
/// instead of the original. Otherwise, if the closure returns `nil`, the
120+
/// issue is suppressed and will not be included in the results.
121+
///
122+
/// The `transformer` closure may be called more than once if the test records
123+
/// multiple issues. If more than one instance of this trait is applied to a
124+
/// test (including via inheritance from a containing suite), the `transformer`
125+
/// closure for each instance will be called in right-to-left, innermost-to-
126+
/// outermost order, unless `nil` is returned, which will skip invoking the
127+
/// remaining traits' closures.
128+
///
129+
/// Within `transformer`, you may access the current test or test case (if any)
130+
/// using ``Test/current`` ``Test/Case/current``, respectively. You may also
131+
/// record new issues, although they will only be handled by issue handling
132+
/// traits which precede this trait or were inherited from a containing suite.
133+
public static func transformIssues(_ transformer: @escaping @Sendable (Issue) -> Issue?) -> Self {
134+
Self(transformer: transformer)
135+
}
136+
137+
/// Constructs a trait that filters issues recorded by a test.
138+
///
139+
/// - Parameters:
140+
/// - isIncluded: The predicate with which to filter issues recorded by the
141+
/// test this trait is applied to. It is passed a recorded issue, and
142+
/// should return `true` if the issue should be included, or `false` if it
143+
/// should be suppressed.
144+
///
145+
/// The `isIncluded` closure is called synchronously each time an issue is
146+
/// recorded by the test this trait is applied to. The closure is passed the
147+
/// recorded issue, and if it returns `true`, the issue will be preserved in
148+
/// the test results. Otherwise, if the closure returns `false`, the issue
149+
/// will not be included in the test results.
150+
///
151+
/// The `isIncluded` closure may be called more than once if the test records
152+
/// multiple issues. If more than one instance of this trait is applied to a
153+
/// test (including via inheritance from a containing suite), the `isIncluded`
154+
/// closure for each instance will be called in right-to-left, innermost-to-
155+
/// outermost order, unless `false` is returned, which will skip invoking the
156+
/// remaining traits' closures.
157+
///
158+
/// Within `isIncluded`, you may access the current test or test case (if any)
159+
/// using ``Test/current`` ``Test/Case/current``, respectively. You may also
160+
/// record new issues, although they will only be handled by issue handling
161+
/// traits which precede this trait or were inherited from a containing suite.
162+
public static func filterIssues(_ isIncluded: @escaping @Sendable (Issue) -> Bool) -> Self {
163+
Self { issue in
164+
isIncluded(issue) ? issue : nil
165+
}
166+
}
167+
}

0 commit comments

Comments
 (0)