From 93521dfbaf31a26729642daf958ea0b2df4e298a Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 2 Apr 2024 13:50:05 -0400 Subject: [PATCH 01/19] [SWT-NNNN] Exit tests One of the first enhancement requests we received for swift-testing was the ability to test for precondition failures and other critical failures that terminate the current process when they occur. This feature is also frequently requested for XCTest. With swift-testing, we have the opportunity to build such a feature in an ergonomic way. Read the full proposal [here](https://github.com/apple/swift-testing/blob/jgrynspan/exit-tests-proposal/Documentation/Proposals/NNNN-exit-tests.md). --- Documentation/Proposals/NNNN-exit-tests.md | 787 ++++++++++++++++++ .../ExitTests/ExitTest.Condition.swift | 41 +- .../Testing/ExitTests/ExitTest.Result.swift | 19 +- Sources/Testing/ExitTests/ExitTest.swift | 30 +- Sources/Testing/ExitTests/StatusAtExit.swift | 14 +- .../Expectations/Expectation+Macro.swift | 10 +- .../ExpectationChecking+Macro.swift | 1 - Sources/Testing/Running/Configuration.swift | 3 +- Sources/Testing/Testing.docc/Expectations.md | 7 + Tests/TestingTests/ExitTestTests.swift | 2 +- 10 files changed, 890 insertions(+), 24 deletions(-) create mode 100644 Documentation/Proposals/NNNN-exit-tests.md diff --git a/Documentation/Proposals/NNNN-exit-tests.md b/Documentation/Proposals/NNNN-exit-tests.md new file mode 100644 index 000000000..06f740d63 --- /dev/null +++ b/Documentation/Proposals/NNNN-exit-tests.md @@ -0,0 +1,787 @@ +# Exit tests + +* Proposal: [SWT-NNNN](NNNN-exit-tests.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Awaiting review** +* Bug: [apple/swift-testing#157](https://github.com/apple/swift-testing/issues/157) +* Implementation: [apple/swift-testing#307](https://github.com/apple/swift-testing/pull/307) +* Review: ([pitch](https://forums.swift.org/t/pitch-exit-tests/78071)) + +## Introduction + +One of the first enhancement requests we received for Swift Testing was the +ability to test for precondition failures and other critical failures that +terminate the current process when they occur. This feature is also frequently +requested for XCTest. With Swift Testing, we have the opportunity to build such +a feature in an ergonomic way. + +> [!NOTE] +> This feature has various names in the relevant literature, e.g. "exit tests", +> "death tests", "death assertions", "termination tests", etc. We consistently +> use the term "exit tests" to refer to them. + +## Motivation + +Imagine a function, implemented in a package, that includes a precondition: + +```swift +func eat(_ taco: consuming Taco) { + precondition(taco.isDelicious, "Tasty tacos only!") + ... +} +``` + +Today, a test author can write unit tests for this function, but there is no way +to make sure that the function rejects a taco whose `isDelicious` property is +`false` because a test that passes such a taco as input will crash (correctly!) +when it calls `precondition()`. + +An exit test allows testing this sort of functionality. The mechanism by which +an exit test is implemented varies between testing libraries and languages, but +a common implementation involves spawning a new process, performing the work +there, and checking that the spawned process ultimately terminates with a +particular (possibly platform-specific) exit status. + +Adding exit tests to Swift Testing would allow an entirely new class of tests +and would improve code coverage for existing test targets that adopt them. + +## Proposed solution + +This proposal introduces new overloads of the `#expect()` and `#require()` +macros that take, as an argument, a closure to be executed in a child process. +When called, these macros spawn a new process using the relevant +platform-specific interface (`posix_spawn()`, `CreateProcessW()`, etc.), call +the closure from within that process, and suspend the caller until that process +terminates. The exit status of the process is then compared against a known +value passed to the macro, allowing the test to pass or fail as appropriate. + +The function from earlier can then be tested using either of the new +overloads: + +```swift +await #expect(exitsWith: .failure) { + var taco = Taco() + taco.isDelicious = false + eat(taco) // should trigger a precondition failure and process termination +} +``` + +## Detailed design + +### New expectations + +We will introduce the following new overloads of `#expect()` and `#require()` to +the testing library: + +```swift +/// Check that an expression causes the process to terminate in a given fashion. +/// +/// - Parameters: +/// - expectedExitCondition: The expected exit condition. +/// - observedValues: An array of key paths representing results from within +/// the exit test that should be observed and returned by this macro. The +/// ``ExitTest/Result/statusAtExit`` property is always returned. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which recorded expectations and +/// issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// - Returns: If the exit test passed, an instance of ``ExitTest/Result`` +/// describing the state of the exit test when it exited. If the exit test +/// fails, the result is `nil`. +/// +/// Use this overload of `#expect()` when an expression will cause the current +/// process to terminate and the nature of that termination will determine if +/// the test passes or fails. For example, to test that calling `fatalError()` +/// causes a process to terminate: +/// +/// await #expect(exitsWith: .failure) { +/// fatalError() +/// } +/// +/// - Note: A call to this expectation macro is called an "exit test." +/// +/// ## How exit tests are run +/// +/// When an exit test is performed at runtime, the testing library starts a new +/// process with the same executable as the current process. The current task is +/// then suspended (as with `await`) and waits for the child process to +/// terminate. `expression` is not called in the parent process. +/// +/// Meanwhile, in the child process, `expression` is called directly. To ensure +/// a clean environment for execution, it is not called within the context of +/// the original test. If `expression` does not terminate the child process, the +/// process is terminated automatically as if the main function of the child +/// process were allowed to return naturally. If an error is thrown from +/// `expression`, it is handed as if the error were thrown from `main()` and the +/// process is terminated. +/// +/// Once the child process terminates, the parent process resumes and compares +/// its exit status against `expectedExitCondition`. If they match, the exit +/// test has passed; otherwise, it has failed and an issue is recorded. +/// +/// ## Child process output +/// +/// By default, the child process is configured without a standard output or +/// standard error stream. If your test needs to review the content of either of +/// these streams, you can pass its key path in the `observedValues` argument: +/// +/// let result = await #expect( +/// exitsWith: .failure, +/// observing: [\.standardOutputContent] +/// ) { +/// print("Goodbye, world!") +/// fatalError() +/// } +/// if let result { +/// #expect(result.standardOutputContent.contains(UInt8(ascii: "G"))) +/// } +/// +/// - Note: The content of the standard output and standard error streams may +/// contain any arbitrary sequence of bytes, including sequences that are not +/// valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). +/// These streams are globally accessible within the child process, and any +/// code running in an exit test may write to it including the operating +/// system and any third-party dependencies you have declared in your package. +/// +/// The actual exit condition of the child process is always reported by the +/// testing library even if you do not specify it in `observedValues`. +/// +/// ## Runtime constraints +/// +/// Exit tests cannot capture any state originating in the parent process or +/// from the enclosing lexical context. For example, the following exit test +/// will fail to compile because it captures an argument to the enclosing +/// parameterized test: +/// +/// @Test(arguments: 100 ..< 200) +/// func sellIceCreamCones(count: Int) async { +/// await #expect(exitsWith: .failure) { +/// precondition( +/// count < 10, // ERROR: A C function pointer cannot be formed from a +/// // closure that captures context +/// "Too many ice cream cones" +/// ) +/// } +/// } +/// +/// An exit test cannot run within another exit test. +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +@discardableResult +@freestanding(expression) public macro expect( + exitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: [any PartialKeyPath & Sendable] = [], + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @convention(thin) () async throws -> Void +) -> ExitTest.Result? = #externalMacro(module: "TestingMacros", type: "ExitTestExpectMacro") + +/// Check that an expression causes the process to terminate in a given fashion +/// and throw an error if it did not. +/// +/// [...] +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +@discardableResult +@freestanding(expression) public macro require( + exitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: [any PartialKeyPath & Sendable] = [], + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @convention(thin) () async throws -> Void +) -> ExitTest.Result = #externalMacro(module: "TestingMacros", type: "ExitTestRequireMacro") +``` + +> [!NOTE] +> These interfaces are currently implemented and available on **macOS**, +> **Linux**, **FreeBSD**, **OpenBSD**, and **Windows**. If a platform does not +> support exit tests (generally because it does not support spawning or awaiting +> child processes), then we define `SWT_NO_EXIT_TESTS` when we build it. +> +> `SWT_NO_EXIT_TESTS` is not defined during test target builds. + +### Representing an exit test in Swift + +A new type, `ExitTest`, represents an exit test: + +```swift +/// A type describing an exit test. +/// +/// Instances of this type describe exit tests you create using the +/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro. You +/// don't usually need to interact directly with an instance of this type. +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +public struct ExitTest: Sendable, ~Copyable { + /// The exit test that is running in the current process, if any. + /// + /// If the current process was created to run an exit test, the value of this + /// property describes that exit test. If this process is the parent process + /// of an exit test, or if no exit test is currently running, the value of + /// this property is `nil`. + /// + /// The value of this property is constant across all tasks in the current + /// process. + public static var current: ExitTest? { get } +} +``` + +### Exit conditions + +These macros take an argument of the new type `ExitTest.Condition`. This type +describes how the child process is expected to have exited: + +- With a specific exit code (as passed to the C standard function `exit()` or a + platform-specific equivalent); +- With a specific signal (on platforms that support signal handling[^winsig]); +- With any successful status; or +- With any failure status. + +[^winsig]: Windows nominally supports signal handling as it is part of the C + standard, but not to the degree that signals are supported by POSIX-like or + UNIX-derived operating systems. Swift Testing makes a "best effort" to emulate + signal-handling support on Windows. See [this](https://forums.swift.org/t/swift-on-windows-question-about-signals-and-exceptions/76640/2) + Swift forum message for more information. + +The type is declared as: + +```swift +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +extension ExitTest { + /// The possible conditions under which an exit test will complete. + /// + /// Values of this type are used to describe the conditions under which an + /// exit test is expected to pass or fail by passing them to + /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(exitsWith:observing:_:sourceLocation:performing:)``. + /// + /// ## Topics + /// + /// ### Successful exit conditions + /// + /// - ``success`` + /// + /// ### Failing exit conditions + /// + /// - ``failure`` + /// - ``exitCode(_:)`` + /// - ``signal(_:)`` + public struct Condition: Sendable { + /// A condition that matches when a process terminates successfully with exit + /// code `EXIT_SUCCESS`. + /// + /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), + /// `EXIT_SUCCESS` and `EXIT_FAILURE` as well as `0` (as a synonym for + /// `EXIT_SUCCESS`.) + public static var success: Self { get } + + /// A condition that matches when a process terminates abnormally with any + /// exit code other than `EXIT_SUCCESS` or with any signal. + public static var failure: Self { get } + + public init(_ statusAtExit: StatusAtExit) + + /// Creates a condition that matches when a process terminates with a given + /// exit code. + /// + /// - Parameters: + /// - exitCode: The exit code yielded by the process. + /// + /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), + /// `EXIT_SUCCESS` and `EXIT_FAILURE`. Platforms may additionally define their + /// own non-standard exit codes: + /// + /// | Platform | Header | + /// |-|-| + /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/_Exit.3.html), [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysexits.3.html) | + /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Exit-Status.html), `` | + /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?exit(3)), [``](https://man.freebsd.org/cgi/man.cgi?sysexits(3)) | + /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | + /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | + /// + /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by + /// the process is yielded to the parent process. Linux and other POSIX-like + /// systems may only reliably report the low unsigned 8 bits (0–255) of + /// the exit code. + public static func exitCode(_ exitCode: CInt) -> Self + + /// Creates a condition that matches when a process terminates with a given + /// signal. + /// + /// - Parameters: + /// - signal: The signal that terminated the process. + /// + /// The C programming language defines a number of [standard signals](https://en.cppreference.com/w/c/program/SIG_types). + /// Platforms may additionally define their own non-standard signal codes: + /// + /// | Platform | Header | + /// |-|-| + /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/signal.3.html) | + /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Standard-Signals.html) | + /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | + /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | + /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | + public static func signal(_ signal: CInt) -> Self + } +} +``` + +### Exit status + +The set of possible status codes reported by the child process are represented +by the `StatusAtExit` enumeration: + +```swift +/// An enumeration describing possible status a process will yield on exit. +/// +/// You can convert an instance of this type to an instance of +/// ``ExitTest/Condition`` using ``ExitTest/Condition/init(_:)``. That value +/// can then be used to describe the condition under which an exit test is +/// expected to pass or fail by passing it to +/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(exitsWith:observing:_:sourceLocation:performing:)``. +#if SWT_NO_PROCESS_SPAWNING +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +public enum StatusAtExit: Sendable { + /// The process terminated with the given exit code. + /// + /// [...] + case exitCode(_ exitCode: CInt) + + /// The process terminated with the given signal. + /// + /// [...] + case signal(_ signal: CInt) +} +``` + +### Exit test results + +These macros return an instance of the new type `ExitTest.Result`. This type +describes the results of the process including its reported exit condition and +the contents of its standard output and standard error streams, if requested. + +```swift +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +extension ExitTest { + /// A type representing the result of an exit test after it has exited and + /// returned control to the calling test function. + /// + /// Both ``expect(exitsWith:observing:_:sourceLocation:performing:)`` and + /// ``require(exitsWith:observing:_:sourceLocation:performing:)`` return + /// instances of this type. + public struct Result: Sendable { + /// The status of the process hosting the exit test at the time it exits. + /// + /// When the exit test passes, the value of this property is equal to the + /// exit status reported by the process that hosted the exit test. + public var statusAtExit: StatusAtExit { get set } + + /// All bytes written to the standard output stream of the exit test before + /// it exited. + /// + /// The value of this property may contain any arbitrary sequence of bytes, + /// including sequences that are not valid UTF-8 and cannot be decoded by + /// [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). + /// Consider using [`String.init(validatingCString:)`](https://developer.apple.com/documentation/swift/string/init(validatingcstring:)-992vo) + /// instead. + /// + /// When checking the value of this property, keep in mind that the standard + /// output stream is globally accessible, and any code running in an exit + /// test may write to it including including the operating system and any + /// third-party dependencies you have declared in your package. Rather than + /// comparing the value of this property with [`==`](https://developer.apple.com/documentation/swift/array/==(_:_:)), + /// use [`contains(_:)`](https://developer.apple.com/documentation/swift/collection/contains(_:)) + /// to check if expected output is present. + /// + /// To enable gathering output from the standard output stream during an + /// exit test, pass `\.standardOutputContent` in the `observedValues` + /// argument of ``expect(exitsWith:observing:_:sourceLocation:performing:)`` + /// or ``require(exitsWith:observing:_:sourceLocation:performing:)``. + /// + /// If you did not request standard output content when running an exit test, + /// the value of this property is the empty array. + public var standardOutputContent: [UInt8] { get set } + + /// All bytes written to the standard error stream of the exit test before + /// it exited. + /// + /// [...] + public var standardErrorContent: [UInt8] { get set } + } +} +``` + +### Usage + +These macros can be used within a test function: + +```swift +@Test func `We only eat delicious tacos`() async { + await #expect(exitsWith: .failure) { + var taco = Taco() + taco.isDelicious = false + eat(taco) + } +} +``` + +Given the definition of `eat(_:)` above, this test can be expected to hit a +precondition failure and crash the process; because `.failure` was the specified +exit condition, this is treated as a successful test. + +It is often interesting to examine what is written to the standard output and +standard error streams by code running in an exit test. Callers can request that +either or both stream be captured and included in the result of the call to +`#expect(exitsWith:)` or `#require(exitsWith:)`. Capturing these streams can be +a memory-intensive operation, so the caller must explicitly opt in: + +```swift +@Test func `We only eat delicious tacos`() async throws { + let result = try await #require(exitsWith: .failure, observing: [\.standardErrorContent])) { ... } + #expect(result.standardOutputContent.contains("ERROR: This taco tastes terrible!".utf8) +} +``` + +There are some constraints on valid exit tests: + +1. Because exit tests are run in child processes, they cannot capture any state + from the calling context (hence their body closures are `@convention(thin)` + or `@convention(c)`.) See the **Future directions** for further discussion. +1. Exit tests cannot recursively invoke other exit tests; this is a constraint + that could potentially be lifted in the future, but it would be technically + complex to do so. + +If a Swift Testing issue such as an expectation failure occurs while running an +exit test, it is reported to the parent process and to the user as if it +happened locally. If an error is thrown from an exit test and not caught, it +behaves the same way a Swift program would if an error were thrown from its +`main()` function (that is, the program terminates abnormally.) + +## Source compatibility + +This is a new interface that is unlikely to collide with any existing +client-provided interfaces. The typical Swift disambiguation tools can be used +if needed. + +## Integration with supporting tools + +SPI is provided to allow testing environments other than Swift Package Manager +to detect and run exit tests: + +```swift +@_spi(ForToolsIntegrationOnly) +extension ExitTest { + /// A type whose instances uniquely identify instances of ``ExitTest``. + public struct ID: Sendable, Equatable, Codable { /* ... */ } + + /// A value that uniquely identifies this instance. + public var id: ID { get set } + + /// Key paths representing results from within this exit test that should be + /// observed and returned to the caller. + /// + /// The testing library sets this property to match what was passed by the + /// developer to the `#expect(exitsWith:)` or `#require(exitsWith:)` macro. + /// If you are implementing an exit test handler, you can check the value of + /// this property to determine what information you need to preserve from your + /// child process. + /// + /// The value of this property always includes ``ExitTest/Result/statusAtExit`` + /// even if the test author does not specify it. + /// + /// Within a child process running an exit test, the value of this property is + /// otherwise unspecified. + public var observedValues: [any PartialKeyPath & Sendable] { get set } + + /// Call the exit test in the current process. + /// + /// This function invokes the closure originally passed to + /// `#expect(exitsWith:)` _in the current process_. That closure is expected + /// to terminate the process; if it does not, the testing library will + /// terminate the process as if its `main()` function returned naturally. + public consuming func callAsFunction() async -> Never + + /// Find the exit test function at the given source location. + /// + /// - Parameters: + /// - id: The unique identifier of the exit test to find. + /// + /// - Returns: The specified exit test function, or `nil` if no such exit test + /// could be found. + public static func find(identifiedBy id: ExitTest.ID) -> Self? + + /// A handler that is invoked when an exit test starts. + /// + /// - Parameters: + /// - exitTest: The exit test that is starting. + /// + /// - Returns: The result of the exit test including the condition under which + /// it exited. + /// + /// - Throws: Any error that prevents the normal invocation or execution of + /// the exit test. + /// + /// This handler is invoked when an exit test (i.e. a call to either + /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(exitsWith:observing:_:sourceLocation:performing:)``) is started. + /// The handler is responsible for initializing a new child environment + /// (typically a child process) and running the exit test identified by + /// `sourceLocation` there. + /// + /// In the child environment, you can find the exit test again by calling + /// ``ExitTest/find(at:)`` and can run it by calling + /// ``ExitTest/callAsFunction()``. + /// + /// The parent environment should suspend until the results of the exit test + /// are available or the child environment is otherwise terminated. The parent + /// environment is then responsible for interpreting those results and + /// recording any issues that occur. + public typealias Handler = @Sendable (_ exitTest: borrowing ExitTest) async throws -> ExitTest.Result +} + +@_spi(ForToolsIntegrationOnly) +extension Configuration { + /// A handler that is invoked when an exit test starts. + /// + /// For an explanation of how this property is used, see ``ExitTest/Handler``. + /// + /// When using the `swift test` command from Swift Package Manager, this + /// property is pre-configured. Otherwise, the default value of this property + /// records an issue indicating that it has not been configured. + public var exitTestHandler: ExitTest.Handler { get set } +} +``` + +Any tools that use `swift build --build-tests`, `swift test`, or equivalent to +compile executables for testing will inherit the functionality provided for +`swift test` and do not need to implement their own exit test handlers. Tools +that directly compile test targets or otherwise do not leverage Swift Package +Manager will need to provide an implementation. + +## Future directions + +### Support for iOS, WASI, etc. + +The need for exit tests on other platforms is just as strong as it is on the +supported platforms (macOS, Linux, FreeBSD/OpenBSD, and Windows). These +platforms do not support spawning new processes, so a different mechanism for +running exit tests would be needed. + +Android _does_ have `posix_spawn()` and related API and may be able to use the +same implementation as Linux. Android support is an ongoing area of research for +Swift Testing's core team. + +### Recursive exit tests + +The technical constraints preventing recursive exit test invocation can be +resolved if there is a need to do so. However, we don't anticipate that this +constraint will be a serious issue for developers. + +### Support for passing state + +Arbitrary state is necessarily not preserved between the parent and child +processes, but there is little to prevent us from adding a variadic `arguments:` +argument and passing values whose types conform to `Codable`. + +The blocker right now is that there is no type information during macro +expansion, meaning that the testing library can emit the glue code to _encode_ +arguments, but does not know what types to use when _decoding_ those arguments. +If generic types were made available during macro expansion via the macro +expansion context, then it would be possible to synthesize the correct logic. + +Alternatively, if the language gained something akin to C++'s `decltype()`, we +could leverage closures' capture list syntax. Subjectively, capture lists ought +to be somewhat intuitive for developers in this context: + +```swift +let (lettuce, cheese, crema) = taco.addToppings() +await #expect(exitsWith: .failure) { [taco, plant = lettuce, cheese, crema] in + try taco.removeToppings(plant, cheese, crema) +} +``` + +### More nuanced support for throwing errors from exit test bodies + +Currently, if an error is thrown from an exit test without being caught, the +test behaves the same way a program does when an error is thrown from an +explicit or implicit `main() throws` function: the process terminates abnormally +and control returns to the test function that is awaiting the exit test: + +```swift +await #expect(exitsWith: .failure) { + throw TacoError.noTacosFound +} +``` + +If the test function is expecting `.failure`, this means the test passes. +Although this behavior is consistent with modelling an exit test as an +independent program (i.e. the exit test acts like its own `main()` function), it +may be surprising to test authors who aren't thinking about error handling. In +the future, we may want to offer a compile-time diagnostic if an error is thrown +from an exit test body without being caught, or offer a distinct exit condition +(i.e. `.errorNotCaught(_ error: Error & Codable)`) for these uncaught errors. +For error types that conform to `Codable`, we could offer rethrowing behavior, +but this is not possible for error types that cannot be sent across process +boundaries. + +### Exit-testing customized processes + +The current model of exit tests is that they run in approximately the same +environment as the test process by spawning a copy of the executable under test. +There is a very real use case for allowing testing other processes and +inspecting their output. In the future, we could provide API to spawn a process +with particular arguments and environment variables, then inspect its exit +condition and standard output/error streams: + +```swift +let result = try await #require( + executableAt: "/usr/bin/swift", + passing: ["build", "--package-path", ...], + environment: [:], + exitsWith: .success +) +#expect(result.standardOutputContent.contains("Build went well!").utf8) +``` + +We could also investigate explicitly integrating with [`Foundation.Process`](https://developer.apple.com/documentation/foundation/process) +or the proposed [`Foundation.Subprocess`](https://github.com/swiftlang/swift-foundation/blob/main/Proposals/0007-swift-subprocess.md) +as an alternative: + +```swift +let process = Process() +process.executableURL = URL(filePath: "/usr/bin/swift", directoryHint: .notDirectory) +process.arguments = ["build", "--package-path", ...] +let result = try await #require(process, exitsWith: .success) +#expect(result.standardOutputContent.contains("Build went well!").utf8) +``` + +## Alternatives considered + +- Doing nothing. + +- Marking exit tests using a trait rather than a new `#expect()` overload: + + ```swift + @Test(.exits(with: .failure)) + func `We only eat delicious tacos`() { + var taco = Taco() + taco.isDelicious = false + eat(taco) + } + ``` + + This syntax would require separate test functions for each exit test, while + reusing the same function for relatively concise tests may be preferable. + + It would also potentially conflict with parameterized tests, as it is not + possible to pass arbitrary parameters to the child process. It would be + necessary to teach the testing library's macro target about the + `.exits(with:)` trait so that it could produce a diagnostic when used with a + parameterized test function. + +- Inferring exit tests from test functions that return `Never`: + + ```swift + @Test func `No seafood for me, thanks!`() -> Never { + var taco = Taco() + taco.toppings.append(.shrimp) + eat(taco) + fatalError("Should not have eaten that!") + } + ``` + + There's a certain synergy in inferring that a test function that returns + `Never` must necessarily be a crasher and should be handled out of process. + However, this forces the test author to add a call to `fatalError()` or + similar in the event that the code under test does _not_ terminate, and there + is no obvious way to express that a specific exit code, signal, or other + condition is expected (as opposed to just "it exited".) + + We might want to support that sort of inference in the future (i.e. "don't run + this test in-process because it will terminate the test run"), but without + also inferring success or failure from the process' exit status. + +- Naming the macro something else such as: + + - `#exits(with:_:)`; + - `#exits(because:_:)`; + - `#expect(exitsBecause:_:)`; + - `#expect(terminatesBecause:_:)`; etc. + + While "with" is normally avoided in symbol names in Swift, it sometimes really + is the best preposition for the job. "Because", "due to", and others don't + sound "right" when the entire expression is read out loud. For example, you + probably wouldn't say "exits due to success" in English. + +- Combining `StatusAtExit` and `ExitTest.Condition` into a single type: + + ```swift + enum ExitCondition { + case failure // any failure + case exitCode(CInt) + case signal(CInt) + } + ``` + + This simplified the set of types used for exit tests, but made comparing two + exit conditions complicated and necessitated a `==` operator that did not + satisfy the requirements of the `Equatable` protocol. + +- Naming `StatusAtExit` something else such as: + + - `ExitStatus`, which could be too easily confusable with exit _codes_ such as + `EXIT_SUCCESS`; + - `ProcessStatus`, but we don't say "process" in our API surface elsewhere; + - `Status`, which is too generic, + - `ExitReason`, but "status" is a more widely-used term of art for this + concept; or + - `terminationStatus` (which Foundation to represent approximately the same + concept), but we don't use "termination" in Swift Testing's API anywhere. + + I settled on `StatusAtExit` because it was distinct and makes it clear that it + represents the status of a process _at exit time_. `ExitStatus` could be + interpreted as the status of the exit itself, i.e.: + + ```swift + enum ExitStatus { + case running + case suspended + case exiting + case exited + } + ``` + +- Changing the implementation of `precondition()`, `fatalError()`, etc. in the + standard library so that they do not terminate the current process while + testing, thus removing the need to spawn a child process for an exit test. + + Most of the functions in this family return `Never`, and changing their return + types would be ABI-breaking (as well as a pessimization in production code.) + Even if we did modify these functions in the Swift standard library, other + ways to terminate the process exist and would not be covered: + + - Calling the C standard function `exit()`; + - Throwing an uncaught Objective-C or C++ exception; + - Sending a signal to the process; or + - Misusing memory (e.g. trying to dereference a null pointer.) + + Modifying the C or C++ standard library, or modifying the Objective-C runtime, + would be well beyond the scope of this proposal. + +## Acknowledgments + +Many thanks to the XCTest and Swift Testing team. Thanks to @compnerd for his +help with the Windows implementation. Thanks to my colleagues Coops, +Danny N., David R., Drew Y., and Robert K. at Apple for +their help with the nuances of crash reporting on macOS. diff --git a/Sources/Testing/ExitTests/ExitTest.Condition.swift b/Sources/Testing/ExitTests/ExitTest.Condition.swift index 10f2a6ff0..7d824c43d 100644 --- a/Sources/Testing/ExitTests/ExitTest.Condition.swift +++ b/Sources/Testing/ExitTests/ExitTest.Condition.swift @@ -10,7 +10,6 @@ private import _TestingInternals -@_spi(Experimental) #if SWT_NO_EXIT_TESTS @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif @@ -21,6 +20,22 @@ extension ExitTest { /// exit test is expected to pass or fail by passing them to /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or /// ``require(exitsWith:observing:_:sourceLocation:performing:)``. + /// + /// ## Topics + /// + /// ### Successful exit conditions + /// + /// - ``success`` + /// + /// ### Failing exit conditions + /// + /// - ``failure`` + /// - ``exitCode(_:)`` + /// - ``signal(_:)`` + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public struct Condition: Sendable { /// An enumeration describing the possible conditions for an exit test. private enum _Kind: Sendable, Equatable { @@ -38,13 +53,20 @@ extension ExitTest { // MARK: - -@_spi(Experimental) #if SWT_NO_EXIT_TESTS @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif extension ExitTest.Condition { /// A condition that matches when a process terminates successfully with exit /// code `EXIT_SUCCESS`. + /// + /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), + /// `EXIT_SUCCESS` and `EXIT_FAILURE` as well as `0` (as a synonym for + /// `EXIT_SUCCESS`.) + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public static var success: Self { // Strictly speaking, the C standard treats 0 as a successful exit code and // potentially distinct from EXIT_SUCCESS. To my knowledge, no modern @@ -59,10 +81,17 @@ extension ExitTest.Condition { /// A condition that matches when a process terminates abnormally with any /// exit code other than `EXIT_SUCCESS` or with any signal. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public static var failure: Self { Self(_kind: .failure) } + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public init(_ statusAtExit: StatusAtExit) { self.init(_kind: .statusAtExit(statusAtExit)) } @@ -89,6 +118,10 @@ extension ExitTest.Condition { /// the process is yielded to the parent process. Linux and other POSIX-like /// systems may only reliably report the low unsigned 8 bits (0–255) of /// the exit code. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public static func exitCode(_ exitCode: CInt) -> Self { #if !SWT_NO_EXIT_TESTS Self(.exitCode(exitCode)) @@ -113,6 +146,10 @@ extension ExitTest.Condition { /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public static func signal(_ signal: CInt) -> Self { #if !SWT_NO_EXIT_TESTS Self(.signal(signal)) diff --git a/Sources/Testing/ExitTests/ExitTest.Result.swift b/Sources/Testing/ExitTests/ExitTest.Result.swift index beb2d56fc..edab995d9 100644 --- a/Sources/Testing/ExitTests/ExitTest.Result.swift +++ b/Sources/Testing/ExitTests/ExitTest.Result.swift @@ -8,7 +8,6 @@ // See https://swift.org/CONTRIBUTORS.txt for Swift project authors // -@_spi(Experimental) #if SWT_NO_EXIT_TESTS @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif @@ -19,11 +18,19 @@ extension ExitTest { /// Both ``expect(exitsWith:observing:_:sourceLocation:performing:)`` and /// ``require(exitsWith:observing:_:sourceLocation:performing:)`` return /// instances of this type. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public struct Result: Sendable { - /// The exit condition the exit test exited with. + /// The status of the process hosting the exit test at the time it exits. /// /// When the exit test passes, the value of this property is equal to the /// exit status reported by the process that hosted the exit test. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public var statusAtExit: StatusAtExit /// All bytes written to the standard output stream of the exit test before @@ -50,6 +57,10 @@ extension ExitTest { /// /// If you did not request standard output content when running an exit /// test, the value of this property is the empty array. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public var standardOutputContent: [UInt8] = [] /// All bytes written to the standard error stream of the exit test before @@ -76,6 +87,10 @@ extension ExitTest { /// /// If you did not request standard error content when running an exit test, /// the value of this property is the empty array. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public var standardErrorContent: [UInt8] = [] @_spi(ForToolsIntegrationOnly) diff --git a/Sources/Testing/ExitTests/ExitTest.swift b/Sources/Testing/ExitTests/ExitTest.swift index 6341ce422..ba6371599 100644 --- a/Sources/Testing/ExitTests/ExitTest.swift +++ b/Sources/Testing/ExitTests/ExitTest.swift @@ -26,10 +26,13 @@ private import _TestingInternals /// A type describing an exit test. /// /// Instances of this type describe exit tests you create using the -/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` +/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or /// ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro. You /// don't usually need to interact directly with an instance of this type. -@_spi(Experimental) +/// +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } #if SWT_NO_EXIT_TESTS @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif @@ -112,7 +115,6 @@ public struct ExitTest: Sendable, ~Copyable { #if !SWT_NO_EXIT_TESTS // MARK: - Current -@_spi(Experimental) extension ExitTest { /// A container type to hold the current exit test. /// @@ -142,6 +144,10 @@ extension ExitTest { /// /// The value of this property is constant across all tasks in the current /// process. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public static var current: ExitTest? { _read { if let current = _current.rawValue { @@ -155,7 +161,7 @@ extension ExitTest { // MARK: - Invocation -@_spi(Experimental) @_spi(ForToolsIntegrationOnly) +@_spi(ForToolsIntegrationOnly) extension ExitTest { /// Disable crash reporting, crash logging, or core dumps for the current /// process. @@ -294,7 +300,7 @@ extension ExitTest: DiscoverableAsTestContent { } } -@_spi(Experimental) @_spi(ForToolsIntegrationOnly) +@_spi(ForToolsIntegrationOnly) extension ExitTest { /// Find the exit test function at the given source location. /// @@ -431,7 +437,7 @@ extension ABI { fileprivate typealias BackChannelVersion = v1 } -@_spi(Experimental) @_spi(ForToolsIntegrationOnly) +@_spi(ForToolsIntegrationOnly) extension ExitTest { /// A handler that is invoked when an exit test starts. /// @@ -467,13 +473,13 @@ extension ExitTest { /// events should be written, or `nil` if the file handle could not be /// resolved. private static let _backChannelForEntryPoint: FileHandle? = { - guard let backChannelEnvironmentVariable = Environment.variable(named: "SWT_EXPERIMENTAL_BACKCHANNEL") else { + guard let backChannelEnvironmentVariable = Environment.variable(named: "SWT_BACKCHANNEL") else { return nil } // Erase the environment variable so that it cannot accidentally be opened // twice (nor, in theory, affect the code of the exit test.) - Environment.setVariable(nil, named: "SWT_EXPERIMENTAL_BACKCHANNEL") + Environment.setVariable(nil, named: "SWT_BACKCHANNEL") var fd: CInt? #if SWT_TARGET_OS_APPLE || os(Linux) || os(FreeBSD) || os(OpenBSD) @@ -504,10 +510,10 @@ extension ExitTest { static func findInEnvironmentForEntryPoint() -> Self? { // Find the ID of the exit test to run, if any, in the environment block. var id: ExitTest.ID? - if var idString = Environment.variable(named: "SWT_EXPERIMENTAL_EXIT_TEST_ID") { + if var idString = Environment.variable(named: "SWT_EXIT_TEST_ID") { // Clear the environment variable. It's an implementation detail and exit // test code shouldn't be dependent on it. Use ExitTest.current if needed! - Environment.setVariable(nil, named: "SWT_EXPERIMENTAL_EXIT_TEST_ID") + Environment.setVariable(nil, named: "SWT_EXIT_TEST_ID") id = try? idString.withUTF8 { idBuffer in try JSON.decode(ExitTest.ID.self, from: UnsafeRawBufferPointer(idBuffer)) @@ -641,7 +647,7 @@ extension ExitTest { // Insert a specific variable that tells the child process which exit test // to run. try JSON.withEncoding(of: exitTest.id) { json in - childEnvironment["SWT_EXPERIMENTAL_EXIT_TEST_ID"] = String(decoding: json, as: UTF8.self) + childEnvironment["SWT_EXIT_TEST_ID"] = String(decoding: json, as: UTF8.self) } typealias ResultUpdater = @Sendable (inout ExitTest.Result) -> Void @@ -687,7 +693,7 @@ extension ExitTest { #warning("Platform-specific implementation missing: back-channel pipe unavailable") #endif if let backChannelEnvironmentVariable { - childEnvironment["SWT_EXPERIMENTAL_BACKCHANNEL"] = backChannelEnvironmentVariable + childEnvironment["SWT_BACKCHANNEL"] = backChannelEnvironmentVariable } // Spawn the child process. diff --git a/Sources/Testing/ExitTests/StatusAtExit.swift b/Sources/Testing/ExitTests/StatusAtExit.swift index 26514ffa5..0f24a5918 100644 --- a/Sources/Testing/ExitTests/StatusAtExit.swift +++ b/Sources/Testing/ExitTests/StatusAtExit.swift @@ -18,7 +18,10 @@ private import _TestingInternals /// expected to pass or fail by passing it to /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or /// ``require(exitsWith:observing:_:sourceLocation:performing:)``. -@_spi(Experimental) +/// +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } #if SWT_NO_PROCESS_SPAWNING @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif @@ -44,6 +47,10 @@ public enum StatusAtExit: Sendable { /// the process is yielded to the parent process. Linux and other POSIX-like /// systems may only reliably report the low unsigned 8 bits (0–255) of /// the exit code. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } case exitCode(_ exitCode: CInt) /// The process terminated with the given signal. @@ -61,12 +68,15 @@ public enum StatusAtExit: Sendable { /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } case signal(_ signal: CInt) } // MARK: - Equatable -@_spi(Experimental) #if SWT_NO_PROCESS_SPAWNING @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif diff --git a/Sources/Testing/Expectations/Expectation+Macro.swift b/Sources/Testing/Expectations/Expectation+Macro.swift index 5111fbddd..8baa1de1e 100644 --- a/Sources/Testing/Expectations/Expectation+Macro.swift +++ b/Sources/Testing/Expectations/Expectation+Macro.swift @@ -582,7 +582,10 @@ public macro require( /// ``` /// /// An exit test cannot run within another exit test. -@_spi(Experimental) +/// +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } #if SWT_NO_EXIT_TESTS @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif @@ -694,7 +697,10 @@ public macro require( /// ``` /// /// An exit test cannot run within another exit test. -@_spi(Experimental) +/// +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } #if SWT_NO_EXIT_TESTS @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif diff --git a/Sources/Testing/Expectations/ExpectationChecking+Macro.swift b/Sources/Testing/Expectations/ExpectationChecking+Macro.swift index aa999395a..0047e6a37 100644 --- a/Sources/Testing/Expectations/ExpectationChecking+Macro.swift +++ b/Sources/Testing/Expectations/ExpectationChecking+Macro.swift @@ -1145,7 +1145,6 @@ public func __checkClosureCall( /// /// - Warning: This function is used to implement the `#expect()` and /// `#require()` macros. Do not call it directly. -@_spi(Experimental) public func __checkClosureCall( identifiedBy exitTestID: (UInt64, UInt64, UInt64, UInt64), exitsWith expectedExitCondition: ExitTest.Condition, diff --git a/Sources/Testing/Running/Configuration.swift b/Sources/Testing/Running/Configuration.swift index 30a2ce303..fd2713bbc 100644 --- a/Sources/Testing/Running/Configuration.swift +++ b/Sources/Testing/Running/Configuration.swift @@ -217,8 +217,7 @@ public struct Configuration: Sendable { /// When using the `swift test` command from Swift Package Manager, this /// property is pre-configured. Otherwise, the default value of this property /// records an issue indicating that it has not been configured. - @_spi(Experimental) - public var exitTestHandler: ExitTest.Handler = { exitTest in + public var exitTestHandler: ExitTest.Handler = { _ in throw SystemError(description: "Exit test support has not been implemented by the current testing infrastructure.") } #endif diff --git a/Sources/Testing/Testing.docc/Expectations.md b/Sources/Testing/Testing.docc/Expectations.md index fd3b0070d..ce451d9d9 100644 --- a/Sources/Testing/Testing.docc/Expectations.md +++ b/Sources/Testing/Testing.docc/Expectations.md @@ -72,6 +72,13 @@ the test when the code doesn't satisfy a requirement, use - ``require(throws:_:sourceLocation:performing:)-4djuw`` - ``require(_:sourceLocation:performing:throws:)`` +### Checking how processes exit + +- ``expect(exitsWith:observing:_:sourceLocation:performing:)`` +- ``require(exitsWith:observing:_:sourceLocation:performing:)`` +- ``ExitTest`` +- ``StatusAtExit`` + ### Confirming that asynchronous events occur - diff --git a/Tests/TestingTests/ExitTestTests.swift b/Tests/TestingTests/ExitTestTests.swift index bc3425e0a..befb856c1 100644 --- a/Tests/TestingTests/ExitTestTests.swift +++ b/Tests/TestingTests/ExitTestTests.swift @@ -8,7 +8,7 @@ // See https://swift.org/CONTRIBUTORS.txt for Swift project authors // -@testable @_spi(Experimental) @_spi(ForToolsIntegrationOnly) import Testing +@testable @_spi(ForToolsIntegrationOnly) import Testing private import _TestingInternals #if !SWT_NO_EXIT_TESTS From f54c9d53263b292b664f918563b9ed10f2e12cda Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Mon, 3 Mar 2025 16:22:09 -0500 Subject: [PATCH 02/19] Remove proposal doc (moving to SE) --- Documentation/Proposals/NNNN-exit-tests.md | 787 --------------------- 1 file changed, 787 deletions(-) delete mode 100644 Documentation/Proposals/NNNN-exit-tests.md diff --git a/Documentation/Proposals/NNNN-exit-tests.md b/Documentation/Proposals/NNNN-exit-tests.md deleted file mode 100644 index 06f740d63..000000000 --- a/Documentation/Proposals/NNNN-exit-tests.md +++ /dev/null @@ -1,787 +0,0 @@ -# Exit tests - -* Proposal: [SWT-NNNN](NNNN-exit-tests.md) -* Authors: [Jonathan Grynspan](https://github.com/grynspan) -* Status: **Awaiting review** -* Bug: [apple/swift-testing#157](https://github.com/apple/swift-testing/issues/157) -* Implementation: [apple/swift-testing#307](https://github.com/apple/swift-testing/pull/307) -* Review: ([pitch](https://forums.swift.org/t/pitch-exit-tests/78071)) - -## Introduction - -One of the first enhancement requests we received for Swift Testing was the -ability to test for precondition failures and other critical failures that -terminate the current process when they occur. This feature is also frequently -requested for XCTest. With Swift Testing, we have the opportunity to build such -a feature in an ergonomic way. - -> [!NOTE] -> This feature has various names in the relevant literature, e.g. "exit tests", -> "death tests", "death assertions", "termination tests", etc. We consistently -> use the term "exit tests" to refer to them. - -## Motivation - -Imagine a function, implemented in a package, that includes a precondition: - -```swift -func eat(_ taco: consuming Taco) { - precondition(taco.isDelicious, "Tasty tacos only!") - ... -} -``` - -Today, a test author can write unit tests for this function, but there is no way -to make sure that the function rejects a taco whose `isDelicious` property is -`false` because a test that passes such a taco as input will crash (correctly!) -when it calls `precondition()`. - -An exit test allows testing this sort of functionality. The mechanism by which -an exit test is implemented varies between testing libraries and languages, but -a common implementation involves spawning a new process, performing the work -there, and checking that the spawned process ultimately terminates with a -particular (possibly platform-specific) exit status. - -Adding exit tests to Swift Testing would allow an entirely new class of tests -and would improve code coverage for existing test targets that adopt them. - -## Proposed solution - -This proposal introduces new overloads of the `#expect()` and `#require()` -macros that take, as an argument, a closure to be executed in a child process. -When called, these macros spawn a new process using the relevant -platform-specific interface (`posix_spawn()`, `CreateProcessW()`, etc.), call -the closure from within that process, and suspend the caller until that process -terminates. The exit status of the process is then compared against a known -value passed to the macro, allowing the test to pass or fail as appropriate. - -The function from earlier can then be tested using either of the new -overloads: - -```swift -await #expect(exitsWith: .failure) { - var taco = Taco() - taco.isDelicious = false - eat(taco) // should trigger a precondition failure and process termination -} -``` - -## Detailed design - -### New expectations - -We will introduce the following new overloads of `#expect()` and `#require()` to -the testing library: - -```swift -/// Check that an expression causes the process to terminate in a given fashion. -/// -/// - Parameters: -/// - expectedExitCondition: The expected exit condition. -/// - observedValues: An array of key paths representing results from within -/// the exit test that should be observed and returned by this macro. The -/// ``ExitTest/Result/statusAtExit`` property is always returned. -/// - comment: A comment describing the expectation. -/// - sourceLocation: The source location to which recorded expectations and -/// issues should be attributed. -/// - expression: The expression to be evaluated. -/// -/// - Returns: If the exit test passed, an instance of ``ExitTest/Result`` -/// describing the state of the exit test when it exited. If the exit test -/// fails, the result is `nil`. -/// -/// Use this overload of `#expect()` when an expression will cause the current -/// process to terminate and the nature of that termination will determine if -/// the test passes or fails. For example, to test that calling `fatalError()` -/// causes a process to terminate: -/// -/// await #expect(exitsWith: .failure) { -/// fatalError() -/// } -/// -/// - Note: A call to this expectation macro is called an "exit test." -/// -/// ## How exit tests are run -/// -/// When an exit test is performed at runtime, the testing library starts a new -/// process with the same executable as the current process. The current task is -/// then suspended (as with `await`) and waits for the child process to -/// terminate. `expression` is not called in the parent process. -/// -/// Meanwhile, in the child process, `expression` is called directly. To ensure -/// a clean environment for execution, it is not called within the context of -/// the original test. If `expression` does not terminate the child process, the -/// process is terminated automatically as if the main function of the child -/// process were allowed to return naturally. If an error is thrown from -/// `expression`, it is handed as if the error were thrown from `main()` and the -/// process is terminated. -/// -/// Once the child process terminates, the parent process resumes and compares -/// its exit status against `expectedExitCondition`. If they match, the exit -/// test has passed; otherwise, it has failed and an issue is recorded. -/// -/// ## Child process output -/// -/// By default, the child process is configured without a standard output or -/// standard error stream. If your test needs to review the content of either of -/// these streams, you can pass its key path in the `observedValues` argument: -/// -/// let result = await #expect( -/// exitsWith: .failure, -/// observing: [\.standardOutputContent] -/// ) { -/// print("Goodbye, world!") -/// fatalError() -/// } -/// if let result { -/// #expect(result.standardOutputContent.contains(UInt8(ascii: "G"))) -/// } -/// -/// - Note: The content of the standard output and standard error streams may -/// contain any arbitrary sequence of bytes, including sequences that are not -/// valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). -/// These streams are globally accessible within the child process, and any -/// code running in an exit test may write to it including the operating -/// system and any third-party dependencies you have declared in your package. -/// -/// The actual exit condition of the child process is always reported by the -/// testing library even if you do not specify it in `observedValues`. -/// -/// ## Runtime constraints -/// -/// Exit tests cannot capture any state originating in the parent process or -/// from the enclosing lexical context. For example, the following exit test -/// will fail to compile because it captures an argument to the enclosing -/// parameterized test: -/// -/// @Test(arguments: 100 ..< 200) -/// func sellIceCreamCones(count: Int) async { -/// await #expect(exitsWith: .failure) { -/// precondition( -/// count < 10, // ERROR: A C function pointer cannot be formed from a -/// // closure that captures context -/// "Too many ice cream cones" -/// ) -/// } -/// } -/// -/// An exit test cannot run within another exit test. -#if SWT_NO_EXIT_TESTS -@available(*, unavailable, message: "Exit tests are not available on this platform.") -#endif -@discardableResult -@freestanding(expression) public macro expect( - exitsWith expectedExitCondition: ExitTest.Condition, - observing observedValues: [any PartialKeyPath & Sendable] = [], - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: @convention(thin) () async throws -> Void -) -> ExitTest.Result? = #externalMacro(module: "TestingMacros", type: "ExitTestExpectMacro") - -/// Check that an expression causes the process to terminate in a given fashion -/// and throw an error if it did not. -/// -/// [...] -#if SWT_NO_EXIT_TESTS -@available(*, unavailable, message: "Exit tests are not available on this platform.") -#endif -@discardableResult -@freestanding(expression) public macro require( - exitsWith expectedExitCondition: ExitTest.Condition, - observing observedValues: [any PartialKeyPath & Sendable] = [], - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: @convention(thin) () async throws -> Void -) -> ExitTest.Result = #externalMacro(module: "TestingMacros", type: "ExitTestRequireMacro") -``` - -> [!NOTE] -> These interfaces are currently implemented and available on **macOS**, -> **Linux**, **FreeBSD**, **OpenBSD**, and **Windows**. If a platform does not -> support exit tests (generally because it does not support spawning or awaiting -> child processes), then we define `SWT_NO_EXIT_TESTS` when we build it. -> -> `SWT_NO_EXIT_TESTS` is not defined during test target builds. - -### Representing an exit test in Swift - -A new type, `ExitTest`, represents an exit test: - -```swift -/// A type describing an exit test. -/// -/// Instances of this type describe exit tests you create using the -/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or -/// ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro. You -/// don't usually need to interact directly with an instance of this type. -#if SWT_NO_EXIT_TESTS -@available(*, unavailable, message: "Exit tests are not available on this platform.") -#endif -public struct ExitTest: Sendable, ~Copyable { - /// The exit test that is running in the current process, if any. - /// - /// If the current process was created to run an exit test, the value of this - /// property describes that exit test. If this process is the parent process - /// of an exit test, or if no exit test is currently running, the value of - /// this property is `nil`. - /// - /// The value of this property is constant across all tasks in the current - /// process. - public static var current: ExitTest? { get } -} -``` - -### Exit conditions - -These macros take an argument of the new type `ExitTest.Condition`. This type -describes how the child process is expected to have exited: - -- With a specific exit code (as passed to the C standard function `exit()` or a - platform-specific equivalent); -- With a specific signal (on platforms that support signal handling[^winsig]); -- With any successful status; or -- With any failure status. - -[^winsig]: Windows nominally supports signal handling as it is part of the C - standard, but not to the degree that signals are supported by POSIX-like or - UNIX-derived operating systems. Swift Testing makes a "best effort" to emulate - signal-handling support on Windows. See [this](https://forums.swift.org/t/swift-on-windows-question-about-signals-and-exceptions/76640/2) - Swift forum message for more information. - -The type is declared as: - -```swift -#if SWT_NO_EXIT_TESTS -@available(*, unavailable, message: "Exit tests are not available on this platform.") -#endif -extension ExitTest { - /// The possible conditions under which an exit test will complete. - /// - /// Values of this type are used to describe the conditions under which an - /// exit test is expected to pass or fail by passing them to - /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or - /// ``require(exitsWith:observing:_:sourceLocation:performing:)``. - /// - /// ## Topics - /// - /// ### Successful exit conditions - /// - /// - ``success`` - /// - /// ### Failing exit conditions - /// - /// - ``failure`` - /// - ``exitCode(_:)`` - /// - ``signal(_:)`` - public struct Condition: Sendable { - /// A condition that matches when a process terminates successfully with exit - /// code `EXIT_SUCCESS`. - /// - /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), - /// `EXIT_SUCCESS` and `EXIT_FAILURE` as well as `0` (as a synonym for - /// `EXIT_SUCCESS`.) - public static var success: Self { get } - - /// A condition that matches when a process terminates abnormally with any - /// exit code other than `EXIT_SUCCESS` or with any signal. - public static var failure: Self { get } - - public init(_ statusAtExit: StatusAtExit) - - /// Creates a condition that matches when a process terminates with a given - /// exit code. - /// - /// - Parameters: - /// - exitCode: The exit code yielded by the process. - /// - /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), - /// `EXIT_SUCCESS` and `EXIT_FAILURE`. Platforms may additionally define their - /// own non-standard exit codes: - /// - /// | Platform | Header | - /// |-|-| - /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/_Exit.3.html), [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysexits.3.html) | - /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Exit-Status.html), `` | - /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?exit(3)), [``](https://man.freebsd.org/cgi/man.cgi?sysexits(3)) | - /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | - /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | - /// - /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by - /// the process is yielded to the parent process. Linux and other POSIX-like - /// systems may only reliably report the low unsigned 8 bits (0–255) of - /// the exit code. - public static func exitCode(_ exitCode: CInt) -> Self - - /// Creates a condition that matches when a process terminates with a given - /// signal. - /// - /// - Parameters: - /// - signal: The signal that terminated the process. - /// - /// The C programming language defines a number of [standard signals](https://en.cppreference.com/w/c/program/SIG_types). - /// Platforms may additionally define their own non-standard signal codes: - /// - /// | Platform | Header | - /// |-|-| - /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/signal.3.html) | - /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Standard-Signals.html) | - /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | - /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | - /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | - public static func signal(_ signal: CInt) -> Self - } -} -``` - -### Exit status - -The set of possible status codes reported by the child process are represented -by the `StatusAtExit` enumeration: - -```swift -/// An enumeration describing possible status a process will yield on exit. -/// -/// You can convert an instance of this type to an instance of -/// ``ExitTest/Condition`` using ``ExitTest/Condition/init(_:)``. That value -/// can then be used to describe the condition under which an exit test is -/// expected to pass or fail by passing it to -/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or -/// ``require(exitsWith:observing:_:sourceLocation:performing:)``. -#if SWT_NO_PROCESS_SPAWNING -@available(*, unavailable, message: "Exit tests are not available on this platform.") -#endif -public enum StatusAtExit: Sendable { - /// The process terminated with the given exit code. - /// - /// [...] - case exitCode(_ exitCode: CInt) - - /// The process terminated with the given signal. - /// - /// [...] - case signal(_ signal: CInt) -} -``` - -### Exit test results - -These macros return an instance of the new type `ExitTest.Result`. This type -describes the results of the process including its reported exit condition and -the contents of its standard output and standard error streams, if requested. - -```swift -#if SWT_NO_EXIT_TESTS -@available(*, unavailable, message: "Exit tests are not available on this platform.") -#endif -extension ExitTest { - /// A type representing the result of an exit test after it has exited and - /// returned control to the calling test function. - /// - /// Both ``expect(exitsWith:observing:_:sourceLocation:performing:)`` and - /// ``require(exitsWith:observing:_:sourceLocation:performing:)`` return - /// instances of this type. - public struct Result: Sendable { - /// The status of the process hosting the exit test at the time it exits. - /// - /// When the exit test passes, the value of this property is equal to the - /// exit status reported by the process that hosted the exit test. - public var statusAtExit: StatusAtExit { get set } - - /// All bytes written to the standard output stream of the exit test before - /// it exited. - /// - /// The value of this property may contain any arbitrary sequence of bytes, - /// including sequences that are not valid UTF-8 and cannot be decoded by - /// [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). - /// Consider using [`String.init(validatingCString:)`](https://developer.apple.com/documentation/swift/string/init(validatingcstring:)-992vo) - /// instead. - /// - /// When checking the value of this property, keep in mind that the standard - /// output stream is globally accessible, and any code running in an exit - /// test may write to it including including the operating system and any - /// third-party dependencies you have declared in your package. Rather than - /// comparing the value of this property with [`==`](https://developer.apple.com/documentation/swift/array/==(_:_:)), - /// use [`contains(_:)`](https://developer.apple.com/documentation/swift/collection/contains(_:)) - /// to check if expected output is present. - /// - /// To enable gathering output from the standard output stream during an - /// exit test, pass `\.standardOutputContent` in the `observedValues` - /// argument of ``expect(exitsWith:observing:_:sourceLocation:performing:)`` - /// or ``require(exitsWith:observing:_:sourceLocation:performing:)``. - /// - /// If you did not request standard output content when running an exit test, - /// the value of this property is the empty array. - public var standardOutputContent: [UInt8] { get set } - - /// All bytes written to the standard error stream of the exit test before - /// it exited. - /// - /// [...] - public var standardErrorContent: [UInt8] { get set } - } -} -``` - -### Usage - -These macros can be used within a test function: - -```swift -@Test func `We only eat delicious tacos`() async { - await #expect(exitsWith: .failure) { - var taco = Taco() - taco.isDelicious = false - eat(taco) - } -} -``` - -Given the definition of `eat(_:)` above, this test can be expected to hit a -precondition failure and crash the process; because `.failure` was the specified -exit condition, this is treated as a successful test. - -It is often interesting to examine what is written to the standard output and -standard error streams by code running in an exit test. Callers can request that -either or both stream be captured and included in the result of the call to -`#expect(exitsWith:)` or `#require(exitsWith:)`. Capturing these streams can be -a memory-intensive operation, so the caller must explicitly opt in: - -```swift -@Test func `We only eat delicious tacos`() async throws { - let result = try await #require(exitsWith: .failure, observing: [\.standardErrorContent])) { ... } - #expect(result.standardOutputContent.contains("ERROR: This taco tastes terrible!".utf8) -} -``` - -There are some constraints on valid exit tests: - -1. Because exit tests are run in child processes, they cannot capture any state - from the calling context (hence their body closures are `@convention(thin)` - or `@convention(c)`.) See the **Future directions** for further discussion. -1. Exit tests cannot recursively invoke other exit tests; this is a constraint - that could potentially be lifted in the future, but it would be technically - complex to do so. - -If a Swift Testing issue such as an expectation failure occurs while running an -exit test, it is reported to the parent process and to the user as if it -happened locally. If an error is thrown from an exit test and not caught, it -behaves the same way a Swift program would if an error were thrown from its -`main()` function (that is, the program terminates abnormally.) - -## Source compatibility - -This is a new interface that is unlikely to collide with any existing -client-provided interfaces. The typical Swift disambiguation tools can be used -if needed. - -## Integration with supporting tools - -SPI is provided to allow testing environments other than Swift Package Manager -to detect and run exit tests: - -```swift -@_spi(ForToolsIntegrationOnly) -extension ExitTest { - /// A type whose instances uniquely identify instances of ``ExitTest``. - public struct ID: Sendable, Equatable, Codable { /* ... */ } - - /// A value that uniquely identifies this instance. - public var id: ID { get set } - - /// Key paths representing results from within this exit test that should be - /// observed and returned to the caller. - /// - /// The testing library sets this property to match what was passed by the - /// developer to the `#expect(exitsWith:)` or `#require(exitsWith:)` macro. - /// If you are implementing an exit test handler, you can check the value of - /// this property to determine what information you need to preserve from your - /// child process. - /// - /// The value of this property always includes ``ExitTest/Result/statusAtExit`` - /// even if the test author does not specify it. - /// - /// Within a child process running an exit test, the value of this property is - /// otherwise unspecified. - public var observedValues: [any PartialKeyPath & Sendable] { get set } - - /// Call the exit test in the current process. - /// - /// This function invokes the closure originally passed to - /// `#expect(exitsWith:)` _in the current process_. That closure is expected - /// to terminate the process; if it does not, the testing library will - /// terminate the process as if its `main()` function returned naturally. - public consuming func callAsFunction() async -> Never - - /// Find the exit test function at the given source location. - /// - /// - Parameters: - /// - id: The unique identifier of the exit test to find. - /// - /// - Returns: The specified exit test function, or `nil` if no such exit test - /// could be found. - public static func find(identifiedBy id: ExitTest.ID) -> Self? - - /// A handler that is invoked when an exit test starts. - /// - /// - Parameters: - /// - exitTest: The exit test that is starting. - /// - /// - Returns: The result of the exit test including the condition under which - /// it exited. - /// - /// - Throws: Any error that prevents the normal invocation or execution of - /// the exit test. - /// - /// This handler is invoked when an exit test (i.e. a call to either - /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or - /// ``require(exitsWith:observing:_:sourceLocation:performing:)``) is started. - /// The handler is responsible for initializing a new child environment - /// (typically a child process) and running the exit test identified by - /// `sourceLocation` there. - /// - /// In the child environment, you can find the exit test again by calling - /// ``ExitTest/find(at:)`` and can run it by calling - /// ``ExitTest/callAsFunction()``. - /// - /// The parent environment should suspend until the results of the exit test - /// are available or the child environment is otherwise terminated. The parent - /// environment is then responsible for interpreting those results and - /// recording any issues that occur. - public typealias Handler = @Sendable (_ exitTest: borrowing ExitTest) async throws -> ExitTest.Result -} - -@_spi(ForToolsIntegrationOnly) -extension Configuration { - /// A handler that is invoked when an exit test starts. - /// - /// For an explanation of how this property is used, see ``ExitTest/Handler``. - /// - /// When using the `swift test` command from Swift Package Manager, this - /// property is pre-configured. Otherwise, the default value of this property - /// records an issue indicating that it has not been configured. - public var exitTestHandler: ExitTest.Handler { get set } -} -``` - -Any tools that use `swift build --build-tests`, `swift test`, or equivalent to -compile executables for testing will inherit the functionality provided for -`swift test` and do not need to implement their own exit test handlers. Tools -that directly compile test targets or otherwise do not leverage Swift Package -Manager will need to provide an implementation. - -## Future directions - -### Support for iOS, WASI, etc. - -The need for exit tests on other platforms is just as strong as it is on the -supported platforms (macOS, Linux, FreeBSD/OpenBSD, and Windows). These -platforms do not support spawning new processes, so a different mechanism for -running exit tests would be needed. - -Android _does_ have `posix_spawn()` and related API and may be able to use the -same implementation as Linux. Android support is an ongoing area of research for -Swift Testing's core team. - -### Recursive exit tests - -The technical constraints preventing recursive exit test invocation can be -resolved if there is a need to do so. However, we don't anticipate that this -constraint will be a serious issue for developers. - -### Support for passing state - -Arbitrary state is necessarily not preserved between the parent and child -processes, but there is little to prevent us from adding a variadic `arguments:` -argument and passing values whose types conform to `Codable`. - -The blocker right now is that there is no type information during macro -expansion, meaning that the testing library can emit the glue code to _encode_ -arguments, but does not know what types to use when _decoding_ those arguments. -If generic types were made available during macro expansion via the macro -expansion context, then it would be possible to synthesize the correct logic. - -Alternatively, if the language gained something akin to C++'s `decltype()`, we -could leverage closures' capture list syntax. Subjectively, capture lists ought -to be somewhat intuitive for developers in this context: - -```swift -let (lettuce, cheese, crema) = taco.addToppings() -await #expect(exitsWith: .failure) { [taco, plant = lettuce, cheese, crema] in - try taco.removeToppings(plant, cheese, crema) -} -``` - -### More nuanced support for throwing errors from exit test bodies - -Currently, if an error is thrown from an exit test without being caught, the -test behaves the same way a program does when an error is thrown from an -explicit or implicit `main() throws` function: the process terminates abnormally -and control returns to the test function that is awaiting the exit test: - -```swift -await #expect(exitsWith: .failure) { - throw TacoError.noTacosFound -} -``` - -If the test function is expecting `.failure`, this means the test passes. -Although this behavior is consistent with modelling an exit test as an -independent program (i.e. the exit test acts like its own `main()` function), it -may be surprising to test authors who aren't thinking about error handling. In -the future, we may want to offer a compile-time diagnostic if an error is thrown -from an exit test body without being caught, or offer a distinct exit condition -(i.e. `.errorNotCaught(_ error: Error & Codable)`) for these uncaught errors. -For error types that conform to `Codable`, we could offer rethrowing behavior, -but this is not possible for error types that cannot be sent across process -boundaries. - -### Exit-testing customized processes - -The current model of exit tests is that they run in approximately the same -environment as the test process by spawning a copy of the executable under test. -There is a very real use case for allowing testing other processes and -inspecting their output. In the future, we could provide API to spawn a process -with particular arguments and environment variables, then inspect its exit -condition and standard output/error streams: - -```swift -let result = try await #require( - executableAt: "/usr/bin/swift", - passing: ["build", "--package-path", ...], - environment: [:], - exitsWith: .success -) -#expect(result.standardOutputContent.contains("Build went well!").utf8) -``` - -We could also investigate explicitly integrating with [`Foundation.Process`](https://developer.apple.com/documentation/foundation/process) -or the proposed [`Foundation.Subprocess`](https://github.com/swiftlang/swift-foundation/blob/main/Proposals/0007-swift-subprocess.md) -as an alternative: - -```swift -let process = Process() -process.executableURL = URL(filePath: "/usr/bin/swift", directoryHint: .notDirectory) -process.arguments = ["build", "--package-path", ...] -let result = try await #require(process, exitsWith: .success) -#expect(result.standardOutputContent.contains("Build went well!").utf8) -``` - -## Alternatives considered - -- Doing nothing. - -- Marking exit tests using a trait rather than a new `#expect()` overload: - - ```swift - @Test(.exits(with: .failure)) - func `We only eat delicious tacos`() { - var taco = Taco() - taco.isDelicious = false - eat(taco) - } - ``` - - This syntax would require separate test functions for each exit test, while - reusing the same function for relatively concise tests may be preferable. - - It would also potentially conflict with parameterized tests, as it is not - possible to pass arbitrary parameters to the child process. It would be - necessary to teach the testing library's macro target about the - `.exits(with:)` trait so that it could produce a diagnostic when used with a - parameterized test function. - -- Inferring exit tests from test functions that return `Never`: - - ```swift - @Test func `No seafood for me, thanks!`() -> Never { - var taco = Taco() - taco.toppings.append(.shrimp) - eat(taco) - fatalError("Should not have eaten that!") - } - ``` - - There's a certain synergy in inferring that a test function that returns - `Never` must necessarily be a crasher and should be handled out of process. - However, this forces the test author to add a call to `fatalError()` or - similar in the event that the code under test does _not_ terminate, and there - is no obvious way to express that a specific exit code, signal, or other - condition is expected (as opposed to just "it exited".) - - We might want to support that sort of inference in the future (i.e. "don't run - this test in-process because it will terminate the test run"), but without - also inferring success or failure from the process' exit status. - -- Naming the macro something else such as: - - - `#exits(with:_:)`; - - `#exits(because:_:)`; - - `#expect(exitsBecause:_:)`; - - `#expect(terminatesBecause:_:)`; etc. - - While "with" is normally avoided in symbol names in Swift, it sometimes really - is the best preposition for the job. "Because", "due to", and others don't - sound "right" when the entire expression is read out loud. For example, you - probably wouldn't say "exits due to success" in English. - -- Combining `StatusAtExit` and `ExitTest.Condition` into a single type: - - ```swift - enum ExitCondition { - case failure // any failure - case exitCode(CInt) - case signal(CInt) - } - ``` - - This simplified the set of types used for exit tests, but made comparing two - exit conditions complicated and necessitated a `==` operator that did not - satisfy the requirements of the `Equatable` protocol. - -- Naming `StatusAtExit` something else such as: - - - `ExitStatus`, which could be too easily confusable with exit _codes_ such as - `EXIT_SUCCESS`; - - `ProcessStatus`, but we don't say "process" in our API surface elsewhere; - - `Status`, which is too generic, - - `ExitReason`, but "status" is a more widely-used term of art for this - concept; or - - `terminationStatus` (which Foundation to represent approximately the same - concept), but we don't use "termination" in Swift Testing's API anywhere. - - I settled on `StatusAtExit` because it was distinct and makes it clear that it - represents the status of a process _at exit time_. `ExitStatus` could be - interpreted as the status of the exit itself, i.e.: - - ```swift - enum ExitStatus { - case running - case suspended - case exiting - case exited - } - ``` - -- Changing the implementation of `precondition()`, `fatalError()`, etc. in the - standard library so that they do not terminate the current process while - testing, thus removing the need to spawn a child process for an exit test. - - Most of the functions in this family return `Never`, and changing their return - types would be ABI-breaking (as well as a pessimization in production code.) - Even if we did modify these functions in the Swift standard library, other - ways to terminate the process exist and would not be covered: - - - Calling the C standard function `exit()`; - - Throwing an uncaught Objective-C or C++ exception; - - Sending a signal to the process; or - - Misusing memory (e.g. trying to dereference a null pointer.) - - Modifying the C or C++ standard library, or modifying the Objective-C runtime, - would be well beyond the scope of this proposal. - -## Acknowledgments - -Many thanks to the XCTest and Swift Testing team. Thanks to @compnerd for his -help with the Windows implementation. Thanks to my colleagues Coops, -Danny N., David R., Drew Y., and Robert K. at Apple for -their help with the nuances of crash reporting on macOS. From f48b44e3841a78858294301adb0b3e0610637566 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Mon, 3 Mar 2025 17:18:45 -0500 Subject: [PATCH 03/19] Try to work around compiler crash --- Sources/Testing/Running/Configuration.swift | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Sources/Testing/Running/Configuration.swift b/Sources/Testing/Running/Configuration.swift index fd2713bbc..3348d6994 100644 --- a/Sources/Testing/Running/Configuration.swift +++ b/Sources/Testing/Running/Configuration.swift @@ -217,7 +217,7 @@ public struct Configuration: Sendable { /// When using the `swift test` command from Swift Package Manager, this /// property is pre-configured. Otherwise, the default value of this property /// records an issue indicating that it has not been configured. - public var exitTestHandler: ExitTest.Handler = { _ in + public var exitTestHandler: ExitTest.Handler = { exitTest in throw SystemError(description: "Exit test support has not been implemented by the current testing infrastructure.") } #endif From 76eb5345732935d36c59c1f0136d990c47d62c34 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Mon, 24 Mar 2025 15:53:12 -0400 Subject: [PATCH 04/19] Create an Exit Testing DocC article --- .../Expectations/Expectation+Macro.swift | 142 ------------------ Sources/Testing/Testing.docc/Expectations.md | 1 + 2 files changed, 1 insertion(+), 142 deletions(-) diff --git a/Sources/Testing/Expectations/Expectation+Macro.swift b/Sources/Testing/Expectations/Expectation+Macro.swift index 8baa1de1e..c4d89e262 100644 --- a/Sources/Testing/Expectations/Expectation+Macro.swift +++ b/Sources/Testing/Expectations/Expectation+Macro.swift @@ -511,78 +511,6 @@ public macro require( /// } /// ``` /// -/// - Note: A call to this expectation macro is called an "exit test." -/// -/// ## How exit tests are run -/// -/// When an exit test is performed at runtime, the testing library starts a new -/// process with the same executable as the current process. The current task is -/// then suspended (as with `await`) and waits for the child process to -/// terminate. `expression` is not called in the parent process. -/// -/// Meanwhile, in the child process, `expression` is called directly. To ensure -/// a clean environment for execution, it is not called within the context of -/// the original test. If `expression` does not terminate the child process, the -/// process is terminated automatically as if the main function of the child -/// process were allowed to return naturally. If an error is thrown from -/// `expression`, it is handed as if the error were thrown from `main()` and the -/// process is terminated. -/// -/// Once the child process terminates, the parent process resumes and compares -/// its exit status against `expectedExitCondition`. If they match, the exit -/// test has passed; otherwise, it has failed and an issue is recorded. -/// -/// ## Child process output -/// -/// By default, the child process is configured without a standard output or -/// standard error stream. If your test needs to review the content of either of -/// these streams, you can pass its key path in the `observedValues` argument: -/// -/// ```swift -/// let result = await #expect( -/// exitsWith: .failure, -/// observing: [\.standardOutputContent] -/// ) { -/// print("Goodbye, world!") -/// fatalError() -/// } -/// if let result { -/// #expect(result.standardOutputContent.contains(UInt8(ascii: "G"))) -/// } -/// ``` -/// -/// - Note: The content of the standard output and standard error streams may -/// contain any arbitrary sequence of bytes, including sequences that are not -/// valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). -/// These streams are globally accessible within the child process, and any -/// code running in an exit test may write to it including the operating -/// system and any third-party dependencies you have declared in your package. -/// -/// The actual exit condition of the child process is always reported by the -/// testing library even if you do not specify it in `observedValues`. -/// -/// ## Runtime constraints -/// -/// Exit tests cannot capture any state originating in the parent process or -/// from the enclosing lexical context. For example, the following exit test -/// will fail to compile because it captures an argument to the enclosing -/// parameterized test: -/// -/// ```swift -/// @Test(arguments: 100 ..< 200) -/// func sellIceCreamCones(count: Int) async { -/// await #expect(exitsWith: .failure) { -/// precondition( -/// count < 10, // ERROR: A C function pointer cannot be formed from a -/// // closure that captures context -/// "Too many ice cream cones" -/// ) -/// } -/// } -/// ``` -/// -/// An exit test cannot run within another exit test. -/// /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } @@ -628,76 +556,6 @@ public macro require( /// } /// ``` /// -/// - Note: A call to this expectation macro is called an "exit test." -/// -/// ## How exit tests are run -/// -/// When an exit test is performed at runtime, the testing library starts a new -/// process with the same executable as the current process. The current task is -/// then suspended (as with `await`) and waits for the child process to -/// terminate. `expression` is not called in the parent process. -/// -/// Meanwhile, in the child process, `expression` is called directly. To ensure -/// a clean environment for execution, it is not called within the context of -/// the original test. If `expression` does not terminate the child process, the -/// process is terminated automatically as if the main function of the child -/// process were allowed to return naturally. If an error is thrown from -/// `expression`, it is handed as if the error were thrown from `main()` and the -/// process is terminated. -/// -/// Once the child process terminates, the parent process resumes and compares -/// its exit status against `expectedExitCondition`. If they match, the exit -/// test has passed; otherwise, it has failed and an issue is recorded. -/// -/// ## Child process output -/// -/// By default, the child process is configured without a standard output or -/// standard error stream. If your test needs to review the content of either of -/// these streams, you can pass its key path in the `observedValues` argument: -/// -/// ```swift -/// let result = try await #require( -/// exitsWith: .failure, -/// observing: [\.standardOutputContent] -/// ) { -/// print("Goodbye, world!") -/// fatalError() -/// } -/// #expect(result.standardOutputContent.contains(UInt8(ascii: "G"))) -/// ``` -/// -/// - Note: The content of the standard output and standard error streams may -/// contain any arbitrary sequence of bytes, including sequences that are not -/// valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). -/// These streams are globally accessible within the child process, and any -/// code running in an exit test may write to it including the operating -/// system and any third-party dependencies you have declared in your package. -/// -/// The actual exit condition of the child process is always reported by the -/// testing library even if you do not specify it in `observedValues`. -/// -/// ## Runtime constraints -/// -/// Exit tests cannot capture any state originating in the parent process or -/// from the enclosing lexical context. For example, the following exit test -/// will fail to compile because it captures an argument to the enclosing -/// parameterized test: -/// -/// ```swift -/// @Test(arguments: 100 ..< 200) -/// func sellIceCreamCones(count: Int) async throws { -/// try await #require(exitsWith: .failure) { -/// precondition( -/// count < 10, // ERROR: A C function pointer cannot be formed from a -/// // closure that captures context -/// "Too many ice cream cones" -/// ) -/// } -/// } -/// ``` -/// -/// An exit test cannot run within another exit test. -/// /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } diff --git a/Sources/Testing/Testing.docc/Expectations.md b/Sources/Testing/Testing.docc/Expectations.md index ce451d9d9..dea1e5d69 100644 --- a/Sources/Testing/Testing.docc/Expectations.md +++ b/Sources/Testing/Testing.docc/Expectations.md @@ -74,6 +74,7 @@ the test when the code doesn't satisfy a requirement, use ### Checking how processes exit +- - ``expect(exitsWith:observing:_:sourceLocation:performing:)`` - ``require(exitsWith:observing:_:sourceLocation:performing:)`` - ``ExitTest`` From 1c1a67a69975183bce6a0e192003fb297ff83c61 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Mon, 24 Mar 2025 17:25:18 -0400 Subject: [PATCH 05/19] Create an Exit Testing DocC article (no rly) --- Sources/Testing/Testing.docc/exit-testing.md | 136 +++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 Sources/Testing/Testing.docc/exit-testing.md diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md new file mode 100644 index 000000000..d545f8a33 --- /dev/null +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -0,0 +1,136 @@ +# Exit testing + + + +Use exit tests to test functionality that may cause a test process to terminate. + +## Overview + +Your code may contain calls to [`precondition()`](https://developer.apple.com/documentation/swift/precondition(_:_:file:line:)), +[`fatalError()`](https://developer.apple.com/documentation/swift/fatalerror(_:file:line:)), +or other functions that may cause the current process to terminate. For example: + +```swift +extension Customer { + func eat(_ food: consuming some Food) { + precondition(food.isDelicious, "Tasty food only!") + precondition(food.isNutritious, "Healthy food only!") + ... + } +} +``` + +In this function, if `food.isDelicious` or `food.isNutritious` is `false`, the +precondition will fail and Swift will terminate the process. You can write an +exit test to validate preconditions like the ones above and to make sure that +your functions correctly catch invalid inputs. + +## Create an exit test + +To create an exit test, call either the ``expect(exitsWith:observing:_:sourceLocation:performing:)`` +or the ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro and +pass a closure containing the code that may terminate the process along with the +expected result of calling that code (success, failure, or a specific exit code +or signal): + +```swift +@Test func `Customer won't eat food unless it's delicious`() async { + let result = await #expect(exitsWith: .failure) { + var food = ... + food.isDelicious = false + Customer.current.eat(food) + } +} +``` + +When an exit test is performed at runtime, the testing library starts a new +process with the same executable as the current process. The current task is +then suspended (as with `await`) and waits for the child process to terminate. +`expression` is not called in the parent process. + +Meanwhile, in the child process, the closure you passed to ``expect(exitsWith:observing:_:sourceLocation:performing:)`` +or to ``require(exitsWith:observing:_:sourceLocation:performing:)`` is called +directly. To ensure a clean environment for execution, the closure is not called +within the context of the original test. Instead, it is treated as if it were +the `main()` function of the child process. + +If the closure returns before the child process terminates, it is terminated +automatically (as if the main function of the child process were allowed to +return naturally.) If an error is thrown from the closure, it is handed as if +the error were thrown from `main()` and the process is terminated. + +Once the child process terminates, the parent process resumes and compares its +exit status against the expected exit condition you passed. If they match, the +exit test has passed; otherwise, it has failed and an issue is recorded. + +## Gather output from the child process + +By default, the child process is configured without a standard output or +standard error stream. If your test needs to review the content of either of +these streams, you can pass its key path in the `observedValues` argument: + +```swift +extension Customer { + func eat(_ food: consuming some Food) { + print("Let's see if I want to eat \(food)...") + precondition(food.isDelicious, "Tasty food only!") + precondition(food.isNutritious, "Healthy food only!") + ... + } +} + +@Test func `Customer won't eat food unless it's delicious`() async { + let result = await #expect( + exitsWith: .failure, + observing: [\.standardOutputContent] + ) { + var food = ... + food.isDelicious = false + Customer.current.eat(food) + } + if let result { + #expect(result.standardOutputContent.contains(UInt8(ascii: "L"))) + } +} +``` + +- Note: The content of the standard output and standard error streams may + contain any arbitrary sequence of bytes, including sequences that are not + valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). + These streams are globally accessible within the child process, and any code + running in an exit test may write to it including the operating system and any + third-party dependencies you have declared in your package. + +The actual exit condition of the child process is always reported by the testing +library even if you do not specify it in `observedValues`. + +## Constraints on exit tests + +### State cannot be captured + +Exit tests cannot capture any state originating in the parent process or from +the enclosing lexical context. For example, the following exit test will fail to +compile because it captures a variable declared outside the exit test itself: + +```swift +@Test func `Customer won't eat food unless it's nutritious`() async { + let isNutritious = false + await #expect(exitsWith: .failure) { + var food = ... + food.isNutritious = isNutritious // ❌ ERROR: trying to capture state here + Customer.current.eat(food) + } +} +``` + +### Exit tests cannot be nested + +An exit test cannot run within another exit test. From 0304bfc256013587b8379328041a2479bbc9439e Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 25 Mar 2025 13:06:21 -0400 Subject: [PATCH 06/19] Avoid the word 'exit' in documentation, add more details about exit conditions --- Sources/Testing/Testing.docc/exit-testing.md | 65 ++++++++++++-------- 1 file changed, 38 insertions(+), 27 deletions(-) diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index d545f8a33..f10dd6e1c 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -10,13 +10,13 @@ See https://swift.org/LICENSE.txt for license information See https://swift.org/CONTRIBUTORS.txt for Swift project authors --> -Use exit tests to test functionality that may cause a test process to terminate. +Use exit tests to test functionality that may cause a test process to exit. ## Overview Your code may contain calls to [`precondition()`](https://developer.apple.com/documentation/swift/precondition(_:_:file:line:)), [`fatalError()`](https://developer.apple.com/documentation/swift/fatalerror(_:file:line:)), -or other functions that may cause the current process to terminate. For example: +or other functions that may cause the current process to exit. For example: ```swift extension Customer { @@ -29,17 +29,14 @@ extension Customer { ``` In this function, if `food.isDelicious` or `food.isNutritious` is `false`, the -precondition will fail and Swift will terminate the process. You can write an -exit test to validate preconditions like the ones above and to make sure that +precondition will fail and Swift will force the process to exit. You can write +an exit test to validate preconditions like the ones above and to make sure that your functions correctly catch invalid inputs. ## Create an exit test To create an exit test, call either the ``expect(exitsWith:observing:_:sourceLocation:performing:)`` -or the ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro and -pass a closure containing the code that may terminate the process along with the -expected result of calling that code (success, failure, or a specific exit code -or signal): +or the ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro: ```swift @Test func `Customer won't eat food unless it's delicious`() async { @@ -51,25 +48,39 @@ or signal): } ``` -When an exit test is performed at runtime, the testing library starts a new -process with the same executable as the current process. The current task is -then suspended (as with `await`) and waits for the child process to terminate. -`expression` is not called in the parent process. - -Meanwhile, in the child process, the closure you passed to ``expect(exitsWith:observing:_:sourceLocation:performing:)`` -or to ``require(exitsWith:observing:_:sourceLocation:performing:)`` is called -directly. To ensure a clean environment for execution, the closure is not called -within the context of the original test. Instead, it is treated as if it were -the `main()` function of the child process. - -If the closure returns before the child process terminates, it is terminated -automatically (as if the main function of the child process were allowed to -return naturally.) If an error is thrown from the closure, it is handed as if -the error were thrown from `main()` and the process is terminated. - -Once the child process terminates, the parent process resumes and compares its -exit status against the expected exit condition you passed. If they match, the -exit test has passed; otherwise, it has failed and an issue is recorded. +The closure or function reference you pass to these macros is the body of the +exit test. When an exit test is performed at runtime, the testing library starts +a new process with the same executable as the current process. The current task +is then suspended (as with `await`) and waits for the child process to +exit. The exit test's body is not called in the parent process. + +Meanwhile, in the child process, the body is called directly. To ensure a clean +environment for execution, the body is not called within the context of the +original test. Instead, it is treated as if it were the `main()` function of the +child process. + +If the body returns before the child process exits, it is allowed to return and +the process exits naturally. If an error is thrown from the body, it is handled +as if the error were thrown from `main()` and the process is forced to exit. + +## Specify an exit condition + +When you create an exit test, you must specify how you expect the child process +will exit by passing an instance of ``ExitTest/Condition``: + +- If the exit test's body should run to completion or exit normally (for + example, by calling `exit(EXIT_SUCCESS)` from the C standard library), pass + ``ExitTest/Condition/success``. +- If the body will cause the child process to exit with some failure, but the + exact status reported by the system is not important, pass + ``ExitTest/Condition/failure``. +- If you need to check for a specific exit code or signal, pass + ``ExitTest/Condition/exitCode(_:)`` or ``ExitTest/Condition/signal(_:)``. + +When the child process exits, the parent process resumes and compares the exit +status of the child process against the expected exit condition you passed. If +they match, the exit test has passed; otherwise, it has failed and the testing +library records an issue. ## Gather output from the child process From fa5aeed593246facd34270d85341754aa032239d Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 25 Mar 2025 13:43:43 -0400 Subject: [PATCH 07/19] Nest subtopics of article correctly, note platform availability --- .../Testing/Testing.docc/OrganizingTests.md | 2 +- Sources/Testing/Testing.docc/exit-testing.md | 19 +++++++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/Sources/Testing/Testing.docc/OrganizingTests.md b/Sources/Testing/Testing.docc/OrganizingTests.md index f2b577eb4..3464db4ae 100644 --- a/Sources/Testing/Testing.docc/OrganizingTests.md +++ b/Sources/Testing/Testing.docc/OrganizingTests.md @@ -124,7 +124,7 @@ struct MenuTests { The compiler emits an error when presented with a test suite that doesn't meet this requirement. -### Test suite types must always be available +#### Test suite types must always be available Although `@available` can be applied to a test function to limit its availability at runtime, a test suite type (and any types that contain it) must diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index f10dd6e1c..b12c4704b 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -10,6 +10,11 @@ See https://swift.org/LICENSE.txt for license information See https://swift.org/CONTRIBUTORS.txt for Swift project authors --> +@Metadata { + @Available(macOS, introduced: 10.15) + @Available(Swift, introduced: 6.2) +} + Use exit tests to test functionality that may cause a test process to exit. ## Overview @@ -33,7 +38,9 @@ precondition will fail and Swift will force the process to exit. You can write an exit test to validate preconditions like the ones above and to make sure that your functions correctly catch invalid inputs. -## Create an exit test +- Note: Exit tests are available on macOS, Linux, FreeBSD, OpenBSD, and Windows. + +### Create an exit test To create an exit test, call either the ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or the ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro: @@ -63,7 +70,7 @@ If the body returns before the child process exits, it is allowed to return and the process exits naturally. If an error is thrown from the body, it is handled as if the error were thrown from `main()` and the process is forced to exit. -## Specify an exit condition +### Specify an exit condition When you create an exit test, you must specify how you expect the child process will exit by passing an instance of ``ExitTest/Condition``: @@ -82,7 +89,7 @@ status of the child process against the expected exit condition you passed. If they match, the exit test has passed; otherwise, it has failed and the testing library records an issue. -## Gather output from the child process +### Gather output from the child process By default, the child process is configured without a standard output or standard error stream. If your test needs to review the content of either of @@ -123,9 +130,9 @@ extension Customer { The actual exit condition of the child process is always reported by the testing library even if you do not specify it in `observedValues`. -## Constraints on exit tests +### Constraints on exit tests -### State cannot be captured +#### State cannot be captured Exit tests cannot capture any state originating in the parent process or from the enclosing lexical context. For example, the following exit test will fail to @@ -142,6 +149,6 @@ compile because it captures a variable declared outside the exit test itself: } ``` -### Exit tests cannot be nested +#### Exit tests cannot be nested An exit test cannot run within another exit test. From cfbd85c96ec7f19d732f9c5d934127db63a2549b Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 25 Mar 2025 13:56:27 -0400 Subject: [PATCH 08/19] Remove macOS availability override in article --- Sources/Testing/Testing.docc/exit-testing.md | 1 - 1 file changed, 1 deletion(-) diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index b12c4704b..76434cd89 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -11,7 +11,6 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors --> @Metadata { - @Available(macOS, introduced: 10.15) @Available(Swift, introduced: 6.2) } From 8b125226b88f3e5f165b09f49e6dee1f1f6e74d9 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 25 Mar 2025 14:35:59 -0400 Subject: [PATCH 09/19] Refine the text around statusAtExit always being set --- Sources/Testing/Testing.docc/exit-testing.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index 76434cd89..d5fc53fb8 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -92,7 +92,8 @@ library records an issue. By default, the child process is configured without a standard output or standard error stream. If your test needs to review the content of either of -these streams, you can pass its key path in the `observedValues` argument: +these streams, you can pass its key path to ``expect(exitsWith:observing:_:sourceLocation:performing:)`` +or ``require(exitsWith:observing:_:sourceLocation:performing:)``: ```swift extension Customer { @@ -126,8 +127,9 @@ extension Customer { running in an exit test may write to it including the operating system and any third-party dependencies you have declared in your package. -The actual exit condition of the child process is always reported by the testing -library even if you do not specify it in `observedValues`. +The testing library always sets ``ExitTest/Result/statusAtExit`` to the actual +exit status of the child process (as reported by the system) even if you do not +pass it. ### Constraints on exit tests From 7507cf04c1cc54835a6236f509f8572df14cbb33 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 25 Mar 2025 14:49:19 -0400 Subject: [PATCH 10/19] Remove cppreference.com links from public docs (not actually an official language standard) --- .../ExitTests/ExitTest.Condition.swift | 24 ++++++++++++------- Sources/Testing/ExitTests/StatusAtExit.swift | 20 ++++++++++++---- 2 files changed, 30 insertions(+), 14 deletions(-) diff --git a/Sources/Testing/ExitTests/ExitTest.Condition.swift b/Sources/Testing/ExitTests/ExitTest.Condition.swift index 7d824c43d..b3789a517 100644 --- a/Sources/Testing/ExitTests/ExitTest.Condition.swift +++ b/Sources/Testing/ExitTests/ExitTest.Condition.swift @@ -60,10 +60,6 @@ extension ExitTest.Condition { /// A condition that matches when a process terminates successfully with exit /// code `EXIT_SUCCESS`. /// - /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), - /// `EXIT_SUCCESS` and `EXIT_FAILURE` as well as `0` (as a synonym for - /// `EXIT_SUCCESS`.) - /// /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } @@ -102,9 +98,9 @@ extension ExitTest.Condition { /// - Parameters: /// - exitCode: The exit code yielded by the process. /// - /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), - /// `EXIT_SUCCESS` and `EXIT_FAILURE`. Platforms may additionally define their - /// own non-standard exit codes: + /// The C programming language defines two standard exit codes, `EXIT_SUCCESS` + /// and `EXIT_FAILURE`. Platforms may additionally define their own + /// non-standard exit codes: /// /// | Platform | Header | /// |-|-| @@ -114,6 +110,11 @@ extension ExitTest.Condition { /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | /// + /// @Comment { + /// See https://en.cppreference.com/w/c/program/EXIT_status for more + /// information about exit codes defined by the C standard. + /// } + /// /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by /// the process is yielded to the parent process. Linux and other POSIX-like /// systems may only reliably report the low unsigned 8 bits (0–255) of @@ -136,8 +137,8 @@ extension ExitTest.Condition { /// - Parameters: /// - signal: The signal that terminated the process. /// - /// The C programming language defines a number of [standard signals](https://en.cppreference.com/w/c/program/SIG_types). - /// Platforms may additionally define their own non-standard signal codes: + /// The C programming language defines a number of standard signals. Platforms + /// may additionally define their own non-standard signal codes: /// /// | Platform | Header | /// |-|-| @@ -147,6 +148,11 @@ extension ExitTest.Condition { /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | /// + /// @Comment { + /// See https://en.cppreference.com/w/c/program/SIG_types for more + /// information about signals defined by the C standard. + /// } + /// /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } diff --git a/Sources/Testing/ExitTests/StatusAtExit.swift b/Sources/Testing/ExitTests/StatusAtExit.swift index 0f24a5918..9b7e1b782 100644 --- a/Sources/Testing/ExitTests/StatusAtExit.swift +++ b/Sources/Testing/ExitTests/StatusAtExit.swift @@ -31,9 +31,9 @@ public enum StatusAtExit: Sendable { /// - Parameters: /// - exitCode: The exit code yielded by the process. /// - /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), - /// `EXIT_SUCCESS` and `EXIT_FAILURE`. Platforms may additionally define their - /// own non-standard exit codes: + /// The C programming language defines two standard exit codes, `EXIT_SUCCESS` + /// and `EXIT_FAILURE`. Platforms may additionally define their own + /// non-standard exit codes: /// /// | Platform | Header | /// |-|-| @@ -43,6 +43,11 @@ public enum StatusAtExit: Sendable { /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | /// + /// @Comment { + /// See https://en.cppreference.com/w/c/program/EXIT_status for more + /// information about exit codes defined by the C standard. + /// } + /// /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by /// the process is yielded to the parent process. Linux and other POSIX-like /// systems may only reliably report the low unsigned 8 bits (0–255) of @@ -58,8 +63,8 @@ public enum StatusAtExit: Sendable { /// - Parameters: /// - signal: The signal that terminated the process. /// - /// The C programming language defines a number of [standard signals](https://en.cppreference.com/w/c/program/SIG_types). - /// Platforms may additionally define their own non-standard signal codes: + /// The C programming language defines a number of standard signals. Platforms + /// may additionally define their own non-standard signal codes: /// /// | Platform | Header | /// |-|-| @@ -69,6 +74,11 @@ public enum StatusAtExit: Sendable { /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | /// + /// @Comment { + /// See https://en.cppreference.com/w/c/program/SIG_types for more + /// information about signals defined by the C standard. + /// } + /// /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } From af9776c798b58422e2a8a0089c8f039832474619 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 25 Mar 2025 16:09:02 -0400 Subject: [PATCH 11/19] Update Linux manpage links to kernel.org --- Sources/Testing/ExitTests/ExitTest.Condition.swift | 4 ++-- Sources/Testing/ExitTests/StatusAtExit.swift | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Sources/Testing/ExitTests/ExitTest.Condition.swift b/Sources/Testing/ExitTests/ExitTest.Condition.swift index b3789a517..b8c3a8980 100644 --- a/Sources/Testing/ExitTests/ExitTest.Condition.swift +++ b/Sources/Testing/ExitTests/ExitTest.Condition.swift @@ -105,7 +105,7 @@ extension ExitTest.Condition { /// | Platform | Header | /// |-|-| /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/_Exit.3.html), [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysexits.3.html) | - /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Exit-Status.html), `` | + /// | Linux | [``](https://www.kernel.org/doc/man-pages/online/pages/man3/exit.3.html), [``](https://www.kernel.org/doc/man-pages/online/pages/man3/sysexits.h.3head.html) | /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?exit(3)), [``](https://man.freebsd.org/cgi/man.cgi?sysexits(3)) | /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | @@ -143,7 +143,7 @@ extension ExitTest.Condition { /// | Platform | Header | /// |-|-| /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/signal.3.html) | - /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Standard-Signals.html) | + /// | Linux | [``](https://www.kernel.org/doc/man-pages/online/pages/man7/signal.7.html) | /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | diff --git a/Sources/Testing/ExitTests/StatusAtExit.swift b/Sources/Testing/ExitTests/StatusAtExit.swift index 9b7e1b782..352c86b47 100644 --- a/Sources/Testing/ExitTests/StatusAtExit.swift +++ b/Sources/Testing/ExitTests/StatusAtExit.swift @@ -38,7 +38,7 @@ public enum StatusAtExit: Sendable { /// | Platform | Header | /// |-|-| /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/_Exit.3.html), [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysexits.3.html) | - /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Exit-Status.html), `` | + /// | Linux | [``](https://www.kernel.org/doc/man-pages/online/pages/man3/exit.3.html), [``](https://www.kernel.org/doc/man-pages/online/pages/man3/sysexits.h.3head.html) | /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?exit(3)), [``](https://man.freebsd.org/cgi/man.cgi?sysexits(3)) | /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | @@ -69,7 +69,7 @@ public enum StatusAtExit: Sendable { /// | Platform | Header | /// |-|-| /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/signal.3.html) | - /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Standard-Signals.html) | + /// | Linux | [``](https://www.kernel.org/doc/man-pages/online/pages/man7/signal.7.html) | /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | From 090fdea24650590ba0da607fd81e843789dc341c Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Wed, 26 Mar 2025 09:44:49 -0400 Subject: [PATCH 12/19] Incorporate editioral feedback --- .../ExitTests/ExitTest.Condition.swift | 20 ++-- .../Testing/ExitTests/ExitTest.Result.swift | 5 +- Sources/Testing/ExitTests/StatusAtExit.swift | 12 +- Sources/Testing/Testing.docc/exit-testing.md | 109 +++++++++--------- 4 files changed, 71 insertions(+), 75 deletions(-) diff --git a/Sources/Testing/ExitTests/ExitTest.Condition.swift b/Sources/Testing/ExitTests/ExitTest.Condition.swift index b8c3a8980..3890a8446 100644 --- a/Sources/Testing/ExitTests/ExitTest.Condition.swift +++ b/Sources/Testing/ExitTests/ExitTest.Condition.swift @@ -57,8 +57,9 @@ extension ExitTest { @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif extension ExitTest.Condition { - /// A condition that matches when a process terminates successfully with exit - /// code `EXIT_SUCCESS`. + /// A condition that matches when a process exits normally. + /// + /// This condition matches the exit code `EXIT_SUCCESS`. /// /// @Metadata { /// @Available(Swift, introduced: 6.2) @@ -75,8 +76,10 @@ extension ExitTest.Condition { #endif } - /// A condition that matches when a process terminates abnormally with any - /// exit code other than `EXIT_SUCCESS` or with any signal. + /// A condition that matches when a process exits abnormally + /// + /// This condition matches any exit code other than `EXIT_SUCCESS` or any + /// signal that causes the process to exit. /// /// @Metadata { /// @Available(Swift, introduced: 6.2) @@ -96,7 +99,7 @@ extension ExitTest.Condition { /// exit code. /// /// - Parameters: - /// - exitCode: The exit code yielded by the process. + /// - exitCode: The exit code reported by the process. /// /// The C programming language defines two standard exit codes, `EXIT_SUCCESS` /// and `EXIT_FAILURE`. Platforms may additionally define their own @@ -116,7 +119,7 @@ extension ExitTest.Condition { /// } /// /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by - /// the process is yielded to the parent process. Linux and other POSIX-like + /// the process is reported to the parent process. Linux and other POSIX-like /// systems may only reliably report the low unsigned 8 bits (0–255) of /// the exit code. /// @@ -131,11 +134,10 @@ extension ExitTest.Condition { #endif } - /// Creates a condition that matches when a process terminates with a given - /// signal. + /// Creates a condition that matches when a process exits with a given signal. /// /// - Parameters: - /// - signal: The signal that terminated the process. + /// - signal: The signal that caused the process to exit. /// /// The C programming language defines a number of standard signals. Platforms /// may additionally define their own non-standard signal codes: diff --git a/Sources/Testing/ExitTests/ExitTest.Result.swift b/Sources/Testing/ExitTests/ExitTest.Result.swift index edab995d9..df4b48a88 100644 --- a/Sources/Testing/ExitTests/ExitTest.Result.swift +++ b/Sources/Testing/ExitTests/ExitTest.Result.swift @@ -23,10 +23,7 @@ extension ExitTest { /// @Available(Swift, introduced: 6.2) /// } public struct Result: Sendable { - /// The status of the process hosting the exit test at the time it exits. - /// - /// When the exit test passes, the value of this property is equal to the - /// exit status reported by the process that hosted the exit test. + /// The exit status reported by the process hosting the exit test. /// /// @Metadata { /// @Available(Swift, introduced: 6.2) diff --git a/Sources/Testing/ExitTests/StatusAtExit.swift b/Sources/Testing/ExitTests/StatusAtExit.swift index 352c86b47..a54646de8 100644 --- a/Sources/Testing/ExitTests/StatusAtExit.swift +++ b/Sources/Testing/ExitTests/StatusAtExit.swift @@ -10,7 +10,7 @@ private import _TestingInternals -/// An enumeration describing possible status a process will yield on exit. +/// An enumeration describing possible status a process will report on exit. /// /// You can convert an instance of this type to an instance of /// ``ExitTest/Condition`` using ``ExitTest/Condition/init(_:)``. That value @@ -26,10 +26,10 @@ private import _TestingInternals @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif public enum StatusAtExit: Sendable { - /// The process terminated with the given exit code. + /// The process exited with the given exit code. /// /// - Parameters: - /// - exitCode: The exit code yielded by the process. + /// - exitCode: The exit code reported by the process. /// /// The C programming language defines two standard exit codes, `EXIT_SUCCESS` /// and `EXIT_FAILURE`. Platforms may additionally define their own @@ -49,7 +49,7 @@ public enum StatusAtExit: Sendable { /// } /// /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by - /// the process is yielded to the parent process. Linux and other POSIX-like + /// the process is reported to the parent process. Linux and other POSIX-like /// systems may only reliably report the low unsigned 8 bits (0–255) of /// the exit code. /// @@ -58,10 +58,10 @@ public enum StatusAtExit: Sendable { /// } case exitCode(_ exitCode: CInt) - /// The process terminated with the given signal. + /// The process exited with the given signal. /// /// - Parameters: - /// - signal: The signal that terminated the process. + /// - signal: The signal that caused the process to exit. /// /// The C programming language defines a number of standard signals. Platforms /// may additionally define their own non-standard signal codes: diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index d5fc53fb8..b79282eb2 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -14,13 +14,13 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors @Available(Swift, introduced: 6.2) } -Use exit tests to test functionality that may cause a test process to exit. +Use exit tests to test functionality that might cause a test process to exit. ## Overview -Your code may contain calls to [`precondition()`](https://developer.apple.com/documentation/swift/precondition(_:_:file:line:)), +Your code might contain calls to [`precondition()`](https://developer.apple.com/documentation/swift/precondition(_:_:file:line:)), [`fatalError()`](https://developer.apple.com/documentation/swift/fatalerror(_:file:line:)), -or other functions that may cause the current process to exit. For example: +or other functions that can cause the current process to exit. For example: ```swift extension Customer { @@ -33,9 +33,9 @@ extension Customer { ``` In this function, if `food.isDelicious` or `food.isNutritious` is `false`, the -precondition will fail and Swift will force the process to exit. You can write -an exit test to validate preconditions like the ones above and to make sure that -your functions correctly catch invalid inputs. +precondition fails and Swift forces the process to exit. You can write an exit +test to validate preconditions like the ones above and to make sure that your +functions correctly catch invalid inputs. - Note: Exit tests are available on macOS, Linux, FreeBSD, OpenBSD, and Windows. @@ -54,30 +54,44 @@ or the ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro: } ``` -The closure or function reference you pass to these macros is the body of the -exit test. When an exit test is performed at runtime, the testing library starts -a new process with the same executable as the current process. The current task -is then suspended (as with `await`) and waits for the child process to -exit. The exit test's body is not called in the parent process. - -Meanwhile, in the child process, the body is called directly. To ensure a clean -environment for execution, the body is not called within the context of the -original test. Instead, it is treated as if it were the `main()` function of the -child process. +The closure or function reference you pass to the macro is the body of the exit +test. When an exit test is performed at runtime, the testing library starts a +new process with the same executable as the current process. The current task is +then suspended (as with `await`) and waits for the child process to exit. + +The parent process never calls the body of the exit test. Instead, the child +process treats the body of the exit test as its `main()` function and calls it +directly. + +- Note: Because the body acts as the `main()` function of a new process, it + can't capture any state originating in the parent process or from its lexical + context. For example, the following exit test will fail to compile because it + captures a variable declared outside the exit test itself: + + ```swift + @Test func `Customer won't eat food unless it's nutritious`() async { + let isNutritious = false + await #expect(exitsWith: .failure) { + var food = ... + food.isNutritious = isNutritious // ❌ ERROR: trying to capture state here + Customer.current.eat(food) + } + } + ``` -If the body returns before the child process exits, it is allowed to return and -the process exits naturally. If an error is thrown from the body, it is handled -as if the error were thrown from `main()` and the process is forced to exit. +If the body returns before the child process exits, the process exits as if +`main()` returned normally. If the body throws an error, Swift handles it as if +it were thrown from `main()` and forces the process to exit abnormally. ### Specify an exit condition -When you create an exit test, you must specify how you expect the child process -will exit by passing an instance of ``ExitTest/Condition``: +When you create an exit test, specify how you expect the child process exits by +passing an instance of ``ExitTest/Condition``: -- If the exit test's body should run to completion or exit normally (for - example, by calling `exit(EXIT_SUCCESS)` from the C standard library), pass - ``ExitTest/Condition/success``. -- If the body will cause the child process to exit with some failure, but the +- If you expect the exit test's body to run to completion or exit normally (for + example, by calling [`exit(EXIT_SUCCESS)`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/exit.3.html) + from the C standard library), pass ``ExitTest/Condition/success``. +- If you expect the body to cause the child process to exit abnormally, but the exact status reported by the system is not important, pass ``ExitTest/Condition/failure``. - If you need to check for a specific exit code or signal, pass @@ -85,15 +99,20 @@ will exit by passing an instance of ``ExitTest/Condition``: When the child process exits, the parent process resumes and compares the exit status of the child process against the expected exit condition you passed. If -they match, the exit test has passed; otherwise, it has failed and the testing -library records an issue. +they match, the exit test passes; otherwise, it fails and the testing library +records an issue. ### Gather output from the child process +The ``expect(exitsWith:observing:_:sourceLocation:performing:)`` and +``require(exitsWith:observing:_:sourceLocation:performing:)`` macros return an +instance of ``ExitTest/Result`` that contains information about the state of the +child process. + By default, the child process is configured without a standard output or standard error stream. If your test needs to review the content of either of -these streams, you can pass its key path to ``expect(exitsWith:observing:_:sourceLocation:performing:)`` -or ``require(exitsWith:observing:_:sourceLocation:performing:)``: +these streams, pass the key path to the corresponding ``ExitTest/Result`` +property to the macro: ```swift extension Customer { @@ -120,36 +139,14 @@ extension Customer { } ``` -- Note: The content of the standard output and standard error streams may - contain any arbitrary sequence of bytes, including sequences that are not - valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). +- Note: The content of the standard output and standard error streams can + contain any arbitrary sequence of bytes, including sequences that aren't valid + UTF-8 and can't be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). These streams are globally accessible within the child process, and any code running in an exit test may write to it including the operating system and any - third-party dependencies you have declared in your package. + third-party dependencies you declare in your package description or Xcode + project. The testing library always sets ``ExitTest/Result/statusAtExit`` to the actual exit status of the child process (as reported by the system) even if you do not pass it. - -### Constraints on exit tests - -#### State cannot be captured - -Exit tests cannot capture any state originating in the parent process or from -the enclosing lexical context. For example, the following exit test will fail to -compile because it captures a variable declared outside the exit test itself: - -```swift -@Test func `Customer won't eat food unless it's nutritious`() async { - let isNutritious = false - await #expect(exitsWith: .failure) { - var food = ... - food.isNutritious = isNutritious // ❌ ERROR: trying to capture state here - Customer.current.eat(food) - } -} -``` - -#### Exit tests cannot be nested - -An exit test cannot run within another exit test. From 11b772dd060bbd23695414515ae9919505b567dd Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Wed, 26 Mar 2025 18:43:34 -0400 Subject: [PATCH 13/19] Minor language tweaks --- Sources/Testing/Testing.docc/exit-testing.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index b79282eb2..cacde439f 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -54,12 +54,12 @@ or the ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro: } ``` -The closure or function reference you pass to the macro is the body of the exit -test. When an exit test is performed at runtime, the testing library starts a -new process with the same executable as the current process. The current task is -then suspended (as with `await`) and waits for the child process to exit. +The closure or function reference you pass to the macro is the _body_ of the +exit test. When an exit test is performed at runtime, the testing library starts +a new process with the same executable as the current process. The current task +is then suspended (as with `await`) and waits for the child process to exit. -The parent process never calls the body of the exit test. Instead, the child +The parent process doesn't call the body of the exit test. Instead, the child process treats the body of the exit test as its `main()` function and calls it directly. From 324e348d7e464e1e9e1f86fe5b0cc417211a2090 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 15 Apr 2025 10:38:12 -0400 Subject: [PATCH 14/19] StatusAtExit -> ExitStatus --- Sources/Testing/CMakeLists.txt | 2 +- .../{StatusAtExit.swift => ExitStatus.swift} | 4 +-- .../ExitTests/ExitTest.Condition.swift | 18 ++++++------ .../Testing/ExitTests/ExitTest.Result.swift | 6 ++-- Sources/Testing/ExitTests/ExitTest.swift | 29 +++++++++---------- Sources/Testing/ExitTests/WaitFor.swift | 12 ++++---- .../Expectations/Expectation+Macro.swift | 4 +-- Sources/Testing/Testing.docc/Expectations.md | 2 +- Sources/Testing/Testing.docc/exit-testing.md | 2 +- Tests/TestingTests/ExitTestTests.swift | 26 ++++++++--------- 10 files changed, 51 insertions(+), 54 deletions(-) rename Sources/Testing/ExitTests/{StatusAtExit.swift => ExitStatus.swift} (98%) diff --git a/Sources/Testing/CMakeLists.txt b/Sources/Testing/CMakeLists.txt index 7e07636d5..014a13d76 100644 --- a/Sources/Testing/CMakeLists.txt +++ b/Sources/Testing/CMakeLists.txt @@ -35,7 +35,7 @@ add_library(Testing ExitTests/ExitTest.Condition.swift ExitTests/ExitTest.Result.swift ExitTests/SpawnProcess.swift - ExitTests/StatusAtExit.swift + ExitTests/ExitStatus.swift ExitTests/WaitFor.swift Expectations/Expectation.swift Expectations/Expectation+Macro.swift diff --git a/Sources/Testing/ExitTests/StatusAtExit.swift b/Sources/Testing/ExitTests/ExitStatus.swift similarity index 98% rename from Sources/Testing/ExitTests/StatusAtExit.swift rename to Sources/Testing/ExitTests/ExitStatus.swift index a54646de8..6e8f2ca21 100644 --- a/Sources/Testing/ExitTests/StatusAtExit.swift +++ b/Sources/Testing/ExitTests/ExitStatus.swift @@ -25,7 +25,7 @@ private import _TestingInternals #if SWT_NO_PROCESS_SPAWNING @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif -public enum StatusAtExit: Sendable { +public enum ExitStatus: Sendable { /// The process exited with the given exit code. /// /// - Parameters: @@ -90,4 +90,4 @@ public enum StatusAtExit: Sendable { #if SWT_NO_PROCESS_SPAWNING @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif -extension StatusAtExit: Equatable {} +extension ExitStatus: Equatable {} diff --git a/Sources/Testing/ExitTests/ExitTest.Condition.swift b/Sources/Testing/ExitTests/ExitTest.Condition.swift index 3890a8446..7ead54829 100644 --- a/Sources/Testing/ExitTests/ExitTest.Condition.swift +++ b/Sources/Testing/ExitTests/ExitTest.Condition.swift @@ -40,7 +40,7 @@ extension ExitTest { /// An enumeration describing the possible conditions for an exit test. private enum _Kind: Sendable, Equatable { /// The exit test must exit with a particular exit status. - case statusAtExit(StatusAtExit) + case exitStatus(ExitStatus) /// The exit test must exit with any failure. case failure @@ -91,8 +91,8 @@ extension ExitTest.Condition { /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } - public init(_ statusAtExit: StatusAtExit) { - self.init(_kind: .statusAtExit(statusAtExit)) + public init(_ exitStatus: ExitStatus) { + self.init(_kind: .exitStatus(exitStatus)) } /// Creates a condition that matches when a process terminates with a given @@ -176,22 +176,22 @@ extension ExitTest.Condition { /// Check whether or not an exit test condition matches a given exit status. /// /// - Parameters: - /// - statusAtExit: An exit status to compare against. + /// - exitStatus: An exit status to compare against. /// - /// - Returns: Whether or not `self` and `statusAtExit` represent the same - /// exit condition. + /// - Returns: Whether or not `self` and `exitStatus` represent the same exit + /// condition. /// /// Two exit test conditions can be compared; if either instance is equal to /// ``failure``, it will compare equal to any instance except ``success``. - func isApproximatelyEqual(to statusAtExit: StatusAtExit) -> Bool { - return switch (self._kind, statusAtExit) { + func isApproximatelyEqual(to exitStatus: ExitStatus) -> Bool { + return switch (self._kind, exitStatus) { case let (.failure, .exitCode(exitCode)): exitCode != EXIT_SUCCESS case (.failure, .signal): // All terminating signals are considered failures. true default: - self._kind == .statusAtExit(statusAtExit) + self._kind == .exitStatus(exitStatus) } } } diff --git a/Sources/Testing/ExitTests/ExitTest.Result.swift b/Sources/Testing/ExitTests/ExitTest.Result.swift index df4b48a88..046ba5207 100644 --- a/Sources/Testing/ExitTests/ExitTest.Result.swift +++ b/Sources/Testing/ExitTests/ExitTest.Result.swift @@ -28,7 +28,7 @@ extension ExitTest { /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } - public var statusAtExit: StatusAtExit + public var exitStatus: ExitStatus /// All bytes written to the standard output stream of the exit test before /// it exited. @@ -91,8 +91,8 @@ extension ExitTest { public var standardErrorContent: [UInt8] = [] @_spi(ForToolsIntegrationOnly) - public init(statusAtExit: StatusAtExit) { - self.statusAtExit = statusAtExit + public init(exitStatus: ExitStatus) { + self.exitStatus = exitStatus } } } diff --git a/Sources/Testing/ExitTests/ExitTest.swift b/Sources/Testing/ExitTests/ExitTest.swift index ba6371599..643c59bfa 100644 --- a/Sources/Testing/ExitTests/ExitTest.swift +++ b/Sources/Testing/ExitTests/ExitTest.swift @@ -92,7 +92,7 @@ public struct ExitTest: Sendable, ~Copyable { /// this property to determine what information you need to preserve from your /// child process. /// - /// The value of this property always includes ``ExitTest/Result/statusAtExit`` + /// The value of this property always includes ``ExitTest/Result/exitStatus`` /// even if the test author does not specify it. /// /// Within a child process running an exit test, the value of this property is @@ -101,8 +101,8 @@ public struct ExitTest: Sendable, ~Copyable { public var observedValues: [any PartialKeyPath & Sendable] { get { var result = _observedValues - if !result.contains(\.statusAtExit) { // O(n), but n <= 3 (no Set needed) - result.append(\.statusAtExit) + if !result.contains(\.exitStatus) { // O(n), but n <= 3 (no Set needed) + result.append(\.exitStatus) } return result } @@ -339,7 +339,7 @@ extension ExitTest { /// - expectedExitCondition: The expected exit condition. /// - observedValues: An array of key paths representing results from within /// the exit test that should be observed and returned by this macro. The -/// ``ExitTest/Result/statusAtExit`` property is always returned. +/// ``ExitTest/Result/exitStatus`` property is always returned. /// - expression: The expression, corresponding to `condition`, that is being /// evaluated (if available at compile time.) /// - comments: An array of comments describing the expectation. This array @@ -376,8 +376,8 @@ func callExitTest( #if os(Windows) // For an explanation of this magic, see the corresponding logic in // ExitTest.callAsFunction(). - if case let .exitCode(exitCode) = result.statusAtExit, (exitCode & ~STATUS_CODE_MASK) == STATUS_SIGNAL_CAUGHT_BITS { - result.statusAtExit = .signal(exitCode & STATUS_CODE_MASK) + if case let .exitCode(exitCode) = result.exitStatus, (exitCode & ~STATUS_CODE_MASK) == STATUS_SIGNAL_CAUGHT_BITS { + result.exitStatus = .signal(exitCode & STATUS_CODE_MASK) } #endif } catch { @@ -402,22 +402,19 @@ func callExitTest( // For lack of a better way to handle an exit test failing in this way, // we record the system issue above, then let the expectation fail below by // reporting an exit condition that's the inverse of the expected one. - let statusAtExit: StatusAtExit = if expectedExitCondition.isApproximatelyEqual(to: .exitCode(EXIT_FAILURE)) { + let exitStatus: ExitStatus = if expectedExitCondition.isApproximatelyEqual(to: .exitCode(EXIT_FAILURE)) { .exitCode(EXIT_SUCCESS) } else { .exitCode(EXIT_FAILURE) } - result = ExitTest.Result(statusAtExit: statusAtExit) + result = ExitTest.Result(exitStatus: exitStatus) } - // How did the exit test actually exit? - let actualStatusAtExit = result.statusAtExit - // Plumb the exit test's result through the general expectation machinery. return __checkValue( - expectedExitCondition.isApproximatelyEqual(to: actualStatusAtExit), + expectedExitCondition.isApproximatelyEqual(to: result.exitStatus), expression: expression, - expressionWithCapturedRuntimeValues: expression.capturingRuntimeValues(actualStatusAtExit), + expressionWithCapturedRuntimeValues: expression.capturingRuntimeValues(result.exitStatus), mismatchedExitConditionDescription: String(describingForTest: expectedExitCondition), comments: comments(), isRequired: isRequired, @@ -710,8 +707,8 @@ extension ExitTest { // Await termination of the child process. taskGroup.addTask { - let statusAtExit = try await wait(for: processID) - return { $0.statusAtExit = statusAtExit } + let exitStatus = try await wait(for: processID) + return { $0.exitStatus = exitStatus } } // Read back the stdout and stderr streams. @@ -741,7 +738,7 @@ extension ExitTest { // Collate the various bits of the result. The exit condition used here // is just a placeholder and will be replaced by the result of one of // the tasks above. - var result = ExitTest.Result(statusAtExit: .exitCode(EXIT_FAILURE)) + var result = ExitTest.Result(exitStatus: .exitCode(EXIT_FAILURE)) for try await update in taskGroup { update?(&result) } diff --git a/Sources/Testing/ExitTests/WaitFor.swift b/Sources/Testing/ExitTests/WaitFor.swift index cc611158f..238ed835a 100644 --- a/Sources/Testing/ExitTests/WaitFor.swift +++ b/Sources/Testing/ExitTests/WaitFor.swift @@ -20,7 +20,7 @@ internal import _TestingInternals /// /// - Throws: If the exit status of the process with ID `pid` cannot be /// determined (i.e. it does not represent an exit condition.) -private func _blockAndWait(for pid: consuming pid_t) throws -> StatusAtExit { +private func _blockAndWait(for pid: consuming pid_t) throws -> ExitStatus { let pid = consume pid // Get the exit status of the process or throw an error (other than EINTR.) @@ -61,7 +61,7 @@ private func _blockAndWait(for pid: consuming pid_t) throws -> StatusAtExit { /// - Note: The open-source implementation of libdispatch available on Linux /// and other platforms does not support `DispatchSourceProcess`. Those /// platforms use an alternate implementation below. -func wait(for pid: consuming pid_t) async throws -> StatusAtExit { +func wait(for pid: consuming pid_t) async throws -> ExitStatus { let pid = consume pid let source = DispatchSource.makeProcessSource(identifier: pid, eventMask: .exit) @@ -80,7 +80,7 @@ func wait(for pid: consuming pid_t) async throws -> StatusAtExit { } #elseif SWT_TARGET_OS_APPLE || os(Linux) || os(FreeBSD) || os(OpenBSD) /// A mapping of awaited child PIDs to their corresponding Swift continuations. -private let _childProcessContinuations = LockedWith]>() +private let _childProcessContinuations = LockedWith]>() /// A condition variable used to suspend the waiter thread created by /// `_createWaitThread()` when there are no child processes to await. @@ -202,7 +202,7 @@ private let _createWaitThread: Void = { /// /// On Apple platforms, the libdispatch-based implementation above is more /// efficient because it does not need to permanently reserve a thread. -func wait(for pid: consuming pid_t) async throws -> StatusAtExit { +func wait(for pid: consuming pid_t) async throws -> ExitStatus { let pid = consume pid // Ensure the waiter thread is running. @@ -239,7 +239,7 @@ func wait(for pid: consuming pid_t) async throws -> StatusAtExit { /// This implementation of `wait(for:)` calls `RegisterWaitForSingleObject()` to /// wait for `processHandle`, suspends the calling task until the waiter's /// callback is called, then calls `GetExitCodeProcess()`. -func wait(for processHandle: consuming HANDLE) async throws -> StatusAtExit { +func wait(for processHandle: consuming HANDLE) async throws -> ExitStatus { let processHandle = consume processHandle defer { _ = CloseHandle(processHandle) @@ -283,6 +283,6 @@ func wait(for processHandle: consuming HANDLE) async throws -> StatusAtExit { } #else #warning("Platform-specific implementation missing: cannot wait for child processes to exit") -func wait(for processID: consuming Never) async throws -> StatusAtExit {} +func wait(for processID: consuming Never) async throws -> ExitStatus {} #endif #endif diff --git a/Sources/Testing/Expectations/Expectation+Macro.swift b/Sources/Testing/Expectations/Expectation+Macro.swift index c4d89e262..c93211201 100644 --- a/Sources/Testing/Expectations/Expectation+Macro.swift +++ b/Sources/Testing/Expectations/Expectation+Macro.swift @@ -490,7 +490,7 @@ public macro require( /// - expectedExitCondition: The expected exit condition. /// - observedValues: An array of key paths representing results from within /// the exit test that should be observed and returned by this macro. The -/// ``ExitTest/Result/statusAtExit`` property is always returned. +/// ``ExitTest/Result/exitStatus`` property is always returned. /// - comment: A comment describing the expectation. /// - sourceLocation: The source location to which recorded expectations and /// issues should be attributed. @@ -533,7 +533,7 @@ public macro require( /// - expectedExitCondition: The expected exit condition. /// - observedValues: An array of key paths representing results from within /// the exit test that should be observed and returned by this macro. The -/// ``ExitTest/Result/statusAtExit`` property is always returned. +/// ``ExitTest/Result/exitStatus`` property is always returned. /// - comment: A comment describing the expectation. /// - sourceLocation: The source location to which recorded expectations and /// issues should be attributed. diff --git a/Sources/Testing/Testing.docc/Expectations.md b/Sources/Testing/Testing.docc/Expectations.md index dea1e5d69..ddc02711f 100644 --- a/Sources/Testing/Testing.docc/Expectations.md +++ b/Sources/Testing/Testing.docc/Expectations.md @@ -77,8 +77,8 @@ the test when the code doesn't satisfy a requirement, use - - ``expect(exitsWith:observing:_:sourceLocation:performing:)`` - ``require(exitsWith:observing:_:sourceLocation:performing:)`` +- ``ExitStatus`` - ``ExitTest`` -- ``StatusAtExit`` ### Confirming that asynchronous events occur diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index cacde439f..916f4cc8e 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -147,6 +147,6 @@ extension Customer { third-party dependencies you declare in your package description or Xcode project. -The testing library always sets ``ExitTest/Result/statusAtExit`` to the actual +The testing library always sets ``ExitTest/Result/exitStatus`` to the actual exit status of the child process (as reported by the system) even if you do not pass it. diff --git a/Tests/TestingTests/ExitTestTests.swift b/Tests/TestingTests/ExitTestTests.swift index befb856c1..ea297183c 100644 --- a/Tests/TestingTests/ExitTestTests.swift +++ b/Tests/TestingTests/ExitTestTests.swift @@ -88,7 +88,7 @@ private import _TestingInternals // Mock an exit test where the process exits successfully. configuration.exitTestHandler = { _ in - return ExitTest.Result(statusAtExit: .exitCode(EXIT_SUCCESS)) + return ExitTest.Result(exitStatus: .exitCode(EXIT_SUCCESS)) } await Test { await #expect(exitsWith: .success) {} @@ -96,7 +96,7 @@ private import _TestingInternals // Mock an exit test where the process exits with a particular error code. configuration.exitTestHandler = { _ in - return ExitTest.Result(statusAtExit: .exitCode(123)) + return ExitTest.Result(exitStatus: .exitCode(123)) } await Test { await #expect(exitsWith: .failure) {} @@ -104,7 +104,7 @@ private import _TestingInternals // Mock an exit test where the process exits with a signal. configuration.exitTestHandler = { _ in - return ExitTest.Result(statusAtExit: .signal(SIGABRT)) + return ExitTest.Result(exitStatus: .signal(SIGABRT)) } await Test { await #expect(exitsWith: .signal(SIGABRT)) {} @@ -126,7 +126,7 @@ private import _TestingInternals // Mock exit tests that were expected to fail but passed. configuration.exitTestHandler = { _ in - return ExitTest.Result(statusAtExit: .exitCode(EXIT_SUCCESS)) + return ExitTest.Result(exitStatus: .exitCode(EXIT_SUCCESS)) } await Test { await #expect(exitsWith: .failure) {} @@ -140,7 +140,7 @@ private import _TestingInternals // Mock exit tests that unexpectedly signalled. configuration.exitTestHandler = { _ in - return ExitTest.Result(statusAtExit: .signal(SIGABRT)) + return ExitTest.Result(exitStatus: .signal(SIGABRT)) } await Test { await #expect(exitsWith: .exitCode(EXIT_SUCCESS)) {} @@ -245,21 +245,21 @@ private import _TestingInternals var result = await #expect(exitsWith: .success) { exit(EXIT_SUCCESS) } - #expect(result?.statusAtExit == .exitCode(EXIT_SUCCESS)) + #expect(result?.exitStatus == .exitCode(EXIT_SUCCESS)) result = await #expect(exitsWith: .exitCode(123)) { exit(123) } - #expect(result?.statusAtExit == .exitCode(123)) + #expect(result?.exitStatus == .exitCode(123)) // Test that basic passing exit tests produce the correct results (#require) result = try await #require(exitsWith: .success) { exit(EXIT_SUCCESS) } - #expect(result?.statusAtExit == .exitCode(EXIT_SUCCESS)) + #expect(result?.exitStatus == .exitCode(EXIT_SUCCESS)) result = try await #require(exitsWith: .exitCode(123)) { exit(123) } - #expect(result?.statusAtExit == .exitCode(123)) + #expect(result?.exitStatus == .exitCode(123)) } @Test("Result is nil on failure") @@ -278,7 +278,7 @@ private import _TestingInternals } } configuration.exitTestHandler = { _ in - ExitTest.Result(statusAtExit: .exitCode(123)) + ExitTest.Result(exitStatus: .exitCode(123)) } await Test { @@ -301,7 +301,7 @@ private import _TestingInternals } } configuration.exitTestHandler = { _ in - ExitTest.Result(statusAtExit: .exitCode(EXIT_FAILURE)) + ExitTest.Result(exitStatus: .exitCode(EXIT_FAILURE)) } await Test { @@ -348,7 +348,7 @@ private import _TestingInternals try FileHandle.stderr.write(String("STANDARD ERROR".reversed())) exit(EXIT_SUCCESS) } - #expect(result.statusAtExit == .exitCode(EXIT_SUCCESS)) + #expect(result.exitStatus == .exitCode(EXIT_SUCCESS)) #expect(result.standardOutputContent.contains("STANDARD OUTPUT".utf8)) #expect(result.standardErrorContent.isEmpty) @@ -357,7 +357,7 @@ private import _TestingInternals try FileHandle.stderr.write(String("STANDARD ERROR".reversed())) exit(EXIT_SUCCESS) } - #expect(result.statusAtExit == .exitCode(EXIT_SUCCESS)) + #expect(result.exitStatus == .exitCode(EXIT_SUCCESS)) #expect(result.standardOutputContent.isEmpty) #expect(result.standardErrorContent.contains("STANDARD ERROR".utf8.reversed())) } From 5d10fd80156cc304d8426827804b8232126acf5b Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 22 Apr 2025 21:57:15 -0400 Subject: [PATCH 15/19] Fix a couple of typos --- Sources/Testing/ExitTests/ExitTest.swift | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/Sources/Testing/ExitTests/ExitTest.swift b/Sources/Testing/ExitTests/ExitTest.swift index ecb74b2cb..9ff732eee 100644 --- a/Sources/Testing/ExitTests/ExitTest.swift +++ b/Sources/Testing/ExitTests/ExitTest.swift @@ -604,18 +604,13 @@ extension ExitTest { try JSON.decode(ExitTest.ID.self, from: UnsafeRawBufferPointer(idBuffer)) } } - guard let id else { + guard let id, var result = find(identifiedBy: id) else { return nil } // If an exit test was found, inject back channel handling into its body. // External tools authors should set up their own back channel mechanisms // and ensure they're installed before calling ExitTest.callAsFunction(). - guard var result = find(identifiedBy: id) else { - return nil - } - - // We can't say guard let here because it counts as a consume. guard let backChannel = _makeFileHandle(forEnvironmentVariableNamed: "SWT_BACKCHANNEL", mode: "wb") else { return result } From d6cdac5db9031a7def2e6201d14a4ef0441366f8 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Tue, 29 Apr 2025 11:17:00 -0400 Subject: [PATCH 16/19] exitsWith: -> processExitsWith: --- Sources/Testing/ExitTests/ExitStatus.swift | 4 +- .../ExitTests/ExitTest.CapturedValue.swift | 2 +- .../ExitTests/ExitTest.Condition.swift | 4 +- .../Testing/ExitTests/ExitTest.Result.swift | 14 +-- Sources/Testing/ExitTests/ExitTest.swift | 39 ++++--- .../Expectations/Expectation+Macro.swift | 8 +- .../ExpectationChecking+Macro.swift | 16 +-- .../SourceAttribution/Expression.swift | 5 +- Sources/Testing/Testing.docc/Expectations.md | 4 +- Sources/Testing/Testing.docc/exit-testing.md | 19 ++-- Sources/TestingMacros/ConditionMacro.swift | 6 +- .../ConditionMacroTests.swift | 12 +-- Tests/TestingTests/AttachmentTests.swift | 2 +- Tests/TestingTests/ConfirmationTests.swift | 6 +- Tests/TestingTests/DiscoveryTests.swift | 4 +- Tests/TestingTests/ExitTestTests.swift | 100 +++++++++--------- Tests/TestingTests/PlanIterationTests.swift | 4 +- Tests/TestingTests/SourceLocationTests.swift | 16 +-- .../Support/FileHandleTests.swift | 2 +- 19 files changed, 133 insertions(+), 134 deletions(-) diff --git a/Sources/Testing/ExitTests/ExitStatus.swift b/Sources/Testing/ExitTests/ExitStatus.swift index afa47d8d6..0dd6d86ab 100644 --- a/Sources/Testing/ExitTests/ExitStatus.swift +++ b/Sources/Testing/ExitTests/ExitStatus.swift @@ -16,8 +16,8 @@ private import _TestingInternals /// ``ExitTest/Condition`` using ``ExitTest/Condition/init(_:)``. That value /// can then be used to describe the condition under which an exit test is /// expected to pass or fail by passing it to -/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or -/// ``require(exitsWith:observing:_:sourceLocation:performing:)``. +/// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(processExitsWith:observing:_:sourceLocation:performing:)``. /// /// @Metadata { /// @Available(Swift, introduced: 6.2) diff --git a/Sources/Testing/ExitTests/ExitTest.CapturedValue.swift b/Sources/Testing/ExitTests/ExitTest.CapturedValue.swift index d4c84e446..1d5c9b18a 100644 --- a/Sources/Testing/ExitTests/ExitTest.CapturedValue.swift +++ b/Sources/Testing/ExitTests/ExitTest.CapturedValue.swift @@ -26,7 +26,7 @@ extension ExitTest { /// exit test: /// /// ```swift - /// await #expect(exitsWith: .failure) { [a = a as T, b = b as U, c = c as V] in + /// await #expect(processExitsWith: .failure) { [a = a as T, b = b as U, c = c as V] in /// ... /// } /// ``` diff --git a/Sources/Testing/ExitTests/ExitTest.Condition.swift b/Sources/Testing/ExitTests/ExitTest.Condition.swift index ac2f10e7d..1b89dc120 100644 --- a/Sources/Testing/ExitTests/ExitTest.Condition.swift +++ b/Sources/Testing/ExitTests/ExitTest.Condition.swift @@ -18,8 +18,8 @@ extension ExitTest { /// /// Values of this type are used to describe the conditions under which an /// exit test is expected to pass or fail by passing them to - /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or - /// ``require(exitsWith:observing:_:sourceLocation:performing:)``. + /// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(processExitsWith:observing:_:sourceLocation:performing:)``. /// /// ## Topics /// diff --git a/Sources/Testing/ExitTests/ExitTest.Result.swift b/Sources/Testing/ExitTests/ExitTest.Result.swift index 046ba5207..ef70a3789 100644 --- a/Sources/Testing/ExitTests/ExitTest.Result.swift +++ b/Sources/Testing/ExitTests/ExitTest.Result.swift @@ -15,9 +15,9 @@ extension ExitTest { /// A type representing the result of an exit test after it has exited and /// returned control to the calling test function. /// - /// Both ``expect(exitsWith:observing:_:sourceLocation:performing:)`` and - /// ``require(exitsWith:observing:_:sourceLocation:performing:)`` return - /// instances of this type. + /// Both ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` + /// and ``require(processExitsWith:observing:_:sourceLocation:performing:)`` + /// return instances of this type. /// /// @Metadata { /// @Available(Swift, introduced: 6.2) @@ -49,8 +49,8 @@ extension ExitTest { /// /// To enable gathering output from the standard output stream during an /// exit test, pass `\.standardOutputContent` in the `observedValues` - /// argument of ``expect(exitsWith:observing:_:sourceLocation:performing:)`` - /// or ``require(exitsWith:observing:_:sourceLocation:performing:)``. + /// argument of ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` + /// or ``require(processExitsWith:observing:_:sourceLocation:performing:)``. /// /// If you did not request standard output content when running an exit /// test, the value of this property is the empty array. @@ -79,8 +79,8 @@ extension ExitTest { /// /// To enable gathering output from the standard error stream during an exit /// test, pass `\.standardErrorContent` in the `observedValues` argument of - /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or - /// ``require(exitsWith:observing:_:sourceLocation:performing:)``. + /// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(processExitsWith:observing:_:sourceLocation:performing:)``. /// /// If you did not request standard error content when running an exit test, /// the value of this property is the empty array. diff --git a/Sources/Testing/ExitTests/ExitTest.swift b/Sources/Testing/ExitTests/ExitTest.swift index fd3c37ee0..81b40fc12 100644 --- a/Sources/Testing/ExitTests/ExitTest.swift +++ b/Sources/Testing/ExitTests/ExitTest.swift @@ -26,9 +26,9 @@ private import _TestingInternals /// A type describing an exit test. /// /// Instances of this type describe exit tests you create using the -/// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or -/// ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro. You -/// don't usually need to interact directly with an instance of this type. +/// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(processExitsWith:observing:_:sourceLocation:performing:)`` macro. +/// You don't usually need to interact directly with an instance of this type. /// /// @Metadata { /// @Available(Swift, introduced: 6.2) @@ -93,10 +93,10 @@ public struct ExitTest: Sendable, ~Copyable { /// observed and returned to the caller. /// /// The testing library sets this property to match what was passed by the - /// developer to the `#expect(exitsWith:)` or `#require(exitsWith:)` macro. - /// If you are implementing an exit test handler, you can check the value of - /// this property to determine what information you need to preserve from your - /// child process. + /// developer to the `#expect(processExitsWith:)` or `#require(processExitsWith:)` + /// macro. If you are implementing an exit test handler, you can check the + /// value of this property to determine what information you need to preserve + /// from your child process. /// /// The value of this property always includes ``ExitTest/Result/exitStatus`` /// even if the test author does not specify it. @@ -234,9 +234,9 @@ extension ExitTest { /// Call the exit test in the current process. /// /// This function invokes the closure originally passed to - /// `#expect(exitsWith:)` _in the current process_. That closure is expected - /// to terminate the process; if it does not, the testing library will - /// terminate the process as if its `main()` function returned naturally. + /// `#expect(processExitsWith:)` _in the current process_. That closure is + /// expected to terminate the process; if it does not, the testing library + /// will terminate the process as if its `main()` function returned naturally. public consuming func callAsFunction() async -> Never { Self._disableCrashReporting() @@ -325,8 +325,8 @@ extension ExitTest { /// /// - Returns: Whether or not an exit test was stored into `outValue`. /// - /// - Warning: This function is used to implement the `#expect(exitsWith:)` - /// macro. Do not use it directly. + /// - Warning: This function is used to implement the + /// `#expect(processExitsWith:)` macro. Do not use it directly. public static func __store( _ id: (UInt64, UInt64, UInt64, UInt64), _ body: @escaping @Sendable (repeat each T) async throws -> Void, @@ -414,12 +414,12 @@ extension ExitTest { /// - sourceLocation: The source location of the expectation. /// /// This function contains the common implementation for all -/// `await #expect(exitsWith:) { }` invocations regardless of calling +/// `await #expect(processExitsWith:) { }` invocations regardless of calling /// convention. func callExitTest( identifiedBy exitTestID: (UInt64, UInt64, UInt64, UInt64), encodingCapturedValues capturedValues: [ExitTest.CapturedValue], - exitsWith expectedExitCondition: ExitTest.Condition, + processExitsWith expectedExitCondition: ExitTest.Condition, observing observedValues: [any PartialKeyPath & Sendable], expression: __Expression, comments: @autoclosure () -> [Comment], @@ -428,7 +428,7 @@ func callExitTest( sourceLocation: SourceLocation ) async -> Result { guard let configuration = Configuration.current ?? Configuration.all.first else { - preconditionFailure("A test must be running on the current task to use #expect(exitsWith:).") + preconditionFailure("A test must be running on the current task to use #expect(processExitsWith:).") } var result: ExitTest.Result @@ -516,11 +516,10 @@ extension ExitTest { /// the exit test. /// /// This handler is invoked when an exit test (i.e. a call to either - /// ``expect(exitsWith:observing:_:sourceLocation:performing:)`` or - /// ``require(exitsWith:observing:_:sourceLocation:performing:)``) is started. - /// The handler is responsible for initializing a new child environment - /// (typically a child process) and running the exit test identified by - /// `sourceLocation` there. + /// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(processExitsWith:observing:_:sourceLocation:performing:)``) is + /// started. The handler is responsible for initializing a new child + /// environment (typically a child process) and running `exitTest` there. /// /// In the child environment, you can find the exit test again by calling /// ``ExitTest/find(at:)`` and can run it by calling diff --git a/Sources/Testing/Expectations/Expectation+Macro.swift b/Sources/Testing/Expectations/Expectation+Macro.swift index c93211201..d14920547 100644 --- a/Sources/Testing/Expectations/Expectation+Macro.swift +++ b/Sources/Testing/Expectations/Expectation+Macro.swift @@ -506,7 +506,7 @@ public macro require( /// causes a process to terminate: /// /// ```swift -/// await #expect(exitsWith: .failure) { +/// await #expect(processExitsWith: .failure) { /// fatalError() /// } /// ``` @@ -519,7 +519,7 @@ public macro require( #endif @discardableResult @freestanding(expression) public macro expect( - exitsWith expectedExitCondition: ExitTest.Condition, + processExitsWith expectedExitCondition: ExitTest.Condition, observing observedValues: [any PartialKeyPath & Sendable] = [], _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, @@ -551,7 +551,7 @@ public macro require( /// causes a process to terminate: /// /// ```swift -/// try await #require(exitsWith: .failure) { +/// try await #require(processExitsWith: .failure) { /// fatalError() /// } /// ``` @@ -564,7 +564,7 @@ public macro require( #endif @discardableResult @freestanding(expression) public macro require( - exitsWith expectedExitCondition: ExitTest.Condition, + processExitsWith expectedExitCondition: ExitTest.Condition, observing observedValues: [any PartialKeyPath & Sendable] = [], _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, diff --git a/Sources/Testing/Expectations/ExpectationChecking+Macro.swift b/Sources/Testing/Expectations/ExpectationChecking+Macro.swift index 5050c3c4a..6d3093f2a 100644 --- a/Sources/Testing/Expectations/ExpectationChecking+Macro.swift +++ b/Sources/Testing/Expectations/ExpectationChecking+Macro.swift @@ -1139,14 +1139,14 @@ public func __checkClosureCall( /// Check that an expression always exits (terminates the current process) with /// a given status. /// -/// This overload is used for `await #expect(exitsWith:) { }` invocations that -/// do not capture any state. +/// This overload is used for `await #expect(processExitsWith:) { }` invocations +/// that do not capture any state. /// /// - Warning: This function is used to implement the `#expect()` and /// `#require()` macros. Do not call it directly. public func __checkClosureCall( identifiedBy exitTestID: (UInt64, UInt64, UInt64, UInt64), - exitsWith expectedExitCondition: ExitTest.Condition, + processExitsWith expectedExitCondition: ExitTest.Condition, observing observedValues: [any PartialKeyPath & Sendable] = [], performing _: @convention(thin) () -> Void, expression: __Expression, @@ -1158,7 +1158,7 @@ public func __checkClosureCall( await callExitTest( identifiedBy: exitTestID, encodingCapturedValues: [], - exitsWith: expectedExitCondition, + processExitsWith: expectedExitCondition, observing: observedValues, expression: expression, comments: comments(), @@ -1170,8 +1170,8 @@ public func __checkClosureCall( /// Check that an expression always exits (terminates the current process) with /// a given status. /// -/// This overload is used for `await #expect(exitsWith:) { }` invocations that -/// capture some values with an explicit capture list. +/// This overload is used for `await #expect(processExitsWith:) { }` invocations +/// that capture some values with an explicit capture list. /// /// - Warning: This function is used to implement the `#expect()` and /// `#require()` macros. Do not call it directly. @@ -1179,7 +1179,7 @@ public func __checkClosureCall( public func __checkClosureCall( identifiedBy exitTestID: (UInt64, UInt64, UInt64, UInt64), encodingCapturedValues capturedValues: (repeat each T), - exitsWith expectedExitCondition: ExitTest.Condition, + processExitsWith expectedExitCondition: ExitTest.Condition, observing observedValues: [any PartialKeyPath & Sendable] = [], performing _: @convention(thin) () -> Void, expression: __Expression, @@ -1191,7 +1191,7 @@ public func __checkClosureCall( await callExitTest( identifiedBy: exitTestID, encodingCapturedValues: Array(repeat each capturedValues), - exitsWith: expectedExitCondition, + processExitsWith: expectedExitCondition, observing: observedValues, expression: expression, comments: comments(), diff --git a/Sources/Testing/SourceAttribution/Expression.swift b/Sources/Testing/SourceAttribution/Expression.swift index dce4ed2a2..b96d227ac 100644 --- a/Sources/Testing/SourceAttribution/Expression.swift +++ b/Sources/Testing/SourceAttribution/Expression.swift @@ -22,9 +22,8 @@ /// let swiftSyntaxExpr: ExprSyntax = "\(testExpr)" /// ``` /// -/// - Warning: This type is used to implement the `#expect(exitsWith:)` -/// macro. Do not use it directly. Tools can use the SPI ``Expression`` -/// typealias if needed. +/// - Warning: This type is used to implement the `#expect()` macro. Do not use +/// it directly. Tools can use the SPI ``Expression`` typealias if needed. public struct __Expression: Sendable { /// An enumeration describing the various kinds of expression that can be /// captured. diff --git a/Sources/Testing/Testing.docc/Expectations.md b/Sources/Testing/Testing.docc/Expectations.md index ddc02711f..e55ff3a1e 100644 --- a/Sources/Testing/Testing.docc/Expectations.md +++ b/Sources/Testing/Testing.docc/Expectations.md @@ -75,8 +75,8 @@ the test when the code doesn't satisfy a requirement, use ### Checking how processes exit - -- ``expect(exitsWith:observing:_:sourceLocation:performing:)`` -- ``require(exitsWith:observing:_:sourceLocation:performing:)`` +- ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` +- ``require(processExitsWith:observing:_:sourceLocation:performing:)`` - ``ExitStatus`` - ``ExitTest`` diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index 916f4cc8e..a55db2940 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -41,12 +41,13 @@ functions correctly catch invalid inputs. ### Create an exit test -To create an exit test, call either the ``expect(exitsWith:observing:_:sourceLocation:performing:)`` -or the ``require(exitsWith:observing:_:sourceLocation:performing:)`` macro: +To create an exit test, call either the ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` +or the ``require(processExitsWith:observing:_:sourceLocation:performing:)`` +macro: ```swift @Test func `Customer won't eat food unless it's delicious`() async { - let result = await #expect(exitsWith: .failure) { + let result = await #expect(processExitsWith: .failure) { var food = ... food.isDelicious = false Customer.current.eat(food) @@ -71,7 +72,7 @@ directly. ```swift @Test func `Customer won't eat food unless it's nutritious`() async { let isNutritious = false - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { var food = ... food.isNutritious = isNutritious // ❌ ERROR: trying to capture state here Customer.current.eat(food) @@ -104,10 +105,10 @@ records an issue. ### Gather output from the child process -The ``expect(exitsWith:observing:_:sourceLocation:performing:)`` and -``require(exitsWith:observing:_:sourceLocation:performing:)`` macros return an -instance of ``ExitTest/Result`` that contains information about the state of the -child process. +The ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` and +``require(processExitsWith:observing:_:sourceLocation:performing:)`` macros +return an instance of ``ExitTest/Result`` that contains information about the +state of the child process. By default, the child process is configured without a standard output or standard error stream. If your test needs to review the content of either of @@ -126,7 +127,7 @@ extension Customer { @Test func `Customer won't eat food unless it's delicious`() async { let result = await #expect( - exitsWith: .failure, + processExitsWith: .failure, observing: [\.standardOutputContent] ) { var food = ... diff --git a/Sources/TestingMacros/ConditionMacro.swift b/Sources/TestingMacros/ConditionMacro.swift index e21938041..49630cfc9 100644 --- a/Sources/TestingMacros/ConditionMacro.swift +++ b/Sources/TestingMacros/ConditionMacro.swift @@ -14,7 +14,7 @@ import SwiftSyntaxBuilder public import SwiftSyntaxMacros #if !hasFeature(SymbolLinkageMarkers) && SWT_NO_LEGACY_TEST_DISCOVERY -#error("Platform-specific misconfiguration: either SymbolLinkageMarkers or legacy test discovery is required to expand #expect(exitsWith:)") +#error("Platform-specific misconfiguration: either SymbolLinkageMarkers or legacy test discovery is required to expand #expect(processExitsWith:)") #endif /// A protocol containing the common implementation for the expansions of the @@ -628,7 +628,7 @@ extension ExitTestExpectMacro { }() } -/// A type describing the expansion of the `#expect(exitsWith:)` macro. +/// A type describing the expansion of the `#expect(processExitsWith:)` macro. /// /// This type checks for nested invocations of `#expect()` and `#require()` and /// diagnoses them as unsupported. It is otherwise exactly equivalent to @@ -637,7 +637,7 @@ public struct ExitTestExpectMacro: ExitTestConditionMacro { public typealias Base = ExpectMacro } -/// A type describing the expansion of the `#require(exitsWith:)` macro. +/// A type describing the expansion of the `#require(processExitsWith:)` macro. /// /// This type checks for nested invocations of `#expect()` and `#require()` and /// diagnoses them as unsupported. It is otherwise exactly equivalent to diff --git a/Tests/TestingMacrosTests/ConditionMacroTests.swift b/Tests/TestingMacrosTests/ConditionMacroTests.swift index 67531dabf..dc36af7cd 100644 --- a/Tests/TestingMacrosTests/ConditionMacroTests.swift +++ b/Tests/TestingMacrosTests/ConditionMacroTests.swift @@ -437,13 +437,13 @@ struct ConditionMacroTests { } #if ExperimentalExitTestValueCapture - @Test("#expect(exitsWith:) produces a diagnostic for a bad capture", + @Test("#expect(processExitsWith:) produces a diagnostic for a bad capture", arguments: [ - "#expectExitTest(exitsWith: x) { [weak a] in }": + "#expectExitTest(processExitsWith: x) { [weak a] in }": "Specifier 'weak' cannot be used with captured value 'a'", - "#expectExitTest(exitsWith: x) { [a] in }": + "#expectExitTest(processExitsWith: x) { [a] in }": "Type of captured value 'a' is ambiguous", - "#expectExitTest(exitsWith: x) { [a = b] in }": + "#expectExitTest(processExitsWith: x) { [a = b] in }": "Type of captured value 'a' is ambiguous", ] ) @@ -463,8 +463,8 @@ struct ConditionMacroTests { @Test( "Capture list on an exit test produces a diagnostic", arguments: [ - "#expectExitTest(exitsWith: x) { [a] in }": - "Cannot specify a capture clause in closure passed to '#expectExitTest(exitsWith:_:)'" + "#expectExitTest(processExitsWith: x) { [a] in }": + "Cannot specify a capture clause in closure passed to '#expectExitTest(processExitsWith:_:)'" ] ) func exitTestCaptureListProducesDiagnostic(input: String, expectedMessage: String) throws { diff --git a/Tests/TestingTests/AttachmentTests.swift b/Tests/TestingTests/AttachmentTests.swift index 0281b4091..be940371e 100644 --- a/Tests/TestingTests/AttachmentTests.swift +++ b/Tests/TestingTests/AttachmentTests.swift @@ -554,7 +554,7 @@ extension AttachmentTests { #if !SWT_NO_EXIT_TESTS @available(_uttypesAPI, *) @Test func cannotAttachCGImageWithNonImageType() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { let attachment = Attachment(try Self.cgImage.get(), named: "diamond", as: .mp3) try attachment.attachableValue.withUnsafeBytes(for: attachment) { _ in } } diff --git a/Tests/TestingTests/ConfirmationTests.swift b/Tests/TestingTests/ConfirmationTests.swift index 5502cb8d2..c4f076268 100644 --- a/Tests/TestingTests/ConfirmationTests.swift +++ b/Tests/TestingTests/ConfirmationTests.swift @@ -76,7 +76,7 @@ struct ConfirmationTests { await confirmation(expectedCount: Int.max...Int.max) { _ in } #if !SWT_NO_EXIT_TESTS await withKnownIssue("Crashes in Swift standard library (rdar://139568287)") { - await #expect(exitsWith: .success) { + await #expect(processExitsWith: .success) { await confirmation(expectedCount: Int.max...) { _ in } } } @@ -87,10 +87,10 @@ struct ConfirmationTests { #if !SWT_NO_EXIT_TESTS @Test("Confirmation requires positive count") func positiveCount() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { await confirmation { $0.confirm(count: 0) } } - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { await confirmation { $0.confirm(count: -1) } } } diff --git a/Tests/TestingTests/DiscoveryTests.swift b/Tests/TestingTests/DiscoveryTests.swift index 8ec185813..24d2eecfa 100644 --- a/Tests/TestingTests/DiscoveryTests.swift +++ b/Tests/TestingTests/DiscoveryTests.swift @@ -49,10 +49,10 @@ struct DiscoveryTests { #if !SWT_NO_EXIT_TESTS @Test("TestContentKind rejects bad string literals") func badTestContentKindLiteral() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { _ = "abc" as TestContentKind } - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { _ = "abcde" as TestContentKind } } diff --git a/Tests/TestingTests/ExitTestTests.swift b/Tests/TestingTests/ExitTestTests.swift index 524e50b3b..02be1a140 100644 --- a/Tests/TestingTests/ExitTestTests.swift +++ b/Tests/TestingTests/ExitTestTests.swift @@ -14,26 +14,26 @@ private import _TestingInternals #if !SWT_NO_EXIT_TESTS @Suite("Exit test tests") struct ExitTestTests { @Test("Exit tests (passing)") func passing() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { exit(EXIT_FAILURE) } if EXIT_SUCCESS != EXIT_FAILURE + 1 { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { exit(EXIT_FAILURE + 1) } } - await #expect(exitsWith: .success) {} - await #expect(exitsWith: .success) { + await #expect(processExitsWith: .success) {} + await #expect(processExitsWith: .success) { exit(EXIT_SUCCESS) } - await #expect(exitsWith: .exitCode(123)) { + await #expect(processExitsWith: .exitCode(123)) { exit(123) } - await #expect(exitsWith: .exitCode(123)) { + await #expect(processExitsWith: .exitCode(123)) { await Task.yield() exit(123) } - await #expect(exitsWith: .signal(SIGSEGV)) { + await #expect(processExitsWith: .signal(SIGSEGV)) { _ = raise(SIGSEGV) // Allow up to 1s for the signal to be delivered. On some platforms, // raise() delivers signals fully asynchronously and may not terminate the @@ -44,7 +44,7 @@ private import _TestingInternals try await Task.sleep(nanoseconds: 1_000_000_000) } } - await #expect(exitsWith: .signal(SIGABRT)) { + await #expect(processExitsWith: .signal(SIGABRT)) { abort() } #if !SWT_NO_UNSTRUCTURED_TASKS @@ -55,7 +55,7 @@ private import _TestingInternals #expect(Test.current != nil) await Task.detached { #expect(Test.current == nil) - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { fatalError() } }.value @@ -91,7 +91,7 @@ private import _TestingInternals return ExitTest.Result(exitStatus: .exitCode(EXIT_SUCCESS)) } await Test { - await #expect(exitsWith: .success) {} + await #expect(processExitsWith: .success) {} }.run(configuration: configuration) // Mock an exit test where the process exits with a particular error code. @@ -99,7 +99,7 @@ private import _TestingInternals return ExitTest.Result(exitStatus: .exitCode(123)) } await Test { - await #expect(exitsWith: .failure) {} + await #expect(processExitsWith: .failure) {} }.run(configuration: configuration) // Mock an exit test where the process exits with a signal. @@ -107,10 +107,10 @@ private import _TestingInternals return ExitTest.Result(exitStatus: .signal(SIGABRT)) } await Test { - await #expect(exitsWith: .signal(SIGABRT)) {} + await #expect(processExitsWith: .signal(SIGABRT)) {} }.run(configuration: configuration) await Test { - await #expect(exitsWith: .failure) {} + await #expect(processExitsWith: .failure) {} }.run(configuration: configuration) } } @@ -129,13 +129,13 @@ private import _TestingInternals return ExitTest.Result(exitStatus: .exitCode(EXIT_SUCCESS)) } await Test { - await #expect(exitsWith: .failure) {} + await #expect(processExitsWith: .failure) {} }.run(configuration: configuration) await Test { - await #expect(exitsWith: .exitCode(EXIT_FAILURE)) {} + await #expect(processExitsWith: .exitCode(EXIT_FAILURE)) {} }.run(configuration: configuration) await Test { - await #expect(exitsWith: .signal(SIGABRT)) {} + await #expect(processExitsWith: .signal(SIGABRT)) {} }.run(configuration: configuration) // Mock exit tests that unexpectedly signalled. @@ -143,13 +143,13 @@ private import _TestingInternals return ExitTest.Result(exitStatus: .signal(SIGABRT)) } await Test { - await #expect(exitsWith: .exitCode(EXIT_SUCCESS)) {} + await #expect(processExitsWith: .exitCode(EXIT_SUCCESS)) {} }.run(configuration: configuration) await Test { - await #expect(exitsWith: .exitCode(EXIT_FAILURE)) {} + await #expect(processExitsWith: .exitCode(EXIT_FAILURE)) {} }.run(configuration: configuration) await Test { - await #expect(exitsWith: .success) {} + await #expect(processExitsWith: .success) {} }.run(configuration: configuration) } } @@ -164,7 +164,7 @@ private import _TestingInternals } await Test { - await #expect(exitsWith: .success) {} + await #expect(processExitsWith: .success) {} }.run(configuration: configuration) } } @@ -186,11 +186,11 @@ private import _TestingInternals configuration.exitTestHandler = ExitTest.handlerForEntryPoint() await Test { - await #expect(exitsWith: .success) { + await #expect(processExitsWith: .success) { #expect(Bool(false), "Something went wrong!") exit(0) } - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { Issue.record(MyError()) } }.run(configuration: configuration) @@ -219,7 +219,7 @@ private import _TestingInternals // // Windows does not have the 8-bit exit code restriction and always reports // the full CInt value back to the testing library. - await #expect(exitsWith: .exitCode(512)) { + await #expect(processExitsWith: .exitCode(512)) { exit(512) } } @@ -232,7 +232,7 @@ private import _TestingInternals @Test("Exit test can be main-actor-isolated") @MainActor func mainActorIsolation() async { - await #expect(exitsWith: .success) { + await #expect(processExitsWith: .success) { await Self.someMainActorFunction() _ = 0 exit(EXIT_SUCCESS) @@ -242,21 +242,21 @@ private import _TestingInternals @Test("Result is set correctly on success") func successfulArtifacts() async throws { // Test that basic passing exit tests produce the correct results (#expect) - var result = await #expect(exitsWith: .success) { + var result = await #expect(processExitsWith: .success) { exit(EXIT_SUCCESS) } #expect(result?.exitStatus == .exitCode(EXIT_SUCCESS)) - result = await #expect(exitsWith: .exitCode(123)) { + result = await #expect(processExitsWith: .exitCode(123)) { exit(123) } #expect(result?.exitStatus == .exitCode(123)) // Test that basic passing exit tests produce the correct results (#require) - result = try await #require(exitsWith: .success) { + result = try await #require(processExitsWith: .success) { exit(EXIT_SUCCESS) } #expect(result?.exitStatus == .exitCode(EXIT_SUCCESS)) - result = try await #require(exitsWith: .exitCode(123)) { + result = try await #require(processExitsWith: .exitCode(123)) { exit(123) } #expect(result?.exitStatus == .exitCode(123)) @@ -282,7 +282,7 @@ private import _TestingInternals } await Test { - let result = await #expect(exitsWith: .success) {} + let result = await #expect(processExitsWith: .success) {} #expect(result == nil) }.run(configuration: configuration) } @@ -305,7 +305,7 @@ private import _TestingInternals } await Test { - try await #require(exitsWith: .success) {} + try await #require(processExitsWith: .success) {} fatalError("Unreachable") }.run(configuration: configuration) } @@ -334,7 +334,7 @@ private import _TestingInternals } await Test { - let result = await #expect(exitsWith: .success) {} + let result = await #expect(processExitsWith: .success) {} #expect(result == nil) }.run(configuration: configuration) } @@ -343,7 +343,7 @@ private import _TestingInternals @Test("Result contains stdout/stderr") func exitTestResultContainsStandardStreams() async throws { - var result = try await #require(exitsWith: .success, observing: [\.standardOutputContent]) { + var result = try await #require(processExitsWith: .success, observing: [\.standardOutputContent]) { try FileHandle.stdout.write("STANDARD OUTPUT") try FileHandle.stderr.write(String("STANDARD ERROR".reversed())) exit(EXIT_SUCCESS) @@ -352,7 +352,7 @@ private import _TestingInternals #expect(result.standardOutputContent.contains("STANDARD OUTPUT".utf8)) #expect(result.standardErrorContent.isEmpty) - result = try await #require(exitsWith: .success, observing: [\.standardErrorContent]) { + result = try await #require(processExitsWith: .success, observing: [\.standardErrorContent]) { try FileHandle.stdout.write("STANDARD OUTPUT") try FileHandle.stderr.write(String("STANDARD ERROR".reversed())) exit(EXIT_SUCCESS) @@ -368,7 +368,7 @@ private import _TestingInternals func nonConstExitCondition() async throws -> ExitTest.Condition { .failure } - await #expect(exitsWith: try await nonConstExitCondition(), sourceLocation: unrelatedSourceLocation) { + await #expect(processExitsWith: try await nonConstExitCondition(), sourceLocation: unrelatedSourceLocation) { fatalError() } } @@ -376,7 +376,7 @@ private import _TestingInternals @Test("ExitTest.current property") func currentProperty() async { #expect((ExitTest.current == nil) as Bool) - await #expect(exitsWith: .success) { + await #expect(processExitsWith: .success) { #expect((ExitTest.current != nil) as Bool) } } @@ -386,7 +386,7 @@ private import _TestingInternals func captureList() async { let i = 123 let s = "abc" as Any - await #expect(exitsWith: .success) { [i = i as Int, s = s as! String, t = (s as Any) as? String?] in + await #expect(processExitsWith: .success) { [i = i as Int, s = s as! String, t = (s as Any) as? String?] in #expect(i == 123) #expect(s == "abc") #expect(t == "abc") @@ -397,7 +397,7 @@ private import _TestingInternals func longCaptureList() async { let count = 1 * 1024 * 1024 let buffer = Array(repeatElement(0 as UInt8, count: count)) - await #expect(exitsWith: .success) { [count = count as Int, buffer = buffer as [UInt8]] in + await #expect(processExitsWith: .success) { [count = count as Int, buffer = buffer as [UInt8]] in #expect(buffer.count == count) } } @@ -407,7 +407,7 @@ private import _TestingInternals @Test("self in capture list") func captureListWithSelf() async { - await #expect(exitsWith: .success) { [self, x = self] in + await #expect(processExitsWith: .success) { [self, x = self] in #expect(self.property == 456) #expect(x.property == 456) } @@ -444,13 +444,13 @@ private import _TestingInternals @Test("Capturing an instance of a subclass") func captureSubclass() async { let instance = CapturableDerivedClass(x: 123) - await #expect(exitsWith: .success) { [instance = instance as CapturableBaseClass] in + await #expect(processExitsWith: .success) { [instance = instance as CapturableBaseClass] in #expect((instance as AnyObject) is CapturableBaseClass) // However, because the static type of `instance` is not Derived, we won't // be able to cast it to Derived. #expect(!((instance as AnyObject) is CapturableDerivedClass)) } - await #expect(exitsWith: .success) { [instance = instance as CapturableDerivedClass] in + await #expect(processExitsWith: .success) { [instance = instance as CapturableDerivedClass] in #expect((instance as AnyObject) is CapturableBaseClass) #expect((instance as AnyObject) is CapturableDerivedClass) #expect(instance.x == 123) @@ -463,28 +463,28 @@ private import _TestingInternals @Suite(.hidden) struct FailingExitTests { @Test(.hidden) func failingExitTests() async { - await #expect(exitsWith: .failure) {} - await #expect(exitsWith: .exitCode(123)) {} - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) {} + await #expect(processExitsWith: .exitCode(123)) {} + await #expect(processExitsWith: .failure) { exit(EXIT_SUCCESS) } - await #expect(exitsWith: .success) { + await #expect(processExitsWith: .success) { exit(EXIT_FAILURE) } - await #expect(exitsWith: .exitCode(123)) { + await #expect(processExitsWith: .exitCode(123)) { exit(0) } - await #expect(exitsWith: .exitCode(SIGABRT)) { + await #expect(processExitsWith: .exitCode(SIGABRT)) { // abort() raises on Windows, but we don't handle that yet and it is // reported as .failure (which will fuzzy-match with SIGABRT.) abort() } - await #expect(exitsWith: .signal(123)) {} - await #expect(exitsWith: .signal(123)) { + await #expect(processExitsWith: .signal(123)) {} + await #expect(processExitsWith: .signal(123)) { exit(123) } - await #expect(exitsWith: .signal(SIGSEGV)) { + await #expect(processExitsWith: .signal(SIGSEGV)) { abort() // sends SIGABRT, not SIGSEGV } } @@ -493,7 +493,7 @@ private import _TestingInternals #if false // intentionally fails to compile @Test(.hidden, arguments: 100 ..< 200) func sellIceCreamCones(count: Int) async throws { - try await #require(exitsWith: .failure) { + try await #require(processExitsWith: .failure) { precondition(count < 10, "Too many ice cream cones") } } diff --git a/Tests/TestingTests/PlanIterationTests.swift b/Tests/TestingTests/PlanIterationTests.swift index 3892b2fb8..21d71f894 100644 --- a/Tests/TestingTests/PlanIterationTests.swift +++ b/Tests/TestingTests/PlanIterationTests.swift @@ -117,11 +117,11 @@ struct PlanIterationTests { #if !SWT_NO_EXIT_TESTS @Test("Iteration count must be positive") func positiveIterationCount() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { var configuration = Configuration() configuration.repetitionPolicy.maximumIterationCount = 0 } - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { var configuration = Configuration() configuration.repetitionPolicy.maximumIterationCount = -1 } diff --git a/Tests/TestingTests/SourceLocationTests.swift b/Tests/TestingTests/SourceLocationTests.swift index 75a8791db..4145687b8 100644 --- a/Tests/TestingTests/SourceLocationTests.swift +++ b/Tests/TestingTests/SourceLocationTests.swift @@ -82,27 +82,27 @@ struct SourceLocationTests { #if !SWT_NO_EXIT_TESTS @Test("SourceLocation.init requires well-formed arguments") func sourceLocationInitPreconditions() async { - await #expect(exitsWith: .failure, "Empty fileID") { + await #expect(processExitsWith: .failure, "Empty fileID") { _ = SourceLocation(fileID: "", filePath: "", line: 1, column: 1) } - await #expect(exitsWith: .failure, "Invalid fileID") { + await #expect(processExitsWith: .failure, "Invalid fileID") { _ = SourceLocation(fileID: "B.swift", filePath: "", line: 1, column: 1) } - await #expect(exitsWith: .failure, "Zero line") { + await #expect(processExitsWith: .failure, "Zero line") { _ = SourceLocation(fileID: "A/B.swift", filePath: "", line: 0, column: 1) } - await #expect(exitsWith: .failure, "Zero column") { + await #expect(processExitsWith: .failure, "Zero column") { _ = SourceLocation(fileID: "A/B.swift", filePath: "", line: 1, column: 0) } } @Test("SourceLocation.fileID property must be well-formed") func sourceLocationFileIDWellFormed() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { var sourceLocation = #_sourceLocation sourceLocation.fileID = "" } - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { var sourceLocation = #_sourceLocation sourceLocation.fileID = "ABC" } @@ -110,11 +110,11 @@ struct SourceLocationTests { @Test("SourceLocation.line and column properties must be positive") func sourceLocationLineAndColumnPositive() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { var sourceLocation = #_sourceLocation sourceLocation.line = -1 } - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { var sourceLocation = #_sourceLocation sourceLocation.column = -1 } diff --git a/Tests/TestingTests/Support/FileHandleTests.swift b/Tests/TestingTests/Support/FileHandleTests.swift index c837ac7cf..4be633ad6 100644 --- a/Tests/TestingTests/Support/FileHandleTests.swift +++ b/Tests/TestingTests/Support/FileHandleTests.swift @@ -85,7 +85,7 @@ struct FileHandleTests { #if !SWT_NO_EXIT_TESTS @Test("Writing requires contiguous storage") func writeIsContiguous() async { - await #expect(exitsWith: .failure) { + await #expect(processExitsWith: .failure) { let fileHandle = try FileHandle.null(mode: "wb") try fileHandle.write([1, 2, 3, 4, 5].lazy.filter { $0 == 1 }) } From 9d007315be8aad73f7d2735655752fadab60d15e Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Wed, 30 Apr 2025 16:55:22 -0500 Subject: [PATCH 17/19] nit: Fix two whitespace issues I saw (tab instead of space indentation) --- Sources/Testing/ExitTests/ExitTest.swift | 2 +- Sources/Testing/SourceAttribution/Expression.swift | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Sources/Testing/ExitTests/ExitTest.swift b/Sources/Testing/ExitTests/ExitTest.swift index 81b40fc12..1e9c29c15 100644 --- a/Sources/Testing/ExitTests/ExitTest.swift +++ b/Sources/Testing/ExitTests/ExitTest.swift @@ -326,7 +326,7 @@ extension ExitTest { /// - Returns: Whether or not an exit test was stored into `outValue`. /// /// - Warning: This function is used to implement the - /// `#expect(processExitsWith:)` macro. Do not use it directly. + /// `#expect(processExitsWith:)` macro. Do not use it directly. public static func __store( _ id: (UInt64, UInt64, UInt64, UInt64), _ body: @escaping @Sendable (repeat each T) async throws -> Void, diff --git a/Sources/Testing/SourceAttribution/Expression.swift b/Sources/Testing/SourceAttribution/Expression.swift index b96d227ac..a294a81e0 100644 --- a/Sources/Testing/SourceAttribution/Expression.swift +++ b/Sources/Testing/SourceAttribution/Expression.swift @@ -23,7 +23,7 @@ /// ``` /// /// - Warning: This type is used to implement the `#expect()` macro. Do not use -/// it directly. Tools can use the SPI ``Expression`` typealias if needed. +/// it directly. Tools can use the SPI ``Expression`` typealias if needed. public struct __Expression: Sendable { /// An enumeration describing the various kinds of expression that can be /// captured. From badca1ef1870992c1581dcde036058cf532c4e8d Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Wed, 30 Apr 2025 16:55:55 -0500 Subject: [PATCH 18/19] Add documentation for ExitTest/Condition/init(_:) --- Sources/Testing/ExitTests/ExitTest.Condition.swift | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Sources/Testing/ExitTests/ExitTest.Condition.swift b/Sources/Testing/ExitTests/ExitTest.Condition.swift index 1b89dc120..f737d8cf6 100644 --- a/Sources/Testing/ExitTests/ExitTest.Condition.swift +++ b/Sources/Testing/ExitTests/ExitTest.Condition.swift @@ -83,6 +83,12 @@ extension ExitTest.Condition { Self(_kind: .failure) } + /// Initialize an instance of this type that matches the specified exit + /// status. + /// + /// - Parameters: + /// - exitStatus: The particular exit status this condition should match. + /// /// @Metadata { /// @Available(Swift, introduced: 6.2) /// } From 9f64079622b5012c4995790fa10cd3de53d03239 Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Thu, 1 May 2025 10:29:56 -0500 Subject: [PATCH 19/19] Add note about exit tests not being able to be nested in another exit test --- Sources/Testing/Testing.docc/exit-testing.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Sources/Testing/Testing.docc/exit-testing.md b/Sources/Testing/Testing.docc/exit-testing.md index a55db2940..06ab53dc9 100644 --- a/Sources/Testing/Testing.docc/exit-testing.md +++ b/Sources/Testing/Testing.docc/exit-testing.md @@ -60,6 +60,8 @@ exit test. When an exit test is performed at runtime, the testing library starts a new process with the same executable as the current process. The current task is then suspended (as with `await`) and waits for the child process to exit. +- Note: An exit test cannot run within another exit test. + The parent process doesn't call the body of the exit test. Instead, the child process treats the body of the exit test as its `main()` function and calls it directly.