Migrate build script to Swift

[kimdv]

Fixes #1465

[kimdv]

Open PR to show the progress.

I'm already open for feedback but it's still WIP

[ahoppen]

One high-level comment: I think the build script should live in the CodeGeneration package (which would then obviously need to be renamed). That way we could also remove the generate-swiftsyntax executable and instead invoke the source generation code directly from the rewritten build-script.

I would like to keep swift-parser-cli separate from the build script because they serve to drastically different purposes. The build script’s purpose is to build SwiftSyntax in CI and provide the glue code with the build script to the build-script in the compiler repo. As such, it’s only interesting during build time. swift-parser-cli is a runtime tool that can be super useful to inspect SwiftSyntax trees and e.g. triage bugs. For example, if someone is hitting a SwiftSyntax crash but can’t share their source code, you might suggest that they run swift-parser-cli reduce and just attach the reduced source code (which with all likelihood contains no sensitive information). As such, I see swift-parser-cli also useful for adopters of SwiftSyntax.

[kimdv]

One high-level comment: I think the build script should live in the CodeGeneration package (which would then obviously need to be renamed). That way we could also remove the generate-swiftsyntax executable and instead invoke the source generation code directly from the rewritten build-script.

Cool. I'll try to merge CodeGeneration package and this one into a single Package.

I would like to keep swift-parser-cli separate from the build script because they serve to drastically different purposes. The build script’s purpose is to build SwiftSyntax in CI and provide the glue code with the build script to the build-script in the compiler repo. As such, it’s only interesting during build time. swift-parser-cli is a runtime tool that can be super useful to inspect SwiftSyntax trees and e.g. triage bugs. For example, if someone is hitting a SwiftSyntax crash but can’t share their source code, you might suggest that they run swift-parser-cli reduce and just attach the reduced source code (which with all likelihood contains no sensitive information). As such, I see swift-parser-cli also useful for adopters of SwiftSyntax.

Just to be clear, I would revert the commit or will you still move it out of the swift-syntax package but just in it's own package? (As now. It is it's own executable)
My tought was to move it out, so the swift-syntax package don't have any dependency on swift-argument-parser.

[ahoppen]

Just to be clear, I would revert the commit or will you still move it out of the swift-syntax package but just in it's own package? (As now. It is it's own executable) My tought was to move it out, so the swift-syntax package don't have any dependency on swift-argument-parser.

Yes, I think moving swift-parser-cli to a different package does make sense. I was just thrown off by the PR title and thought that swift-parser-cli would be the vehicle for migrating build-script and that’s definitely not what we want.

[kimdv]

Yes, I think moving swift-parser-cli to a different package does make sense. I was just thrown off by the PR title and thought that swift-parser-cli would be the vehicle for migrating build-script and that’s definitely not what we want.

Will remove this PR so it focus on migrate the build script only then

[ahoppen]

I think a separate PR for moving swift-parser-cli would be great, but that’s a different topic.

[kimdv]

The CLI look like

OVERVIEW: Build and test script for SwiftSyntax.

The build script can also drive the test suite included in the SwiftSyntax
repo. This requires a custom build of the compiler project since it accesses
test utilities that are not shipped as part of the toolchains. See the Testing
section for arguments that need to be specified for this.

USAGE: swift-syntax-cli <subcommand>

OPTIONS:
  -h, --help              Show help information.

SUBCOMMANDS:
  build
  generate-source-code
  test
  verify-source-code

  See 'swift-syntax-cli help <subcommand>' for detailed help.

Build help

USAGE: swift-syntax-cli build <toolchain> [<build-dir>] [<multiroot-data-file>] [--disable-sandbox] [--release] [--enable-rawsyntax-validation] [--enable-test-fuzzing] [--verbose]

ARGUMENTS:
  <toolchain>             The path to the toolchain that shall be used to build SwiftSyntax.
  <build-dir>             The directory in which build products shall be put. If omitted
                          a directory named ".build" will be put in the swift-syntax
                          directory. (default: .build)
  <multiroot-data-file>   Path to an Xcode workspace to create a unified build of SwiftSyntax with
                          other projects.

OPTIONS:
  --disable-sandbox       Disable sandboxes when building with SwiftPM
  --release               Build in release mode.
  --enable-rawsyntax-validation
                          When constructing RawSyntax nodes validate that their layout matches that
                          defined in `CodeGeneration` and that TokenSyntax nodes have a `tokenKind`
                          matching the ones specified in `CodeGeneration`.
  --enable-test-fuzzing   For each `assertParse` test, perform mutations of the test case based on
                          alternate token choices that the parser checks, validating that there are
                          no round-trip or assertion failures.
  --verbose               Enable verbose logging.
  -h, --help              Show help information.

Generate source code help

USAGE: swift-syntax-cli generate-source-code <destination> [--verbose]

ARGUMENTS:
  <destination>           The path to the source directory (i.e. 'swift-syntax/Sources') where the source files are to be generated

OPTIONS:
  --verbose               Enable verbose logging.
  -h, --help              Show help information.

Test help

USAGE: swift-syntax-cli test <toolchain> [<build-dir>] [<multiroot-data-file>] [--disable-sandbox] [--release] [--enable-rawsyntax-validation] [--enable-test-fuzzing] [--verbose] [--skip-lit-tests] [<filecheck-exec>]

ARGUMENTS:
  <toolchain>             The path to the toolchain that shall be used to build SwiftSyntax.
  <build-dir>             The directory in which build products shall be put. If omitted
                          a directory named ".build" will be put in the swift-syntax
                          directory. (default: .build)
  <multiroot-data-file>   Path to an Xcode workspace to create a unified build of SwiftSyntax with
                          other projects.
  <filecheck-exec>        Path to the FileCheck executable that was built as part of the LLVM repository.
                          If not specified, it will be looked up from PATH.

OPTIONS:
  --disable-sandbox       Disable sandboxes when building with SwiftPM
  --release               Build in release mode.
  --enable-rawsyntax-validation
                          When constructing RawSyntax nodes validate that their layout matches that
                          defined in `CodeGeneration` and that TokenSyntax nodes have a `tokenKind`
                          matching the ones specified in `CodeGeneration`.
  --enable-test-fuzzing   For each `assertParse` test, perform mutations of the test case based on
                          alternate token choices that the parser checks, validating that there are
                          no round-trip or assertion failures.
  --verbose               Enable verbose logging.
  --skip-lit-tests        Don't run lit-based tests
  -h, --help              Show help information.

Generate source code help

USAGE: swift-syntax-cli verify-source-code <destination> [--verbose]

ARGUMENTS:
  <destination>           The path to the source directory (i.e. 'swift-syntax/Sources') where the source files are to be generated

OPTIONS:
  --verbose               Enable verbose logging.
  -h, --help              Show help information.

[ahoppen]

Thanks for doing this, @kimdv. I think the Swift implementation is really so much easier to read and work with. I’ve got a few comments though. Just to warn you, once you’ve addressed those, I might come up with another big-ish set of comments, but we’re definitely heading in the right direction.

[ahoppen]

One thought just came to me: With the current design, you won't be able to build the build script (or swift-Syntax-cli or whatever the name ends up being) if SwiftSyntax doesn't build itself, because the build script depends on SwiftSyntax. I think that's quite unfortunate and very surprising.

With that in mind, I think CodeGeneration will need to continue being its own package so that the build script itself can continue to be free of a SwiftSyntax dependency.

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[ahoppen]

To debug the latest failure (and in general), I think it would be good if swift-syntax-dev-utils would print the commands for all subprocesses it runs if it has --verbose, similar to

https://github.com/apple/swift-syntax/blob/861333cd66fa80e6dc0d0d11d0c05e218418e1b8/build-script.py#L95-L96C22

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[ahoppen]

🥳 macOS is passing. Linux can’t be far

[kimdv]

swiftlang/swift#66573

@swift-ci please test

[kimdv]

swiftlang/swift#66573

@swift-ci please test windows

[ahoppen]

swiftlang/swift#66573

@swift-ci please test windows