Use local git package for code gen

[kimdv]

I'm not sure if it need to be enabled in the CI or if I need to do something in the build-script.py

Tested locally by adding some white space in the template and got

** Running code generation **
Building for debugging...
[11/11] Linking generate-swiftsyntax
Build complete! (4.33s)
** Verifing code generated files **
diff --recursive --exclude .* --context=0 /var/folders/w7/s0l8n0b56dg5jnl6r9w6tl380000gn/T/tmp7yc_cvx1/SwiftBasicFormat/generated/BasicFormat.swift /Users/kimdevos/repository/kimdv/swift-src/swift-syntax/Sources/SwiftBasicFormat/generated/BasicFormat.swift
*** /var/folders/w7/s0l8n0b56dg5jnl6r9w6tl380000gn/T/tmp7yc_cvx1/SwiftBasicFormat/generated/BasicFormat.swift	Fri Apr 14 22:52:45 2023
--- /Users/kimdevos/repository/kimdv/swift-src/swift-syntax/Sources/SwiftBasicFormat/generated/BasicFormat.swift	Fri Apr 14 22:46:31 2023
***************
*** 63,67 ****
- 
- 
- 
- 
- 
--- 62 ----
FAIL: code-generated files committed to repository do not match generated ones. Please re-generate the code-generated-files using the following command, open a PR to the SwiftSyntax project and merge it alongside the main PR.$ swift-syntax/build-script.py generate-source-code --toolchain /path/to/toolchain.xctoolchain/usr

[kimdv]

@swift-ci plese test

[ahoppen]

The problem is that CI checks out swift-syntax with --depth 1, so it doesn’t contain any git history 😬

[kimdv]

Oh shot... 🙈

Se discussed something in a PR before.
Can't find it. But you pointed out a possible solution. Do you remember what it was?

[ahoppen]

I thought that we could reference the git repository and use HEAD as the branch name but unfortunately that doesn’t work either because SwiftPM doesn’t accept HEAD as a branch name.

And I have a horrible idea. Inside Package.swift, we could invoke git to find the current HEAD commit and then depend on that. Here’s a code snippet that I just tried and it seems to work. Of course, we would need some more proper error handling and I think if we fail to retrieve the git commit, we could fall back to just using a local dependency.

One thing that I haven’t figured out yet is how to launch git from PATH without hard-coding its path. I’m not even sure if that’s possible… But then, if every/most clients have git installed at /usr/bin/git and we have a decent fallback, maybe it’s fine, I’m not sure yet.

Do you want to give it a try?

import Foundation

let stdout = Pipe()
let stderr = Pipe()
var data = Data()

let process = try! Process()
process.executableURL = URL(fileURLWithPath: "/usr/bin/git")
process.arguments = ["-C", "/Users/alex/syntax", "rev-parse", "HEAD"]
process.standardOutput = stdout
stdout.fileHandleForReading.readabilityHandler = { handle in
  data.append(handle.availableData)
}
stderr.fileHandleForReading.readabilityHandler = { _ in /* ignored */ }

try process.run()
process.waitUntilExit()

let commit = String(decoding: data, as: UTF8.self)

[kimdv]

After some thinking I tought that just referring to the local path would give me the same?
HEAD will be the latest commit on the current branch. So instead of making your example above I would just refer to the local folder.

When referring to the local folder there will be a possibility to generate some code that will break code generation.
We moved away of that reason before because there was many annoying changes that did break.
But I think that the code generation is pretty stable now, so I think it's a fair trade of to use the local path for swift-syntax.

What do you think?
Right now I'm referring to a commit, but that doesn't work good as generating locally might have a different result than the CI.

[ahoppen]

After some thinking I tought that just referring to the local path would give me the same? HEAD will be the latest commit on the current branch. So instead of making your example above I would just refer to the local folder.

There is a subtle difference here. While you’re developing locally, your HEAD will not have any changes in your working copy, so even if you mess up your CodeGeneration locally, you will still be able to run it because it references the last commit that you’ve made. At least that has been my thinking.

[kimdv]

After some thinking I tought that just referring to the local path would give me the same? HEAD will be the latest commit on the current branch. So instead of making your example above I would just refer to the local folder.

There is a subtle difference here. While you’re developing locally, your HEAD will not have any changes in your working copy, so even if you mess up your CodeGeneration locally, you will still be able to run it because it references the last commit that you’ve made. At least that has been my thinking.

I see. I'll try to come up with something 😁

[kimdv]

@swift-ci please test

[kimdv]

@ahoppen i tried to use HEAD as revision, and that worked.
I triggered the CI. But I'm not sure if it verifies the generates code. As there was code gen changes locally

[ahoppen]

Oh, that’s beautiful. I wonder what I did wrong to not get it to work. Maybe I was using branch instead of revision 🤔

I don’t think this will affect CI in any way right now since we aren’t building CodeGeneration in CI at all. You’d need to add that flag to https://github.com/apple/swift/blob/main/utils/swift_build_support/swift_build_support/products/swiftsyntax.py

[kimdv]

swiftlang/swift#65439

@swift-ci please test macOS

[kimdv]

Got an error 🥳

FAIL: code-generated files committed to repository do not match generated ones. Please re-generate the code-generated-files using the following command, open a PR to the SwiftSyntax project and merge it alongside the main PR.$ swift-syntax/build-script.py generate-source-code --toolchain /path/to/toolchain.xctoolchain/usr
** Running code generation **
/Users/ec2-user/jenkins/workspace/swift-syntax-PR-macOS/branch-main/build/buildbot_incremental/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift run --package-path /Users/ec2-user/jenkins/workspace/swift-syntax-PR-macOS/branch-main/swift-syntax/CodeGeneration generate-swiftsyntax /var/folders/7n/r31lzfjx6556jgc6vg55_cg80000gn/T/tmpgq9a2alc --verbose
** Verifing code generated files **

[kimdv]

swiftlang/swift#65439

@swift-ci please test

[kimdv]

swiftlang/swift#65439

@swift-ci please test

[kimdv]

@swift-ci please test windows

[kimdv]

swiftlang/swift#65439

@swift-ci please test Linux

[ahoppen]

Wohooo. This is great 🎉 Could you add a sentence of two of how CodeGeneration references to CodeGeneration/README.md?

[kimdv]

@swift-ci please test

[kimdv]

@swift-ci please test windows