Improve releases

[rjmholt]

Meta issue for release improvements

Automation

• Improve the changelog generation script to create changelogs in the correct format.
• Work out a single place to set the extension version number (across client and server).
• Improve VSIX publication - currently the build produces a VSIX, but vsce only wants to publish from
  a code base, meaning the VSIX needs to be decompressed to publish it (maybe I missed something here, I honestly can't understand why this is the case)
• Introduce testing on the VSIX artefact:
  • Have a list of files that must be signed and verify the signatures on those files
  • Crack open the VSIX and run the test suite on it
  • Ensure the VSIX can be installed
• The ultimate ideal is to have a "prepare release" script that takes a version number as argument, and goes through and sets the version numbers and updates the changelog.

Another possibility here is to have the changelog updated with every PR, as a merge requirement. Then to release we just add a header...

Process

We should consider a document that describes the step-by-step process we follow for builds, not just in terms of "how to build", but also what our strategy is for release/version management and what our "contract" (too strong a word) is for things like feature-freezing a release.

The ideal release process is:

• Every commit to master is built as a potential release
• Actual releases are just branched off a selected commit with version numbers set at build time
  (i.e. normal commit builds are e.g. 1.8.1-preview, but when selected for release drop the preview)

There may still be a little bit of work to do on this, so an intermediate process might be:

• Select a commit as "feature-complete" for the coming release and branch it from master
• Move it to VSTS (and merge any needed additions there as PRs/cherry-picks)
• Build and prepare the release publication
• On release date, release the assets built
• Update GitHub with any VSTS commits, then release on GitHub

[rjmholt]

/cc @tylerl0706 @rkeithhill @SeeminglyScience

[TylerLeonhardt]

Improve the changelog generation script to create changelogs in the correct format.

What is the correct format?

Work out a single place to set the extension version number (across client and server).

YES. I bookmarked this a while ago and really want to mess with it:
https://github.com/semantic-release/semantic-release

Improve VSIX publication - currently the build produces a VSIX...

Well... I just stumbled upon this so I guess we COULD release from the code base instead of the vsix:
microsoft/vscode-vsce#191

but this tells me that you should be able to specify a vsix already...
microsoft/vscode-vsce#146

Introduce testing on the VSIX artefact

NEED.

The ultimate ideal is to have a "prepare release" script that takes a version number as argument, and goes through and sets the version numbers and updates the changelog.

This should be a part of the Package task or with a new Release task in the InvokeBuild script.

Another possibility here is to have the changelog updated with every PR, as a merge requirement. Then to release we just add a header...

I'm not too sure about this.... the merge conflicts would be really annoying. Pretty sure PowerShell does a (better) version of what we're doing now.

[rjmholt]

What is the correct format?

The script currently produces output like:

- Clean up uses of `CommandTypes.Workflow` and `WorkflowInfo` (#6708)

Which is what the PowerShell repo publishes. But our changelog format has historically been:

- [PowerShell/vscode-powershell #1327](https://github.com/PowerShell/vscode-powershell/pull/1327) -
  Switch to named pipes

So writing changelog updates for us means doing some manual stuff at the moment.

[rjmholt]

but this tells me that you should be able to specify a vsix already...
microsoft/vscode-vsce#146

Ah! I couldn't find that before. Good to know. Ideally we should record or even script it somewhere

[rjmholt]

I'm not too sure about this.... the merge conflicts would be really annoying. Pretty sure PowerShell does a (better) version of what we're doing now.

Yeah the conflicts are a problem. But trying to go back through history to work out what the changes were seems like a second-class approach -- the person implementing the PR knows what the changes were best.

I think the commit/PR title+description are supposed to be the true record of the change. So perhaps the script should use the commit message as it does currently, but as maintainers (@rkeithhill this is a good time to tag you 😄) we should enforce the policy that the squashed commit message (which comes from the PR title - so the PR title) should be a description of the change for the changelog?